Correcting Faulty FillValue (i.e. Blank) Declarations in NetCDF Grids
time to time you will encounter NetCDF grids that visualize (in Saga, or
IDV or in ncBrowse, for examples) with areas expected to be blanked
actually being shown (usually with odd colors that indicate extreme
values. This is an indicated that the special value in the file
used for blanking (the "fillvalue") has not been properly
recognized by the displaying program. While rare, this situation
can occur at any time; experience indicates it is common with new web
products that have not been widely tested by users. The method
below shows how this can be fixed. Setting up for the procedure is
a lot more trouble than the actual fix, which only takes a minute; but
once you are set up, you can perform the fix again very quickly.
Preliminary Reading (in
OceanTeacher, unless otherwise indicated):
(NCO) - "NCO manipulates data stored in
A dozen standalone, command-line programs that take
as input, then operate (e.g., derive new data, average, print, hyperslab,
manipulate metadata) and output the results to screen or files in text,
binary, or netCDF formats." [From the website]
ncatted netCDF ATTribute EDitor - Link to ncatted documentation
"Releases receive unique CVS
tags and their tarballs are stored
on SourceForge" - Long list of NCO releases, including source code that
must be compiled and executable versions for Windows
- Latest installation package of NCO for Windows; located in the
SourceForge list about, half-way down.
sssal_20120712_global_caps_aquarius_l3.nc - Example NetCDF file
of surface salinity
from early stages of the Aquarius program, with an apparent fillvalue
problem. Aquarius files were fixed immediately and this issue went
away, but the file was saved by the author to illustrate the correction
procedure described below. New NetCDF products from various
programs sometimes include this same problem, hence the need to keep
this exercise handy. In most cases you should inform the
publisher, so files can be fixed and this method made unnecessary.
Batch Files - Wikipedia articles
|1. If you open the above
NetCDF grid in Saga (using MODULES > IMPORT/EXPORT - GDAL/OGR > IMPORT
RASTER, this is what you see (using the rainbow preset color palette).
expected salinity values are all crowded into the very lowest color value,
and a large part of the globe is taken up by an extremely large value,
This situation is typical of a faulty fillevalue in NetCDF. The
value is being displayed by Saga, instead of being ignored, which is what we
|2. Here we've opened the same
grid in ncBrowse (top half of image) and selected the CAP_SSS variable for
display (lower half of image). CAP_SSS is identifed in the lower half
as the global data grid of salinity values, a piece of information needed
|3. When you click GRAPH
VARIABLE, you'll see this very similar image.
|4. But what is
going wrong? If you look in Panel 2, you'll see this is how the
fillvalue is "declared" by the NetCDF file:
- :FillValue = 32767.0f; // float
After some long experience with NetCDF, you'll learn that the correct
syntax for the declaration is different by only a single character, as you
- :_FillValue = 32767.0f; // float
|5. The lack of
the underscore character after the colon makes the fillvalue declaration
useless. We need to fix that. To do the job we'll use a powerful
set of NetCDF tools, called the NetCDF Operators (NCO). There are a
dozen tools, of which we'll only use one. But you should consider
keeping all of these tools on your computer if you intend to do much work
with operational data in NetCDF format.
|6. Open the NCO website, and
take a few minutes to read over the main materials.
|7. If you scroll down the same page,
you'll find this list of the separate NCO tools -- we're going to use
ncatted for our work below.
NOTE: You can't get the EXE file
What is NCO?
|8. But you can click on the
ncatted link to take you to a long documentation page for all 12
Here is an excerpt from the documentation for ncatted.
It shows you how to write a BAT file that runs ncatted.
BAT files will be explained below.
|9. Experience with ncatted
has shown that correcting the fillvalue mistake only takes this smaller part
of the complete syntax for ncatted. So we can ignore most of
the options you see above in Panel 8.
|10. Further down the
documentation page for ncatted you see that the argument att_dsc,
above, actually consists of 5 parts, defined here.
There should be no empty spaces
between these five consecutive arguments.
att_dsc = att_nm, var_nm,
mode, att_type, att_val
att_nm - Attribute name. Example: units
var_nm - Variable name. Regular expressions
are accepted and will select any matching variable names. Example:
mode - Edit mode abbreviation. Example: a.
Complete listing included in the NCO docs
att_type - Attribute type abbreviation.
Example: c. Complete listing included in the NCO docs
att_val - Attribute value. Example: 32767
|11. First we'll
get the ncatted software, and then we'll use the above information to
write a BAT file that does the job.
|12. Look again at the main
NCO page you opened in Panel 6. About halfway down the page, you'll
find this information about obtaining the software.
Open the SOURCEFORGE
Releases receive unique CVS tags and their tarballs are
stored here on SourceForge and
here on our server at UCI. You
can also retrieve the source of tagged versions directly
|13. Viewing the SourceForge
list, with files sorted by name, you'll see a file named
nco-4.3.4.windows.mvs.exe, the latest version [possibly something later, in
which case you should use it instead].
Save the 4.3.4 file to a convenient
place on your computer.
|14. Run the 4.3.4
installation file to install the NCO software in the default location,
|15. Now you have a permanent
collection of the NCO utility programs, as listed here.
|16. Make a working copy of
the ncatted.exe file and place it in a working folder with the
erroneously written NetCDF salinity file.
Create a new text file, in the
same folder, named ncatted.txt and rename it ncatted.bat.
|17. Open the BAT
file in any ASCII editor and write this text. It is colored here to
show the separate sections:
- ncatted -a
This is exactly the same structure as you see above in Panel 9, but the
items all now have their real-world values. There are 3 spaces,
and only 3. Notice that the final product has the same
filename as the input, with the addition of the token "_blanked" so we'll
know that it has been fixed.
|18. Now run the
ncatted.exe file by opening the BAT file. Watch with Windows
Explorer to see the new file created in the same folder. If you have
any problems, then check again to make sure your BAT file is correctly
written. If you make changes in the BAT file, just run it again to
write over any previous attempts.
|19. And here is the final
blanked file, loaded into ncBrowse for display. Notice that the old
(incorrect) fillvalue declaration is still there, alongside the new correct
|20. And if you click on GRAPH
VARIABLE, here is the clean image. Notice the realistic value range
along the bottom.
|21. And here we're loading
the new grid into Saga, using the settings shown here.
|22. With the rainbow preset
colors palette, this is what the final product looks like.
|23. Make sure to
save any work in Saga with an appropriate filename and in the PRODUCTS >
SAGA > GRIDS folder.
|24. Now it would
be easy to correct any other file, simply by editing the BAT file to have
the desired incoming and outgoing filenames. You can do this
correction work in the same "work folder" you used above, and transfer the
blanked NetCDF files back to their proper locations. Don't keep the
older, incorrect versions unless you have a particular reason.