Marine Data Literacy 2.0

Providing instruction for managing, converting, analyzing and displaying oceanographic station data, marine meteorological data, GIS-compatible marine and coastal data or model simulations, and mapped remote sensing imagery

 

 

 

 

Home > 5. Gridded Data > 5.9 Faulty FillValues

5.9 Correcting Faulty FillValue (i.e. Blank) Declarations in NetCDF Grids

  • Exercise Title:  Correcting Faulty FillValue (i.e. Blank) Declarations in NetCDF Grids

  • Abstract:  From time to time you will encounter NetCDF grids that visualize (in Saga, or IDV or in ncBrowse, for examples) with areas expected to be blanked actually being shown (usually with odd colors that indicate extreme values.  This is an indicated that the special value in the file used for blanking (the "fillvalue") has not been properly recognized by the displaying program.  While rare, this situation can occur at any time; experience indicates it is common with new web products that have not been widely tested by users.  The method below shows how this can be fixed.  Setting up for the procedure is a lot more trouble than the actual fix, which only takes a minute; but once you are set up, you can perform the fix again very quickly.

  • Preliminary Reading (in OceanTeacher, unless otherwise indicated):

    • N/A

  • Required Software:

    • ncBrowse - Widely used browse/view tool for NetCDF files; for quick inspection and checking variable names and file declarations

    • ncatedd.exe - NCO (see below) editor for NetCDF files

  • Other Resources: 

    • NetCDF Operators (NCO) - "NCO manipulates data stored in netCDF format.  A dozen standalone, command-line programs that take netCDF files as input, then operate (e.g., derive new data, average, print, hyperslab, manipulate metadata) and output the results to screen or files in text, binary, or netCDF formats." [From the website]

      • ncatted netCDF ATTribute EDitor - Link to ncatted documentation within NCO.

      • "Releases receive unique CVS tags and their tarballs are stored here on SourceForge" - Long list of NCO releases, including source code that must be compiled and executable versions for Windows

      • nco-4.3.4.windows.mvs.exe - Latest installation package of NCO for Windows; located in the SourceForge list about, half-way down.

      • sssal_20120712_global_caps_aquarius_l3.nc - Example NetCDF file of surface salinity from early stages of the Aquarius program, with an apparent fillvalue problem.  Aquarius files were fixed immediately and this issue went away, but the file was saved by the author to illustrate the correction procedure described below.  New NetCDF products from various programs sometimes include this same problem, hence the need to keep this exercise handy.  In most cases you should inform the publisher, so files can be fixed and this method made unnecessary.

    • BAT or Batch Files - Wikipedia articles

  • Author:  Murray Brown

  • Version:  9-9-13

http://wiht.link/NetCDF-resources

1.  If you open the above NetCDF grid in Saga (using MODULES > IMPORT/EXPORT - GDAL/OGR > IMPORT RASTER, this is what you see (using the rainbow preset color palette).

The expected salinity values are all crowded into the very lowest color value, and a large part of the globe is taken up by an extremely large value, >32000.

This situation is typical of a faulty fillevalue in NetCDF.  The value is being displayed by Saga, instead of being ignored, which is what we usually expect.

2.  Here we've opened the same grid in ncBrowse (top half of image) and selected the CAP_SSS variable for display (lower half of image).  CAP_SSS is identifed in the lower half as the global data grid of salinity values, a piece of information needed below.
3.  When you click GRAPH VARIABLE, you'll see this very similar image.
4.  But what is going wrong?  If you look in Panel 2, you'll see this is how the fillvalue is "declared" by the NetCDF file:
  • :FillValue = 32767.0f; // float

After some long experience with NetCDF, you'll learn that the correct syntax for the declaration is different by only a single character, as you see here:

  • :_FillValue = 32767.0f; // float

 

5.  The lack of the underscore character after the colon makes the fillvalue declaration useless.  We need to fix that.  To do the job we'll use a powerful set of NetCDF tools, called the NetCDF Operators (NCO).  There are a dozen tools, of which we'll only use one.  But you should consider keeping all of these tools on your computer if you intend to do much work with operational data in NetCDF format.
6.  Open the NCO website, and take a few minutes to read over the main materials.
7.  If you scroll down the same page, you'll find this list of the separate NCO tools -- we're going to use ncatted for our work below.

NOTE:  You can't get the EXE file here.

What is NCO?

8.  But you can click on the ncatted link to take you to a long documentation page for all 12 programs

Here is an excerpt from the documentation for ncatted.  It shows you how to write a BAT file that runs ncatted.

BAT files will be explained below.

SYNTAX

  • ncatted [-a att_dsc] [-a ...] [-D dbg] [-h] [--hdr_pad nbr] [-l path] [-O] [-o output-file] [-p path] [-R] [-r] [--ram_all]
         input-file [[output-file]]
9.  Experience with ncatted has shown that correcting the fillvalue mistake only takes this smaller part of the complete syntax for ncatted.  So we can ignore most of the options you see above in Panel 8.
  • ncatted [-a att_dsc] input-file [[output-file]]

10.  Further down the documentation page for ncatted you see that the argument att_dsc, above, actually consists of 5 parts, defined here.

There should be no empty spaces between these five consecutive arguments.

  • att_dsc = att_nm, var_nm, mode, att_type, att_val

    • att_nm - Attribute name. Example: units

    • var_nm - Variable name. Regular expressions are accepted and will select any matching variable names. Example: pressure, '^H2O'

    • mode - Edit mode abbreviation. Example: a. Complete listing included in the NCO docs

    • att_type - Attribute type abbreviation.  Example: c. Complete listing included in the NCO docs

    • att_val - Attribute value. Example: 32767

 

11.  First we'll get the ncatted software, and then we'll use the above information to write a BAT file that does the job.
12.  Look again at the main NCO page you opened in Panel 6.  About halfway down the page, you'll find this information about obtaining the software.

Open the SOURCEFORGE link.

Version Summary

Releases receive unique CVS tags and their tarballs are stored here on SourceForge and here on our server at UCI. You can also retrieve the source of tagged versions directly with CVS.

13.  Viewing the SourceForge list, with files sorted by name, you'll see a file named nco-4.3.4.windows.mvs.exe, the latest version [possibly something later, in which case you should use it instead].

Save the 4.3.4 file to a convenient place on your computer.

14.  Run the 4.3.4 installation file to install the NCO software in the default location, usually C:\NCO.
15.  Now you have a permanent collection of the NCO utility programs, as listed here.
16.  Make a working copy of the ncatted.exe file and place it in a working folder with the erroneously written NetCDF salinity file.

Create a new text file, in the same folder, named ncatted.txt and rename it ncatted.bat.

17.  Open the BAT file in any ASCII editor and write this text.  It is colored here to show the separate sections:
  • ncatted  -a  _FillValue,CAP_SSS,o,f,32767  sssal_20120712_global_caps_aquarius_l3.nc  sssal_20120712_global_caps_aquarius_l3_blanked.nc

This is exactly the same structure as you see above in Panel 9, but the items all now have their real-world values.  There are 3 spaces, and only 3.  Notice that the final product has the same filename as the input, with the addition of the token "_blanked" so we'll know that it has been fixed.

18.  Now run the ncatted.exe file by opening the BAT file.  Watch with Windows Explorer to see the new file created in the same folder.  If you have any problems, then check again to make sure your BAT file is correctly written.  If you make changes in the BAT file, just run it again to write over any previous attempts.
19.  And here is the final blanked file, loaded into ncBrowse for display.  Notice that the old (incorrect) fillvalue declaration is still there, alongside the new correct one.
20.  And if you click on GRAPH VARIABLE, here is the clean image.  Notice the realistic value range along the bottom.
21.  And here we're loading the new grid into Saga, using the settings shown here.
22.  With the rainbow preset colors palette, this is what the final product looks like.
23.  Make sure to save any work in Saga with an appropriate filename and in the PRODUCTS > SAGA > GRIDS folder.
24.  Now it would be easy to correct any other file, simply by editing the BAT file to have  the desired incoming and outgoing filenames.  You can do this correction work in the same "work folder" you used above, and transfer the blanked NetCDF files back to their proper locations.  Don't keep the older, incorrect versions unless you have a particular reason.