Marine Data Literacy 2.0

Providing instruction for managing, converting, analyzing and displaying oceanographic station data, marine meteorological data, GIS-compatible marine and coastal data or model simulations, and mapped remote sensing imagery

 

 

 

 

Home > 5. Gridded Data > 5.14 NetCDF/HDF/GRIB

5.14 Managing NetCDF, HDF4/5 and GRIB Rasters with Principal Display Programs

Software Used Below:    Saga    ncBrowse    Integrated Data Viewer (IDV)  

 

1.  This extremely basic overview table has been written by the author to summarize, in a very incomplete and crude way, the current relationships between the principal raster formats (except ESRI's Arc Grid) and the 3 most widely used grid viewing programs in MDL.  The only really safe comment is that specific compatibilities are suggested by this table, but you must always test any specific file/software combination to see if it really "goes in."

FORMATS Saga1 ncBrowse Integrated Data Viewer (IDV) Comment
NetCDF Easy loading and display, in most cases Easy to use, via OPENDAP,  THREDDS or local files Usually very easily displayed on all platforms; European resources often have problems with blanking, orientation, grid location, etc.
HDF42

Completely incompatible

Latest format specifications effectively "merge" HDF and NC; files can be labeled HDF or NC externally and/or internally, sometimes oppositely.3
HDF52
 GRIB2 Usually no problem, but check everything

 

Completely incompatible

 
Usually no problem, but check everything Oceanographers are becoming more aware of GRIB, and it may see wider use due to extremely robust documentation and widespread use in the meteorology community.
  1. Saga is now completely compatible with NetCDF
  2. As long as the HDF4 file is supplied with latitudes and longitudes -- sometimes explicitly listed as fields and sometimes concealed -- the files can be labeled NC or HDF, interchangeably, but still loaded with equal ease into Saga via the GDAL Tool.  Fewer tests with HDF5 indicate this also applies.
  3. This short exercise does not cover the further complications added by the EOS variants to HDF; curious users should set aside a big chunk of their lives to try to learn what this is rally all about.  Best of luck to you.

2.  Before you begin anything, download the above 4 example files to DATA > OCEAN > EXAMPLES. 

  • Then unzip the BZ file in place

  • Add the extension .NC to the name

3.  This is what you should have now.
4.  In the panels that follow we'll load as many of these as we can into the programs Saga, ncBrowse and Integrated Data Viewer (IDV) to see how these formats and programs interact.  There is no attempt to provide a step-by-step training, as this is provided elsewhere (as indicated).
5.  Saga Summary  
6.  NetCDF, HDF and GRIB2 files are loaded into Saga with IMPORT/EXPORT-GDAL/OGR > IMPORT RASTER.

Click on the ellipsis (>>>) to the right of FILES to begin the selection process.

 

7.  You can select all of the NC, HDF or GRIB2 files to load at the same time.

Experiments with these files indicate that you can rename the files from NC to HDF, or the reverse, it it won't make any difference to Saga!  Either version works fine.

8.  Back on the main window, uncheck the MULTIPLE BANDS and TRANSFORMATION boxes.

Then click OK.

9.  Saga will take you through the files, one at a time, to select the data you want (i.e. the "bands") even though you did not check that item.

Select the most obvious data item (i.e. the one you really want), and click OK.

10.  Do that again for the next file.
11.  And again for the next file.
12.  And you will finally have these 4 grids loaded.
13.  Go back into the descriptions, and change the names as you see here, so you don't get them mixed up.

The author always tries to remember their original names, with disastrous results, so avoid his mistakes. 

14.  Now we have the loaded grids.  We can take an initial look at them, with ADD TO MAP, to see what else needs to be done.

15.  The original publishers may have provided wonderful data, but Saga often has trouble reading and displaying the files as we want them to appear.  Here's an analysis of the maps:
NAVO SST
  • Upside down
  • Land is black - no-data value problem
  • Sea is white - palette problem
SCRIPPS MIXED LAYER DEPTH
  • Upside down
  • Land is white - no-data seems fine
  • Sea seems well paletted (but needs color)
SURFACE TEMPERATURE (LAND & SEA)
  • Upside down?
  • Seems to be a different globe than the -180 to +180 we usually want
  • Palette looks ok, but needs color
MC SST
  • Orientation correct
  • No-data values seem fine
  • Data values seem well paletted (but need color)

16.  And here the author has begun the above "fixes" to the grids to make them look better.  The work is not complete, and you are invited to finish the improvements yourself, based on material in the MDL exercises.

17.  Now you have the maps begun, but much remains to be done.  Here's a list of suggestions:

18.  Aside from the visualization (i.e. the maps) you might want to extract data for your area of interest.  In saga, so this with GRID-TOOLS < RESAMPLING, the tool shown here.

Use the Google search engine on the MDL main page to look for RESAMPLING to see examples of this.

19.  Or, for the original global grid or for the resampled grid, you can convert the format to many different grid formats with the tool IMPORT/EXPORT - GDAL/OGR > EXPORT RASTER.

Use the Google search engine on the MDL main page to look for EXPORT RASTER to see examples of this.

20.  You can export it in any one of 37 formats, as you see here.  Take care with all the settings, and make experiments to make sure you're exporting the grid exactly formatted for the intended target software.
21.  Saga Image Export:  The usual methods to capture Saga images for use elsewhere are covered in the exercise 5.5 Creating "Standard" Grid Products with Saga
22.  ncBrowse Summary - This program is a small "utility" type program that can be used for investigating of the integrity of NetCDF files and quick visualization.  NetCDF metadata are partially displayed, which is useful but not completely satisfying the needs of some data managers.  The ncBrowse data export facility is nearly useless, because the 2 file formats it can create are not widely used.  The author sets ncBrowse as the default program for NC files, for quick checking.  Most NetCDF files that Saga initially displays incorrectly are handled properly by ncBrowse, so it can be considered as the "acid test" for NetCDF conformity.

NOTE:  ncBrowse also reads CDF format, a close relative of NetCDF.

23.  Open ncBrowse and you'll see this relatively simple main page.
24.  Select FILE > OPEN FILE then navigate to these example files.  Select the first one ("NAVO") and then click OPEN.
25.  Here you see the file in ncBrowse.  There are ATTRIBUTE, DIMENSION and VARIABLES statistics shown on the left.  On the right you can see the 3 "variables".

NOTE:  It's not known why VARIABLES on the left doesn't match up with the actual list on the right.

Double-click on the variable you want to explore, ANALYSED SST.

26.  This control window opens for the SST variable.

27.  Due to a glitch in ncBrowse coding you can't see all the metadata at once, but here they are.

These metadata are critical, and need to be examine.  A classic case of using ncBrowse to "fix" a dataset is found in 5.9 Correcting Faulty FillValue (i.e. Blank) Declarations in NetCDF Grids

 short analysed_sst(time=1, lat=1801, lon=3600);
:long_name = "Analyzed Sea Surface Temperature";
:standard_name = "sea_surface_temperature";
:type = "depth 1m";
:units = "kelvin";
:_FillValue = -32768S; // short
:add_offset = 273.15f; // float
:scale_factor = 0.1f; // float
:valid_min = -20S; // short

:valid_max = 350S; // short

28.  Check the X and Y selections (above GRAPH VARIABLE) and make sure they match up with the relative sizes of the START and END ranges beside them.  [They are correctly selected above.]  Then click on GRAPH VARIABLE.
29.  ncBrowse is almost always able to display earth science data grids in NetCDF correctly, because it has been written by an expert in that format.  So whereas Saga might give you a strange map, you almost always see something reasonable at first.

Here's a really good global map of SST, with a scale at the bottom (in Kelvin, as expected).

30.  Sometimes (but not always) you can easily use the START and END coordinates to select an area of interest, by simply changing the values to the lat/lon coordinates you want.  Here we set them to the Liberia area.

Click on GRAPH VARIABLE again.

31.  And here you see the location you want.  This is often possible, but in the case of very high-resolution global grids ncBrowse may simply refuse to return a map.
32.  Here we've opened the second grid listed above ("MC").
33.  You can set the X and Y axis, as you did above, and click GRAPH VARIABLE.
34.  But ncBrowse cannot handle such a large grid, and does not return a map.  This INFORMATION window just stays there until you close it.
35.  And here is the third grid.  We've selected MLD MEAN (mixed layer depth - average) to display.
36.  And here's the map.  You can see understandably deep MLD near the poles, due to strong winds.
37.  For any grid, you can try to export data, but only these relatively obscure formats are available.  Saga's main list of importable rasters doesn't include either one.

So it's good we have NetCDF files to begin with and don't need ncBrowse for file creation.

38.  ncBrowse Image Export:  The only method offered is a PRINT option, that includes PRINT TO FILE.  No usual image options are offered.
39.  Integrated Data Viewer (IDV) Summary
40.  Run IDV.  Typically, you should use PROJECTIONS > PREDEFINED > WORLD to get this basic start-point map. `
41.  Select DASHBOARD > DATA CHOOSERS, then navigate to your example files.  Select the 2 shown here and click ADD SOURCE (at bottom center)

NOTE:  The A200 and climatology NetCDF files (the middle 3) are not recognized by IDV as valid grids, so they don't work.  This indicates IDV does not include recognition algorithms for HDF, and the publishers haven't chosen to join the merge-HDF-and-NetCDF movement.

42.  For example, this is what you'd see if you tried to open either A200 file in IDV.  Just click CANCEL to get out of this if you try.

43.  Here's what you'll see after loading the recognized grids.
44.  To view the first grid (NAVO), make these choices in the dashboard.  Then click CREATE DISPLAY.
45.  Here's the default map of that grid.  Everything looks fine, but you probably would want to work on the color palette, etc. to make it best for your purposes.

46.  You can set the properties to focus in on the Liberia area of interest, as demonstrated in many IDV exercises, for example 9.8 Visualizing Satellite-Measured Sea Surface Height & Geostrophic Current Estimates in IDV: AVISO.
47.  And here's the product of that subsetting. 

NOTE:  Be sure to work carefully with the color palette and value range to get a usable figure with understandable features (i.e. like this one).  The global values would render it all a featureless yellow.

48.  You can use IDV to do more than visualize the data.  You can extract the data (global or subset) in 2 formats, as shown below.  Select the NetCDF option.

49. You're asked to specify which display to export.  Select the one you want.  Here the current display is selected.

Then click OK.

50.  Now you're asked to locate and name the exported product file.  Be sure to include the NC extension, because IDV doesn't do this automatically.

Click OPEN when finished.

51.  Now you're asked to specify the data grid.  Pick the one you want and click OK.

This completes the process, and the new file should appear where you specified.

52.  Here's your new file, shown in ncBrowse to check its validity.  It seems fine, so you succeeded.

53.  Here is the second loaded file (the GRIB2 format file).  You can see that it has problems associated with the 0-360 orientation, which either is badly formed, or IDV can't automatically fix.  When the map first appears, it is a 0-to-360 map.  If you force it to center on the Atlantic, then the left half looks OK, but the right half is a blurred image of the center-line values.  The IDV publishers have been asked for assistance/comment.

54.  IDV Image Export: