Marine Data Literacy 2.0

Providing instruction for managing, converting, analyzing and displaying oceanographic station data, marine meteorological data, GIS-compatible marine and coastal data or model simulations, and mapped remote sensing imagery

 

 

 

 

Home > 2. Marine GIS > 2.21 Color Web

2.21 Importing Climatological or Operational NetCDF Satellite Temp, Chloro, and Sal Grids into Marine GIS: OceanColor Web

  • Exercise Title:  Importing Climatological or Operational NetCDF Satellite Temperature, Chlorophyll-a and Salinity Grids into Marine GIS: OceanColor Web

  • Abstract:  Now that the Ocean Color Web (OCW) delivers many of its principal products in NetCDF format (including HDF5, a subset of NetCDF) it is much easier to import these rasters into a GIS, as you'll learn  below.  "Chlorophyll" is used here as a placeholder for many optical parameters provided at OCW.

  • Preliminary Reading (in OceanTeacher, unless otherwise indicated):

  • Required Software:

  • Other Resources: 

    • US NASA Ocean Color Web - Huge archive of marine optics parameters and SST from a long series of satellites, all in a non-standard HDF format that previously required laborious conversion to ASCII before use in GIS.  Latest version of Saga can accommodate these data directly.

    • frame_grid_liberia_0.05deg.sgrd - Dummy 0.05-degree grid for Liberia, made in 2.5 Creating Grid Templates in Saga

    • frame_grid_liberia_1deg.sgrd - Dummy 1.0-degree grid for Liberia, made in 2.5 Creating Grid Templates in Saga

    • A20030012015031.L3m_MC_SST_sst_4km_jan.nc - Climatological raster of surface temperature for January, made in 2.36

    • A20030012015031.L3m_MC_CHL_chlor_a_4km_jan.nc - Climatological raster of chlorophyll for January, made in 2.36

    • Q20120012015031.L3m_MC_SCI_V4.0_SSS_1deg_jan.hd5 - Climatological raster of salinity for January, made in 2.36

  • Author:  Murray Brown

  • Version:  2016-1-13

TEMPERATURE GRID     CHLOROPHYLL GRID     SALINITY GRID     GRID SUBSETTING

 
1.  Perform the exercise 2.36 Obtaining Operational & Climatological Satellite T, Chlorophyll and S Ocean Data: OceanColor Web to obtain data needed in this exercise.  Be sure to read over all the ancillary information so you have a good idea of what you're working with and how it was obtained.
2.  Run Saga.
3.  Select TOOLS > IMPORT/EXPORT-GDAL > GDAL: IMPORT RASTER.  For some reason, the files can't bea loaded with the expected IMPORT NETCDF tool, which may relate to specific formatting problems.
4.  Within the tool, you'll need to set up these options.  We picked the easiest method for interpolation, but you can explore this later on your own.

Click OK to continue.
5.  Make this selection, because we only want the SST data.  Then click OK.
6.  The SST grid appears in the DATA > GRIDS list.
7.  You can use ADD TO MAP  to see the grid.
  • Data value shown, 40718 for a temperature clearly shows that "scaling" of raster temperature values has occurred, so a formula will be needed to remove scaling
  • No color on land indicates a __FillValue has been correctly identified and used
  • X and Y values are too large to be lon/lat, so reprojection will be needed

 
8.  In the PROPERTIES panel, select DESCRIPTION to see the underlying metadata for the grid.  Read through and try to understand all the lines.
  • The SST_SCALE_FACTOR and the SST_ADD_OFFSET both show that the SST data values have been scaled.  This allowed them to be stored here as "unsigned 2 byte integer" values, to save space.
  • The UNDEFINED COORDINATE SYSTEM refers to the use of raw X and Y axis values, rather than longitudes and latitudes.

9.  To unscale the data values, select TOOLS > GRID > CALCULUS > GRID CALCULATOR.
  • Select the above system and SST grid for input.
  • For FORMULA enter the expression shown here:
    • (g1*0.00071718497)-2
  • For NAME, enter something like SST DESCALED so you know exactly what the result is called

Then click OK to proceed.

 
10.  This new grid should appear beside the old SST grid.
11.  If you use ADD TO MAP to see the new grid, you will still see the non-geographic axis coordinates.  But wherever the cursor is placed, you should see ordinary temperature values, e.g. 28.55 in this example.
12.  To check the unscaled values, you can right-click on the object and select HISTOGRAM to see this distribution.

It's interesting to note that there are lots of values below zero, down to about -1.99.  Those are typical temp values very near the poles.  Their presence in the data is the reason the descaling formula contained the minus 2 value.
13.  To continue on and fix the latitude/longitude problem, select TOOLS > PROJECTION > GEOREFERENCING > DEFINE GEOREFERENCE FOR GRIDS
  • Select the SST DESCALED grid for georeferencing.
  • Then click OK
14.  Make these selections to project the grid.  Be careful with each item, because some of them have very similar options.

The CELLSIZE has been calculated from total map width (360º) divided by number of columns (8640)

When finished, click OK.
15.  Look in the data objects, and you'll see a new grid with the physical descriptors shown here.  Rename it as shown so you don't lose it later.
16.  Before any other steps, you should save this new grid in the folder LIBERIA > PRODUCTS > SAGA> GRIDS > COLORWEB > SST with the filename sst_11mu_day_jan_global_modisa_colorweb_4km_descaled_geoproj.sgrd (or 0p05deg instead of 4km if you prefer; they are roughly the same). 

Here's how Saga saves a grid/raster:

  • FILENAME.SGRD - Saga file with the grid's physical and format characteristics
  • FILENAME.MGRD - Saga file with information about how to display the file, i.e. SETTINGS-related items
  • FILENAME.SDAT - Saga file with the basic raster data values for the grid

17.  If everything went well, you can open this new grid with ADD TO MAP, as you see here.  The settings should all be fine, because Saga created and saved an MGRD file above.
18.  Everything looks ok, so let's move on to the second grid containing the chlorophyll data.
  19.  Open the chlorophyll data in Saga, and when asked to SELECT FROM SUBDATASETS, check the CHLOR-A item.

Then click OK.
20.  These data have been handled somewhat differently from the SST data above:
  • Data value shown, 0.165274 seems to hint that no "scaling" of raster chlorophyll-a values has occurred, so no formula will be needed to remove scaling
  • Black color on land indicates no __FillValue has been correctly identified or used; can be fixed by:
    • Method in 5.9; takes time, or
    • Manual adjustment to Saga properties settings; fast and easy
  • X and Y values are too large to be lon/lat, so reprojecting will be needed, as above.

21.  In the PROPERTIES panel, select DESCRIPTION to see the underlying metadata for the grid.
22.  In the PROPERTIES panel, select SETTINGS.  You can see that the FILLVALUE should have been -32767, but it's actually -99999.  And the value range should bottom out at 0, but it uses the FILL VALUE.  That's why the data image above has black continents and blank white oceans.
23.  Set the NO DATA minimum and maximum to -32767 to make the continents white (i.e. no data).

Change the COLORS > VALUE RANGE to show 0 for the minimum.

Then click APPLY to see how this changes things.
24.  The continents now look fine (white = NO VALUE).  But the sea is still completely black.  Why?
25.  Right-click on the data object in the menu, and select HISTOGRAM to see what values we have for the sea.  Maybe this will explain why it's completely black.
26.  Now you can see that the value range (0-99) is entirely too big.  The real data range is about 0-8.
27.  Make that change in the VALUE RANGE, and also with the the COLORS control to increase the palette to a much larger number and a RAINBOW PALETTE.

28.  Now the data map will look much better.  Spend some time working with your own data to select the best value range, palette, etc.  Once you have the best set of conditions, then use it for all data so they can be visually compared.  These things are all up to you, because you're the scientist.
29.  Now we need to geoproject the data onto a global lat/lon system. Use the same tool and settings as above, because the grid system is the same for both.

It would be a disaster if they weren't the same, so check carefully.

 
30.  Now you can see the newly projected grid listed below a new "system" line, that describes the raster in geographic terms.  The new name (that you apply) indicates that it has been changed by the above settings conversions
31.  Take a moment to save the new grid in the folder LIBERIA > PRODUCTS > SAGA> GRIDS > COLORWEB > CHLORO with the filename chloro_jan_globe_modisa_ocweb_0p05deg_nonull_geoproj.sgrd
  • 0p05deg is roughly equivalent to the original cellsize, 4km
  • nonull indicates the land value are all removed, i.e. not shown in black
  • geoproj indicates the grid is now in true geographic coordinates

32.  Here we've used FILE > LOAD > GRID to load this new raster as a check to make sure it has been properly formed.
33.  Everything looks ok, so let's move on to the second grid containing the chlorophyll data.
34.  As you did above for SST and chlorophyll, load the salinity grid for January.  When asked which subset, select the "l3m_data", i.e. the Level 3 salinity from Aquarius.

Then click OK to continue.
35.  Use ADD TO MAP  to see that specific visualization problems occur here that are not quite the same as above:
  • Data value shown, 34.655769 indicates that no "scaling" of raster salinity values has occurred, so no formula will be needed to remove scaling
  • Black color on land indicates no __FillValue has been correctly identified or used; can be fixed by:
    • Method in 5.9; takes time, or
    • Manual adjustment to Saga properties settings; fast and easy
  • X and Y values have the correct raw value range (0º-360º) but they don't center on 0º,0º in the Atlantic, so reprojection is needed

 

36.  In the PROPERTIES panel, select DESCRIPTION to see the underlying metadata for the grid.  Read through and try to understand all the lines.
37.  In the OPTIONS panel, you can see the familiar NO DATA problem.
38.  To fix the NO DATA problem change the NO DATA MINIMUM and MAXIMUM both to -32767.  And further down, change the VALUE RANGE MINIMUM to 0.

Then click on APPLY to see if this fixes things.
39.  It seems to be working, but there is little differentiation in the at-sea colors.  This indicates the data values are all close together, but the palette covers a much wider range of possible values.
40.  Right-click on the data grid in the DATA menu, and select HISTOGRAM to see the actual range of values.

You can see here that the range is very narrow, about 31 to 37.  So we need to re-set the values in the SETTINGS panel.
41.  Go back to the PROPERTIES > SETTINGS panel, and insert 31 and 37 for the VALUE RANGE MIN and MAX.

Then click APPLY to see the changes.

42.  This looks great.

But now we need to fix the longitude range.  It currently goes from 0-360, but for this particular map (and indeed, nearly always) we want -180 to +180.
43.  Open TOOLS > PROJECTION > GEOREFERENCING > DEFINE GEOREFERENCE FOR GRIDS

Then enter these values.  Note carefully the differences with the previous two projections.

Then click OK to continue.
44.  This new system and grid object appear.  Notice that its name should be changed to reflect the correction of the NO DATA problem ("nonull") and the projection fix ("geoproj")
45.  Now you can right-click on the new object, and use SAVE AS to save it to the folder LIBERIA > PRODUCTS > SAGA> GRIDS > COLORWEB > SALINITY with the filename salt_globe_jan_aquarius_colorweb_1deg_nonull_geoproject.sgrd

46.  As before you should use FILE > GRID to add this grid to the map, and make sure it looks OK.
47.  So that takes care of basic reading and display of the global versions of these three grids.  Now we need to see how to subset them down to the Liberia area of interest, through the process of "resampling".
48.  Clean house right now and leave open only the three final, renamed global grids from each of the above.
49.  Also open the three Liberia-area frame grids you made in 2.5 Creating Grid Templates in Saga.

N.B. For your own work, you should have a "library" of local grids for subsetting purposes, at various resolutions.
50.  Now run TOOLS > GRID > TOOLS > RESAMPLING.  You can see that if you don't have the dummy grid you need, you can simply enter the physical coordinates in the TARGET GRID SYSTEM spaces.
51.  But you should make these choices:
  • For system and grid (at the top) select the global chlorophyll grid.
  • For TARGET GRID SYSTEM, pick GRID OR GRID SYSTEM
  • Then for GRID SYSTEM (which quickly changes to a set of drop-down choices) pick the high-resolution ("0.05") regional-sized ("400x300") grid

This tool will sample your existing grid to calculate what would be the values for each new grid cell, and then will create it in the Saga grid format.

Click OK to continue.
52.  The new grid appears in the desired system position.  If you made a small mistake then just delete it, and go back one step.
53.  If everything technical is OK, then change the name of the grid slightly to indicate it is not global now.  In this case "globe" has been changed to "liberia".

You make this change in the top line of the SETTINGS > OPTIONS > GENERAL > NAME panel, and then click APPLY

54.  Here, you can use ADD TO MAP (new, of course) to see the sampled grid and all its settings, including the color LEGEND on the right.  It looks fine, so you can continue.  If anything looks wrong, then try to fix it in the SETTINGS
55.  Now, use SAVE AS to save this newly made, newly renamed grid to the folder PROJECTS > LIBERIA > PRODUCTS > SAGA > GRID with the filename chloro_jan_liberia_modisa_ocweb_0p05deg_nonull_geoproj.sgrd or similar depending on your naming preferences.
56.  Use exactly the same tool and make the same name change to create the Liberia-area SST grid you see here.

57.  And here's what it looks like.
58.  Now, use SAVE AS to save this newly made, newly renamed grid to the folder PROJECTS > LIBERIA > PRODUCTS > SAGA > GRID with the filename sst_jan_liberia_modisa_ocweb_0p05deg_nonull_geoproj.sgrd or similar depending on your naming preferences.
59.  But now we're going to move on to the salinity grid, which has a very different grid cell size of 1 degree.  Be careful to make the necessary adjustments in the method.

60.  Here is the RESAMPLING TOOL with the necessary settings.  The original grid is rather crude, i.e. 1 degree, so you MUST resample to the same resolution to keep your product "honest".

 
61.  Now, use SAVE AS to save this newly made, newly renamed grid to the folder PROJECTS > LIBERIA > PRODUCTS > SAGA > GRID with the filename salt_jan_liberia_modisa_ocweb_1deg_nonull_geoproj.sgrd or similar depending on your naming preferences.

62.  And here you can see the new grid, reloaded into Saga to check it.  Everything looks fine.
63.  A final word about these new OceanColor Web grids:  It is wonderful news that the OCW is now using NetCDF or HDF5 (which is nearly equivalent) for their data.  Now they are moving in that direction, it will be infinitely easier for all users to obtain and manage their important products.  The author urges OCW to study the NetCDF protocols to provide files that do not require the above manipulations for easy viewing, i.e.:
  • Please use the correct NO DATA expression within the files, so that the manual adjustment to force recognition of the value -32767 (or whatever they choose) is not necessary.  See MDL's exercise 5.9 Correcting Faulty FillValue (i.e. Blank) Declarations in NetCDF Grids
  • Please include geographic projections in all the files
  • Please consider using -180 to +180 for the longitudinal range, not 0 to 360, when projected