-
Exercise Title:
Importing Ocean Data Portal (ODP) Spreadsheet with International
Organization for Standardization (ISO) Date & Time into an Ocean Data
View (ODV) Collection
-
Abstract: This
exercise addresses primarily the methods to import a
spreadsheet containing ISO dates and times into an ODV collection.
The ODP is a work in progress, so the current data formats and the catalog
access system are expected to evolve. Writing this exercise is part
of that process, because feedback to the system administrators will be
offered to assist them in their work. Essentially this exercise
simply drops itself onto the very robust table import functions of Ocean
Data View.
-
Preliminary Reading (in
OceanTeacher, unless otherwise indicated):
-
Required Software:
-
Other Resources:
-
Author: Murray
Brown
-
Version: 6-1-2014
|
PART A:
Importing spreadsheet with completely ISO-compliant date and time (yyyy-mm-ddThh:mm:ss.sss) |
1. Open the main Ocean Data
Portal (ODP) page, and familiarize yourself with the project scope and
goals. Click on DATA ACCESS along the top row of commands.
Then in the next window select ODP NETWORK > SEARCH & DOWNLOAD |
 |
2. This separate page opens,
showing a browse-type menu of the data resources already in the ODP. |
 |
3. Click on ALL PAGES in the
lower right corner, so you can see all 89 items on one page, as you see
here. NOTE: The list of
items has only grown from 65 to 89 in a little over 2 years, so the ODP is
not adding categories at a great rate. The number of individual data
objects within these categories is, however, considerable. |
 |
4. Near the middle of the
page, find and select the SECCHI DISK DATA RU_NODC_02. |
 |
5. Then find and click DATA
ACCESS near the top left corner of the page. |
 |
6. In a short time, these
options will appear to the right of the selection. If the data are too
big to display, then only the DOWNLOAD control appears. NOTE: Right
now, three different types of resource are provided, as indicated by the
icons on the far right: STRUCTURED DATA FILE, DATABASE, and OBJECT
FILE. |
 |
7. Right-click on the DISPLAY
DATA control, and select OPEN LINK IN NEW TAB (or equivalent in your
browser). |
 |
8. This spreadsheet opens. The variables are:
- Data unit: index - Appear to be unique station identifiers
- Data unit: iIdentifier - Lots of repetition, so they might be program,
vessel, or system groupings
- Date and time - Good ISO formats
- Latitude point
- Longitude point
- Water transparency - Secchi disk depth in m
- Quality check sign - Apparently always zero
Only 50 lines of data are available here, never the whole file, so
COPY/PASTE from here is not useful. |
 |
9. Find the download options
at the bottom left of the spreadsheet. Click on CSV (for "comma
separated variables"). |
 |
10. Navigate to DATA > OCEAN > ODP and save the file as
RU_RNODC_02_1-1330940285043_csv.txt |
11. Always check ASCII files
with a good editor to see what they contain. Here are the first few
lines of the TXT file. |
 |
12. Note that the DATE AND
TIME records are absolutely compliant with the ISO standard, including the
"+4:00" offset to indicate that the times are 4 hours ahead of Z time at the
Greenwich Meridian. |
 |
13. Run Ocean Data View (ODV) |
 |
14. Select FILE >
NEW (to create a new collection) and navigate to the folder
PRODUCTS > ODV> COLLECTIONS and enter the name
RU_RNODC_02_1-1330940285043_csv. Then click SAVE. |
15. To define the collection
structure, select USE TXT, ODV, VAR, OR OTHER FILE IS TEMPLATE.
NOTE: The term "Customized Collection Creation" does not appear in
the ODV documentation, but it's a good guess that the following steps are
customization, so we check it here.
Then click OK. |
 |
16. This window appears, very
similar in function and appearance to the Excel window that helps the user
to make incoming spreadsheets intelligible. The COLUMN LABELS are not
separate here. so the COLUMN SEPARATOR must not be correct. Try all
the choices to see what happens. |
 |
17. When the comma is
selected, then the COLUMN LABELS appear separately, so this is the correct
setting. The other settings are self-explanatory.
Click OK. |
 |
18. This windows appear,
where you set up the structure of the new collection.
- The META VARIABLES are the fields that ODV "wants" to have. They
are not necessarily available in this dataset, but they are desireable.
- The initial set of COLLECTION VARIABLES are the variables that the
import process had discovered in the spreadsheet.
|
 |
19. Of these,
DATA UNIT INDEX, DATA UNIT IDENTIFIER and DATA AND TIME all appear to be
more appropriate for the metadata section. |
20. Select these 3 variables
in the lower section, and use the << control to remove them. It is also
possible to create NEW variables here, with the control on the right side.
For example, according to the ODV guide if you have true time-series data, a
collection of sequential data from only one location, then create the
variable
time_ISO8601 for use as the organizing variable (instead of depth, the
usual quantity).
Now click OK. |
 |
21. Due to the very different
nature of this dataset and collection, these selections are probably
sufficient. Click OK. |
 |
22. After a few seconds, the
collection is created, and this empty map appears. |
 |
23. Select IMPORT > ODV SPREADSHEET,
then navigate to the TXT spreadsheet and open it. |
24. This is a repeat of the
spreadsheet recognition step above. Make the same selection, and click
OK. |
 |
25. Now we must make the
specific match-ups between the incoming data (in the spreadsheet) and the
fields in the collection structure. |
 |
26. Select DATA UNIT INDEX
and STATION, and click ASSOCIATE. Asterisks (*) will appear by them to
indicate the match-up. |
 |
27. Now, select the DATA AND
TIME on the source side, and YEAR on the meta variables side. Then
click CONVERT. |
 |
28. From the conversions that
are offered for DATES and DAYS, select the first one. NOTE: In the string that
defines the format (yyyy-mm-dd hh:mm:ss.sss) there is no "T" in the middle
(as required by the ISO standard). This indicates that ODV doesn't
require it.
Then click OK. |
 |
29. Notice how ODV has placed
asterisks by all of the date and time elements on the right side. Now
click OK. |
 |
30. The two relevant
variables are already marked, so just click OK. |
 |
31. This message, and the
appearance of red dots on the map indicate that the import has been
successful. Click OK. |
 |
32. This is the map of the
stations, zoomed by you or automatically. |
 |
33. Now we need
to see how to use the data and time information in analyses. This
requires Station, Scatter or Surface mode graphics. |
34. Select VIEW > LAYOUT
TEMPLATES > 1 SCATTER WINDOW to see this graphic. The only variables
we have are used here is default axes. |
 |
35. The ISO date
and time information are in the metadata, but they do not yet constitute a
numeric variable that can be used as an axis or surface variable in any
analysis. We must first create a derived variable from these fields. |
36. Select VIEW > DERIVED
VARIABLES to see this list. There are many choices on the right which
could be calculated, but nothing yet on the left. |
 |
37. Close the open choices,
and open the TIME choices. Select TIME (STATION DATA & TIME) and click
ADD. Do not click OK. |
 |
38. This window is asking you
to define an "epoch" that defines the beginning of the system time for your
ODV collection. "System time is measured by
... the number of ticks that have transpired since some arbitrary
starting date, called the epoch. For example, Unix ... systems encode system
time ... as the number of seconds elapsed since ... 1 January 1970 00:00:00,
with exceptions for leap seconds. Systems that implement the 32-bit and
64-bit versions of the Windows API ... provide the system time as both
SYSTEMTIME,
represented as a year/month/day/hour/minute/second/milliseconds value, and
FILETIME,
represented as a count of the number of 100-nanosecond ticks since 1 January
1601 00:00:00 ..." [From Wikipedia article
http://en.wikipedia.org/wiki/System_time ] NetCDF time often
observes an epoch of 1900-01-01T00:00:00. And the "BPE" (before
present era) terminology refers to 1950-01-01. Etc. Etc.
Make whatever choices you want, and click
OK. |
 |
39. This new variable appears
on the left. Click OK to close this window. |
 |
40. Now, right-click on the
graphic and select X-VARIABLE. |
 |
41. Select the TIME variable,
and click OK. |
 |
42. And here you can see the
ISO data and time successfully used in ODV analyses. |
 |
43. Select VIEW >
SAVE VIEW AS and save this analysis as
1scatter_secchi_time so you can always get back to this figure in
this collection. If you want to close the collection, select FILE >
NEW. |
PART B:
Importing spreadsheet with nearly ISO-compliant date and time (yyyy-mm-dd
hh:mm:ss.sss) |
44. Now, just to
make sure that dates and times that do not include the "T" in the middle, as
specified by ISO 8601, we'll create a collection with a different
spreadsheet that we already know lacks this feature. |
45. Find the dataset
SEVASTOPOL BAY SECCHI UA_IBSS_06. Use the same steps as above to
access the spreadsheet. |
 |
46. Save the file in the folder DATA > OCEAN > ODP with the filename
UA_IBSS_06_1-1330939175118_csv.txt |
47. Examine the TXT file to
see what it contains. It is very similar to the above file, with some
differences |
 |
48. The variables are:
- Data unit: index - Not unique; only 2 values (1 or 2)
- Date and time - ISO, but lacks the "T" in the middle
- Latitude point
- Longitude point
- Water transparency
|
49. Use the same
steps as above to create the collection and import the data. The DATA
UNIT: INDEX can be set to STATION, but this association isn't certain. |
50. This figure shows the
collection, visualized in the same way as the previous collection. |
 |
51. Select VIEW >
SAVE VIEW AS and save this analysis as
1scatter_secchi_time so you can always get back to this figure in
this collection. If you want to close the collection, select FILE >
NEW. [It is in a different collection from the previous one, so the
same view name can be used.] |
52. Inspection of
other data choices in the ODP catalog has indicated that there are some very
different contents in other items. These will be treated in one or
more separate exercises. |