Quick Start to Retrieve or Plot Data

Table of Contents

Getting Started

Using FTP to Retrieve Data

Using HTTP to Retrieve Data

Using wget to Retrieve Data

Using GDS Clients to Plot Data

Use Our Server to Create Plots

Explanation of "Online"/"Offline" data

Obtaining Offline Data

Offline Order Limitations

Getting Started

For new users, we recommend you start with the NOMADS Web Interface. At the top of this page, you can sort products either by data provider or data type using the links at the top of the page. The interface allows users to:

1. Use FTP and HTTP to retrieve data
2. Remotely create and download graphics
3. Utilize distributed access to create products locally on your desktop

The sections below tell you how to quickly begin using these features.

Also, we have developed additional resources and links at the NOMADS Home Page to help you quickly learn and efficiently use distributed access methods within the NOMADS OPeNDAP framework. These references include getting started guides, tutorials, and information about available OPeNDAP enabled clients and servers. Please see the Distributed Data Access section of this guide.

Using FTP to Retrieve Data

In the table on the Model Data Access page, find the model-grid entry you desire and choose the "plot/ftp" link. After you build and review your dataset collection, you can use ftp to retrieve files or subsets from files. Press "Select Files for FTP" to access our "ftp4u" service. First you select any file or files from your dataset collection. You can also select support files, such as .ctl (GrADS descriptor ), .grb (GRIB data), or .idx (gribmap).

For GRIB files you can select individual parameters and levels. This can greatly reduce the volume of data you have to copy and thus reduce transfer times. For each file you select, we create a file with the same name, but it will only contain the parameters and levels you select. You can also extract a subregion from the GRIB files to further reduce the data volume. The files remain in GRIB format. If you perform spatial (region) subsetting on files, it is important to note that this operation will render all support files invalid. It is up to the user to recreate their own .ctl and .idx files that may be required to utilize the GRIB data.

After specifying your retrieval attributes, enter your email address in the form at the bottom of the screen, and press "Start FTP".

Your files are prepared for retrieval in two stages. First they are copied to a private ftp server and then to the public ftp server at ftp.ncdc.noaa.gov. Depending on the volume of data, it may take several minutes to complete the transfer. They will be deposited in directory /pub/download/nomads. All files from your dataset collection will have an extention which is your unique 4 digit order number. They will remain on our anonymous server for three days. You can use the instructions below to retrieve your files:

To determine when your files are available for retrieval, you can monitor our ftp site.

Note for AWIPS Workstation Users:
You can request model subsets or complete model runs via the NOMADS Web Interface. Since the current NCDC holdings contain the NOAAPort Broadcast model output, we expect that NWS and NOAAPort users will use this service for papers, review of synoptic conditions against model analysis and forecast products, and other analysis tasks. Users may elect to use the NOMADS distributed remote access methods using OpeNDAP and their selected desktop client (see Distributed Data Access); browse and plot selected output for retrieval; or ftp the raw GRIB using our ftp4u service.

Once you obtain the raw grib for display on an AWIPS workstation, you must convert the GRIB to the AWIPS style NetCDF format. Unidata (Kambic, Oram, et al.), and FSL (Ramer, et al.), developed tables for use with the gribtonc routines to convert GRIB to NetCDF. Please see Unidata NetCDF Libraries and Decoders for more information.

Using HTTP to Retrieve Data

NCDC provides a web server to allow you to browse our datasets, select files, and then use your web browser to download the files to your host computer. On the Model Data Access page, select the "http" link for the desired dataset. Navigate to the desired file. Do not left click on the ".grb" or ".idx" files. These are binary files, and your browser will attempt to display them. To download the files:

• Using Netscape - right click and choose "Save Link Target As".
• Using Internet Explorer - right click and choose "Save Target As".

Using wget to Retrieve Data

GNU Wget is a freely available network utility to retrieve files from the World Wide Web, using HTTP (Hyper Text Transfer Protocol) and FTP (File Transfer Protocol), the two most widely used Internet protocols. Since wget is a noninteractive downloading utility, you can execute scripts containing wget commands. For detailed instructions, see the GNU wget Manual. For downloading instructions, see GNU wget - GNU Project. On the Model Data Access page, select the "http" link for the desired dataset. Navigate to the desired file. Do not left click on the ".grb" or ".idx" files. These are binary files, and your browser will attempt to display them. To download the files:

• Using copy and paste, construct the URL to the desired file.
• Copy and paste the constructed URL into a wget command.

  Example:

  wget -O OUTPUT_LOCAL_FILEPATH http://nomads.ncdc.noaa.gov/data/eta/
  200305/20030501/early-eta_212_20030501_0000_012.grb
  

** Note ** Some operating systems and/or shells require you to place the OPeNDAP Constraint URL's in single quote characters

Using GDS Clients to Plot Data

NCDC provides a GrADS OPeNDAP Server (GDS) to access model data. On the Model Data Access page, select the "gds" link for the desired dataset. Use the GDS to navigate to the GRIB file you want to access. Select the "info" link to find the name and URL of the data on our server you desire to access with your client running on your desktop. Each example below shows you how to access a single dataset from the Eta model.

• Grid Analysis and Display System (GrADS) Example
  gradsdods
  sdfopen http://nomads.ncdc.noaa.gov:9090/dods/
  NCDC_NOAAPort_ETA/200305/20030501/
  early-eta_212_20030501_0000_fff
  display t2m
  
• Unidata's Integrated Data Viewer (IDV)
  Just paste the selected NOMADS URL into the IDV data loader.
  
  http://nomads.ncdc.noaa.gov:9090/dods/
  NCDC_NOAAPort_ETA/200305/20030501/
  early-eta_212_20030501_0000_fff
  
  
• MATLAB

  To load data served by NOMADS GDS into Matlab one needs the
  Matlab-OPeNDAP command line client. Follow the instructions found at this link to download and install the "Matlab Command Line Tool" for the platform that they are using. Make sure that the place in which the command line tool is put is in both the unix/linux or whatever path as well as in the Matlab path. The easiest way to do this is to simply startup Matlab from the directory with the tool in it.

  Below is a example Matlab session that was used to get a subset of the data as well as to plot them.

  >> path(path,'/Applications/DODS/bin/ml-toolbox/')
  >>loaddods('http://nomads.ncdc.noaa.gov:9091/dods/
  NCEP_NARR_DAILY/197912/19791210/
  narr-a_221_19791210_0000_000
  ?clwmrprs[0:0][0:28:28][0:4:239][0:4:585]')
  >> whos
  Name Size Bytes Class
  clwmrprs 60x147x2 141120 double array
  lat 60x1 480 double array
  lev 2x1 16 double array
  lon 147x1 1176 double array
  time 1x1 8 double array
  Grand total is 17850 elements using 142800 bytes
  >> imagesc(lon,lat,squeeze(clwmrprs(:,:,1)))
  ;set(gca,'ydir','normal')
  >> max(clwmrprs(:))
  ans =
  9.9990e+20
  >> min(clwmrprs(:))
  ans =
  1.0000e-12
  >> nn=find(clwmrprs>1e+20);
  >> clwmrprs(nn)=nan;
  >> imagesc(lon,lat,squeeze(clwmrprs(:,:,1)))
  ;set(gca,'ydir','normal')
  >> load coast
  >> hold on
  >> plot(long,lat,'w')
  >> print -dpng /users/petercornillon/desktop/figure.png
  << END EXAMPLE >>
  
  
  

  A Matlab OPeNDAP Oceanographic Toolbox is available. This toolbox provides Graphical User Interfaces (GUIs) to access a number of ocean data sets via OPeNDAP as well as tools to plot the acquired data. To download, See this site .
  
  *Note* To use some of the GUIs you will also need the timeplt package developed by Rich Signell of the USGS. This package is available Here.

  For further support or to report problems on OPeNDAP access users can go to OPeNDAP user support.

• Interactive Data Language  (IDL)
  

  Refer to your user guide.

• Ferret Example
  set data "http://nomads.ncdc.noaa.gov:9090/dods/NCDC_NOAAPort_ETA/
  200305/20030501/early-eta_212_20030501_0000_fff"
  
  

When you give a url to a OPeNDAP client, it actually sends a url with one of the following six file extensions: .das, .dds, .dods, .info, .ver or .help. The extensions tell the OPeNDAP server which object to return. No extension defaults to OPeNDAP, and retrieves the data subset.

Using Our Server to Create Plots

This web site provides a built in GrADS front-end application to allow users to select data, specify numerous plot options, and then use GrADS on our server to create plots. You can create contour maps, timeseries plots, lat/lon vs time plots, and animations. After viewing plots you can download the plot in either postscript format or GrADS metadata format. Use this function if you are unfamilar with GrADS, but want to quickly create some plots:

1. Point your browser to the Model Data Access page,
2. Select the "plot/ftp" link for the desired dataset.
3. Wait for the system to build you collection of data, build time can vary immensely depending on which dataset you are requestion. Anywhere from seconds to a few hours.
4. When this page fully loads, Click the "Plot" Button in the lower right.

Explanation of "Online"/"Offline" data

Due to the vast volume of model data NCDC NOMADS archives, it is not possible to keep it all online all the time. Eventually, online data needs to be pushed back into NCDC's High Performance Storage System

Upon entering a datasets entry point from the data access page, the following captions will display under "Data Availability":

Online
Archived Offline

Both captions are followed by their corresponding date ranges. "Online" data is stored directly on the back-end data server and is readily accessible using the NOMADS web interface: HTTP, FTP, GDS, and other NOMADS services. Offline data is stored in NCDC's deep archive and must first be retrieved from the HPSS then placed into a public staging area to make it accessible. The data will remain staged for a specified, limited amount of time. The following two sections explain the offline order interface in more detail.

Obtaining Offline Data

Because data is listed "Archived Offline", this does not indicate that users cannot access this data. However, more steps, and a waiting period are required to obtain it. To submit an order for offline data, enter the date range of interest and click "Build Order". The web interface will recognize when a request includes an offline date range, and require you to enter a valid e-mail address on the following web page. This address will be used only for notifying the user of the orders status. The system does not check for invalid addresses so if one is used the user will not obtain notifications regarding the order.

To begin the offline data request process, enter a valid E-mail address in the text box then click the "Confirm Offline Order" button to the left. Doing so will initiate automated processing on our server to pull the data out of NCDC's deep archive and place it in the staging area set aside for users. When the order is picked up by our system, an email will be sent to the address you just entered echoing details about the order and a URL to check on the order and eventually retrieve the data when the process completes. Another e-mail message will be sent upon completion, notifying the user the order is complete. Occasionally missing or corrupt files will fail to be retrieved from the deep archive: a list of such files is appended to the completion e-mail.

The system processes requests in the order they are received. The time it takes to complete the order depends on how many are in front of yours, how large your order is, HPSS load, and to a lesser extent, NOMADS server load.

Offline Order Limitations

When offline data has been staged, it will appear the same as online data for a period of five days (from the time the order was completed). If the user cannot or does not access and retrieve the data within this allotted time, the staging area will be purged and the user will need to resubmit the offline data request. As of May 2006, the staging area has 1.75 terabytes (1,750,000 Megabytes) of space allocated to ALL users, when this system is in demand, the space can fill up quickly. A low disk space warning has been put in place to notify users before they enter their email to begin an offline order. No orders placed when this warning is up can be guaranteed to complete successfully. Please be considerate of other users, all orders larger then 50% of the total staging area volume may be terminated without notice. If you have urgent need for a large volume of offline data, please contact NOMADS.ncdc@noaa.gov. In some cases, we will not be able to service large requests.

In addition, offline requests cannot currently (May 2006) be subset in any fashion. Hence, a request for a year of offline High Resolution GFS 500 millibar heights will need to pull the entire dataset from deep archive, which is currently not possible given the current amount of staging space. This is due to the fact that the archived files are stored in a compressed format. We currently are looking into ways to improve this.

Throttles:
We have put the following measures in place to prevent overload of the offline order system, and prevent orders from failing to complete, or "completing" unsuccessfully:

1. Orders of High Resolution NWP, which span more than 15 days will be automatically truncated to 15 days starting with the data request start date.
2. When there are 40 offline orders in the order queue, the system will block order submission until orders can complete.
3. If staging area disk space is critically low, submission of orders will be blocked until the space frees up. In some cases, this can take up to 5 days for enough staged data to expire.
4. If staging area disk space becomes critical with orders still queued, Processing of orders will halt until disk space frees up.
5. Lastly, if the system is experiencing technical difficulties for any reason, or we feel the system is being overloaded (too many orders in a short period of time), submission of orders will be blocked until the issues are resolved.

These throttles have worked well, however there are still instances where orders may fail. Most of the time this is due to something going wrong in the automated process which retrieves the data, or the HPSS being reset when an order is processing. If you receive a notification that your order was completed but data is missing, in most cases the best course of action is to resubmit the order. If they continue to fail, contact NOMADS.ncdc@noaa.gov. Certain files will fail repeatedly because they are missing from our archive.

Top of Page