RDA Delayed mode processing back online

RDA subset request, format conversion, and data restaging from tape services are back online following system maintenance.  Queued requests will now be processed as compute resources allow.


RDA delayed mode request processing offline for maintenance.

RDA subset request, format conversion, and data restaging from tape services are currently offline due to system maintenance.  Requests will remain in status "Queue" until the data processing servers come back online, likely later tomorrow, 01/03/18.


NCAR data experts seek input at AGU fall meeting for summer 2018 workshop on digital data repository service

NCAR data management experts will engage the community in a poster session on December 11 at the AGU 2017 Fall Meeting to help shape a 2018 National Science Foundation workshop focused on developing requirements and expectations for a prospective new digital repository service.
The envisioned Geoscience Digital Data Resource and Repository Service (GeoDaRRS) would complement existing NSF-funded data facilities by providing data management planning support resources for the general community and repository services for researchers who have data that do not fit in any existing repository. GeoDaRRS would support NSF-funded researchers in meeting data archiving requirements set by the NSF and publishers for geosciences, thereby ensuring the availability of digital data for use and reuse in scientific research going forward.
See this abstract on the AGU Fall Meeting site for more information.
For additional information on the workshop, please see the workshop website.


RDA partial outages Dec 6 and 9, 2017

The tape drives that host some of our data holdings may be down sporadically for maintenance tomorrow, December 6. This will NOT affect services for disk-accessible (mostly more recent) data. If your request requires reading data from our tape library, you may experience a few hours delay. Plan accordingly or get your data today.

START: Wed Dec 6 2017 11:00 AM MST
END: Wed Dec 6 2017 3:00 PM MST

Globus Data Transfer service will be down for system upgrades for a few hours Saturday, December 9. Ordinarily, Globus is the fastest way to transfer data from the RDA. However, during this outage, Globus transfers will be delayed a few hours. Other data transfer methods will be available during this time.

START: Sat Dec 9 2017 9:00 AM MST
END: Sat Dec 9 2017 1:00 PM MST


Data subset, format conversion, and re-staging of data for download request processing offline through 11/3/17

Due to system maintenance, processing of subset, format conversion, and re-staging of data for download requests will be suspended until the afternoon of 2017/11/03.  Additionally, some dataset updates will be delayed until the same date.  Please check back for updates.  We apologize for the inconvenience.


Accessing RDA OPeNDAP endpoints with authentication

All RDA OPeNDAP supported datasets found under https://rda.ucar.edu/thredds now require user authentication for access.  Here are details on how to configure your system, such that most applications including NCL, will work properly with the authentication step (these instructions are based on what is provided by the hdfeos group):

1. RDA Registration and Setting Up Cookies

NCAR RDA OPeNDAP server access requires user registration and cookies. If you do not have RDA Login, please register first and have RDA username and password ready.

Once you have RDA Login username and password, and verify that they work by logging into https://rda.ucar.edu, you need to set up cookies to access data.  We highlight the key steps here.

To set up cookies properly, you will need the following 3 files in your home directory (e.g., /home/rdahelp on Linux or /Users/rdahelp on Mac assuming that rdahelp is your UNIX system's login name).
  1. /home/rdahelp/.netrc
  2. /home/rdahelp/.rda_cookies
  3. /home/rdahelp/.dodsrc
The first file .netrc should have RDA login and password. For example, if your username is rdahelp and password is 1234abcd, the file should have the following line.
machine rda.ucar.edu login rdahelp password 1234abcd
Please edit the above file to match your username and password. Since the file contains the password, make sure that others cannot see it by changing permission.

$chmod go-rwx /home/rdahelp/.netrc
The second file, .rda_cookies, should be created automatically by either wget or curl command.
If you like to use wget, issue the following command to create the cookie file. 
$wget --load-cookies ~/.rda_cookies --save-cookies ~/.rda_cookies --auth-no-challenge=on --keep-session-cookies https://rda.ucar.edu/thredds/dodsC/files/g/ds083.2/grib1/2000/2000.02/fnl_20000201_06_00.grib1.dds -O test.dds
If you like to use curl, issue the following command to create the cookie file. 
$curl -n -c ~/.rda_cookies -b ~/.rda_cookies -L -g --url https://rda.ucar.edu/thredds/dodsC/files/g/ds083.2/grib1/2000/2000.02/fnl_20000201_06_00.grib1.dds -o test.dods
If the above command succeeds, you will get the following output when you check the content of test.dods file.
$more test.dods
Dataset {                                                        Int32 LatLon_Projection;                                    Float32 lat[lat = 181];                                          Float32 lon[lon = 360];                                                         ... }files/g/ds083.2/grib1/2000/2000.02/fnl_20000201_06_00.grib1
If you don't see the above output, your login/password and system is not working properly with NCAR RDA OPeNDAP server. You cannot access data until this step works. If you get the same output, you can delete the temporary output test.dods file.
The final step is to create .dodsrc file if you don't already have one and add the following lines.

2. Test NCL open and read of OPeNDAP data (NCL version 6.4.0 is required)

The following example shows an example of how to test if NCL can open and read an OPeNDAP file after the above steps have been completed.
$ ncl                                                             ncl 0> f = addfile("https://rda.ucar.edu/thredds/dodsC/files/g/ds083.2/grib1/2000/2000.02/fnl_20000201_06_00.grib1","r")                     ncl 1> print(f)                                               Variable: f                                                       Type: file                                                         filename:    fnl_20000201_06_00                                   path: https://rda.ucar.edu/thredds/dodsC/files/g/ds083.2/grib1/2000/2000.02/fnl_20000201_06_00.grib1                                         file global attributes:                                              Originating_or_generating_Center : US National Weather Service, National Centres for Environmental Prediction (NCEP)          Originating_or_generating_Subcenter : 0                            GRIB_table_version : 0,2                                          ... etc                         
In the above code, the url points to the remote FNL file being served by OPeNDAP. Use print command to check if the url file is opened successfully and to see what variables are available inside the file.


All RDA data transfer and processing services restored to production

All RDA data transfer and processing services have been restored to production.  Thank you for your patience during the service outage.  Please let us know if you experience any continued problems.


RDA Service Outage July 14-18, 2017

All RDA data access, processing, and distribution services will be unavailable during system maintenance scheduled to take place from 5AM MDT, Friday, July 14 until 9AM MDT, Tuesday, July 18.

RDA web interface supported data discovery and request process submission capabilities are expected to remain online during this maintenance window.

Please note that if you have custom data requests where the data will expire during the outage, you can go into your user dashboard and extend the dates so that you can resume downloading when services are restored.  In your dashboard, under "Customized Data Requests", click the calendar icon next to the specific request that you want to extend.

An update will be posted when all services come back online. We apologize for the inconvenience.


RDA web services down for maintenance at 1PM MDT on May 3, 2017

All RDA web services will be down for maintenance starting at 1PM MDT on May 3, 2017.  We hope to have these services back online within 2 hours of the start of the downtime.  We apologize for the inconvenience.


Transferring custom file lists with Globus

The RDA has supported Globus file transfers for over two years, which enables users to transfer data files from RDA datasets to their destination endpoint using the Globus data transfer service (see http://ncarrda.blogspot.com/2015/06/transferring-rda-data-with-globus.html for details).  Transferring files using Globus is an efficient, reliable, and secure method to transfer large amounts of data from one endpoint to another.

But what if you're interested in a smaller or specific collection of files within a dataset?  In this scenario, the user would navigate to the Globus web app transfer interface and then browse through the files on the RDA shared endpoint to locate the files of interest.  This workflow, however, is not ideal from a search and discovery perspective, since users don't have the convenience of filtering the file sets via faceted browsing or viewing the supporting metadata alongside each file listed on the Globus web interface.

The standard Globus web app transfer interface.  In this workflow, users must browse through the full set of files within a RDA dataset to locate specific files to be transferred.

The good news: RDA users now have the ability to select and initiate Globus transfers of custom file sets directly from the RDA web interface without having to search for the files on the Globus transfer interface.  This method of transferring data uses the Globus Browse Endpoint helper API, and the only step required to be done by the user on the Globus web app is to select the destination endpoint that will receive the data files in the transfer.  The advantage to this approach is users are able to make use of the faceted search and discovery tools on the RDA website and avoid the bottleneck of making "blind" searches of the files on the Globus web interface.

How to make a Globus transfer of custom file sets

Starting from any RDA dataset page, go to the 'Data Access' tab, then select the 'Web file listing' link associated with any of the data products listed.  At this point, you may choose either the 'Faceted browse' or 'Complete file list' option, and then follow the instructions to locate your files of interest.

Procured file lists using the 'Complete file list' (left) and 'Faceted browse' search function (right) on the RDA website. Once a collection of files has been selected, the user initiates a Globus transfer by selecting the 'Globus' button.

To initiate a Globus transfer, select the files you wish to download, then select the 'Globus download' button.  This will redirect you to the Globus web app where you will authenticate with Globus and select the destination endpoint that will receive the data files.  After a destination endpoint has been chosen, select the 'Submit' button.  You will then be redirected back to the RDA website, at which point the data transfer will be submitted on your behalf.  A status message displaying the status of your data transfer will appear, and a notification e-mail will be sent to your RDA user e-mail address when the transfer has successfully completed.

The browse endpoint interface on the Globus web app, where users select the destination endpoint.

As a reminder, RDA users can log into the Globus web app with their RDA user e-mail login and password credentials; this is the recommended method for logging into Globus.  To do this, simply choose the 'NCAR RDA' organizational login on the Globus login page.