|
 |
|
What's new with the STORET Data Warehouse?
The STORET Data Warehouse now provides downloaded data in a compressed format
using immediate or overnight batch processing. This means that when
downloading a file from the STORET Warehouse, you will no longer see a "pop-up window" filled with raw results in a
text file format. Instead, a link to a zipped file containing your data is now delivered to you via email. When querying data
from the Warehouse, the system will ask you to provide an email address to which the link is sent. The link then allows you download
the zipped file containing the raw data you requested, metadata for this data, and other associated reference documents.
This page helps answer many of the questions that users have regarding these
changes in the way data from the STORET Warehouse is being delivered. Please see
How to Query and Download Data for detailed
instructions on how to generate, download,
export (to Microsoft Excel or Access), and analyze a data query from the
STORET Data Warehouse
Why did the STORET Team do this?
- When the older "pop-up window" was used for immediate downloads, it was
possible for users to lose data if the file was saved before the real-time
download was completed. This is why the immediate "pop-up" option was discontinued.
- A requirement of making the documented quality of the data more accessible
meant adding the metadata file and other project-level reference documents to
the compressed file. Having multiple files as part of the download mandated
a compressed file format and is the reason why an email address is now needed
for both Immediate and Overnight Downloads.
- Smaller datasets can still be downloaded immediately. Processing smaller,
immediate queries is still completed and downloadable from the URL within 1-30
minutes and they are not overnight batches unless selected as such.
- As part of a future upgrade, using this compressed format will allow us
to eliminate the need for three separate download queries (Regular, Biological,
Habitat) which has confused a number of users in the past. Soon a single
query will return all three result files in a compressed format.
Return to Top
What is a .tar.gz file and why isn't my download a textfile?
- The downloaded .tar.gz file is a compressed (zipped)
file. This means that you will need compression software like WINZIP® to open
the file. It was necessary to move to this compressed format for both Immediate
and Overnight downloads as all downloads now contain multiple files, including
your results textfile.
Return to Top
Why are there more files than just my query results in the downloaded file?
- In addition to the results data that you queried, you now automatically
receive the metadata file and any (if any) reference documents that are
associated with the organizations that own the data. This is so that you
can better determine the quality of the data you downloaded. The result
file(s) contain the raw data that you queried; regular, biological, and habitat
result queries are found in individual files. The metadata file includes
information about the organizations that own the data including contacts, methods,
labs, and other info. The reference documents can be pictures, datalogger
results, QAPPs, or any other project-level documents associated with the data owning organizations.
The Warehouse currently does not serve up station or result level documents.
Return to Top
I can't make sense of my Results file when I open it.
Return to Top
Is there any useful information in the metadata file?
- The metadata file contains the following summaries that can be used to
contact the data owners, create a station list, describe the methods and procedures
used, qualify the labs, correctly cite the data, and generally determine the
quality of the data for yourself:
- Organization summary
- Cooperating Organization Summary
- Project Summary
- Sample Collection/Creation Procedure Summary
- Sample Gear and Equipment Configuration Summary
- Sample Preservation and Handling Profile Summary
- Analytical Procedure and Equipment Detail Summary
- Laboratory Summary
- Lab Sample Preparation Procedure Summary
- Bibliographic Citation Summary
Return to Top
How do I make sense of downloaded file names?
The files found in a download have four main components:
- Type of Document: denotes whether the downloaded document file
contains raw data ("Data") or is a reference document associated with the raw data ("RefDoc").
Examples:
Data_XYZ_20070322_205714_RegResults.txt
RefDoc_XYZ_20070322_205714_Project_COPSBDC1_cua0001.pdf
- Unique Identifier: user provided 3 character ID, followed by the date/time stamp
to ensure user can uniquely identify a download.
Examples:
Data_XYZ_20070322_205714_RegResults.txt
RefDoc_XYZ_20070322_205714_Project_COPSBDC1_cua0001.pdf
- Type of Data: denotes what type of raw results data the downloaded document file
contains ("RegResults" for regular results, "BioResults" for biological results, and "HabResults"
for habitat results), which file is the metadata file ("Metadata"), or provides the Project ID and file name for
the downloaded reference documents.
Examples:
Data_XYZ_20070322_205714_BioResults.txt
Data_XYZ_20070322_205714_HabResults.txt
Data_XYZ_20070322_205714_RegResults.txt
Data_XYZ_20070322_205714_Metadata.txt
RefDoc_XYZ_20070322_205714_Project_COPSBDC1_cua0001.pdf
- Type of File: denotes the extension for the format of the document file
(.txt, .pdf, .bmp, .gif, .jpg).
Examples:
Data_XYZ_20070322_205714_RegResults.txt
RefDoc_XYZ_20070322_205714_Project_COPSBDC1_cua0001.pdf
Return to Top
|
