|
Updating NOAA Weather Data |
|
rev. 04/05/2011
|
|
|
Updating NOAA Weather Data |
|
|
rev. 04/05/2011
|
|
The process of updating weather data is fully automated by a Managing Energy service that runs overnight. The manual process documented below is no longer used, however these instructions have been retained to explain how the automated process works.
Weather data is downloaded from the NCDC weather data site (Global surface summary of day). Then agree to the terms presented. You should be at a page that looks like this.
Go to Data Files (Current) and pick the year you want to download for (likely the most current year). You will be taken to an FTP site that looks like the one below.
Within this folder you may select the year of interest. For example the directory called 2006 will contain all weather information for all stations over the course of the 2006 year.
Each file shown contains the 2006 weather data for one station. Downloading and importing each stations weather data one at a time would be a time consuming process. We can obtain all weather information for all stations in one single download by scrolling to the bottom of this list and selecting the last and largest file.
For the 2006 year, this file is called 'gsod_2006.tar'. Though the name of this file may vary from year to year, it should be an archive of all the files shown in this directory. Download this file to Q:\Weather\NCDC and decompress it to the same directory (7-Zip or winRAR will both work).
The decompression process will create a large number of files. As mentioned, each file contains weather information for its respective weather station. Many are for stations not used in MEcom, which the import application will automatically ignore.
Now that you have the weather data files decompressed into a single directory, it's time to use the MEcom Weather Data Importer to automate the process of extracting and importing this weather data.
Installing the MEcom Weather Data Importer
To install this utility simply copy the folder Q:\ManagingEnergy Software\MEC Weather Data Importer' to your local machine. Use the file MECWeatherDataImporter.exe to run.
Importing Weather Data using MEcom Weather Data Importer
To import weather data run MECWeatherDataImporter.exe.
Select one of the data files to indicate to the application where the files are stored.
NOTE: The NCDC weather data is stored using Fahrenheit values. Make sure you place a check in the box, otherwise the data will not be internally converted to Celsius, which is the basic temperature unit for storage in MEcom.
To perform the import click GO. The application will automatically extract the data from each file and import it into MEcom (using the MEcom API). If data is missing the application will use linear interpolation to fill in gaps.
Once completed, review the log and take action if necessary.
Defining Import Fields in MEcom Weather Data Importer
From time to time the format of the input file may change. To accomodate this, a file called "FieldMap.xml", located in the application directory, defines the import fields and were in the rows of the data file each field is located.
<?xml version="1.0" encoding="utf-8" ?>
<Fields>
<Field name="StationID" startIndex="0" endIndex="6" />
<Field name="Date" startIndex="14" endIndex="22" />
<Field name="Temperature" startIndex="25" endIndex="30" />
</Fields>
MEcom Weather Data Importer needs to find 3 fields, StationID, Date, and Temperature. The startIndex and endIndex specify the character index in the row each field can be found. For example, the settings above tell the application that the station ID can be located in each row from index 0 through 6.
To import data of a non-standard format you will need to change these index values to suit the format of the file you are importing.
MEcom Weather Data Importer Technical Notes
A configuration file ('BOSSWeatherDataImporter.exe.config') is used to define application level settings.
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<appSettings>
<!-- The Connection string to the database used by the MEcom API -->
<add key="ConnectionString" value="Integrated Security=true;Data Source=SQLSERVER;"/>
<!--
Variables provided by the application... (replaced by runtime values. for use in the settings below)
[ApplicationDirectory] - The starting application directory
[SourceDirectory] - The directory containing all the source files
[DestinationDirectory] - The directory which extracted files will be placed
-->
<!-- The path to the application used to perform the extraction of compressed files -->
<add key="ExtractApplication" value="[ApplicationDirectory]\7-zip\7z.exe"/>
<!--The arguments passed to the extraction application -->
<add key="ExtractArgs" value="e -y -o"[DestinationDirectory]" "[SourceDirectory]\*.gz""/>
<!--The path to the fields definition file -->
<add key="Fields" value="[ApplicationDirectory]\FieldMap.xml"/>
</appSettings>
</configuration>