6 More Columns Added to Environment Canada Climate Stations (Shapefile) to Help Check Data Availability

Quick Download Links

CSV, Shapefile, KMZ, ZIP

Please note that this data was generated on Dec 31, 2013 and the last day of data is depend on the day when the data is retrieved from EC website. If one of the last day is 2013-12-30, it doesn’t mean there is no data after Dec 30, 2013. I will update the data once a while to make sure the dataset is up-to-date.

Background

  1. Environment Canada climate data is available in hourly, daily and monthly. However, this is not true for every station. Some may just have daily and monthly data and don’t have hourly data.

  2. Some stations stopped working at some point and don’t have up-to-date data. For these stations, only a period of data is available.

For climate station selection, it’s useful to know which type of data is available from a station and the time period of available data. These information could be compared with simulation period of a model to help select right climate stations.

Solution

Fortunately,  these information could be retrieved from EC website. Following columns was added to the CSV and shapefile published in previous blog

  1. HOURLY_FIRST_DAY: First day of hourly data; will be null if there is no hourly data

  2. HOURLY_LAST_DAY: Last day of hourly data; will be null if there is no hourly data

  3. DAILY_FIRST_DAY: First day of daily data; will be null if there is no daily data

  4. DAILY_LAST_DAY: Last day of daily data; will be null if there is no daily data

  5. MONTHLY_FIRST_DAY: First day of monthly data; will be null if there is no monthly data

  6. MONTHLY_LAST_DAY: Last day of monthly data; will be null if there is no monthly data

Please note that the name of these columns in shapefile is shortened due to the limitation of shapefile on column name.

Advertisements

Environment Canada Climate Stations (Shapefile and KMZ)

Quick Download Links

CSV, ShapefileKMZ, ZIP

Updates

Dec 31, 2013

To help determine the data availability of each station, 6 more columns (HOURLY_FIRST_DAY, HOURLY_LAST_DAY, DAILY_FIRST_DAY, DAILY_LAST_DAY, MONTHLY_FIRST_DAY, MONTHLY_LAST_DAY) are added to show the available time range of hourly, daily and monthly data. Please note that the name of these columns in shapefile is shortened due to the limitation of shapefile on column name.

Available Data

EC stations data is the basic information of all EC climate stations. It’s available in three formats.

  1. CSV, which has following columns: ID, NAME, PROVINCE, LATITUDE, LONGITUDE, ELEVATION. The ID column is the station ID required in ECReader .
  2. Shapefile, which is created from the CSV file with “Add XY Data” tool in ArcMap. Geographic coordinate system (GCS_North_American_1983) is used. The columns are same with CSV file.

Please note that the 5 stations in the right bottom corner has missing latitude and longitude in the Environment Canada website. In the data here, the latitude and longitude is set as 0.

  1. KMZ, which is converted from shapefile with “Layer To KML” tool in ArcGIS. It’s intent to be used in Google Earth for further analysis.

A zip file, which contains all these three formats, is also available.

Calculation of Latitude and Longitude

Latitude and longitude is calculated based on information given in the daily data report page. The degree, minute and second format displayed on the page is converted to decimal degree using following equation.

Decimal Degree = Degree + (minute + second/60.0) / 60.0

For climate station 100 Mile House in BC,

Latitude = 51 + (38 + 49.2/60)/60 = 51.64695

Longitude = 121+ (18 + 9.06/60) 60 = 121.302516666667 (saved as -121.302516666667)

Steps Implemented in ECReader to Generate the CSV File

  1. Get all stations using “Search by Province” in Advance Search. 8467 stations are displayed in several pages. In each page, read station ID, station name, province and available data type (hourly, daily and monthly) and corresponding time period of each station. The station ID and available data type is save as hidden input in the result page.

  1. For each station, go to the daily report page and read latitude, longitude and elevation.

  1. For each station, write station ID, name, province, latitude, longitude and elevation into the CSV file. More information is available.

These steps are implemented in ECReader as simple as clicking a button. Please note that this button is not added to the interface for performance consideration. In case you want to test this function yourself, please reply at the bottom of this blog.

Background

When a SWAT model is setup, it would be convenient if the location of all EC climate stations is available in some GIS formats, which could be used to determine which stations should be used in the model.

EC doesn’t give this map. Instead, a tool is supplied in the Advanced Search to help locate stations. It’s not good enough for GISers, who wants to work in a more visual way. I’m that guy.

Google internet, a shapefile is found here. Interestingly, it happens to be created by University of Guelph, where I worked for three years starting from 2010. I was in Department of Geography, which also should be where the data come from. I could know the professor or student who created this data. Anyway, this data is just meant to handle stations with climate normals and there are 1480 stations compared to 8467 stations got from advance search in EC website. Apparently, we couldn’t use this data for the modelling work.

Solution

EC give the location (latitude and longitude) of each station in the daily data report page. Retrieving this page for all 8467 stations and then reading the location from the page would be a perfect solution for computers.

At the same time, it’s possible to get the station ID required in ECReader and create a simple lookup relation between station name and station ID, which means user won’t need to find the station ID themselves as before. ECReader would be more friendly.

Implementation

ECReader is modified to do this work. The response from EC website is in html format. To read this, System.Xml.XmlDocument is first used. It didn’t work well on parsing html. Exceptions everywhere. Then, 3rd party library HtmlAgilityPack was found and it was perfect.

The process is implemented as a static method of class EC. It saves all the information in CSV format. To get all the stations, depends on the computer configuration, half an hour is needed.

The final data is compared with data generated by University of Guelph. All the stations are in the same location.

Future Work

There are several ways to utilize the station data in ECReader in the future development.

  1. Lookup station ID from station name. No one want to get station ID himself.
  2. Add more methods to define working stations, including the interactive map selection (import watershed boundary or select manually on the map) which could be implemented with DotSpatial.

Uncompleted Data Bug Fixed, Please Update

Please follow my blog to get the latest updates on ECReader as soon as they are available.

After releasing ECReader (Environment Canada Climate Data Reader) , several users reports that the data of some years is uncompleted. It does start from Jan 1st but would end at any arbitrary day of that year, which will make the result files unusable. Since the program doesn’t check if the data is completed, the problem could be only found in manually data analysis. It’s a vital bug. Sorry for any inconvenience.

Good news is the bug has been fixed. It’s highly recommended to update your program and re-download your data. The new version is located in the same place as the previous version. Please click here to download.

Other minor changes are also added to this version and the structure of the package has changed. Some files are removed. It looks like this.

Other minor changes are list below.

  1. The program got an icon .
  2. No more DLLs. Only one executable file.
  3. Button is removed. Instead a help link is added.
  4. is added to select all data types for free format txt/csv.

How to Get Environment Canada Climate Station ID From Name

  1. Go to http://climate.weather.gc.ca/.
  2. In Step 1, select “Daily”.
  3. In Step 2, choose a year in which there could be some daily data.
  4. In Step 3, input the station name

  1. Hit the “Search” Button.
  2. If more than one station is given, select the station you want to use.
  3. Then you will be navigated to the daily data report page.

The address of this page would be in following format:

http://climate.weather.gc.ca/climateData/dailydata_e.html?timeframe=2&Prov=MAN%20&StationID=29886&dlyRange=2000-01-01|2013-12-11&Year=2013&Month=12&Day=11

A key named StationID will be found in this address. The value for this key is the station ID we need. In this case, the station ID is 29886.

Environment Canada Climate Data Reader

Please follow my blog to get the latest updates on ECReader as soon as they are available.

Please download the program package here and the quick guide is here. The .NET source codes is host in Google Codes.

Questions, bugs, more formats and functions, please send me email. I couldn’t be more happy to hear from you:)

Updates

Jan 10, 2014 Save and Load Defined Stations in ECReader1.1

Jan 6, 2014 File Name Convention in ECReader1.1

Jan 2, 2014  Version 1.1, Modify how to define query stations

Dec 28, 2013  New function to retrieve all EC climate stations and save as CSV file, based on which shapefile and kmz file could be also generated to help select climate stations in modelling.

Dec 20, 2013  Fix the uncompleted data bug.

What We Have

In Canada, Environment Canada is the major source of the climate data (precipitation and temperature). The climate data could be downloaded from Environment Canada website (http://climate.weather.gc.ca) for each year in CSV format which contains 27 columns data. To be used in ArcSWAT or SWAT, these files must be processed in following way:

  1. Extract min and max temperature and total precipitation from each file;
  2. Combine the data of different years from the same station into one dbf or txt file in a format required by ArcSWAT.

Let’s take a look. If 5 climate stations are located in/around the study watershed and the simulation period is 20 years, total 5*20 = 100 CSV files need to be downloaded first and then be processed into 5 dbf/txt files. It may take one or two days to finish this work. No modeler wants this.

Besides time consuming, this process is also error-prone. Large amount of files will be process in a same way. The brain and eye is easy to become tired. Wrong file may be processed or the same file may be processed twice. And the most important is that it’s difficult to find these errors. Modeler usually never check the climate data again after they are there, which leave it impossible to figure out climate data related problems in the model.

What We Could Have

Since the climate data preparation process involves many repeated work, a program is the natural solution. Select stations, click a button and then go to get a coffee. All the files are ready when you get back. Ideally? Yes, that’s the “Environment Canada Climate Data Reader” is trying to do.

Environment Canada Climate Data Reader, or just call it reader, is a .NET windows program designed to download climate data from Environment Canada and generate result files in a specific format (of course, it supports ArcSWAT dbf and txt format). With it, you could enjoy your coffee when doing workJ

Quick Guide

   1. Give Stations – Which stations will be used?

Station ID is the unique ID used by Environment Canada to identify each climate station. Before using reader, these IDs must be retrieved based on station name. If you don’t how to get the station id, just hit and a short tutorial will show you how to do that.

Already got station IDs? Let’s give the ID(s) to the reader first.

  • Only got one station? Good, just select option and input the station id in the box. Want to know the information of this station? Hit to check basic information of this station (to make sure correction station is used).

  • Got more than on stations? That’s normal. First prepare a csv file in Excel or any test editor, in which the first column is the station name and the second column is station id. It may look like this:

name,id

deerwood,29886

CSV file is ready? Select option and hit to load the file.

   2. Give Time Range – What’s your simulation period?

The simulation period may or may not be decided yet. No problem, just make it longer. ArcSWAT and SWAT would read the right data based on date.

   3. Select an Output Folder – Where do you want to save the result files?

The default output folder is the my document folder. Hit if you don’t like this. will open the given folder, which will allow you check the result files.

   4. Select an Output Format and Data Fields – What format do you prefer and Which data do you like?

Want to just use climate data in ArcSWAT? No problem, just select or and you are good to go (only min and max temperature and total precipitation will be exported).

Want some free air and do some analysis on the data? No problem, select or and select any climate data you want in the list.

Want the final SWAT format? Yes, for people who don’t use ArcSWAT, just select (only min and max temperature and total precipitation will be exported).

   5. Start Download and Drink Coffee

Ready? Hit to start the work. Progress message will be given on the bottom.

When you see , it’s done.

   6. Check Result Files

Curious about the results? Hit to open the output folder and open the result files. You may want to know the name convention for different format.

  1. P[Station ID].dbf for precipitation and T[Station ID].dbf for temperature, e.g. P29886.dbf and T29886.dbf.
  2. P[Station ID].txt for precipitation and T[Station ID].txt for temperature, e.g. P29886.txt and T29886.txt.
  3. [Station ID].txt, e.g. 29886.txt.
  4. [Station ID].csv, e.g. 29886.csv.
  5. pcp1.pcp for precipitation and tmp1.tmp for temperature.