Northern Enlightenment – Splunking Canadian Weather Extremes

When I mentioned to folks that I had an upcoming engagement in Yellowknife recently, their first reaction was to make a joke about how cold it was up there and the weather in general. For those who are not up on their Canadian geography Yellowknife is “North of 60” and is the capital of the Northwest Territories. Once home to many gold mines, it’s now the diamond capital of North America and one of the best places to see the northern lights. So, to see what I was getting myself into, I decided that I should analyze the weather data for the past decade and see how things were going to shape up.

After searching the web for a few minutes I was easily able to get historical weather data from Environment Canada (http://climate.weather.gc.ca/). I could see a month at a time on their website, but there was no easy way to do comparisons and I had to download the data one year at a time as a CSV file and do the comparison myself.

Of course I knew it would be simple to use Splunk to do all the work for me and thanks to Splunk’s data input capabilities, it was very easy to import this CSV data with only minor tweaks to the configuration. This was all doable via the web interface and did not require any backend text config file changes. Here’s an example of the format of the data:

Sample CSV weather data

The dataset I’m working with is from 2003-2013. I noticed there were a few gaps in the data but that is ok for this purpose. Since I was to be in Yellowknife at some point during the fall, I focused my analysis from September to December. Sometimes just a simple view can be useful and so for all of my analysis, I am using an average of the values over these months, with the different years as the series.

I started with a simple view of the average temperature. The first thing I noticed was how over the past decade the variance in September is much smaller than the variance by the time we roll into December, with almost a 20’c difference between 2005 and 2013!!

Temperature analysis results

So from the look of things Sept/Oct are most likely going to be somewhat normal, but by the time November/December it will be a mystery as to how it goes.

Now that’s just the temperature. What about the snowfall?

snowfall analysis results

Here’s where you start to get worried being in northern versus southern Canada. There has previously been snow in September, but there has not been any yet so far this month. However, it looks like by October there will be some. November seems to be when it really snows and back in 2006 and 2008 they had a lot of snow – over 70cm! The last few years seems to be a bit lower which will hopefully stay true for this year.

One of the other factors to consider is always the windchill. That is when you can take a balmy -25C day and make it feel even colder!

windchill analysis results

Unfortunately, the dataset did not have any metrics on windchill values, but did have data on max wind gusts and we can see that for the past few years there have been some big gusts in the 40-45 km/h range, which will add at least another -10C to the temperature.

Now Splunk does have some capabilities that could be leveraged to try and predict what the temperature, snowfall, and wind gusts could be but I will leave that for another time. I am in no way a weather expert and I will leave predicting the real forecasts to weather people and climatologists, but I will say that Environment Canada is predicting a warmer winter this year so that might be good news. I know one thing for sure, once 2014 is done I will add that data to my index and see how it actually did.

If I wanted to be more proactive about this I could leverage some online services such as Weather Underground (http://www.wunderground.com/) or Forecast.io (http://forecast.io/) and pull live weather data and store it in Splunk. Doing that could let me do some more targeted analysis on more recent data and with more precision. I could also setup some alerts to notify me when the temperature drops below -10C for example. But, I might save that for another post.

D.

Looking to expedite your success with Splunk? Click here to view our Splunk service offerings.

© Discovered Intelligence Inc., 2014. Unauthorised use and/or duplication of this material without express and written permission from this site’s owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Discovered Intelligence, with appropriate and specific direction (i.e. a linked URL) to this original content.