Subscribe / Unsubscribe Enewsletters | Login | Register

Pencil Banner

How to collect and analyze data from 100,000 weather stations

John Brandon | June 17, 2015
If you want to understand what it takes to collect, track and analyze reams of data, just check the weather. There are constant fluctuations, scores of data points and intense interest from all over the planet. Analyze the data correctly and someone in the state of Washington knows whether or not to wear a raincoat. Do it poorly and there might be a massive traffic pileup from people driving too fast on slick roads.

If you want to understand what it takes to collect, track and analyze reams of data, just check the weather. There are constant fluctuations, scores of data points and intense interest from all over the planet. Analyze the data correctly and someone in the state of Washington knows whether or not to wear a raincoat. Do it poorly and there might be a massive traffic pileup from people driving too fast on slick roads. 

Bryson Koehler understands this dynamic. As CIO of The Weather Company, he is charged with increasing the accuracy of weather forecasting for the various entities the company owns, which include the Weather Channel and the Weather Underground mobile app. 

The app in particular uses a massive personal sensor network to increase accuracy. Even a smartphone can be a basic weather station: The Weather Company uses algorithms that can determine the outside temperature for that user based on what the phone is reporting. 

There are 100,000 sensors sending data worldwide (and 40,000 in the United States alone). Understandably, processing the data is no easy task. 

"Some of the data is interesting such as lightning data or pollen data and it doesn't always help us create a forecast, but we can tell people who have allergies what to expect," Koehler says. "Other types of data we get in real time, such as aircraft telemetry data installations on commercial aircrafts that we bring down in real time to see atmospheric conditions." 

Koehler says the flight data is incredibly helpful. It can be used to alert airlines about possible changes in flight plans, or let them know the wear and tear on a plane is not as significant as it might have seemed during a flight. This data can help minimize delays, since the airlines are required to do extra safety checks related to severe weather. The Weather Company can tell if the real-time weather data did not reach as high a threshold as the pilot might have reported. 

The analysis is intense. Stations provide data for humidity, barometric pressure, dew point, UV load, rainfall, wind and many other factors. There are billions of reports sent in each month, according to Koehler. The station data is repurposed into a format people can use and understand. 

"People can pull up different layers of maps, and they can pull up forecasts from all over the globe," he says. "In contrast, the National Weather Service in the U.S. has about 3,500 recording stations that they own and operate on behalf of U.S. taxpayers." 

More instruments mean more data

It is an interesting dilemma to have such an abundance of data to process. Koehler says that the NWS is one of the world's most "most instrumented" government agencies. Yet, the Weather Company has to deal with many thousands of personal weather stations worldwide. Some of the stations are not easily accessible they could be in a remote region of Iceland. Some of the weather sensors are as small as a Coke can and some involve an antenna that is three-feet tall. 

 

1  2  Next Page 

Sign up for CIO Asia eNewsletters.