To get the speed of insight required to deliver this real time data, Open Energi first shifted from an SQL Server database to a Hadoop environment in 2015, before embracing more streaming analytics.
"At the time our database was under increasing pressure, so we looked at the Hadoop ecosystem," Bironneau said. "We wasted a lot of time trying to set it up from scratch on Windows, and if you have ever tried that, don't. It is an absolute nightmare. So we downloaded the Hortonworks sandbox on Windows and got up and running with a proof of concept data warehouse in a couple of weeks."
He said that this move brought the infrastructure spend down to a fifth of what it was spending with SQL Server. This move to Hortonworks also opened the company's eyes to streaming analytics capabilities with the Hortonworks Data Flow platform.
So the data science team started to test out some machine learning algorithms and real-time analytics for customer assets.
"We work in an environment which is traditionally very data poor where you have limited control," Bironneau said. "So we thought if we can take the data that we have captured over the last five years and apply it to a data-poor environment fast enough, what kind of savings could you achieve?"
The key is what Bironneau calls "real-time validation".
"Because all of our forecasts happen in near real time for events that will occur within the next five minutes to hour, you get an observable quantity back to see if your original forecast was right or not," he explained. The data science team can then start to implement counters to the errors in the forecast so that the system gets smarter over time.
This has opened up capabilities like "peak price avoidance, tariff optimisation, and imbalance services, which when you bundle these things together, the uplift can be almost a twofold increase in revenue for customers," Bironneau said.
Sign up for CIO Asia eNewsletters.