How Utopus Insights Reduced Data Processing Time from Weeks to Hours

Deepak Prabhu • August 12, 2022

Going Serverless on Amazon Web Services

Renewable energy analytics provider Utopus Insights (Utopus) wanted to generate more computing power so that it could improve the performance of Scipher, its industrial Internet of Things (IoT) analytics solution platform that assesses the performance of wind and solar energy assets using data in near real time. When the company initially launched the platform, it ran the solution on premises, which required ongoing investments in hardware to scale its infrastructure to meet increasing demand.


To meet its data processing needs, Utopus explored going serverless and decided to migrate to Amazon Web Services (AWS) in 2018. A year later, Utopus became fully native to the cloud, which has helped it scale quickly, accelerate its time to market, and achieve compute cost optimization. The company has maintained high availability on Scipher and reduced the time that it takes to process data from weeks down to hours. By working on AWS and expanding its data processing capabilities, the company can focus on innovating its digital solutions and facilitating the global transition to clean energy.


Exceeding Data Processing Capacity on Premises


Founded in 2017, Utopus aims to accelerate the integration of renewable energy into the modern grid by optimizing energy production using data analytics and insights. Utopus serves over 235 customers globally and holds over 90 granted and pending patents across its suite of products, ranging from asset management coverage to power forecasting insights. This suite is underpinned by the company’s platform, Scipher, which collects data from wind and solar assets to drive reporting and actionable insights using machine learning (ML). Using this solution, businesses can proactively monitor their assets worldwide and predict maintenance needs or part failures, helping to increase the longevity and performance of their renewable energy assets. 


When launching Scipher in 2017, Utopus used an on-premises data center to power its digital solutions and process petabytes of energy data at any given time. As more customers adopted its solution, the company faced challenges in quickly scaling its infrastructure. “The amount of data that we had exceeded our processing capacity,” says Ziad Rida, technical director of architecture at Utopus. “Our main goal was to ingest the data and make it available for data science.” In 2018, Utopus engaged AWS to accelerate its data processing capabilities and optimize its compute resources. To test the viability of working in the cloud, the company developed proofs of concept working alongside the AWS team. Utopus received resources and technical training with subject matter experts, and it kicked off its cloud migration that same year. “AWS helps us scale more rapidly than we could have on our own,” says Michael Wilkinson, chief product officer at Utopus. “At a time when both climate change and energy security are urgent global concerns, making data-driven innovations on AWS is the key to unlocking reliable, smart solutions both superfast and at scale.”


Taking a Serverless-First Approach to Scale to Demand


When migrating to AWS, Utopus opted for fully managed, serverless solutions when possible. “Using serverless solutions on AWS means that we don’t have to worry about scalability, manageability, installation, or configuration,” says Rida. For instance, the company adopted Amazon Kinesis Data Streams, which companies use to easily stream data at virtually any scale. On average, the company streams 7 TB of data through Scipher every day. 


After streaming its data, the company processes it using AWS Lambda, a serverless, event-driven compute service that companies use to run code for virtually any type of application or backend service without provisioning or managing servers. “AWS Lambda is highly scalable and very powerful,” says Rida. “We can manage latency, make our processing faster, and decide how much we want to spend.” Using AWS Lambda, the company processes over 200 billion signals from wind and solar sources every day, and in under 2 hours, it can process the same amount of data that would have taken 2 weeks to process on premises. By gaining deeper and timelier insights, its customers can improve the performance of their renewable energy assets and minimize their carbon footprints. 


Decarbonizing the Electricity System on AWS


Utopus completed its migration to AWS in 2019 and continues to try new services to improve the performance of Scipher. To further reduce its time to market and introduce new capabilities, Utopus is trialing AWS IoT Greengrass, which helps companies build IoT devices faster. The company aims to accelerate the adoption of renewable energy across all industries. “Frankly, we couldn’t be doing the work of trying to decarbonize the electricity system without using AWS,” says Wilkinson. 


Ream more here.




News and Events

Real-Time Data Monitoring: The Key to Optimizing Renewable Energy Performance

By Deepak Prabhu • May 8, 2024
Centralize data from various platforms into a single interface with Scipher.Vx+ real time data monitoring software.

The Benefits of Power Forecasting for Renewable Energy Professionals

By Deepak Prabhu • May 8, 2024
As the global shift towards renewable energy continues, wind and solar power are emerging as crucial sources of electricity generation. However, unlike traditional energy sources like coal and natural gas, the effectiveness of wind and solar power is highly dependent on weather patterns. This makes accurate power forecasting digital solutions such as Scipher.Fx essential for wind and solar power producers. Maximizing Energy Production Wind and solar power producers heavily rely on precise power forecasts to maximize their energy production. By understanding the expected energy output from their renewable energy assets, producers can ensure that they meet the demand and avoid any disruptions in energy supply. Accurate power forecasting also helps in preventing over-generation, which leads to wasted energy and revenue. With Scipher.Fx, producers can optimize their energy generation and improve overall efficiency. Reduce Imbalance Costs Implementing power forecasting software such as Scipher.Fx can lead to significant cost savings for wind and solar power producers. By leveraging accurate forecasts, producers can effectively manage their energy generation, resulting in optimized operations, reduced maintenance costs, and improved labor efficiency. S cipher.Fx enables producers to avoid penalties associated with over-generation or under-generation, thereby preventing financial losses. Grid Integration Integrating wind and solar energy production with the power grid requires precise power forecasting. Accurate forecasts enable producers to plan their energy generation effectively and ensure a timely supply of the required amount of energy to the grid. This aspect is especially crucial in regions with high levels of renewable energy generation, as maintaining grid stability becomes a challenge. Regulatory Compliance Compliance with regulations pertaining to energy generation is a crucial responsibility for wind and solar power producers. Many regulatory bodies require accurate power forecasts from producers, which can be a daunting task without the appropriate software tools. Scipher.Fx provides wind and solar power producers with the means to comply with regulatory requirements effectively. By utilizing the power forecasting capabilities of Scipher.Fx, producers can ensure accurate forecasts, meet regulatory standards, and avoid penalties associated with non-compliance. As wind and solar power take center stage in the renewable energy landscape, power forecasting software like Scipher.Fx plays a vital role in optimizing energy production, achieving cost savings, enabling seamless grid integration, and ensuring regulatory compliance for wind and solar power producers. By harnessing the power of accurate forecasting, producers can unlock the full potential of renewable energy and contribute to a sustainable and greener future.

Forecasting to Asset Management: T.P. Aeolian Dynamics Journey with Scipher™

By Deepak Prabhu • August 24, 2023
Sotiris Shiacallis, Operations Manager at Cyprus-based wind farm operator T.P. Aeolian Dynamics, shares about the company's experience using Scipher.Fx and Scipher.Vx+. Shiacallis describes how renewable power forecasting and asset management digital solutions by Utopus Insights have become pivotal in optimizing their wind farm operations, enhancing forecasting accuracy, and driving O&M with reliable insights.
Show More