HPE’s vision for helping customers is to slash the time it takes to turn ideas into value, which is transforming industries, markets, and lives. The company’s InfoSight product aims to deliver on this vision by providing customers with insight into their infrastructure in seconds and minutes, rather than hours or days.
InfoSight works by receiving infrastructure information from a network of over 20 billion sensors deployed at data centers all around the globe, which send trillions of metrics to InfoSight every day.
“We’re watching what’s going on in the environment and putting the data through expert systems using various AI and machine learning techniques,” says Jeff Dutton, Data Platform Architect for HPE InfoSight. “The goal is to tell the customer what to do to prevent problems and optimize their experience.”
Increasing demand from customers and internal stakeholders at HPE to deliver insight in real time necessitated a major overhaul to the InfoSight architecture, as Dutton explains:
“Moving to a near real-time user experience requires an infrastructure that self-heals, scales massively and is always available to process streaming workloads, no matter what. This is fast becoming table stakes for every business.”
The speed by which Dutton and the team could deliver insights to customers was gated by the time it took to run batch processes against very large, extremely complex data sets. To deliver value faster, InfoSight needed to evolve beyond its classic, batch mode, big data architecture.
As an initial experiment, the team migrated from an enterprise database to Apache Spark to see if faster batch processing was the answer. However, they quickly realized they would need to transition to work with the data in real time to realize their business vision. This meant introducing stream processing.
“Batch analytics is something nearly every developer and architect understands fundamentally, regardless of their tech stack,” notes Dutton. “Batch continues to play an important role in the InfoSight architecture; it’s leveraging several petabytes of historical data. The source for near real-time insight, however, becomes the stream. And it’s the streaming part of the journey that has been the most challenging.”
Dutton turned to Akka Platform from Lightbend for the elasticity and resilient self-healing required to deliver big data at speed.
“I have been working with Lightbend since 2014 and knew they could deliver on the elasticity and resilience we required,” shares Dutton. “When they packaged their Akka Platform stack with Apache Kafka, Apache Spark, and a multitude of other technologies into a de facto fast data distribution, it made good business sense to use their platform for the next generation of InfoSight.”
Akka Platform provides HPE with microservices frameworks for processing continuous application logic; multiple streaming engines for handling tradeoffs between data latency, volume, transformation, and integration; machine learning and deep learning tools for applying algorithms from the data science team; and intelligent management and monitoring tools for reducing the risk of running this always-on system in production.
Data services in a dynamic, streaming application must run forever and be able to scale horizontally across multiple nodes. This is a fundamental advantage of Akka Platform’s concurrency model, which is built around independent, self-contained processes that communicate asynchronously via messages. This means that they can be distributed and run across many machines.
In addition to scaling elastically, Akka Platform enables the HPE InfoSight streaming applications to recover from failures quickly. Failure is treated a first class concept: failures are normal, anticipated, and handled within the application, which enables self-healing.
Dutton’s approach is fast becoming a reference for other HPE business units looking to deliver similar real-time customer value. The solution that he and his team have built with Akka platform for HPE InfoSight is making such sound business sense that HPE Storage and HPE Pointnext are now teaming to bring the streaming solution to HPE’s top customers.
Download PDFHewlett Packard Enterprise (HPE) is the global edge-to-cloud platform as-a-service company that helps organizations accelerate outcomes by unlocking value from all of their data, everywhere. Established in 2015, HPE works with around 90% of Fortune Global 500 companies, serves customers in more than 170 countries, and generated revenues of $27 billion in 2020.
