State generation – Toegevoegde waarde door continu data inzicht

Over the last decades, large amounts of data have been stored in all kinds of databases, data lakes and data warehouses. Data sources that organizations often know they possess but which are rarely used to support real-time decision making. What if we introduced the ability to persist specific data points from existing 'stateless data' in transit to produce stateful data? In this blog Geert will tell you all about it.

In the last years, eMagiz has empowered many organizations with the ability to integrate any system, any application, any service and any file with each other. Primary business processes in these client environments rely on eMagiz to transmit data in a secure manner with guaranteed delivery of messages. In many cases data that was transmitted were orders, shipments, data updates, and data retrievals. Often parts of that data is already known by the receiving system. But in other cases, systems, applications and devices are sending stateless data such as temperature, number of clicks, or another value that is isolated from previous data transmissions. Never has eMagiz stored a single byte in these years – all data was regarded as data in transit. But what if we introduce the ability to persist specific data points from existing stateless data in transit to produce stateful data? What added value does that provide and what use cases could we solve? persisteren uit bestaande ‘stateless data’ in transit om stateful data te produceren? Welke toegevoegde waarde levert dat op en welke use cases zouden we hierdoor kunnen oplossen?  

In the last decades, high volumes of data are persisted into all kinds of databases, data lakes and data warehouses. Companies often know that they own lots of data sources, but they are seldom used to support real-time decision making. Data is mostly used for analysis and reporting, explaining the past and at best predicting the future.  

Furthermore, technology lacked the ability to capture data and turn that into an action, resulting in a dismissal of such capabilities by organizations. Technology improvements have been made in the recent years which were required to open up this capability. Among those you’ll find the requirement to filter out data – only relevant data points (or alerts) are to be submitted or interpreted. And the ability to define an action based on a data point or a derived data point called a state in this context (average values, thresholds reached, etc.).  

A state of an object can be for instance the number of clicks on a webpage in the last hour, the average temperature of a device in the last 30 minutes or the event that in the last hour a business process didn’t report a completed status. Lastly, the ability to combine several data streams to produce a new state is relevant to mention in this context.  

The state is the key to open up the ability for real-time decision making as it can be the trigger for an action or alert to a specific device or application. Data streams are all around us, from events generated in the supply chain, website clickstreams and server logs, to data produced by IoT devices. Companies are seeking to process this data instantly, or as close to real time as possible. eMagiz State Generation enables organizations to do this, letting them sense, query, analyze and process data in real-time. eMagiz State Generation gives customers the ability to develop value-add stream processing applications with the benefits of the eMagiz low-code platform.  

With this ability we believe we have taken a next major step in our journey to help companies to leverage integrations to add value for their own organization and customer base. eMagiz State Generation components can be used in any integration pattern we offer (Messaging, API Gateway or Event Streaming) to ensure clients can create the solution they need.

Please feel free to contact me directly if you have any questions.

By Geert-Jan Waanders, Product Manager @ eMagiz