Event streaming

Read everything about our solid Event streaming module and see how we embedded into our platform. In this blog we use a use-case of a logistic supplier and walk you through the capabilities of our platform and navigate you through each of the five phases.

In deze blog

Hi there, eMagiz developers! A special delivery for you, in which we zoom into how you can use the Event Streaming module in eMagiz. Over the last period, we have taken a lot of steps towards releasing a solid Event Streaming module that can really help you solve your business problems.

It won’t come as a surprise therefore that we have lots to share with you today in this release blog. Among others, we made it possible to use the existing ILM phases in tandem with implementing Event Streaming in the platform. This means we offer a managed solution to orchestrate topics, the storage for these topics and offer a solution to filter out and/or transform messages between multiple topics. In this release blog, we will delve into the specifics of how eMagiz can be used to implement a managed event streaming solution.

To illustrate what the capabilities of the eMagiz platform are we will navigate through each of the five phases in eMagiz with the help of a use case of Event Streaming. The use case that we are going to set up is as follows:

  • As a logistic supplier of goods, one of my main costs is the fuel consumed by my fleet of trucks. To get a proper insight into the amount of fuel stops done by my fleet of trucks in my Transport Management System I would like to know, based on my on-board computer data when my trucks needed to stop for fuel.
  • In order to keep track of spend, I would like to send the ensuing receipt to my Financial Management System for processing.

Capture

The first step of developing any integration is filling in the Capture phase. In Capture you can easily connect systems to one another in a visual manner. The same applies for Event Streaming. See below the sketch made based on the use case described above.

On system level you can define whether the system(s) you want to connect have the ability to connect to an event streaming topic (managed by eMagiz). By selecting this option you can simplify your life as a developer, as eMagiz will already set up parts of the Design and Create phase for you based on the choices you make in Capture.


Design

In Design the first thing you see is your project overview. In this visual representation of your integration landscape, you can see which data is exchanged between which systems. If we zoom into the Event Streaming part of this overview you see that on integration level, we store the following information:

  • Number of ‘producers’ on a topic.
  • Number of event processors for which this data of the topic (integration) is the input of the event processor.
  • Number of event processors for which this data of the topic (integration) is the output of the event processor.
  • Number of ‘consumers’ on a topic.

This information helps you discern the number of producers and consumers on a specific data element that is stored on a topic.

Topic and Topic Properties

A crucial factor in setting up your managed eMagiz topics is the maximum size of the topic in bytes as well as the maximum retention period, these are the two of the topic properties that are configurable. These two drive how long information is stored and therefore drive the costs of the total topic storage that is needed to run all your topics within your landscape.

eMagiz gives you the possibility to change these values based on the expected throughput of data on topic level. To help you make the decision a pre-defined setting is given by eMagiz.

Topic Storage

On the Design Architecture in eMagiz you can see the amount of Topic Storage configured for your eMagiz project and you can also see the impact of your topic settings on the available topic storage.

This way you can assess the impact of adding new topics to your managed Event Streaming solution and prepare for expansion based on your current contractual agreements.

Event Processor

As you probably noticed in the Design overview we showed, there is one integration that has a red indicator. This red indicator tells us that we have a consuming system that wants to retrieve data from one of our topics, but does not yet have a producing system that can deliver the data to the topic.

If you recall, the use case was intended to filter out the fuel stops of our trucks. To make this possible eMagiz offers you the possibility to place an event processor between two topics.

The moment you have added the processor you will see a change in the Design overview as depicted below.


Create

To transfer your designed solution to Create you only need to select the integrations we have just designed in our add integrations overview.

Adding these integration leads to a nice overview in Create that shows the topic(s) and the event processor(s) in one overview.

You can even zoom in on parts of your event streaming solution by clicking on a topic, processor or system as illustrated below:

If your Event Streaming solution only had topics and no processors, the Create phase would be done as eMagiz automatically generates everything for you.

However in this case, as we use the event processor, we need to add the filter (and only the filter) to our event processor. The rest of the event processor is generated by eMagiz for you.


Deploy

The moment you are satisfied with the collection of topic(s) and event processor(s) you can navigate to the next phase of eMagiz, the Deploy phase. In the deploy phase, you make the transition from designing and building the solution to actually running the solution to make sure that others can reach the topics.

Luckily eMagiz is there to assist you in doing so. For example, eMagiz automatically creates the topics based on what you have designed and build in Create.

External users and rights

The only thing remaining for you is to register the external users and give them the correct rights on the specific topics. This to ensure that no breach of data can occur. To add external users you can easily navigate to the Event Streaming section in Deploy and select the tab called External Users.

After you have created each external user you can assign them the correct rights via the ACL tab under Cluster.

Configuration

Obviously the external parties, in this case the on-board computer, TMS and FMS need to be able to connect to the topics we have activated. To do so they need the following three configuration items:

  • Keystore (specifying the rights of a specific external user)
  • Truststore (checking if the external user tries to connect to the correct cluster)
  • Bootstrap Server URL (the URL you need to connect to a specific cluster)

All three of these configuration items are located in Deploy and are therefore easily transferable to the external parties that need to connect to the topics.


Manage

The moment you have activated the topics, set the correct rights per user per topic and have shared the configuration details with all external parties that need to produce or consume data from the topic, the solution works and data can be exchanged between parties.

However, to stay in control of the data traffic and see whether a consumer is falling behind eMagiz offers several statistics that can help you identify the amount of data stored on the various topics and see how many messages a specific consumer is behind (identifying the so-called consumer lag).

*As no traffic has taken place on this example no data is recorded and therefore there is also no consumer lag yet.


Learning

In this release blog we created a broad overview of the capabilities of the platform when implementing Event Streaming.

The moment you are interested in implementing your business case with Event Streaming via the eMagiz Platform you can browse to the Documentation Portal of eMagiz located within the eMagiz Platform to learn the specifics with the help of our micro learnings on the subject matter or contact us at productmanagement@emagiz.com for more information.


Look into the future

The next step for us would be to enable functionality that gives you the tools to analyze the events within the stream as a collection of events (i.e. determining the average fuel price on a certain day of the week over the complete fleet of trucks) instead of basing decision on a single event within the event stream and implement the logic afterwards in the application itself.

This would unlock even more potential for the use of event stream processing via the eMagiz Platform.


Key takeaways

Thanks to all that help build this nice feature, to those who tested and gave feedback to make the product even better. To end this post here are some key takeaways for you:

  • With the help of eMagiz you can easily manage your event streaming cluster.
  • eMagiz offers tools to transform or filter on event level making the solution more flexible.
  • eMagiz uses the visual representation in all five ILM phases to give you a powerful insight into what data is being processed, who has access and how much it is used.
  • We are continuously developing our product also for Event Streaming. If you see potential for improvement please let us know at productmanagement@emagiz.com

Let’s stay in touch and till next time!

* indicates a Beta feature. If you would like to get access to this beta feature, please contact productmanagement@emagiz.com

Twitter
LinkedIn
WhatsApp
Email