Azure Time Series Insights For Iot Devices Azure Time Series Insights – How To Harness The Power Of IoT Devices

When you’re dealing with thousands or more of IoT devices, it is difficult to monitor them all at once. However, using the new technology from Microsoft, Azure Time Series Insights, it is possible to organize your data and analyze them in a relatively simple way. It can help you find device issues early and enable features like predictive maintenance. Here is our experience of working with this tool and some tips on using it with your devices.

I hope you’ve enjoyed my previous article about Power BI and its integration with Dynamics 365. This time we will discuss another analytical tool. I would like to share my and Predica’s team’s experience with implementing a complex solution for IoT devices. We used Azure Time Series Insights as one of the core components to organize data.

I will explain the benefits of introducing this software to the project. Additionally, I’ve included a demonstration of querying data using C# code. With the use of web API, it is quite easy to implement. Let’s get to it!

How does it work?

In my view, Azure Time Series Insights (TSI) has two sides.

The first one is a user-friendly analytical tool for data aggregation and visualization – perfect for monitoring trends of your IoT devices using line charts or heat maps. You can do it without any programming knowledge, the only thing you need is a little analytical thinking and intuition.

With the use of programming, we can access the second side which has much wider capabilities. Using available APIs, you can gather raw data and aggregated data, taking into account the different conditions. More about this later…

Time Series Insights dashboard gives you an at-a-glance view of your devices’ performance

Introducing Azure Time Series Insights to the project

Microsoft has recently announced the release of a public preview of Azure Time Series Insights, and it was a signal for us to consider whether it could help us with the project. If you want to know more about what Azure Time Series Insights is, please visit its dedicated web page. In this article, I will focus on our experience with the service.

The Microsoft team describes it as follows:

(Source)

The project where we used Time Series Insights, is for a global manufacturer. We are delivering a complex analytical system for device monitoring and predictive maintenance.

The whole solution is fully cloud-based and consists of Azure services. We are implementing this project in line with CFS architecture, but also expanding it with additional elements.

How to reduce the number of services with Time Series Insights?

Azure Time Series Insights only supports the IoT Hub and Event Hub as direct sources. In Microsoft documentation located under this link, you can find a handy guide, explaining how to connect your devices to Time Series Insights and how to configure your service correctly.

However, once connected to devices, how to deal with data aggregation, how to store them efficiently and visualize? This is where Azure TSI comes in.

In my current project, data flowing from the devices must be aggregated in a very flexible way, mostly depending on user preferences. For this purpose, Stream Analytics sends data to Azure SQL, and then the web API gets data directly from the database.

We used stream analytics and an SQL database using advanced logic to aggregate data. Stored Procedures and many views were included to retrieve the most recent data points for a particular device. However, this became problematic in terms of performance and cost generation when the number of devices exceeded several hundred thousand.

After the introduction of Time Series Insights, we have reduced the number of services because we no longer required Stream Analytics to preprocess this data. We’ve also replaced the storage and database procedures with ready-made mechanisms implemented in the Azure TSI.

The first figure below shows which Azure services were used before the introduction of TSI, and what the architecture of the solution looked like.

Figure 1. Architecture before Azure TSI implementation

These elements of the architecture form only a part of the project related to data aggregation, which is what we’re focusing on right now. The whole architecture is much more complex and is made according to the Connected Field Services concept.

The next diagram shows how it looked after the introduction of the Azure TSI.

Figure 2. Architecture after Azure TSI implementation

What are the benefits?

To summarize, here are the main advantages of adding this tool to our project:

1. Reducing the number of services, and therefore costs – thanks to replacing Stream Analytics and databases which we no longer needed
2. Simplicity – the whole logic of data aggregation is prepared in one tool
3. Real-time analytics – there is a live data preview via line graph and a heat map
4. Flexibility – the solution is accessible via APIs

The downside of this decision was the fact that it is a very fresh tool, and development is definitely longer. It’s because at this early stage, some errors came up within the documentation (however, they were resolved through our intervention).

Data aggregation and visualization

Now you know more about the project, I can demonstrate the greatest strength of Time Series Insights. It is amazing to see how quickly you can aggregate data and visualize it at the same time – assuming we are dealing with several hundred thousand devices sending data every few seconds.

During the project, we used several key capabilities related to aggregations:

1. The ability to freely manipulate the size of the aggregation window
2. Easy data range manipulation
3. A heat map chart where we can clearly see any anomalies.

All these functionalities merged into one tool, allowed us to prepare solution where we could quickly detect devices that weren’t working properly. Moreover, we could use aggregated data from Time Series Insights and present it on an external portal.

The following video shows you how you can start your journey with data analysis in Time Series Insights.

Dawid is demonstrating how to simply analyze data in Azure TSI

Of course, this is not a tool for visualizing different types of data, because we have a restriction on chart types here: both of them work along the time axis. We do not have the ability to create our custom metrics or edit data either.

However, I don’t think you will find a faster tool to slice and dice your aggregated data flowing from IoT devices.

With Azure TSI charts you can view how your devices are performing over time

C# showcase

As promised, here are a few tips on how to prepare a good JSON request to collect aggregated data from the Azure TSI, containing key and useful sections.

From the following video, you will also find out how to collect raw data.

Learn about the methods of analyzing data using C#

In my opinion, the most useful parts which you should remember are:

1. Build easy to manage down-sampling based on dynamic aggregation window.

2. Predicate with the ‘AND’ property – you will gain the possibility to set multiple conditions to filter your data.

3. SearchSpan clause – this is a mandatory clause to define the date range for your data.

If you need more explanation about collecting data using C#, contact us. I will be happy to share my experience with you.

Conclusion

Azure Time Series Insights can solve your problems with fast data aggregation and monitoring. It is a great tool that helps you reduce the number of services and prepare architecture.

If you are planning to implement an IoT solution, don’t hesitate to contact us. We will help you plan architecture in the cloud and apply it quickly.

And if you have any questions, I would like to invite you to my upcoming Q&A session about this tool and other analytical topics. Stay tuned for more news on our Facebook page.