Custom Vision Ai Azure Iot Custom Vision AI with Azure IoT solutions

In my previous article, I wrote about the Azure Digital Twins service that enables the visualization of physical environments and associated devices. This time I would like to present another interesting scenario, in which IoT devices and Azure cloud services play a crucial part too, thanks to Custom Vision solution.

I talked about data generated by IoT devices, like temperature or humidity, but there are even more exciting scenarios. Imagine that there is an IoT device with a camera that can analyze the crowd (to see how many people are in the room or what emotions they are showing) or analyze packages on the production lines to detect damages.

If you did not hear about Azure Cognitive Services, let me introduce this topic and explain how these tools support the Internet of Things solutions.

What are Azure Cognitive Services?

Azure Cognitive Services are services and tools available in Azure cloud that allow developers to build intelligent applications without having direct AI or data science skills and knowledge. The main purpose of Azure Cognitive Services is to help developers create applications that can see, hear, speak, and understand.

The Azure Cognitive Services catalog consists of five main pillars:

• Vision – provides you with access to advanced algorithms for image processing and returning information. We will cover this area more deeply later in the article.
• Speech – adds speech-enabled features to applications.
• Language – allows the application to understand and process natural language to recognize what the users want to express.
• Web Search – return results on the basis of specific users’ requests.
• Decision – enables content analysis that improves the decision-making process.

Image analysis with Custom Vision service

As said above, the cloud-based Custom Vision service gives developers access to advanced algorithms for image processing and returning information. In a simple scenario, after uploading an image or providing an image URL, Microsoft Custom Vision algorithms can analyze visual content in different ways based on inputs and user choices and then return appropriate results. Below you can see an example of face analysis.

Another interesting scenario where Custom Vision service can be used is related to production lines and packages. For instance, this solution can consist of an IoT device with a connected camera to detect potentially damaged boxes.

How can we use Custom Vision service with IoT devices?

Internet of Things device market is growing fast right now. There are many manufacturers that provide IoT devices not only with integrated sensors (e.g. temperature sensor) but there are also devices like GPS trackers or devices with camera to constantly monitor the selected area. During the recent Microsoft conference called Build, there were many interesting announcements, including IoT devices for monitoring purposes. Below I present an example of IoT product from Lenovo which will be available in the Azure IoT Device Catalog this summer. This solution enables running Custom Vision module on IoT devices with camera.

Another great product released by Microsoft is the Azure Kinect camera. It enables real-time video analysis and works with Custom Vision service. I encourage you to watch this short video below.

What are some real use cases for Custom Vision service and IoT devices?

The main question is: How can we use Custom Vision service together with IoT devices? Let’s go over some examples.

Monitor workers on the construction site

Safety on the construction site is the most important aspect. Each worker should wear a helmet together with a jacket. Without the protective equipment, workers are not allowed to be on the construction. To avoid danger, it’s possible to implement IoT devices with a camera to analyze if workers wear helmets and jackets. If there is someone without the protective clothing, an alert can be sent, based on the Custom Vision analysis results.

Monitor packages on the production line

Another useful example of using IoT devices together with Custom Vision service is to monitor packages on the production lines. It would be much more complicated to manually check each package to see if there are any damages. In this case, IoT device with a camera could be used to analyze packages on the production line and alert if some of these packages are damaged or not properly closed.

Monitor crowd during the event

Crowd analysis can also be a good example of a case where Custom Vision service and IoT devices play a significant role. As an organizer of some event, you may need to measure how many attendees were in a specific area or check if there were more women or men. Of course, there are many other parameters you could collect and analyze.