Azure Custom Vision Integration with Azure IoT Hub

6 min readAug 15, 2024

Introduction

In the era of the Internet of Things (IoT), the integration of machine learning and artificial intelligence with IoT devices offers transformative capabilities. Azure Custom Vision, a part of Azure’s suite of AI services, allows developers to build custom image classifiers that can identify objects and patterns in images. By integrating Azure Custom Vision with Azure IoT Hub, you can create a powerful solution that leverages image classification models on IoT devices, enabling automated and real-time image analysis.

Why Integrate Azure Custom Vision with Azure IoT Hub?

Integrating Azure Custom Vision with Azure IoT Hub offers several benefits:

Enhanced Automation: Automatically classify and analyze images as soon as they are captured by IoT devices.
Real-time Insights: Gain immediate insights from images, enabling quick decision-making.
Scalability: Deploy custom vision models to multiple IoT devices, ensuring consistent and scalable image analysis.
Edge Computing: Run machine learning models at the edge, reducing latency and bandwidth usage by processing data locally.

Solution Overview

Lets start with the solution part to see how we can achieve this

Uploading Images: Lets Start by uploading images to Azure Custom Vision. Tag and classify these images to create a labeled dataset for training.

2. Tag them: I have tagged each and every image based on different compressor gauge level.

for ex, here the meter reading is at 7 so I have tagged them as 7

Like wise if you have a single image and if you want to classify multiple objects yes you can do it. You just need to tag them accordingly.

3. Quick Prediction: Once you tag the images you are good to run prediction. One of the important point to remember to run prediction you need to have atleast 15 images with each tag or else you will receive error message.

4. Train: Once you are ready you can click on Train button, it will take atleast 5–7 mins based on the size (sometimes it may vary).

5. Performance: Once train is completed you can see the performance results for each and every tagged item. It will give the results based on precision, recall and mAP.

you can find more about these 3 classifiers here: https://learn.microsoft.com/en-us/legal/cognitive-services/custom-vision/custom-vision-cvs-characteristics-and-limitations

6. Publish: Once you train them based on your requirement, you will get an option to publish them.

7. Quick test: Now in order to test published model to make sure it really checks images and classifies them. you can click on “Quick Test” → lets upload a sample image file.

Here you can see the result, it has detected our image which was tagged with 0 when gauge was at 0.

Enhancing Custom vision with Azure IoT / Industrial IoT.

Now we have seen solution works and it can do object classification. In order to Integrate with IoT/ Industrial IoT solution where you might be generating lot of images and you may need to identify the issue as soon as it gets detect.

Step-by-Step Guide

In order to setup integration between Azure Custom vision to Azure IoT, lets get the package. Once you publish your model in Azure Custom vision you can download the package based on different types.

2. So in our case I am going to download docker file for Azure IoT Edge → with OS as Linux.

3. Now lets open Azure IoT hub →

Add a new Edge module → I am using image URI as python:3.11.3 (choose the latest version)
Make sure to pass required environment variables.
Under Container create options → add required hostname → add working directory, any other parameters such as hostconfig details and finally entry point.
and finally add required settings under Module Twin settings.

Finally Apply and Deploy.

After it gets deploy please check runtime status if it is running or failed(failure).

Docker Package deploy

Now the edge module is ready we need to make sure other supporting files also have to be setup.

Based on your edge device → login to your IoT Device → check your docker container to make sure all required files have been deployed. In my case I am using Industrial IoT → so i go to specified path and check all the files.

open hostname file → and it will show the right deployed name.

2. Now we need to push previously downloaded zip file contents, it includes Prediction python file and TensorFlow file.

I have uploaded couple of images under “test_images” folder.

Testing Results:

Now we need to test if my IoT Edge device picks up the images and show some predicted values.

Now for testing I have created sample python file which can detect images and show the prediction.

import io
import json
import logging


from flask import Flask, request, jsonify

from PIL import Image

from predict import initialize, predict_image, predict_url

app = Flask(__name__)

app.config['MAX_CONTENT_LENGTH'] = 4 * 1024 * 1024

if __name__ == '__main__':
    
    img = "file:///home/host/test_images/20240805133120_poi3.png"

    logging.basicConfig(level=logging.INFO)
  
    initialize()
          
     try:
        results = predict_url(img)
        print("Complete")
        print(results)

    except Exception as e:
        print('EXCEPTION:', e)

2. Lets test it from portainer.io → go to specific “vision_ai” container → click on Console →

In a console window → running it as

python vision.py

Node-Red:

Lets try the same using Node-Red to injest some images and check its prediction. Just created a sample flow to inject a new test image and function will check the payload and then populating an output.

Now cross validate the same with our Azure Custom vision to make sure IoT edge device is really picking up and it matches the result.

I am getting right TagName and then probability is around 99.1%

Overall steps Reference:

1. Uploading Images to Custom Vision

2. Training the Model

3. Quick Prediction

4. Publishing and Exporting the Model

5. Deploying to IoT Hub

6. Automated Predictions with MQTT

Conclusion

Integrating Azure Custom Vision with Azure IoT Hub empowers developers to build intelligent IoT solutions that automatically analyze and classify images. This integration enhances automation, provides real-time insights, and supports scalable deployments, making it a valuable addition to any IoT ecosystem.