This hands-on lab poses a set of challenges to get you started with hands-on experience to the Azure IoT stack.
The rough architecture that you will build in this training looks like this - if you follow the one path. However, your solution might look very different and still be absolutely valid!
- You need an Azure subscription. If you do not have an Azure Subscription yet, you can also do this lab with a free trial subscription
- You should have a general idea about Azure. This lab does not include an Azure primer.
- Very important: This training does not provide a step-by-step guide which you just need to click through or paste stuff around. Instead this lab requires you to figure solutions out on your own! If the instructions or hints mention concepts which you are not familiar with, please do not hesitate to either ask your buddies in the training or your favorite search engine :)
The each step describes the desired outcome. Often there are multiple ways to achive this goal. Every way to solve it, is correct! To help you towards a solution, each step gives you a few hints and links to documentation - mostly to the documentation of related Azure services. Do read those! Sometimes the hints also contain pointers to different alternative ways to solve a challenge.
Some services may take a few minutes to come up when you deploy them in Azure. Use the provisioning time to read further documentation or plan your next step for the challenge you are working on.
The lab is generally the following scenario: We want to measure certain readings from different environments on our factory shopfloor. As there are currently no sensors installed which would allow us to easily get their data for analysis, we decided to start with a quick and easy prototype. For this we bought a few cheap Raspberry Pi boards with a sensor-head installed. The sensor can measure temperature, humidity and pressure.
If our project proves some real value in getting and analysing this kind of data, we might later put some more ruggedized hardware in place. But for now it's really about getting the data out of the shopfloor and onto the cloud where we have a all the tools we need for quick prototyping without putting expensive hardware in our IT data center.
For this lab, we use a simulated version of the real Raspi board and the sensor-head. You can find it here: https://azure-samples.github.io/raspberry-pi-web-simulator/ The simulator is written in Javascript but you do not need knowledge of the programming language to use it. Although, if you do, feel free to extend the simulator later in the lab to send additional data etc.
The code is already all there, you do not need to do any more coding! The only thing it needs, is a connection string to an Azure IoT Hub. (Hint: The connection string goes into line 15).
So, first you probably need to provision an IoT Hub and then create a device identity for your simulated raspi.
You have data flowing from the simulated raspi into an Azure IoT Hub.
Create a new IoT Hub from the Azure Portal
For this step there actually is a full step-by-step guide. Only refer to it if you are really stuck! https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-raspberry-pi-web-simulator-get-started#register-a-new-device-in-the-iot-hub
To verify that our data is flowing into Azure and also to quickly see what the data looks like, we want to deploy a web-based frontend that can to used without much or any coding, schema definition etc.
There is the Azure Time Series Insights service which could suite well for this purpose. But again, you are also free to try other tools as long as they fulfil the goal of the challenge.
Without the need to define a schema on your data, you have deployed a tool that allows you to see your data in a web-frontend and drill into it to see the raw telemetry events.
One key concept in working with IoT data is the differentiation between hot, warm and cold data layer. An architecture that builds on this principle is also known as a lamdba architecture.
There are various reasons for the cold layer but to name just two important ones here:
- Our processing pipeline might change at some point in the future. If we keep the historical data in its original, unprocessed form, we can do a replay of that data into our new pipeline.
- Most analytics, and especially machine learning, happens in batch mode at some later point. For this we might not even know yet which data points exactly to keep. However, it might not be a very good idea from a cost perspective to load all the telemetry data in a fast but expensive storage such as an relational (SQL) database.
To start with this architecture, we now want to store our incoming data in an unprocessed form for long-term storage.
While there are various storage solutions available, we recommend to go with the easiest one here: Azure Blob Storage, which is a general-purpose object storage. It is integrated with also all Azure services.
Also, there are various ways in Azure how to achieve this and get telemetry data from the IoT Hub into storage. Two simple ways are worth looking into: Azure Stream Analyics (ASA) or IoT Hub routing. You will also use ASA in a later challenge, so it might be helpful to already get started with it here. IoT Hub routing on the other side is even easier to set up. Your choice!
The unprocessed telemetry events are flowing into a storage that suits the needs of a cold storage. The data should be stored in a common text format that can be easily read by any tool (e.g. CSV or JSON).
What is Azure Stream Analytics
Quick start with Stream Analytics
Now that we solved our cold-storage requirements, let's take a look on where we store data for faster access than in a cold storage. This will be our warm layer. Later we want to build some visual dashboards based on this, so the storage that we choose here should be optimized for that.
To reduce the amount of data that we need to store here (think about our costs!), we want to pre-process the incoming data. Our senor is sending data in a high frequency (every 2 seconds). For now we don't need that data in such high resolution for our dashboard. Hence, we only want to store the average readings of the temperature and humidity in 30 second buckets (a.k.a windows).
These kind of calculations on continuously flowing data are called stream processing. In Azure there is the fully managed Azure Stream Analytics service for this. However, if you are more advanced already, you can also take a look at Spark Structured Streaming with Azure Databricks.
There are various services available in Azure which we could use as our warm layer. To name just two possible options here: Azure Cosmos DB or Azure SQL Database. While they will both work here, they are very different offerings. One is a NoSQL database, whereas the other one is a more traditional relational database.
Telemetry data that was averaged over 30 seconds is being transmitted into a storage service that can serve as a warm storage to which we will later connect a dashboard solution to.
Introduction to Steam Analytics Windowing
Spark Streaming from Azure Event Hubs in Azure Databricks (Hint: this also works if your source is an IoT Hub)
Since now our pre-processed data is available in a database that can serve it for end-user reporting, it's time to build a first dashboard.
Again, there are many possible ways to solve this challenge. Anything from Excel to a custom visualization using Python. There is also the Microsoft Power BI tool which can be handy for this use case.
You have a simple but interactive dashboard in place that you can share with end-users to work with the telemetry data from your sensors. The data source is the warm storage that you built up before.
The dashboard could is showing for example average temperature and humidity for our device over time.
Note: If you cannot install the Power BI Desktop tool on your local machine (missing permissions etc.), you can easily just spin up a VM in Azure for the purpose of this lab. For instance, there is the "Data Science VM" template, which comes with Power BI pre-installed. You can deploy it to your Azure subscription from here.
Get data from Azure into PowerBI
Since we don't want to be in charge in the long run to provision and set up our devices manually, we plan to introduce some self-service for this in the future. However, we would still like to know whenever new devices get created in the IoT Hub or when a device connects for the first time. For this we would like to get a simple email notification.
Using services like Azure Event Grid and Azure Logic Apps, this is again possible without any coding.
Whenever a new device gets created in the IoT Hub or when a device connects for the first time, you will be notified by email.
React to IoT Hub events by using Event Grid to trigger actions
Getting started with Logic Apps
IoT Hub Events tutorial (this is the fast track for this challenge ;-) ) (Note: if you cannot use Office365 to send out emails from Logic Apps, there are also other connectors such as Gmail available)