Real-Time Data Ingestion and Analytics with Azure and .NET

The article explores the challenges of managing continuous data flows from a multitude of devices and outlines a basic architectural overview for building telemetry pipelines. It highlights how combining the Azure ecosystem, Azure Data Explorer, and .NET enables efficient data collection, real-time data ingestion across various protocols, and the transformation of data into actionable intelligence with ultra-low latency.

In a hyper-connected world, data is constantly flowing from billions of devices, probably even your toaster. There is a significant challenge not just in capturing this data, but also aggregating it across various protocols, processing it in real time, and storing it for meaningful analysis. This article will show how to use Azure and .NET for this purpose.

Azure ecosystem

Azure provides a structured and enterprise-grade architecture to ingest, route, and process high-volume, continuous telemetry streams (logs, metrics, and IoT). Azure IoT is a collection of services for connecting, managing, and deriving intelligence from IoT devices and industrial equipment at scale.

Azure IoT support the following protocols:

• Modbus (RTU/TCP) – Azure IoT supports Modbus through custom IoT Edge modules, which can act as protocol converters
• M-Bus (Meter-Bus) – M-Bus communication can be enabled through IoT Edge by using protocol adapters or creating custom modules that communicate with M-Bus meters.
• TCP/IP – Native support for TCP/IP communications is integrated into Azure IoT solutions.
• BACnet (IP-based) – Azure IoT can integrate with BACnet devices via gateways that convert BACnet/IP to a compatible protocol for Azure, such as MQTT or HTTP. IoT Edge can also be used to manage these interactions.
• LoRa – Azure IoT Hub supports LoRaWAN via integrations with external network servers like The Things Network (TTN) or by using LoRaWAN-compatible IoT Edge modules that convert LoRa data for cloud ingestion.
• RX/TX meters – Using IoT Edge or custom hardware, data can be captured and forwarded to Azure IoT Hub using supported protocols like MQTT, AMQP, or HTTP.

From all those devices and protocols flows data to the ingestion layer. It’s a layer where data is collected from multiple sources and transported into a central storage system for processing and analysis. Azure provides a suite of specialized ingestion tools:

• Azure IoT Hub – Enables highly secure, bidirectional (two-way) communication with millions of connected devices. Natively speaks standard IoT protocols including MQTT, AMQP, and HTTPS.
• Azure Event Hubs – A massive data-streaming pipeline which acts as a pure, high-throughput data scraper designed to ingest millions of events per second.
• Azure Event Grid – The reactive traffic controller used for discrete, event-driven routing based on specific triggers

Telemetry is only valuable if you can act on it immediately, so there needs to be an analytics layer. Azure Data Explorer (ADX) is a fast, fully managed big data analytics service optimized for real-time, time-series analysis. If the ingestion layer acts as the plumbing that transports your data, ADX is the powerful engine at the end of the pipeline that processes it.

To balance performance and speed, ADX offers two distinct ingestion methods:

• Batched Ingestion – Optimizes the traffic and reduces costs by grouping incoming data and processing it every few minutes.
• Streaming Ingestion – Designed for ultra-low latency, making data searchable within seconds of leaving the device.

ADX uses Kusto Query Language (KQL), a language specifically designed for time-series exploration. While it organizes data into a familiar SQL-like hierarchy of databases, tables, and columns, KQL is vastly easier to read, write, and understand. Instead of writing complex, nested SQL joins just to parse a string, KQL provides native operators for text search, time-series data, and statistical modeling. 

KQL query looks like this:

A practical example

Consider a scenario where you are building an API using .NET with the requirement to consume data from Azure Data Explorer. In that case, within your application’s repository layer, you would need to implement something like this:

As seen in the implementation, interfacing with the ADX cluster requires the Microsoft.Azure.Kusto.Data client package. 

You must also provide the specific URI for the ADX cluster, which follows a structured endpoint format. The following components constitute the structural breakdown of the endpoint:

• cluster-name – The unique name assigned to your cluster instance.
• azure-region – The specific Azure region where the resource is provisioned.
• kusto.windows.net – The standard domain for ADX/Kusto communication.

A critical detail to emphasise is the authentication flow. Security is handled via Microsoft Entra (formerly AAD), with the mechanism typically pivoting based on the environment:

• WithAadSystemManagedIdentity() – The application leverages its own managed identity, eliminating the risk of credential leakage or the need for secret rotation.
• WithAadAzCliAuthentication() – Used in local development by silently inheriting the developer’s active Azure CLI session.
• WithAadUserPromptAuthentication() – Triggers an interactive browser-based login when explicit user verification is required.

Once the connection is established, ADX can be queried. A nice touch is to use pre-defined KQL functions on the cluster, and the API just calls them with parameters if needed. We can rewrite are previous KQL query into the function to illustrate this approach: 

This function can be called from .NET application:

This call also sends request metadata to ADX in the form of ClientRequestProperties object. ClientRequestId gives the request a unique ID so it can be traced in ADX , and Application tags which app issued it. A fresh GUID is generated per call, so each query is individually identifiable. _queryProvider!.ExecuteQueryAsync(…) runs the query. Also important is the argument “adx-database” which is the target database on the cluster, since there can be numerous databases, based on the environment (dev, test, production).

Once the data is received you can process it and store to the collection:

From there, you can process the data additionally before it is served to the user in API response.

Conclusion

So this is the basic overview of how to architect a pipeline for receiving and managing telemetry data. By leveraging Azure ecosystem alongside the analytical capabilities of Azure Data Explorer and the versatility of .NET, the complexities of modern data streams can be effectively handled. This integrated approach ensures that data is not only collected efficiently but is also transformed into actionable intelligence with ultra-low latency.