Integrating third party SIEM through Azure Event Hub

Author – Gunjan Chaudhari, Associate Cloud Engineer

After reading this title I am pretty sure you are confused about what we are going to learn in this blog. But right now, let’s focus on two important words Azure Event Hub and SIEM.

What is Azure Event Hub?

As the name says, it is a service provided by Azure to store events. But this service is much more than that the name says, Azure event hub handles data that is continuously generated from different sources i.e. it is a data streaming platform and event ingestion service, which can process millions of events per second. Due to the low latency, it can handle large distributed streaming capacity in seconds. It comes under Platform as a Service (PaaS) in Azure. It is a fully managed service that is simple and scalable but with little bit more configurations in Azure. For example, Event Hubs enable behavior tracking in mobile apps, traffic information from web farms, in-game event capture in console games, or telemetry collected from industrial machines, connected vehicles, or other devices.

Terminologies: 

1. Event Producers: An entity that sends the messages (events) to the Event hub is known as Event producer or publisher. Event producers/publishers can publish events using HTTPS or AMQP. Using these protocols, an Event Producer can ingest the data in the event hub.
2. Partitions: In the event hub, we can have multiple partitions. This multiple partition enables parallel processing of the event. We can create 32 partitions and if users need more partitions, they can contact the Azure team. Each partition has a partition key/id. Partitions are used to segregate the data. The events get stored in the partition in sequence, as the newer events arrive, they are added to the end of the sequence.
3. Consumer Groups: It is a view of the entire event hub. Consumer groups enable consuming applications to have a separate view of the event stream. In the real-world scenario, if you have a partition full of events generally you will have two applications that are reading those events within that partition. One application typically reads the events and puts them into storage, because the event in the event hub will be expired in a certain period and another application is used to read the events and to identify some issues. Both applications will have different offset keys. An offset is a position of an event within the partition. It is basically a curser and this curser value needs to be stored by each application. So, they will know where they need to start reading again.
4. Throughput unit: It is a unit of the capacity to handle the traffic coming and going out through the event hub.
5. Event Receivers: An entity that receives/reads the event data from the event hub is known as an event receiver.

What is SIEM Tool?

Security Information and Event Management (SIEM) is a software that collects security data from various sources like applications, domain controllers, servers, and many more. This collected data is normalized and aggregated for analyzing the data to discover and detect threats and notifying the administrator using alerts. SIEM is important because it makes it easy for enterprises to filter the massive amount of data and set security alerts as per the organization. For instance, a user account that generates 25 failed login attempts in 25 minutes could be flagged as suspicious but still be set at a lower priority because the login attempts were probably made by the user who had probably forgotten his login information. However, a user account that generates 130 failed login attempts in five minutes would be flagged as a high-priority event because it’s most likely a brute-force attack in progress. Popular SIEM Tools are Splunk, IBM QRadar, Log Rhythm.

The main challenge was that we were not able to see IIS logs. To see all the logs, we used the SIEM tool. Another difficulty we faced was connecting EventHub with the virtual machine and storage account. This was solved by using a diagnostic script which is mentioned below.

Demo:

Step 1:  Create a storage account to create one container inside it to store all the logs.

Step 2: Create Event hub namespace in the Azure portal. While creating this keep the pricing tier as Standard so that we can create multiple consumer groups if needed. After this, Click on Review + create.

Step 3: Create an event hub inside the namespace which we created and enable the capture option. Select the storage account and container which we created to store logs.

Step 4: After creating EventHub and namespace, we need to make changes in the script. This script is used so that we can get diagnostics of the virtual machine.

• Replace your Subscription ID, resource group name, and VM name or you can directly go into the virtual machine’s properties and copy-paste resource ID in the script.

<Metrics resourceId=“/subscriptions/SubscriptionID/resourceGroups/resourceGroupName/providers/Microsoft.Compute/virtualMachines/VM name“>

• Replace EventHubNamespace name and event hub name.

<EventHub Url=“https://EventHubNamespaceName.servicebus.windows.net/EventHubName“ SharedAccessKeyName=“RootManageSharedAccessKey“ />

• Replace the Storage account name and storage account key     

            <StorageAccount name=“Storage Account Name” key=”StorageAccount Key“ />

• Replace EventHubNamespace name and event hub name in a Private config tag.

              <PrivateConfig> 

              <StorageAccount name=“Storage Account Name” key=”StorageAccount Key“ />

              <EventHub Url=“https://EventHubNamespaceName.servicebus.windows.net/EventHubName“

                               SharedAccessKeyName=“RootManageSharedAccessKey“ 

                               SharedAccessKey= “Primary key of  RootManageSharedAccessKey”/>

              </PrivateConfig>

After all the changes the code will look like this :

Step 5: Save this document with .xml extension. For eg: Diagnostic.xml and open the PowerShell to create a diagnostic extension for Virtual machine.

Connect-AzAccount

$rgName = “[Resource Group Name]”

$vmName = “[VM name]”

$diagnosticsconfig_path = “[Diagnostic file local path eg: C:\Users\Gunjan\Desktop\Diagnostic.xml]”

$diagnosticsstorage_name = “[storage Name]”

$diagnosticsstorage_key = “[Storage Key]”

Set-AzVMDiagnosticsExtension –ResourceGroupName $rgName –VMName $vmName –DiagnosticsConfigurationPath $diagnosticsconfig_path –StorageAccountName $diagnosticsstorage_name –StorageAccountKey $diagnosticsstorage_key

We have successfully created a diagnostic extension for our Virtual Machine. We can see the logs in our storage account by clicking on Tables. We can also use a third-party tool such as Azure Service Bus explorer for receiving the logs.

Step 6: Install Service Bus explorer and connect it with Event Hub Namespace by using a connection string. We can get this connection string in the shared access policies of Event Hub Namespace.

For eg:

           Endpoint=sb://demonsblog.servicebus.windows.net/;

          SharedAccessKeyName=RootManageSharedAccessKey;

         SharedAccessKey=vQw3GAsvKNZmn8MznOZ/OF68bUiqr3mCSOitoF9YKCo=

Step 7: After connecting our Service Bus Explorer with Azure Event Hub we can receive logs by creating a consumer group and then right-click on that consumer group to create a consumer group listener.

Voilà!!

We have successfully implemented Azure Event Hub and Service Bus Explorer. But before rushing towards closing this blog lets read some benefits of azure event hub.

Benefits:

• Shortens the time taken to notify and for the identification of threats.
• Offers a holistic view of an organization’s information security environment, making it easier to gather and analyze security information to keep systems safe. The organization’s data goes into a centralized repository where it is stored and easily accessible.
• IT team has a choice of frameworks to implement the Azure Event Hub- .NET, JAVA, Python, Nods.js, Apache Storm, etc.
• Can perform detailed forensic analysis in the event of major security breaches