Keeping Your Cloud Secure And Compliant – Tools For Azure Monitoring And Auditing Azure monitor

Moving your business to the cloud comes with many benefits. However, it is also a responsibility and Azure monitoring is a must when using the platform. The data you store needs to be safe from leaks and security breaches. Additionally, there are legal requirements pertaining to data storage which you need to fulfil. Azure monitoring can help you with fulfilling these obligations – here are the key aspects you need to consider.

This is it – you are entering the cloud era! The decision has been made, your business is moving its infrastructure to the cloud, or you are building your first application which will be based on cloud technology.

This is it – your organization’s AHA! moment! The tipping point. From now on, the applications will be faster, deployments easier, and everything will run smoothly.

Your enterprise is truly entering the DIGITAL TRANSFORMATION era.

Isn’t this what the cloud vendor, analytics, and all the consultants told you? Isn’t the cloud the new promised land of effortlessness and easy to handle IT operations?

Well, not exactly. At least not if you don’t prepare yourself, your team and your company for it.

Where’s the catch? 

To put it into a more realistic scenario – let’s consider an example of one of the companies we are working with at Predica. It is a manufacturing organization building a new solution for its customers. The solution will gather data, analyze it and manage the customers’ devices. All of it built on top of the Azure platform and managing mission-critical, physical infrastructure.

Isn’t this the perfect case of digital transformation? A device manufacturer entering an area of services and cloud computing?

Indeed, it is. But at the same time, it puts this organization into a completely new area of management and operations of a cloud solution. More specifically, something they need to prepare for in different areas: people skills, tooling, and operations procedures.

OPERATIONS!

This is the keyword here. What we have learned is that for an organization, going to the cloud is the easiest step. However, operating in a cloud environment is a completely different story. A typical situation is that the move to the cloud, especially in applications and using Platform-as-a-Service areas, is something that internal operations teams are not prepared for.

One of the aspects of operations is getting to know what is happening with your solution. You need to gather audit and log operations, process them, get the key facts and metrics, and act on them.

But how can you do it on Azure platform? Is there a ready solution for Azure monitoring?

This article will show you 4 areas of auditing and logging that you should focus on in your solutions. Further, I will show you what mechanisms are available to you. In addition, I will also present a recommendation on how to gather and process this information from the platform.

Plus, there will be a BONUS link to an article that will give you additional advice.

Our application

Here is our example of application infrastructure built on top of PaaS Azure services. Its goal is to gather telemetry data and provide access to it for end users.

For simplicity, we will not describe the details on how it gathers this telemetry data, what is the logic behind it and how exactly it is built.

The key architecture components are:

• Event Hub which receives telemetry information and processes it, storing the outcome in Azure Storage (cold storage) and Azure SQL Data Warehouse
• Web API and Web Application exposing information processed from Event Hub stored in Azure SQL Data Warehouse
• Azure Active Directory that secures the entire platform (access to Azure and RBAC model) and provides secure environment for applications (API and web app)
• Azure Key Vault used by the application to store credentials and keys, protected with Azure Active Directory.

This is a simple application, and a typical example of many modern applications built on top of Azure. However, even this simple architecture is generating lots of information. Gathering it will allow you to effectively manage the platform, and detect problems and security issues.

Read through the whole article and check the above application diagram with the content applied to it. It might be a surprise.

What are the four areas and tools you should focus on in terms of logging and auditing in this architecture, which provide this information?

#1 General auditing

Azure is a complex service. At every moment the services on Azure generate event logs, performance data, hosts metrics. These are available in two major data sources:

• Azure Diagnostics Logs: Azure Diagnostics logs provide performance and events log monitoring for events from Azure resources and services. The type of information provided and content of the logs varies by resource.
• Azure Activity Log: Who did what in Azure configuration, who was granted which role – it is all provided for you in Azure Audit Logs. This is an operational audit log from all the resources within Azure platform and operations performed through Azure Resources Management.

These sources simultaneously provide the vast majority of information on diagnostics logs, metrics and auditing information on Azure operations.

#2 Security auditing

General auditing is good for monitoring events and detecting problems, but what about security monitoring?

From a security point of view, Azure is managed by Azure Active Directory. Azure Active Directory audit and sign-in logs are your first and most important source of information about users and their activity on the platform.

Azure platform security model is based on role-based access control (RBAC). Roles are granted to Azure Active Directory users and groups. If you want to know who was granted which role and when – Azure Activity Log will provide this information for you.

How to get quick insights from Azure Activity Log?

Specific Azure services might have their own security logs and you should always look into it within the documentation. Our example application has three logs already available:

• Azure Key Vault has an audit log, providing e.g. information on who has accessed sensitive material stored there. The good news is that this stream is available in the Activity log as well.
• Azure Storage has its own analytics log, gathering audit information on storage access.
• Azure SQL Data Warehouse has its threat detection mechanism – when enabled, it provides audit information and stores it in Azure Storage.

Always check whether the service in your architecture has its audit source – it might be a valuable source of information.

#3 Application Telemetry

So far we were only showing tools for logging and auditing the Azure platform and services. But what if we want to monitor our components built on the Azure platform? In our case, the web API and application or customer access.

What we’ve found is that not many people know about Azure Application Insights.

Application Insights allow you to monitor your applications at application level – you can deploy an Application Insights agent to Azure App Service and gather important metrics on its performance and application. What is more important is that you can integrate it with your code and:

• Issue telemetry information from your application, which Application Insights can present as trends and metrics.
• Capture exceptions and errors from your code and push them to Application Insights. You can see faulty paths in your application and check the error after it happened.

#4 Azure Monitor

It is a lot of information to process. And, it comes from multiple sources. Is there one place where you can gather it and process at once? Maybe not for all of it, but for most, right now we have Azure Monitor.

It is a tool providing built-in capabilities for the review and analysis of audit logs. Azure Monitor service gathers all information from Azure Audit and Diagnostics logs and allows to access it from a single location in the Azure portal.

Azure Monitor also allows alert rules to be specified to trigger alerts sent over e-mail and SMS, or to initialize a call to external service through Webhooks in case of a specific event or metric. It is available through a dedicated Azure portal blade – Monitor.

Introduction to Azure Monitor with Tomasz Onyszko

Logs, logs, logs

You have to admit: there is a lot of information, logs and metrics generated from Azure services. And there are many ways you can gather them, consume and store them on the platform.

Let’s sum it up for our simple application. Here are all logs and audit sources we have covered, together with information on how they can be accessed and stored:

Not that simple, huh?

So much info and so many sources. What to do next?

Wow, there’s a lot of it.

Which approach to take to gather it all in one place? How to process it? And finally, which tool to use to generate alerts and visualization?

Here is the general approach we use as a starting point in discussions with our clients.

You can take it from here and adjust it to your specifications.

1. Use Azure Monitor. First of all, you should monitor Azure events through Azure Monitor which has both the Azure Audit and Diagnostics logs in one place. It is the first tool you can use to detect interesting events and trigger alerts.

Hint: Use Webhooks to push information from Azure Monitor to your SIEM systems or built-in capabilities for alerts.

2. Application Insights for telemetry. Solution-specific metrics? Need to measure process SLA on the application? Integrate your solution with Azure Application Insights. It is a great tool to show you what is happening with your application and detect anomalies.

Hint: Application map is a quick visual which allows you to spot problems with your app easily.

3. Some services have separate logs – bring them all together. Use Azure Functions or other Azure services (but Functions fits well into this scenario) to extract and ingress services logs into your monitoring platform. Usually, there is an API which allows you to pull this information.
4. Long term data storage? Store log data in Azure Storage blobs for other solutions to pick it up and consume, and for the long term. Azure Monitor can store this data for you there.
5. How to get data for current trends and event monitoring? Push log events to Azure Event Hub for processing and store the outcome in Azure SQL. This will allow you to consume current and historical data with Power BI for easy analytics.

Sounds like an awful amount of work and integration to do, doesn’t it? Isn’t there something easier which will gather all this information and provide some visuals? In fact, there is.

Power BI integration

For a quick visualization and access to data, you can leverage Power BI which provides ready packs for some of the services. Two of such solutions, which you will find immediately ready for your Azure application, are:

• Azure Audit Log
• Azure Active Directory

If you need something more advanced, Microsoft provides its Operations Management Suite (OMS) as SaaS offering. It gathers information from various data sources and allows you to manage and analyze them. You can find the supported data sources in this article.

And, if OMS is not enough, there are third party services which provide such capabilities. One used by many organizations is Splunk with its add-on for Microsoft Cloud service.

Let’s look at our application architecture again

Do you still remember our little application? Let’s take a look at its infrastructure again, this time with all the elements we have covered applied to it.
 

It is a bit more complex, isn’t it?

But YOU’VE MADE IT to this point. You deserve the bonus link, and here it is. This article on Azure Logging and Auditing covers everything I’ve discussed, and more. It is the ultimate guide to logging and auditing on the Azure platform.

Enjoy!

Remember, information is not everything

This article covered lots of information on how you can gather and process diagnostics and audit information from the Azure platform.

But ultimately, having information is not enough. What’s most important is what you will do with it! Make sure your operations team and procedures are ready for it.

We will gladly help you with this process if necessary – get in touch now!