How to push Dynamics365 logs into elasticsearch SIEM System

Written by: Thorsten Jakoby, Immanuel Haag and Tamira Horn.

Why

Customer Relation Management systems (CRM) typically host tons of Personally Identifiable Information (PII). Therefore they are a very attractive target for attackers and definitely highly relevant for security perspectives. Can teams be notified if large amounts of data are stolen? Are there mechanisms to interrupt potentially unwanted data exfiltration? Yes, if you at least built a Security Information & Event Management (SIEM) system into your Security Operations Center (SOC). Even though the theft was not noticed, security forensics or investigations benefit from many information that are put into an event context to analyze attacks or leaks. The CRM system market is very huge and comes up with different product designs. One player is Microsoft Dynamics365 – a widely used SaaS offer, which can be used as a CRM system. In contrast, there aren’t many SIEM systems available and they are highly specialized. In this scenario, we used the ELK Stack on Elasticcloud as log management and SIEM solution. We show how to push Dyncamics365 logs into SIEM System with a Logic App.

What – the sink: ELK Stack

First let’s have a look at the sink.

The sink is the easy part: it’s the ELK Stack, a well known place where you can drop log entries via different options: Filebeat, Fluentd or many other direct integrations through a provided REST-API. ELK is an acronym which stands for Elasticsearch, Logstash and Kibana. Elasticsearch is a search and analytics engine, with Logstash you can ingest data and transform/manipulate data through so-called pipelines. And Kibana helps to access the ingested data within your browser through personally designed dashboards. You could say Kibana is the GUI for the users who want to search for log files. Data is stored in Elasticsearch within an index. It can be used to group data of the same origin. You could compare an index with a namespace. Under the hood an index presents shards which are used to manage data replication in the cluster.

In our case we use three stages (“development”, “integration”, “production”) that get data from the source and enrich the data by a label for each stage. So it’s possible to use only one index for storing the log-files but a user could search for a specific stage if he selects one of these labels. Using one index is a recommended best practice to keep the overview if you want to add many data sources. Furthermore the sharding of the data in a cluster is easier with one index per log type.

As said before there is an ELK Stack REST-API – to be precise, there are many REST-APIs to be able to model various activities. In our concrete example we only use the document API – as we want to push “document based” log files into ELK. In the following we give an example how to POST a JSON file into the used Elasticcloud (SaaS offering) with the CLI tool curl:
curl -XPOST "ELASTICCLOUD-URL-ENDPOINT:9243/INDEXNAME/_doc" -H 'Content-Type: application/json' -d'{ "test": "test1" }' -H "Authorization: ApiKey ADDYOURAPIKEYHERE"


In the later described Low-Code-Application we use the update ability of the Elastic document API. So we do use PUT Calls with a unique document id. If it happens that we send a log file multiple times to the API – the document in the index only will be updated and not stored several times. So we don’t spam the SIEM system with too many files.

What – the source: Dynamics 365 and Dataverse/ CommonDataService

Getting the needed logs turned out to be not very clear. We simply did not know where they were.
Dynamics365 is storing data into a Microsoft PaaS/SaaS system called “Common Data Service (CDS)” or “Dataverse”. That system is top notch for integration with Microsoft’s PowerPlatform and many other Microsoft products. It is like “database as a service with a cool Web UI”. You can build relations with entities which are directly stored and accessible. The cool thing: those entities are accessible via oData API for non-Microsoft-systems. And on top of that: audit mode for entities is built in.

Dataverse does not only enable integration with external systems by default – it also has audit logs built in. You can even grab audit logs via oData. In conclusion: Dynamics365 stores its data in CDS. CDS can provide audit logs. That’s it. Now we just need to ship those logs into the SIEM system.

What – audit logs: some information

The wanted log files from now on called “Audit logs”, contain information about user access and actions within Dynamics. Actions include everything from changing an address in a customer record to activating or deactivating the monitoring functionality.

Dataverse provides the “audit”-entity which is structured as follows:

{
"_objectid_value": "XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX",
"_userid_value": "XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX",
"operation": 4,
"createdon": "2021-03-15T14:08:04Z",
"auditid": "example-audit-idXXX-XXXXXX-XXXXXX",
"attributemask": null,
"action": 64,
"objecttypecode": "user",
"_regardingobjectid_value": null,
"useradditionalinfo": null,
"_callinguserid_value": null,
"transactionid": null
},
{
"_objectid_value": "YYYYYY-YYYYY-YYYYYY-YYYYYY-YYYYYY",
"_userid_value": "YYYYYY-YYYYYY-YYYYYY-YYYYYY-YYYYYY",
...
},
{
...},

As you can see we receive a lot of IDs which refer to further information in other entities. We need to gather more information in order to generate as much security context for SIEM as possible.

For example: get more details by the referenced IDs.

The official documentation of the Dynamics 365 WebAPI provides more information: https://docs.microsoft.com/en-us/dynamics365/customer-engagement/web-api/audit?view=dynamics-ce-odata-9

Design considerations

We quickly realized that it will not be as easy as with other cloud platform components. CDS offers audit log streaming to other Microsoft products – but no streaming to external systems. Also, we did not find any easy integration from Elasticcloud to Dataverse or oData endpoints. It turned out that we need to design a solution. The process would look something like this:

1. Get credentials for Service Principal from KeyVault
2. Get oAuth token from Azure Active Directory
3. Ask CDS API for all audit logs in time frame
4. Request audit-details from CDS for all audit logs
5. Put all results into Elasticsearch

That sounds not too hard. Writing a function, hosted on Azure Functions was our very first idea. Using Quarkus as a lightweight framework seems to be a good idea.
But then we considered that this application will not be maintained by us. We would task our customer to maintain an application, for which he never planned resources.
Then we realized that this is a situation for “putting responsibilities as much as possible into the platform”. Creating an Azure Logic App instead of writing a function seems to be a good idea. Some advantages we see: visual process flows, automated and managed executions in conjunction with direct integration to the cloud platform.

So we’ll give it a try.

How to

How to push Dyncamics365 logs into SIEM system (elasticsearch cloud)

In this section we’ll create the necessary components to run an Azure Logic App for pushing logs to Elasticcloud.

Create Service Principal and store credentials

Requests to APIs or systems must be authenticated. In this case the requesting system is not a human user – it is our Azure Logic App asking for audit logs. Thus we’ll create a security identity to represent that Logic App. Those identities are sometimes referenced as “technical user”. In Azure they are called “Service Principal” (SPI) (also check here: https://stackoverflow.com/a/48105935/1297981)

There are several ways to create SPI’s. We’ll use the quickest via Azure CLI.
az ad sp create-for-rbac --name dynamicsToSiemPush --password "changeToStrongPassword"AZ CLI will return important and sensitive information like following:
{
"appId": "a487e0c1-82af-47d9-9a0b-af184eb87646d",
"displayName": "dynamicsToSiemPush",
"name": "http://dynamicsToSiemPush",
"password": changeToStrongPassword,
"tenant": "yourAadTenantId"
}

These identity information will be used later on to automate our solution. So let’s put them into a secret storage. We’ll use Azure KeyVault in this setup and assume that you already created one in default configuration.

Our new SPI will be used by the Logic App later. But first we need to allow it to call Dynamics 365 API. This is fairly easy, thanks to the integration with AAD and Dynamics. Go to your AAD, type “dynamicsToSiemPush” into “Search your tenant” and click on your App Registration. Choose “API permissions” from the menu and add Dynamics CRM user_impersonation.

Depending on administration permissions in your specific Active Directory, Admin Consent might be needed.
Once done your SPI is allowed to send requests to Dynamics365 and CDS API. Next up, vice versa: configure Dynamics to accept and process requests from this SPI.

Allow SPI to request Dynamics Audit Logs

The next step will be performed in your dynamics tenant. Login to Dynamics365 with administrator permissions and follow these steps. Please notice that the settings area is in fact the settings for “Power Apps” – which are part in Microsoft’s “Power Platform” Click on gear wheel (settings) in the upper right corner and choose extended settings. And unfortunately, some screenshots are in German language. But it should be enough for an illustration of the procedure.

Choose “Security” and add a new “SecurityRole” with a proper name e. g. “DynamicsSiemPush”.

You’re now entering Dynamic’s fine granular permission model for Security Roles. Choose wisely which permissions you want to permit to your new role. As we only want to read data, there is no need for write/modify permissions. At this point you have to keep your nerves as there are (depending on your system) a lot of entities and you have to click on each of them several times to give them the appropriate rights. Beware there are multiple tabs – go through each of them!

Now that we created the new role, let’s put our shiny new SPI into that role. Unfortunately those SPI are not always auto discovered by Dynamics. If you can not find your SPI, please apply the following steps.
In the next step we need to create an Application User which is a link to the SPI that you have created before. Again navigate to Security -> Users and choose “Application User” in the gray dropdown menu.
And again be aware that you choose in the new user context “Application User”:

Copy the Application ID from AAD of your created SPI and paste it here. Save the new user.

If everything works as expected, the user should be linked to the SPI of the Azure tenant. You can see in the Users Overview that the Username and URI/Object ID was automatically filled through the information of AAD. But the new user doesn’t have any rights – so now you have to link the created Security Role by clicking Manage Roles and choose the previously created security role DynamicsSiemPush.

With this setup you’re to consume the oData API of Dynamics with your SPI. In the next step we will give a short example how to achieve this with an Azure Logic App and push data into a SIEM System.

Build the Low-Code-Application

We use the Azure low-code approach so called Logic Apps. There are different low-code application flavours like power-apps and flows available in the MS Azure Cloud. As we are familiar with Logic Apps we chose this offer.
Within a Resource Group create a new Logic App with e. g. with the name “dx2siem”. After a while the deployment succeeded and you can begin with development in the visual app designer.

The trigger (or how often your application is executed)

Each Logic App has to be triggered through an event. You can trigger them by calling an extra API-Endpoint or using a time based recurrence approach. We decided that we nearly need real-time data, so we added a trigger which executes the app every 10 minutes:

The Secrets (or how to get the dynamics oData auth tokens)

As we want to consume the oData API of a dynamics O365 tenant first make sure to get the credentials of the SPI from the configured keyvault you saved earlier. Add a new action named keyvault connection within the Logic App designer:

Add a SPI which is able to access your keyvault in that the SPI of your dynamics tenant is saved.

After these steps – the logic is able to use credentials for accessing Dynamics. Further it’s necessary to add more secrets in the keyvault. For example the needed connection information of the ELK Stack API which consists of an API key and API URL.

Set variables, get the dynamics data and process it

After a deep investigation of the API we found out that we have to make multiple calls for all log files which should appear in the SIEM system. The structure is the following. The are Audits which represent the last performed actions within Dynamics. It’s an Array with multiple Entries for each log. If you want more details about these logs, e.g. who has performed what there are so called AuditDetails – to get them you have to use the AuditID of the response you’ll get from the Audits Array. There is a GET endpoint within the API which allows you to fetch every X hours of MS Dynamics audit logs. The response will give you multiple auditlogs which are handled in the Logic App. But before this could happen it’s smart to set some variables in the App Designer (see screenshot below)

• Hours: Specifies how many hours should be retrieved by the API in the GET LastXHours call.
• DX_URL: The oData URL of the o365 Dynamics tenant.
• SIEM_URL: URL endpoint of the SIEM system.
• SIEM_index: Specifies the index to write to in the SIEM system.
• SIEM_ingestpipeline: Specifies the ingest pipeline to be used on the part of SIEM. In it the field mapping of the view for Kibana is done.

The initial GET call retrieves the last X hours of audit logs from the Dynamics API via its endpoint (DX_URL variable). Here the variables and secrets are added to the Logic App statement as seen in the screenshot below. The API call is limited by a filter to the number of hours set (Hours variable).

To make the data fields accessible within the Logic App a Parse JSON Action must be used. It is applied to the response (body) of the getLastXHours call. Through a schema, the Logic App knows the available data fields. Since we want to have a solution that is as generic as possible, we defined the data field value as a type array. This ensures that all fields in this array can be used later on. If something changes in the API response in the future, the Logic App will still work. The following screenshot shows this visually:

The remaining steps of the Logic App are executed in a For-Each loop.The entries of the JSON array contained by get LastXHours audit logs can be accessed with the value and the statement current item within the loop as the current object of desire in Parse JSON. Beware you have to add a correct schema here as well, to gain access on the auditid in the next steps.

So this step in the loop passes us the appropriate needed auditid for the next call of the GET auditdetails. So in each iteration of the loop a different auditid is output until the array is processed:

The auditid is stored in the URI field as a Logic App access value. This value changes per loop cycle. A corresponding HTTP GET call is executed in each case. The authentication is also integrated as in the GetLastXHours step. The result of the two HTTP GET calls must now be prepared before they are sent to the SIEM system. This is the task of the next step. To be able to generate a new, extended JSON object from the retrieved data, a Compose Action is used in the Logic App.

The action receives the following expression as input:
addProperty(items('For_each'),'Details',body('GET_AuditDetails'))

Details of this can be seen in the reference.

It instructs the Logic App to append the results (body) of the GET AuditDetail call as a new property with the key Details for each audit log item. Thus, a new JSON object is created from an AuditLog and AuditLogDetail.

For clarification, AuditLog:
{
"_objectid_value": "XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX",
"_userid_value": "XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX",
"operation": 4,
"createdon": "2021-03-15T14:08:04Z",
"auditid": "example-audit-idXXX-XXXXXX-XXXXXX",
"attributemask": null,
"action": 64,
"objecttypecode": "systemuser",
"_regardingobjectid_value": null,
"useradditionalinfo": null,
"_callinguserid_value": null,
"transactionid": null
}

AuditLogDetails:

{
"@odata.context": "https://XXXXXXXXXXXX.dynamics.com/api/data/v9.0/$metadata#Microsoft.Dynamics.CRM.RetrieveAuditDetailsResponse",
"AuditDetail": {
"@odata.type": "#Microsoft.Dynamics.CRM.UserAccessAuditDetail",
"Interval": 4,
"AccessTime": "2021-03-15T14:08:04Z"
}
}

would merged into a new JSON object:
{
"_objectid_value": "XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX",
"_userid_value": "XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX",
"operation": 4,
"createdon": "2021-03-15T14:08:04Z",
"auditid": "example-audit-idXXX-XXXXXX-XXXXXX",
"attributemask": null,
"action": 64,
"objecttypecode": "systemuser",
"_regardingobjectid_value": null,
"useradditionalinfo": null,
"_callinguserid_value": null,
"transactionid": null,
"Details": {
"@odata.context": "https://XXXXXXXXXXXX.dynamics.com/api/data/v9.0/$metadata#Microsoft.Dynamics.CRM.RetrieveAuditDetailsResponse",
"AuditDetail": {
"@odata.type": "#Microsoft.Dynamics.CRM.UserAccessAuditDetail",
"Interval": 4,
"AccessTime": "2021-03-15T14:08:04Z"
}
}
}
This new JSON object is then transferred to the SIEM system by the Logic App. This REST call is a PUT. This is due to the fact that without a return channel we are currently unable to distinguish whether the retrieved data already exists in the SIEM system. Fortunately, the interface of SIEM offers the possibility to create a document with a unique identifier. If the same identifier is used again, the document will be updated in the SIEM system if it already exists. To get this identifier, the combination of createdon-auditid is set per log file. It can be assumed that a timestamp and the auditid are sufficiently unique in their combination to be used for this. Similarly, the previously defined SIEM variables are set in the URI field of the Logic App. The output (the newly generated JSON object from ComposeNewJSON) is used as the body. For the authentication of the SIEM system a corresponding API key, which is stored in the KeyVault, comes into use.

After this the log data is successfully transferred from MS Dynamics into the SIEM system.

Conclusion

Pushing audit logs for CRM entities is definitely a crucial security enabling task. Unfortunately there was no useful integration available for us. Low-Code-Applications can help to lower integration complexity. This was like an idea for an experiment. It turned out to be very reliable and effective, to put integration tasks into the platform. So this article showed how to push Dyncamics365  logs into a SIEM system with a Logic App.

Downside: we discovered that CRM bulk processes trigger A LOT of log data. Logic Apps are priced per operation. Our real-world-scenario had around 5 subqueries. This leads to very high costs. Another issue is that testing and code review is not straight forward within a Logic App as you may know it from your daily coding tasks.