Stream CEF logs to Microsoft Sentinel with AMA Connector (2023)

  • Article
  • 9 minutes to read

This article describes how to use theCommon Event Format (CEF) über AMAConnector for filtering and fast loading of CEF (Common Event Format) logs from multiple local devices via Syslog.

The connector uses the Azure Monitor Agent (AMA), which uses data collection rules (DCRs). DCRs allow you to filter logs before ingestion for faster loading, more efficient analysis and querying.

AMA is installed on a Linux machine that acts as a log forwarder and AMA collects logs in CEF format.

  • Configure connector
  • Learn more about the connector


The CEF Connector via AMA is currently in PREVIEW. HimAzure Preview Supplemental TermsAdd additional legal terms that apply to Azure features that are in beta, preview, or not yet released for general availability.


We will present on February 28, 2023CommonSecurityLog table schema changes. This means custom queries need to be reviewed and updated. Microsoft Sentinel updates out-of-the-box content (results, searches, workbooks, analyzers, etc.).

general description

What is the CEF Collection?

Many networks, security appliances, and appliances send their logs in CEF format over syslog. This format contains more structured information than syslog, with the information being presented in a parsed key-value array.

If your device or system sends logs via syslog with CEF, integration with Microsoft Sentinel allows you to easily perform analysis and query on the data.

CEF normalizes the data and immediately makes it more useful for analysis with Microsoft Sentinel. Microsoft Sentinel also allows you to ingest syslog events without parsing them and analyze them with query time analysis.

(Video) Onboarding Windows Security Events to Microsoft Sentinel via AMA - Full Demo

How Common Event Format (CEF) collection works via the AMA Connector

Stream CEF logs to Microsoft Sentinel with AMA Connector (1)

  1. Your organization sets up a log forwarder (Linux VM) if it doesn't already exist. Forwarding can be on-premises or cloud-based.
  2. Your organization uploads CEF records from your source devices to the forwarder.
  3. The AMA Connector installed on the log forwarder collects and analyzes the logs.
  4. The connector streams events to the Microsoft Sentinel workspace for further analysis.

When installing a log forwarder, the source device must be configured to send syslog events to the syslog daemon on that forwarder and not to the local daemon. The forwarding syslog daemon sends events over UDP to the Azure Monitor agent. When this Linux forwarder is expected to collect a large amount of syslog events, its syslog daemon sends events over TCP to the agent. In either case, the agent sends the events from there to your Log Analytics workspace in Microsoft Sentinel.

Stream CEF logs to Microsoft Sentinel with AMA Connector (2)

Configure the Common Event Format (CEF) via the AMA connector

previous requirements

Before you begin, make sure you have the following:

  • Microsoft Sentinel solution enabled.
  • A defined Microsoft Sentinel workspace.
  • A Linux machine for collecting logs.
    • Python 2.7 or 3 must be installed on the Linux computer. Use thepython --versionÖpython3 --versionVerification command.
  • or thesyslog-deÖrsyslogdemon activated.
  • To collect events from systems other than an Azure virtual machine, ensure the followingarco azurit is installed.

Configure a log forwarder

To ingest syslog and CEF logs into Microsoft Sentinel, you must identify and configure a Linux computer that collects the logs from your devices and forwards them to the Microsoft Sentinel workspace. This machine can be a physical or virtual machine in your on-premises environment, an Azure virtual machine, or a virtual machine in another cloud. If this machine is not an Azure VM, it must have Azure Arc installed (seeprevious requirements).

This machine has two components involved in this process:

  • To daemon syslog,rsyslogÖsyslog-de, which collects the records.
  • The AMA that forwards logs to Microsoft Sentinel.

When configuring the connector and DCRrun a scripton the Linux machine running the built-in Linux syslog daemon (rsyslog.d/syslog-de) to listen to syslog messages from your security solutions on TCP/UDP port 514.

DCR installs AMA to collect and analyze the logs.

Log Forwarding: Security Considerations

Be sure to configure machine security according to your company's security policy. For example, you can configure your network to conform with your corporate network security policy and change the ports and protocols on the daemon to suit your needs. To improve the security settings of your device,Back up your VM in Azure, check this one outNetwork security best practices.

If your devices send syslog and CEF logs over TLS (because your log forwarding is in the cloud, for example), you need to configure the syslog daemon (rsyslogÖsyslog-de) for communication via TLS:

Configure connector

You can configure the connector in two ways:

  • Microsoft's Sentinel Portal. With this configuration, you can create, manage, and delete DCRs per workspace.
  • API. These settings allow you to create, manage, and delete DCRs. This option is more flexible than the user interface. For example, the API allows you to filter for specific logging levels, while the UI only allows you to select a minimal logging level.

Configure or connector not Microsoft Sentinel Portal (UI)

  1. Open the connector page and create the DCR
  2. Define resources (VM)
  3. Select data source type and create DCR
  4. Run the installation script
Open the connector page and create the DCR
  1. open thisblue portaland navigate toMicrosofts SentinelService.

  2. Selectdata ports, and type in the search barMCE.

  3. ChooseCommon Event Format (CEF) via AMA (preview)Connector

    (Video) AMA Agent With Sentinel

  4. Select in the connector descriptionOpen connector page.

  5. nosettingsChoose areaCreate a data collection rule.

  6. UnderThe essential:

    • Enter a DCR name
    • Choose your subscription
    • Select the resource group where your sink is defined

Define resources (VM)

Select the computers you want to install AMA on. These machines are virtual machines or local Linux machines with Arc installed.

  1. Chooseresourcestap and selectadd resources).

  2. Select the virtual machines on which you want to install the Log Collection Connector.

  3. Review your changes and selectcollect.

Select data source type and create DCR


Use the same machine to forward both plain syslogsjCEF notifications

If you intend to use this log forwarding computer to forward Syslog and CEF messages to avoid duplicate events in the Syslog and CommonSecurityLog tables:

On any source machine that sends logs to the forwarder in CEF format, you must edit the syslog configuration file to remove the functions used to send CEF messages. This way the installs that are sent in CEF are not also sent in syslog.

(Video) How to send Unix OS logs to Microsoft Sentinel [Microsoft Sentinel FAQ]

  1. Choosecollecttap and selectLinux system registryas data source type.

  2. Set the minimum logging level for each installation. When you select a log level, Microsoft Sentinel collects logs for the selected level and other higher severity levels. For example if you selectLOG_ERR, Microsoft Sentinel collects logs for theLOG_ERR,LOG_CRIT,LOG_ALERT, SheLOG_NOTlevels

    Stream CEF logs to Microsoft Sentinel with AMA Connector (5)

  3. Review your changes and selectNext: Review and create.

  4. nocheck and createtab, selectCreate.

Run the installation script
  1. Log in to the Linux forwarding machine on which you want to install AMA.

  2. Run this command to start the installation script:

    sudo wget -O python

    The installation script configures thersyslogÖsyslog-dedaemon to use the required protocol and restarts the daemon.


    Avoidfull disk scenarioswhere the agent cannot work, we recommend that you configure itsyslog-deÖrsyslogConfiguration to not save unnecessary records. A disk full scenario stops the installed AMA from functioning. Read more aboutRSyslogNameÖSystemprotokoll.

Configure the connector using the API

You can create DCR with theAPI. learn more aboutDCR.

Run this command to start the installation script:

sudo wget -O python

The installation script configures thersyslogÖsyslog-dedaemon to use the required protocol and restarts the daemon.

(Video) Connecting Data to Microsoft Sentinel

Request URL and Headers


order body

Edit the template:

  • check thestreamsThe field is defined asMicrosoft-CommonSecurityLog.
  • Add the filter and installation log levels in theInstallationsnamejRecord levelParameter
{ "propiedades": { "immutableId": "dcr-bcc4039c90f0489b80927bbdf1f26008", "fontes de dados": { "syslog": [ { "streams": [ "Microsoft-CommonSecurityLog" ], "facilityNames": [ "*" ] , " logLevels": [ "*" ], "name": "sysLogsDataSource-1688419672" } ] }, "destinations": { "logAnalytics": [ { "workspaceResourceId": "/subscriptions/{Your-Subscription-Id} /resourceGroups /{resourceGroupName}/providers/Microsoft.OperationalInsights/workspaces/{SentinelWorkspaceName}", "workspaceId": "123x56xx-9123-xx4x-567x-89123xxx45","name": "la-140366483" } ] }, " dataFlows" : [ { "streams": [ "Microsoft-CommonSecurityLog" ], "destinations": [ "la-140366483" ] } ], "provisioningState": "Succeeded" }, "ubicación": "westeurope", "tags ": {}, "tipo": "Linux", "id": "/suscripciones/{Id. de su subscripción}/resourceGroups/{resourceGroupName}/providers/Microsoft.Insights/dataCollectionRules/{DCRName}", "name ": "{DCRName}", "tipo": "Microsoft.Insights/dataCollectionRules", "etag": "\"2401b6f3-0000-0d00-0000-618bb f4300 00\""}

After editing the template, usePOSTÖPUT ONto use it:


Examples of plant sections and logging levels

Check out these example log-level installs and configurations. HimNameThe field contains the name of the filter.

This example collects events from theCron,Devil,local0,local3juucpfacilities with whichEmbargo,mistake,Critical,Alarm, SheEmergencyLogging Levels:

"syslog": [ { "streams": [ "Microsoft-CommonSecurityLog" ], "facilityNames": [ "cron", "daemon", "local0", "local3", "uucp" ], "logLevels": [ " Warning", "Error", "Critical", "Alert", "Emergency" ], "name": "sysLogsDataSource-1688419672" }]

This example collects events for:

  • Himprivate permitjMarcosfacilities witheducation,realize,Embargo,mistake,Critical,Alarm, SheEmergencyRecord level
  • HimDevilInstallation with theEmbargo,mistake,Critical,Alarm, SheEmergencyRecord level
  • HimAder,local0,local5, Shenewsfacilities withCritical,Alarm, SheEmergencyRecord level
  • HimPostjuucpfacilities withEmergencyRecord level
"syslog": [ { "streams": [ "Microsoft-CommonSecurityLog" ], "facilityNames": [ "authpriv", "mark" ], "logLevels": [ "Informações", "Aviso", "Aviso", " Error", "Critical", "Alert", "Emergency" ], "name": "sysLogsDataSource--1469397783" }, { "streams": [ "Microsoft-CommonSecurityLog" ], "facilityNames": [ "daemon" ] , "logLevels": [ "Warnung", "Fehler", "Kritisch", "Alarm", "Notfall" ], "name": "sysLogsDataSource--1343576735" }, { "streams": [ "Microsoft-CommonSecurityLog" ], "installnames": [ "kern", "local0", "local5", "news" ], "níveis de log": [ "Critical", "Alert", "Emergency" ], "name": "sysLogsDataSource - - 1469572587" }, { "streams": [ "Microsoft-CommonSecurityLog" ], "facilityNames": [ "mail", "uucp" ], "logLevels": [ "Emergency" ], "name": "sysLogsDataSource- 1689584311 " } ]}

test a connector

  1. Run this command to verify that the syslog daemon is running on the UDP port and that AMA is listening:


    You must see themrsyslogÖsyslog-deDaemon listens on port 514.

  2. To capture messages sent from a logger or connected device, run this command in the background:

    tcpdump -I beliebiger Port 514 -A vv &
  3. After validation is complete, we recommend exitingTCP Vulkan: He writesfgand then selectCheck+C.

  4. To send demo messages, do one of the following:

    • Use the netcat utility. In this example, the utility reads the data sent byEcoCommand with line break option disabled. The utility then writes the data to the UDP port514on local server without timeout. To run the netcat utility, you may need to install an additional package.

      echo -n "<164>CEF:0|Teste simulado|MOCK|common=test-event-format|end|TRAFFIC|1|rt=$common=time-received-with-event-format -event" | nc -u -w0 hosten lokal 514
    • Use the registrar. This example writes the message to the4 locationsInstallation, on the severityEmbargotowards the port514, on localhost, in CEF RFC format. Him-tj--rfc3164the flags are used to match the expected RFC format.

      logger -p local4.warn -P 514 -n --rfc3164 -t CEF "0|Mock test|MOCK|common=test-event-format|end|TRAFFIC|1|rt=$common =formatado por evento -receive_time"
  5. To verify that the connector is installed correctly, run the debug script with this command:

    sudo wget -O python

Next Steps

In this article you learned how to configure Windows CEF on the AMA Connector to load data from CEF-enabled devices via syslog. For more information about Microsoft Sentinel, see the following articles:

(Video) Troubleshooting Microsoft Sentinel CEF environment

  • learn howGain insight into your data and potential threats.
  • BeginThreat detection with Microsoft Sentinel.
  • Use workbooksto monitor your data.


How do I send Active Directory logs to Sentinel? ›

In Microsoft Sentinel, select Data connectors from the navigation menu. From the data connectors gallery, select Azure Active Directory and then select Open connector page. Mark the check boxes next to the log types you want to stream into Microsoft Sentinel (see above), and select Connect.

How do I send Syslog to Sentinel? ›

From the Microsoft Sentinel navigation menu, select Data connectors. From the connectors gallery, select Syslog and then select Open connector page. If your device type is listed in the Microsoft Sentinel Data connectors gallery, choose the connector for your device instead of the generic Syslog connector.

What is CEF in Azure Sentinel? ›

What is CEF collection? Most network and security systems support either Syslog or CEF (which stands for Common Event Format) over Syslog as means for sending data to a SIEM. Azure Sentinel provides the ability to ingest data from an external solution.

Does Azure Sentinel require Log Analytics? ›

A Log Analytics workspace is required to house all of the data that Microsoft Sentinel will be ingesting and using for its detections, analytics, and other features. For more information, see Microsoft Sentinel workspace architecture best practices.

How do I send a .LOG file? ›

Right-click on the log file and choose Send to > Compressed (zipped) folder. This will create a zip archive of the logs, you can then attach that to an email or provide the file however you'd like to. After you send the compressed logs to us it is safe to delete both the compressed and uncompressed log files.

How do I import logs into Sentinel? ›

From the connector page, select the Open your workspace custom logs configuration link. Or, from the Log Analytics workspace navigation menu, select Custom logs. In the Custom tables tab, select Add custom log. In the Sample tab, upload a sample of a log file from your device (e.g. access.

How do you send a Syslog to a remote server? ›

Forwarding Syslog Messages
  1. Log on to the Linux device (whose messages you want to forward to the server) as a super user.
  2. Enter the command - vi /etc/syslog. conf to open the configuration file called syslog. ...
  3. Enter *. ...
  4. Restart the syslog service using the command /etc/rc.

Is Microsoft Sentinel the same as Azure Sentinel? ›

Azure Sentinel, now known as Microsoft Sentinel, centralizes your threat collection, detection, response, and investigation efforts. It provides threat intelligence and intelligent security analytic capabilities that facilitate threat visibility, alert detection, threat response, and proactive hunting.

What is the use of data connectors in Sentinel? ›

Azure Sentinel enables you to use data connectors to configure connections with different Microsoft services, partner solutions, and other resources. There are several out-of-the-box data connectors available in Azure Sentinel, and there are different ways to ingest data when a connector is not available.

Which connector do you use to collect Windows security events? ›

The Microsoft Sentinel Data Connector that utilizes the modern agent (AMA) for collecting Windows Security Events is for a couple of months general available.

How do I connect my office 365 to Sentinel? ›

In Microsoft Sentinel, select Data connectors, select Microsoft 365 Defender from the gallery and select Open connector page.
Mark the check boxes of the tables with the event types you wish to collect:
  1. Defender for Endpoint.
  2. Defender for Office 365.
  3. Defender for Identity.
  4. Defender for Cloud Apps.
  5. Defender alerts.
Feb 2, 2023

What is CEF log format? ›

The common event format (CEF) is a standard for the interoperability of event- or log generating devices and applications. The standard defines a syntax for log records. It comprises of a standard prefix and a variable extension that is formatted as key-value pairs.

What is common event format CEF over syslog? ›

Many networking and security devices and appliances send their system logs over the Syslog protocol in a specialized format known as Common Event Format (CEF). This format includes more information than the standard Syslog format, and it presents the information in a parsed key-value arrangement.

How do I view raw logs in Sentinel? ›

To get it, one has to take the RawDataRecordId from the All view of the event in the WebUI, then go to More -> Get raw data -> check if the appropriate Event source hierarchy and Event source is selected -> select the right file based on date and time -> click Download .

How do I send Azure monitor logs to Sentinel? ›

To log a service to Sentinel, pick the service (1), select "Activity Log" from the menu (2), and then click the "Logs" button (3). Note that on this screen, before pressing "Logs," you can review the information that will be sent to Sentinel.

What is the difference between Log Analytics and sentinel? ›

Log Analytics is a service to store and query logs and metrics. Azure Sentinel is using certain features of Azure Monitor as a platform. For example, Azure Sentinel uses Log Analytics for storing logs and metrics. When you enable Sentinel you choose to which Log Analytics workspaces the service is enabled.

What is the difference between Azure Sentinel Basic and Analytic logs? ›

Analytics logs can be retained for 730 days, but they are also the most expensive log type. Basic Logs can be enabled on a per table level and are cheaper than analytics logs ($ 0.50 compared to $2.6 per GB), but they have three main limitations: Retention is limited to 8 days.

How do I FTP a log file? ›

In the Home pane, double-click FTP Logging. - In the Log File Rollover section, click Select W3C Fields... Select the fields for your log files, then click OK. Under Directory, specify the path where the log file should be stored.

How do I make a log file executable? ›

Enable Installer Logs
  1. Open the Command Prompt.
  2. Run the following command: "C:\setup.exe" /v"/l*v LogFile. log", where "C:\setup.exe" is the path to the EXE installation file, and "LogFile. log" is the path to the file to be used to output the log file. As you continue with the installation, make sure that LogFile.
Oct 22, 2014

How do I Create a custom connector in Azure Sentinel? ›

The most direct way to create a custom connector is to use the Log Analytics agent. The Log Analytics agent is based on Fluentd and can use any Fluentd input plugin bundled with the agent to collect events and then forward them to an Azure Sentinel workspace.

Does Sentinel have an API? ›

Microsoft Sentinel REST APIs allow you to create and manage data connectors, analytic rules, incidents, bookmarks, and get entity information.

Which data sources can be used by Azure Sentinel? ›

Azure data sources
TypeData sourceLog Analytics tablename
AzureAzure Key VaultAzureDiagnostics
NetworkIIS LogsW3CIISLog
6 more rows
Jan 30, 2023

How do I forward Azure logs to SIEM? ›

From Azure Monitor, you export your logs using the Azure Monitoring single pipeline to an Event Hub. Finally, on the SIEM server, you need to install a partner SIEM connector. Then you can stream from the Event Hub your logs into the SIEM solution.

How do I send custom logs to Azure monitor? ›

Define a custom log
  1. In the Azure portal, select Log Analytics workspaces > your workspace > Settings.
  2. Select Custom logs.
  3. By default, all configuration changes are automatically pushed to all agents. For Linux agents, a configuration file is sent to the Fluentd data collector.
  4. Select Add to open the Custom Log wizard.
Jan 19, 2023

How do you integrate defender with Sentinel? ›

In Microsoft Sentinel, select Data connectors, select Microsoft 365 Defender from the gallery and select Open connector page.
Connect events
  1. Defender for Endpoint.
  2. Defender for Office 365.
  3. Defender for Identity.
  4. Defender for Cloud Apps.
  5. Defender alerts.
Feb 2, 2023

How do I send logs to event hub? ›

Stream logs to an event hub
  1. Sign in to the Azure portal.
  2. Select Azure Active Directory > Audit logs.
  3. Select Export Data Settings.
  4. In the Diagnostics settings pane, do either of the following: ...
  5. Select the Stream to an event hub check box, and then select Event Hub/Configure.
Oct 31, 2022

How do I forward logs to a syslog server? ›

Go to Device > Server Profiles > Syslog. Go to Device > Log settings > System.
Forwarding System logs to a syslog server requires three steps:
  1. Create a syslog server profile.
  2. Configure the system logs to use the Syslog server profile to forward the logs.
  3. Commit the changes.
Sep 25, 2018

How do I forward logs to syslog? ›

  1. Create a syslog server profile. Go to Device > Server Profiles > Syslog. ...
  2. Create a log forwarding profile. Go to Objects > Log forwarding. ...
  3. Use the log forwarding profile in your security policy. Go to Policies > Security. ...
  4. Don't forget to commit your changes when you're finished.
Sep 25, 2018

What security logs should be sent to SIEM? ›

There are six different types of logs monitored by SIEM solutions:
  • Perimeter device logs.
  • Windows event logs.
  • Endpoint logs.
  • Application logs.
  • Proxy logs.
  • IoT logs.

How do I add custom logs to Azure Sentinel? ›

Similar to Syslog, there are two steps to configuring custom log collection:
  1. Install the Log Analytics agent on the Linux or Windows machine that will be generating the logs.
  2. Configure your application's logging settings.
  3. Configure the Log Analytics agent from within Microsoft Sentinel.
Dec 1, 2022

What is the difference between Log Analytics and Azure Monitor? ›

Re: Difference between Log Analytics and Monitor

Monitor is the brand, and Log Analytics is one of the solutions. Log Analytics and Application Insights have been consolidated into Azure Monitor to provide a single integrated experience for monitoring Azure resources and hybrid environments.

How do you send Azure logs to log in Analytics? ›

On the Diagnostic settings page, click Add diagnostic setting. Under Category details, select AuditLogs and SigninLogs. Under Destination details, select Send to Log Analytics, and then select your new log analytics workspace. Click Save.

What is the difference between Sentinel and Defender? ›

Microsoft 365 Defender only integrates with other Microsoft cloud products, while Microsoft Sentinel allows you to add third-party (on-premises) products. For example, how can you secure your environment if you can't correlate data from the cloud with your firewall logs? Incident handling.

How do you make a playbook in Microsoft Sentinel? ›

Follow these steps to create a new playbook in Microsoft Sentinel: From the Microsoft Sentinel navigation menu, select Automation.
In the Basics tab:
  1. Select the Subscription, Resource group, and Region of your choosing from their respective drop-down lists. ...
  2. Enter a name for your playbook under Playbook name.
Jan 18, 2023

How do I set up event log forwarding? ›

This is one way to configure Windows Event forwarding.
Right-click Subscriptions and select Create Subscription.
  1. Enter a name and description for the subscription.
  2. For Destination Log, confirm that Forwarded Events is selected. ...
  3. Select Source computer initiated and click Select Computers Groups. ...
  4. Click Select Events.
Jan 18, 2023

Which method is used to send events to Eventhub? ›

Sending events to an Event Hub is accomplished either using HTTP POST or via an AMQP 1.0 connection. The choice of which to use when depends on the specific scenario being addressed.

How do you send event logs? ›

How to send Windows Event Logs?
  1. Open Event Viewer. ...
  2. On the left side, navigate to Event Viewer > Windows Logs > Application.
  3. Right-click on the Application and select Save All Events As.
  4. Name the file and click Save.
  5. Select Display information for these languages and then English.
  6. Click OK.
Feb 19, 2020


1. Azure Sentinel Lab Series | Setup Syslog Collector and install Azure Sentinel Agent | EP1
2. Codeless Connector Platform: Create Your Data Connector in Microsoft Sentinel
(Microsoft Security Community)
3. Quick Demo - Send OCI Audit Logs to Azure Sentinel SIEM with OCI Streaming and Azure Functions
(Onur Senturk)
4. Azure Sentinel webinar: Log forwarder deep dive on filtering CEF and syslog events
(Microsoft Security)
5. Transforming Data at Ingestion Time in Microsoft Sentinel | Microsoft Sentinel Webinar
(Microsoft Security Community)
6. Using Azure Sentinel with Logstash


Top Articles
Latest Posts
Article information

Author: Mrs. Angelic Larkin

Last Updated: 05/04/2023

Views: 6157

Rating: 4.7 / 5 (67 voted)

Reviews: 82% of readers found this page helpful

Author information

Name: Mrs. Angelic Larkin

Birthday: 1992-06-28

Address: Apt. 413 8275 Mueller Overpass, South Magnolia, IA 99527-6023

Phone: +6824704719725

Job: District Real-Estate Facilitator

Hobby: Letterboxing, Vacation, Poi, Homebrewing, Mountain biking, Slacklining, Cabaret

Introduction: My name is Mrs. Angelic Larkin, I am a cute, charming, funny, determined, inexpensive, joyous, cheerful person who loves writing and wants to share my knowledge and understanding with you.