In this post I will be adding custom log files to OMS Log Analytics. Custom log files give you the ability to add plain text logs into Log Analytics. Depending on your situation it might be easier to first explore structured logging options like Windows Event Log, Syslog or Application Insights since custom logs have a few limitations.
Configure Custom Logs
At the time of writing custom logs was still in preview, to use it you have to enable the feature in the OMS portal under Settings -> Preview Features. If you are using the Azure portal and the feature is not enabled you won’t see the + button to add a custom log. Once you have custom logs enabled you can use the OMS portal or Azure portal to add a custom log. In the OMS portal open the settings menu by clicking the gear icon in the top right. Under the Data -> Custom Logs menu you will see an Add button to add a custom log.
It is a pretty simple process, just follow the wizard to select a sample file, choose the record delimiter which can be a timestamp or newline, specify the paths to monitor and provide a name for the custom log. Make sure you give the custom log a reasonable name since you will be using it as the identifier in queries.
Take note of the restrictions for custom logs which can be found here. If your custom logs violate any of the criteria they won’t show up in Log Analytics. My custom logs took 30 minutes to show up in Log Analytics but your mileage can vary.
Custom Fields
Log Analytics will store data from the custom log text files in a single field called RawData. To get anything useful out of the custom logs you have to create custom fields over the data. Custom fields are not unique to custom logs you can extract custom fields from any existing fields.
To create a custom field execute a search query that displays the field you want to extract from. In the case of your custom log the table name will be the custom log name. Once you have the results, click on the ellipse to the left of the field name and choose ‘Extract Fields From …’.
On the next screen you can highlight the data you want to extract and match it against current records to refine the extraction process. You can click on records in the search results to further modify the extraction process.
Once you are satisfied with the result save the extraction, detailed instructions here.
Take note, if you create a new custom field your existing data won’t be updated with the new custom field. The custom field will only show on new records ingested by Log Analytics after the custom field was created.
Hello Francois
I am trying the same to pull my custom logs to OMS but my logs are there in a VM which is connected to the OMS Workspace, does it be fine to just give path without specifying any VM name..
Yes, you don’t specify the VM name only the path to the logfiles. The agent will be collecting the logs in that path on all VMs.
Hi Francois,
Do you know if it is possible to configure custom log collection via an ARM template or something similar. I would love to configure settings in a JSON format versus having to use the OMS Custom Log setup wizard, especially when I have several different Custom Logs to setup.
Thanks in advance.
Hi Brandon,
I never tried it before but I don’t think it is possible with ARM templates. The Custom Log resources are not showing in the ARM reference documentation. You can do it with PowerShell it that helps link.