top of page
Search

Troubleshooting CEF Log and syslog Ingestion Issues in Microsoft Sentinel Due to Disk Space Constraints

Writer's picture: Keshav JainKeshav Jain

When collecting Common Event Format (CEF) and syslog data from devices using a syslog collector host in the Microsoft Sentinel SIEM platform, it is possible to encounter scenarios where log ingestion unexpectedly stops. Among the various potential causes, one common issue is disk space exhaustion. This blog explores how to identify and resolve this issue effectively. Symptom

In the log file /var/opt/microsoft/azuremonitoragent/log/mdsd.err, you may encounter the following error message:

Error while inserting item to Local persistent store syslog.error: IO error: No space left on device: While appending to file: /var/opt/microsoft/azuremonitoragent/events/syslog.error/000555.log: No space left on device

This error clearly indicates that the disk has run out of space, preventing further logging.

Resolution Steps

1. Check Disk Space

  • Log in to the syslog collector host.

  • Use the following command to check the disk usage:

df -h

  • Focus on the /var directory. If its usage exceeds 90%, you need to clear some space.

2. Check Log File Sizes

  • Navigate to the log directory and check the sizes of log files:

cd /var/log du -h messages* or du -h syslog*


  • Identify files that are unusually large (e.g., over 4 GB).

3. Backup Log Files (Optional but Recommended)

Before clearing log files, create a backup to avoid losing important data. Note that this step may fail if the disk space is critically low.

  • Use the following command to zip a backup of the log file:

zip messagebackup.zip messages


4. Clear Log Files (Be very careful while performing below step. Take the appropriate approval. below steps will delete the log files. )

If disk space is critically low, you can empty the log files to free up space:

  • To clear a specific log file:

    cat /dev/null > messages-xxxx or cat /dev/null > syslog.xxxx

    (Replace xxxx with the file name.)

  • Alternatively, use the truncate command:

    sudo truncate -s 0 /var/log/messages or sudo truncate -s 0 /var/log/syslog

5. Restart Services

After clearing space, restart the relevant services to ensure proper functioning:

  • Restart rsyslog and azuremonitoragent services:

    sudo systemctl restart rsyslog sudo systemctl restart azuremonitoragent

6. Verify Disk Space

Re-run the disk usage command to ensure sufficient space has been freed:

df -h

Important Notes

By following these steps, you can quickly restore log ingestion in Microsoft Sentinel while addressing disk space issues. Always exercise caution when clearing or truncating log files to avoid unintentional data loss. Disclaimer


The views expressed in this blog post are solely my own and do not represent those of my employer or any clients. Be carefull while performing the above steps. It will delete the log files. The views and opinions expressed in this blog post are based on references from Microsoft articles. Assistance from ChatGPT is taken for only customizing and formatting the blog.


14 views0 comments

Comments


Subscribe Form

Thanks for submitting!

©2021 by Azure Secops and Azure Monitoring. Proudly created with Wix.com

bottom of page