When collecting Common Event Format (CEF) and syslog data from devices using a syslog collector host in the Microsoft Sentinel SIEM platform, it is possible to encounter scenarios where log ingestion unexpectedly stops. Among the various potential causes, one common issue is disk space exhaustion. This blog explores how to identify and resolve this issue effectively. Symptom
In the log file /var/opt/microsoft/azuremonitoragent/log/mdsd.err, you may encounter the following error message:
Error while inserting item to Local persistent store syslog.error: IO error: No space left on device: While appending to file: /var/opt/microsoft/azuremonitoragent/events/syslog.error/000555.log: No space left on device
This error clearly indicates that the disk has run out of space, preventing further logging.
Resolution Steps
1. Check Disk Space
Log in to the syslog collector host.
Use the following command to check the disk usage:
df -h |
Focus on the /var directory. If its usage exceeds 90%, you need to clear some space.
2. Check Log File Sizes
Navigate to the log directory and check the sizes of log files:
cd /var/log du -h messages* or du -h syslog* |
Identify files that are unusually large (e.g., over 4 GB).
3. Backup Log Files (Optional but Recommended)
Before clearing log files, create a backup to avoid losing important data. Note that this step may fail if the disk space is critically low.
Use the following command to zip a backup of the log file:
zip messagebackup.zip messages |
4. Clear Log Files (Be very careful while performing below step. Take the appropriate approval. below steps will delete the log files. )
If disk space is critically low, you can empty the log files to free up space:
To clear a specific log file:
cat /dev/null > messages-xxxx or cat /dev/null > syslog.xxxx
(Replace xxxx with the file name.)
Alternatively, use the truncate command:
sudo truncate -s 0 /var/log/messages or sudo truncate -s 0 /var/log/syslog
5. Restart Services
After clearing space, restart the relevant services to ensure proper functioning:
Restart rsyslog and azuremonitoragent services:
sudo systemctl restart rsyslog sudo systemctl restart azuremonitoragent
6. Verify Disk Space
Re-run the disk usage command to ensure sufficient space has been freed:
df -h
Important Notes
Temporary Fix:Â The steps provided above are a temporary workaround. For a permanent resolution, review and optimize log rotation and retention settings.
Approvals:Â Ensure proper approvals are obtained before executing these commands, especially in production environments.
Reference:Â For additional guidance, refer to the https://learn.microsoft.com/en-us/azure/azure-monitor/agents/azure-monitor-agent-troubleshoot-linux-vm-rsyslog
By following these steps, you can quickly restore log ingestion in Microsoft Sentinel while addressing disk space issues. Always exercise caution when clearing or truncating log files to avoid unintentional data loss. Disclaimer
The views expressed in this blog post are solely my own and do not represent those of my employer or any clients. Be carefull while performing the above steps. It will delete the log files. The views and opinions expressed in this blog post are based on references from Microsoft articles. Assistance from ChatGPT is taken for only customizing and formatting the blog.
Comments