When setting up Microsoft Sentinel, there's a common question: How do we store data for a really long time? Some compliances and industries regulation require us to keep data for upto 7 years or more because of rules. Addressing this challenge involves evaluating various options offered by Microsoft Log Analytics for prolonged data storage. In this blog post, we'll explore these options and consider their pros and cons.
There are different ways to do this using Microsoft Log Analytics:
Exporting telemetry Data in Blob Storage: You can put your data in Azure Blob Storage. This method involves exporting data to Azure Blob Storage. While it provides flexibility, it also comes with certain limitations. You need to develop automation (Playbooks) for data export, and parsing data in storage accounts can be a complex undertaking.
Using Azure Data Explorer: This is a tool made for analyzing data. Using Azure Data Explorer enable us to run cross-platform queries and visualize data across both Azure Data Explorer and Microsoft Sentinel. However, utilizing Azure Data Explorer for data storage may come with additional costs, as export capabilities are not free now. Also, its relatively complex to setup the ADX.
Archival Tables: Storing data in "archival tables" within Log Analytics is a relatively simple and straightforward approach. The data is easy to parse, making it accessible when needed. However, it's essential to consider how frequently you'll need to access this data, as restoring data from archival tables incurs cost
Analytical Tables for Frequent Access: If frequent data access is a requirement, consider using "analytical tables" within Log Analytics. While they may have slightly higher costs for storage, they provide more straightforward and quicker access to data. The enhanced user interface for data restoration in Microsoft Sentinel further facilitates efficient data retrieval. In my opinion, the best way to store data for a long time is to use "archival" tables. They are relatively simple to work with. However, keep in mind how often you'll need to use this data since restoring data from these tables comes with a cost.
If you're archiving data for compliance and don't expect to frequently restore large amounts of data, archival tables are a good choice. But if you need regular access to the data, it's wise to calculate costs using the Azure Pricing Calculator. Based on experience, older data is often needed less frequently. In such cases, "analytical" tables are a simpler option.
Microsoft Sentinel has also improved its user interface for data restoration, making it easier to retrieve data through the "Search" feature.
Conclusion Deciding how to store data for a long time in Microsoft Sentinel depends on your organization's needs. Archival tables are simple and good for rules, but they can cost when you want the data back. Analytical tables are better if you need to use the data a lot. Microsoft Sentinel also makes it easier to get your data back. Disclaimer: Please note that the views expressed here are my own and do not represent my employer\organization. I'm sharing my personal perspective, and there may be better options available. It's important to use this information carefully and make cost calculations using the Azure Price Calculator to make an informed decision.