Sentinel Hunting Queries and the Data Lake

What it is

Microsoft Sentinel holds data from many sources, so it is where you hunt across everything, including non Microsoft logs and long term history. You hunt with hunting queries, you save evidence with bookmarks, and you reach data that is no longer in the hot tier by using the Microsoft Sentinel data lake with KQL jobs and summary rules.

How it works

Hunting queries

The Hunting page lists built in queries from the Content hub and your own. Each query is KQL tagged with MITRE tactics and techniques, so you can pick queries for a technique. In the Defender portal you run them from advanced hunting, where Defender XDR tables and Sentinel workspace tables can be queried together. That lets one query join DeviceProcessEvents with SecurityEvent, for example.

Typical workflow:

1. Choose or write a hypothesis.
2. Run the relevant hunting queries and review the results.
3. Add interesting rows as bookmarks, with notes and entity mappings.
4. Attach bookmarks to an incident, or create an incident from them.
5. If the query is valuable, create an analytics rule from it, so it runs automatically.

Bookmarks and livestream

A bookmark saves rows of hunting results, with comments and tags, so evidence is not lost. Bookmarks can be mapped to entities and attached to incidents, and they can be searched later.

Livestream runs a hunting query continuously in the background and shows new results as they arrive, with a notification, so you can watch for a pattern in real time during an investigation, for example a specific IP address. It does not create alerts, and it is meant for short term monitoring. Turn the query into an analytics rule if the monitoring must be permanent.

The data lake

The Sentinel data lake is a low cost store for large volumes of security data, kept for up to 12 years. Data in the data lake tier is not available for real time analytics or hunting in the Analytics tier, but you can query it whenever you need.

Data lake exploration in the Defender portal lets you run interactive KQL queries directly on lake data, over multiple workspaces. This suits an analyst who needs history for an incident when the data is older than the Analytics tier retention, and wants to use familiar KQL.

KQL jobs

KQL jobs are one time or scheduled asynchronous KQL queries that run on data in the data lake and on federated tables. Federated tables are external sources, such as Microsoft Entra ID, Microsoft 365 and Microsoft Resource Graph tables, which you can query next to lake tables without ingesting them.

Use jobs for:

• Long running, one time queries for incident investigations and response.
• Aggregation tasks that support enrichment, using low fidelity logs.
• Historical threat intelligence matching, scanning old data for indicators that you learned about recently.
• Anomaly detection scans across several tables.

Jobs are effective when the query uses joins or unions across datasets. The key idea is promotion: a job writes its results into a table in the Analytics tier, where you can query them with the normal advanced hunting editor. Promotion has benefits, which are combining current and historical data, lowering query costs, and combining data from several workspaces into one. It also has a cost, because storage in the Analytics tier costs more than in the lake, so project only the columns you need and filter the rows before you promote.

Summary rules

Summary rules aggregate large sets of data in the background and write the results to a custom table in the Analytics tier. The summary data is precompiled, so queries over it are fast, and it can be derived from low cost tiers.

They help with:

• Fast analysis and reports over very large datasets and long time ranges.
• Cost savings. Keep verbose logs in a cheap tier, and send only the summarised data to the Analytics table for analysis.
• Privacy. Remove or hide sensitive details in the summary and limit access to the raw tables.

A summary rule has a query, a time bin that sets how often it runs and the size of the window, and a destination table, whose name ends in _CL. Templates exist for common cases.

Search jobs and restore

For older data in Analytics tables with long retention, search jobs pull matching rows into a new table (with the _SRCH suffix), and restore brings a time range of archived data back into a table (with the _RST suffix) so you can query it. The usual order is start the job, wait for the table to be ready, query it, then delete the restored table when the investigation ends.

A worked scenario

A SOC learns of a new campaign with a list of 500 malicious IP addresses. Hunting queries from the content hub find nothing in the last 30 days. The team runs a KQL job on the data lake firewall logs for the past year, matching against the indicator list, and the job promotes the matches to the Analytics tier, where they confirm one old connection. For a monthly report, a summary rule aggregates daily sign in counts into a custom table. During the incident, an analyst keeps a livestream running on the attacker's IP address, and bookmarks the key rows to the incident.