Microsoft Sentinel Attack Range + Data Lake

A complete, battle-tested deployment of the MS-Attack-Range with Data Lake integration, custom entity/graph work, IOC hunting via KQL Jobs, and comprehensive detection engineering capabilities.

Built and validated live on a Pay-As-You-Go Azure tenant with Defender for Servers Plan 2.

What This Repo Contains

Infrastructure

• Modified Terraform — Dual DCR architecture (Analytics + Bulk), 90-day retention, Defender for Cloud + MDE connectors, Standard SKU public IPs
• Data Collection Rules — 6 DCRs: Analytics (XPath-filtered high-signal events), Bulk (all events), Linux (Syslog), ThreatIntel (custom table), VMInventory (custom table), Cross-workspace (attack-range → SentinelUdemy)
• ARM Templates — DCR deployments for custom table routing, analytics rules

Scripts

Script	Purpose
preflight.sh	Pre-deployment validation (auth, quota, tools, IP)
create-datalake-tables.sh	Creates _CL tables + DCE for Data Lake routing
validate.sh	Post-deploy validation (VMs, DCRs, tables, data flow)
install-sysmon.ps1	Installs Sysmon + SwiftOnSecurity config on Windows VMs
configure-mde-passive.sh	Sets DC to MDE passive mode, removes MDE from workstation/Kali
send-threatintel.sh	Sends test IOCs to ThreatIntel_CL via Logs Ingestion API
collect-vminventory.sh	Collects VM inventory and sends to VMInventory_CL

Detection Engineering

• 34 analytics rules — 20 from the original repo ARM template + 7 built-in Content Hub templates + 5 custom entity-mapped rules + 2 additional custom rules
• 5 entity-mapped rules with Account, Host, IP, and Process entities feeding the investigation graph
• 1 automated KQL Job — hourly IOC matching against 1.2M MDTI indicators
• 1 workbook — Kill Chain Graph with 4 visualization panels
• 1 hunting query — Kill Chain Entity Graph for full attack path tracing
• 1 automation rule — auto-tagging entity rule incidents
• 1 watchlist — AttackRangeAssets for IP/hostname correlation

Custom Tables

Table	Location	Purpose
ThreatIntel_CL	attack-range-law	Custom threat intel indicators via Logs Ingestion API
VMInventory_CL	attack-range-law	VM posture data (OS, patches, ports, admins)
IOCMatches_CL	SentinelUdemy	KQL Job output: IOC matches from MDTI scanning
SecurityEventLake_CL	attack-range-law	Bulk SecurityEvent for Data Lake tier
SyslogLake_CL	attack-range-law	Bulk Syslog for Data Lake tier
WindowsEventLake_CL	attack-range-law	Sysmon events for Data Lake tier

Notebooks

• attack-range-graph.ipynb — Custom Sentinel Data Lake graph using GraphSpecBuilder API with nodes (Host, Account, IP, ThreatIndicator, Alert, Identity) and edges (LoggedInto, AuthenticatedTo, ExecutedProcessOn, MatchesThreatIntel, TriggeredAlert)

Architecture

Attack Range VMs (attack-range-rg)
  ├── Windows DC (win-dc)        ──┐
  ├── Windows Workstation         ──┼── DCR Analytics ──► SecurityEvent (Analytics tier)
  └── Kali Linux                  ──┘    High-signal: 4624/25, 4688, 4698, Sysmon 1/8/10/12/13
                                   │
                                   ├── DCR Bulk ──► SecurityEvent (same tables, bulk)
                                   │
                                   ├── DCR Cross-Workspace ──► SentinelUdemy SecurityEvent
                                   │    (for IOC matching against 1.2M MDTI indicators)
                                   │
                                   └── DCR Linux ──► Syslog (Analytics tier)

  Entra ID ──── Diagnostic Settings ──► AuditLogs (Analytics tier)

  MDE (DC passive) ──► Defender for Cloud connector ──► SecurityAlert

  Logs Ingestion API ──► ThreatIntel_CL, VMInventory_CL

  KQL Job (hourly) ──► Scans SecurityEvent × ThreatIntelIndicators
                   ──► Writes matches to IOCMatches_CL
                   ──► Analytics rule fires ──► High severity incident with entities

Deployment Results

Metric	Value
Total analytics rules	34
Rules actively firing	13+
Total alerts generated	122+ (in first session)
MITRE tactics covered	8 (Recon, Initial Access, Execution, Persistence, PrivEsc, DefEvasion, CredAccess, C2)
Attack simulations run	40+ individual techniques
Custom tables created	6
Data connectors active	Azure Activity, Entra AuditLogs, Defender for Cloud
MDTI indicators available	1,229,056

Lessons Learned (Hard-Won)

These are issues we hit during live deployment that aren't documented elsewhere:

Infrastructure

1. Azure Basic SKU public IPs are retired — The original repo uses Dynamic/Basic. New subscriptions get quota=0 for Basic. Use allocation_method = "Static" and sku = "Standard" instead.
2. Resource group name casing matters — Azure stores RG names case-insensitively but Terraform is case-sensitive. SentinelUdemy ≠ sentineludemy. When importing existing RGs, use the exact Azure casing or Terraform will try to destroy and recreate.
3. Never use terraform import on a production resource group — We imported an existing RG, and the next terraform apply attempted to destroy it (including all production Sentinel data). Always use a separate RG for lab resources.
4. vCPU quota is the #1 deployment blocker — Dev/Pay-As-You-Go subscriptions default to 4-10 vCPUs. The attack range needs 12. Request increase BEFORE running Terraform.
5. NSG rules must include WinRM (5985) — The original repo's Terraform creates the rule but it can get lost during state issues. Without it, Ansible can't reach the VMs.
6. Standard SKU public IPs require subnet-level NSG — NIC-level NSG association isn't sufficient. Associate NSG to the subnet as well.

Detection Engineering

7. Command-line logging must be enabled separately from process creation auditing — auditpol /set /subcategory:'Process Creation' /success:enable enables 4688 events, but the CommandLine field is EMPTY without the registry key ProcessCreationIncludeCmdLine_Enabled = 1. Both are required.
8. Sysmon is NOT installed by default — The DCR collects the Sysmon channel, but the binary must be installed on each VM. The repo doesn't do this.
9. macOS fork() safety kills Ansible — Set export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES before running attack-range.py on macOS.
10. Azure Linux VMs require RSA SSH keys — ed25519 keys are rejected. Generate with ssh-keygen -t rsa -b 4096.

Data Lake & Custom Tables

11. Azure CLI --plan only supports Analytics and Basic — NOT DataLake. The Data Lake tier switch must be done in the Defender portal UI. No CLI/API automation available.
12. Data Lake is only available on the primary workspace — Non-primary connected workspaces in the Defender portal don't get Data Lake access, table management, or KQL Jobs.
13. 90-120 minute latency after table creation/tier switch — Don't troubleshoot during this window.
14. KQL Job schema validation is strict — Column types must match exactly. Confidence field from MDTI is int but resists casting. We dropped it rather than fight the type system.
15. KQL Jobs can't cross-workspace query — workspace('other') syntax fails in KQL Jobs. Solution: create a cross-workspace DCR that routes data to the primary workspace.

Connectors & Licensing

16. MDE Sentinel connector requires standalone license — Defender for Servers P2 provisions MDE on VMs but doesn't grant tenant-level consent for the MDE data connector. Device* tables stay empty.
17. Entra ID SignInLogs require P1/P2 per user — But AuditLogs, ServicePrincipalSignInLogs, and ManagedIdentitySignInLogs work on the free tier via diagnostic settings.
18. XDR graph templates require Device tables* — Without them, built-in graph templates can't find entities or draw paths. Custom investigation graphs (from entity-mapped analytics rules) work fine.
19. Defender for Cloud connector works; MDE connector doesn't — Both are controlled by enable_defender_for_cloud in our Terraform. The DfC connector creates successfully; the MDE connector returns 401 "Missing consent".
20. MDTI connector provides 1.2M+ IOCs — These are in the ThreatIntelIndicators table and are free with Sentinel. Massive value for IOC matching.

Entity & Graph Work

21. Entities only appear on incidents if the analytics rule has entity mappings — No mappings = no entities = empty investigation graph.
22. Entity grouping on incidents requires selecting entity types — Not Name or Severity.
23. Blast radius requires Exposure Management data — Which requires MDE license for device graph data.
24. Custom Data Lake graphs require VS Code + Sentinel extension + Spark compute — The sentinel_lake and sentinel_graph Python libraries only exist in the managed Spark environment.

Prerequisites

• Azure subscription (Pay-As-You-Go or higher)
• 12+ vCPUs available in your region
• Terraform >= 1.0.0
• Python 3.7+ with Ansible 2.9+
• Azure CLI
• SSH key pair (RSA, not ed25519)

Quick Start

# 1. Pre-flight
cd scripts && ./preflight.sh ../terraform/terraform.tfvars

# 2. Deploy
cd ../terraform && terraform init && terraform plan -out=plan.tfplan && terraform apply plan.tfplan

# 3. Create Data Lake tables
cd ../scripts && ./create-datalake-tables.sh <rg> <workspace> <location>

# 4. Install Sysmon
az vm run-command invoke --resource-group <rg> --name <vm> --command-id RunPowerShellScript --scripts @install-sysmon.ps1

# 5. Enable command-line logging
az vm run-command invoke --resource-group <rg> --name <vm> --command-id RunPowerShellScript --scripts "reg add 'HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System\Audit' /v ProcessCreationIncludeCmdLine_Enabled /t REG_DWORD /d 1 /f"

# 6. Run attacks
export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
cd /path/to/ms-attack-range
python3 attack-range.py attack -a credential_dump

# 7. Deallocate when done
az vm deallocate --resource-group <rg> --name <vm> --no-wait

Documentation

Document	Description
GUIDE.md	Complete 9-phase deployment guide
ENTITIES-AND-CUSTOM-TABLES.md	Entity creation, custom tables, investigation graphs, IOC hunting (11 phases)
ATTACK-DETECTION-MAP.md	Maps every attack technique to its analytics rule, table, and tier

Cost

Item	Estimate
VMs (3x Standard_D2s_v3)	~$10/day when running
Sentinel ingestion (Analytics)	~$3-14/month at lab volumes
Data Lake KQL Jobs	<$1/month
Custom table ingestion	Negligible
Total with VMs deallocated	~$0/day
Total with VMs running	~$10/day

Credits

• Original attack range: oloruntolaallbert/MS-Attack-Range
• Atomic Red Team: redcanaryco/atomic-red-team
• IOC hunting approach: Microsoft Sentinel Blog
• Google SecOps composite detection research: Adoption Guide by Chris Martin

License

MIT