# Incident report — NS-2026-1008

**Northstar Systems · controlled simulation**
Classification: internal · Status: closed · Author: case study author

> **Controlled simulated environment.** All identities, infrastructure,
> telemetry, indicators and incidents in this report are synthetic. See
> `lab/environment.md`.

---

## Executive summary

At 08:43:17 UTC on 2026-10-08 the `deploy` service account on `app-01`
authenticated from `198.51.100.24` after a burst of failed password attempts.
Within two minutes the session had spawned an interactive shell; within seven
minutes the account had elevated to root through a pre-existing
`NOPASSWD: ALL` sudo entry and established two forms of persistence. The
attacker then read an application secret file, exported 412,880 customer and
order rows from `db-01` (224,395,201 bytes), staged them locally, moved
laterally to the workstation `ws-02` over WinRM using a password found in a
deployment inventory, established a scheduled task there, and transferred the
staged data to `198.51.100.60`. The activity was detected 34m23s after initial
access by a correlation rule that was written during this investigation.

The incident was contained at 09:44:12 and eradicated by ~10:35. The database
export and its transfer to the collection endpoint are **corroborated** by
three independent sources. The workstation transfer is **probable** but its
contents cannot be confirmed. The mechanism by which the `deploy` password was
obtained is **unresolved**.

**Severity: HIGH.** Compromised valid account + root privilege + persistence +
lateral movement + confirmed bulk export of customer data.

---

## 1. Incident metadata

| Field | Value |
| --- | --- |
| Case ID | NS-2026-1008 |
| Analyst | case study author |
| Environment | Northstar Systems (simulated) |
| Incident window | 2026-10-08 08:43:17 – 2026-10-08 10:35 UTC |
| Initial alert | ALERT-001, 09:17:40 UTC |
| Affected assets | `app-01` (compromised), `db-01` (data accessed), `ws-02` (compromised), `edge-01` (exposure vector only) |
| ATT&CK | see `attack-mapping/attack-mapping.md` |

---

## 2. Initial alert (ALERT-001)

```
ALERT-001

Severity:  High
Detection: Authentication anomaly followed by privileged activity
User:      deploy
Host:      app-01
Source:    198.51.100.24
Time:      08:43 UTC

Fired at 09:17:40 UTC (34m23s after the successful authentication).
```

The alert deduplicated a failed-login cluster and a later sudo event into one
lead. It named no attacker, intent or path — that had to be established from
telemetry.

---

## 3. Triage

Each question was answered from evidence before moving on.

### Q1 — Did the authentication actually occur?

- **Data source:** `data/auth/app01_auth.log`
- **Evidence:** AUTH-014 (`Accepted password for deploy … 198.51.100.24 port 48219`)
- **Answer:** Yes. **FACT** — a valid credential was accepted.

### Q2 — Is the source normal?

- **Data source:** same file, full-day baseline
- **Result:** the only previously-seen source for `deploy` is `192.0.2.77`
  (AUTH-031, the backup host). `198.51.100.24` has no prior history on
  `app-01`.
- **Answer:** New source. **FACT** for the absence of history; **INFERENCE**
  that it is hostile (supported later by the command trail).

### Q3 — What happened immediately afterward?

- **Data source:** `data/linux/app01_auditd.log`
- **Evidence:** PROC-003 (shell at 08:43:19), PROC-004…008 (discovery).
- **Answer:** An interactive shell ran discovery commands within five seconds.
  **CORROBORATED FACT** (auth session + auditd execve).

### Q4 — Did the user perform privileged actions?

- **Data source:** `data/auth/app01_auth.log` + auditd
- **Evidence:** PRIV-001 (`sudo -l`), PRIV-002 (`sudo -i`, root shell at
  08:45:10).
- **Answer:** Yes, to root. **CORROBORATED** (sudo log + auditd EXECVE).

### Q5 — Is there persistence?

- **Data source:** auditd PATH watches, Windows Security
- **Evidence:** PERSIST-001 (cron), PERSIST-002 (authorized_keys),
  PERSIST-003 (scheduled task on `ws-02`).
- **Answer:** Three mechanisms across two hosts. **FACT** (path watches).

Triage conclusion: confirmed compromise, high severity, escalate to IR.

---

## 4. Investigation steps

The format used throughout: question → hypothesis → source → query → evidence
→ interpretation → confidence → alternative → next action. Repeat queries are
reproducible via `scripts/queries/q00-all-queries.sh`.

### STEP 1 — Establish initial access

- **Question:** How did the attacker get in?
- **Hypothesis:** H-001 (account compromised) vs. a web-tier exploit.
- **Data source:** `app01_auth.log`, `edge01_nginx.log`, firewall.
- **Query:** Q1, Q13, Q15.
- **Evidence:** AUTH-001…014; edge-01 shows scan noise only (0 exploit
  indicators).
- **Interpretation:** valid-account access, not exploitation. The failure
  burst immediately precedes the success.
- **Confidence:** High.
- **Alternative:** a web app pivot that left no nginx log. Rejected: the
  firewall shows the SSH connection arriving directly from the internet.
- **Next:** what privileges did the account hold?

### STEP 2 — Determine privilege and persistence

- **Question:** Could the attacker persist and escalate on `app-01`?
- **Hypothesis:** H-003 (persistence established).
- **Data source:** `app01_auditd.log`, `app01_auth.log`, PATH watches.
- **Query:** Q4, Q5, Q11.
- **Evidence:** PRIV-001/002; PERSIST-001/002; PROC-015.
- **Interpretation:** root was available through a pre-existing sudo entry and
  two persistence mechanisms were created. No exploit was needed at any stage.
- **Confidence:** High.
- **Alternative:** legitimate automation making these changes. Rejected: the
  timing follows the hostile session and the parent process is the attacker's
  sudo shell, not the deploy tooling.
- **Next:** was data accessed?

### STEP 3 — Assess data access

- **Question:** Did the attacker reach the sensitive data repository?
- **Hypothesis:** H-006 (data exfiltrated).
- **Data source:** `db01_postgresql.log`, flows.
- **Query:** Q6, Q7.
- **Evidence:** APP-001…004, NET-001.
- **Interpretation:** the app's read-only credential was used to run a
  412,880-row `COPY`.
- **Confidence:** High.
- **Alternative:** a legitimate reporting job. Rejected: the query's source is
  a psql client on `app-01` opened 14 minutes after `FILE-001` was read, not
  the application pool (which connects as `app`, not `app_ro`).
- **Next:** did anything leave the network?

### STEP 4 — Trace egress

- **Question:** Was the exported data transferred externally?
- **Hypothesis:** H-006 (upheld or not).
- **Data source:** `app01_flows.jsonl`, `firewall.log`.
- **Query:** Q7.
- **Evidence:** FILE-004 (224,395,201 bytes) + PROC-017 (curl POST) +
  NET-002 (224,402,933 bytes to `198.51.100.60`).
- **Interpretation:** the byte counts agree within HTTP/TLS framing; the flow
  window matches the curl process. Transfer confirmed.
- **Confidence:** High for transfer, Medium for what the destination did with
  it (U-002).
- **Alternative:** a backup or telemetry upload to a legitimate service.
  Rejected: `198.51.100.60` has no business relationship and no DNS history.
- **Next:** was there movement beyond `app-01`?

### STEP 5 — Lateral movement

- **Question:** Did the attacker reach other hosts?
- **Hypothesis:** H-004 (db-01 direct access), H-007 (edge pivot).
- **Data source:** flows, Windows Security, firewall.
- **Query:** Q8, Q9, Q11, Q16, HUNT-003.
- **Evidence:** NET-003/006, PRIV-003, PERSIST-003.
- **Interpretation:** one hop to `ws-02` over WinRM; no evidence elsewhere.
- **Confidence:** High. H-004 and H-007 disproven.
- **Alternative:** movement that produced no completed flow. Not excluded
  (U-004).
- **Next:** confirm scope with hunts.

### STEP 6 — Threat hunts

HUNT-001 (other accounts: **disproven**), HUNT-002 (persistence: **3
mechanisms, no others found**), HUNT-003 (lateral: **one hop; db-01 direct
access disproven**), HUNT-004 (staging: **corroborated on app-01, probable on
ws-02**). Details in `hunts/`.

---

## 5. Hypothesis register

| ID | Hypothesis | Status | Evidence | Confidence |
| --- | --- | --- | --- | --- |
| H-001 | `deploy` account compromised | Confirmed | AUTH-014, PROC-003 | High |
| H-002 | Credential obtained via reuse/stuffing/prior breach | Unresolved | AUTH-001…013 | Low |
| H-003 | Persistence established | Confirmed | PERSIST-001/002/003 | High |
| H-004 | `db-01` accessed directly from the internet | Disproven | HUNT-003, Q16 | High |
| H-005 | Additional accounts compromised externally | Disproven | HUNT-001 | High |
| H-006 | Customer data exfiltrated | Probable | APP-003, FILE-004, NET-002/007 | Medium–High |
| H-007 | `edge-01` exploited and used to pivot | Disproven | nginx log, firewall, HUNT-003 | High |
| H-008 | `ws-02` staged files were transferred | Probable | FILE-005, NET-007 | Medium |

---

## 6. Correlated evidence

| Finding | Auth | Endpoint | Network | File/App | Confidence |
| --- | :-: | ---: | ---: | ---: | --- |
| Initial access | ✓ | ✓ | — | — | High |
| Privilege escalation | ✓ | ✓ | — | — | High |
| Persistence (app-01) | — | ✓ | — | ✓ | High |
| Credential access (files) | — | ✓ | — | ✓ | High |
| Internal reconnaissance | — | ✓ | ✓ | — | High |
| Database export | — | ✓ | ✓ | ✓ | High |
| Lateral movement | ✓ | ✓ | ✓ | — | High |
| Workstation collection | — | ✓ | ✓ | ✓ | Medium |
| Exfiltration (app-01) | — | ✓ | ✓ | ✓ | High |
| Exfiltration (ws-02) | — | — | ✓ | ✓ | Medium |
| Persistence (ws-02) | — | ✓ | — | ✓ | High |

---

## 6b. Evidence and visibility gaps

| ID | Gap | Effect on confidence |
| --- | --- | --- |
| GAP-001 | No packet capture on the internal or egress segments. | Content of the ws-02 transfer cannot be confirmed; exfil is inferred from flow volume (U-003). |
| GAP-002 | DNS resolver log gap 08:52:10–09:03:22 (11m12s). | The gap overlaps credential access and persistence. The attacker's exfil used raw IPs, so the gap probably hides nothing relevant — but "probably" is the ceiling. |
| GAP-003 | `ws-02` clock runs +38s fast. | All ws-02 timestamps must be normalised; the timeline labels them. Within the ±1s of other hosts, ordering is unaffected. |
| GAP-004 | No domain-controller or VPN log collection. | A second identity path cannot be searched (U-006). |
| GAP-005 | No file-read audit beyond the secrets-file watches. | The `.env` read is evidenced because a watch existed; other files may have been read silently. |
| GAP-006 | Flow export records only completed/refused connections. | Silently-dropped probes leave no record (U-004). |
| GAP-007 | `db-01` has no host-level EDR. | Only server-log evidence exists for DB activity; no process tree on the database host. |

---

## 7. Timeline

The full timeline is machine-generated in `timeline/timeline.csv` from the
evidence manifest. Extract (UTC; ws-02 records normalised from +38s):

```text
08:41:12  app-01  AUTH-001..013  13 failed password attempts from 198.51.100.24
08:43:17  app-01  AUTH-014       Accepted password for deploy (INITIAL ACCESS)
08:43:19  app-01  PROC-003       interactive shell
08:43:22  app-01  PROC-004       id; uname -a; hostname -I
08:43:47  app-01  FILE-001       /opt/northstar/app/.env read
08:45:10  app-01  PRIV-002       sudo -i -> root shell
08:45:31  app-01  FILE-002       inventory.ini read (ws-02 credential)
08:49:15  app-01  PERSIST-001    /etc/cron.d/.sysupdate written
08:50:22  app-01  PERSIST-002    attacker key -> authorized_keys
08:52:10  app-01  DNS-002        resolver gap begins
08:57:44  db-01   APP-001        app_ro session authorised
08:58:04  db-01   APP-003        COPY (412,880 rows) begins
09:03:22  app-01  DNS-003        resolver gap ends
09:03:41  db-01   APP-003        COPY completes, 224,395,201 bytes
09:03:41  app-01  FILE-004       staged to /tmp/.cache/cust.csv
09:03:51  app-01/ws-02 NET-003/NET-006  WinRM lateral logon as svc_backup
09:06:12  ws-02   FILE-005       work share staged locally (11,632,640 bytes)
09:06:59  ws-02   PERSIST-003    scheduled task \OneDriveUpdate created
09:08:12  app-01  NET-002        HTTPS POST of export to 198.51.100.60
09:11:31  ws-02   NET-007        HTTPS transfer to 198.51.100.60
09:17:40  siem    ALERT-001      DETECTION (MTTD 34m23s)
09:44:12  app-01  ACCOUNT-001    deploy disabled, session killed (CONTAINMENT)
09:47:31  app-01  AUTH-017       same source rejected
09:52:08  app-01  AUTH-018       attacker key rejected
10:05:44  app-01  PERSIST-004    cron drop-in removed
10:07:12  app-01  PERSIST-005    authorized_keys restored
10:14:03  app-01  FILE-006       staged export shredded
10:34:08  ws-02   PERSIST-006    scheduled task deleted
10:38:12  db-01   APP-005        app_ro credential rotated, app restarted
10:44:02  app-01  AUTH-033       rebuilt host reachable via key-only account
```

---

## 8. Severity

```
Severity: HIGH

Reason:
  - valid account with root escalation (PRIV-002)
  - three persistence mechanisms across two hosts
  - confirmed bulk export of customer data (APP-003, NET-002)
  - lateral movement (NET-006)

Uncertainty:
  - whether the exported data was retained (U-002)
  - whether the ws-02 files were transferred (U-003)
```

Severity was raised from Medium to High by the confirmed root privilege plus
the confirmed data export, not by the alert's initial rating.

---

## 9. Response decisions (evidence → risk → decision → action → validation)

| Step | Evidence | Risk | Decision | Action | Validation |
| --- | --- | --- | --- | --- | --- |
| Contain account | AUTH-014, PRIV-002 | Further privileged use | Disable `deploy` | ACCOUNT-001 at 09:44:12 | AUTH-017 (09:47:31) and AUTH-018 (09:52:08) rejected |
| Block source | AUTH-001…018 | Return access | Firewall block | NET-013 at 09:44:30 | firewall drops logged at 09:47:31 / 09:52:08 |
| Remove persistence | PERSIST-001/002/003 | Re-entry / re-execution | Remove all three | 10:05:44 / 10:07:12 / 10:34:08 | PERSIST-004/005/006; `find /etc/cron.d` empty |
| Rotate secrets | FILE-001, FILE-002 | Credential reuse | Rotate DB + `svc_backup` | 10:38 | APP-005 (new credential in use) |
| Rebuild host | full session evidence | Unknown residue | Rebuild `app-01`, re-image `ws-02` | 10:38–12:26 | AUTH-033 via key-only account; health checks green |

---

## 10. Containment validation

```text
Before containment
  AUTH-014  08:43:17  Accepted password for deploy from 198.51.100.24

Action
  ACCOUNT-001  09:44:12  deploy disabled, session killed

After containment
  AUTH-017  09:47:31  Failed password for deploy from 198.51.100.24
  AUTH-018  09:52:08  Failed publickey for deploy from 198.51.100.24

Conclusion
  No successful authentication for deploy from the attacker source after
  containment, in the available telemetry.
```

---

## 11. Eradication validation

| Artifact | Removed | Validation evidence | Result |
| --- | --- | --- | --- |
| `/etc/cron.d/.sysupdate` | 10:05:44 | PERSIST-004 + crontab artifact | path absent; `find` empty |
| attacker key in `authorized_keys` | 10:07:12 | PERSIST-005 + restored artifact | key absent from restored file |
| `%TEMP%\msupd` staging | with re-image | FILE-006 (app-01 staging shredded) | local copy gone |
| `\OneDriveUpdate` task | 10:34:08 | PERSIST-006 (4699) | task deleted |

Eradication does not undo the transfers at NET-002/NET-007; those are
historical.

---

## 12. Recovery validation

- `app-01` rebuilt and reachable via the key-only `ops-deploy` account from the
  bastion (AUTH-033, 10:44:02).
- `db-01` application credential rotated and the application restarted
  (APP-005, 10:38:12).
- Health checks green from 12:26:40 (`edge01_nginx.log`, monitor poll).
- Re-run hunts HUNT-001…004 against the post-containment window: no new
  attacker telemetry.

---

## 13. Root cause

- **Immediate cause:** valid `deploy` credentials were used from the internet
  against an SSH service exposed via a NAT rule (8022 → `app-01:22`).
- **Contributing factors:** a shared, long-lived service-account password;
  password authentication enabled for that account; a pre-existing
  `NOPASSWD: ALL` sudo grant; plaintext secrets in `.env` and `inventory.ini`;
  a flat internal network allowing `app-01` → `ws-02`; unrestricted egress.
- **Detection failure:** no correlation between an authentication-failure
  burst and a subsequent success; no alert on cron/authorized_keys writes; no
  egress-volume alert. The activity was detected after the fact, by a rule
  written during this investigation.
- **Control failure:** no bastion or MFA in front of the service-account SSH;
  over-broad sudo; secrets readable by the service account; no egress policy.
- **Root cause:** service-account credential management and network exposure —
  a shared password with excessive privilege, reachable directly from the
  internet, with no compensating control.

---

## 14. Detection gap analysis

| Gap | Behaviour missed | New detection | Effect |
| --- | --- | --- | --- |
| Failures not correlated with success | AUTH-001…014 | DET-001 v2 | 34m MTTD for this class becomes seconds |
| Service-account sudo unseen | PRIV-002 | DET-002 | Root escalation by a service account alerts |
| Cron writes unseen | PERSIST-001 | DET-003 | Persistence by interactive session alerts |
| Login→privilege not correlated | AUTH-014 + PRIV-002 | DET-004 v2 | Whole sequence alerts with one rule |
| authorized_keys writes unseen | PERSIST-002 | existing site rule `ssh-authorized-keys-modified` | Password-independent persistence alerts |
| Egress volume unseen | NET-002/NET-007 | DET-006 | Largest flows on the segment alert |
| Remote logon not baselined | NET-006 | DET-007 | Non-management WinRM alerts |

---

## 15. Detections

Each detection was derived from observed behaviour and tested against the
dataset (`detections/tests/run-tests.sh`, 8/8 pass). The v1→v2 tuning for
DET-001, DET-004, DET-006 and DET-007 is documented in the query scripts; the
short version:

| ID | Detects | v1 result | v2 result |
| --- | --- | --- | --- |
| DET-001 | failure burst → success | 1 TP, 1 FP (backup retry) | 1 TP, 0 FP |
| DET-002 | service-account sudo root | 1 TP, 0 FP | 1 TP, 0 FP |
| DET-003 | cron drop-in write | 1 TP, 0 FP (1 latent FP noted) | 1 TP, 0 FP |
| DET-004 | login → privileged exec | 1 TP, 1 FP (ops maintenance) | 1 TP, 0 FP |
| DET-005 | authorized_keys modified | existing site rule | n/a |
| DET-006 | large egress | 2 TP, 1 FP (backup appliance, prod) | 2 TP, 0 FP |
| DET-007 | WinRM from non-management host | 1 TP, 1 FP (agent path, prod) | 1 TP, 0 FP |

Sigma rules: `detections/sigma/`. Runnable queries: `detections/queries/`.

---

## 16. SOC metrics (simulated)

| Metric | Value |
| --- | --- |
| Initial malicious activity | 08:43:17 |
| Detection (ALERT-001) | 09:17:40 |
| **MTTD** | **34m23s** |
| Triage start | 09:19:05 |
| Alert-to-triage | 1m25s |
| Escalation | 09:31:12 |
| Containment | 09:44:12 |
| Alert-to-containment | 26m32s |
| Activity-to-containment | 60m55s |
| Eradication complete | ~10:35 |
| Alert-to-eradication | ~1h17m |
| Recovery validated | 12:26:40 |
| Alert-to-recovery | 3h09m |

All values are simulated, computed from the dataset.

---

## 16b. Findings

### F-001 — Valid-account access from the internet

- **Finding:** `deploy` was used to access `app-01` from `198.51.100.24`
  without authorisation.
- **Evidence:** AUTH-014, AUTH-015, PROC-003.
- **Confidence:** High.
- **Limitation:** how the password was obtained is unresolved (U-001).
- **Impact:** full interactive access to the application server.

### F-002 — Excessive privilege held by the service account

- **Evidence:** PRIV-001, PRIV-002.
- **Confidence:** High.
- **Impact:** one command produced root; this turned a stolen password into a
  host compromise.

### F-003 — Persistence established on two hosts

- **Evidence:** PERSIST-001, PERSIST-002, PERSIST-003.
- **Confidence:** High.
- **Impact:** access survived the session; all three required explicit removal.

### F-004 — Bulk export of customer data from `db-01`

- **Evidence:** APP-001…004, FILE-004.
- **Confidence:** High (export), Medium (retention).
- **Impact:** 412,880 customer/order rows staged and prepared for transfer.

### F-005 — Data transferred to an external endpoint

- **Evidence:** NET-002, PROC-017; NET-007 (probable).
- **Confidence:** High for app-01, Medium for ws-02.
- **Impact:** the exported data left the network.

### F-006 — Lateral movement to `ws-02`

- **Evidence:** NET-003, NET-006, PRIV-003, FILE-002.
- **Confidence:** High.
- **Impact:** a second host was compromised with local administrator rights.

### F-007 — Detection gaps permitted a 34-minute dwell time

- **Evidence:** AUTH-001…014 vs. the absence of any alert before ALERT-001.
- **Confidence:** High.
- **Impact:** persistence and exfiltration completed before detection.

### F-008 — No evidence of direct `db-01` compromise

- **Evidence:** HUNT-003, firewall/flow records.
- **Confidence:** High within the searched sources.
- **Impact:** scope limited to `app-01` and `ws-02`; negative finding.

---

## 16c. Limitations

- No packet capture: content of transfers, and any traffic that produced no
  flow record, cannot be examined (GAP-001, GAP-006).
- DNS gap 08:52:10–09:03:22 (GAP-002).
- `ws-02` clock skew +38s (GAP-003).
- No DC/VPN logs (GAP-004).
- `db-01` and `edge-01` have no host-level EDR.
- The dataset is synthetic; any resemblance in timing or naming to a real
  incident is coincidental.

The full uncertainty register is `uncertainty.md`.

---

## 17. Reproduction

```bash
# 1. regenerate the dataset (byte-identical)
python3 scripts/generate_dataset.py --check

# 2. verify file hashes
cd data && sha256sum -c SHA256SUMS && cd ..

# 3. verify every evidence citation in this report
python3 scripts/verify_evidence.py --strict

# 4. run the investigation queries (outputs quoted in section 4)
bash scripts/queries/q00-all-queries.sh

# 5. run the detection suite
bash detections/tests/run-tests.sh

# rebuild the timeline from the manifest
python3 scripts/build_timeline.py
```

Expected results are recorded in `detections/tests/*.expected` and compared
automatically by `run-tests.sh`.

---

## 17b. Detection engineering notes

Detections were not written first and fitted to the story afterwards. The order
was: run the query on the raw dataset → observe what it returned → tune against
the benign baseline in the same dataset (`AUTH-031`, the ops maintenance login,
the backup appliance) → re-run and record. The false positives are real rows in
this dataset, not invented examples, which is why the v1 outputs in
`detections/tests/` contain them.

---

## 17c. Final quality audit

Every finding traces to evidence IDs that resolve in
`evidence/manifest.yaml` and re-hash cleanly under
`scripts/verify_evidence.py`. Timeline rows are generated from the same
manifest. ATT&CK techniques without evidence were left out (see
`attack-mapping/attack-mapping.md`). Detections were all executed against the
dataset. Response actions all carry a validation record.

## 18. Recommendations

1. Remove password authentication from service accounts; require keys through a
   bastion with MFA. Remove the internet NAT rule to `app-01:22`.
2. Reduce sudo: no `NOPASSWD: ALL` for service accounts; scope commands.
3. Move secrets out of files readable by the service account into a secret
   store; rotate `app_ro` and `svc_backup` credentials.
4. Segment the internal network so `app-01` cannot reach the workstation
   segment; apply default-deny egress with an allow-list.
5. Deploy DET-001, DET-002, DET-003, DET-004, DET-006, DET-007; gate alerts on
   the benign baselines identified here.
6. Onboard domain-controller and VPN logs, and close the DNS gap.
7. Enable packet capture or egress content logging on the boundary so future
   transfers can be characterised rather than estimated.