macos-incident-postmortem

macOS Incident Postmortem

When to Use This Skill

Use this skill when...	Use something else when...
GUI froze and you're not sure if the machine rebooted	Live-debugging a hung process — use sample / spindump
Investigating recent kernel panics or watchdog timeouts	Application crashes — open the per-app .crash / .ips directly
Cross-referencing "what was I doing at time T?" against logs	Active CPU diagnosis — see launchservices-health
Auditing whether the system is "due for a reboot"	Pre-incident hardening — wrong skill, this is forensics

Platform Guard

This skill is macOS-only. /Library/Logs/DiagnosticReports/, kern.boottime, pmset -g log, and the log command are Darwin-specific. Refuse to act if uname -s is not Darwin.

test "$(uname -s)" = "Darwin" || { echo "macos-plugin: not Darwin, refusing"; exit 1; }

Core Expertise

A macOS "incident" can mean any of:

• A kernel panic (system reset by the kernel)
• A WindowServer userspace watchdog timeout (the GUI froze and the user power-cycled)
• A LaunchServices / coreaudiod / mds-stores XPC stall (one daemon dragged the GUI down)
• A jetsam memory-pressure event (the kernel killed apps to reclaim RAM)
• A thermal throttle (CPU clamped to base frequency for minutes)
• A user-initiated force-reboot after a hang (no panic; just lost state)

The first job in a postmortem is distinguishing between actual reboots and GUI hangs. The 2026-04-22 incident that motivated this plugin looked like a crash to the user but last reboot showed no reboot — the kernel was fine, only the GUI stack hung.

The second job is timeline reconstruction: cross-reference Diagnostic Reports, kern.boottime, last reboot, last shutdown, and shell history to answer "what happened around time T?".

Gather the deterministic signals

Run the bundled script to gather every mechanical signal in one pass — reboot-vs-hang detection, per-category DiagnosticReports counts, the CPU-event offender histogram, the jetsam victim list, and the fixed reboot/hang classifier:

bash "${CLAUDE_SKILL_DIR}/scripts/macos-incident-postmortem.sh" --home-dir "$HOME"

Parse STATUS= and ISSUES: from the output. The script emits CLASSIFICATION= (REBOOT_PANIC / REBOOT_CLEAN / HANG_UI / HANG_CPU / HANG_JETSAM / HANG_OR_POWERCYCLE / UNKNOWN) with a CLASSIFICATION_REASON=, plus PANIC_COUNT= / HANG_COUNT= / JETSAM_COUNT= / CPU_RESOURCE_COUNT= etc., LATEST_PANIC= / LATEST_HANG= paths, and CPU_OFFENDERS: / JETSAM_VICTIMS: histograms.

Pass the incident time T as an epoch to enable the classifier (MACOS_PM_INCIDENT_EPOCH=<epoch> bash …), or supply an alternate report source with --reports-dir <path>. With no incident time the classifier reports UNKNOWN rather than guessing — feed it T once you know it.

The reboot/hang decision tree the script encodes:

Pattern	Interpretation
Boot at/after T with a .panic report	True reboot from a kernel panic
Boot at/after T, no panic	Clean restart (user-initiated or watchdog)
Boot before T with a .hang/.spindump.txt	GUI hang — machine did not reboot
Boot before T with a .cpu_resource.diag	Daemon CPU storm
Boot before T with a JetsamEvent-*	Memory-pressure kill
Boot before T, none of the above	Power loss or hard power-cycle

Diagnostic report category reference

/Library/Logs/DiagnosticReports/ collects everything macOS thinks is worth keeping; the script classifies each by filename suffix:

Pattern	Category	Severity
*.panic	Kernel panic	Critical
*.ips (process-specific)	Userspace crash report (Apple's modern format)	Per-process
*.crash	Legacy userspace crash	Per-process
*.cpu_resource.diag	Process exceeded CPU threshold (typ. 80% / 90s)	Hot daemon
*.wakeups_resource.diag	Process woke the system too often	Power drain
*.diskwrites_resource.diag	Process wrote too much to disk	I/O drain
*.hang	UI thread hang detection	GUI freeze
*.spindump.txt	Spindump capture from a hang	GUI freeze
JetsamEvent-*.ips	Kernel killed processes for memory pressure	RAM exhaustion

Note: Apple migrated most categories to the .ips extension circa Monterey. Older systems and some categories still produce legacy extensions. The script matches by suffix, not by exact filename.

last reboot reads /var/log/wtmp.X rotated logs. On modern macOS, also check the unified log when wtmp has rotated past the incident:

log show --predicate 'eventType == "stateEvent" AND (event == "boot" OR event == "shutdown")' \
  --last 7d --style syslog

Timeline Reconstruction (judgment)

With the deterministic classification and counts in hand, the remaining steps are the agent's job: read the panic backtrace, name suspect kexts, correlate with logs and shell activity, and write the narrative.

Step 1: Inspect the unified log around T

# Adjust the time window to bracket T
log show --start "2026-04-22 08:15:00" --end "2026-04-22 08:25:00" \
  --predicate 'subsystem == "com.apple.WindowServer" OR process == "launchservicesd" OR process == "coreaudiod"' \
  --style syslog \
  | head -500

Common signatures to grep for:

Signature	Meaning
WindowServer: watchdog	UI froze long enough to trip the watchdog
posix_spawn failures	Fork/exec storm — usually shell loops or runaway scripts
Jetsam Killing	Kernel killing processes for memory
_dispatch_*_timeout	Daemon stuck on a synchronous IPC call
Thermal pressure	CPU thermal-throttled

Step 2: Inspect the panic / hang report

The script's LATEST_PANIC= / LATEST_HANG= lines give you the file to read. Key fields in a panic report:

Field	Meaning
panic(cpu N caller ...)	The instruction that panicked; first line says why
Backtrace (CPU N)	Call stack at the time of panic
Mac OS version, Kernel version	OS state
System uptime in nanoseconds	Uptime at the moment of panic
last loaded kext / loaded kexts	Likely third-party suspect

Hang reports are spindump-style: one column per thread of the hung process (typically WindowServer or the offender daemon), with stack traces at sample intervals.

Step 3: Correlate with shell activity

# Most recent zsh history entries (assumes default zsh)
fc -l -t '%Y-%m-%d %H:%M:%S' -100

# Or directly:
tail -50 ~/.zsh_history

The zsh EXTENDED_HISTORY format : <epoch>:<elapsed>;<cmd> lets you grep for commands run within the incident window.

Step 4: Synthesize

Write a one-paragraph timeline including:

• Type of incident (panic / hang / jetsam / power-cycle)
• Time T and time-since-boot at T
• Top suspects (loaded kexts, busy daemons, low-memory processes)
• What recovery looked like (clean reboot / hard power-cycle / GUI returned on its own)

Common Patterns

The reboot-vs-hang one-liner, per-category report counts, the CPU-event offender histogram, and the jetsam victim list are all produced in one pass by the bundled script (scripts/macos-incident-postmortem.sh — see "Gather the deterministic signals" above). Read CLASSIFICATION=, the *_COUNT= keys, CPU_OFFENDERS:, and JETSAM_VICTIMS: from its output rather than re-running the equivalent shell pipelines. A sudden rise in any category count is a leading indicator before a major hang or panic; offenders/victims that recur are running near their CPU/memory budget.

Correlate with sleep/wake history

pmset -g log | grep -E 'Sleep|Wake|DarkWake' | tail -50

A spurt of DarkWake entries followed by a hang report often means a peripheral or sleep-assertion-holder is the trigger.

Skip List (Common False Suspects)

Suspect	Why it's usually NOT the cause
mds, mds_stores (Spotlight)	Heavy I/O is normal after large file changes; rarely panics
cloudd	High network use is normal during iCloud sync
bird (CloudKit)	Same as cloudd
Time Machine	Throttles itself; almost never the proximate cause
kernel_task at 100%	This is thermal management running, not a bug

If one of these is the only thing visible in your timeline, look harder — the real cause is usually a daemon that blocked on one of these, not the daemon itself.

Agentic Optimizations

Context	Command
All deterministic signals + classifier	bash "${CLAUDE_SKILL_DIR}/scripts/macos-incident-postmortem.sh" --home-dir "$HOME"
Classify against a known T	MACOS_PM_INCIDENT_EPOCH=<epoch> bash "${CLAUDE_SKILL_DIR}/scripts/macos-incident-postmortem.sh"
Last 5 boots	last reboot | head -5
Last 5 shutdowns	last shutdown | head -5
WindowServer log slice	log show --predicate 'subsystem == "com.apple.WindowServer"' --last 1h --style syslog | head -200

Quick Reference

Key paths

Path	Contents
/Library/Logs/DiagnosticReports/	All system-wide reports
~/Library/Logs/DiagnosticReports/	Per-user reports (rare; mostly legacy)
/var/log/wtmp.X	Reboot / shutdown record (read via last)
/var/log/asl/	ASL legacy logs (mostly unused in 2026)
/var/db/diagnostics/	Unified log binary database

Useful log show predicates

Predicate	Use
subsystem == "com.apple.WindowServer"	GUI hangs
process == "launchservicesd"	LS XPC stalls
process == "coreaudiod"	Audio daemon issues
eventType == "stateEvent"	Boot/shutdown/sleep
eventMessage CONTAINS[c] "hang"	Hang detection events
category == "ttsd"	Speech synthesis stalls

Time selectors

Selector	Example
--last <duration>	--last 1h, --last 1d
--start <ts> --end <ts>	--start "2026-04-22 08:00:00"
--info / --debug	Include lower-priority entries
--style syslog	Compact, grep-friendly

Decision Flow

Did `last reboot` advance near time T?
├─ YES → Kernel-level event
│   ├─ *.panic file present? → Panic; read backtrace
│   └─ No panic file → Clean restart (user-initiated or watchdog)
└─ NO → Userspace event
    ├─ *.hang or *.spindump.txt near T? → UI thread hang
    ├─ *.cpu_resource.diag spike near T? → Daemon CPU storm
    ├─ JetsamEvent-* near T? → Memory-pressure kill
    └─ None of the above → Power loss or hard power-cycle

Error Handling

Symptom	Cause	Fix
find: ...DiagnosticReports: Permission denied	Some user-level reports require sudo	Stick to system-wide; don't sudo unless necessary
last reboot empty	wtmp rotated past the incident	Use log show --predicate 'event == "boot"' instead
log show very slow / huge output	Default predicate is too broad	Narrow with --predicate and tighter time range
Reports only go back a few days	Apple rotates the diag dir aggressively	Check ~/Library/Logs/DiagnosticReports/ for backups; some events only persist as log show entries
Filenames with .ips not .crash	Modern macOS format change	Treat both as equivalent; same parser tools work

Related Skills

• launchservices-health — when the timeline points at launchservicesd, dig deeper there
• kitty-session-persistence — recover terminal state lost during the incident