simulation-validator

Simulation Validator

Goal

Provide a three-stage validation protocol: pre-flight checks, runtime monitoring, and post-flight validation for materials simulations.

Requirements

• Python 3.10+
• No external dependencies (uses Python standard library only)
• Works on Linux, macOS, and Windows

Inputs to Gather

Before running validation scripts, collect from the user:

Input	Description	Example
Config file	Simulation configuration (JSON/YAML)	simulation.json
Log file	Runtime output log	simulation.log
Metrics file	Post-run metrics (JSON)	results.json
Required params	Parameters that must exist	dt,dx,kappa
Valid ranges	Parameter bounds	dt:1e-6:1e-2

Decision Guidance

When to Run Each Stage

Is simulation about to start?
├── YES → Run Stage 1: preflight_checker.py
│         └── BLOCK status? → Fix issues, do NOT run simulation
│         └── WARN status? → Review warnings, document if accepted
│         └── PASS status? → Proceed to run simulation
│
Is simulation running?
├── YES → Run Stage 2: runtime_monitor.py (periodically)
│         └── Alerts? → Consider stopping, check parameters
│
Has simulation finished?
├── YES → Run Stage 3: result_validator.py
│         └── Failed checks? → Do NOT use results
│                            → Run failure_diagnoser.py
│         └── All passed? → Results are valid

Choosing Validation Thresholds

Metric	Conservative	Standard	Relaxed
Mass tolerance	1e-6	1e-3	1e-2
Residual growth	2x	10x	100x
dt reduction	10x	100x	1000x

Script Outputs (JSON Fields)

Script	Output Fields
scripts/preflight_checker.py	report.status, report.blockers, report.warnings
scripts/runtime_monitor.py	alerts, residual_stats, dt_stats (alerts include NaN/Inf/overflow detection, residual growth, and dt collapse)
scripts/result_validator.py	checks, confidence_score, failed_checks, status (PASS / FAIL / INSUFFICIENT_DATA); confidence_score is null when no check ran
scripts/failure_diagnoser.py	probable_causes, recommended_fixes

Three-Stage Validation Protocol

Stage 1: Pre-flight (Before Simulation)

1. Run scripts/preflight_checker.py --config simulation.json
2. BLOCK status: Stop immediately, fix all blocker issues
3. WARN status: Review warnings, document accepted risks
4. PASS status: Proceed to simulation

Note: preflight_checker.py validates required keys, numeric ranges, output-directory access, and disk space. It does not evaluate numerical stability (CFL / diffusion-Fourier). For explicit stability gating use skills/core-numerical/numerical-stability/scripts/cfl_checker.py.

python3 scripts/preflight_checker.py \
    --config simulation.json \
    --required dt,dx,kappa \
    --ranges "dt:1e-6:1e-2,dx:1e-4:1e-1" \
    --min-free-gb 1.0 \
    --json

Stage 2: Runtime (During Simulation)

1. Run scripts/runtime_monitor.py --log simulation.log periodically
2. Configure alert thresholds based on problem type
3. Stop simulation if critical alerts appear

python3 scripts/runtime_monitor.py \
    --log simulation.log \
    --residual-growth 10.0 \
    --dt-drop 100.0 \
    --json

Stage 3: Post-flight (After Simulation)

1. Run scripts/result_validator.py --metrics results.json
2. All checks PASS: Results are valid for analysis
3. Any check FAIL: Do NOT use results, diagnose failure

python3 scripts/result_validator.py \
    --metrics results.json \
    --bound-min 0.0 \
    --bound-max 1.0 \
    --mass-tol 1e-3 \
    --json

For variational / gradient-flow models (Allen-Cahn, Cahn-Hilliard), add --variational to enforce a strict monotone non-increasing energy check.

Failure Diagnosis

When validation fails:

python3 scripts/failure_diagnoser.py --log simulation.log --json

Conversational Workflow Example

User: My phase field simulation crashed after 1000 steps. Can you help me figure out why?

Agent workflow:

1. First, check the log for obvious errors:

   python3 scripts/failure_diagnoser.py --log simulation.log --json
   

2. If diagnosis suggests numerical blow-up, check runtime stats:

   python3 scripts/runtime_monitor.py --log simulation.log --json
   

3. Recommend fixes based on findings:
   • If residual grew rapidly → reduce time step
   • If dt collapsed → check stability conditions
   • If NaN detected → check initial conditions

Error Handling

Error	Cause	Resolution
Config not found	File path invalid	Verify config path exists
Non-numeric value	Parameter is not a number	Fix config file format
out of range	Parameter outside bounds	Adjust parameter or bounds
Output directory not writable	Permission issue	Check directory permissions
Insufficient disk space at <path>	Disk nearly full on the output volume	Free up space or reduce output
Invalid parameter name	--required name has disallowed characters	Use only letters, digits, _, ., -
range max ... must be greater than min	Inverted/degenerate --ranges or bounds	Ensure max > min
must be a finite positive number	nan/inf/negative threshold supplied	Pass a finite positive value
Log file too large	Log exceeds the 500 MB parse cap	Truncate or pre-filter the log

Interpretation Guidance

Status Meanings

Status	Meaning	Action
PASS	All checks passed	Proceed with confidence
WARN	Non-critical issues found	Review and document
BLOCK	Critical issues found	Must fix before proceeding

Confidence Score Interpretation

Score	Meaning
1.0	All validation checks passed → proceed with confidence
0.75+	Most checks passed, minor issues
0.5-0.75	Significant issues, review carefully
< 0.5	Major problems, do not trust results
null (status INSUFFICIENT_DATA)	No recognized metrics fields; no check ran — NOT a pass. Inspect the metrics file.

A requested bound (--bound-min/--bound-max) with no matching field_min/field_max in the metrics is reported as a failed bounds_unverifiable check, never a vacuous pass. For variational/gradient-flow runs, pass --variational (or set "energy_variational": true in the metrics) to enforce a strict monotone non-increasing energy check (energy_monotone); otherwise a weaker energy_net_decrease check is used, which does not detect mid-run spikes.

Common Failure Patterns

Pattern in Log	Likely Cause	Recommended Fix
NaN, Inf, overflow	Numerical instability	Reduce dt, increase damping
max iterations, did not converge	Solver failure	Tune preconditioner, tolerances
out of memory	Memory exhaustion	Reduce mesh, enable out-of-core
dt reduced	Adaptive stepping triggered	May be okay if controlled

Verification checklist

Do not trust a validation verdict until each applicable item below is satisfied with the concrete artifact named. Record these in your summary to the user.

• Ran result_validator.py --json and confirmed results.status is PASS (not INSUFFICIENT_DATA) AND results.confidence_score == 1.0; a null score or INSUFFICIENT_DATA means no check ran — treat as unverified, not as a pass.
• Listed results.checks and confirmed every requested check actually appears (e.g. mass_conserved, bounds_satisfied, no_nan, and energy_monotone/energy_net_decrease); confirmed results.failed_checks is empty and contains no bounds_unverifiable entry (which means a requested bound had no field_min/field_max to compare against).
• For variational/gradient-flow models (Allen-Cahn, Cahn-Hilliard), passed --variational (or set "energy_variational": true) so energy_monotone is enforced; recorded that the weaker energy_net_decrease was NOT relied on, since it cannot detect mid-run energy spikes.
• Recorded the mass drift tolerance used (--mass-tol, default 1e-3) and confirmed it matches the Conservative/Standard/Relaxed column appropriate to the run; did not silently accept the default for a tight-conservation problem.
• Ran runtime_monitor.py --json and recorded residual_stats (min/max/last) and dt_stats; confirmed there are no alerts for NaN/Inf/overflow, residual growth above --residual-growth, or dt collapse below --dt-drop.
• Confirmed numerical stability was gated separately via core-numerical/numerical-stability/scripts/cfl_checker.py (CFL/Fourier limit) — preflight_checker.py does NOT evaluate CFL/Fourier and a PASS preflight says nothing about temporal/spatial stability.
• On any FAIL or alert, ran failure_diagnoser.py --json and recorded the probable_causes/recommended_fixes, rather than reusing the results.

Common pitfalls & rationalizations

Tempting shortcut	Why it's wrong / what to do
"Preflight passed, so the run is numerically stable."	preflight_checker.py checks required keys, ranges, output-dir writability, and disk space only. It does NOT compute CFL/Fourier. Gate stability with cfl_checker.py separately.
"result_validator printed a confidence score, so results are good."	An empty or unrecognized metrics file returns confidence_score: null and status INSUFFICIENT_DATA — that is "no check ran", not a pass. Verify recognized fields are present and status == PASS.
"Energy ends lower than it started, so the dissipative run is fine."	The default energy_net_decrease only compares first vs last and misses mid-run spikes. For gradient-flow models use --variational to enforce the strict monotone energy_monotone check.
"I asked for bounds and didn't get a bounds_satisfied: false, so bounds hold."	If field_min/field_max are absent the validator emits bounds_unverifiable (a FAILED check), never a vacuous pass. Ensure the metrics file actually carries the field extrema.
"The simulation finished without crashing, so the results are trustworthy."	Run completion is not correctness. Verify mass conservation, energy behavior, physical bounds, and a clean runtime_monitor alert list before using results.
"dt got smaller during the run, so the solver is failing."	runtime_monitor dt-collapse is direction-aware (running-max vs current) and only alerts past --dt-drop; a controlled adaptive ramp is expected. Check the actual dt_stats and whether an alert fired.
"I'll just use the default thresholds."	Defaults (--mass-tol 1e-3, --residual-growth 10, --dt-drop 100) are the Standard column; a conservation-critical problem needs the Conservative tolerances. Pick thresholds for the physics, then record them.

Security

Input Validation

• Config file paths are validated for existence before parsing; non-existent paths produce clear errors (exit code 2)
• --required parameter names are validated against a safe-character allowlist (^[A-Za-z0-9_.-]+$); names with shell metacharacters are rejected
• --ranges entries are parsed as name:min:max with finite numeric bounds enforced and max > min required
• --min-free-gb is validated as a finite positive number (negatives, zero, nan, inf rejected)
• --residual-growth and --dt-drop thresholds are validated as finite positive numbers
• --bound-min and --bound-max are validated as finite numbers (nan/inf rejected), and --bound-max > --bound-min is enforced; --mass-tol is validated as a finite positive number
• Invalid input exits with code 2 and an explanatory message

File Access

• preflight_checker.py reads a single user-specified config file (JSON/YAML) and checks disk space on the volume hosting the resolved output directory
• runtime_monitor.py reads a single log file specified by --log; log files are size-limited (500 MB max) and rejected before parsing if larger
• result_validator.py reads a single metrics file (JSON) specified by --metrics
• failure_diagnoser.py reads a single log file specified by --log; log files are size-limited (500 MB max) before parsing
• No scripts write to the filesystem; all output goes to stdout

Tool Restrictions

• Read: Used to inspect script source, references, config files, and simulation logs
• Bash: Used to execute the four Python validation scripts (preflight_checker.py, runtime_monitor.py, result_validator.py, failure_diagnoser.py) with explicit argument lists
• Write: Used to save validation reports; writes are scoped to the user's working directory
• Grep/Glob: Used to locate log files, config files, and search references

Safety Measures

• No eval(), exec(), or dynamic code generation
• All subprocess calls use explicit argument lists (no shell=True)
• failure_diagnoser.py uses hardcoded, pre-compiled diagnostic regex patterns; runtime_monitor.py accepts optional --residual-pattern / --dt-pattern overrides that are compiled with re.compile (no eval) and applied only to the user's own log
• Diagnostic strings emitted in output are drawn from the skill's fixed cause/fix table, not interpolated from raw log content

Limitations

• Not a real-time monitor: Scripts analyze logs after-the-fact
• Regex-based: Log parsing depends on pattern matching; may miss unusual formats
• No automatic fixes: Scripts diagnose but don't modify simulations

References

• references/validation_protocol.md - Detailed checklist and criteria
• references/log_patterns.md - Common failure signatures and regex patterns

Version History

• v1.2.2 (2026-06-24): Added a Verification checklist (evidence-based, tied to the four scripts' JSON outputs) and a Common pitfalls & rationalizations table to harden agent interpretation of validation verdicts.
• v1.2.0 (2026-06-23): Corrected diagnostic regexes (no false convergence/blow-up on healthy logs), direction-aware dt-collapse detection, NaN/Inf scan in runtime monitor, strict variational energy check, non-vacuous bounds/confidence, config-relative output-dir + correct-volume disk check, and implemented the documented input-validation/file-size safeguards
• v1.1.0 (2024-12-24): Enhanced documentation, decision guidance, Windows compatibility
• v1.0.0: Initial release with 4 validation scripts