Run Earthly E2E validation for YAML schema contracts. Use when validating YAML schema changes, testing schema contracts against live ClickHouse, or regenerating Python types, DDL, and docs from YAML. For SQL schema design and optimization, use clickhouse-architect skill instead.
Limited to specific tools
Additional assets for this skill
This skill is limited to using the following tools:
Earthly requires Docker. Start Colima before running:
colima start
Check if running:
docker ps # Should not error
Required for +test-schema-validate and +test-schema-e2e:
doppler configure set token <token_from_1password>
doppler setup --project gapless-network-data --config prd
brew install earthly
cd /Users/terryli/eon/gapless-network-data
colima start # If not already running
earthly +test-schema-generate
cd /Users/terryli/eon/gapless-network-data
colima start # If not already running
./scripts/earthly-with-doppler.sh +test-schema-e2e
cd /Users/terryli/eon/gapless-network-data
earthly +all
After running +test-schema-generate or +test-schema-e2e, check ./earthly-artifacts/:
| Path | Contents |
|---|---|
types/blocks.py | Pydantic + TypedDict models |
types/__init__.py | Package init |
ddl/ethereum_mainnet.sql | ClickHouse DDL |
docs/ethereum_mainnet.md | Markdown documentation |
For E2E, artifacts are under e2e/types/, e2e/ddl/, e2e/docs/.
| Target | Secrets | Purpose |
|---|---|---|
+deps | No | Install uv + dependencies |
+build | No | Copy source files |
+test-unit | No | Run pytest |
+test-schema-generate | No | Generate types/DDL/docs |
+test-schema-validate | Yes | Validate vs ClickHouse |
+test-schema-e2e | Yes | Full workflow + artifacts |
+all | No | Run all non-secret targets |
Cause: Docker/Colima not running
Fix:
colima start
# Wait for "done" message, then retry
earthly +test-schema-generate
Cause: Wrong flag name or malformed secrets file
Fix: The correct flag is --secret-file-path (NOT --secret-file). The wrapper script handles this, but if running manually:
# WRONG
earthly --secret-file=/path/to/secrets +target
# CORRECT
earthly --secret-file-path=/path/to/secrets +target
Also ensure secrets file has no quotes around values:
# WRONG format
CLICKHOUSE_HOST="host.cloud"
# CORRECT format
CLICKHOUSE_HOST=host.cloud
Cause: hatchling build backend requires README.md in container
Fix: Ensure Earthfile copies README.md in deps target:
deps:
COPY pyproject.toml uv.lock README.md ./ # README.md required!
Cause: Doppler not configured or secrets not passed
Fix:
# Verify Doppler has the secrets
doppler secrets --project gapless-network-data --config prd | grep CLICKHOUSE
# Use the wrapper script (handles secret injection)
./scripts/earthly-with-doppler.sh +test-schema-validate
Force rebuild without cache:
earthly --no-cache +test-schema-e2e
The wrapper script scripts/earthly-with-doppler.sh:
CLICKHOUSE_* variablesKEY="value", Earthly needs KEY=value)--secret-file-path flag| Secret | Purpose |
|---|---|
CLICKHOUSE_HOST_READONLY | ClickHouse Cloud host |
CLICKHOUSE_USER_READONLY | Read-only user |
CLICKHOUSE_PASSWORD_READONLY | Read-only password |
| File | Purpose |
|---|---|
/Users/terryli/eon/gapless-network-data/Earthfile | Main build file |
/Users/terryli/eon/gapless-network-data/scripts/earthly-with-doppler.sh | Secret injection wrapper |
/Users/terryli/eon/gapless-network-data/schema/clickhouse/ethereum_mainnet.yaml | SSoT schema |
/Users/terryli/eon/gapless-network-data/docs/adr/2025-12-03-earthly-schema-e2e.md | ADR |
--secret-file-path not --secret-file (Earthly v0.8.16)--format env outputs quotes, must strip with sed 's/"//g'This skill validates schemas but does not design them. For schema design guidance (ORDER BY, compression, partitioning), invoke quality-tools:clickhouse-architect first.
| Skill | Purpose |
|---|---|
quality-tools:clickhouse-architect | Schema design before validation |
devops-tools:clickhouse-cloud-management | Cloud credentials for E2E tests |
devops-tools:clickhouse-pydantic-config | Client configuration |