experiment

Run Experiments

Run a computational experiment. Given a research question that can be answered via software, this skill will write and run the necessary software and generate a report on the results.

This skill can also be used to analyze experimental data and generate a research report from it, even if the experiment itself was not run by Asta. In that case, the user can provide the experimental data as a file input, and Asta will analyze it and generate a report.

Environment Variables

asta experiment (Panda) reads its model choice and credentials from the environment. These are not exposed as CLI flags, so set them before invoking the skill if you want to override the defaults.

Model selection:

• PANDA_LLM — selects the underlying LLM (e.g., a Claude or GPT model name). If unset, the default defined in panda_agent/config.py is used.

Provider credentials (set the ones matching the model you select via PANDA_LLM):

• OPENAI_API_KEY — required for GPT models; also required for Panda's default configuration.
• ANTHROPIC_API_KEY — required to use Claude models.
• TOGETHER_API_KEY — required to use Mistral or Llama models.
• INFERD_TOKEN — required to use OLMo models.

Example — run the experiment with Claude:

export PANDA_LLM="claude-3-5-sonnet-latest"
export ANTHROPIC_API_KEY="..."
asta experiment --task "..." --outputs_dir "$OUTPUTS_DIR" --result_file "$OUTPUTS_DIR/result.json"

For the authoritative list of supported models and any additional variables, see asta experiment --help and the Panda README.

Workflow

The user will either:

1. describe the research task (TASK) and (optionally) background knowledge (BACKGROUND_KNOWLEDGE) or
2. provide paths to files that contain a. the task (TASK_FILE) b. (optionally) the background knowledge (BACKGROUND_KNOWLEDGE_FILE)

Default Output Locations

IMPORTANT: Always specify output locations to keep experiments organized in .asta/experiment/:

• OUTPUTS_DIR: .asta/experiment/<YYYY-MM-DD-slug>/ where:
  • YYYY-MM-DD is the current date
  • slug is a short descriptive name derived from the task (e.g., "french-translation", "gpt4-eval")
• RESULT_FILE: <OUTPUTS_DIR>/result.json

Example directory structure:

.asta/experiment/
├── 2024-01-15-french-translation/
│   ├── result.json
│   └── [experiment outputs]
└── 2024-01-16-model-comparison/
    ├── result.json
    └── [experiment outputs]

The user may optionally override these locations by providing: 3. a custom directory for experimental outputs (OUTPUTS_DIR) 4. a custom path for the result summary JSON (RESULT_FILE)

Task and Background Knowledge

If the user describes a task and (optionally) provides background knowledge, then run as follows:

With default output locations (recommended):

# Create output directory with date and slug
OUTPUTS_DIR=".asta/experiment/$(date +%Y-%m-%d)-<task-slug>"
mkdir -p "$OUTPUTS_DIR"

asta experiment \
  --task "TASK" \
  --background_knowledge "BACKGROUND_KNOWLEDGE" \
  --force_report \
  --outputs_dir "$OUTPUTS_DIR" \
  --result_file "$OUTPUTS_DIR/result.json"

Example:

# Task: Assess GPT-4 translation quality
OUTPUTS_DIR=".asta/experiment/$(date +%Y-%m-%d)-gpt4-french-translation"
mkdir -p "$OUTPUTS_DIR"

asta experiment \
  --task "Perform an experiment to assess how good gpt-4o is at translating into French. Use just 5 test examples." \
  --force_report \
  --outputs_dir "$OUTPUTS_DIR" \
  --result_file "$OUTPUTS_DIR/result.json"

With custom output locations (if user specifies):

asta experiment \
  --task "TASK" \
  --background_knowledge "BACKGROUND_KNOWLEDGE" \
  --force_report \
  --outputs_dir "/custom/path/experiments/" \
  --result_file "/custom/path/result.json"

Task and Background Knowledge Files

When the user provides files containing the task and background knowledge:

With default output locations (recommended):

# Create output directory with date and slug (derived from task filename or content)
OUTPUTS_DIR=".asta/experiment/$(date +%Y-%m-%d)-<task-slug>"
mkdir -p "$OUTPUTS_DIR"

asta experiment \
  --task_file "TASK_FILE" \
  --background_knowledge_file "BACKGROUND_KNOWLEDGE_FILE" \
  --force_report \
  --outputs_dir "$OUTPUTS_DIR" \
  --result_file "$OUTPUTS_DIR/result.json"

Example:

# Files: my_project/task.txt, my_project/background_knowledge.txt
OUTPUTS_DIR=".asta/experiment/$(date +%Y-%m-%d)-custom-experiment"
mkdir -p "$OUTPUTS_DIR"

asta experiment \
  --task_file "my_project/task.txt" \
  --background_knowledge_file "my_project/background_knowledge.txt" \
  --force_report \
  --outputs_dir "$OUTPUTS_DIR" \
  --result_file "$OUTPUTS_DIR/result.json"

With custom output locations (if user explicitly specifies paths):

asta experiment \
  --task_file "my_project/task.txt" \
  --background_knowledge_file "my_project/background_knowledge.txt" \
  --force_report \
  --outputs_dir "my_project/experiments/" \
  --result_file "my_project/result.json"

Notes

• Output Directory: Always create .asta/experiment/<YYYY-MM-DD-slug>/ directory before running the experiment using mkdir -p
• Task Slug: Create a short descriptive slug from the task (e.g., "gpt4-translation", "model-comparison"). Keep it lowercase with hyphens.
• Result File: Always save to <OUTPUTS_DIR>/result.json for consistency
• Background Knowledge: Optional - can be omitted if not provided by user
• Custom Paths: If user explicitly provides custom output paths, use those instead of defaults
• Always return the result to the user and inform them where outputs were saved