> For the complete documentation index, see [llms.txt](https://layerlens.gitbook.io/stratix-python-sdk/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://layerlens.gitbook.io/stratix-python-sdk/cli/examples.md). # Workflow Examples Fifteen copy-paste workflows covering the most common CLI tasks. *** ## 1. Quick start: first trace to evaluation Set up, list traces, inspect one, and evaluate it with a judge. ```bash # Configure export LAYERLENS_STRATIX_API_KEY="sk-..." # See what traces are available stratix trace list # Inspect a specific trace stratix trace get # Create a judge and test it against the trace stratix judge create --name "Accuracy" --goal "Rate factual accuracy of the response" --model-id stratix judge test --judge-id --trace-id ``` *** ## 2. Run an evaluation and wait for results ```bash # Run and block until done stratix evaluate run \ --model openai/gpt-4o \ --benchmark arc-agi-2 \ --wait # Or fire and check later stratix evaluate run --model openai/gpt-4o --benchmark arc-agi-2 stratix evaluate list --status in-progress stratix evaluate get ``` *** ## 3. Compare models on the same benchmark ```bash stratix evaluate run --model openai/gpt-4o --benchmark arc-agi-2 --wait stratix evaluate run --model anthropic/claude-3-opus --benchmark arc-agi-2 --wait # List results sorted by accuracy stratix evaluate list --sort-by accuracy --order desc ``` *** ## 4. Judge workflow: create, test, tune, apply ```bash # Create a judge stratix judge create \ --name "Helpfulness" \ --goal "Rate how helpful and actionable the response is on a 1-5 scale" \ --model-id # Test against a sample trace stratix judge test --judge-id --trace-id # Review the result stratix --format json judge get # Iterate: create a refined version stratix judge create \ --name "Helpfulness v2" \ --goal "Rate helpfulness on a 1-5 scale with justification" \ --model-id ``` *** ## 5. Trace search and export ```bash # Search for traces matching a keyword stratix trace search "customer support" --page-size 5 # Export a trace to a file stratix trace export -o trace_backup.json # Export as JSON for piping stratix --format json trace get | jq '.id' ``` *** ## 6. Bulk evaluation from a JSONL file ```bash # Create a jobs file cat > jobs.jsonl <<'EOF' {"model": "openai/gpt-4o", "benchmark": "arc-agi-2"} {"model": "openai/gpt-4o-mini", "benchmark": "arc-agi-2"} {"model": "anthropic/claude-3-opus", "benchmark": "arc-agi-2"} EOF # Dry-run to preview stratix bulk eval --file jobs.jsonl --dry-run # Execute and wait stratix bulk eval --file jobs.jsonl --wait ``` *** ## 7. Bulk trace evaluation with a judge ```bash # Create a trace ID file stratix --format json trace list | jq -r '.[].id' > trace_ids.txt # Dry-run to preview stratix bulk eval \ --judge-id \ --traces trace_ids.txt \ --dry-run # Run trace evaluations for all traces stratix bulk eval \ --judge-id \ --traces trace_ids.txt ``` *** ## 8. CI/CD pipeline integration ```bash # In your GitHub Actions workflow: stratix evaluate run \ --model openai/gpt-4o \ --benchmark arc-agi-2 \ --wait # Generate a summary for the GitHub job stratix ci report -o "$GITHUB_STEP_SUMMARY" # Or output JSON for custom processing stratix ci report -o report.json ``` *** ## 9. Integration monitoring ```bash # List all integrations stratix integration list # Test a specific integration stratix integration test # JSON output for scripting stratix --format json integration list | jq '.[] | select(.status != "active")' ``` *** ## 10. Scorer management ```bash # List existing scorers stratix scorer list # Create a scorer with dry-run stratix scorer create \ --name "Code Quality" \ --description "Evaluates generated code for correctness, readability, and best practices" \ --model-id \ --prompt "Score the following code on a 1-10 scale for quality..." \ --dry-run # Create for real stratix scorer create \ --name "Code Quality" \ --description "Evaluates generated code for correctness, readability, and best practices" \ --model-id \ --prompt "Score the following code on a 1-10 scale for quality..." # Delete with confirmation stratix scorer delete # Delete without prompt stratix scorer delete -y ``` *** ## 11. Evaluation spaces ```bash # List spaces stratix space list # Create a private space stratix space create \ --name "Q1 Model Comparison" \ --description "Comparing GPT-4o vs Claude 3 Opus for Q1 release" \ --visibility private # Create a public space (dry-run first) stratix space create \ --name "Public Leaderboard" \ --visibility public \ --dry-run # Get space details by slug or ID stratix space get q1-model-comparison # Clean up stratix space delete -y ``` *** ## 12. JSON output and scripting ```bash # Pipe trace IDs into a loop stratix --format json trace list | jq -r '.[].id' | while read id; do echo "Exporting $id..." stratix trace export "$id" -o "traces/${id}.json" done # Get evaluation accuracy as a number ACCURACY=$(stratix --format json evaluate get | jq -r '.accuracy') echo "Accuracy: $ACCURACY" # Filter evaluations by status stratix --format json evaluate list | jq '[.[] | select(.status == "success")]' ``` *** ## 13. Pagination and sorting ```bash # Page through traces stratix trace list --page 1 --page-size 20 stratix trace list --page 2 --page-size 20 # Sort evaluations stratix evaluate list --sort-by accuracy --order desc --page-size 5 stratix evaluate list --sort-by submitted_at --order asc # Sort spaces stratix space list --sort-by created_at --order desc ``` *** ## 14. Verbose mode and debugging ```bash # Enable verbose output to see HTTP requests stratix -v trace list # Combine with JSON output stratix -v --format json evaluate get # Debug authentication issues stratix -v integration list ``` *** ## 15. Clean up resources ```bash # Delete a trace (with confirmation prompt) stratix trace delete # Delete without prompting stratix trace delete -y # Delete a scorer (dry-run first) stratix scorer delete --dry-run stratix scorer delete -y # Delete a space stratix space delete -y ``` --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://layerlens.gitbook.io/stratix-python-sdk/cli/examples.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.