Skip to main content

​
CLI Overview

Flightline is a headless-first platform. It lives in your terminal, configuration files, and CI pipelines, providing the tools needed to test and validate AI systems without complex dashboards.

​
Core Workflow

The Flightline CLI is organized around a 5-step workflow designed to move AI features from discovery to a verified ship-ready state.

​
1. Discover

flightline discover
Scans your codebase to find where LLM calls happen. It identifies the models being used, extracts prompt snippets, and assigns risk tiers based on data sinks.

​
2. Generate

flightline generate --from-discover flightline.discovery.json --count <n>
Uses discovery context to manufacture high-fidelity synthetic data directly from your code and prompts. This ensures you have realistic test scenarios for every edge case.

​
3. Trace

fltrace <your-test-command>
A wrapper that instruments your application code. It records every LLM call, capturing inputs, outputs, latency, and token counts for analysis.

​
4. Evaluate

flightline eval scenarios
Runs a comparison between expected outcomes (from your generated scenarios) and actual AI outputs (from your traces) to identify logic failures.

​
5. Check

flightline check --traces
The final ship-readiness gate. It runs deterministic and reasoning-based checks against your traces to answer the 7 ship-blocking questions.

​
Advanced Refinement

​
Learn

flightline learn <path/to/sample.json>
If code-based generation is not specific enough for your domain, use learn to analyze a sample dataset. This creates a metadata profile that can be used to enhance the realism of generated scenarios.

​
Shorthand and Aliases

Many commands have shorter aliases for faster typing:
CommandAlias
generategen
discoverscan

​
Common Options

Most commands support these shared flags:
  • --model, -m: Choose a specific model via OpenRouter (e.g., google/gemini-3-flash-preview).
  • --verbose, -v: Display detailed logs and execution steps.
  • --output, -o: Specify a custom path for results.

​
Configuration

Flightline uses a flightline.yaml file at the root of your repository to manage project-wide settings and custom evaluation rubrics.

Configuration Reference

Learn how to customize Flightline for your project.