Skip to main content

​
fltrace

fltrace is a CLI wrapper that instruments your application code to capture LLM activity. It works similarly to observability tools like ddtrace-run, but is specifically optimized for capturing the full context of AI interactions.

​
Usage

fltrace <your-command>
Example: 
fltrace python app.py
fltrace npm test
fltrace ./my_custom_script.sh

​
Language Support

Flightline provides auto-instrumentation for both Python and Node.js applications.

​
Setup

# No additional setup required!
# Python auto-instrumentation is built into the CLI
pip install flightline-ai

​
Instrumentation

# No code changes required for Python!
# fltrace automatically hooks into OpenAI, Anthropic, and other SDKs
from openai import OpenAI

client = OpenAI()
response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Hello!"}]
)

​
Running with fltrace

# Run your Python application with tracing
flightline fltrace python app.py

# Or run your tests
flightline fltrace pytest tests/
You can also enable tracing by setting the FLIGHTLINE_RECORD=1 environment variable.

​
How it Works

When instrumentation is enabled, Flightline automatically records:
  • Request and Response Metadata: The configuration parameters and raw completions from the model.
  • Performance and Cost Signals: Detailed timing data and exact token usage counts for every call.
  • Payload Capture: Optional full-content recording based on your privacy settings and testing mode.

​
Key Options

OptionDescription
--traces, -tDirectory to save captured traces (default: .flightline/traces).
--prod, -pProduction mode. Enables stricter PII handling.
--quiet, -qSuppress fltrace status messages, showing only your command’s output.

​
Why Trace?

Static evaluation is limited. By capturing real execution traces, Flightline can:
  1. Analyze Behavior: Verify if the AI actually followed instructions in a real-world context.
  2. Detect Regressions: Compare traces from a new PR against a “Golden Set” of traces from main.
  3. Identify Latency Bottlenecks: See which specific AI calls are slowing down your users.
  4. Audit for Safety: Check for PII leakage or unsafe outputs in production-like environments.

​
Example Output

$ fltrace python app.py

🛫 fltrace: Recording to .flightline/traces (synthetic mode)
   Running: python app.py

... your application output ...

🛬 fltrace: Saved 4 trace file(s)
   Check with: flightline check --traces .flightline/traces/

​
Next Steps

Captured traces are the raw evidence used by Flightline’s intelligence layer to grade your system.

Run Ship-Readiness Check

Evaluate your captured traces for ship-readiness.