Skip to main content

​
Observability

Flightline’s observability layer is built on the principle that to test AI effectively, you must see exactly what it is doing at runtime. We use a non-invasive instrumentation approach to capture the full context of every LLM interaction.

​
The fltrace Mechanism

Instead of requiring you to add complex logging code or decorators to every function, Flightline uses a CLI wrapper called fltrace. When you run a command through fltrace, it initializes our instrumentation layer before your application starts. This captures LLM calls during execution with minimal code changes.

​
What gets captured?

Every captured “Trace” is a complete record of a single LLM interaction, organized into three primary buckets:
  1. Request and Response Metadata: The configuration parameters and raw completions from the model.
  2. Performance and Cost Signals: Detailed timing data and exact token usage counts for every call.
  3. Payload Capture: Optional full-content recording based on your privacy settings and testing mode.

​
Synthetic vs. Production Mode

Flightline handles data differently based on your privacy posture.

​
Synthetic Mode (Default)

In this mode, Flightline assumes you are testing against synthetic or non-sensitive data. It records full payloads, which are necessary for deep reasoning-based evaluations.

​
Production Mode

When enabled via the --prod flag, Flightline activates strict PII handling. It can redact sensitive information or only record metadata and structural patterns, ensuring that customer data never leaves your secure environment.

​
The Trace Bundle

Traces are collected into a “Trace Bundle,” which represents a complete execution run (such as a test suite run or a specific user session). This bundle is the primary input for the flightline check and flightline eval commands.

​
Why this mental model matters

Engineers should think of fltrace as a “flight recorder” for their AI. You don’t need to worry about what to log; you simply fly your mission, and Flightline records the data needed to prove the mission was successful.

Intelligence Layer

See how Flightline uses these traces to analyze system performance.