Skip to main content

​
Scenario Space & Coverage

When testing traditional software, we use code coverage to measure test quality. For AI, code coverage is often misleading because the complexity lives in the Input Space (the “Latent Space”), not just the code paths. Flightline uses the concept of Scenario Space to ensure your AI is tested systematically.

​
What is Scenario Space?

Scenario Space is a multi-dimensional map of all the possible inputs your AI might encounter. During discovery, Flightline identifies the “Dimensions of Variance” in your domain.

​
Example: Support Ticket Automation

For an AI handling customer support, the Scenario Space dimensions might include:
  • Urgency: Low, Medium, High.
  • Topic: Billing, Technical Issue, Feature Request.
  • Language: English, Spanish, French.
  • Complexity: Simple question vs. multi-step troubleshooting.

​
Coverage-Aware Generation

Instead of generating 100 random test cases (which might all be simple billing questions in English), Flightline’s generate command uses Coverage-Aware Targeting. We aim to cover your Scenario Space by ensuring we have test cases for meaningful combinations of dimensions. This ensures that no corner of the input space is left unexamined.

​
Why this mental model matters

Engineers should stop asking “How many tests do we have?” and start asking “How much of our Scenario Space have we covered?” By moving from random sampling to systematic coverage, you can be certain that your AI won’t fail when it encounters a rare combination of inputs in production, even if you’ve never seen that specific scenario before.

Generate Command

Learn how to create coverage-aware test scenarios.