Sign in
Open console
Docs/Best practices/Scoping an assistant well
docs · best practices

Scoping an assistant well

concept

Start narrow, measure, widen. A field-tested approach to scope.

Scope is the single biggest lever on whether an assistant feels trustworthy or flaky. The reliable path is not to guess the perfect boundary up front — it is to start narrow, watch how real questions land, and widen only where the evidence says you can. This page explains how the Scope Gate decides what is in or out of scope, and how to tune that decision with signals you can actually measure.

The whole method in one line — Launch with one corpus and a tight gate, let real traffic accumulate, then loosen the boundary one step at a time against measured deflection accuracy and unanswered rate — never by feel.

How the Scope Gate actually decides

The Scope Gate runs before retrieval. Each incoming question is classified as in-scope or out-of-scope first; only in-scope questions go on to search the corpus and generate a cited answer. The classification happens in two stages: a deterministic Layer-1 pass applies your explicit rules (keyword and category matches that resolve without a model call), and anything those rules do not settle falls through to an LLM-classifier tail that makes the judgment call.

The gate is fail-open by default — When the gate is uncertain, the default behavior is to let the question through rather than refuse it. That keeps you from silently turning away legitimate questions, but it also means a too-loose gate will happily answer things it should have deflected. Tightening is your job, and you tighten with rules — not by hoping the classifier gets stricter on its own.

Start narrow

For a new assistant, point it at a single, well-curated corpus and write a deliberately tight set of Layer-1 rules — the topics you are certain it should answer, and explicit out-of-scope categories for the obvious adjacents. A narrow gate produces more deflections at launch, and that is the point: you would rather route a borderline question to a human now and learn from it than ship a confident answer the sources never supported.

Measure with real signals

Once questions are flowing, read the analytics rather than your intuition. Three signals drive the widen-or-tighten decision:

Deflection accuracycorrect OOS refusals / total OOS refusals
Of the questions the gate refused as out-of-scope, how many were genuinely out of scope. Low accuracy means the gate is refusing things it should answer — your rules are too tight.
Unanswered rate%
Share of in-scope questions that produced no answer. A climbing rate points at corpus gaps, not the gate — widening scope will not help if the sources are not there.
Escalationscount over time
Questions routed to a human. A spike after a rule change tells you the change moved the boundary the wrong way.
There is no "deflection rate" metric — VeriCite measures deflection accuracy — the correctness of refusals — not a raw "deflection rate." If a dashboard or report cites a "deflection rate," it is wrong; the shipped signal is accuracy. Anchor your tuning decisions on deflection accuracy, unanswered rate, and escalations only.

Then widen

With a few weeks of signal, look for the safe places to loosen: high deflection accuracy paired with a cluster of refused questions that you can see are legitimate is the signal to add an in-scope rule or attach another corpus. Make one change, then watch the same three signals. If escalations or unanswered rate climb, you widened past where the sources can support — roll the change back. The loop is narrow, measure, widen, measure again. It never ends; it just gets slower as the boundary settles.

What you can tune — and what you cannot

Tuning scope means editing the Layer-1 rules and the classifier configuration, then measuring. Two controls look like tuning knobs but are not, and you should not build a process around them:

  • Minimum confidence on the tenant Scope Gate is a reserved no-op. The field exists in the schema, but the classifier does not enforce it in the current build — setting it changes nothing.
  • Disable now, the gate kill switch, is superadmin-only. A Tenant Administrator cannot turn the gate off from the workspace, so do not plan to disable it as a tuning step.
Two config paths, two different configs — Scope-gate configuration lives in two places: the tenant Scope Gate settings page and the per-assistant scope panel. They write to different configuration, so a change on one does not propagate to the other. Decide which level you are tuning and edit there consistently — mixing the two is the usual cause of "I changed it but nothing happened."
featureScope Gate0.0–1.0; reserved knob — sets the classifier confidence floor.
featureAnalyticsAnswer rate, escalation rate, and citation coverage over time.
conceptEscalation by defaultWhen sources are silent, VeriCite escalates to a human instead of guessing.
featureFallback policyWhat the assistant does when no approved source supports an answer.
journeyQuickstartStand up your first assistant — corpus, scope, channels — in about ten minutes.
Was this page helpful?