Monitor the evidence plane
Monitor ingestion delay, source availability, dropped records, schema failures and clock skew. A functioning agent with incomplete evidence can produce a worse result than a visible service outage. Show the coverage interval in each investigation and expose the missing sources before an operator relies on the answer.
Set separate service expectations for collection, investigation and enforcement. The auditor’s queue can grow without changing the policy already protecting a connection. Document what happens when a source is late or an investigation times out: retain the partial evidence, mark the limitation, and avoid silently turning an incomplete result into a definitive finding.
Review findings and actions
- Can each factual statement be traced to an event or a bounded query?
- Does the entity join rely on identity evidence or merely a shared address?
- Is an inferred intention presented as an inference?
- Did the model inspect only the allowed time window and tenant?
- Does the proposed action have an owner, expiry and rollback?
Use a regular review sample that includes both accepted and rejected recommendations. Track why operators disagree: missing context, overbroad suggested action, wrong identity join or unsupported narrative. Those categories lead to different improvements in source mapping, policy, retrieval and reasoning.
Regression tests for the auditor
Maintain replayable evidence bundles with synthetic identifiers and no live secrets. Include benign high-volume traffic, misconfigured certificates, interrupted uploads, compromised identities and adversarial text embedded in content. The expected result should include when the auditor must abstain.
Finding correctness: supported by observed evidence
Citation integrity: referenced events exist and match the claim
Boundary integrity: tenant, role, source and time constraints hold
Uncertainty: missing evidence changes the conclusion
Action safety: recommendations obey approval and rollback policyA model upgrade is an operational change. Compare the same evidence pack across versions, record regressions and keep the prior version available until acceptance criteria pass. Evaluate the complete tool-using workflow, not only the final language output.