DocsBook a demo

Platform

Platform overviewArchitectureWorkflow orchestrationGitOps configurationGovernance and AAAAI and MCPKnowledge and contextRuntime and executionEvidence and monitoring

Use cases

All use casesProduction incidentRelease preparationHotfix to productionSecurity scan triage
Why NopsAIIntegrationsSecurity

Resources

All resourcesAI agent governanceMCP governanceMCP securitySelf-hosted platforms
PricingGitHub

Company

How a run worksAboutContactBook a demo

Analysis reviewers and AI Evaluation

Read-only analysis of pipelines, runs, and resources, with an optional AI-reviewed health score.

How-toOperatorAutomation author

Key points

  • The health score starts at 100 and subtracts weighted visible findings: critical ×25, high ×15, medium ×8, low ×3, opportunity ×1.
  • Category scores use the same weights filtered to that category.
  • A structured AI Evaluation can replace the displayed number with an AI-reviewed score.
  • Analysis is read-only. Analyse Run never reruns a failed step; it opens existing logs and resources or copies the diagnosis until a user chooses a separate confirmed action.
  • AI Evaluation uses the authenticated POST /v1/analysis/evaluate endpoint with a usable model, a redacted reviewer snapshot, and bounded page-owned evidence.
  • It does not create an Assistant conversation and does not ask hosted MCP to inspect the pipeline, team, resource, or run.

Before you start

Feature enabled
AI evaluation enabled for the install; the route refuses otherwise
Model access
A model the caller is allowed to use in the target scope
A pipeline to analyse
A saved pipeline, opened at its Health tab

Steps

  1. 01

    Open the Health tab

    Pipeline detail owns the analysis result on Health. The Analyse Pipeline header action switches there rather than opening a separate modal.

    Verify
    • The tab shows the current analysis for the pipeline.
  2. 02

    Run an evaluation

    Evaluation is a request against the analysis surface, not a pipeline run: it does not consume runner capacity.

    Request an evaluationbash
    curl -sX POST "$NOPSAI_URL/v1/analysis/evaluate" \
  -H "Authorization: Bearer $NOPSAI_TOKEN" \
  -H "Content-Type: application/json" \
  --data @evaluation.json | jq
    Replace before running
    • evaluation.json carries the analysis payload the Health tab submits.
    Expected result
    • A structured evaluation the Health tab renders alongside the analysis.
  3. 03

    Read the result as advice, not a gate

    An evaluation is a reviewer opinion. Enforcement comes from governance level, guardrail knowledge, and approvals, which are separate mechanisms.

How it works

Structured AI reviews are cached in browser-local review history by subject type, subject ID, and snapshot revision. Reopening the same analysis shows the reviewed score immediately.

When only an older same-subject review exists, the modal labels it as a previous-snapshot score and asks you to regenerate for current evidence.

Deterministic analysis still works when Assistant debugging or model access is unavailable — you lose the AI-reviewed score, not the findings.

AI Evaluation selects the configured default from the unscoped Assistant profile picker and still sends the team and resource scope path to the evaluation endpoint for policy validation.

Examples

Why the score movedtext
Base                      100
2 critical findings       -50   (2 x 25)
1 high finding            -15   (1 x 15)
3 medium findings         -24   (3 x 8)
                          ----
Displayed health score      11

Implementation evidence

  • services/nopsai/analysis_evaluation_handlers.go

    Evaluation endpoint, snapshot redaction, and policy validation.

  • doc/feature-reference.md

    Analysis reviewer behavior and scoring.