Reporting

A TRX is a file on the agent. Reporting is the other direction: the run's results going out to somewhere people already look — a ReportPortal dashboard, a Teams channel, the Test Case work items in Azure DevOps — or somewhere only your team has.

Three integrations ship, and the interface behind them is the same one you would implement yourself.

Reference the package and register it in your startup; everything else is configuration:

public sealed class Startup : IPreceptStartup
{
    public void ConfigureServices(IServiceCollection services, PreceptSettings settings)
    {
        services.AddPreceptReportPortalReporting();
        services.AddPreceptTeamsReporting();
        services.AddPreceptAzureDevOpsReporting();
    }
}

{
  "Reporting": {
    "ReportPortal": { "Endpoint": "https://rp.example.com", "Project": "acme_regression" },
    "Teams": { "Variant": "Pipeline", "NotifyOn": "Failure" },
    "AzureDevOps": { "OrganizationUrl": "https://dev.azure.com/acme", "Project": "Web", "TestPlanId": 412 }
  }
}

Endpoints belong in precept.json; keys and webhook URLs do not. They are credentials, and the environment-variable override is what keeps them out of source control — PRECEPT_REPORTING__REPORTPORTAL__APIKEY, PRECEPT_REPORTING__TEAMS__WEBHOOKURL, PRECEPT_REPORTING__AZUREDEVOPS__PERSONALACCESSTOKEN.

Each package's page documents its own settings: ReportPortal, Teams, Azure DevOps. What a pipeline has to pass them, and where the credentials come from, is Running a suite on CI.

What gets attached, and what does not#

ReportPortal and Azure DevOps upload the files a test captured — screenshots, Playwright traces, request logs — as attachments of its item or result. Two settings decide which files go, and both live in the reporter's own section:

The cap is what turns a failure that has a screenshot and a trace on disk into a dashboard item with only the screenshot on it. A Playwright trace of a scenario with more than a handful of steps is routinely over 8 MB, so this is the case to expect rather than the exception. The TRX has no such cap: the same trace is still copied next to the .trx and reaches the pipeline's published attachments, which is the quickest way to tell a skipped upload from a trace that was never recorded.

Raise the cap where traces are worth opening from the dashboard:

{ "Reporting": { "ReportPortal": { "MaxAttachmentBytes": 52428800 } } }

or per agent as PRECEPT_REPORTING__REPORTPORTAL__MAXATTACHMENTBYTES; the Azure DevOps reporter has the same setting under its own section. The server has a limit of its own past that — a ReportPortal deployment's upload size is set on its reverse proxy, and Azure DevOps refuses an attachment over 100 MB — and a file that clears Precept's cap but not the server's fails with the server's error, which the reporter does report. Details of where files are written and how they reach the TRX are on Artifacts and logging.

What the Teams card looks like#

A channel is where a run is noticed. It is not where it is diagnosed — a stack trace pasted into one buries the next suite's card and is unreadable where it lands. So the card answers the questions a channel actually asks, and links to the place that answers the rest:

┌──────────────────────────────────────────────┐
│ ❌  Checkout regression                       │   ← tinted green or red, edge to edge
│     4 of 133 failed · 7 m 24 s                │
├──────────────────────────────────────────────┤
│    126        4         3         2           │
│  Passed    Failed   Skipped    Flaky          │
│                                               │
│  Pass rate                  97% · 126 of 130  │
│  ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓░░░░  │
│                                               │
│  Environment   staging                        │
│  [ Open build 20260814.3 ]                    │
└──────────────────────────────────────────────┘

Four layouts ship, and Variant picks one — in code, in precept.json, or as PRECEPT_REPORTING__TEAMS__VARIANT for a single run:

services.AddPreceptTeamsReporting(teams => teams.Variant = PreceptTeamsCardVariant.Pipeline);

The build the card points at is the agent's, not a setting. BUILD_BUILDID and the collection and project variables address an Azure DevOps build; GITHUB_RUN_ID and the repository address a workflow run. Both are things the agent already knows and a test project cannot, so RunUrl is only for pointing somewhere else — a dashboard of your own — and the Pipeline variant's facts come from the same place. Off an agent there is no build to describe, and that variant renders the summary it is built on rather than a table of blanks.

Flaky and inconclusive tiles appear only when the run had any, a count of zero is stated in grey rather than shouted in red, and the pass rate leaves skipped tests out of both sides — a suite that skips half its scenarios against an environment is not 50% broken.

An image is optional and yours: a pipeline's status badge, a team logo, a shield for the environment.

{
  "Reporting": {
    "Teams": {
      "ImageUrl": "https://dev.azure.com/acme/Web/_apis/build/status/nightly",
      "ImagePlacement": "Banner"
    }
  }
}

ImagePlacement takes Thumbnail (the default, beside the heading), Banner (full width, under it) or None — which is how an overlay drops the image for one environment without deleting the URL from the shared file. The address has to be one every reader's Teams client can fetch without signing in: the card is rendered in their client, not on the agent, so a link into a private artifact store renders as a broken image for the whole channel.

Linking a run's reports to each other#

A run that reports to ReportPortal and to Teams should say so on the card:

[ Open Azure DevOps ]  [ Open ReportPortal ]

Those buttons need no configuration, and could not have any: a launch id is assigned by ReportPortal when the launch is opened, and a test run id by Azure DevOps when the run is created. So each reporter posts what it opened to PreceptRunLinks — a noticeboard registered for every run — and the Teams card renders whatever is on it. Register the reporters and the buttons appear; "LinkToOtherReports": false turns them off.

One button per service, not one per URL. The agent's build and the test run filed against it are two ways into the same Azure DevOps, so the detected build gives way when a reporter has posted a link into the same host — the test run holds the results, the messages and the screenshots this card leaves out, and its build is one click further on. Without the Azure DevOps reporter registered there is nothing to give way to, and the card links the build as before. A RunUrl you configured by hand is never the one dropped: you asked for it.

It is a noticeboard rather than a protocol, and the distinction is visible in one place. Reporters run on pumps of their own, so nothing orders ReportPortal opening its launch against Teams building its card; each destination posts its address as soon as it has one — at the start of the run, not the end — and the card renders whatever is there when it is written. A run long enough to be worth a card has always had both. The alternative was a reporter that waits on another reporter's web service before the first test runs, and that is a worse trade than a button that is very occasionally one short.

A reporter of your own can post to the same board, and is then linked from the card exactly like the shipped ones — see writing a reporter.

CI and local runs#

The three shipped reporters only report from a build agent. A launch, a test run and a channel message all belong to a pipeline; a developer's machine has neither the credentials for them nor a reason to add its runs to the project's statistics. So the registration above is the whole story on CI, and on a laptop the same suite reports nothing and asks for nothing — the reporters are never started, which is why a missing API key is not an error there either. One line says what happened, and it is not a warning:

[Precept] ReportPortal, Teams, AzureDevOps reporting is CI-only and this run is not on a build agent, so nothing is sent.

A run counts as CI when the agent's own variable is set — TF_BUILD, GITHUB_ACTIONS, GITLAB_CI, JENKINS_URL, TEAMCITY_VERSION, BUILDKITE and the rest, or a CI that does not say false. Nothing to configure on the pipeline: the agent has already said so.

To report from your own machine — checking a card's layout, or filling a dashboard while writing the integration — one key in precept.local.json brings them all in:

{ "Reporting": { "CiOnly": false } }

PRECEPT_REPORTING__CIONLY=false does the same for a single run, and Reporting:<Reporter>:CiOnly for a single reporter. From there the settings are ordinary settings again: a missing key is an error, because you have just asked this machine to report.

The detection itself is a setting as well, at the top level rather than under Reporting: PRECEPT_ISCONTINUOUSINTEGRATION=true makes a local run behave as a CI one throughout, and "IsContinuousIntegration": false in an overlay takes an agent back out of it.

This is a different axis from the environment, which says what is under test. A pipeline running against staging and a developer running against staging want the same precept.staging.json and different reporting, which is exactly what the two separate switches give.

A reporter you write yourself is not CI-only unless it says so — a CSV file or a local dashboard keeps working everywhere. Return true from CiOnly, or bind a setting to it, when it needs a credential. Reporting:CiOnly overrides every reporter either way, in both directions.

Each reporter also keeps an Enabled flag of its own, for switching one off outright:

{ "Reporting": { "ReportPortal": { "Enabled": false } } }

Linking scenarios to Azure DevOps test cases#

The mapping lives beside the scenario rather than in a spreadsheet, and a scenario can carry it either in its title or in a tag:

Scenario: [41207] A customer checks out with a stored card

@smoke @tc:41207
Scenario: A customer checks out with a stored card

Either way that result is filed against Test Case 41207, which is what makes the work item show its own latest automated outcome and a test plan report coverage from the run. A scenario covering two test cases names both — [41207][41208], or two tags — and is reported against each; naming one work item in both places still files one result.

The title is read with TestCaseTitlePattern, a regular expression with a group named id, so the convention is yours:

{
  "Reporting": {
    "AzureDevOps": {
      "TestCaseTitlePattern": "^\\s*(?<id>\\d+)\\s*[:-]\\s*",
      "TestCaseTagPrefix": ""
    }
  }
}

The default, ^\s*(?:\[(?<id>\d+)\]\s*)+, is anchored at the start of the title deliberately: an unanchored pattern also finds ids nobody wrote — Scenario: The cart shows [3] items — and Azure DevOps rejects a whole batch of results when one of them names a work item that is not a Test Case. Set either setting to "" to switch that source off; a test that names no work item at all is still part of the run unless OnlyMappedTests says otherwise.

Who a run belongs to#

Azure DevOps shows a run's Run by as Unassigned unless it is told who ran it — it does not take that from the token that posted the results. Precept asks the organization who the token is and files the run and every result under that identity: your own account from a laptop, the build service from a pipeline. It is one call at the start of the run, and a token not allowed to make it leaves the field empty rather than failing the run.

RunBy attributes a run to somebody else — an identity GUID, or a unique name for Azure DevOps to resolve:

{ "Reporting": { "AzureDevOps": { "RunBy": "qa@acme.com" } } }

This is unrelated to BuildId, which associates the run with a build so it appears on that build's Tests tab and nothing more.

Running against a test plan#

TestPlanId turns the run into a planned one, and Azure DevOps then accepts a result only when it names the test point it fills — a test case in a suite, under a configuration — rather than just the Test Case work item. Precept resolves those points from the ids the scenarios already carry, so the mapping stays in the feature files:

{
  "Reporting": {
    "AzureDevOps": { "TestPlanId": 412, "TestSuiteId": 413 }
  }
}

TestSuiteId is optional and narrows the run. Without it, a test case the plan holds in two suites — or under two configurations — is two planned slots, and the scenario covering it is reported into each; naming a suite files only that suite's point. Reading the plan's points is a test API, so Test Management (read & write) is still the whole requirement.

A planned result also has to carry the test case's revision, and that is a work item read — a scope a token filing test results does not otherwise need. Precept asks for it, takes a refusal as the answer, and files revision 1 rather than failing the run; Azure DevOps records the number it is given without checking it. Add Work Items (read) to the token for the real one. Either way the run says which it did.

What a planned run cannot hold is a result that fills no slot: a test naming no work item, or naming one that is not in the plan, is left out of it. That is not silent — the count and the reason are written into the run's comment when it completes:

Precept run 3f2a…, 240 tests against 'staging'. 12 test(s) named no Test Case work item and were
left out of this planned run. 3 result(s) named a Test Case with no test point in plan 412.

Leave TestPlanId unset if that is not what you want. The link to the work item comes from the result's test case, not from the plan, so an unplanned run still updates each Test Case's latest automated outcome — and it keeps the unmapped tests.

Writing a reporter#

IPreceptReporter has four events and a default implementation for each, so a reporter overrides only what it needs:

public sealed class SlowTestReporter : IPreceptReporter
{
    private readonly List<PreceptTestReport> _slow = [];

    public string Name => "SlowTests";

    public Task OnTestFinishedAsync(PreceptTestReport test, CancellationToken cancellationToken)
    {
        if (test.Duration > TimeSpan.FromSeconds(30))
        {
            _slow.Add(test);
        }

        return Task.CompletedTask;
    }

    public async Task OnRunFinishedAsync(PreceptRunSummary run, CancellationToken cancellationToken) =>
        await File.WriteAllLinesAsync("slow-tests.txt", _slow.Select(t => $"{t.Duration:mm\\:ss} {t.DisplayName}"), cancellationToken);
}

services.AddPreceptReporter<SlowTestReporter>();

A reporter is resolved from the container, so it can take dependencies — including settings of its own, bound from a subsection of Reporting by the overload that takes them:

services.AddPreceptReporter<SlowTestReporter, SlowTestSettings>();   // binds Reporting:SlowTest

PreceptTestReport carries the verdict, the timings, the attempt count, the categories, the lines the test logged, the files it captured and the exception itself. It names the test twice over: ClassName and MethodName are the code behind it, while SuiteName and DisplayName are what it is called — the Gherkin feature and scenario titles for a generated test, so a dashboard groups by "Checkout" rather than by Acme.Tests.CheckoutFeature. SuiteName falls back to ClassName for a class whose [TestSuite] was given no title, and SuiteDescription carries the feature narrative — the lines under the Feature: heading, dedented so a destination that renders markdown does not read them as a code block — or null when there is none. PreceptRunSummary carries the counts, the wall-clock duration, and the failures — up to MaxTrackedFailures of them — because accumulating those is otherwise the first thing every summary notifier has to write.

A reporter that opens something with an address of its own — a launch, a test run, a dashboard page — can post it to PreceptRunLinks, which is registered for every run and injectable like any other dependency. That is how the Teams card carries a button to the ReportPortal launch without either package knowing about the other; see linking a run's reports to each other.

public sealed class DashboardReporter(PreceptRunLinks links) : IPreceptReporter
{
    public string Name => "Dashboard";

    public async Task OnRunStartingAsync(PreceptRunStarting run, CancellationToken cancellationToken) =>
        links.Add("Dashboard", await OpenPageAsync(run, cancellationToken));
}

Return IsEnabled => false when a reporter has nothing to do; it is then never given a queue at all. Return CiOnly => true when it belongs to a pipeline rather than to a machine — see CI and local runs — and it is left out of a developer's run before it is ever started. Throw from OnRunStartingAsync when it is switched on but has no endpoint or credential to talk to: that is the moment somebody is still watching the console, and by then the run has already established that this reporter was meant to report from here.

What reporting costs a run#

Nothing on the test's thread, by construction.

Each reporter gets a bounded queue and a pump task of its own. Publishing a result is one small record and a non-blocking write per reporter, so a destination that takes two seconds to answer, or has stopped answering altogether, delays no test and no other reporter. A reporter is called one event at a time and in order, which is also why it needs no locking of its own.

The consequences of never blocking are worth knowing about:

• A queue that fills drops events. Backpressure here would mean a test thread waiting on somebody's web service. Drops are counted and reported at the end of the run, so a short report says it is short instead of looking complete. Raise QueueCapacity, or batch inside the reporter, as Precept.Reporting.AzureDevOps does.
• A reporter that throws is contained. The first failure is a warning; after MaxFailures of them the reporter is dropped for the rest of the run, so an endpoint that is down costs one message rather than one per test.
• The run waits at the end, but not forever. After the last test the queues are drained under ShutdownTimeoutSeconds, shared across all reporters. Whatever has not gone out by then is abandoned and said to be abandoned.

A run with no reporters registered, or with Reporting:Enabled set to false, constructs nothing and starts no queue — and neither does a discovery request or --list-tests, so listing the tests in an IDE never opens a launch or posts to a channel.