Reporting
A TRX is a file on the agent. Reporting is the other direction: the run's results going out to somewhere people already look, such as a ReportPortal dashboard, a Teams channel or the Test Case work items in Azure DevOps, or to somewhere only your team has.
Three reporters ship, and they implement the same interface you would implement yourself.
| Package | Sends |
|---|---|
Precept.Reporting.ReportPortal |
A launch per run, a suite per feature, a test item per scenario, with logs and captured files attached. |
Precept.Reporting.Teams |
One Adaptive Card at the end of a run: the verdict, the counts, the pass rate, and a link to the build. |
Precept.Reporting.AzureDevOps |
A test run whose results are filed against the Test Case work items your scenarios name, in their titles or in a tag. |
Reference the package and register it in your startup. Everything else is configuration:
public sealed class Startup : IPreceptStartup
{
public void ConfigureServices(IServiceCollection services, PreceptSettings settings)
{
services.AddPreceptReportPortalReporting();
services.AddPreceptTeamsReporting();
services.AddPreceptAzureDevOpsReporting();
}
}
{
"Reporting": {
"ReportPortal": { "Endpoint": "https://rp.example.com", "Project": "acme_regression" },
"Teams": { "Variant": "Pipeline", "NotifyOn": "Failure" },
"AzureDevOps": { "OrganizationUrl": "https://dev.azure.com/acme", "Project": "Web", "TestPlanId": 412 }
}
}
Endpoints belong in precept.json. Keys and webhook URLs do not: they are credentials, and the
environment-variable override is what keeps them out of source control.
PRECEPT_REPORTING__REPORTPORTAL__APIKEY, PRECEPT_REPORTING__TEAMS__WEBHOOKURL and
PRECEPT_REPORTING__AZUREDEVOPS__PERSONALACCESSTOKEN are the three.
Each package's page documents its own settings: ReportPortal, Teams, Azure DevOps. What a pipeline has to pass them, and where the credentials come from, is on Running a suite on CI.
What gets attached, and what does not#
ReportPortal and Azure DevOps upload the files a test captured (screenshots, Playwright traces, request logs) as attachments of its item or result. A trace or a request log holds the test user's cookies and anything the test typed, passwords included; see Traces and HARs hold credentials. Three settings in the reporter's own section decide which files go:
| Setting | Default | What it does |
|---|---|---|
AttachArtifacts |
true |
Upload a test's files at all. false uploads none. |
AttachOnSuccess |
false |
Attach a passing test's files too. By default only a failure's files are uploaded. |
MaxAttachmentBytes |
8388608 (8 MB) |
Largest single file uploaded. Anything larger is skipped, silently: no error, no log line, and the other files still go. |
The cap is what turns a failure with a screenshot and a trace on disk into a dashboard item with
only the screenshot. A Playwright trace of a scenario with more than a handful of steps is routinely
over 8 MB, so expect this. The TRX has no such cap: the same trace is still copied next to the
.trx and reaches the pipeline's published attachments, which is the quickest way to tell a
skipped upload from a trace that was never recorded.
Raise the cap where traces are worth opening from the dashboard:
{ "Reporting": { "ReportPortal": { "MaxAttachmentBytes": 52428800 } } }
Or per agent, as PRECEPT_REPORTING__REPORTPORTAL__MAXATTACHMENTBYTES. The Azure DevOps reporter
has the same setting under its own section. The server has a limit of its own past that. A
ReportPortal deployment's upload size is set on its reverse proxy, and Azure DevOps refuses an
attachment over 100 MB. A file that clears Precept's cap but not the server's fails with the
server's error, which the reporter does report. Where files are written and how they reach the TRX
is on Artifacts and logging.
What the Teams card looks like#
A channel is where a run is noticed, not where it is diagnosed. A stack trace pasted into one buries the next suite's card and is unreadable where it lands. So the card answers the questions a channel asks, and links to the place that answers the rest:
┌──────────────────────────────────────────────┐
│ ❌ Checkout regression │ ← tinted green or red, edge to edge
│ 4 of 133 failed · 7 m 24 s │
├──────────────────────────────────────────────┤
│ 126 4 3 2 │
│ Passed Failed Skipped Flaky │
│ │
│ Pass rate 97% · 126 of 130 │
│ ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓░░░░ │
│ │
│ Environment staging │
│ [ Open build 20260814.3 ] │
└──────────────────────────────────────────────┘
Four layouts ship. Variant picks one, in code, in precept.json, or as
PRECEPT_REPORTING__TEAMS__VARIANT for a single run:
| Variant | Card |
|---|---|
Summary (default) |
The above: verdict, counts, pass rate, environment. No failure messages. |
Compact |
One line: verdict, pass rate, duration, button. For a channel that hears from a dozen suites a day. |
Detailed |
Summary, then the failed tests and their messages, capped by MaxFailuresListed and MaxFailureMessageLength. |
Pipeline |
Summary, then the build: pipeline, build number, branch, commit, who queued it. |
services.AddPreceptTeamsReporting(teams => teams.Variant = PreceptTeamsCardVariant.Pipeline);
The fact list under the pass rate always names the environment. Facts adds rows of your own,
one per key. A key is shown as written, with one exception: a key with no lowercase letter is
written as a label, so a fact set from CI as PRECEPT_REPORTING__TEAMS__FACTS__BUILD reads
Build on the card rather than BUILD.
{
"Reporting": {
"Teams": {
"Facts": { "Browser": "chromium", "Suite": "Checkout" }
}
}
}
The build the card points at comes from the agent, not from a setting. BUILD_BUILDID and the
collection and project variables address an Azure DevOps build; GITHUB_RUN_ID and the repository
address a workflow run. The agent already knows those and a test project cannot, so RunUrl is
only for pointing somewhere else, such as a dashboard of your own. The Pipeline variant's facts
come from the same place. Off an agent there is no build to describe, and that variant renders the
summary it is built on rather than a table of blanks.
Flaky and inconclusive tiles appear only when the run had any. A count of zero is stated in grey rather than red. The pass rate leaves skipped tests out of both sides, because a suite that skips half its scenarios against an environment is not 50% broken.
An image is optional and yours: a pipeline's status badge, a team logo, a shield for the environment.
{
"Reporting": {
"Teams": {
"ImageUrl": "https://dev.azure.com/acme/Web/_apis/build/status/nightly",
"ImagePlacement": "Banner"
}
}
}
ImagePlacement takes Thumbnail (the default, beside the heading), Banner (full width, under
it) or None, which is how an overlay drops the image for one environment without deleting the URL
from the shared file. The address has to be one every reader's Teams client can fetch without
signing in. The card is rendered in their client, not on the agent, so a link into a private
artifact store renders as a broken image for the whole channel.
Linking a run's reports to each other#
A run that reports to ReportPortal and to Teams should say so on the card:
[ Open Azure DevOps ] [ Open ReportPortal ]
Those buttons need no configuration, and could not have any. ReportPortal assigns a launch id when
the launch is opened, and Azure DevOps assigns a test run id when the run is created. So each
reporter posts what it opened to PreceptRunLinks, a noticeboard registered for every run, and the
Teams card renders whatever is on it. Register the reporters and the buttons appear.
"LinkToOtherReports": false turns them off.
One button per service, not one per URL. The agent's build and the test run filed against it
are two ways into the same Azure DevOps. When a reporter has posted a link into the same host, the
detected build gives way to it, because the test run holds the results, the messages and the
screenshots this card leaves out, and its build is one click further on. Without the Azure DevOps
reporter registered there is nothing to give way to, and the card links the build as before. A
RunUrl you configured by hand is never the one dropped.
The board is a noticeboard rather than a protocol, and the difference shows in one place. Reporters run on pumps of their own, so nothing orders ReportPortal opening its launch against Teams building its card. Each destination posts its address as soon as it has one, at the start of the run, and the card renders whatever is there when it is written. A run long enough to be worth a card has always had both. The alternative, a reporter that waits on another reporter's web service before the first test runs, is a worse trade than a button that is very occasionally missing.
A reporter of your own can post to the same board, and is then linked from the card exactly like the shipped ones. See writing a reporter.
CI and local runs#
The three shipped reporters only report from a build agent. A launch, a test run and a channel message all belong to a pipeline. A developer's machine has neither the credentials for them nor a reason to add its runs to the project's statistics. So the registration above is the whole story on CI, and on a laptop the same suite reports nothing and asks for nothing. The reporters are never started, which is why a missing API key is not an error there either. One line says what happened, and it is not a warning:
[Precept] ReportPortal, Teams, AzureDevOps reporting is CI-only and this run is not on a build agent, so nothing is sent.
A run counts as CI when the agent's own variable is set: TF_BUILD, GITHUB_ACTIONS, GITLAB_CI,
JENKINS_URL, TEAMCITY_VERSION, BUILDKITE and the rest, or a CI that does not say false.
There is nothing to configure on the pipeline.
To report from your own machine, for example to check a card's layout or to fill a dashboard while
writing the integration, one key in precept.local.json brings them all in:
{ "Reporting": { "CiOnly": false } }
PRECEPT_REPORTING__CIONLY=false does the same for a single run, and Reporting:<Reporter>:CiOnly
for a single reporter. From there the settings are ordinary settings again: a missing key is an
error, because you have just asked this machine to report.
The detection itself is a setting as well, at the top level rather than under Reporting.
PRECEPT_ISCONTINUOUSINTEGRATION=true makes a local run behave as a CI one throughout, and
"IsContinuousIntegration": false in an overlay takes an agent back out of it.
This is a different axis from the environment, which says what
is under test. A pipeline running against staging and a developer running against staging want the
same precept.staging.json and different reporting, which is what the two separate switches give.
A reporter you write yourself is not CI-only unless it says so. A CSV file or a local dashboard
keeps working everywhere. Return true from CiOnly, or bind a setting to it, when it needs a
credential. Reporting:CiOnly overrides every reporter either way, in both directions.
Each reporter also keeps an Enabled flag of its own, for switching one off outright:
{ "Reporting": { "ReportPortal": { "Enabled": false } } }
Linking scenarios to Azure DevOps test cases#
The mapping lives beside the scenario rather than in a spreadsheet. A scenario can carry it in its title or in a tag:
Scenario: [41207] A customer checks out with a stored card
@smoke @tc:41207
Scenario: A customer checks out with a stored card
Either way the result is filed against Test Case 41207. That is what makes the work item show its
own latest automated outcome, and a test plan report coverage from the run. A scenario covering two
test cases names both, as [41207][41208] or two tags, and is reported against each. Naming one
work item in both places still files one result.
The title is read with TestCaseTitlePattern, a regular expression with a group named id, so
the convention is yours:
{
"Reporting": {
"AzureDevOps": {
"TestCaseTitlePattern": "^\\s*(?<id>\\d+)\\s*[:-]\\s*",
"TestCaseTagPrefix": ""
}
}
}
The default, ^\s*(?:\[(?<id>\d+)\]\s*)+, is anchored at the start of the title on purpose. An
unanchored pattern also finds ids nobody wrote, such as Scenario: The cart shows [3] items, and
Azure DevOps rejects a whole batch of results when one of them names a work item that is not a Test
Case. Set either setting to "" to switch that source off. A test that names no work item at all
is still part of the run unless OnlyMappedTests says otherwise.
Who a run belongs to#
Azure DevOps shows a run's Run by as Unassigned unless it is told who ran it. It does not take
that from the token that posted the results. Precept asks the organization who the token is and
files the run and every result under that identity: your own account from a laptop, the build
service from a pipeline. It is one call at the start of the run. A token not allowed to make it
leaves the field empty rather than failing the run.
RunBy attributes a run to somebody else, as an identity GUID or a unique name for Azure DevOps to
resolve:
{ "Reporting": { "AzureDevOps": { "RunBy": "qa@acme.com" } } }
This is unrelated to BuildId, which associates the run with a build so it appears on that build's
Tests tab, and nothing more.
Running against a test plan#
TestPlanId turns the run into a planned one. Azure DevOps then accepts a result only when it
names the test point it fills, which is a test case in a suite under a configuration, rather
than just the Test Case work item. Precept resolves those points from the ids the scenarios already
carry, so the mapping stays in the feature files:
{
"Reporting": {
"AzureDevOps": { "TestPlanId": 412, "TestSuiteId": 413 }
}
}
TestSuiteId is optional. It names the branch of the plan this run is for: that suite and every
suite below it. Without it, the run covers the whole plan. This is how one static plan serves several
automated runs, each naming its own suite. A test case sits in the suite it was added to, not in the
suites above it, so a suite that groups its test cases into child suites still gets every result.
A test case held twice in that branch, in two suites or under two configurations, is two planned
slots, and the scenario covering it is reported into each. Precept reads the points once per run, from
the plan itself, so test cases shared with other plans make no difference. Reading them is a test
API, so Test Management (read & write) is still the whole requirement.
A planned result also has to carry the test case's revision, and that is a work item read, a
scope a token filing test results does not otherwise need. Precept asks for it, takes a refusal as
the answer, and files revision 1 rather than failing the run. Azure DevOps records the number it is
given without checking it. Add Work Items (read) to the token for the real one. Either way the
run says which it did.
What a planned run cannot hold is a result that fills no slot. A test naming no work item, or naming one that is not in the plan, is left out of it. That is not silent: the count and the reason are written into the run's comment when it completes:
Precept run 3f2a…, 240 tests against 'staging'. 12 test(s) named no Test Case work item and were
left out of this planned run. 3 result(s) named a Test Case with no test point in plan 412.
Leave TestPlanId unset if that is not what you want. The link to the work item comes from the
result's test case, not from the plan, so an unplanned run still updates each Test Case's latest
automated outcome, and it keeps the unmapped tests.
Writing a reporter#
IPreceptReporter has four events and a default implementation for each, so a reporter overrides
only what it needs:
public sealed class SlowTestReporter : IPreceptReporter
{
private readonly List<PreceptTestReport> _slow = [];
public string Name => "SlowTests";
public Task OnTestFinishedAsync(PreceptTestReport test, CancellationToken cancellationToken)
{
if (test.Duration > TimeSpan.FromSeconds(30))
{
_slow.Add(test);
}
return Task.CompletedTask;
}
public async Task OnRunFinishedAsync(PreceptRunSummary run, CancellationToken cancellationToken) =>
await File.WriteAllLinesAsync("slow-tests.txt", _slow.Select(t => $"{t.Duration:mm\\:ss} {t.DisplayName}"), cancellationToken);
}
services.AddPreceptReporter<SlowTestReporter>();
A reporter is resolved from the container, so it can take dependencies, including settings of its
own, bound from a subsection of Reporting by the overload that takes them:
services.AddPreceptReporter<SlowTestReporter, SlowTestSettings>(); // binds Reporting:SlowTest
PreceptTestReport carries the verdict, the timings, the attempt count, the categories, the lines
the test logged, the files it captured and the exception itself. It names the test twice over.
ClassName and MethodName are the code behind it. SuiteName and DisplayName are what it is
called, which for a generated test are the Gherkin feature and scenario titles, so a dashboard
groups by "Checkout" rather than by Acme.Tests.CheckoutFeature. SuiteName falls back to
ClassName for a class whose [TestSuite] was given no title. SuiteDescription carries the
feature narrative, the lines under the Feature: heading, dedented so a destination that renders
markdown does not read them as a code block, or null when there is none. PreceptRunSummary
carries the counts, the wall-clock duration, and up to MaxTrackedFailures of the failures, because
accumulating those is otherwise the first thing every summary notifier has to write.
A reporter that opens something with an address of its own, such as a launch, a test run or a
dashboard page, can post it to PreceptRunLinks, which is registered for every run and injectable
like any other dependency. That is how the Teams card carries a button to the ReportPortal launch
without either package knowing about the other. See
linking a run's reports to each other.
public sealed class DashboardReporter(PreceptRunLinks links) : IPreceptReporter
{
public string Name => "Dashboard";
public async Task OnRunStartingAsync(PreceptRunStarting run, CancellationToken cancellationToken) =>
links.Add("Dashboard", await OpenPageAsync(run, cancellationToken));
}
Three more members shape when a reporter runs:
- Return
IsEnabled => falsewhen a reporter has nothing to do. It is then never given a queue. - Return
CiOnly => truewhen it belongs to a pipeline rather than to a machine. It is then left out of a developer's run before it is started. See CI and local runs. - Throw from
OnRunStartingAsyncwhen the reporter is switched on but has no endpoint or credential to talk to. That is the moment somebody is still watching the console, and by then the run has already established that this reporter was meant to report from here.
What reporting costs a run#
Nothing on the test's thread, by construction.
Each reporter gets a bounded queue and a pump task of its own. Publishing a result is one small record and a non-blocking write per reporter. A destination that takes two seconds to answer, or has stopped answering altogether, delays no test and no other reporter. A reporter is called one event at a time and in order, so it needs no locking of its own.
Never blocking has consequences:
- A queue that fills drops events. Backpressure would mean a test thread waiting on somebody's
web service. Drops are counted and reported at the end of the run, so a short report says it is
short instead of looking complete. Raise
QueueCapacity, or batch inside the reporter, asPrecept.Reporting.AzureDevOpsdoes. - A reporter that throws is contained. The first failure is a warning. After
MaxFailuresof them the reporter is dropped for the rest of the run, so an endpoint that is down costs one message rather than one per test. - The run waits at the end, but not forever. After the last test the queues are drained under
ShutdownTimeoutSeconds, shared across all reporters. Whatever has not gone out by then is abandoned, and the run says so.
| Setting | Default | |
|---|---|---|
Reporting:Enabled |
true |
false builds no pipeline at all |
Reporting:CiOnly |
unset | overrides every reporter: false reports from anywhere, true from a build agent only |
Reporting:<Reporter>:CiOnly |
true for the shipped three |
where that one reporter runs |
Reporting:QueueCapacity |
4096 |
events one reporter may have waiting |
Reporting:ShutdownTimeoutSeconds |
60 |
how long the run waits for reporters to flush |
Reporting:MaxFailures |
10 |
throws before a reporter is dropped |
Reporting:MaxTrackedFailures |
100 |
failures kept on PreceptRunSummary.Failures |
A run with no reporters registered, or with Reporting:Enabled set to false, constructs nothing
and starts no queue. Neither does a discovery request or --list-tests, so listing the tests in an
IDE never opens a launch or posts to a channel.