Running a Precept suite on Azure Pipelines

Two files, meant to be copied into the repository that holds your tests:

They are split because the schedule, the triggers and the dialog belong to a pipeline while the run itself does not: a nightly, a per-release smoke check and an on-demand regression are three entry files over one job template.

• Before the first run
• The run dialog
• How a pipeline sets any Precept setting
• Environments
• The two filters
• Parallelism and retries
• Reporting
• Azure DevOps test plans
• Results, attachments and exit codes
• Scheduling a different configuration
• Gating a deployment on the suite
• When it goes wrong

Before the first run#

The test project#

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net10.0</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Precept.TestPlatform" Version="0.10.0" />
    <PackageReference Include="Precept.Reqnroll" Version="0.10.0" />
    <PackageReference Include="Precept.Web" Version="0.10.0" />
    <PackageReference Include="Precept.Api" Version="0.10.0" />

    <!-- Only for the reporters the suite actually uses. -->
    <PackageReference Include="Precept.Reporting.ReportPortal" Version="0.10.0" />
    <PackageReference Include="Precept.Reporting.Teams" Version="0.10.0" />
    <PackageReference Include="Precept.Reporting.AzureDevOps" Version="0.10.0" />
  </ItemGroup>

</Project>

global.json#

dotnet test on the .NET 10 SDK has to be told to drive Microsoft.Testing.Platform. The job template runs dotnet test from the root of the checkout, and the SDK reads global.json from the directory the command is run in and upward — not from the project's — so this file has to be at the repository root:

{
  "test": { "runner": "Microsoft.Testing.Platform" }
}

Without it the run fails with "Testing with VSTest target is no longer supported".

The startup#

Reporters are registered in code, once. Nothing in the pipeline turns them on — they are CI-only by default, so they wake up on the agent because it sets TF_BUILD, and stay asleep on a developer's machine without asking for the credentials that machine has no business holding.

public sealed class Startup : IPreceptStartup
{
    public void ConfigureServices(IServiceCollection services, PreceptSettings settings)
    {
        services.AddPreceptReportPortalReporting();
        services.AddPreceptTeamsReporting();
        services.AddPreceptAzureDevOpsReporting();
    }
}

precept.json and its overlays#

What is not a credential belongs in the repository, and the environment overlays are what make one pipeline serve five environments:

precept.json          the defaults, and the endpoints that never change
precept.dev.json
precept.test.json
precept.auto.json
precept.stage.json
precept.prod.json

// precept.json
{
  "Environment": "local",
  "MaxParallelism": 8,
  "RunTimeoutMinutes": 180,

  "Web":  { "Browser": "chromium", "Headless": true },
  "Api":  { "TimeoutMilliseconds": 30000 },

  "Reporting": {
    // Endpoints and project names, yes. Keys and webhook URLs, no — those come from the
    // variable group as PRECEPT_* variables.
    "ReportPortal": { "Endpoint": "https://rp.acme.com", "Project": "acme_regression" },
    "Teams":        { "Variant": "Pipeline", "NotifyOn": "FailureOrFlaky" },
    "AzureDevOps":  { "TestCaseTagPrefix": "tc:" }
  }
}

// precept.prod.json — production is read-only and has no test tenancy
{
  "Web": { "BaseUrl": "https://acme.com" },
  "Api": { "BaseUrl": "https://api.acme.com" },
  "Filter": {
    "Exclude": "destructive or seeded-data or @wip"
  }
}

Precept.TestPlatform copies precept*.json to the output directory on build, so there is no <None Update> to write.

The variable group#

Create a variable group named precept-secrets (Pipelines → Library) holding the two credentials, both marked secret:

A secret variable is not put into the environment automatically — that is the whole point of the explicit env: mapping in the job template. If a name here does not match, the value arrives at the test process as the literal string $(reportPortalApiKey) and the reporter fails authenticating rather than silently doing nothing.

The Azure DevOps token#

The Azure DevOps reporter uses the build's own token, so there is no third secret to create — but two things have to be true:

1. The job must be allowed to see it. That is what PRECEPT_REPORTING__AZUREDEVOPS__PERSONALACCESSTOKEN: $(System.AccessToken) in the template does; on a classic pipeline it is the Allow scripts to access the OAuth token checkbox.
2. The build service identity — " Build Service ()" — needs test management rights on the project. Without them the run is created and the results are refused.

A personal access token with Test Management (read & write) works instead, and is what you need if the results should be filed into a different project from the one running the pipeline. Add Work Items (read) as well when the run is against a test plan; see test plans for what it buys.

The run dialog#

Parameters are a compile-time thing in Azure Pipelines, which is why adding one is an edit to azure-pipelines.yml and never to the template — and why a scheduled run cannot have its own.

How a pipeline sets any Precept setting#

The table above is not the limit. Every setting Precept has can be set from a pipeline, because the last source in the configuration chain is the environment:

precept.json  →  precept.{environment}.json  →  PRECEPT_* environment variables

The variable name is the configuration key with : written as __:

That table is a selection. Environment variables is the complete list — every setting Precept has, with its default and what it does.

The rule holds for settings of your own too — an AuthSettings class bound to the Auth section reads PRECEPT_AUTH__CLIENTSECRET, which is how a client secret reaches the suite from a variable group without ever being in the repository. A dictionary's last segment is the key, which is what makes ATTRIBUTES__BUILD add an attribute named build.

Two things are deliberately not settings and so are not spelled this way: --precept-environment and --precept-filter are command-line options, because they say what this run is rather than how the suite behaves. PRECEPT_ENVIRONMENT does work as a fallback for the first of them; there is no PRECEPT_FILTER for the second, and the name would be misleading if there were — Filter is the section holding Filter:Include and Filter:Exclude, which is the other filter.

PRECEPT_ is a namespace with an owner. The settings loader strips the prefix and binds everything after it as a configuration key, so a variable put there is read into the run whether you meant it as a setting or not — and one that collides with a real key changes the run silently. The job template keeps its own working variables — the filter it was handed, the project path, the TRX file name — under SUITE_* for that reason, and only the real settings are PRECEPT_*. Follow the same split in anything you add.

One trap. An environment variable wins over the overlay whatever its value, and an empty string is a value. Setting PRECEPT_FILTER__EXCLUDE to nothing does not mean "the pipeline has no opinion" — it means "this run excludes nothing", and it will resurrect every test precept.prod.json was keeping out. The job template sets that variable from inside the run script, only when the parameter is non-blank, for exactly this reason. Apply the same care to any variable you add.

Environments#

The environment is resolved once per run, first source to say anything winning:

The pipeline uses (1) so it is in charge whatever a developer's machine or the project file says. A name with no overlay is not an error, and the run says so:

[Precept] Environment 'stage'. Filter excluded 5 of 233 tests.
[Precept] There is no 'precept.stag.json' beside the test binary, so the run is using
          'precept.json' alone.

Watch for that line the first time a new environment name is added to the parameter list — it is what a typo looks like, and a suite quietly running on defaults against production is the failure mode it exists to catch.

Precept.TestData follows the same name: testdata.stage.json overlays testdata.json.

The environment is not the same axis as CI. A pipeline against stage and a developer against stage want the same overlay and different reporting, which is why IsContinuousIntegration is a separate switch that the agent sets by itself.

The two filters#

Both take the same expression language — tag / name: / class: operands, and/or/not, parentheses, * and ? wildcards, quoting for anything with spaces — and they do different jobs.

--precept-filter selects among the tests a run has. It is the Filter parameter, and it is what "run the smoke pack against stage" means:

smoke and not slow
(@api or @web) and not @wip
name:*checkout* and tag:regression
class:*.PaymentTests

Filter:Include / Filter:Exclude decide which tests the run has at all. What they remove is gone before discovery publishes anything: absent from the run, from the report and from a test explorer's tree, not reported as skipped, not reachable by the filter above — and when a class loses every test it had, its [BeforeSuite]/[AfterSuite] hooks never run either. That is the environment's own policy, and it belongs in precept.{environment}.json, with the Exclude parameter there for a one-off.

The leading @ restriction is Microsoft.Testing.Platform's, not Precept's: a command-line argument beginning with @ is read as a response file name. Write smoke and not @slow, or wrap the whole thing in parentheses. The job template refuses the value up front so the failure names the parameter rather than a file nobody wrote.

Parallelism and retries#

MaxParallelism bounds how many units of work are in flight, and ParallelScope says what a unit is — a whole class by default, or a single test.

The default matters more than it looks. The Reqnroll generator emits one class per .feature file, so a suite of six features never puts more than six units in flight however high the limit, and the run is as slow as its longest feature. Precept says so at the start of a run when the gap is large enough to matter. Setting the scope to Test schedules every scenario individually; what you give up is ordering within a feature, and the classes that genuinely need it take [NonParallelizable]. The full trade-off is in README.md.

Retries count reruns: 0 reruns nothing, 2 means a test may run three times in all. Only hard failures are rerun — a skip or an inconclusive verdict will not change. A test that was actually rerun keeps every attempt in its output, each headed by how that attempt ended, so a scenario that failed twice and passed on the third does not report one clean green run.

Turning retries up on a pipeline hides flakiness rather than fixing it. FailureOrFlaky on the Teams card is the counterweight: the channel hears about a test that only passed on a rerun.

Reporting#

Reporting never touches the execution path: each reporter gets a bounded queue and a pump task of its own, so a hung dashboard delays no test and no other reporter. What it costs a run is one record and a non-blocking write per test.

The ReportPortal launch description carries a link back to the Azure DevOps build, built from the agent's own variables — BUILD_BUILDID, SYSTEM_TEAMFOUNDATIONCOLLECTIONURI, SYSTEM_TEAMPROJECT. Nothing to configure: a launch someone opened from a chat message leads back to the pipeline that produced it.

To try a card's layout or fill a dashboard from your own machine, one key in precept.local.json brings every reporter in — { "Reporting": { "CiOnly": false } } — and from there a missing API key is an error, because you have just asked that machine to report.

Azure DevOps test plans#

The mapping from scenario to Test Case work item lives beside the scenario, in its title or in a tag:

Scenario: [41207] A customer checks out with a stored card

@smoke @tc:41207
Scenario: A customer checks out with a stored card

Either files that result against Test Case 41207, which is what makes the work item show its own latest automated outcome. A scenario covering two test cases names both and is reported against each.

Setting Test plan id makes it a planned run, and Azure DevOps then accepts a result only when it names the test point it fills — a test case in a suite, under a configuration. Precept resolves those points from the ids the scenarios already carry, so there is no mapping file. What a planned run cannot hold is a result that fills no slot: a test naming no work item, or naming one with no point in the plan, is left out. That is not silent — the count and the reason go into the run's comment:

Precept run 3f2a…, 240 tests against 'stage'. 12 test(s) named no Test Case work item and were
left out of this planned run. 3 result(s) named a Test Case with no test point in plan 412.

Test suite id narrows it. Without one, a test case the plan holds in two suites — or under two configurations — is two planned slots and the scenario is reported into each.

Leave the plan at 0 if that is not what you want: an unplanned run still updates each Test Case's latest automated outcome, and it keeps the unmapped tests.

A planned result also carries the test case's revision, and that is a work item read rather than a test one. Precept asks, takes a refusal as the answer and files revision 1 rather than failing the run; add Work Items (read) to the identity for the real number. Either way the run says which it did.

Results, attachments and exit codes#

--report-trx is available in every Precept suite: the TRX writer is a dependency of Precept.TestPlatform, so there is nothing to install for it. Precept implements the platform's TRX capability, so the file carries the declaring class, each test's categories, the lines it logged — the Gherkin step trace included — and its failure split into message and stack trace.

Recorded files (screenshots, Playwright traces, videos) are copied next to the .trx and listed as <ResultFile> entries, which is what publishRunAttachments: true uploads. The template also publishes the raw artifact directory as a pipeline artifact, because a Playwright trace is worth downloading whole and opening in the trace viewer.

Exit codes worth knowing:

Code 8 is the one to think about. A filter that matches nothing, or an overlay Filter that removes everything, produces it — and it is right that it fails, because a green run of zero tests is exactly what a pipeline should never report as success. The template adds a message naming the filter, since the platform's own says only that nothing ran.

Add --minimum-expected-tests 200 after the -- for a suite whose size you know: it turns "the filter quietly matched twelve of two hundred tests" into a failure as well.

Scheduling a different configuration#

A scheduled run takes the parameter defaults and nothing else — Azure Pipelines has nowhere to put per-schedule values. A nightly that needs a different configuration is therefore a second entry file over the same job template, and that is the whole file:

# azure-pipelines.nightly.yml
trigger: none
pr: none

schedules:
  - cron: '0 2 * * *'
    displayName: Nightly full regression against stage
    branches:
      include: [main]
    always: true

variables:
  - group: precept-secrets

stages:
  - stage: Test
    jobs:
      - template: templates/precept-test-job.yml
        parameters:
          jobSuffix: nightly
          testProject: tests/Acme.Tests/Acme.Tests.csproj
          testProjectDirectory: tests/Acme.Tests
          environment: stage
          excludeFilter: 'wip'
          maxParallelism: 12
          parallelScope: Test
          defaultRetries: 1
          teamsNotifyOn: FailureOrFlaky
          testPlanId: 412

The same shape gives a per-PR smoke pipeline: filter: 'smoke', environment: dev, reportPortal: false, and a pr: trigger instead of a schedule.

Gating a deployment on the suite#

Put the test stage after the deployment stage and let the exit code do the work — a job that fails fails the stage, and a stage that fails stops the ones after it:

stages:
  - stage: DeployStage
    jobs: [ ... ]

  - stage: VerifyStage
    dependsOn: DeployStage
    jobs:
      - template: templates/precept-test-job.yml
        parameters:
          jobSuffix: smoke
          environment: stage
          filter: 'smoke'
          testProject: tests/Acme.Tests/Acme.Tests.csproj
          testProjectDirectory: tests/Acme.Tests

  - stage: DeployProd
    dependsOn: VerifyStage
    condition: succeeded()
    jobs: [ ... ]

For production, use an Azure DevOps environment with an approval check rather than a pipeline parameter — a run against prod should be something a person signed off, not something anyone with queue permission can select from a dropdown.

When it goes wrong#

The full framework reference is README.md.