Execution model
Tests are grouped by their declaring class. MaxParallelism bounds how many units of work run at
once, and ParallelScope decides what a unit is:
ParallelScope |
One unit is | Tests of one class |
|---|---|---|
"Class" (default) |
a whole class | run one after another |
"Test" |
a single test | may run at the same time |
Anything marked [NonParallelizable] runs afterwards, one class at a time, whichever scope is set.
A single non-parallelizable test takes its whole class into that phase, so the class's
[BeforeSuite] and [AfterSuite] pair stays together.
Why a high MaxParallelism can look ignored#
The default unit is a class, and the Reqnroll generator emits one class per .feature file. A
suite of two feature files therefore never runs more than two units at once. With
"MaxParallelism": 8, six slots stay empty and the run takes as long as its longest feature.
Precept prints a warning at the start of a run when the gap is large enough to matter.
Setting "ParallelScope": "Test" schedules every scenario individually, so a suite with a few long
feature files can use the whole limit. What you give up is ordering within a feature:
- Scenarios of one feature no longer see each other's leftovers in a fixed order: a static field, a row a previous scenario inserted, a value stashed in a step-definition field.
- Reqnroll takes a pooled runner per scenario, so its own
[BeforeFeature]and[AfterFeature]hooks run once per runner rather than once per feature. Reqnroll behaves this way under every parallel-capable provider. - Precept's own
[BeforeSuite]still completes before the class's first test starts, and[AfterSuite]still runs after its last one finishes.
Put [NonParallelizable] on the few classes that need the old ordering, and leave the rest
test-scoped, rather than holding the whole run back for them.
Retries, timeouts and reporting#
Retries#
[Retry(n)] counts reruns, not runs. A test always runs once, and n is how many more times
it runs while it keeps failing. [Retry(1)] runs a failing test twice; [Retry(0)] is the same as
no attribute. "DefaultRetries" in precept.json does the same for tests without the attribute,
and defaults to 0.
Only hard failures are rerun. A skip or an inconclusive verdict will not change on a rerun.
A test that was rerun keeps every attempt in its output. Each block opens with the attempt it belongs to and closes with how that attempt ended:
[Attempt 1 of 3]
Given the catalogue service is reachable
When the client requests the catalogue
[Attempt 1 of 3] Failed: Expected status 200 but received 503.
[Attempt 2 of 3]
Given the catalogue service is reachable
When the client requests the catalogue
[Attempt 2 of 3] Passed
Without this, a scenario that failed twice and passed on the third would report one clean green
run. The labels appear only when a rerun happened, so a test that passes first time reads as it
always did. A test's duration covers every attempt, including the DelayMilliseconds between them.
The same goes for files. A rerun writes into its own directory under the test's, attempt-2,
attempt-3 and so on, instead of writing its failure.png and trace.zip over the first
attempt's. The result attaches every attempt's files, each labelled [Attempt n of m] like the log.
A test that passed on its rerun therefore still carries the screenshot and trace of the failure it
retried past, which is what a flaky test has to show.
Timeouts#
A single test has no time limit by default. An end-to-end test takes as long as the system under test makes it take, and a limit chosen in advance for every test is either too short for the slowest test or too long to mean anything.
What a run needs protecting from is never ending: a step waiting on a socket that will not answer,
a loop that never exits. One ceiling on the whole run is enough for that. "RunTimeoutMinutes" in
precept.json defaults to 12 hours (720); 0 removes the limit. When it fires:
- every test still running is cut off and reported as failed with the reason,
- every test not yet started is reported as failed without running,
- the runner writes one line saying the run was over time.
The TRX and the reporters get a verdict for everything the run selected, which a build agent killing the process would never give them.
A test that wants its own limit can have one: [Timeout(ms)] on the test or its class, or
"DefaultTimeoutMilliseconds" for every test without the attribute, which defaults to 0. Either
limit is enforced in two steps:
PreceptTestContext.Current.CancellationTokenis cancelled. A body that observes the token fails with aTimeoutExceptionthere and then.- A body that never looks at the token, which is every Reqnroll step and most tests, is abandoned.
The verdict is the same
TimeoutException, reported at the timeout rather than whenever the body would have finished. The[AfterTest]hook, the module verifications and disposal all run as they do for any failure, so a browser test that overran still gets its screenshot and trace.
The abandoned body keeps running in the background until what it holds is disposed under it, which is usually at once. Anything it throws after that is logged to the test, not raised. Observe the token where a test can, in a wait, an HTTP call or a database command, and it ends cleanly instead.
No limit is enforced while a debugger is attached, neither the run's nor a test's. The clock keeps running at a breakpoint, and a limit that fired there would fail the test being looked at and close its browser. Debug for as long as you need; the limits return on the next run without a debugger.
What reaches the reporters#
Each result carries its timing, category metadata and artifact paths through to the platform's reporters and to every Precept reporter.