Feature request: `skip_remaining`

[AdamWill]

In os-autoinst/openQA#5349 we're dealing with a situation where a set of tests calls a setup function that, for convenience, uses two get_ok calls, which are considered to be tests. After the setup, we may want to skip the actual tests from 27-plugin_obs_rsync_status_details.t - but we can't use skip_all to do this, because we've already run two "tests", and Test::More doesn't seem to like it if you try to skip_all after any "tests" have already run.

It would be convenient in this case if there was a skip_remaining, or just if skip_all could behave as skip_remaining when called after some tests have already run...

[exodist]

Assuming you use done_testing and not a fixed plan, and you are not in a subtest you can probably just put:

if (SHOULD_SKIP) {
    done_testing;
    exit(0)
}

A real implementation would need more logic to account for subtests and fixed plans, here is one with some pseudo-code. Everything here is possible, I just do not remember the implementation details off the top of my head, but if anyone wants tthis and wants to give it a go implementing this, I would gladly merge a PR that introduces this to Test2::Tools::Basic;

sub skip_remaining {
    my $ctx = context();

    if (PSEUDO_CODE_NO_PLAN_SET) {
        $ctx->hub->finalize($ctx->trace, 1);
    }
    else {
        $ctx->PSEUDO_SEND_SKIPPED_OK for $PSEUDO_PLAN - $PSEUDO_CURRENT_COUNT;
    }

    $ctx->release;
    exit(0) if PSEUDOCODE_NOT_IN_SUBTEST; # Exit if we are not in a subtest
    last T2_SUBTEST_WRAPPER; # Terminate the subtest
    die "Scope Leak"; # Should never get here
}

[AdamWill]

Well, openQA's driver_missing thing did use done_testing (then exit) - but it seems like on recent versions of...something (not sure exactly what update changed it, because I started running into this on a distribution upgrade), that breaks, because doing that before any tests have run started to be considered as a failure. So my PR is actually intended exactly to switch away from done_testing to skip_all, heh.

Basically it seems like on recent versions of everything, you can only safely use done_testing if at least one test has run, and you can only safely use skip_all if no tests have run. Which makes things a bit inconvenient. There's nothing you can use to just mean "whether any tests have already run or not, I want to skip the rest of them without counting exactly how many there are".

I will have a go at doing the implementation you suggested if I find time (thanks for the starting point!), can't promise I will though, unfortunately :(

[exodist]

Then add some logic to skip_remaining to simply call skip_all if no tests have been run.

[AdamWill]

it occurred to me to put some logic like that into openQA's driver_missing thing, yeah - have it figure out which one to call - but at least on a quick look through the docs I couldn't immediately see how you tell whether any tests have run yet. Is there a simple way to do that?

[exodist]

$ctx->hub->count will be 0 if no tests have run, a positive integer if tests have run.
$ctx->hub->plan can also give you the current plan state, undef for no plan at all, a number for how many tests are expected, 'SKIP' if a skip-all has happened, and 'NO PLAN' for some legacy Test::Builder tests that explicitly say they have no plan (In which case you can just exit or end the subtest with no need for done_testing)

[exodist]

oh, and $ctx->hub->nested can be used to detect a subtest, it will be 0 outside of subtests, a positive integer for inside 1 or more subtests

[haarg]

Test::Needs may work for your use case. It tries to handle skipping tests with skip_all if possible, or skipping the remaining tests if there was already a test plan issued.

[AdamWill]

FWIW, this discussion inspired me to poke around in the code a bit more and I wound up finding a decent workaround for this specific downstream case which makes this need less important for now. I'll leave it up to you folks to decide if you want to still keep it open or not.