Allow disabling test discovery

[rburchell]

Description

In some contexts, discovery might not be wanted, for example:

• if there are a large number of test binaries, link times can grow unbounded
• if the test binaries are prone to crashing due to some odd build configuration

Add a new opt-in way to do this by setting the discovery mode to "DISABLED".

GitHub Issues

n/a

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.38%. Comparing base (e83218c) to head (e4eb35d).
⚠️ Report is 30 commits behind head on devel.

Additional details and impacted files

@@           Coverage Diff           @@
##            devel    #3089   +/-   ##
=======================================
  Coverage   91.38%   91.38%           
=======================================
  Files         204      204           
  Lines        8916     8916           
=======================================
  Hits         8147     8147           
  Misses        769      769           

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[horenmar]

I do not believe the motivation here is sufficient.

1. If you hardcode the discovery mode to DISABLED, that's IMO worse than just not calling catch_discover_tests at all.

2. If you use configure-time logic to decide between the different discovery modes, you can have an if branch there as well.

In addition,

• if there are a large number of test binaries, link times can grow unbounded

If the issue is that you don't want to run discovery after link, e.g. because you find yourself rebuilding lot more often than running unit tests, PRE_TEST discovery mode already solves this.

• if the test binaries are prone to crashing due to some odd build configuration

If a build configuration leads to some test binaries crashing, your build system should already know which ones these are. This allows you to workaround the problem with disabling that call with 2).

[rburchell]

If you hardcode the discovery mode to DISABLED, that's IMO worse than just not calling catch_discover_tests at all.

If a machine is either underpowered enough, or configured in such a way that it is going to waste 5 minutes of time configuring for a developer, I would much rather that they just turn it off rather than make the experience worse for CI (and everyone else).

If you use configure-time logic to decide between the different discovery modes, you can have an if branch there as well.

Sure, I could patch every single test binary across both my own code and all of our dependencies. Or I could just patch Catch.cmake, which I suppose I will have to continue to do so on my own.

If the issue is that you don't want to run discovery after link, e.g. because you find yourself rebuilding lot more often than running unit tests, PRE_TEST discovery mode already solves this.

PRE_TEST has its own set of problems. I don't remember what they were right now because time has passed. One of which is that it just shifts the point of the problem from building to testing, so even something straightforward like "what are the tests I can run" now takes 5 minutes, which is unacceptable from a dev UX perspective. The other one was something related to cross compilation, I think.

I could go on, but while you took a while to reply, you've made it pretty clear you aren't interested in this. I'll take it off your plate and just keep patching it locally myself.