Reorganize CI test execution to minimize platform redundancy

[gfraiteur]

Objective

Minimize redundancy in CI test execution. PostSharp runs ~1,800 tests across up to 18 TeamCity platform configurations (OS × Architecture × TFM × SDK version). Many tests are platform-neutral, so executing them on every configuration wastes CI time and resources without improving coverage.

The goal of this work item is to use test platform-constraint metadata to run each test only on the configurations where it provides unique coverage.

Background

A cataloging tool (Build/Scripts/Generate-TestDatabase.ps1) was written to inventory every CI-run test with its platform constraints and a derived classification. It produces a JSON database (Build/bin/Temp/test-database.json) covering 5 test types: build, unit, issue, distribution, and misc tests.

Current measured distribution (branch release/2024.0)

Metric	Count
Total tests	1,809
Active	1,764
Inactive	45
Platform-neutral	1,533
Platform-specific	231

Platform-specific breakdown (active tests):

	.NET FW only	.NET Core only	Both FW+Core	Total
Windows-only	28	9	114	151
Any OS	0	80	0	80
Total	28	89	114	231

The single biggest finding: ~85% of tests are platform-neutral. A platform-neutral test only needs to run on one representative configuration, not all 18.

Key observations

• Architecture is not a discriminating dimension. No test has a host-architecture (x64 vs ARM64) constraint. The x64/ARM64 redundancy in CI can be reduced for the vast majority of tests.
• Platform-neutral tests dominate. Running the 1,533 platform-neutral tests on a single TFM/OS/arch (rather than all configs) is the largest available saving.
• Each test already records its CI-suite membership across the four suite groups (netFramework, netCoreWindows, netCoreLinux, distribution).

Proposed scope

1. Promote Generate-TestDatabase.ps1 from WIP (currently untracked) into the repo, with regeneration wired into the build so the database stays current.
2. Define a policy mapping each classification → minimal set of CI configurations it must run on.
3. Adjust TeamCity / AutoTest.proj test selection so platform-neutral tests run on a single representative config, and platform-specific tests run only where their constraints require.
4. Validate that total coverage is preserved (no test silently dropped) while redundant executions are eliminated.

Open questions

• Which single configuration is "canonical" for platform-neutral tests?
• How to keep the database current and prevent drift (regenerate per-build vs. committed snapshot)?
• Handling of the 6 known MSBuild-condition patterns in Phase B, plus a guard for novel ("unresolved") patterns.

---

Numbers are a snapshot; regenerate the database against the current tree before acting.