Pipeline Transition Plan#4381
Conversation
… proposed directory and sharing structures.
paulmedynski
added
the
Area\Engineering
There was a problem hiding this comment.
Pull request overview
Adds a new eng/pipelines/README.md describing the intended transition from the legacy Azure DevOps pipeline layout to a new grouped layout (ci/pr/official/shared), to help contributors understand where new pipeline work should land during the migration.
Changes:
- Documented the target directory structure and naming conventions for pipeline entry YAMLs.
- Captured the legacy layout and current migration status table.
- Added contributor guidelines for where to place new pipelines/templates during the transition.
| Every leaf directory above (each pipeline group with entry files, plus the `shared/` directories) | ||
| also contains the standard template subdirectories, omitted above for brevity: |
| `shared/`. A template that is only used within a single pipeline group stays in that group's own | ||
| subdirectory, for example `ci/shared/` | ||
|
|
||
| ## Legacy Layout (being removed) |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4381 +/- ##
==========================================
- Coverage 65.45% 63.62% -1.84%
==========================================
Files 285 280 -5
Lines 43373 66257 +22884
==========================================
+ Hits 28388 42153 +13765
- Misses 14985 24104 +9119
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
benrr101
left a comment
benrr101
left a comment
There was a problem hiding this comment.
Decent enough. I'm not a fan of "shared" over "common" but ... whatever 🤷♂️
|
|
||
| Each directory may contain multiple entry files when there are distinct pipelines in the same | ||
| logical group (e.g. `official/` has separate official and non-official pipelines). In `ci/`, each | ||
| pipeline (package, test, stress, kerberos) lives in its own subdirectory with its own `jobs/`, |
There was a problem hiding this comment.
I personally don't think this is necessary, but maybe there will be a mess of mixed up steps, etc without it
There was a problem hiding this comment.
I agree, multiple directories can be avoided. Things should be parameterized to re-use core execution scripts that when modified/updated, updates all pipelines together.
| ├── sqlclient-pr-project-ref-pipeline.yml | ||
| ├── github-sync-pipeline.yml # ⚠ Legacy — moving to github/ | ||
| ├── common/ # ⚠ Legacy — shared templates | ||
| │ ├── templates/ |
There was a problem hiding this comment.
I think its worth noting that the files in this folder should be going away entirely.
In other words:
- common/templates -> old, old "common" yaml (that really wasn't common at all)
- common/[everything else] -> "legacy" common yaml, that can be moved directly to shared
| │ │ └── sqlclient-ci-kerberos-pipeline.yml | ||
| │ ├── package/ | ||
| │ │ └── sqlclient-ci-package-pipeline.yml | ||
| │ ├── shared/ # Templates shared across CI pipelines only |
There was a problem hiding this comment.
Why does CI need its own shared directory? Can we aim for only 1 shared directory to avoid duplication of CI artifacts with PR/test/official pipelines, etc.
| ```text | ||
| eng/pipelines/ | ||
| ├── ci/ # CI pipelines (package, test, stress, kerberos, …) | ||
| │ ├── kerberos/ |
There was a problem hiding this comment.
I would recommend moving kerberos and stress pipelines to test\ directory. I would imagine test to contain all testing related yamls (expandable in future). We don't need separate directories for 1 yaml file each.
|
|
||
| Each directory may contain multiple entry files when there are distinct pipelines in the same | ||
| logical group (e.g. `official/` has separate official and non-official pipelines). In `ci/`, each | ||
| pipeline (package, test, stress, kerberos) lives in its own subdirectory with its own `jobs/`, |
There was a problem hiding this comment.
I agree, multiple directories can be avoided. Things should be parameterized to re-use core execution scripts that when modified/updated, updates all pipelines together.
| ## Target Layout | ||
|
|
||
| ```text | ||
| eng/pipelines/ |
There was a problem hiding this comment.
We also need docs generation, localization pipelines to be included.
| ### `shared/` Directories | ||
|
|
||
| Templates that are consumed by multiple top-level pipelines live in the top-level | ||
| `shared/`. A template that is only used within a single pipeline group stays in that group's own |
There was a problem hiding this comment.
It's debatable for CI - since most of the CI steps are similar to PR pipeline, if you can parameterize them well.
| └── onebranch/ # ⚠ Legacy — renaming to official/ | ||
| ``` | ||
|
|
||
| ## Migration Status |
There was a problem hiding this comment.
Please include other remaining pipelines as well.
Also, could you add phase-based approach as well? What is the step-wise approach here to get this migration completed?
And finally, also provide clarity about how much progress has been made for the items 'In Progress'.
e.g. Official pipelines need to run all tests as package pipeline too, should be captured as one of the pending tasks for the official/non-official pipelines.
github-project-automation
Bot
moved this from In review
to Waiting for customer
in SqlClient Board
cheenamalhotra
added
the
Author attention needed
|
@paulmedynski This pull request has been marked as Author attention needed. When you have addressed the reviewer feedback and are ready for another review, please post a comment with |
Description
Added a README with some initial notes on the transition plan from old -> new pipeline layouts and sharing.