Add pscale import d1 for Cloudflare D1 offline migration#1278
Open
no-itsbackpack wants to merge 23 commits into
Open
Add pscale import d1 for Cloudflare D1 offline migration#1278no-itsbackpack wants to merge 23 commits into
no-itsbackpack wants to merge 23 commits into
Conversation
Introduce `pscale import d1` (doctor, lint, convert-schema, start, verify, status, complete) backed by internal/import/d1, with pgloader-based loading, local migration state, Slack lifecycle notifications, and Postgres helpers. Co-authored-by: Cursor <cursoragent@cursor.com>
18731b9 to
8aa1a34
Compare
Persist migration state before imported/verified Slack notifications, track schema_applied so failed pgloader runs can resume without re-applying DDL, and match column UNIQUE constraints with word boundaries only. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Unit tests do not shell out to pgloader. SQLite integration tests use requireSQLite3 and skip when the CLI is absent. Co-authored-by: Cursor <cursoragent@cursor.com>
Named returns were cleared by return nil, err after lint/plan, so start JSON errors and failure Slack payloads lost migration_id, issues, and method metadata. Co-authored-by: Cursor <cursoragent@cursor.com>
dgraham
approved these changes
Jun 30, 2026
Add sample-driven type coercion and complete command library code; fix resume/dest probing, pgloader retry and row validation, verify input-path guards, and complete ordering with verified-phase requirement. Co-authored-by: Cursor <cursoragent@cursor.com>
Fix pgloader transforms and table filters, gate boolean coercion on sampled 0/1 values, improve column parsing and FK ordering, and tighten verify/resume error handling for the core start→verify→complete flow. Co-authored-by: Cursor <cursoragent@cursor.com>
Validate per-table pgloader loads against staged SQLite row counts, normalize input paths for resume, route start/verify client failures through the JSON envelope, and consolidate related tests and pgloader helpers. Co-authored-by: Cursor <cursoragent@cursor.com>
Stale-role deletion was matching every pscale_api_* username and could revoke shell or admin credentials mid-import; scope cleanup to roles this flow creates by name. Co-authored-by: Cursor <cursoragent@cursor.com>
Notify once when pgloader starts loading tables and once when row-count verification begins, instead of posting per-table updates. Co-authored-by: Cursor <cursoragent@cursor.com>
Resolve go.mod/go.sum conflict by taking planetscale-go v0.174.0 from main. Co-authored-by: Cursor <cursoragent@cursor.com>
Warn on SQLite views, triggers, partial indexes, and expression indexes during D1 lint, and skip non-portable indexes during schema conversion instead of emitting invalid Postgres DDL. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Resolve db_name from saved migration state before building the destination URI unless --dbname was explicitly passed. Co-authored-by: Cursor <cursoragent@cursor.com>
Share ResolveMigrationDBName across start and verify, and avoid overwriting a saved non-default db_name with the CLI default during state updates. Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 66a27a1. Configure here.
Pass DBNameExplicit from start into Import so saved state does not override an explicit dbname flag. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Summary
Adds
pscale import d1— an offline migration path from Cloudflare D1 (SQLite export) into PlanetScale Postgres.Workflow: export with wrangler → lint/plan → import → verify → complete
CLI commands (
pscale import d1 …)doctorlintconvert-schemastart--dry-runfor preview only)verifystatuscompleteteardownalias)Export is done with wrangler, not the CLI:
Example import flow:
Also includes
internal/import/d1/— schema conversion, lint/plan, pgloader import, verify, local migration stateinternal/postgres/— connection URI parsing, pgx open, psql discovery/version check (no logical replication / pg_dump pipeline)internal/import/d1/andinternal/cmd/importcmd/How to test
Prerequisites:
pscaleauth, a Postgres database branch,pgloader,sqlite3, and a D1 SQL export (from wrangler or the repo fixture).1. Smoke test (no PlanetScale load)
Dry-run returns a
migration_id— save it for verify/complete.2. Full import (against a dev branch)
3. Automated tests
go test ./internal/import/d1/... ./internal/cmd/importcmd/...4. Slack notifications (optional)
Run against local api-bb with Sidekiq running. Import lifecycle events should appear in
#d1-migrations. Pass--no-notifyon start/verify/complete to skip.Test plan
doctorpasses with pgloader + sqlite3 installedlintandstart --dry-runsucceed on the sample fixturego test ./internal/import/d1/... ./internal/cmd/importcmd/...passes in CI