Skip to content

Add pscale import d1 for Cloudflare D1 offline migration#1278

Open
no-itsbackpack wants to merge 23 commits into
mainfrom
import-d1
Open

Add pscale import d1 for Cloudflare D1 offline migration#1278
no-itsbackpack wants to merge 23 commits into
mainfrom
import-d1

Conversation

@no-itsbackpack

@no-itsbackpack no-itsbackpack commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds pscale import d1 — an offline migration path from Cloudflare D1 (SQLite export) into PlanetScale Postgres.

Workflow: export with wrangler → lint/plan → import → verify → complete

CLI commands (pscale import d1 …)

Command Purpose
doctor Check prerequisites (pgloader, sqlite3, etc.)
lint Analyze a D1 SQL export for migration issues
convert-schema Convert SQLite schema to PostgreSQL DDL
start Lint + plan + load into Postgres (--dry-run for preview only)
verify Row counts, sequences, and content checks
status Show local migration state
complete Mark migration finished (teardown alias)

Export is done with wrangler, not the CLI:

wrangler d1 export <name> --remote --output ./d1-export.sql

Example import flow:

pscale import d1 start mydb dev --input ./d1-export.sql --method pgloader
pscale import d1 verify mydb dev --migration-id <id> --input ./d1-export.sql
pscale import d1 complete mydb dev --migration-id <id>

Also includes

  • Core library: internal/import/d1/ — schema conversion, lint/plan, pgloader import, verify, local migration state
  • Slim Postgres helpers: internal/postgres/ — connection URI parsing, pgx open, psql discovery/version check (no logical replication / pg_dump pipeline)
  • Slack notifications: lifecycle events are reported to the api
  • Tests: unit tests under internal/import/d1/ and internal/cmd/importcmd/

How to test

Prerequisites: pscale auth, a Postgres database branch, pgloader, sqlite3, and a D1 SQL export (from wrangler or the repo fixture).

1. Smoke test (no PlanetScale load)

pscale import d1 doctor
pscale import d1 lint --input internal/import/d1/testdata/sample_d1_export.sql --format json
pscale import d1 start mydb --input internal/import/d1/testdata/sample_d1_export.sql --dry-run --format json

Dry-run returns a migration_id — save it for verify/complete.

2. Full import (against a dev branch)

pscale import d1 start mydb dev --input ./d1-export.sql --method pgloader
pscale import d1 verify mydb dev --migration-id <id> --input ./d1-export.sql
pscale import d1 complete mydb dev --migration-id <id>

3. Automated tests

go test ./internal/import/d1/... ./internal/cmd/importcmd/...

4. Slack notifications (optional)

Run against local api-bb with Sidekiq running. Import lifecycle events should appear in #d1-migrations. Pass --no-notify on start/verify/complete to skip.


Test plan

  • doctor passes with pgloader + sqlite3 installed
  • lint and start --dry-run succeed on the sample fixture
  • Full start → verify → complete against a dev Postgres branch
  • go test ./internal/import/d1/... ./internal/cmd/importcmd/... passes in CI
  • (Optional) API import notifications

@no-itsbackpack no-itsbackpack requested a review from a team as a code owner June 26, 2026 19:39
Comment thread internal/import/d1/plan.go
Comment thread internal/migrate/d1/prepare.go Outdated
Comment thread internal/migrate/d1/verify_checks.go Outdated
Comment thread internal/cmd/importcmd/d1.go Outdated
Comment thread internal/cmd/importcmd/d1.go Outdated
Comment thread internal/import/d1/sqlite_load.go
Comment thread internal/cmd/importcmd/d1_doctor.go Outdated
Comment thread internal/migrate/d1/verify.go Outdated
Comment thread internal/migrate/d1/pgloader.go Outdated
Comment thread internal/import/d1/import.go Outdated
Comment thread internal/import/d1/verify.go Outdated
Comment thread internal/import/d1/import.go
Comment thread internal/import/d1/parse.go
Introduce `pscale import d1` (doctor, lint, convert-schema, start, verify,
status, complete) backed by internal/import/d1, with pgloader-based loading,
local migration state, Slack lifecycle notifications, and Postgres helpers.

Co-authored-by: Cursor <cursoragent@cursor.com>
no-itsbackpack and others added 4 commits June 29, 2026 23:12
Persist migration state before imported/verified Slack notifications, track
schema_applied so failed pgloader runs can resume without re-applying DDL,
and match column UNIQUE constraints with word boundaries only.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Unit tests do not shell out to pgloader. SQLite integration tests use
requireSQLite3 and skip when the CLI is absent.

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread internal/import/d1/import.go Outdated
Named returns were cleared by return nil, err after lint/plan, so start JSON
errors and failure Slack payloads lost migration_id, issues, and method metadata.

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread internal/import/d1/import.go
no-itsbackpack and others added 2 commits June 30, 2026 16:35
Add sample-driven type coercion and complete command library code; fix resume/dest probing, pgloader retry and row validation, verify input-path guards, and complete ordering with verified-phase requirement.

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread internal/import/d1/convert.go
Comment thread internal/import/d1/lint.go
Comment thread internal/import/d1/convert.go Outdated
Comment thread internal/import/d1/verify.go
Comment thread internal/import/d1/constraints.go Outdated
Comment thread internal/import/d1/import.go
Comment thread internal/import/d1/parse.go Outdated
Fix pgloader transforms and table filters, gate boolean coercion on sampled 0/1 values, improve column parsing and FK ordering, and tighten verify/resume error handling for the core start→verify→complete flow.

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread internal/import/d1/pgloader.go Outdated
Comment thread internal/import/d1/import.go Outdated
Comment thread internal/cmd/importcmd/d1_verify.go Outdated
Validate per-table pgloader loads against staged SQLite row counts, normalize input paths for resume, route start/verify client failures through the JSON envelope, and consolidate related tests and pgloader helpers.

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread internal/import/d1/pgloader.go
Comment thread internal/import/d1/state.go
Comment thread internal/import/d1/schema_reset.go
Stale-role deletion was matching every pscale_api_* username and could revoke shell or admin credentials mid-import; scope cleanup to roles this flow creates by name.

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread internal/import/d1/import.go
@planetscale planetscale deleted a comment from cursor Bot Jul 1, 2026
Notify once when pgloader starts loading tables and once when row-count
verification begins, instead of posting per-table updates.

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread internal/cmd/importcmd/d1_convert_schema.go
Comment thread internal/import/d1/import.go
Resolve go.mod/go.sum conflict by taking planetscale-go v0.174.0 from main.

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread internal/import/d1/import.go
no-itsbackpack and others added 2 commits July 5, 2026 22:42
Warn on SQLite views, triggers, partial indexes, and expression indexes during D1 lint, and skip non-portable indexes during schema conversion instead of emitting invalid Postgres DDL.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread internal/cmd/importcmd/d1_verify.go
Resolve db_name from saved migration state before building the destination URI unless --dbname was explicitly passed.

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread internal/cmd/importcmd/d1_start.go
Share ResolveMigrationDBName across start and verify, and avoid overwriting a saved non-default db_name with the CLI default during state updates.

Co-authored-by: Cursor <cursoragent@cursor.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 66a27a1. Configure here.

Comment thread internal/import/d1/import.go Outdated
no-itsbackpack and others added 2 commits July 5, 2026 22:59
Pass DBNameExplicit from start into Import so saved state does not override an explicit dbname flag.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants