[SPARK-57681][SQL] Support dynamic table options for UPDATE by anuragmantri · Pull Request #56792 · apache/spark

anuragmantri · 2026-06-25T23:07:22Z

What changes were proposed in this pull request?

This PR add the dynamic table options syntax to UPDATE statements.

Why are the changes needed?

This is a continuation to adding options syntax in DMLs SPARK-49098

Does this PR introduce any user-facing change?

Yes, it adds new SQL syntax. For example:

 UPDATE catalog.db.table WITH (`k` = 'v' )
 SET status = 'archived' WHERE created_at < '2024-01-01'.

How was this patch tested?

Added tests to the following suites.

Parser (DDLParserSuite)
End-to-end (UpdateTableSuiteBase)
LogicalWriteInfo (InMemoryRowLevelOperationTable)

Was this patch authored or co-authored using generative AI tooling?

I used Claude Code (Claude Opus 4.8) to generate the code and tests and verified manually.

anuragmantri · 2026-06-25T23:09:37Z

@szehon-ho, tagging you since you made the INSERT counterpart :)

uros-b · 2026-06-26T11:33:30Z

          // don't rewrite as the table supports truncation
          d

        case r @ ExtractV2Table(t: SupportsRowLevelOperations) =>


Please consider this concern: silent option-drop on non-rewrite delete paths: for a table implementing both SupportsRowLevelOperations and SupportsDeleteV2 (e.g. Iceberg), a DELETE … WITH(...) WHERE can take the metadata-only/deleteWhere path (OptimizeMetadataOnlyDeleteFromTable / DataSourceV2Strategy), which has no options parameter; so the user's options are silently ignored. Same for the TruncatableTable truncate path. This would be a new user-visible "silently ignored" surprise, and no test currently covers this path.

Great catch. I looked at those paths. The options are indeed not being passed all the way. Fixing this requires DSv2 API changes something like this with default methods.

// SupportsDeleteV2 default void deleteWhere(Predicate[] predicates, CaseInsensitiveStringMap options) { deleteWhere(predicates); // default ignores options → fully back-compatible } // TruncatableTable default boolean truncateTable(CaseInsensitiveStringMap options) { return truncateTable(); }

I can make this change in this same PR or can do a follow up if DSv2 changes need separate PRs. Let me know what you think is the best way forward.

uros-b · 2026-06-26T11:34:27Z

-    | UPDATE identifierReference tableAlias setClause whereClause?                 #updateTable
+    | DELETE FROM identifierReference optionsClause? tableAlias whereClause?       #deleteFromTable
+    | UPDATE identifierReference optionsClause? tableAlias setClause whereClause?  #updateTable
    | MERGE (WITH SCHEMA EVOLUTION)? INTO target=identifierReference targetAlias=tableAlias


So this PR makes INSERT/DELETE/UPDATE accept WITH(...), but how about MERGE?

I intentionally did not add MERGE as the semantics are a bit different since MERGE statements have source and target relations, I'm still thinking about how options would look like there. I can file a separate JIRA and PR for MERGE later.

uros-b · 2026-06-26T11:35:36Z

        stop = 56))
  }

+  test("delete from table: with options") {


Please make sure to either update the SQL reference docs, or file a followup Jira ticket to do so later.

For sure. I took a look and looks like INSERT docs are also not updated. I will create a follow up JIRA to update all options related DMLS at once.

anuragmantri · 2026-06-30T21:44:16Z

@dongjoon-hyun - Could you please review this PR since you had reviewed its precedent (SPARK-49098) as well? Thank you!

dongjoon-hyun

Thank you for pinging me, @anuragmantri .

In general, given the code contribution, the claim of this PR title is too broad. When you make a PR, please revise the PR title by narrowing down as much as possible to the exact contribution scope.

We don't need to add MERGE here. Instead, I'd like to recommend to proceed DELETE FROM syntax in another JIRA issue after removing this PR because it could be misleading if we ignore options. For UPDATE syntax, IIUC, there is no silent ignoring code path.

Lastly, follow-up have a special meaning in the community. So, please don't say follow-up of something especially when the Fix Version is different.

If you narrow down the scope according to the above, I believe we can get a higher chance of merge to Apache Spark 4.3.0, @anuragmantri .

anuragmantri · 2026-07-01T20:33:30Z

Thanks for the valuable feedback @dongjoon-hyun. As you suggested, I have narrowed the scope of this JIRA and the PR to only UPDATE as it is the most unambiguous. I will work on the other DMLs and the docs on separate JIRAs.

This is ready for another round. Thanks!

dongjoon-hyun

What happens for the following queries?

UPDATE t WITH (...) ...
WHERE id IN (SELECT ... FROM t)

We don't consider these new options for analyzer relation caches, do we?

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala

Lines 238 to 242 in 972f696

    
           case CatalogAndIdentifier(catalog, ident) => 
        
             val key = toCacheKey(catalog, ident, finalTimeTravelSpec) 
        
             val planId = u.getTagValue(LogicalPlan.PLAN_ID_TAG) 
        
             relationCache 
        
               .get(key)

Given that, it seems that there are more corner cases with analyzer relation cache. Could you add more test coverage and verity them? Typically, Subquery and CTE are the corner cases we need to consider carefully, @anuragmantri .

dongjoon-hyun · 2026-07-01T22:03:37Z

    | fromClause multiInsertQueryBody+                                             #multiInsertQuery
    | DELETE FROM identifierReference tableAlias whereClause?                      #deleteFromTable
-    | UPDATE identifierReference tableAlias setClause whereClause?                 #updateTable
+    | UPDATE identifierReference optionsClause? tableAlias setClause whereClause?  #updateTable


Although this seems to follow read path, this is inconsistent with our existing INSERT syntax.

spark/sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4

Lines 592 to 595 in 972f696

| INSERT (WITH SCHEMA EVOLUTION)? INTO TABLE? identifierReference tableAlias optionsClause? (BY NAME)?

REPLACE (WHERE | ON) replaceCondition=booleanExpression #insertIntoReplaceBooleanCond

| INSERT (WITH SCHEMA EVOLUTION)? INTO TABLE? identifierReference tableAlias optionsClause? (BY NAME)?

REPLACE USING identifierList #insertIntoReplaceUsing

UPDATE should follow INSERT style.

Done. I moved the options clause after tableAlias.

dongjoon-hyun · 2026-07-01T22:05:51Z

        stop = 70))
  }

+  test("update table: with options") {


Please use test prefix when it's possible.

- test("update table: with options") { + test("SPARK-57681: update table: with options") {

Done. I added SPARK-57681: prefix to all tests added as part of this PR.

dongjoon-hyun · 2026-07-01T22:13:12Z

I finished the second-round review, @anuragmantri .

Please follow INSERT's syntax instead of SELECT. tableAlias should come first.
Please double-check RelationResolution class. Especially, subqueries and CTE cases. At least, this PR should add a test coverage to clarify what is supposed by this PR.
And, a minor comment about test prefixes.

peter-toth · 2026-07-02T12:49:49Z

+        |""".stripMargin)
+
+    checkRowLevelOperationOptions(
+      sql(s"UPDATE $tableNameAsString WITH (`write.split-size` = 10) SET salary = -1 WHERE pk = 1"),


This test only covers a simple WHERE pk = 1 predicate, which takes the plain buildReplaceDataPlan / buildWriteDeltaPlan path. Two option-relevant paths go unexercised:

The subquery rewrite buildReplaceDataWithUnionPlan (triggered when the condition contains a subquery) -- a different plan shape that also has to carry r.options through.

The analyzer relation-cache case @dongjoon-hyun raised: with the same table referenced in a WHERE ... IN (SELECT ... FROM <same table>), the cache is keyed without options (RelationResolution.toCacheKey, and the cache hit at RelationResolution.scala:241 ignores the current relation's finalOptions), so the options may be dropped or leaked between the target and the subquery relation.

Suggest adding a variant that reuses the target table in the predicate, e.g.:

checkRowLevelOperationOptions( sql(s"UPDATE $tableNameAsString WITH (`write.split-size` = 10) " + s"SET salary = -1 WHERE pk IN (SELECT pk FROM $tableNameAsString WHERE salary > 50)"), "write.split-size" -> "10")

This exercises the union path and would surface any option loss on the self-reference.

Thank you, I added tests that exercise the buildReplaceDataWithUnionPlan. I added my analysis of ResolveResolution code and table cache as a PR comment. Please let me know what you think.

anuragmantri

Thanks for the useful pointers on the table cache @dongjoon-hyun and @peter-toth . I traced the cache loads in RelationResolution. For the UPDATE case, the target table is always resolved first. The subquery or CTE resolution happens later. This ensures that the options are not lost.

I added a test each with subquery and CTE as @peter-toth suggested and that proves this.

However, for INSERT query like this

INSERT INTO t WITH (...) SELECT ... FROM t

The reused read is an operator child and is resolved before the target in the same bottom-up pass. I was able to reproduce it with a test, but that issue is pre-existing and out of scope for this PR.

This PR is ready for another round.

anuragmantri · 2026-07-07T01:41:33Z

    | fromClause multiInsertQueryBody+                                             #multiInsertQuery
    | DELETE FROM identifierReference tableAlias whereClause?                      #deleteFromTable
-    | UPDATE identifierReference tableAlias setClause whereClause?                 #updateTable
+    | UPDATE identifierReference optionsClause? tableAlias setClause whereClause?  #updateTable


Done. I moved the options clause after tableAlias.

anuragmantri · 2026-07-07T01:41:57Z

        stop = 70))
  }

+  test("update table: with options") {


Done. I added SPARK-57681: prefix to all tests added as part of this PR.

anuragmantri · 2026-07-07T01:44:05Z

+        |""".stripMargin)
+
+    checkRowLevelOperationOptions(
+      sql(s"UPDATE $tableNameAsString WITH (`write.split-size` = 10) SET salary = -1 WHERE pk = 1"),


Thank you, I added tests that exercise the buildReplaceDataWithUnionPlan. I added my analysis of ResolveResolution code and table cache as a PR comment. Please let me know what you think.

dongjoon-hyun · 2026-07-07T02:10:52Z

Thank you for updating, @anuragmantri .

dongjoon-hyun

+1, LGTM. Thank you, @anuragmantri .

## What changes were proposed in this pull request? This PR add the dynamic table options syntax to UPDATE statements. ## Why are the changes needed? This is a continuation to adding options syntax in DMLs [SPARK-49098](https://issues.apache.org/jira/browse/SPARK-49098) Does this PR introduce any user-facing change? Yes, it adds new SQL syntax. For example: ```sql UPDATE catalog.db.table WITH (`k` = 'v' ) SET status = 'archived' WHERE created_at < '2024-01-01'. ``` ## How was this patch tested? Added tests to the following suites. - Parser (DDLParserSuite) - End-to-end (UpdateTableSuiteBase) - LogicalWriteInfo (InMemoryRowLevelOperationTable) ## Was this patch authored or co-authored using generative AI tooling? I used Claude Code (Claude Opus 4.8) to generate the code and tests and verified manually. Closes #56792 from anuragmantri/delete-update-sql-options. Authored-by: Anurag Mantripragada <amantripragada@apple.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 6b9d033) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

dongjoon-hyun · 2026-07-08T00:20:42Z

Merge Summary:

merged into master 6b9d033
merged into branch-4.x 81ea5df

Posted by merge_spark_pr.py

dongjoon-hyun · 2026-07-08T00:24:09Z

Thank you all.

To @anuragmantri , I added you to the Apache Spark contributor group and assigned SPARK-57681 to you.
Congratulations for your first commit.

uros-b reviewed Jun 26, 2026

View reviewed changes

dongjoon-hyun requested changes Jun 30, 2026

View reviewed changes

anuragmantri force-pushed the delete-update-sql-options branch from 398cadc to 50072c3 Compare July 1, 2026 20:30

anuragmantri changed the title ~~[SPARK-57681][SQL] Support dynamic table options for DELETE and UPDATE~~ [SPARK-57681][SQL] Support dynamic table options UPDATE Jul 1, 2026

anuragmantri changed the title ~~[SPARK-57681][SQL] Support dynamic table options UPDATE~~ [SPARK-57681][SQL] Support dynamic table options UPDATE Jul 1, 2026

anuragmantri changed the title ~~[SPARK-57681][SQL] Support dynamic table options UPDATE~~ [SPARK-57681][SQL] Support dynamic table options for UPDATE Jul 1, 2026

dongjoon-hyun reviewed Jul 1, 2026

View reviewed changes

peter-toth reviewed Jul 2, 2026

View reviewed changes

anuragmantri added 2 commits July 6, 2026 18:30

[SPARK-57681][SQL] Support dynamic table options for UPDATE

38ea4fa

Address review comments

ff7e484

anuragmantri force-pushed the delete-update-sql-options branch from 50072c3 to ff7e484 Compare July 7, 2026 01:58

anuragmantri commented Jul 7, 2026

View reviewed changes

dongjoon-hyun approved these changes Jul 7, 2026

View reviewed changes

peter-toth approved these changes Jul 7, 2026

View reviewed changes

dongjoon-hyun closed this in 6b9d033 Jul 8, 2026

	case CatalogAndIdentifier(catalog, ident) =>
	val key = toCacheKey(catalog, ident, finalTimeTravelSpec)
	val planId = u.getTagValue(LogicalPlan.PLAN_ID_TAG)
	relationCache
	.get(key)

	\| INSERT (WITH SCHEMA EVOLUTION)? INTO TABLE? identifierReference tableAlias optionsClause? (BY NAME)?
	REPLACE (WHERE \| ON) replaceCondition=booleanExpression #insertIntoReplaceBooleanCond
	\| INSERT (WITH SCHEMA EVOLUTION)? INTO TABLE? identifierReference tableAlias optionsClause? (BY NAME)?
	REPLACE USING identifierList #insertIntoReplaceUsing

Uh oh!

Conversation

anuragmantri commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

anuragmantri commented Jun 25, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anuragmantri Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anuragmantri commented Jun 30, 2026

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

anuragmantri commented Jul 1, 2026

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anuragmantri left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Jul 7, 2026

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Jul 8, 2026

Uh oh!

dongjoon-hyun commented Jul 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

anuragmantri commented Jun 25, 2026 •

edited

Loading

anuragmantri Jun 27, 2026 •

edited

Loading

dongjoon-hyun commented Jul 1, 2026 •

edited

Loading

anuragmantri left a comment •

edited

Loading