Spark: Add session-level split size override by gerashegalov · Pull Request #16154 · apache/iceberg

gerashegalov · 2026-04-29T08:51:42Z

What changes were made in this PR?

Add a Spark 4.1 session configuration key, spark.sql.iceberg.read.split-size, that allows overriding the read.split.target-size table property at the session level without requiring DDL changes to table metadata or source code changes to read call sites.

This is useful when GPU and CPU workloads read the same Iceberg table concurrently: GPU sessions benefit from significantly larger splits while CPU sessions perform better with the default 128MB. Hardware accelerators like RAPIDS Accelerator for Apache Spark are designed as drop-in replacements requiring no application code changes, so a session-level knob is useful.

Changes

Spark 4.1 only:

SparkSQLProperties: add READ_SPLIT_SIZE = "spark.sql.iceberg.read.split-size".
SparkReadConf: add .sessionConf(SparkSQLProperties.READ_SPLIT_SIZE) to both splitSize() and splitSizeOption() parser chains, and document the updated precedence.
Docs: add the new session configuration key.
Tests: cover global session config, camelCase session config, read-option precedence, and the adaptive split-size interaction.

This PR intentionally does not add a table-scoped session key. Per-read overrides continue to use the existing split-size read option; table-scoped session behavior can be considered separately if needed.

Resolution precedence

Read option (split-size)
Session conf (spark.sql.iceberg.read.split-size)
Table property (read.split.target-size)
Default (128MB)

For split size, session overrides are treated like explicit read options and disable adaptive split-size adjustment.

How was this patch tested?

Verified locally against current apache/main (20bf72be):

./gradlew :iceberg-spark:iceberg-spark-4.1_2.13:test --tests org.apache.iceberg.spark.TestSparkReadConf --tests org.apache.iceberg.spark.source.TestSparkScan
./gradlew :iceberg-spark:iceberg-spark-4.1_2.13:spotlessCheck
./gradlew :iceberg-spark:iceberg-spark-4.1_2.13:checkstyleMain :iceberg-spark:iceberg-spark-4.1_2.13:checkstyleTest
git diff --check

Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

… configurations - Removed the session configuration for split size from SparkReadConf and SparkSQLProperties. - Updated SparkReadConf documentation to clarify the precedence of table-scoped session configurations over global settings. - Added tests to verify that table-scoped session configurations take precedence over global configurations and that options take precedence over table-scoped configurations. Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

…ation for scan planning

…sing

# Conflicts: # spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/SparkConfParser.java # spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/SparkReadConf.java # spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/SparkSQLProperties.java

wombatu-kun · 2026-06-05T03:57:58Z

+    return confParser
+        .longConf()
+        .option(SparkReadOptions.SPLIT_SIZE)
+        .sessionConf(SparkSQLProperties.SPLIT_SIZE)


splitSizeOption() is the gate term in SparkScan.java:370 (splitSizeOption() == null && adaptiveSplitSizeEnabled()), so adding SPLIT_SIZE here makes setting the session conf disable adaptive split sizing, which defaults to on. The table-property path leaves splitSizeOption() null and keeps adaptive sizing enabled, so the same split-size value behaves differently depending on whether it comes from this session conf or read.split.target-size. If that is intended (honor an explicit session split size exactly, as the read option does), document it in the precedence javadoc and add a test for the interaction; otherwise the session conf should not suppress adaptive sizing.

wombatu-kun · 2026-06-05T03:59:56Z

  public static final String ADVISORY_PARTITION_SIZE = "spark.sql.iceberg.advisory-partition-size";

+  // Overrides the split target size for scan planning
+  public static final String SPLIT_SIZE = "spark.sql.iceberg.split-size";


This adds a user-facing session conf but nothing in docs/docs/spark-configuration.md, where every other spark.sql.iceberg.* conf is listed (the adaptive-split confs from #16088 are at lines 210-211). Add spark.sql.iceberg.split-size, and the table-scoped spark.sql.iceberg.split-size.<table-name> form, to that table.

wombatu-kun · 2026-06-05T04:11:42Z

Context for reviewers - this capability has prior history worth reconciling before a deep review.

The same feature was proposed twice before and closed without merging (#10336 and its revival #13677). On #10336, @szehon-ho objected to a bare global spark.sql.iceberg.split-size key on design grounds: "The problem with this is that it does lead to some ambiguity as to what table the config is applying to (many queries read from several tables, for example)" (#10336 (comment)). He pointed at per-scan SQL OPTIONS on the Spark side as the alternative (apache/spark#46707) and later noted it had become possible there (#10336 (comment)).

This PR's table-scoped key (spark.sql.iceberg.split-size.<table-name>) directly answers that multi-table ambiguity. The global tier (spark.sql.iceberg.split-size) it also adds reintroduces the same ambiguity for queries reading several tables, so a maintainer call on whether the global tier should stay would help before deeper review.

There is also a competing open PR #16185 proposing the same capability with different naming (spark.sql.iceberg.split.target-size, plus lookback / open-file-cost / adaptive knobs, mirroring the table-property names). It is currently stale. Worth deciding which naming and scope to converge on before either is reviewed in depth.

Two related notes for whoever picks this up:

whether per-scan SQL OPTIONS (now available in Spark) already cover the original use case is worth re-confirming.
the resolution change here is generic - it adds the .<table-name> table-scoped suffix to every spark.sql.iceberg.* session conf, not only split-size.

wombatu-kun · 2026-06-05T04:14:12Z

+          }
+
+          String sparkTableSessionConfValue =
+              sessionConf.get(toCamelCase(tableSessionConfName), null);


toCamelCase is applied to the full composed key including the table.name() suffix, so any hyphen in the catalog/database/table name (legal via backticks) gets folded into camelCase and this fallback can never match the key a user would set. The primary kebab lookup at line 286 uses the raw name and is fine. If the camelCase variant is only meant to cover the conf-name part, camel-case just that: sessionConf.get(toCamelCase(sessionConfName) + "." + tableName, null).

gerashegalov · 2026-06-10T20:03:24Z

Thanks for reviewing @wombatu-kun , especially for the context.

Having a discussion about #16153 with the community is the most important part. I am open to alternative user-friendly solutions that do not involve code change on the application side or fine tuning individual tables of which it could be hundreds or even thousands.

liucao-dd · 2026-06-11T20:11:11Z

Hi folks, author of #16185 here - leaving that one as closed and +1 on the general feature added on this PR. Given SQL statements such as MERGE INTO, UPDATE, and DELETE do not expose DataFrame read options, and changing table properties affects all readers, having a session-level override would help standard use cases as well, not only GPU users.

I do have other thoughts on naming / interface implementation detail, will leave separate comments

liucao-dd · 2026-06-11T20:14:59Z

  public static final String ADVISORY_PARTITION_SIZE = "spark.sql.iceberg.advisory-partition-size";

+  // Overrides the split target size for scan planning
+  public static final String SPLIT_SIZE = "spark.sql.iceberg.split-size";


Should this be spark.sql.iceberg.read.split-size instead?

The closest merged precedent is #16088, which added the split-planning session configs under the read namespace:

spark.sql.iceberg.read.adaptive-split-size.enabled

spark.sql.iceberg.read.adaptive-split-size.parallelism

Since this is the fixed split-size counterpart to adaptive split sizing, keeping it under spark.sql.iceberg.read.* seems more consistent

liucao-dd · 2026-06-11T20:19:35Z

 | spark.sql.iceberg.delete-planning-mode                 | AUTO                                                           | Scan planning mode for delete files (`AUTO`, `LOCAL`, `DISTRIBUTED`)                                                            |
 | spark.sql.iceberg.advisory-partition-size              | Table default                                                  | Advisory size (bytes) used for writing to the Table when Spark's Adaptive Query Execution is enabled. Used to size output files |
+| spark.sql.iceberg.split-size                          | Table default                                                  | Overrides `read.split.target-size` for scan planning. Session values are honored like read options and disable adaptive split-size adjustment |
+| spark.sql.iceberg.split-size.&lt;table-name&gt;       | Global session default                                         | Table-scoped split size override using the fully qualified table name as a suffix                                                |


Could we consider making table-scoped session configs a generic identity-first pattern instead of a split-size-specific suffix? Having the table name at the end of the config seems confusing.

An alternative is to resolve table-scoped session configs as:

spark.sql.iceberg.<catalog>.<namespace...>.<table>.<setting-suffix>

For this config, if the global key becomes spark.sql.iceberg.read.split-size, the table-scoped key would be:

spark.sql.iceberg.<catalog>.<namespace...>.<table>.read.split-size

That keeps the table identity together, then applies the same setting suffix used by the global session config, making it a generic pattern to support table specific session config.

It would be a slightly larger change admittedly, and i'm happy to help or open a separate PR just to add the table level generic session config support if needed.

I'm not super excited about allowing for table level defaults through the session parameters. Feels like we are working around an issue (the inability to set config per identifier). Are we ok with just allowing the global default here?

The alternative proposed here means we are essentially adding a completely new paradigm to parsing these config entries.

RussellSpitzer

In general when we do modifications to Spark code we only do the latest Spark version first, so review can happen on a single copy of the changes before moving them back in a subsequent PR to other versions.

My larger comment here is that this PR adds the session level split override but also introduces a new mechanism for table scoping session properties. I would put that off for a separate PR because this is a much larger capability that would apply to all sessionConf properties (not just the one being added here) and we should probably have a bit more of a general review on that.

RussellSpitzer · 2026-06-23T15:53:41Z

Ah I missed - #16154 (comment) which has this same commentary.

RussellSpitzer · 2026-06-23T15:59:59Z

@anuragmantri Could you check this out, you are more deeply involved in the Spark code these days than me

anuragmantri · 2026-06-24T04:53:23Z

I just took a look at the PR and all the historical context. The motivation is clear to me. I echo @RussellSpitzer and @wombatu-kun's comments.

The global session conf is the right starting point. Every existing Iceberg session conf (vectorization.enabled, data-planning-mode, advisory-partition-size, aggregate-push-down.enabled) is a global key. This follows that pattern exactly and is easy to reason about.

I'd also suggest removing the table-scoped mechanism from this PR. A few reasons:

Spark itself has no per-table session config pattern anywhere. The closest analog is spark.sql.catalog..* for catalog plugin, and that uses prefix-based extraction (iterate all configs, regex-match the prefix, strip it). There is no suffix approach.
Every sessionConf() caller in SparkConfParser now silently supports table-scoping. Which is a larger behavior change that needs more discussion as the earlier reviewers pointed out.
Table.name() format varies across catalog implementations. The user must know exactly what string their catalog returns to construct the session key, which is fragile.

I know this design was rejected earlier citing table ambiguity, but since then Spark has added SQL options in SELECT (apache/spark#46707) and INSERT (apache/spark#47591). I think we should add the support for MERGE, UPDATE and DELETE statements as well in Spark. I can attempt to make these changes but I believe that will be the cleanest way to do this.

Also, I'm not a GPU programmer but I'm curious, isn't having a uniform split size for all the tables involved in a session be better for GPU bandwidth utilization?

szehon-ho

my thought can go here as well: #16153 (comment)

gerashegalov · 2026-07-08T21:07:35Z

Thank you for reviews @anuragmantri @liucao-dd @RussellSpitzer @szehon-ho @wombatu-kun

Updated the branch to address the review feedback:

Removed the table-scoped session configuration mechanism from this PR.
Narrowed the code changes to Spark 4.1 only.
Renamed the session key to spark.sql.iceberg.read.split-size to align with the existing read/adaptive split-size configs.
Kept docs and tests explicit that a session split-size override behaves like a read option and disables adaptive split-size adjustment.

Backports or table-scoped behavior can be handled separately if needed.

…preserve-remote

gerashegalov · 2026-07-08T23:43:14Z

Also, I'm not a GPU programmer but I'm curious, isn't having a uniform split size for all the tables involved in a session be better for GPU bandwidth utilization

My starting point was actually a single global default size. I don't see per-table tuning as a GPU-specific concern. It is like with the CPU the ideal split size depends on compression ratios, table size (e.g. dim vs fact in a join). The GPU specific part is that you often want it much bigger relative to the CPU setting.

gerashegalov added 6 commits April 10, 2026 11:53

Iceberg reads split size as a conf

c852e48

Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

Merge branch 'split-size-conf' into split-size-conf-main

9fa325a

Enhance SparkSQLProperties and SparkReadConf: Add SPLIT_SIZE configur…

e188468

…ation for scan planning

Update SparkReadConf to include sessionConf for SPLIT_SIZE option par…

3cb8a8c

…sing

Update SparkReadConf to include sessionConf for SPLIT_SIZE option par…

9713856

…sing

github-actions Bot added the spark label Apr 29, 2026

gerashegalov added 6 commits April 29, 2026 02:10

formatting

dab99e5

Spark 4.1: Add split-size session config

b31e96a

Spark: Fix split-size Javadoc example

05a865a

Trigger CI

6cbf2cf

Trigger CI again

842723a

wombatu-kun reviewed Jun 5, 2026

View reviewed changes

Spark: Address split-size session conf review comments

a1f0947

github-actions Bot added the docs label Jun 10, 2026

liucao-dd mentioned this pull request Jun 11, 2026

Spark: Add session-level read split configs #16185

Closed

liucao-dd reviewed Jun 11, 2026

View reviewed changes

RussellSpitzer reviewed Jun 23, 2026

View reviewed changes

anuragmantri mentioned this pull request Jun 25, 2026

[SPARK-57681][SQL] Support dynamic table options for UPDATE apache/spark#56792

Closed

szehon-ho reviewed Jun 27, 2026

View reviewed changes

Spark: Narrow split-size session conf scope

2b58c60

Merge remote-tracking branch 'origin/main' into split-size-conf-main-…

28f1f66

…preserve-remote

Uh oh!

Conversation

gerashegalov commented Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were made in this PR?

Changes

Resolution precedence

How was this patch tested?

Uh oh!

wombatu-kun Jun 5, 2026

Choose a reason for hiding this comment

Uh oh!

wombatu-kun Jun 5, 2026

Choose a reason for hiding this comment

Uh oh!

wombatu-kun commented Jun 5, 2026

Uh oh!

wombatu-kun Jun 5, 2026

Choose a reason for hiding this comment

Uh oh!

gerashegalov commented Jun 10, 2026

Uh oh!

liucao-dd commented Jun 11, 2026

Uh oh!

liucao-dd Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liucao-dd Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RussellSpitzer Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RussellSpitzer left a comment

Choose a reason for hiding this comment

Uh oh!

RussellSpitzer commented Jun 23, 2026

Uh oh!

RussellSpitzer commented Jun 23, 2026

Uh oh!

anuragmantri commented Jun 24, 2026

Uh oh!

szehon-ho left a comment

Choose a reason for hiding this comment

Uh oh!

gerashegalov commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gerashegalov commented Jul 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

gerashegalov commented Apr 29, 2026 •

edited

Loading

liucao-dd Jun 11, 2026 •

edited

Loading

liucao-dd Jun 11, 2026 •

edited

Loading

RussellSpitzer Jun 23, 2026 •

edited

Loading

gerashegalov commented Jul 8, 2026 •

edited

Loading