Skip to content

MINOR: Close already-opened input readers when ParquetRewriter setup fails#3662

Open
anxkhn wants to merge 1 commit into
apache:masterfrom
anxkhn:patch-7
Open

MINOR: Close already-opened input readers when ParquetRewriter setup fails#3662
anxkhn wants to merge 1 commit into
apache:masterfrom
anxkhn:patch-7

Conversation

@anxkhn

@anxkhn anxkhn commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Rationale for this change

ParquetRewriter.getFileReaders opens a TransParquetFileReader for each input
file into a local list. If opening a later file throws IOException, it is
rethrown as IllegalArgumentException without closing the readers already opened
for the earlier files, so their SeekableInputStream handles leak.
ParquetRewriter.close() only ends the writer and does not close the input
readers, so nothing else releases them either. A single corrupt, missing, or
permission-denied file among several valid inputs therefore leaks up to N open
input streams per failed rewrite/merge/join setup.

What changes are included in this PR?

In getFileReaders, close the readers already opened before rethrowing, using
org.apache.parquet.util.AutoCloseables.close(...) so every reader is released
and any close failures are aggregated as suppressed exceptions on the original
IllegalArgumentException. This matches the existing cleanup idiom in the module.
The change is limited to the catch (IOException) block; the success path is
unchanged.

Are these changes tested?

Yes. A new ParquetRewriterTest case
(testInputFileReadersClosedWhenLaterInputFileFailsToOpen) supplies a valid
input file (wrapped so it records whether the stream it hands out is closed)
followed by an InputFile whose newStream() throws, asserts that constructing
the ParquetRewriter throws IllegalArgumentException, and asserts the first
reader's stream was closed. It fails on the pre-fix code (the first stream is
left open) and passes with the fix. The full ParquetRewriterTest class passes
(Tests run: 96, Failures: 0, Errors: 0).

Are there any user-facing changes?

No public API changes. The only behavioral difference is that a failed
ParquetRewriter setup no longer leaks the input streams it had already opened.

…fails

ParquetRewriter.getFileReaders opens a TransParquetFileReader for each input
file into a local list. If opening a later file throws IOException, it was
rethrown as IllegalArgumentException without closing the readers already
opened for the earlier files, leaking their SeekableInputStream handles
(ParquetRewriter.close() only ends the writer and does not close input
readers). One corrupt, missing, or permission-denied file among several
valid ones could therefore leak up to N open input streams per failed
rewrite/merge/join setup.

Close the readers already opened before rethrowing, using
AutoCloseables.close so every reader is released and any close failures are
aggregated as suppressed exceptions on the original IllegalArgumentException.

Add a ParquetRewriterTest case that supplies a valid input file followed by
one that fails to open and asserts the first reader's stream is closed.

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seams reasonable to me 👍

@wgtmac

wgtmac commented Jul 15, 2026

Copy link
Copy Markdown
Member

I would recommend creating an issue for this. Or at least do not mark it as MINOR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants