repository: add PackWriter and two-phase chunk index update

[mr-raj12]

Description

PackWriter buffers (chunk_id, cdata) pairs and flushes them as a pack file via borgstore once max_count chunks accumulate. At N=1 (max_count=1), pack_id == chunk_id and pack files land at packs/{chunk_id_hex}. No changes needed to get() or delete(). UNKNOWN_INT32 = 0xFFFFFFFF is the sentinel for pack location fields that are not yet written. 0xFFFFFFFF is above MAX_DATA_SIZE (~20 MB), so it can never collide with a real obj_offset. chunks.add() writes the placeholder; update_pack_info() fills in real values after flush().

flush() clears _pieces in a try/finally. Without this, a store failure would leave the chunk in the buffer and it would get re-bundled with the next chunk, pushing the N>1 code path and writing under a hash-derived key instead of the chunk's own id.

Changes:

• repository.py: add PackWriter; put() delegates to
  _pack_writer.add() and returns pack results.
• constants.py: add UNKNOWN_INT32 = 0xFFFFFFFF.
• hashindex.pyx: add() uses UNKNOWN_INT32 placeholders; new
  update_pack_info().
• hashindex.pyi: type stub for update_pack_info().
• cache.py: add_chunk() calls update_pack_info() from pack results.
• archive.py: add_reference() in rebuild_archives() does the same.

refs #8572

Checklist

• PR is against master
• New code has tests and docs where appropriate
• Tests pass
• Commit messages are clean and reference related issues

[codecov]

Codecov Report

❌ Patch coverage is 95.65217% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.78%. Comparing base (a1e8e53) to head (0395cc1).
⚠️ Report is 7 commits behind head on master.

Files with missing lines	Patch %	Lines
src/borg/repository.py	94.59%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #9723      +/-   ##
==========================================
+ Coverage   84.72%   84.78%   +0.05%     
==========================================
  Files          92       92              
  Lines       15007    15047      +40     
  Branches     2243     2250       +7     
==========================================
+ Hits        12715    12757      +42     
+ Misses       1592     1590       -2     
  Partials      700      700              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

[ThomasWaldmann]

Looks good overall. Some small optimizations could be done.

Later, when introducing a size limit, it will get a bit more complicated, if we want to absolutely obey that limit and not possibly exceed it by a maximum chunksize in the worst case.

Update: thinking about it, we could also just accept that it is no strict limit. Simpler code. E.g. when setting 50MB as limit, it could be also 70MB.

[ThomasWaldmann]

range-load using the pack values from index within this PR or in next one?

[mr-raj12]

range-load using the pack values from index within this PR or in next one?

in the next one

[ThomasWaldmann]

OK, so please finish this one.

For the next one:

Guess that will be a small and simple PR, just use the obj_offset and obj_size from the index to do a range-load. As that is always 0 and filesize right now, nothing should break.

Idea for the next one after that:

As you now update the index after you can know the sha256 pack_id, add a env var (SHA256_PACK_ID=1 or so) to disable the pack_id == chunk_id hack and use the real sha256 pack_id from the index. Still stay at max_count=1.

Likely, that will show a lot of problems in the existing code, pointing to all the places that now need to use the index to get the pack_id / work based on packs. Make a priority list of what needs fixing.

The CI could get 1 informative but otherwise ignored job that runs the tests with that env var, so we can see how less and less tests fail while you fix more and more stuff.

[ThomasWaldmann]

LGTM.

Guess the real test will come when actually using the indexed pack-related values.