feat(python): add user header and origin timestamp support by jiengup · Pull Request #3613 · apache/iggy

jiengup · 2026-07-06T07:44:33Z

Add user headers to SendMessage and ReceiveMessage, expose origin timestamp, and fix Docker test infrastructure.

Which issue does this PR address?

Rationale

The Python SDK could send and receive message payloads, but it did not expose the typed user headers already carried by the underlying Rust IggyMessage. This left Python behind the Rust, Node, Go, Java, and C# SDKs for message metadata.

What changed?

Python messages can attach user headers and read them back from received messages. SendMessage accepts user_headers and an optional custom id, while ReceiveMessage exposes user_headers() and origin_timestamp().

According to the discussion below, unlike the original proposal in #3601 (which exposed only a plain dict[str, str | bytes | bool | int | float]), the binding now supports the full Rust header type surface through two exposed classes, plus a plain-scalar convenience layer:

HeaderKey / HeaderValue — typed complex enums covering every HeaderKind: Raw, String, Bool, Int8/16/32/64/128, UnsignedInt8/16/32/64/128, Float32, Float64.
UserHeaders — the mapping returned by ReceiveMessage.user_headers(). It is a real dict subclass (dict[HeaderKey, HeaderValue]), so all mapping operations work, and it adds a chainable to_scalar_dict() for the convenient scalar form: message.user_headers().to_scalar_dict().

Sending (`plain → Rust`)

Each key/value pair is converted independently, so typed and plain forms can be freely mixed in one dict (e.g. {HeaderKey.String("a"): 7, "b": HeaderValue.Bool(True), "c": "plain"}):

Plain scalars map by best-effort, lossless conversion for both keys and values (keys are no longer restricted to str):
- str → String, bytes → Raw, bool → Bool
- int → the smallest header kind that holds it exactly: non-negative → Uint8/16/32/64/128, negative → Int8/16/32/64/128; values beyond the 128-bit range raise ValueError.
- float → Float32 when the value is exactly representable as f32, otherwise Float64 (always lossless).
Explicit typed values are range-checked: integer variants are enforced by the binding at construction (OverflowError for out-of-range, e.g. HeaderValue.Int8(300)), and an explicit Float32 whose value overflows f32 (→ inf) is rejected with ValueError.

Receiving (`Rust → plain`)

Every HeaderKind maps losslessly onto a Python scalar (128-bit ints fit Python's arbitrary-precision int; f32 → f64 is exact), so:

user_headers() returns typed UserHeaders (dict[HeaderKey, HeaderValue]), or None when the message carries no user headers.
UserHeaders.to_scalar_dict() returns dict[str | bytes | bool | int | float, str | bytes | bool | int | float] — a direct, lossless conversion (no logging); it only raises ValueError on a genuine decode error of a stored field.

Header decode errors from known-but-invalid or unknown semantic kinds surface as ValueError. SendMessage(id=...) (mapped to u128) is part of the same binding update.

Minor Fix

The Python test compose setup starts the server with fresh default root credentials, binds HTTP/TCP/QUIC addresses explicitly, and uses iggy ping for the healthcheck instead of the HTTP stats endpoint.

TODO: make pre-commit and other scripts compatible with other shells (e.g. zsh); commands like mapfile don't work in some shells.

Discussion Notes

Choices from the issue discussion, updated to the final implementation:

Python supports both a plain dict API for the common case and explicit HeaderKey/HeaderValue wrapper classes; the two can be mixed per entry.
Integers are stored in the narrowest lossless kind (unsigned for non-negative, signed for negative); only values beyond ±128-bit raise ValueError.
Floats select Float32/Float64 by exact representability instead of always Float64.
Non-string keys are now supported (any plain scalar or HeaderKey), instead of being rejected.
ReceiveMessage.user_headers() returns None for no headers.
origin_timestamp() is exposed on received messages.

API Usage

Sending — plain scalars (common case)

from apache_iggy import SendMessage as Message

# Keys and values are plain Python scalars; each is converted losslessly.
Message(
    "order-created",
    user_headers={
        "content-type": "application/json",  # -> String
        "trace-blob": b"\x00\x01",           # -> Raw
        "retryable": False,                  # -> Bool
        "attempt": 3,                        # -> UnsignedInt8 (smallest fit)
        "created-at-ms": 1_700_000_000_000,  # -> UnsignedInt64
        "score": 1.25,                       # -> Float32 (exactly representable)
    },
    id=42,  # optional custom message id (u128)
)

Sending — explicit typed kinds

from apache_iggy import HeaderKey, HeaderValue
from apache_iggy import SendMessage as Message

Message(
    "order-created",
    user_headers={
        HeaderKey.String("schema-version"): HeaderValue.UnsignedInt16(1),
        HeaderKey.UnsignedInt32(7): HeaderValue.String("order-id"),  # non-string key
    },
)

Sending — mixed typed + plain in one dict

Message(
    "order-created",
    user_headers={
        HeaderKey.String("typed"): HeaderValue.Int32(-5),
        "plain-key": HeaderValue.Bool(True),
        HeaderKey.String("plain-val"): "hello",
        "both-plain": 9,
    },
)

Receiving

message = polled_messages[0]

message.origin_timestamp()          # int

headers = message.user_headers()    # UserHeaders | None
if headers is not None:
    # Typed access (full wire-type fidelity):
    value = headers.get(HeaderKey.String("schema-version"))
    if isinstance(value, HeaderValue.UnsignedInt16):
        print(value.value)          # 1

    # Convenient plain form (lossless):
    plain = headers.to_scalar_dict()      # dict[str|bytes|bool|int|float, str|bytes|bool|int|float]
    print(plain["schema-version"])  # 1

Validation / errors

Message("p", user_headers={"k": 2**128})            # ValueError: 128-bit range
Message("p", user_headers={"k": HeaderValue.Int8(300)})   # OverflowError
Message("p", user_headers={"k": HeaderValue.Float32(1e40)})  # ValueError: 32-bit float
Message("p", user_headers={object(): "v"})          # ValueError: keys must be str/bytes/bool/int/float/HeaderKey

Local Execution

Passed: cargo fmt --check --manifest-path foreign/python/Cargo.toml
Passed: cargo check --manifest-path foreign/python/Cargo.toml
Passed: cargo test --manifest-path foreign/python/Cargo.toml
Passed: uv run --extra dev ruff check tests/test_message_operations.py tests/test_consumer_group.py
Passed: .venv/bin/python -m pytest tests/test_message_operations.py::TestMessageOperations::test_invalid_user_headers_are_rejected -q
Successfully run: example/message-headers/typed-headers/consumer.py, example/message-headers/typed-headers/producer.py, example/message-headers/plain-headers/consumer.py, example/message-headers/plain-headers/producer.py

AI Usage

Codex was used to inspect the existing Python, Rust, Node, and Go SDK behavior, implement the Python binding changes, add tests. All the modification was reviewed carefully by the human.

hubcio · 2026-07-06T07:52:56Z

one additional request from my side: please update python examples to include a code that would show how to use iggy message user headers. there is one for rust already.

codecov · 2026-07-06T07:58:39Z

Codecov Report

❌ Patch coverage is 80.71066% with 114 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.25%. Comparing base (45fac44) to head (209ecd7).
⚠️ Report is 16 commits behind head on master.

Files with missing lines	Patch %	Lines
foreign/python/src/user_headers.rs	79.27%	114 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #3613      +/-   ##
============================================
- Coverage     73.47%   73.25%   -0.23%     
  Complexity      937      937              
============================================
  Files          1286     1287       +1     
  Lines        141019   141307     +288     
  Branches     116908   116658     -250     
============================================
- Hits         103619   103511     -108     
- Misses        34175    34519     +344     
- Partials       3225     3277      +52

Components	Coverage Δ
Rust Core	`73.44% <ø> (-0.27%)`	⬇️
Java SDK	`62.44% <ø> (ø)`
C# SDK	`71.40% <ø> (-0.68%)`	⬇️
Python SDK	`87.11% <80.71%> (-4.54%)`	⬇️
PHP SDK	`84.29% <ø> (ø)`
Node SDK	`91.26% <ø> (-0.10%)`	⬇️
Go SDK	`42.28% <ø> (ø)`

Files with missing lines	Coverage Δ
foreign/python/src/lib.rs	`100.00% <100.00%> (ø)`
foreign/python/src/receive_message.rs	`88.37% <100.00%> (+3.99%)`	⬆️
foreign/python/src/send_message.rs	`97.91% <100.00%> (+1.14%)`	⬆️
foreign/python/src/user_headers.rs	`79.27% <79.27%> (ø)`

... and 45 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

jiengup · 2026-07-06T08:08:03Z

one additional request from my side: please update python examples to include a code that would show how to use iggy message user headers. there is one for rust already.

That will be included when the PR is ready.

slbotbm · 2026-07-08T16:12:53Z

@hubcio @jiengup

This would allow us to have two advantages: users that do not have much experience/do not require explicit control can use dict[str, str | bytes | bool | int | float], while users that require precise control on the types of headers can use dict[HeaderKey, HeaderValue]. Also, by providing dict[HeaderKey, HeaderValue] as an option, we allow AI Agents to make less mistakes as compared to dict[str, str | bytes | bool | int | float], which is a little vague from the agents' point of view.

What do you guys think?

jiengup · 2026-07-09T02:23:46Z

change the signature of the headers from dict[str, str | bytes | bool | int | float] | None to dict[str, str | bytes | bool | int | float] | dict[HeaderKey, HeaderValue] | None

I think it's a good idea and well aligned with the first point worth discussing in my PR. I'll push a new patch soon.

hubcio · 2026-07-09T10:49:40Z

@slbotbm for me this sounds fine, however one thing comes to my mind: will it have any impact on performance?

jiengup · 2026-07-09T11:52:35Z

@slbotbm for me this sounds fine, however one thing comes to my mind: will it have any impact on performance?

Actually, I think additional typed HeaderKey support will not introduce more overhead compared to the current implementation.

The only overhead we introduce is the iteration of user_header and the transformation of Rust and Python types, which need to be considered if we have a large number of user header pairs in some user cases.

hubcio · 2026-07-09T14:14:18Z

alright, then don't worry about it now.

jiengup · 2026-07-10T10:01:52Z

@hubcio @slbotbm

Hi, I just reflected the change and merged all the commits into a final one. The PR message is also updated.

Instead of a plain-only dict[str, ...] with int → Int64 / float → Float64 and string-only keys, the binding now exposes the full typed HeaderKey/HeaderValue surface with smallest-fit kind selection, supports non-string keys, and allows typed and plain pairs to be freely mixed in one dict.

Please have a look at it when you are free.

slbotbm

first round of review

slbotbm · 2026-07-12T07:08:31Z

+    await produce_messages(client)
+
+
+async def init_system(client: IggyClient):


Current system catches stream creation and topic creation independently, logs errors, and continues with an invalid system. Unify the try/catches and exit if stream/topic is not created

I don't get this point.
The message header examples are implemented accroding to the existing getting-started example.

In common.py, in the init_system function, topic creation will be tried even if the stream creation fails. Since topics reside inside streams in iggy, it will not be created and will fail. Better to change the code so that topic should only be created if the stream exists/is created.

slbotbm · 2026-07-12T07:22:48Z

/author

Add user headers to SendMessage and ReceiveMessage, expose origin timestamp, and fix Docker test infrastructure: - explicit typed header and plain Python header support - type transformation and validation across Rust and Python - user header usage examples - minor fix: Docker test config file

to_scalar_dict Split message-headers example into plain-headers and typed-headers variants sharing logic via common.py. Rename UserHeaders.to_plain() to to_scalar_dict() to better describe the returned dict type.

jiengup · 2026-07-12T11:06:53Z

/ready

slbotbm

Some more comments:

You have set the type of user headers in some places to [Any, Any], which makes any type checker incapable of catching mistakes. Would be better to explicitly declare the types.

slbotbm · 2026-07-16T06:43:07Z

+    pub fn to_scalar_dict<'a>(slf: &Bound<'a, Self>) -> PyResult<Bound<'a, PyDict>> {
+        let py = slf.py();
+        let dict = slf.as_any().cast::<PyDict>()?;
+        let headers = py_user_headers_to_rust(py, dict)?;
+        rust_user_headers_to_plain_py(py, headers)
+    }


This function is not lossless - if the user supplies UnsignedInt8(1) and UnsignedInt16(1), both get converted to python int 1. In this case, the conversion should fail, and be documented properly.

Also, the current code converts twice before returning the headers back to the user: python typed headers -> rust headers -> python plain headers. It would be preferable to do a direct conversion python typed header -> python plain header. TryFrom can be defined and utilized.

if the user supplies UnsignedInt8(1) and UnsignedInt16(1)

I don't get this.
Lossless here means Rust type conversion into Python type should not cost any data loss (e.g. uint32 to py Int because Python can represent any large int).
You mean we should reject any coexisting UnsignedInt8(1) and UnsignedInt16(1) or we should reject UnsignedInt16(-1)? I think the former is not reasonable.

Let me think a little about this and get back to you

slbotbm · 2026-07-16T07:09:52Z

+HTTP_CREATED = 201
+JsonValue: TypeAlias = (
+    None | bool | int | float | str | list["JsonValue"] | dict[str, "JsonValue"]
+)


These are never used. Can be removed.

slbotbm · 2026-07-16T07:13:00Z

The logging setup in the tests you have defined aren't really useful.

slbotbm · 2026-07-16T07:14:01Z

+        typed_key = HeaderKey.UnsignedInt128(42)
+        typed_value = HeaderValue.UnsignedInt128(2**96)
+        user_headers: dict[HeaderKey, HeaderValue] = {
+            typed_key: typed_value,
+            HeaderKey.String("float32"): HeaderValue.Float32(1.25),
+        }


This can be made into one dict. No need to declare extras vars.

slbotbm · 2026-07-16T07:29:36Z

+    await init_system(client)
+    await produce_messages(client, build_plain_headers)


Even if init_system fails, this code will not know since init_system logs the error and returns normally.

In produce_messages, the loop termination condition depends on successful sends, but if the send fails, sent_batches is not incremented after a failed send, a persistent configuration, authorization, or initialization failure makes the example run forever. It prints repeated errors instead of terminating with a useful failure.

slbotbm · 2026-07-16T07:31:12Z

+
+
+def generate_orders() -> Iterator[Order]:
+    order_id = 0


In this function, the generated order_ids are not coherent. The current code will effectively produce the following 6 events:

OrderConfirmed(order-0) OrderRejected(order-0) OrderCreated(order-3) OrderConfirmed(order-1) OrderRejected(order-1) OrderCreated(order-6)

jiengup force-pushed the python/user-header branch from a5a6357 to 6d29d49 Compare July 6, 2026 15:23

jiengup marked this pull request as ready for review July 6, 2026 15:34

github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Jul 6, 2026

jiengup force-pushed the python/user-header branch from 0fdd454 to 8daf546 Compare July 10, 2026 09:46

jiengup force-pushed the python/user-header branch from 8daf546 to 90b153d Compare July 10, 2026 13:18

slbotbm reviewed Jul 12, 2026

View reviewed changes

github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Jul 12, 2026

jiengup added 2 commits July 12, 2026 18:52

chore(python): refactor Python header examples and rename to_plain to

209ecd7

to_scalar_dict Split message-headers example into plain-headers and typed-headers variants sharing logic via common.py. Rename UserHeaders.to_plain() to to_scalar_dict() to better describe the returned dict type.

jiengup force-pushed the python/user-header branch from 90b153d to 209ecd7 Compare July 12, 2026 10:53

github-actions Bot added S-waiting-on-review PR is waiting on a reviewer and removed S-waiting-on-author PR is waiting on author response labels Jul 12, 2026

slbotbm requested changes Jul 16, 2026

View reviewed changes

slbotbm reviewed Jul 16, 2026

View reviewed changes

github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Jul 16, 2026

slbotbm requested changes Jul 16, 2026

View reviewed changes

		await produce_messages(client)


		async def init_system(client: IggyClient):

		await init_system(client)
		await produce_messages(client, build_plain_headers)

Uh oh!

Conversation

jiengup commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR address?

Rationale

What changed?

Sending (plain → Rust)

Receiving (Rust → plain)

Minor Fix

Discussion Notes

API Usage

Sending — explicit typed kinds

Sending — mixed typed + plain in one dict

Receiving

Validation / errors

Local Execution

AI Usage

Uh oh!

hubcio commented Jul 6, 2026

Uh oh!

codecov Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jiengup commented Jul 6, 2026

Uh oh!

slbotbm commented Jul 8, 2026

Uh oh!

jiengup commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hubcio commented Jul 9, 2026

Uh oh!

jiengup commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hubcio commented Jul 9, 2026

Uh oh!

jiengup commented Jul 10, 2026

Uh oh!

slbotbm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

slbotbm commented Jul 12, 2026

Uh oh!

jiengup commented Jul 12, 2026

Uh oh!

slbotbm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jiengup commented Jul 6, 2026 •

edited

Loading

Sending (`plain → Rust`)

Receiving (`Rust → plain`)

codecov Bot commented Jul 6, 2026 •

edited

Loading

jiengup commented Jul 9, 2026 •

edited

Loading

jiengup commented Jul 9, 2026 •

edited

Loading