Skip to content

Conversation

@the-mikedavis
Copy link
Collaborator

Previously the READ_AHEAD_LIMIT was a constant fixed at 4096. For some usage patterns setting the limit higher helps consumer throughput when the chunk size is medium/small - specifically streams published at medium throughput being consumed over a TLS connection.

See rabbitmq/rabbitmq-server#14877

I will make a follow-up in the server to add some Cuttlefish config for this.

@the-mikedavis the-mikedavis self-assigned this Nov 3, 2025
@the-mikedavis
Copy link
Collaborator Author

the-mikedavis commented Nov 3, 2025

Maybe it makes more sense for this to be passed in the config map, so it would be consistent with the read_ahead option?

Edit: yeah that would be nicer so we could pass in values like "32KB" in config and all of that parsing logic lives in Rabbit.

@the-mikedavis the-mikedavis changed the title osiris_log: Read read-ahead limit from application environment osiris_log: Add reader config option for read-ahead limit Nov 3, 2025
Previously the `READ_AHEAD_LIMIT` was a constant fixed at 4096. For some
usage patterns setting the limit higher helps consumer throughput when
the chunk size is medium/small - specifically streams published at
medium throughput being consumed over a TLS connection.
@lukebakken lukebakken self-requested a review November 3, 2025 20:50
@michaelklishin michaelklishin changed the title osiris_log: Add reader config option for read-ahead limit Optimization: osiris_log: add reader config option for read-ahead limit Nov 3, 2025
@michaelklishin michaelklishin added this to the 1.10.1 milestone Nov 3, 2025
@michaelklishin michaelklishin merged commit 0134cf0 into rabbitmq:main Nov 3, 2025
4 checks passed
@michaelklishin
Copy link
Contributor

Note: the default read-ahead limit is still 4096 with this PR but it now can be tweaked and eventually we might discover a more optimal default.

@kjnilsson
Copy link
Contributor

I'm thinking this would have been an opportunity to combine the on and limit fields into 1 where a limit of 0 equals no read-ahead. Unfortunrately this was merged and by the look of things tagged before I noticed there was a PR.

@the-mikedavis
Copy link
Collaborator Author

Ah yeah we could keep these both in the same option. Let me send some follow-up changes. We probably don't need to care about the semantic versioning so much in this dependency. But we could have the stream.read_ahead option in RabbitMQ accept either a boolean or a number of bytes, so it would be backwards compatible with 4.2.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants