Exploring Delayed Message Support in Iggy: Use Cases & Implementation Ideas

[l-7-l]

Iggy is absolutely fantastic, delayed messages is sometimes quite an useful feature. In our e-commerce system, we currently rely on delayed messages for three critical workflows:

1. Order timeout closure (30-minute threshold)

2. Payment settlement initiation (N days post-fulfillment)

3. Scheduled product listing management (launches/takedowns)

While Iggy's performance is excellent, native delayed message support would be a valuable feature. This capability would eliminate the need for additional components, reducing technical debt.

As noted by @spetz, delayed messages serve crucial roles in certain scenarios. They proposed a potential implementation approach:

"Create dedicated topics with background jobs that move messages to destination topics upon schedule expiration. This could be treated as a special topic type – either allowing direct message pulls regardless of timing, or adding a behavioral flag."

Open questions from the proposal:

1. How might this affect consumer access patterns for standard vs. delayed topics?

We welcome architecture insights and alternative implementation approaches from the community.

[l-7-l]

I found an article about introducing qmq delayed messages:
QMQ offers delayed/scheduled messaging for any time within a configurable period (up to two years by default). You can specify the delivery time of a message at any point within this future timeframe.

I placed QMQ last because I consider its design for delayed messages to be the most rational among open-source message queues (MQ). The core design, simply put, is multi-level timing wheel + lazy loading + dedicated disk storage for delayed messages.

QMQ implements delayed/scheduled messages using a two-layer hash wheel:

Disk Layer: Uses hourly intervals as the unit (default; adjustable via configuration). Each interval generates a log file (schedule log). Since QMQ supports delayed messages up to two years (configurable), this results in a maximum of 2 * 366 * 24 = 17,568 files (fewer files are needed if the maximum delay time is shorter).

Memory Layer: When a message's delivery time approaches, the indices for messages within that hour (containing the offset and size within the schedule log) are loaded from disk into a hash wheel in memory. This in-memory hash wheel uses 500ms intervals.

Summary of Key Design Highlights:

1. The Timing Wheel algorithm is ideal for delayed/scheduled messaging. It eliminates the need for sorting delayed messages, offering O(1) time complexity for both insertion and deletion operations.

2. The multi-level timing wheel design enables support for delayed messages with extremely large time spans.

3. Lazy loading ensures only messages nearing their delivery time reside in memory. Messages scheduled further in the future are stored on disk, making the design memory efficient.

4. Storing delayed messages separately in schedule log files prevents them from interfering with the space reclamation of normal messages.

[spetz]

Thank you for the sharing the very interesting design for this feature. As discussed already on the Discord, while having the delayed messages is quite a common thing for the message queues, it's not that "popular" for the message streaming engines. I think that the initial idea, where there could be a dedicated "system" topic storing the delayed messages and then appending them (when the time comes), to the desired topic e.g. based on the message metadata/headers could work. Currently, there's a huge refactoring as we move to the io_uring runtime with thread-per-core & shared nothing design, but it's certainly the feature worth considering in the future :)