feat: implement gossipsub partial messages extension #577

jxs · 2025-09-16T18:47:13Z

Description

This is a draft implementation of partial messages for gossipsub following the spec PR and based on the Go implementation. Still WIP but should give a good idea of the direction we're heading.

…sipsub-partial-messages

MarcoPolo · 2025-09-16T21:33:16Z

protocols/gossipsub/src/behaviour.rs

+        {
+            // Return err if trying to publish the same partial message state we currently have.
+            if existing.available_parts() == partial_message.available_parts() {
+                return Err(PublishError::Duplicate);


I don't think this is correct.

Imagine you have parts 1,2,3.

You tell your peers about those parts.

A peer comes back and says I want part 2.

You republish with the same parts in order to respond.

You get this error and fail to respond.

thanks for explaining Marco, updated as we spoke

MarcoPolo · 2025-09-16T21:33:54Z

protocols/gossipsub/src/behaviour.rs

    data_transform: D,

+    /// Partial messages received.
+    partial_messages: HashMap<TopicHash, HashMap<Vec<u8>, P>>,


Do you need to store P here? I think it's better if P is owned solely by the application.

yup, you are right, thanks Marco!

MarcoPolo · 2025-09-16T21:35:51Z

protocols/gossipsub/src/types.rs

+pub(crate) struct PartialData {
+    pub(crate) ihave: Vec<u8>,
+    pub(crate) iwant: Vec<u8>,
+    pub(crate) message: Vec<u8>,


Is it useful to store the message here? It seems like wasted space, you only use it to check if the peer is sending you a duplicate.

Might be simpler to let the application handle dupes.

thanks Marco, I only left wanted and has wanted to avoid sending the same message and has to avoid notifying the application layer of duplicates.
Thanks!

Pull-Request: libp2p#6164.

and create a paralel list for the partial_only topics.

Pull-Request: libp2p#6171.

dknopik · 2025-10-14T08:54:02Z

protocols/gossipsub/src/partial.rs

 /// 5. Received partial data is integrated using `extend_from_encoded_partial_message()`
 /// 6. The `group_id()` ties all parts of the same logical message together
 pub trait Partial {
+    type Metadata: Metadata;


should parts_metadata return this?

After discussion, I think the way to go is to remove the assoc type and keep parts_metadata the way it is for now :)

Pull-Request: libp2p#6180.

dknopik · 2025-10-15T09:19:43Z

protocols/gossipsub/src/behaviour.rs

+    pub fn publish_partial<P: Partial>(
+        &mut self,
+        topic: impl Into<TopicHash>,
+        partial_message: P,


…-partial-messages

I noticed two issues during testing. 1. We allow our mesh to grow greater than `mesh_n_high`, intentionally 2. There is a potential underflow in the heartbeat that can cause a panic For the first 1. This looks like its intentional but I can't recall why we would have added it. I think its counter-intuitive to allow our mesh to grow larger than the specified parameter. I suspect we added it to prevent our mesh from being filled with inbound peers and potentially being eclipsed. I suspect the best approach here is to remove inbound peers in the mesh maintenance rather than exceeding the mesh_n_high configuration. For 2. There is an underflow which this PR prevents. It can be triggered for low mesh_n_high values, i.e 0. This shouldn't be a concern for regular users, but we shouldn't have code that can panic based on user configuration. Pull-Request: libp2p#6183.

Pull-Request: libp2p#6181.

Pull-Request: libp2p#6178.

Adds links to related tech mentioned in examples/README. This is useful to people new to the field, like me. Pull-Request: libp2p#6016.

Split off from libp2p#6183, to quote: >I noticed two issues during testing. > >We allow our mesh to grow greater than mesh_n_high, intentionally >This looks like its intentional but I can't recall why we would have added it. I think its counter-intuitive to allow our mesh to grow larger than the specified parameter. I suspect we added it to prevent our mesh from being filled with inbound peers and potentially being eclipsed. I suspect the best approach here is to remove inbound peers in the mesh maintenance rather than exceeding the mesh_n_high configuration. Pull-Request: libp2p#6184.

ackintosh · 2025-10-25T22:43:17Z

protocols/gossipsub/src/behaviour.rs

            .connected_peers
            .iter()
-            .filter(|(_, p)| p.topics.contains(&topic_hash))
+            .filter(|(_, peer)| {


Additionally, I think we need a filter like this here:

.filter(|(_, p)| p.topics.contains(&topic_hash))

ackintosh · 2025-10-26T06:17:45Z

protocols/gossipsub/src/behaviour.rs

 /// Helper function to get a subset of random gossipsub peers for a `topic_hash`
 /// filtered by the function `f`. The number of peers to get equals the output of `n_map`
 /// that gets as input the number of filtered peers.
+#[allow(unused, reason = "partial is used with partial_messages feature")]


Suggested change

#[allow(unused, reason = "partial is used with partial_messages feature")]

#[allow(unused, reason = "exclude_partial is used with partial_messages feature")]

ackintosh · 2025-10-26T06:18:04Z

protocols/gossipsub/src/behaviour.rs


 /// Helper function to get a set of `n` random gossipsub peers for a `topic_hash`
 /// filtered by the function `f`.
+#[allow(unused, reason = "partial is used with partial_messages feature")]


Suggested change

#[allow(unused, reason = "partial is used with partial_messages feature")]

#[allow(unused, reason = "exclude_partial is used with partial_messages feature")]

ackintosh · 2025-10-26T06:28:32Z

protocols/gossipsub/src/behaviour.rs

    let mut gossip_peers = connected_peers
        .iter()
-        .filter(|(_, p)| p.topics.contains(topic_hash))
+        .filter_map(|(peer_id, peer)| {


Additionally, I think we need a filter like this here:

.filter(|(_, p)| p.topics.contains(&topic_hash))

@RolandSherwin

## Description Changelog was incorrect as we didn't release version `0.5.1` cc @RolandSherwin

jxs added 8 commits September 9, 2025 10:48

feat(gossipsub): implement gossipsub 1.3

6313068

implement test extesion

53cc7e8

Add PartialMessage trait implementation

4249ccb

implement sending and receiving partial messages at te protocol level

9aa7ead

add partial_publish method

bc45d3e

update trait extend method to receive slice instead

8b1c086

implement Partial message handling and publishing at the behaviour level

b1b8dbe

Merge branch 'sigp-gossipsub' of github.com:sigp/rust-libp2p into gos…

e7250f5

…sipsub-partial-messages

MarcoPolo reviewed Sep 16, 2025

View reviewed changes

dependabot bot and others added 2 commits September 18, 2025 08:11

deps: bump Swatinem/rust-cache from 2.8.0 to 2.8.1

b433556

Pull-Request: libp2p#6164.

address Marco's review

6370514

jxs force-pushed the gossipsub-partial-messages branch 5 times, most recently from cb0e925 to e6f1ae4 Compare September 26, 2025 11:35

jxs added 3 commits September 26, 2025 15:23

feature gate partial messages

909a68f

remove PartialIDONWANT from proto to follow spec

046ad47

update trait return types and remove non used impl

69c2d95

jxs force-pushed the gossipsub-partial-messages branch from e6f1ae4 to 69c2d95 Compare September 26, 2025 14:24

jxs and others added 10 commits September 26, 2025 15:29

fixup! feature gate partial messages

45c2bc1

update to spec changes

85b92f3

update extension message priority to highest

b6478d5

add missing proto field SubOpts partial

5e98f1a

implement subscription options partial

f77490c

publish only full messages to partial false peers

1bfbc55

revert to only using TopicHash,

33783bd

and create a paralel list for the partial_only topics.

add partial option to subscribe and unsubscribe methods

438a297

chore(ci): remove usage of self-hosted runners (libp2p#6170)

e1929e0

chore: Address latest clippy lints

9d2eb00

Pull-Request: libp2p#6171.

dknopik reviewed Oct 14, 2025

View reviewed changes

jxs added 3 commits October 14, 2025 11:24

address Daniel's review

4ea2e45

address missing review comment

ea73a71

fix(libp2p): delegate metrics feature to gossipsub when enabled

0e52451

Pull-Request: libp2p#6180.

dknopik reviewed Oct 15, 2025

View reviewed changes

jxs and others added 4 commits October 15, 2025 10:24

impl Daniel's suggestions for the public API

c415f37

borrow metadata mutably for metadata.update

3048649

impl penalization for failed partials

1685aae

address Marco review

69cde69

jxs changed the title ~~Gossipsub partial messages~~ feat: implement gossipsub partial messages extension Oct 15, 2025

jxs and others added 8 commits October 15, 2025 11:56

cargo fmt

19e359d

Merge branch 'master' of github.com:libp2p/rust-libp2p into gossipsub…

22a7b74

…-partial-messages

chore(dpes): bump prometheus-client

0a81a74

Pull-Request: libp2p#6181.

chore(deps): bump quic-go/quic-go from 0.50.1 to 0.54.1 in /wasm-tests

6b3c57a

Pull-Request: libp2p#6178.

chore(README.md): Add Links to mentioned tech

dd3fb5d

Adds links to related tech mentioned in examples/README. This is useful to people new to the field, like me. Pull-Request: libp2p#6016.

Require Metadata to be Send + Sync

54228bf

ackintosh reviewed Oct 25, 2025

View reviewed changes

ackintosh reviewed Oct 26, 2025

View reviewed changes

chore: prepare libp2p-upnp v0.6.0 (libp2p#6188)

61d436a

## Description Changelog was incorrect as we didn't release version `0.5.1` cc @RolandSherwin

jxs force-pushed the gossipsub-partial-messages branch from 9efdb00 to f728315 Compare October 28, 2025 09:55

merge master@origin

2220ae4

jxs force-pushed the gossipsub-partial-messages branch 3 times, most recently from 58d211b to c256ac6 Compare October 29, 2025 14:13

fix sending partial messages to peers not supporting them

9545140

jxs force-pushed the gossipsub-partial-messages branch from c256ac6 to 9545140 Compare October 29, 2025 14:32

jxs added 2 commits October 30, 2025 12:53

update to latest spec changes

dd21893

emmit gosipsup event whenever partial data was updated

d97ae1f

	#[allow(unused, reason = "partial is used with partial_messages feature")]
	#[allow(unused, reason = "exclude_partial is used with partial_messages feature")]

feat: implement gossipsub partial messages extension #577

Are you sure you want to change the base?

feat: implement gossipsub partial messages extension #577

Uh oh!

Conversation

jxs commented Sep 16, 2025

Description

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dknopik Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

dknopik Oct 14, 2025 •

edited

Loading