Different Gossipsub versions per topic (class)

[yiannisbot]

Different Gossipsub topics at the CL are carrying traffic with different characteristics. Attestations topics see thousands of very small messages per second, while the block topic is only carrying a few larger messages per epoch.

It’s becoming clear that the default configuration of Gossipsub (e.g., for message diffusion, gossip control messages and recently IDONTWANT messages) does not fit the characteristics of all topics. It has been reported, for instance, that although IDONTWANT messages were introduced to reduce the bandwidth consumption of nodes, they have actually resulted in increased bandwidth consumption, due to the vast amount of IDONTWANTs sent for the thousands of attestation messages (observed by Lighthouse, at least).

Given the “one size does not fit all” situation, it has been proposed that we could use different versions of Gossipsub, with different configuration parameters for different topics (or topic-classes).

A first high-level categorisation could look like:

• Topics with high traffic but small messages would benefit from a few changes:
  • No IDONTWANT messages would be sent in those topics and gossip messages (e.g., IHAVE , IWANT) could be sent to less remote nodes.
  • An alternative to not sending any IDONTWANT messages would be to only send IDONTWANTs for messages larger than a specific value (e.g., 1KB).
  • Increasing the heartbeat for these topics could also be something to consider, although it needs to be done with care.
  • Under the current PeerDAS plan to increase the blob count, the sampling side of the approach would greatly benefit by increasing the peer count at each node (relevant ethresear.ch post). Thus, in order to avoid excessive bandwidth requirements, one approach would be to reduce the gossip factor of these topics that have a large number of msgIDs, or reduce the number of msgIDs at each topic’s IHAVE control message.
• Topics with less traffic, but larger messages, such as the blocks topic, would likely keep the current configuration. We could explore reducing slightly the mesh size, as sending large messages to fewer concurrent peers could speed up the propagation time to the next hop. Furthermore, we could also consider increasing the heartbeat of these messages. Intuitively, a lower heartbeat value increases the number of duplicates received through IWANT replies, while the original message still propagates.

---

This issue is intended to gather feedback on whether having different Gossipsub versions per topic (class) is something that should be added to the roadmap, as well as proposals for configuration parameters per topic (class).

[Ideas in this issue have been discussed with @cortze @AgeManning @cskiraly]

[arnetheduck]

An alternative to not sending any IDONTWANT messages would be to only send IDONTWANTs for messages larger than a specific value (e.g., 1KB).

nim-libp2p already does this regardless of topic - this is a function of message size and/or message id size (ie for a large message id, the savings must be proportionally larger), not topic so it should not need to depend on the topic. it is not a given that all messages on a topic are of the same size (a block is anything from a few hundred bytes to several mb).

also, filtering by version is an odd way to look at things - it seems broadly more useful to reason about features rather than versions - what if 1.3 introduces some structure that is also useful for small messages? will we then have "1.1 and 1.3 but not 1.2" in the logic?

[arnetheduck]

Also, if some implementation is sending IDONTWANT for attestations, that implementation needs to be tracked down and fixed - that's a trivial bug, thouhg in theory the spec could give a suggested minimum IDONTWANT activation size.

Something to remember about idontwant and its savings is that it's often useful to think about tcp frames (1500 byte increments) rather than individual bytes - it largely doesn't matter for most practical purposes if you send 1 or 1500 bytes - it's more about the number of packets.

[ppopth]

An alternative to not sending any IDONTWANT messages would be to only send IDONTWANTs for messages larger than a specific value (e.g., 1KB).

nim-libp2p already does this regardless of topic

go-libp2p is already doing this as well. I'm not sure what are the pros to do it per topic.

also, filtering by version is an odd way to look at things - it seems broadly more useful to reason about features rather than versions

I agree with that. Eventually everyone should use the latest version for all topics.

[Nashatyrev]

go-libp2p is already doing this as well. I'm not sure what are the pros to do it per topic.

Same for jvm-libp2p. And it looks like this solution is good enough.

Generally speaking it's a good idea to localize some gossip parameters (not Gossip versions) to be topic-specific. This may include core parameters such as D and heartbeat. However this looks like a pretty significant change in both gossip spec and implementation so we probably need use cases strong enough to push forward the proposal

[AgeManning]

We also size limit IDONTWANT. Still waiting for more nodes to update on the network to gather some more metrics around its efficacy.

In terms of specifically different versions per topic, I think this would be hard to implement. We do multistream-select on connection to find the highest compatible version of a protocol. If we supported more than 1 version of gossipsub, our peer would just select the highest, it would be difficult to engineer our implementation to negotiate multiple versions and then align them with specific topics.

Rather than versions tho, I think it is useful to allow different gossipsub parameters per topic. As @yiannisbot mentioned, we could modify the degree, heartbeat and potentially IDONTWANT message sizes per-topic. I think this doesn't really require much of a spec change and implementers could do this locally if they wanted.

It would then be a matter of deciding what parameters suit which topics.

[yiannisbot]

Thanks for the feedback and the input everyone! It sounds like topic-specific Gossipsub parameters is an approach seen favourably.

We do multistream-select on connection to find the highest compatible version of a protocol.

I think that’s the biggest blocker here for versions.

I think this doesn't really require much of a spec change and implementers could do this locally if they wanted.

It would then be a matter of deciding what parameters suit which topics.

Unless there are strong objections, I’d say the next step is to come up with: i) topic characteristics and ii) recommended settings for the parameters per topic - happy to take that up and coordinate. Once there's agreement on that we could proceed with some testing or simulation.