Bootstrap server - get metadata request to support future clustering

[spetz]

In the near future, we will start working on the clustering feature, which will also require changes in SDKs.
One of the first requirements will be to allow the clients to discover the cluster nodes so that the actual leader can be resolved and connected to.

The overall idea and design are rather simple and could be already added transparently to the existing Iggy server (and its SDKs) without having the clustering in place yet.

Let's say that we could introduce a new command, e.g., GetMetadata, which would return as the response the cluster details, for example, some unique ID, and most importantly, the list of all nodes (IP address, hostname, etc.). Of course, for the time being, there'd always be just a single node (leader).

When the client connects to the Iggy server, once the connection is established and authentication is completed, it should send GetMetadata and then connect to the returned leader node to perform all the further actions, such as producing/consuming the messages and so on.

If for any reason, the leader becomes unavailable, the client should, once again, invoke GetMetadata and connect to the next leader node (of course, with clustering in place, such internal cluster updates will also be notified via particular events) - this should be done internally by the SDK (as part of the existing reconnection mechanism).

From now on, the client should accept the list of Iggy servers to connect to (not just a single server), and similarly, when using the connection string approach, the servers could be comma-separated (which is pretty much an industry standard).

By default, each Iggy server also becomes the bootstrap server (since it's going to be aware of the whole cluster and all its nodes), yet from the client perspective, it only needs to specify 2 or 3 servers (even if there are tens of nodes), as GetMetadata will return the cluster details anyway.

What we could currently implement in both server and client is the following:

• Introduce a new GetMetadata command (and response) along with the schema and (de)serialization
• Implement the handler on the server side to return the cluster details (just a list of a single node for now)
• Add a new get_metadata() command to the SDK (in case someone would like to invoke it manually)
• When connecting the client to the server, first authenticate with the valid credentials, and then invoke GetMetadata
• Pick the leader node and connect to it (if it's already connected to, ignore it - which will be the case for now)
• In case of reconnection, automatically fetch the latest metadata and connect to the new leader
• Ensure that the client accepts a list of servers to connect to (instead of a single record)

As mentioned before, this will be a fully transparent change - there's no requirement (for now) to even use GetMetadata and the described flow. Server and all the existing SDKs can function without any changes to them, since we still operate on a single node, however, I think that we could already work on it, and add it to the next release (in the coming weeks).

[zicklag]

That all sounds good to me! 🚀

[T1B0]

few questions comes to mind :

• are read to non-leader(s) allowed in this scheme ? would this be config flag returned by get_metadata() maybe ?
• do we introduce a new error / code for the case when a client attempts to write to a non-leader ?
• multi AZ scenario: will returned nodes description support of some sort of tagging ? (ex. zone = "eu-west-2")

[spetz]

few questions comes to mind :

* are read to non-leader(s) allowed in this scheme ? would this be config flag returned by get_metadata() maybe ?

* do we introduce a new error / code for the case when a client attempts to write to a non-leader ?

* multi AZ scenario: will returned nodes description support of some sort of tagging ? (ex. zone = "eu-west-2")

Valid points! I think that at this stage, it's all about adding the support for new metadata request & response and making the SDKs aware of this updated flow after establishing the connection. Once we get to the VSR clustering implementation, the response will most likely be updated with the additional fields, tags, etc.

Speaking of the reads from non-leader(s), I think that this will be allowed. Once the clustering implementation gets more and more features, we may also introduce multiple leaders (based on partitions) along with other useful ideas.

As for writing to the non-leader node, this should return an error.

[hubcio]

As a result of your discussion, I implemented first part of new command GetClusterMetadata.
It contains binary interface and serialization/deserialization of command.
#2152

Next steps:

• implement actual Rust interface so that clients can call it and get ClusterMetadata
• add CLI implementation so our users will be able to call it from command line and get info about cluster state
• add tests if overwriting cluster metadata configs via env variables works just fine

Comments are welcome!