On listeners, addresses, and ports

This change reworks how we define and manage network ports, and achieves a bunch of goals in one go, at the cost of being quite big, of course.

## Unix-socket all the things

Restate server now supports listening on unix-sockets on all services (fabric, admin, ingress, and tokio-console). But even better, we listen on *both* the inet socket and unix-socket by default. Unix sockets get automatically created under the `restate-data/*.sock` and get cleaned up on shutdown (even if not, they are cleaned up on the next start). The unix-socket support include restatectl and restate CLIs. `restatectl -s unix:restate-data/admin.sock status` and setting env variables like `RESTATE_ADMIN_URL=unix:restate-data/admin.sock` will work for `restate svc status`. You can also use `curl` to call ingress like `curl --unix-socket restate-data/ingress.sock [http://local/Counter/123/add](http://test/Counter/123/add) --silent --json 1` (note that the hostname in the URL is ignored when connecting with unix-sockets.

Listening on unix-sockets can be disabled on all ports or on certain services, with a new option `listen-mode` that can be supplied in env-variable, config file, or `restate-server --listen-mode=tcp`. Listen modes support `tcp`, `unix`, and `all (default)`.  When using `unix` we’ll only listen on unix-sockets and all advertised addresses will automatically be derived to show the unix-socket address.

As a result of this change, all unit tests now use unix-sockets, no more port conflicts with your locally running services and potentially less flaky tests on CI. I’ve updated (and simplified) the local-cluster-runner utility to make use of it. There is more room for more improvements there still.

## Let’s talk about ports

**Random Ports**

Restate can now select random ports on startup by `restate-server --use-random-ports=true` or `RESTATE_USE_RANDOM_PORTS=true`. Those are conflict-free (os-selected) and because unix-sockets are also created, users (in the future) can use restate/restatectl to by pointing them to the unix-socket until they figure the ports. We print the advertised addresses for all services on startup and with the new `restate-server --no-logo` advertiseds address will be the first thing printed on stdout.

**Socket Activation**

Another cool feature is the support for LISTEN_FD/systemd compatible file-descriptor passing for listener sockets from the parent process to restate-server. A parent process can open the tcp listeners and even the unix-socket listeners (except for fabric port) and pass the file descriptors to restate-server (i.e. via systemd socket activation, or a utility like `systemfd` ).

for instance `systemfd --no-pid -s http::9000 -- restate-server` where `9000` becomes the ingress port. You can pass multiple ports and restate has a certain order to assign those ports.

*What does this bring to the table?*

1. Restarting restate-server without losing the socket listeners (ingress is the biggest winner), so clients will not observe connection errors during restart or upgrades.
2. Test harnesses, wrappers, or even our own tools can pre-allocate the tcp ports, and listen on them before starting restate. Those external wrappers don’t need to wait any more for restate-server to start before they try to connect to it (no connection retries needed). **This unlocks embedding restate in tests or shipping a restate-lite version that only listens on anonymous unix sockets**. In fact, we are a couple of steps away from making it possible to fully embed restate for use-cases that don’t need a server. The only small thing that’s missing is the invoker using a pre-supplied file-descriptor and/or support unix-sockets to connect to deployments.
3. Restate server will now attempt to bind on all required ports and unix-sockets very early in its startup, before starting any roles or opening the database. This reduces the downtime window, and allows us to centralize port assignment (for random) and gives us a nice place to print all addresses.

## Advertised Addresses

The PR unifies how we manage and configure advertised addresses for all services, it deprecates some of the old inconsistently named configuration keys (admin and ingress advertised addresses). But most importantly, restate-server will now attempt to detect a reasonable value for the advertised address. If the restate is listen mode is `unix` (only), it’ll now print `unix:/` advertised addresses, and if it’s tcp, it’ll try and detect the public routable IP address of the node instead of using `127.0.0.1` This makes docker deployments much nicer while maintaining to override all of them as needed. There is also a new option to override the hostname part of this address only without interfering with random ports `RESTATE_ADVERTISED_HOST=my_host.com` (or via cli, config) for global override, and it can be applied per service (`RESTATE_ADMIN__ADVERTISED_HOST=`). In fact, all new options can be overridden per service.

Additionally, all addresses and ports are now managed by a new component `AddressBook` that’s available via task-center. The address book is what powers handing off listeners down to services and it provides an interface to query all bound addresses such that we can return them in future `GetIdent` responses (not implemented yet).

A related improvement is how we configure `metadata-client` ’s `addresses`. Nodes now do not need to supply their own node advertised address in `addresses`. They only need to know about one or more of their peers but we'll now automatically include our own node if it's running a metadata server, thanks to early port binding and the `AddressBook`, this makes `addresses` field in config completely optional and for single-node setups, it's now empty by default.

This opens the door (not implemented) to adding support for `restate-server --peer=<address>` which would allow restate nodes to join a cluster and bootstrap completely by connecting to known peer address. This will let is figure its own correct advertised address, its metadata configuration without passing a configuration file.

## Misc

- New config options (global and with per-service override) `bind-port`, `bind-ip`, `advertised-host`, `use-random-ports` and `listen-mode`
- `restate-hyper-uds` crate to support using unix-socket with hyper clients, we should have a **config-gated** option to allow invoking deployments via unix-sockets too (any takers?)
- Port numbers, unix socket names, and service names are defined in a set of zero-cost types in `restate-types::net::address`
- Type-checked usage of addresses in all the codebase to denote which services they're meant to refer, this avoid confusion where a type like `AdvertisedAddress` didn't make it clear which service. You’ll see types like `AdvertisedAddress<AdminPort>` everywhere now.
- Documentation and configuration json schema express the service name and defaults according to the `ListenerPort` type parameter.

# What did we lose?

- For simplicity and to reduce confusion, unix-socket paths are not configurable anymore through `bind-address`, they will now be always created
under restate_data directory (fabric.sock, ingress.sock, admin.sock, and tokio.sock (if enabled). The socket files are deleted on process shutdown.
The benefit is that their locations are predictable for tools, users, and system operators.
`restatectl -s unix:restate_data/admin.sock status`
- Unix socket names have a limit of ~108 bytes in most unix systems, this puts a limit over the path length of restate-data, I've included a small optimization
that converts the path into relative if CWD is a prefix of the data-dir but this is not guaranteed solution. I'd say we evaluate how much
is this going to be a problem in practice and we can provide a configurable base directory for unix-sockets via config and env variable as needed. It's literally a single variable in AddressBook.

Author: AhmedSoliman

Cargo.lock

@@ -50,6 +50,7 @@ restate-encoding = { path = "crates/encoding" }
 restate-errors = { path = "crates/errors" }
 restate-fs-util = { path = "crates/fs-util" }
 restate-futures-util = { path = "crates/futures-util" }
+restate-hyper-uds = { path = "crates/hyper-uds" }
 restate-ingress-http = { path = "crates/ingress-http" }
 restate-ingress-kafka = { path = "crates/ingress-kafka" }
 restate-invoker-api = { path = "crates/invoker-api" }
@@ -190,7 +191,7 @@ rangemap = "1.5.1"
 rayon = { version = "1.10" }
 regex = { version = "1.11" }
 regress = { version = "0.10" }
-reqwest = { version = "0.12.5", default-features = false, features = [
+reqwest = { version = "0.12", default-features = false, features = [
     "json",
     "rustls-tls",
     "stream",

Cargo.toml

@@ -9,13 +9,21 @@
 // by the Apache License, Version 2.0.
 
 #![allow(clippy::async_yields_async)]
-
 //! Utilities for benchmarking the Restate runtime
+
+use std::net::SocketAddr;
+use std::time::Duration;
+
 use anyhow::anyhow;
 use futures_util::{TryFutureExt, future};
 use http::Uri;
 use http::header::CONTENT_TYPE;
 use pprof::flamegraph::Options;
+use tokio::net::TcpListener;
+use tokio::runtime::Runtime;
+use tokio::sync::oneshot;
+use tracing::warn;
+
 use restate_core::{TaskCenter, TaskCenterBuilder, TaskKind, cancellation_token, task_center};
 use restate_node::Node;
 use restate_rocksdb::RocksDbManager;
@@ -26,11 +34,8 @@ use restate_types::config::{
 };
 use restate_types::config_loader::ConfigLoaderBuilder;
 use restate_types::logs::metadata::ProviderKind;
+use restate_types::net::listener::AddressBook;
 use restate_types::retries::RetryPolicy;
-use std::time::Duration;
-use tokio::runtime::Runtime;
-use tokio::sync::oneshot;
-use tracing::warn;
 
 pub fn discover_deployment(current_thread_rt: &Runtime, address: Uri) {
     let client = reqwest::Client::builder()
@@ -102,16 +107,23 @@ pub fn spawn_restate(config: Configuration) -> task_center::Handle {
         .build()
         .expect("task_center builds")
         .into_handle();
+
     let mut prometheus = Prometheus::install(&config.common);
     restate_types::config::set_current_config(config.clone());
+
+    let mut address_book = AddressBook::new(restate_types::config::node_filepath(""));
     let live_config = Configuration::live();
 
     tc.block_on(async {
         RocksDbManager::init();
         prometheus.start_upkeep_task();
 
+        if let Err(err) = address_book.bind_from_config(&live_config.pinned()).await {
+            panic!("Failed to bind address book: {err}");
+        }
+
         TaskCenter::spawn(TaskKind::SystemBoot, "restate", async move {
-            let node = Node::create(live_config, prometheus)
+            let node = Node::create(live_config, prometheus, address_book)
                 .await
                 .expect("Restate node must build");
             node.start().await
@@ -126,9 +138,12 @@ pub fn spawn_mock_service_endpoint(task_center_handle: &task_center::Handle) {
     task_center_handle.block_on(async {
         let (running_tx, running_rx) = oneshot::channel();
         TaskCenter::spawn(TaskKind::TestRunner, "mock-service-endpoint", async move {
+            let addr: SocketAddr = ([127, 0, 0, 1], 9080).into();
+            let listener = TcpListener::bind(addr).await?;
+
             cancellation_token()
                 .run_until_cancelled(mock_service_endpoint::listener::run_listener(
-                    "127.0.0.1:9080".parse().expect("valid socket addr"),
+                    listener,
                     || {
                         let _ = running_tx.send(());
                     },

benchmarks/src/lib.rs

@@ -20,6 +20,7 @@ use serde::{Deserialize, Serialize};
 use url::Url;
 
 use restate_cli_util::OsEnv;
+use restate_types::net::address::{AdminPort, AdvertisedAddress, HttpIngressPort};
 
 use crate::app::GlobalOpts;
 
@@ -47,8 +48,8 @@ pub const EDITOR_ENV: &str = "RESTATE_EDITOR";
 pub struct CliConfig {
     pub environment_type: EnvironmentType,
 
-    pub ingress_base_url: Option<Url>,
-    pub admin_base_url: Option<Url>,
+    pub ingress_base_url: Option<AdvertisedAddress<HttpIngressPort>>,
+    pub admin_base_url: Option<AdvertisedAddress<AdminPort>>,
     pub bearer_token: Option<String>,
 
     #[cfg(feature = "cloud")]
@@ -62,8 +63,8 @@ impl CliConfig {
         Self {
             environment_type: EnvironmentType::Default,
 
-            ingress_base_url: Some(Url::parse("http://localhost:8080/").unwrap()),
-            admin_base_url: Some(Url::parse("http://localhost:9070/").unwrap()),
+            ingress_base_url: Some(AdvertisedAddress::default()),
+            admin_base_url: Some(AdvertisedAddress::default()),
             bearer_token: None,
 
             #[cfg(feature = "cloud")]
@@ -251,7 +252,7 @@ impl CliEnv {
         }
     }
 
-    pub fn ingress_base_url(&self) -> Result<&Url> {
+    pub fn ingress_base_url(&self) -> Result<&AdvertisedAddress<HttpIngressPort>> {
         match self.config.ingress_base_url.as_ref() {
             Some(ingress_base_url) => Ok(ingress_base_url),
             None => Err(anyhow!(
@@ -262,7 +263,7 @@ impl CliEnv {
         }
     }
 
-    pub fn admin_base_url(&self) -> Result<&Url> {
+    pub fn admin_base_url(&self) -> Result<&AdvertisedAddress<AdminPort>> {
         match self.config.admin_base_url.as_ref() {
             Some(admin_base_url) => Ok(admin_base_url),
             None => Err(anyhow!(
@@ -380,15 +381,15 @@ mod tests {
                 .ingress_base_url
                 .expect("ingress_base_url must be provided")
                 .to_string(),
-            "http://localhost:8080/".to_string()
+            "http://127.0.0.1:8080/".to_string()
         );
         assert_eq!(
             cli_env
                 .config
                 .admin_base_url
                 .expect("admin_base_url must be provided")
                 .to_string(),
-            "http://localhost:9070/".to_string()
+            "http://127.0.0.1:9070/".to_string()
         );
 
         // Defaults are templated over RESTATE_HOST