dpd, MGS APIs marked server-side-versioned, but that's not valid

[davepacheco]

See #9704.

There are a few paths here, not wholly exclusive:

• Make it client-side-versioned and fully implement RFD 567 (client-side-versioning).
• Split this API into two: one that's server-side-versioned and used by Nexus; and one that's client-side-versioned and used by Sled Agent.
  • Pro: Nexus can still treat this as server-side-versioned.
  • Pro: Clearly reflects two different interfaces with different stability.
  • Con: Duplication at the API layer (two APIs with overlapping endpoints and types and not necessarily the same types). This probably wouldn't be too bad at first, but it could get unwieldy over time.
    This would only make sense if Sled Agent likely doesn't need new API versions from Dendrite. For example, if we keep rev'ing these APIs, but can still provide a compatible "shim" for sled agent indefinitely and it doesn't need the new parts, then this would work. But if it did need the new parts (say, new required arguments), this won't help. That said, it can't work at all for sled agent to need the new parts in the same release they're shipped precisely because old sled agents still exist talking to new dpd during an online update.
• Update ls-apis (and the dropshot-api-manager? not sure if it cares) to support the idea that edges are determined to be server-side-versioned vs. client-side-versioned, rather than APIs themselves. The idea here would be that we don't need to have two different APIs, but the tool understands that Sled Agent treats it as client-side-versioned and Nexus treats it as server-side-versioned.

I'm increasingly convinced that we probably should do RFD 567, but that's also going to take time. Either of the other two options may buy us time here. But neither of those can solve the problem of actually wanting to ship changes to the Dendrite API in the same release that new Sled Agents use them. Only RFD 567 can do that.

[davepacheco]

This applies to MGS, too, per #9707 (comment).

[sunshowers]

Here's some investigation I did today on this with Claude Code's help:

Commit references:

• omicron: 6cef874
• dendrite: 606c0be
• maghemite: 396bb3c
• lldp: 61479b6
• propolis: 2aa7f9d

Summary

Server	Client	Status	Evidence
ddmd	dpd-client	verified	services.rs:3127
dpd	gateway-client	verified	switch_identifiers.rs:60
lldpd	dpd-client	verified	dendrite.rs:194
mgd	ddm-admin-client	verified	ddm.rs:110
mgd	dpd-client	verified	dendrite.rs:491
mgd	gateway-client	verified	main.rs:93
omicron-sled-agent	ddm-admin-client	verified	ddm_reconciler.rs:56
omicron-sled-agent	gateway-client	bug: not localhost	early_networking.rs:376
omicron-sled-agent	dpd-client	bug: not localhost	services.rs:1090
omicron-sled-agent	mg-admin-client	bug: not localhost	early_networking.rs:483
omicron-sled-agent	propolis-client	bug: not localhost	instance.rs:2283
tfportd	dpd-client	verified	services.rs:2852
tfportd	lldpd-client	verified	tfport.rs:88
wicketd	ddm-admin-client	verified	bootstrap_addrs.rs:162
wicketd	dpd-client	verified	uplink.rs:83
wicketd	gateway-client	verified	wicketd.rs:35

Detailed findings

ddmd -> dpd-client

ddmd is configured via SMF. On Oxide sleds, dpd_host is set to [::1]:

// https://github.com/oxidecomputer/omicron/blob/6cef874f3b/sled-agent/src/services.rs#L3125-L3132
if is_oxide_sled {
    mg_ddm_config = mg_ddm_config
        .add_property("dpd_host", "astring", "[::1]")
        .add_property("dpd_port", "astring", &DENDRITE_PORT.to_string())
}

The binary default is also localhost (ddmd/src/main.rs:72).

---

dpd -> gateway-client

dpd connects to the local MGS to fetch switch identifiers, defaulting to
[::1]:{MGS_PORT}:

// https://github.com/oxidecomputer/dendrite/blob/606c0be888/dpd/src/switch_identifiers.rs#L52-L60
// Note that we do _not_ use internal DNS to resolve this address --
// we care about our _local_ MGS only.
let url = match mgs_socketaddr {
    Some(socket_addr) => format!("http://{socket_addr}"),
    None => format!("http://[::1]:{MGS_PORT}"),
};

MGS is configured to listen on [::1] (services.rs:2606).

---

lldpd -> dpd-client

lldpd defaults to localhost for dpd. The SMF manifest doesn't override this:

// https://github.com/oxidecomputer/lldp/blob/61479b6922/lldpd/src/dendrite.rs#L187-L202
let host = opts.host.unwrap_or_else(|| "localhost".to_string());
let port = opts.port.unwrap_or(dpd_client::default_port());
let client = Client::new(&format!("http://{host}:{port}"), client_state);

---

mgd -> ddm-admin-client

mg-lower hardcodes localhost:8000:

// https://github.com/oxidecomputer/maghemite/blob/396bb3c570/mg-lower/src/ddm.rs#L109-L111
pub(crate) fn new_ddm_client(log: &Logger) -> Client {
    Client::new("http://localhost:8000", log.clone())
}

---

mgd -> dpd-client

mg-lower hardcodes localhost. The comment notes this is intentional:

// https://github.com/oxidecomputer/maghemite/blob/396bb3c570/mg-lower/src/dendrite.rs#L483-L494
/// Create a new Dendrite/dpd client. The lower half always runs on the same
/// host/zone as the underlying platform.
pub(crate) fn new_dpd_client(log: &Logger) -> DpdClient {
    DpdClient::new(
        &format!("http://localhost:{}", dpd_client::default_port()),
        client_state,
    )
}

---

mgd -> gateway-client

mgd defaults to [::1]:12225:

// https://github.com/oxidecomputer/maghemite/blob/396bb3c570/mgd/src/main.rs#L92-L94
#[arg(long, default_value = "[::1]:12225")]
mgs_addr: SocketAddr,

---

omicron-sled-agent -> ddm-admin-client

sled-agent uses Client::localhost():

// https://github.com/oxidecomputer/omicron/blob/6cef874f3b/sled-agent/src/ddm_reconciler.rs#L56
let client = Client::localhost(base_log)?;

// https://github.com/oxidecomputer/omicron/blob/6cef874f3b/clients/ddm-admin-client/src/lib.rs#L47-L49
pub fn localhost(log: &Logger) -> Result<Self, DdmError> {
    Self::new(log, SocketAddrV6::new(Ipv6Addr::LOCALHOST, DDMD_PORT, 0, 0))
}

---

omicron-sled-agent -> gateway-client (bug)

sled-agent uses the switch zone's underlay IP, not localhost:

// https://github.com/oxidecomputer/omicron/blob/6cef874f3b/sled-agent/src/bootstrap/early_networking.rs#L376-L378
let mgs_client = MgsClient::new(
    &format!("http://[{}]:{}", switch_zone_underlay_ip, MGS_PORT),
    self.log.new(o!("component" => "MgsClient")),
);

sled-agent runs in the global zone; MGS runs in the switch zone.

---

omicron-sled-agent -> dpd-client (bug)

sled-agent uses the switch zone's underlay IP, not localhost:

// https://github.com/oxidecomputer/omicron/blob/6cef874f3b/sled-agent/src/services.rs#L1090-L1102
let dpd_clients: Vec<DpdClient> = uplinked_switch_zone_addrs
    .iter()
    .map(|addr| {
        DpdClient::new(&format!("http://[{}]:{}", addr, DENDRITE_PORT), ...)
    })
    .collect();

---

omicron-sled-agent -> mg-admin-client (bug)

sled-agent uses the switch zone's underlay IP, not localhost:

// https://github.com/oxidecomputer/omicron/blob/6cef874f3b/sled-agent/src/bootstrap/early_networking.rs#L483-L489
let mgd = MgdClient::new(
    &format!("http://{}", &SocketAddrV6::new(switch_zone_underlay_ip, MGD_PORT, 0, 0)),
    self.log.clone(),
);

---

omicron-sled-agent -> propolis-client (bug)

sled-agent uses the propolis zone's IP, not localhost:

// https://github.com/oxidecomputer/omicron/blob/6cef874f3b/sled-agent/src/instance.rs#L2283-L2286
let client = Arc::new(PropolisClient::new_with_client(
    &format!("http://{}", &self.propolis_addr),
    reqwest_client,
));

Propolis instances run in their own zones.

---

tfportd -> dpd-client

tfportd is configured via SMF to use [::1]:

// https://github.com/oxidecomputer/omicron/blob/6cef874f3b/sled-agent/src/services.rs#L2850-L2860
tfport_config = tfport_config
    .add_property("dpd_host", "astring", &format!("[{}]", Ipv6Addr::LOCALHOST))
    .add_property("dpd_port", "astring", &format!("{}", DENDRITE_PORT));

---

tfportd -> lldpd-client

tfportd hardcodes localhost:

// https://github.com/oxidecomputer/dendrite/blob/606c0be888/tfportd/src/tfport.rs#L87-L88
let client = lldpd_client::Client::new(
    &format!("http://localhost:{}", lldpd_client::default_port()),

---

wicketd -> ddm-admin-client

wicketd uses DdmAdminClient::localhost():

// https://github.com/oxidecomputer/omicron/blob/6cef874f3b/wicketd/src/bootstrap_addrs.rs#L162
let ddm_admin_client = DdmAdminClient::localhost(log)?;

---

wicketd -> dpd-client

wicketd hardcodes [::1]:

// https://github.com/oxidecomputer/omicron/blob/6cef874f3b/wicketd/src/preflight_check/uplink.rs#L83
let dpd_client = dpd_client::Client::new(
    &format!("http://[::1]:{DENDRITE_PORT}"),
    client_state,
);

---

wicketd -> gateway-client

wicketd receives the MGS address via CLI, documented as expected to be localhost:

// https://github.com/oxidecomputer/omicron/blob/6cef874f3b/wicketd/src/bin/wicketd.rs#L35
/// The address (expected to be on localhost) for MGS.
#[clap(long, action)]
mgs_address: SocketAddrV6,

MGS is configured to listen on [::1] (services.rs:2606).