Questions on Self-Hosting: Custom Templates, Cleanup & K8s Co-hosting

[uucloud]

I've successfully got E2B self-hosted and running on a single machine, and it's working great! As I'm thinking about scaling this up (especially on Kubernetes), I have a few questions:

1. Custom Template Startup Speed
   I'm wondering how to handle custom templates at scale.

If I have many users creating their own templates, it's not feasible to cache all of them on every node. I noticed the scheduler doesn't seem to check for template cache hits when placing a sandbox.

This seems like it could cause slow "cold starts" for custom templates. How do you recommend solving this?

2. Resource Cleanup on Orchestrator Crash
   My second question is about cleanup. I've observed that if the orchestrator process dies unexpectedly, it does leak resources—specifically orphaned Firecracker VMs and veth pairs from the network pool.

How are you currently handling this? Is there a garbage collector, a reconciliation loop, or some other process that cleans up these resources after the orchestrator restarts? I'm curious about your maintenance strategy here.

3. Co-hosting with Kubernetes (kubelet)

we have internally clusters that are already running on AWS and k8s (EKS).

I saw you mentioned that you have already running e2b in k8s. This is exactly my goal.

I'm very curious how you manage running the orchestrator and kubelet on the same node without resource conflicts. Specifically:

How do you manage resources like hugepages so they don't conflict?

How do you make kubelet aware of the CPU and Memory that the orchestrator (and its VMs) are consuming, so k8s doesn't over-provision pods?

If kubelet is using the static CPU policy (with cpuset), how do you prevent the orchestrator's VMs from using those reserved CPUs?

Are you running the orchestrator as a DaemonSet, or just as a separate process on the host?

4. Flexibility for envd (and Core Binaries)
   My last question is about updating core components like envd that are baked into the templates.

It seems that if I want to add a feature to envd or update it, I would need to rebuild all existing templates (including all custom ones) to get the new version. This maybe difficult to manage at scale.

Is this correct, or do you have another solution? I was wondering if it would be technically feasible to mount the envd binary into the VM (e.g., right before boot) rather than baking it into the filesystem? This seems like it would be much more flexible for rolling out updates.

Thanks for any insights, and thanks for building this!

[linear]

ENG-3245

[sitole]

If I have many users creating their own templates, it's not feasible to cache all of them on every node. I noticed the scheduler doesn't seem to check for template cache hits when placing a sandbox.

We are working on a scale where it is not feasible to cache the template on each node. In the infra repo, there is an NFS cache layer that speeds up template caching. We are working on improvements here that will speed things up even more.

My second question is about cleanup. I've observed that if the orchestrator process dies unexpectedly, it does leak resources—specifically orphaned Firecracker VMs and veth pairs from the network pool.

Orchestrator should not die unexpectedly (at least in production mode), and if it does, please report it. Otherwise, a safe way to re-deploy Orchestrator now is to replace the whole VM. We are working on better deployments here.

I'm very curious how you manage running the orchestrator and kubelet on the same node without resource conflicts.
I will group all k8s-related responses under this one block, as it's very custom and specific, and it's not open-sourced yet.

We are running customer K8s nodes for orchestrating sandboxes. On this node, there is an orchestrator job (similar to Nomad) that, during pod start, copies the orchestrator binary from the Docker image to the host, and then, during the main pod Docker task, it jumps to the host namespace and runs the orchestrator there. This way, it can still be orchestrated via k8s, but it can also run as a binary in host Linux namespaces, which are required for the orchestrator to work properly. There is more work for us to do to make it more standard and less hacky.

We still feel like the Nomad setup in this repository is easy to go with, which is why k8s stuff is not yet open-sourced, as it requires a lot of care and code changes from our side.

It seems that if I want to add a feature to envd or update it, I would need to rebuild all existing templates (including all custom ones) to get the new version. This may be difficult to manage at scale.
Yes, if a new envd is released, you need to update the template. This is a current limitation; feel free to offer some ideas on how to overcome this in a safe way.

[sitole]

Please feel free to follow issue related to selfhosting #864.

[uucloud]

@sitole , thanks so much for the detailed answers! This is really helpful.

replace the whole VM

This surprised me a little, as I was under the impression that the orchestrator needed to run on a bare-metal machine to launch Firecracker (since nested virtualization isn't always available). Are you running the main orchestrator inside a VM, which then launches the Firecracker microVMs? I just want to make sure I'm understanding the architecture correctly.

feel free to offer some ideas on how to overcome this in a safe way.

For the envd update, thanks for confirming! You asked for ideas, and I had a couple of thoughts:

• One idea was a self-updating mechanism for envd itself, but this wouldn't work for sandboxes where internet access is disabled.

• Another idea, is what if the envd binary could be dynamically mounted into the VM at boot time? Perhaps it could be passed in via virtio-fs or as a small, read-only block device. This seems like it would cleanly decouple the envd version from the base template and make updates much easier.

Thanks again for your time!

[sitole]

replace the whole VM

This surprised me a little, as I was under the impression that the orchestrator needed to run on a bare-metal machine to launch Firecracker (since nested virtualization isn't always available). Are you running the main orchestrator inside a VM, which then launches the Firecracker microVMs? I just want to make sure I'm understanding the architecture correctly.

Now everything is run in VMs, even for AWS it acts like VM. We have some internal prototypes on how to do redeployments without shutting down the whole node, but it's not production-ready yet.

For the envd update, thanks for confirming! You asked for ideas, and I had a couple of thoughts

Thank! We considered both in the past, but as all sandbox starts are basically VMs resumes (with envd already running and program memory in RAM) we worry that restarting the service during each sandbox start can harm start performance, but we need to investigate this more.

[sitole]

I will close this issue as duplicated, fell free to continue with selfhosting questions in core issue #864.