Skip to content

Commit 85b35a7

Browse files
eagleonhilleagleonhill
committed
Add article about NoteBook on Slurm
1 parent b41529d commit 85b35a7

File tree

3 files changed

+184
-1
lines changed

3 files changed

+184
-1
lines changed

‎.vitepress/data/blogPosts.ts‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,27 @@ export interface BlogPost {
1212
}
1313

1414
export const blogPosts: BlogPost[] = [
15+
{
16+
"title": "How to Run Jupyter Notebook on an HPC Cluster: SLURM & Velda Guide",
17+
"slug": "run-juypter-notebook-on-hpc",
18+
"description": "Master running Jupyter Notebooks on HPC clusters with SLURM or simplify the process with Velda. Learn SSH tunneling, port forwarding, and streamlined alternatives for interactive data science workflows on remote compute resources.",
19+
"excerpt": "Learn how to run Jupyter Notebooks on HPC clusters using SLURM with SSH tunneling, or discover how Velda simplifies the entire process to a single command for seamless interactive data science workflows.",
20+
"date": "2025-12-01",
21+
"author": "Chuan Qiu",
22+
"readingTime": "7 min",
23+
"category": "Technical Blog",
24+
"image": "https://cdn-images-1.medium.com/max/2400/1*plZPR9YMdDmyc1KXsShkJQ.png",
25+
"tags": [
26+
"jupyter",
27+
"hpc",
28+
"slurm",
29+
"data-science",
30+
"remote-computing",
31+
"velda",
32+
"ssh-tunneling",
33+
"interactive-development"
34+
]
35+
},
1536
{
1637
"title": "Optimizing ML Development at Kumo.ai with Velda",
1738
"slug": "kumo-optimizing-ml-development",

‎blog/run-juypter-notebook-on-hpc.md‎

Lines changed: 161 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,161 @@
1+
---
2+
sidebar: false
3+
title: "How to Run Jupyter Notebook on an HPC Cluster: SLURM & Velda Guide"
4+
description: "Master running Jupyter Notebooks on HPC clusters with SLURM or simplify the process with Velda. Learn SSH tunneling, port forwarding, and streamlined alternatives for interactive data science workflows on remote compute resources."
5+
date: 2025-12-01
6+
author: Chuan Qiu
7+
tags: [jupyter, hpc, slurm, data-science, remote-computing, velda, ssh-tunneling, interactive-development]
8+
keywords: ["Jupyter Notebook HPC", "SLURM Jupyter setup", "SSH tunnel Jupyter", "remote Jupyter notebook", "HPC data science", "Velda Jupyter", "port forwarding Jupyter", "interactive computing cluster", "GPU Jupyter notebook", "remote development workflow"]
9+
image: "https://cdn-images-1.medium.com/max/2400/1*plZPR9YMdDmyc1KXsShkJQ.png"
10+
excerpt: "Learn how to run Jupyter Notebooks on HPC clusters using SLURM with SSH tunneling, or discover how Velda simplifies the entire process to a single command for seamless interactive data science workflows."
11+
readingTime: "7 min"
12+
category: "Technical Blog"
13+
---
14+
15+
# **How to Run Jupyter Notebook on an HPC Cluster: SLURM & Velda Guide**
16+
17+
Launch, Tunnel, and Access Interactive Sessions on Remote Machines
18+
19+
## **Why Run Jupyter Notebooks on HPC Clusters?**
20+
21+
Jupyter Notebooks have become the de facto standard for interactive data analysis, visualization, and collaborative research in the data science community. However, as datasets grow in size and complexity, the computational demands often exceed the capabilities of a local machine. This is where High-Performance Computing (HPC) clusters, managed by systems like SLURM, become essential.
22+
23+
**HPC clusters provide critical advantages for data scientists:**
24+
25+
* **Massive Parallel Processing**: Access to hundreds of CPU cores and multiple GPUs
26+
* **Extensive Memory**: Handle datasets that exceed local machine limitations
27+
* **Dedicated Resources**: Run long-running experiments without tying up your laptop
28+
* **Cost Efficiency**: Share expensive computational resources across teams
29+
30+
Using Jupyter Notebooks on an HPC cluster is therefore a critical skill for data scientists and machine learning engineers working with large-scale data processing, deep learning models, or computationally intensive simulations.
31+
32+
Unlike running code on a local machine, working on a remote HPC cluster requires specific steps to manage resources and connectivity. This guide explains how to start an interactive Jupyter server on a compute node, configure SSH tunneling for your network setup, and access your notebook in your local browser.
33+
34+
## **1. The Standard SLURM Method for Running Jupyter Notebooks**
35+
36+
Most SLURM cluster environments require you to allocate compute resources, launch the Jupyter server, and manually configure SSH tunneling for connectivity. Here's the complete workflow:
37+
38+
### **Step 1: Allocate a Compute Node with SLURM**
39+
40+
It's almost never recommended to run heavy processing on the login node, and you cannot access GPU in that way. Instead, request an interactive job on a dedicated compute node using the `srun` command.
41+
42+
```bash
43+
srun -p gpu --gres=gpu:1 --time=02:00:00 --cpus-per-task=4 --mem=32G --pty bash
44+
```
45+
46+
Result: The scheduler moves you from the login node to a gpu node (e.g., `node123`).
47+
48+
**Pro tip:** Adjust the `--time`, `--cpus-per-task`, and `--mem` parameters based on your workload requirements. For deep learning tasks, request GPU resources using `--gres=gpu:1` or more.
49+
50+
### **Step 2: Launch Jupyter Notebook on the Compute Node**
51+
52+
Once logged into the compute node, load your Python environment (e.g., using conda environment or module load) and launch Jupyter. You must use the `--no-browser` flag because the remote terminal has no display.
53+
54+
```bash
55+
module load anaconda
56+
conda activate myenv
57+
58+
# Run Jupyter without opening a browser window
59+
jupyter notebook --no-browser --port=8888
60+
```
61+
62+
Note: The terminal will output a URL containing a token (e.g., `http://localhost:8888/?token=...`). Keep this terminal open and save the token for later use.
63+
64+
### **Step 3: Configure SSH Tunnel for Remote Access**
65+
66+
Because compute nodes are typically on a private network, you cannot access them directly from your laptop. You must create an SSH tunnel to forward the Jupyter port. Choose the option below that matches your HPC cluster's network configuration:
67+
68+
#### Option A: Double Tunneling (Standard for most HPCs)
69+
70+
* Scenario: Compute nodes are firewall-protected and only reachable via the login node.
71+
* Action: Open a new terminal on your local machine and run: \
72+
```bash
73+
# Syntax: ssh -t -L [LocalPort]:localhost:[RemotePort] user@login ssh -L [RemotePort]:localhost:[RemotePort] node_name
74+
ssh -t -L 8888:localhost:8888 username@login \
75+
ssh -L 8888:localhost:8888 node123
76+
```
77+
* (Replace <code>node123</code> with the **node name** found via <code>hostname</code> in your Slurm session).
78+
<img src="https://cdn-images-1.medium.com/max/1600/1*6t1-JDyMKvbWAC-qAgBPgg.png" />
79+
80+
#### Option B: Internal Network Forwarding (Restricted Access)
81+
82+
* Scenario: You cannot SSH into compute nodes at all (even from the login node), but the login node can "see" the compute node on the internal network.
83+
* Action: Start notebook with `--ip=0.0.0.0` and run this on your local machine:
84+
```bash
85+
ssh -N -L 8888:node123:8888 username@login
86+
```
87+
<img src="https://cdn-images-1.medium.com/max/1600/1*2kjh3fo2m322oErKG6-m9A.png" />
88+
89+
#### Option C: Direct Tunneling
90+
91+
* Scenario: Your institution allows direct SSH access to compute nodes (rare).
92+
* Action: Run this on your local machine \
93+
```bash
94+
ssh -N -L 8888:localhost:8888 username@node123
95+
```
96+
* Alternatively, you can directly access the notebook with node123:8888 if you start jupyter notebook with public ip & port.
97+
98+
### **Step 4: Open in Local Browser**
99+
100+
Now that the tunnel is active, your local machine can communicate with the Jupyter server running on the HPC cluster.
101+
102+
1. Open your web browser.
103+
2. Navigate to `http://localhost:8888`.
104+
3. Paste the token generated in Step 2.
105+
4. You should see the jupyter dashboard in your browser.
106+
107+
**Troubleshooting Common Issues:**
108+
* **Port already in use:** Change the port number (e.g., use 8889 instead of 8888)
109+
* **Connection refused:** Verify the SSH connection is still active
110+
111+
## **2. The Alternative: Simplified Cloud-Native Jupyter Access with Velda**
112+
113+
Managing SSH sessions, tracking node names, and dealing with firewall restrictions is complex and error prone. Velda offers a modern, cloud-native alternative, so your browser can access to Jupyter Notebooks without manual port forwarding or complex networking configuration.
114+
115+
### **Step 1: Launch Jupyter with Velda's vrun Command**
116+
117+
Use the `vrun` command to allocate a compute instance and start the Jupyter Notebook service simultaneously:
118+
119+
```bash
120+
vrun -P gpu-h200-1 -s notebook jupyter notebook --no-browser --port=8888
121+
```
122+
123+
* **`-P gpu-h200-1`**: Specifies the required resources (GPU node with 1 nvidia H200)
124+
* **`-s notebook`**: Names the session for easy identification and proxy access.
125+
126+
**Key advantages:**
127+
* No manual resource allocation or node tracking
128+
* Automatic environment synchronization
129+
* Isolate with other workload on the node
130+
* Built-in security and authentication
131+
132+
### **Step 2: Access Your Notebook Instantly**
133+
134+
Instead of configuring SSH tunnels, simply open the automatically-generated secure URL in your browser:
135+
136+
**URL Format:** `https://8888-notebook-[instance-id].velda.cloud`
137+
138+
*(Or use your custom domain in your self-hosted Velda deployment).*
139+
140+
This method immediately connects your local browser to the remote Jupyter Notebook session with enterprise-grade security, even if you're behind a strict corporate firewall or VPN.
141+
142+
Regardless of which method you choose, follow these best practices:
143+
144+
1. **Always use compute nodes**, never use login nodes for Jupyter sessions
145+
2. **Request appropriate resources** - don't over-allocate GPUs or memory
146+
3. **Use environment management** (conda/venv) for reproducibility
147+
4. **Save work frequently** - HPC sessions can timeout
148+
5. **Monitor resource usage** with tools like `nvidia-smi` for GPUs
149+
6. **Clean up sessions** when finished to free resources for others
150+
151+
## **Ready to Simplify Your HPC Jupyter Workflow?**
152+
153+
Stop wrestling with SSH tunnels and complex HPC configurations. Velda makes running Jupyter Notebooks on cloud and HPC resources as simple as running them locally.
154+
155+
**Experience the difference:** One command to launch, instant browser access, and zero networking headaches. Learn more about how [Velda compares Slurm](/slurm-alternative).Join data scientists and ML engineers who've already streamlined their workflows with Velda.
156+
157+
**Get started today:**
158+
159+
* [**Try hosted Velda cloud**](https://velda.cloud/): Immediately access cloud compute with one click
160+
* [**Velda Open Source**](https://github.com/velda-io/velda): Deploy on your own infrastructure
161+
* [**Enterprise Edition**](https://velda.io/book): Get SSO, RBAC, and advanced features

‎sitemap.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ description: Complete site navigation for Velda - find all pages, blog posts, an
2323

2424
## 📰 Blog
2525
- [Blog Home](/blog/) - Latest posts and updates
26+
- [How to Run Jupyter Notebook on an HPC Cluster: SLURM & Velda Guide](/blog/run-juypter-notebook-on-hpc) - *November 30, 2025*
2627
- [Optimizing ML Development at Kumo.ai with Velda](/blog/kumo-optimizing-ml-development) - *October 20, 2025*
2728
- [Velda Blog - Cloud Development Insights & Updates](/blog/) - *October 19, 2025*
2829
- [Building a Scalable ML Workflow with Velda](/blog/build-machine-learning-workflow) - *September 24, 2025*
@@ -37,4 +38,4 @@ description: Complete site navigation for Velda - find all pages, blog posts, an
3738

3839
---
3940

40-
*Last updated: 2025-10-22*
41+
*Last updated: 2025-12-02*

0 commit comments

Comments
 (0)