Skip to content

Commit 38243e8

Browse files
authored
Add flexible scaling patterns to FAQ (#491)
* Add flexible scaling patterns to FAQ * Final edits
1 parent 6f0f1a6 commit 38243e8

File tree

2 files changed

+68
-0
lines changed

2 files changed

+68
-0
lines changed

dev/vignettes/_v07-questions.Rmd

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -128,3 +128,37 @@ func <- carrier::crate(\(x) fn(x), fn = fn)
128128
```
129129

130130
- For environments, consider using `parent.env(e) <- emptyenv()`. This is not required for R6 classes as they are already isolated by default. Regardless of any parent environment, an environment / R6 class could still contain many items that are not needed by the parallel process, in which case consider passing individual members (`env$x`, `env$y`) rather than the entire object (`env`) where possible.
131+
132+
### 5. How to create daemons on-demand or shutdown daemons when they're not used
133+
134+
Setting daemons can be totally separate from the task of actually launching (deploying) them. To just set daemons when you're using only your local machine:
135+
136+
```r
137+
daemons(url = local_url())
138+
```
139+
or if you're using your local machine and/or remote machines:
140+
141+
```r
142+
daemons(url = host_url())
143+
```
144+
This sets up daemons. You can think of this as a 'base station' that listens for incoming daemon connections.
145+
146+
When you actually want to launch (deploy) a daemon, you may use either:
147+
```r
148+
launch_local()
149+
```
150+
or
151+
```r
152+
launch_remote(remote = ssh_config("ssh://servername")) # or cluster_config()
153+
```
154+
The key to adopting a flexible-use pattern that lets you scale up and down is to specify one of the following arguments to the `...` argument of `launch_local()` or `launch_remote()`. You may also supply these to the initial `daemons()` call for them to apply by default for all launches:
155+
156+
- `maxtasks`: integer number of tasks to perform before exiting.
157+
- `idletime`: milliseconds idle time before exiting.
158+
- `walltime`: milliseconds (soft) wall time before exiting (i.e. at least this amount of time, and possibly more as there is no forcible timeout mid-task)
159+
160+
As an example, to launch a daemon to perform one task only:
161+
```r
162+
launch_remote(remote = ssh_config("ssh://servername"), maxtasks = 1L)
163+
```
164+
In this way, you may use `cluster_config()` to launch jobs on an HPC cluster on demand, without using persistent daemons. Note that you will incur the latency costs of however long it takes to launch the job.

vignettes/v07-questions.Rmd

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -136,3 +136,37 @@ func <- carrier::crate(\(x) fn(x), fn = fn)
136136
```
137137

138138
- For environments, consider using `parent.env(e) <- emptyenv()`. This is not required for R6 classes as they are already isolated by default. Regardless of any parent environment, an environment / R6 class could still contain many items that are not needed by the parallel process, in which case consider passing individual members (`env$x`, `env$y`) rather than the entire object (`env`) where possible.
139+
140+
### 5. How to create daemons on-demand or shutdown daemons when they're not used
141+
142+
Setting daemons can be totally separate from the task of actually launching (deploying) them. To just set daemons when you're using only your local machine:
143+
144+
```r
145+
daemons(url = local_url())
146+
```
147+
or if you're using your local machine and/or remote machines:
148+
149+
```r
150+
daemons(url = host_url())
151+
```
152+
This sets up daemons. You can think of this as a 'base station' that listens for incoming daemon connections.
153+
154+
When you actually want to launch (deploy) a daemon, you may use either:
155+
```r
156+
launch_local()
157+
```
158+
or
159+
```r
160+
launch_remote(remote = ssh_config("ssh://servername")) # or cluster_config()
161+
```
162+
The key to adopting a flexible-use pattern that lets you scale up and down is to specify one of the following arguments to the `...` argument of `launch_local()` or `launch_remote()`. You may also supply these to the initial `daemons()` call for them to apply by default for all launches:
163+
164+
- `maxtasks`: integer number of tasks to perform before exiting.
165+
- `idletime`: milliseconds idle time before exiting.
166+
- `walltime`: milliseconds (soft) wall time before exiting (i.e. at least this amount of time, and possibly more as there is no forcible timeout mid-task)
167+
168+
As an example, to launch a daemon to perform one task only:
169+
```r
170+
launch_remote(remote = ssh_config("ssh://servername"), maxtasks = 1L)
171+
```
172+
In this way, you may use `cluster_config()` to launch jobs on an HPC cluster on demand, without using persistent daemons. Note that you will incur the latency costs of however long it takes to launch the job.

0 commit comments

Comments
 (0)