Question about RAM usage across pipeline steps (coarse → partitions → merge) #195
Description
First of all, thank you for the excellent work and for open-sourcing this project — it’s been very helpful.
I’m training with my own dataset and I’m a bit concerned about host RAM (CPU memory) usage across different steps of the pipeline. My dataset is about ~1000 4K images, and I observed quite high RAM consumption.
Here are my current observations:
Coarse training
python main.py fit
--config configs/$COARSE_NAME.yaml
-n $COARSE_NAME
Peak RAM: ~250 GB
Partition training (8 blocks)
python utils/train_citygs_partitions.py -n $NAME
It partitions into 8 blocks
Peak RAM: roughly ~70–90 GB per block (depending on the block)
My questions:
Could you share expected / typical RAM ranges for each major step in the pipeline, especially for datasets at my scale (≈1000 4K images)? In particular, do you have experience-based estimates for the RAM usage of the later steps, including but not limited to:
checkpoint merging, e.g. python utils/merge_citygs_ckpts.py outputs/$NAME any other post-processing / export / evaluation steps that may load multiple blocks or large tensors into RAM
Additionally:
Are there known flags or configuration options to reduce peak RAM (e.g., caching behavior, dataloader settings, preprocessing strategy, depth/mask loading, image downsample suggestions)?
Is the coarse-stage RAM dominated by any specific component (e.g., image caching, depth maps, feature extraction, point cloud, etc.)?
Any guidance on how to plan RAM requirements for the full pipeline would be greatly appreciated. Thanks again for your great work!