RAM limits for big datasets

[Jaruba20]

First, @HuguesTHOMAS thanks a lot for your work, it is trully amazing.

I am trying to train a model with a bunch of datasets, but I am running into a problem with RAM limits in the preprocessing steps.
I have 128GB of RAM, and I am getting an OOM error during the subsampling stage when loading/creating+loading the .ply and .pkl files from all clouds.

Have you had a similar problem? I've check almost all related issues both in this repo and in KPConv and KPConv-Pytorch, and I cannot find any solution.

Do you recommend changing parameters? A lazy-load? (I understand that this one doesn't work because it would still need to load everything before the training starts) A shared memory with disc? Or maybe a more efficient method of storing the values in memory?

Thanks a lot.

[HuguesTHOMAS]

Hi @Jaruba20,

Thanks for your interest in my work.

If your dataset is too big to all fit in memory, then you need to create a custom data loader. I have not faced this issue in my work, but I had some ideas on how to tackle this.

For every batch, you want your dataloader to read the input data directly from a file. To avoid long loading times, ideally, the files should be small. So a good strategy would be to split your datasets into tiles and save each tile in a file. Then, if you want to keep the sphere input, you will need a smart datalaoder that loads the file for each sphere (potentially up to 4 tile files per sphere. Keep your tile size a bit larger than the sphere diameter for the most efficient loading. There are multiple ways to decide the center of the sphere you want to sample. You can keep a heavily sunsampled version of your whole dataset from which you sample, or you can randomly select tiles and then randomly select points in these tiles, etc. Just remember that random sampling is usually worse than balanced class sampling or balanced spatial sampling.

This is just some ideas, there are many ways to tackle this, so feel free to explore your own.

Best,
Hugues

[Jaruba20]

@HuguesTHOMAS Thanks a lot for your response.

I've managed to run it following your advices and some ideas of my own, although the results are not as good as I expected.

For context, I am trying to train a "general" model for outdoor segmentation. I've obtained some classified datasets and remapped the classes so they all have the same ones. There are some things that I've noticed:

1. Probably due to class imbalance (ground and building classes, but primarly the first one are much more present in all clouds than others like vehicles or fences), there are inconsistencies in cloud segmentation, like roofs being classified as floor although I am using big radius during testing (I've also read in other issues that you recommend adding height as a feature, but I am afraid that scenarios with multiple floor heights might be wrongly segmented). I am trying to do a new run with the c-sampler, as it is stated in the config to manage with this problems, but I am open to suggestions if you believe there are better/additional approaches.

2. Another concern I have is about RGB. Some of the datasets I am using do not have a color feature, and for the moment, I've decided to not include them during training. I don't really remember if it was in this repo or in KPconv's, where I read that in order to not overfit categories with colors and make the model learn the geometric features, a percentage of points had its rgb values changed to 0. My main question here is if it is safe to "mix" clouds with and without rgb features, or if that could affect the models performance in your opinion.

3. I don't quite understand the Neighbors Calibration Report printed. Could you briefly explain how to comprehend the output? For example, this one corresponds to my last training:

Finally, I have the same problem as #12 . No log.txt file is being written during training, which also do not let me compare perfomances, just in case you knew the cause.

Thanks again for everything.

Have a nice week.