[question] - Do you have any advice for loading and working with multiple sp2b files at once?

[evanlouis563]

Hello!

Thank you for providing this wonderful tool for working with SP2 data. I'm interested in exploring it, and your examples are very helpful. I noticed the examples are based on working with one SP2 file. I watched your YouTube video (https://www.youtube.com/watch?v=Kb6I8sd2IKM&t=85s), and if I understand correctly it sounds like PySP2 was developed due to the sheer number of sp2b files you would get when running the SP2 at a ground station continuously. I was wondering if you have any guidance or if there is functionality (that I missed) in PySP2 for combining multiple files so that you can look at a time period that spans the time coverage of multiple sp2b files with the different functions in 'util.' I'm thinking of something akin to the way that Igor concatenates the sp2b files into very long waves of 'particle by particle' information, which can then be post-processed all together.

I am analyzing SP2 data from a research campaign with 20 flights (each flight was ~ 8 hours), so I have about 60 files per flight. I used PySp2 to read the sp2b files one by one into datasets, and then I saved each dataset as a netCDF file. I then looped through those netCDF files and saved them as individual slices in a larger netCDF file. Now, to access the information from each sp2b file, I will just need to loop through those slices. To utilize PySP2 functions like particle_properties and leo_fit, I will need to call those functions for each Dataset as I loop through the slices of a larger Dataset encompassing all the information originally in the individual sp2b files.

While trying to aggregate all of the information from the datasets containing individual sp2b file information, I have been running into a lot of memory issues (my laptop only has 15.6 GB of RAM), which is why I went the netCDF route. Trying to concatenate the data in each sp2b file (as Igor does) into one large dataset also gave me a memory error.

I was eventually able to save a netCDF file containing 66 slices (corresponding to the 66 sp2b files for a single research flight; see screenshot below). However, I am wondering if there is a better way or if you would mind sharing how you go about tackling working with the large number of sp2b files containing information about a period of interest. I talked to my advisor, and he is helping me to get more RAM, but I was curious if you, the PySP2 experts, would mind sharing with me how you typically accomplish accomplish this task.

For context, I also wanted to avoid working with Igor for SP2 data processing after seeing how long things were taking. Before finding PySP2, I exported all the 'particle-by-particle' and housekeeping information out of Igor into HDF5 files for each flight. I have been working with those HDF5 files on my own in python for many of the postprocessing steps, but I would love to try and take advantage of the work that has already been done with PySP2 if possible.

I will also apologize in advance if my question is not appropriate to post here. If it's more of an issue with my understanding of python, then please don't be afraid to let me know. I used MATLAB throughout my PhD, and have only started to use Python in my postdoc so I have much to learn.

Take care,
Eva-Lou

[rcjackson]

@evanlouis563 sorry for the very late response here. I was extremely busy with various DOE projects and was unable to devote my attention to PySP2 until now. On your laptop, I would keep your particle files as separate slices like you did there. I would then do the processing for the day with just one file, and save the result. Once you have 66 separate daily concentration/PSDs from this processing, just add together all of the concentrations and PSDs, since the total concentration will be the sum over all files. This should not take too much memory. You can even use dask to speed this up.

[evanlouis563]

@rcjackson Thank you for the response! No apologies needed! Your response makes sense.