hitting the integer limit for MPI-I/O on fat and dense GPU machines

[kostrzewa]

On AMD MI250X (64 GB of GPU memory per GCD), the MG for an 80c160 lattice comfortably fits on two nodes with four GPUs each (eight GCDs each). This means that the 44 GB gauge configuration has to be read by 16 MPI tasks. This means that every MPI task reads around 2.7 GB (2949120086 bytes), which exceeds the silly 2^31 = 2147483648 byte limit of MPI I/O.

This means that we need to split reads and writes exceeding this limit into multiple I/O operations.

[fernandezdlg]

I am running into the exact same issue when trying to make jobs smaller for faster slurm queueing in daint.alps. From the "Chunks" branch, I assume there is not a solution working with tmLQCD at this moment, correct?

If not, I'll try to look into this issue a bit further next month. I believe having support for larger local lattices could be very beneficial.

[kostrzewa]

@fernandezdlg Indeed, this has not been resolved. For correlation function calculations (where you only need to read the gauge field) just run without LEMON. LIME is sufficiently fast unless your jobs are super short, which I doubt.

[kostrzewa]

It's a different matter entirely when running the HMC or if you want to use LEMON to write out a propagator or eigenvector. LIME is not a viable alternative in that case...

[kostrzewa]

In principle I think there's a relatively simple solution with MPI 4.0 now as functions have been added which support "large counts". See https://www.mpi-forum.org/docs/mpi-4.1/mpi41-report/node499.htm.

The MPI procedures which now support "large counts" are suffixed with _c and take MPI_Count or MPI_Offset instead of int in these critical places.

I'm not sure how easy the change to these functions would be. It could be that "chunking" is more of a drop-in replacement but I haven't looked into it enough.

[kostrzewa]

A relevant function would be for example this one: https://www.mpich.org/static/docs/v4.0.x/www3/MPI_File_write_at_all_c.html

as compared to this one:

https://www.mpich.org/static/docs/v3.2/www3/MPI_File_write_at_all.html