Large/sparse file support #2
Description
Right now we use 1/4096 of a file's size in RAM. For a 64GB file, this is 16MB. For an 8TB file, this is 2GB of RAM.
We can bound the amount of RAM used by calling mincore on parts of the file at a time. Perhaps do no more than 64GB? This would still use up to 16MB per thread, so perhaps also a global semaphore should guard overall memory usage.