Infamous memory usage #761
Description
Preliminary checks
- I am running the latest version
- I have read the Code of Conduct
- I have checked there is not other Issues or Pull Requests covering the same topic to open
Describe the bug
So I already read about the OOM killing mdedup because if high memory usage, but I'm quite a bit puzzled here.
I ran the following on a set of 590k emails:
/mnt/venv/bin/mdedup -b raw --action move-selected --strategy select-biggest --export /mnt/DEDUP /mnt/EML /mnt/EML-OST /mnt/EML-PST
Once about 23k emails were done, mdedup had a memory usage of 42GB, which is roughly 1,8MB per email ! At that pace, I'd need 1TB to dedup my emails.
This number climbed almost proportionnaly, which makes me think that, since I used the -b raw parameter. mdedup does not only keep paths and corresponding sets of hashes in memory, but full emails.
I've done another test without -b switch, and this time I got about 15GB for 23k emails which still is a very high number. At that pace I'd need about 400GB for my emails to be deduped.
I've made a quick calculation, keeping 10 SHA256 hashes (32 bytes) in memory per email would require about 180MB of data (regardless of any data representation) plus let's say 1KB per filename which would require another whopping 576MB, for a total of less than a GB of data.
Now I can imagine that the dedup table would take some memory, but 1TB of memory ? Even ZFS isn't that hungry.
This program is highly memory inefficient, since I think it keeps way more than just sets of hashes. Does it keep the whole in memory representation of an email object per email ?
Are my findings normal or off the charts ?
I've had earlier issues with high memory consumption with lots of json data.
I'd advise to have a look at msgspec library which saved me about 40% memory usage on a json dict with millions of entries.
Anyway, I'd advise you to only keep hashes in memory instead of whatever class object repr you're keeping.
Additional context
mdedup output just before I had to stop it:
Hashed mails [#-----------------------------------] 23756/596456 04:35:21
Memory usage of mdedup
ps -eo size,pid,user,command --sort -size | awk '{ hr=$1/1024 ; printf("%13.2f Mb ",hr) } { for ( x=4 ; x<=NF ; x++ ) { printf("%s ",$x) } print "" }' | cut -d "" -f2 | cut -d "-" -f1 | head
42956.84 Mb /mnt/venv/bin/python /mnt/venv/bin/mdedup
17007.59 Mb /usr/libexec/qemu
13616.19 Mb /usr/libexec/qemu
10567.80 Mb /usr/libexec/qemu
9330.69 Mb /usr/libexec/qemu
8982.71 Mb /usr/libexec/qemu
8976.46 Mb /usr/libexec/qemu
8935.57 Mb /usr/libexec/qemu
8746.17 Mb /usr/libexec/qemu
Memory map of mdedup
pmap $(ps aux | grep [m]dedup | awk '{print $2}') 13073: /mnt/venv/bin/python /mnt/venv/bin/mdedup -b raw --action move-selected --strategy select-biggest --export /mnt/DEDUP /mnt/EML /mnt/EML-OST /mnt/EML-PST
00005609cf128000 4K r---- python3.9
00005609cf129000 4K r-x-- python3.9
00005609cf12a000 4K r---- python3.9
00005609cf12b000 4K r---- python3.9
00005609cf12c000 4K rw--- python3.9
00005609d0564000 18028940K rw--- [ anon ]
00007fe4b0f36000 256K rw--- [ anon ]
00007fe4b19f6000 3828736K rw--- [ anon ]
00007fe59b4ff000 9425956K rw--- [ anon ]
00007fe7daa0d000 13230060K rw--- [ anon ]
00007feb0223f000 935788K rw--- [ anon ]
00007feb3b41b000 8704K rw--- [ anon ]
00007feb3bc9c000 5120K rw--- [ anon ]
00007feb3c19d000 2560K rw--- [ anon ]
00007feb3c45c000 2816K rw--- [ anon ]
00007feb3c729000 2816K rw--- [ anon ]
00007feb3ca0a000 2048K rw--- [ anon ]
00007feb3cc44000 2048K rw--- [ anon ]
00007feb3ce5b000 1792K rw--- [ anon ]
00007feb3d053000 1536K rw--- [ anon ]
00007feb3d1ef000 1536K rw--- [ anon ]
00007feb3d373000 1792K rw--- [ anon ]
00007feb3d561000 9216K rw--- [ anon ]
00007feb3de61000 12K r---- libgcc_s-11-20231218.so.1
00007feb3de64000 72K r-x-- libgcc_s-11-20231218.so.1
00007feb3de76000 12K r---- libgcc_s-11-20231218.so.1
00007feb3de79000 4K ----- libgcc_s-11-20231218.so.1
00007feb3de7a000 4K r---- libgcc_s-11-20231218.so.1
00007feb3de7b000 4K rw--- libgcc_s-11-20231218.so.1
00007feb3de7c000 20K r---- libnss_myhostname.so.2
00007feb3de81000 68K r-x-- libnss_myhostname.so.2
00007feb3de92000 44K r---- libnss_myhostname.so.2
00007feb3de9d000 4K ----- libnss_myhostname.so.2
00007feb3de9e000 16K r---- libnss_myhostname.so.2
00007feb3dea2000 4K rw--- libnss_myhostname.so.2
00007feb3dead000 1024K rw--- [ anon ]
00007feb3dfad000 24K r---- _decimal.cpython-39-x86_64-linux-gnu.so
00007feb3dfb3000 172K r-x-- _decimal.cpython-39-x86_64-linux-gnu.so
00007feb3dfde000 64K r---- _decimal.cpython-39-x86_64-linux-gnu.so
00007feb3dfee000 4K r---- _decimal.cpython-39-x86_64-linux-gnu.so
00007feb3dfef000 12K rw--- _decimal.cpython-39-x86_64-linux-gnu.so
00007feb3dff2000 1536K rw--- [ anon ]
00007feb3e172000 20K r---- _pickle.cpython-39-x86_64-linux-gnu.so
00007feb3e177000 64K r-x-- _pickle.cpython-39-x86_64-linux-gnu.so
00007feb3e187000 24K r---- _pickle.cpython-39-x86_64-linux-gnu.so
00007feb3e18d000 4K ----- _pickle.cpython-39-x86_64-linux-gnu.so
00007feb3e18e000 4K r---- _pickle.cpython-39-x86_64-linux-gnu.so
00007feb3e18f000 8K rw--- _pickle.cpython-39-x86_64-linux-gnu.so
00007feb3e191000 256K rw--- [ anon ]
00007feb3e1d1000 60K r---- libsqlite3.so.0.8.6
00007feb3e1e0000 964K r-x-- libsqlite3.so.0.8.6
00007feb3e2d1000 196K r---- libsqlite3.so.0.8.6
00007feb3e302000 16K r---- libsqlite3.so.0.8.6
00007feb3e306000 12K rw--- libsqlite3.so.0.8.6
00007feb3e309000 24K r---- _sqlite3.cpython-39-x86_64-linux-gnu.so
00007feb3e30f000 40K r-x-- _sqlite3.cpython-39-x86_64-linux-gnu.so
00007feb3e319000 16K r---- _sqlite3.cpython-39-x86_64-linux-gnu.so
00007feb3e31d000 4K ----- _sqlite3.cpython-39-x86_64-linux-gnu.so
00007feb3e31e000 4K r---- _sqlite3.cpython-39-x86_64-linux-gnu.so
00007feb3e31f000 8K rw--- _sqlite3.cpython-39-x86_64-linux-gnu.so
00007feb3e321000 256K rw--- [ anon ]
00007feb3e361000 88K r---- libreadline.so.8.1
00007feb3e377000 184K r-x-- libreadline.so.8.1
00007feb3e3a5000 40K r---- libreadline.so.8.1
00007feb3e3af000 12K r---- libreadline.so.8.1
00007feb3e3b2000 24K rw--- libreadline.so.8.1
00007feb3e3b8000 1800K rw--- [ anon ]
00007feb3e57a000 20K r---- _yaml.cpython-39-x86_64-linux-gnu.so
00007feb3e57f000 308K r-x-- _yaml.cpython-39-x86_64-linux-gnu.so
00007feb3e5cc000 36K r---- _yaml.cpython-39-x86_64-linux-gnu.so
00007feb3e5d5000 4K r---- _yaml.cpython-39-x86_64-linux-gnu.so
00007feb3e5d6000 8K rw--- _yaml.cpython-39-x86_64-linux-gnu.so
00007feb3e5d8000 516K rw--- [ anon ]
00007feb3e659000 16K r---- libexpat.so.1.8.10
00007feb3e65d000 124K r-x-- libexpat.so.1.8.10
00007feb3e67c000 40K r---- libexpat.so.1.8.10
00007feb3e686000 4K ----- libexpat.so.1.8.10
00007feb3e687000 8K r---- libexpat.so.1.8.10
00007feb3e689000 4K rw--- [ anon ]
00007feb3e68d000 4K r---- fcntl.cpython-39-x86_64-linux-gnu.so
00007feb3e68e000 8K r-x-- fcntl.cpython-39-x86_64-linux-gnu.so
00007feb3e690000 8K r---- fcntl.cpython-39-x86_64-linux-gnu.so
00007feb3e692000 4K r---- fcntl.cpython-39-x86_64-linux-gnu.so
00007feb3e693000 4K rw--- fcntl.cpython-39-x86_64-linux-gnu.so
00007feb3e694000 16K r---- pyexpat.cpython-39-x86_64-linux-gnu.so
00007feb3e698000 28K r-x-- pyexpat.cpython-39-x86_64-linux-gnu.so
00007feb3e69f000 12K r---- pyexpat.cpython-39-x86_64-linux-gnu.so
00007feb3e6a2000 4K ----- pyexpat.cpython-39-x86_64-linux-gnu.so
00007feb3e6a3000 4K r---- pyexpat.cpython-39-x86_64-linux-gnu.so
00007feb3e6a4000 8K rw--- pyexpat.cpython-39-x86_64-linux-gnu.so
00007feb3e6a6000 1280K rw--- [ anon ]
00007feb3e7e6000 12K r---- _multibytecodec.cpython-39-x86_64-linux-gnu.so
00007feb3e7e9000 20K r-x-- _multibytecodec.cpython-39-x86_64-linux-gnu.so
00007feb3e7ee000 8K r---- _multibytecodec.cpython-39-x86_64-linux-gnu.so
00007feb3e7f0000 4K r---- _multibytecodec.cpython-39-x86_64-linux-gnu.so
00007feb3e7f1000 4K rw--- _multibytecodec.cpython-39-x86_64-linux-gnu.so
00007feb3e7f2000 52K r---- md__mypyc.cpython-39-x86_64-linux-gnu.so
00007feb3e7ff000 120K r-x-- md__mypyc.cpython-39-x86_64-linux-gnu.so
00007feb3e81d000 36K r---- md__mypyc.cpython-39-x86_64-linux-gnu.so
00007feb3e826000 4K ----- md__mypyc.cpython-39-x86_64-linux-gnu.so
00007feb3e827000 4K r---- md__mypyc.cpython-39-x86_64-linux-gnu.so
00007feb3e828000 20K rw--- md__mypyc.cpython-39-x86_64-linux-gnu.so
00007feb3e82d000 4K r---- md.cpython-39-x86_64-linux-gnu.so
00007feb3e82e000 4K r-x-- md.cpython-39-x86_64-linux-gnu.so
00007feb3e82f000 4K r---- md.cpython-39-x86_64-linux-gnu.so
00007feb3e830000 4K r---- md.cpython-39-x86_64-linux-gnu.so
00007feb3e831000 4K rw--- md.cpython-39-x86_64-linux-gnu.so
00007feb3e832000 768K rw--- [ anon ]
00007feb3e8f2000 8K r---- _csv.cpython-39-x86_64-linux-gnu.so
00007feb3e8f4000 12K r-x-- _csv.cpython-39-x86_64-linux-gnu.so
00007feb3e8f7000 8K r---- _csv.cpython-39-x86_64-linux-gnu.so
00007feb3e8f9000 4K ----- _csv.cpython-39-x86_64-linux-gnu.so
00007feb3e8fa000 4K r---- _csv.cpython-39-x86_64-linux-gnu.so
00007feb3e8fb000 4K rw--- _csv.cpython-39-x86_64-linux-gnu.so
00007feb3e8fc000 2560K rw--- [ anon ]
00007feb3eb7c000 8K r---- _json.cpython-39-x86_64-linux-gnu.so
00007feb3eb7e000 28K r-x-- _json.cpython-39-x86_64-linux-gnu.so
00007feb3eb85000 8K r---- _json.cpython-39-x86_64-linux-gnu.so
00007feb3eb87000 4K r---- _json.cpython-39-x86_64-linux-gnu.so
00007feb3eb88000 4K rw--- _json.cpython-39-x86_64-linux-gnu.so
00007feb3eb89000 1024K rw--- [ anon ]
00007feb3ec89000 16K r---- _asyncio.cpython-39-x86_64-linux-gnu.so
00007feb3ec8d000 20K r-x-- _asyncio.cpython-39-x86_64-linux-gnu.so
00007feb3ec92000 16K r---- _asyncio.cpython-39-x86_64-linux-gnu.so
00007feb3ec96000 4K r---- _asyncio.cpython-39-x86_64-linux-gnu.so
00007feb3ec97000 8K rw--- _asyncio.cpython-39-x86_64-linux-gnu.so
00007feb3ec99000 256K rw--- [ anon ]
00007feb3ecd9000 8K r---- binascii.cpython-39-x86_64-linux-gnu.so
00007feb3ecdb000 16K r-x-- binascii.cpython-39-x86_64-linux-gnu.so
00007feb3ecdf000 8K r---- binascii.cpython-39-x86_64-linux-gnu.so
00007feb3ece1000 4K ----- binascii.cpython-39-x86_64-linux-gnu.so
00007feb3ece2000 4K r---- binascii.cpython-39-x86_64-linux-gnu.so
00007feb3ece3000 4K rw--- binascii.cpython-39-x86_64-linux-gnu.so
00007feb3ece4000 256K rw--- [ anon ]
00007feb3ed24000 120K r---- libssl.so.3.0.7
00007feb3ed42000 372K r-x-- libssl.so.3.0.7
00007feb3ed9f000 116K r---- libssl.so.3.0.7
00007feb3edbc000 40K r---- libssl.so.3.0.7
00007feb3edc6000 16K rw--- libssl.so.3.0.7
00007feb3edca000 72K r---- _ssl.cpython-39-x86_64-linux-gnu.so
00007feb3eddc000 44K r-x-- _ssl.cpython-39-x86_64-linux-gnu.so
00007feb3ede7000 52K r---- _ssl.cpython-39-x86_64-linux-gnu.so
00007feb3edf4000 4K ----- _ssl.cpython-39-x86_64-linux-gnu.so
00007feb3edf5000 4K r---- _ssl.cpython-39-x86_64-linux-gnu.so
00007feb3edf6000 36K rw--- _ssl.cpython-39-x86_64-linux-gnu.so
00007feb3edff000 1792K rw--- [ anon ]
00007feb3efbf000 12K r---- _struct.cpython-39-x86_64-linux-gnu.so
00007feb3efc2000 20K r-x-- _struct.cpython-39-x86_64-linux-gnu.so
00007feb3efc7000 16K r---- _struct.cpython-39-x86_64-linux-gnu.so
00007feb3efcb000 4K r---- _struct.cpython-39-x86_64-linux-gnu.so
00007feb3efcc000 4K rw--- _struct.cpython-39-x86_64-linux-gnu.so
00007feb3efcd000 8K r---- libuuid.so.1.3.0
00007feb3efcf000 20K r-x-- libuuid.so.1.3.0
00007feb3efd4000 4K r---- libuuid.so.1.3.0
00007feb3efd5000 4K r---- libuuid.so.1.3.0
00007feb3efd6000 4K rw--- [ anon ]
00007feb3efdb000 8K r---- _queue.cpython-39-x86_64-linux-gnu.so
00007feb3efdd000 4K r-x-- _queue.cpython-39-x86_64-linux-gnu.so
00007feb3efde000 4K r---- _queue.cpython-39-x86_64-linux-gnu.so
00007feb3efdf000 4K r---- _queue.cpython-39-x86_64-linux-gnu.so
00007feb3efe0000 4K rw--- _queue.cpython-39-x86_64-linux-gnu.so
00007feb3efe4000 56K r---- libtinfo.so.6.2
00007feb3eff2000 60K r-x-- libtinfo.so.6.2
00007feb3f001000 56K r---- libtinfo.so.6.2
00007feb3f00f000 16K r---- libtinfo.so.6.2
00007feb3f013000 4K rw--- libtinfo.so.6.2
00007feb3f018000 8K r---- resource.cpython-39-x86_64-linux-gnu.so
00007feb3f01a000 4K r-x-- resource.cpython-39-x86_64-linux-gnu.so
00007feb3f01b000 4K r---- resource.cpython-39-x86_64-linux-gnu.so
00007feb3f01c000 4K r---- resource.cpython-39-x86_64-linux-gnu.so
00007feb3f01d000 4K rw--- resource.cpython-39-x86_64-linux-gnu.so
00007feb3f01e000 12K r---- readline.cpython-39-x86_64-linux-gnu.so
00007feb3f021000 12K r-x-- readline.cpython-39-x86_64-linux-gnu.so
00007feb3f024000 8K r---- readline.cpython-39-x86_64-linux-gnu.so
00007feb3f026000 4K ----- readline.cpython-39-x86_64-linux-gnu.so
00007feb3f027000 4K r---- readline.cpython-39-x86_64-linux-gnu.so
00007feb3f028000 4K rw--- readline.cpython-39-x86_64-linux-gnu.so
00007feb3f029000 12K r---- termios.cpython-39-x86_64-linux-gnu.so
00007feb3f02c000 4K r-x-- termios.cpython-39-x86_64-linux-gnu.so
00007feb3f02d000 8K r---- termios.cpython-39-x86_64-linux-gnu.so
00007feb3f02f000 4K r---- termios.cpython-39-x86_64-linux-gnu.so
00007feb3f030000 8K rw--- termios.cpython-39-x86_64-linux-gnu.so
00007feb3f032000 768K rw--- [ anon ]
00007feb3f0f2000 4K r---- libpthread.so.0
00007feb3f0f3000 4K r-x-- libpthread.so.0
00007feb3f0f4000 4K r---- libpthread.so.0
00007feb3f0f5000 4K r---- libpthread.so.0
00007feb3f0f6000 4K rw--- libpthread.so.0
00007feb3f0f7000 4K r---- _contextvars.cpython-39-x86_64-linux-gnu.so
00007feb3f0f8000 4K r-x-- _contextvars.cpython-39-x86_64-linux-gnu.so
00007feb3f0f9000 4K r---- _contextvars.cpython-39-x86_64-linux-gnu.so
00007feb3f0fa000 4K r---- _contextvars.cpython-39-x86_64-linux-gnu.so
00007feb3f0fb000 4K rw--- _contextvars.cpython-39-x86_64-linux-gnu.so
00007feb3f0fc000 4K r---- _uuid.cpython-39-x86_64-linux-gnu.so
00007feb3f0fd000 4K r-x-- _uuid.cpython-39-x86_64-linux-gnu.so
00007feb3f0fe000 4K r---- _uuid.cpython-39-x86_64-linux-gnu.so
00007feb3f0ff000 4K r---- _uuid.cpython-39-x86_64-linux-gnu.so
00007feb3f100000 4K rw--- _uuid.cpython-39-x86_64-linux-gnu.so
00007feb3f101000 64K r---- _regex.cpython-39-x86_64-linux-gnu.so
00007feb3f111000 220K r-x-- _regex.cpython-39-x86_64-linux-gnu.so
00007feb3f148000 512K r---- _regex.cpython-39-x86_64-linux-gnu.so
00007feb3f1c8000 4K r---- _regex.cpython-39-x86_64-linux-gnu.so
00007feb3f1c9000 36K rw--- _regex.cpython-39-x86_64-linux-gnu.so
00007feb3f1d2000 12K r---- unicodedata.cpython-39-x86_64-linux-gnu.so
00007feb3f1d5000 16K r-x-- unicodedata.cpython-39-x86_64-linux-gnu.so
00007feb3f1d9000 1056K r---- unicodedata.cpython-39-x86_64-linux-gnu.so
00007feb3f2e1000 4K r---- unicodedata.cpython-39-x86_64-linux-gnu.so
00007feb3f2e2000 4K rw--- unicodedata.cpython-39-x86_64-linux-gnu.so
00007feb3f2e3000 768K rw--- [ anon ]
00007feb3f3a3000 12K r---- liblzma.so.5.2.5
00007feb3f3a6000 108K r-x-- liblzma.so.5.2.5
00007feb3f3c1000 44K r---- liblzma.so.5.2.5
00007feb3f3cc000 4K ----- liblzma.so.5.2.5
00007feb3f3cd000 4K r---- liblzma.so.5.2.5
00007feb3f3ce000 4K rw--- [ anon ]
00007feb3f3cf000 12K r---- _lzma.cpython-39-x86_64-linux-gnu.so
00007feb3f3d2000 16K r-x-- _lzma.cpython-39-x86_64-linux-gnu.so
00007feb3f3d6000 12K r---- _lzma.cpython-39-x86_64-linux-gnu.so
00007feb3f3d9000 4K r---- _lzma.cpython-39-x86_64-linux-gnu.so
00007feb3f3da000 4K rw--- _lzma.cpython-39-x86_64-linux-gnu.so
00007feb3f3db000 8K r---- libbz2.so.1.0.8
00007feb3f3dd000 52K r-x-- libbz2.so.1.0.8
00007feb3f3ea000 8K r---- libbz2.so.1.0.8
00007feb3f3ec000 4K r---- libbz2.so.1.0.8
00007feb3f3ed000 4K rw--- libbz2.so.1.0.8
00007feb3f3f2000 8K r---- grp.cpython-39-x86_64-linux-gnu.so
00007feb3f3f4000 4K r-x-- grp.cpython-39-x86_64-linux-gnu.so
00007feb3f3f5000 4K r---- grp.cpython-39-x86_64-linux-gnu.so
00007feb3f3f6000 4K r---- grp.cpython-39-x86_64-linux-gnu.so
00007feb3f3f7000 4K rw--- grp.cpython-39-x86_64-linux-gnu.so
00007feb3f3f8000 8K r---- _bz2.cpython-39-x86_64-linux-gnu.so
00007feb3f3fa000 8K r-x-- _bz2.cpython-39-x86_64-linux-gnu.so
00007feb3f3fc000 4K r---- _bz2.cpython-39-x86_64-linux-gnu.so
00007feb3f3fd000 4K ----- _bz2.cpython-39-x86_64-linux-gnu.so
00007feb3f3fe000 4K r---- _bz2.cpython-39-x86_64-linux-gnu.so
00007feb3f3ff000 4K rw--- _bz2.cpython-39-x86_64-linux-gnu.so
00007feb3f400000 2048K rw--- [ anon ]
00007feb3f600000 692K r---- libcrypto.so.3.0.7
00007feb3f6ad000 2416K r-x-- libcrypto.so.3.0.7
00007feb3f909000 820K r---- libcrypto.so.3.0.7
00007feb3f9d6000 4K ----- libcrypto.so.3.0.7
00007feb3f9d7000 344K r---- libcrypto.so.3.0.7
00007feb3fa2d000 12K rw--- libcrypto.so.3.0.7
00007feb3fa30000 12K rw--- [ anon ]
00007feb3fa36000 20K r---- _datetime.cpython-39-x86_64-linux-gnu.so
00007feb3fa3b000 48K r-x-- _datetime.cpython-39-x86_64-linux-gnu.so
00007feb3fa47000 24K r---- _datetime.cpython-39-x86_64-linux-gnu.so
00007feb3fa4d000 4K r---- _datetime.cpython-39-x86_64-linux-gnu.so
00007feb3fa4e000 12K rw--- _datetime.cpython-39-x86_64-linux-gnu.so
00007feb3fa51000 256K rw--- [ anon ]
00007feb3fa93000 8K r---- _posixsubprocess.cpython-39-x86_64-linux-gnu.so
00007feb3fa95000 8K r-x-- _posixsubprocess.cpython-39-x86_64-linux-gnu.so
00007feb3fa97000 4K r---- _posixsubprocess.cpython-39-x86_64-linux-gnu.so
00007feb3fa98000 4K r---- _posixsubprocess.cpython-39-x86_64-linux-gnu.so
00007feb3fa99000 4K rw--- _posixsubprocess.cpython-39-x86_64-linux-gnu.so
00007feb3fa9a000 256K rw--- [ anon ]
00007feb3fada000 16K r---- _socket.cpython-39-x86_64-linux-gnu.so
00007feb3fade000 40K r-x-- _socket.cpython-39-x86_64-linux-gnu.so
00007feb3fae8000 36K r---- _socket.cpython-39-x86_64-linux-gnu.so
00007feb3faf1000 4K r---- _socket.cpython-39-x86_64-linux-gnu.so
00007feb3faf2000 4K rw--- _socket.cpython-39-x86_64-linux-gnu.so
00007feb3faf3000 256K rw--- [ anon ]
00007feb3fb33000 8K r---- _blake2.cpython-39-x86_64-linux-gnu.so
00007feb3fb35000 24K r-x-- _blake2.cpython-39-x86_64-linux-gnu.so
00007feb3fb3b000 8K r---- _blake2.cpython-39-x86_64-linux-gnu.so
00007feb3fb3d000 4K r---- _blake2.cpython-39-x86_64-linux-gnu.so
00007feb3fb3e000 4K rw--- _blake2.cpython-39-x86_64-linux-gnu.so
00007feb3fb3f000 12K r---- libz.so.1.2.11
00007feb3fb42000 56K r-x-- libz.so.1.2.11
00007feb3fb50000 24K r---- libz.so.1.2.11
00007feb3fb56000 4K ----- libz.so.1.2.11
00007feb3fb57000 4K r---- libz.so.1.2.11
00007feb3fb58000 4K rw--- [ anon ]
00007feb3fb59000 8K r---- select.cpython-39-x86_64-linux-gnu.so
00007feb3fb5b000 12K r-x-- select.cpython-39-x86_64-linux-gnu.so
00007feb3fb5e000 8K r---- select.cpython-39-x86_64-linux-gnu.so
00007feb3fb60000 4K ----- select.cpython-39-x86_64-linux-gnu.so
00007feb3fb61000 4K r---- select.cpython-39-x86_64-linux-gnu.so
00007feb3fb62000 4K rw--- select.cpython-39-x86_64-linux-gnu.so
00007feb3fb63000 16K r---- _hashlib.cpython-39-x86_64-linux-gnu.so
00007feb3fb67000 24K r-x-- _hashlib.cpython-39-x86_64-linux-gnu.so
00007feb3fb6d000 16K r---- _hashlib.cpython-39-x86_64-linux-gnu.so
00007feb3fb71000 4K ----- _hashlib.cpython-39-x86_64-linux-gnu.so
00007feb3fb72000 4K r---- _hashlib.cpython-39-x86_64-linux-gnu.so
00007feb3fb73000 4K rw--- _hashlib.cpython-39-x86_64-linux-gnu.so
00007feb3fb74000 8K r---- _random.cpython-39-x86_64-linux-gnu.so
00007feb3fb76000 8K r-x-- _random.cpython-39-x86_64-linux-gnu.so
00007feb3fb78000 4K r---- _random.cpython-39-x86_64-linux-gnu.so
00007feb3fb79000 4K r---- _random.cpython-39-x86_64-linux-gnu.so
00007feb3fb7a000 4K rw--- _random.cpython-39-x86_64-linux-gnu.so
00007feb3fb7b000 4K r---- _bisect.cpython-39-x86_64-linux-gnu.so
00007feb3fb7c000 4K r-x-- _bisect.cpython-39-x86_64-linux-gnu.so
00007feb3fb7d000 4K r---- _bisect.cpython-39-x86_64-linux-gnu.so
00007feb3fb7e000 4K r---- _bisect.cpython-39-x86_64-linux-gnu.so
00007feb3fb7f000 4K rw--- _bisect.cpython-39-x86_64-linux-gnu.so
00007feb3fb80000 512K rw--- [ anon ]
00007feb3fc00000 160K r---- libc.so.6
00007feb3fc28000 1492K r-x-- libc.so.6
00007feb3fd9d000 352K r---- libc.so.6
00007feb3fdf5000 4K ----- libc.so.6
00007feb3fdf6000 16K r---- libc.so.6
00007feb3fdfa000 8K rw--- libc.so.6
00007feb3fdfc000 52K rw--- [ anon ]
00007feb3fe0a000 16K r---- math.cpython-39-x86_64-linux-gnu.so
00007feb3fe0e000 28K r-x-- math.cpython-39-x86_64-linux-gnu.so
00007feb3fe15000 16K r---- math.cpython-39-x86_64-linux-gnu.so
00007feb3fe19000 4K r---- math.cpython-39-x86_64-linux-gnu.so
00007feb3fe1a000 4K rw--- math.cpython-39-x86_64-linux-gnu.so
00007feb3fe1b000 768K rw--- [ anon ]
00007feb3fedf000 8K r---- zlib.cpython-39-x86_64-linux-gnu.so
00007feb3fee1000 16K r-x-- zlib.cpython-39-x86_64-linux-gnu.so
00007feb3fee5000 12K r---- zlib.cpython-39-x86_64-linux-gnu.so
00007feb3fee8000 4K r---- zlib.cpython-39-x86_64-linux-gnu.so
00007feb3fee9000 4K rw--- zlib.cpython-39-x86_64-linux-gnu.so
00007feb3feea000 4K r---- _opcode.cpython-39-x86_64-linux-gnu.so
00007feb3feeb000 4K r-x-- _opcode.cpython-39-x86_64-linux-gnu.so
00007feb3feec000 4K r---- _opcode.cpython-39-x86_64-linux-gnu.so
00007feb3feed000 4K r---- _opcode.cpython-39-x86_64-linux-gnu.so
00007feb3feee000 4K rw--- _opcode.cpython-39-x86_64-linux-gnu.so
00007feb3feef000 16K r---- array.cpython-39-x86_64-linux-gnu.so
00007feb3fef3000 24K r-x-- array.cpython-39-x86_64-linux-gnu.so
00007feb3fef9000 16K r---- array.cpython-39-x86_64-linux-gnu.so
00007feb3fefd000 4K ----- array.cpython-39-x86_64-linux-gnu.so
00007feb3fefe000 4K r---- array.cpython-39-x86_64-linux-gnu.so
00007feb3feff000 4K rw--- array.cpython-39-x86_64-linux-gnu.so
00007feb3ff00000 1024K rw--- [ anon ]
00007feb40000000 360K r---- libpython3.9.so.1.0
00007feb4005a000 1748K r-x-- libpython3.9.so.1.0
00007feb4020f000 976K r---- libpython3.9.so.1.0
00007feb40303000 4K ----- libpython3.9.so.1.0
00007feb40304000 24K r---- libpython3.9.so.1.0
00007feb4030a000 220K rw--- libpython3.9.so.1.0
00007feb40341000 136K rw--- [ anon ]
00007feb40366000 4K r---- _heapq.cpython-39-x86_64-linux-gnu.so
00007feb40367000 8K r-x-- _heapq.cpython-39-x86_64-linux-gnu.so
00007feb40369000 12K r---- _heapq.cpython-39-x86_64-linux-gnu.so
00007feb4036c000 4K r---- _heapq.cpython-39-x86_64-linux-gnu.so
00007feb4036d000 4K rw--- _heapq.cpython-39-x86_64-linux-gnu.so
00007feb4036e000 256K rw--- [ anon ]
00007feb403ae000 340K r---- LC_CTYPE
00007feb40403000 12K rw--- [ anon ]
00007feb40406000 52K r---- libm.so.6
00007feb40413000 448K r-x-- libm.so.6
00007feb40483000 368K r---- libm.so.6
00007feb404df000 4K r---- libm.so.6
00007feb404e0000 4K rw--- libm.so.6
00007feb404e4000 28K r--s- gconv-modules.cache
00007feb404eb000 8K rw--- [ anon ]
00007feb404ed000 8K r---- ld-linux-x86-64.so.2
00007feb404ef000 152K r-x-- ld-linux-x86-64.so.2
00007feb40515000 44K r---- ld-linux-x86-64.so.2
00007feb40521000 8K r---- ld-linux-x86-64.so.2
00007feb40523000 8K rw--- ld-linux-x86-64.so.2
00007ffc65d45000 136K rw--- [ stack ]
00007ffc65dab000 16K r---- [ anon ]
00007ffc65daf000 8K r-x-- [ anon ]
ffffffffff600000 4K --x-- [ anon ]
total 45531420K
Environment
$ mdedup --version
mdedup, version 7.5.0
Running on RHEL 9.4 x64.
Additional context
Sorry if my wording was a bit harsh, I'm just totally frightened by those high memory usage numbers.