Blobfuse2 (version 2.5.0) memory/swap issue

[samy44444]

Currently, we have a BLOB storage account in Azure with six containers to store custom application data, with some containers containing files and remaining containers are empty. There files are uploaded to the storage account through Storage Explorer. In the future, the setup is expected to scale to hundreds of containers. During testing, after mounting the containers using blobfuse2 on Ubuntu Linux VM on Azure, the memory and swap usage consistently increase to around 80% and above, even on a test machine where no users are connected, no files are uploaded, and no processes are running. Once the containers are unmounted and the machine is rebooted, memory usage decreases to around 5–6%. To validate this behavior, we used the sar -r command to monitor memory/swap utilization.

Also, instead of mounting all 6 containers through /etc/fstab entries, if we manually mount container by container, the memory + swap space usage goes by 8.5% per container.

We created a ticket with Azure support initially. After troubleshooting for a few days, Azure support team came back and suggested us to create a issue with github.

We would like to find out the RCA on why the memory spikes up to 80% with only few files in the container.

-If further tweaks can be done to relief the memory footprint?

-Whether mounting 100 containers on the same VM is a feasible use case?

[vibhansa-msft]

Kindly share the mount cli and config file that you are using.

[samy44444]

/etc/fstab entry:

/root/BLOBALL_MOUNT.sh /root/BLOBALL fuse3 defaults,_netdev 0 0

The content of /root/BLOBALL_MOUNT.sh is listed below

#!/bin/bash
sudo blobfuse2 mount all /root/BLOBALL --config-file=/root/BLOBALL.yaml --foreground=false

[samy44444]

Please note that we are testing this in the DEV machine when no users are connect.

[samy44444]

We started with the following entries in the config file first

---

allow-other: true

logging:
type: syslog
level: log_debug

components:

• libfuse
• stream
• attr_cache
• azstorage

libfuse:
attribute-expiration-sec: 120
entry-expiration-sec: 120
negative-entry-expiration-sec: 240

stream:
block-size-mb: 8
blocks-per-file: 3
cache-size-mb: 256

attr_cache:
timeout-sec: 600

azstorage:
type: block
account-name:
mode: msi
endpoint: https://.blob.core.windows.net/
container:

file_cache:
path: /tmp/blobfuse2tmp
timeout-sec: 120
max-size-mb: 1024

[samy44444]

Microsoft support team also asked us to try by deleting the following entry but it did not make any difference.
attr_cache:
timeout-sec: 600

[samy44444]

We also tried to test by changing the value for "cache-size-mb" from 256mb to 64mb, but it did not make any difference.

[vibhansa-msft]

"sudo blobfuse2 mount all " how many containers in one go you are trying to mount here ?

[vibhansa-msft]

Can you try with below config once and let me know if this works:

logging:
type: syslog
level: log_debug

components:

• libfuse
• block_cache
• attr_cache
• azstorage

block_cache:
block-size-mb: 16
mem-size-mb: 128
prefetch: 12
parallelism: 24

azstorage:
account-name:
mode: msi
container:

[samy44444]

1. When the server reboot, it will mount 6 containers at the same time. Some containers have files and some don't
2. Am I supposed to put the correct values under account-name? Is there any other value I need to put in the config file?

[samy44444]

Just to make sure everything is in order, can you put your recommendation of the config entries in a file and attach here so I don't accidently miss out anything?

[samy44444]

I'm getting the following error.

Failed to mount container : Error: failed to initialize new pipeline [config error in block_cache [memory limit too low for configured prefetch]]

Here is the updated config based on your input.

[samy44444]

logging:
type: syslog
level: log_debug

components:

• libfuse
• block_cache
• attr_cache
• azstorage

block_cache:
block-size-mb: 16
mem-size-mb: 128
prefetch: 12
parallelism: 24

azstorage:
account-name:
mode: msi
container:

[samy44444]

It is NOT mounting even after I reboot. I also get the same error when I try to mount manually.