Blobfuse2 2.5.0 Change to How Re-mount is Handled

[michaelschmit]

We might be an edge case, but we have hundreds of containers we mount up and that list is constantly changing. We periodically run a mount all to add new containers that have been added to our storage account. Previously this worked fine because only the new containers would be mounted and any existing containers failed (skipped essentially). On upgrade from 2.4.2 to 2.5.0 this functionality changed to an unmount and remount which takes significantly, significantly longer. So much so we had to downgrade to 2.4.2 until we can get something else figured out.

Do you have a suggestion about how to handle this scenario post 2.5.0?

Thanks !!!

[syeleti-msft]

Hi @michaelschmit, In the container list that you are maintaining, are all the containers valid(i.e., every container is present in the storage acnt) in that list when you are mounting every time? Also may I know how much more time it is taking in the new release?
Are you also seeing unmount and remount of running mountpoints?

[michaelschmit]

Again, we aren't maintaining a list today since new containers are being added constantly. Instead, we use the blobfuse2 mount all. Generally we are only adding containers, but occasionally a container or two might go away as well.

In regard to timings, on release 2.4.2 running the mount all command executes in 1m39s. Since most of the containers are already mounted. Depending on how many new containers are added, it can take a few seconds or minutes longer.

On release 2.5.0, the amount of time to run the mount all command jumps significantly. On the order of 10-30 seconds per container (Again we are mounting ~780 containers today), due to the attempt to unmount and remount I imagine.

In regards to also seeing unmount and remount of running mount points. It seemed as though the unmount and remount is still failing with the error:

Failed to mount container [container name] : Error: directory is already mounted

[michaelschmit]

Just checking in to see if you need any additional debug information. Thanks !!!

[jainakanksha-msft]

@michaelschmit, we made an enhancement with our mount all feature.
Now, whenever it mounts all containers, it first checks if any container are already mounted, unmounts them gracefully, and then proceeds to mount everything.
This update is intended to make the product more stable, ensuring that mounting does not fail.
Could you please provide more details about your use case? This will help us recommend the best way for you to perform this process.
Also, why do you need to run mount all so frequently? Keep in mind, the application's initial bootup time might increase slightly.

[michaelschmit]

Why not check if the container is already mounted and do nothing, rather than error? I think the unmount and remount is really only a scalable solution for a few containers which means it is not suitable for mount all.

I think I did a good job describing our use case already, but I will do it again. We have hundreds of containers and that list is constantly changing, and generally growing. These containers are being created dynamically by other code in our solution. When a container gets added, in a short amount of time it needs to be available on the file system. Rather than trying to maintain both a list of directories currently mounted and the new ones that need to be mounted by an outside process, we use the mount all command (as I think mount all was intended).

Beyond that, what I don't understand is why I am still seeing failures even after the change to unmount and remount. The containers still fail with the error:

Failed to mount container [container name] : Error: directory is already mounted

Is that a bug in the logic?

I really think some consideration should be made for solutions with hundred of containers. If it is not a viable solution for hundreds of containers, then it really isn't a scalable solution.

[jainakanksha-msft]

@michaelschmit ,
I’d like to correct my earlier statement. The error:
Failed to mount container [container name] : Error: directory is already mounted
is expected when a container is already mounted. In this case, Blobfuse simply skips re-mounting during the next mount all execution.
I verified your scenario with a large and dynamically growing set of containers.
In this case, Blobfuse correctly mounts only the new containers, while the already-mounted ones are skipped (with the above message). From my testing:

• Mounting an empty container typically takes <1s.
• If a container has a large number of files, mount time increases since the kernel performs a directory listing (list call).
• To mitigate this and avoid mount latency being impacted by the number of files, you can use the block-list-on-mount-sec attribute in your configuration (e.g., set to 10s).
• Validation of existing container mounts usually takes ~100ms.

So, the overall performance should scale reasonably well, but it’s worth evaluating at your end based on your specific workload.

If you still observe that mount all is taking longer than expected, could you please share logs for two consecutive mount all operations? We’d be happy to analyze them further to identify any bottlenecks.

[michaelschmit]

I'm seeing a significant increase in the amount of time before a container fails with the: `Failed to mount container [container name] : Error: directory is already mounted` error message on 2.5.0 vs 2.4.2. Is there a particular log I should look at? Thanks.

[vibhansa-msft]

You can enable debug logs and that can help us understand why it took time. If there are large number of containers being mounted then iterating through them all and identifying current directory is mounted or not, will be time consuming. Also, if there are large number of mounts running already your system will be under stress with low free memory and high cpu usage already. Kindly validate these parts and let us know if you see something else here.

[vibhansa-msft]

Closing this as there is no update. Feel free to reopen when you have required info.