Stale file in blobfuse2 after mounting

[surrealmemoir]

Which version of blobfuse was used?
Blobfuse2 2.5.0~preview.1

Which OS distribution and version are you using?
Ubuntu,
20.04.5 LTS

If relevant, please share your mount command.
/usr/bin/blobfuse2 mount ${MNTDIR} --container-name=$1 --attr-cache-timeout=0 --config-file=$CONFIG

What was the issue encountered?
File is "stuck" with the first version after mounting. Aterwards, it does not update to any newer version coming from the storage account.

Have you found a mitigation/solution?
So far, only either 1) remount or 2) sudo sysctl -w vm.drop_caches=3 can solve it

config.yaml

allow-other: true
logging:
type: base
level: log_debug
file-path: /home/username/mount_log/data.log
libfuse:
attribute-expiration-sec: 20
entry-expiration-sec: 240
azstorage:
type: adls
account-name: xx123xx
endpoint: https://xx123xx.dfs.core.windows.net
mode: msi
update-md5: false
validate-md5: false
virtual-directory: true
max-results-for-list: 0
disable-compression: false
telemetry: ""
honour-acl: false
cpk-enabled: false
cpk-encryption-key: ""
cpk-encryption-key-sha256: ""
preserve-acl: false
filter: ""
user-assertions: ""
components:
- libfuse
- azstorage

Please share logs if available.

Below are the operations carried out: after a fresh mount. First ls the file sss.txt.
-rw-r----- 1 username username 74 May 1 16:18 sss.txt

Then go to another machine and modify the file. Save, I can verify that the file is now
-rw-r----- 1 username username 84 May 1 17:01 sss.txt

Back to the original machine, the file stays stale unless I remount or flush kernel by "sudo sysctl -w vm.drop_caches=3". Logs excerpt are the following. It's able to see that the file size is now 84 instead of 74, but does not download it.

Thu May 1 17:04:11 EDT 2025 : blobfuse2[29476] : [/mcmnt/synapse-data] LOG_DEBUG [utils.go (162)]: SDK(Response) : ==> REQUEST/RESPONSE (Try=1/6.953398ms, OpTime=6.975498ms) -- RESPONSE RECEIVED
HEAD https://xx123xx.blob.core.windows.net/synapse-data/AllData%2Fbarra%2Fusfast%2Fbackup%2Ftest_estu%2Fsss.txt
Accept: application/xml
Authorization: REDACTED
User-Agent: Azure-Storage-Fuse/2.5.0~preview.1 (Ubuntu 20.04.5 LTS) azsdk-go-azblob/v1.6.1 (go1.24.0; linux)
x-ms-version: 2025-05-05

---

RESPONSE Status: 200 OK
Accept-Ranges: bytes
Content-Length: 84
Content-Md5: REDACTED
Content-Type: application/octet-stream
Date: Thu, 01 May 2025 21:04:11 GMT
Etag: "0x8DD88F349B28B7C"
Last-Modified: Thu, 01 May 2025 21:01:05 GMT
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
X-Ms-Access-Tier: Hot
X-Ms-Access-Tier-Inferred: true
X-Ms-Acl: user::rw-,group::r--,other::---
X-Ms-Blob-Type: BlockBlob
X-Ms-Creation-Time: Wed, 30 Apr 2025 19:43:24 GMT
X-Ms-Group: $superuser
X-Ms-Lease-State: available
X-Ms-Lease-Status: REDACTED
X-Ms-Owner: 442b2576-f515-447a-9900-c8311206e92b
X-Ms-Permissions: rw-r-----
X-Ms-Request-Id: 0c03c99f-401e-004c-0edc-ba429f000000
X-Ms-Resource-Type: file
X-Ms-Server-Encrypted: REDACTED
X-Ms-Version: 2025-05-05

Thu May 1 17:04:11 EDT 2025 : blobfuse2[29476] : [/mcmnt/synapse-data] LOG_DEBUG [utils.go (162)]: SDK(Retry) : response 200
Thu May 1 17:04:11 EDT 2025 : blobfuse2[29476] : [/mcmnt/synapse-data] LOG_DEBUG [utils.go (162)]: SDK(Retry) : exit due to non-retriable status code
Thu May 1 17:04:11 EDT 2025 : blobfuse2[29476] : [/mcmnt/synapse-data] LOG_TRACE [libfuse_handler.go (728)]: Libfuse::libfuse_open : AllData/barra/usfast/backup/test_estu/sss.txt, handle 7
Thu May 1 17:04:11 EDT 2025 : blobfuse2[29476] : [/mcmnt/synapse-data] LOG_TRACE [libfuse_handler.go (805)]: Libfuse::libfuse_flush : AllData/barra/usfast/backup/test_estu/sss.txt, handle: 7
Thu May 1 17:04:11 EDT 2025 : blobfuse2[29476] : [/mcmnt/synapse-data] LOG_TRACE [libfuse_handler.go (861)]: Libfuse::libfuse_release : AllData/barra/usfast/backup/test_estu/sss.txt, handle: 7
Thu May 1 17:04:11 EDT 2025 : blobfuse2[29476] : [/mcmnt/synapse-data] LOG_TRACE [azstorage.go (412)]: AzStorage::CloseFile : AllData/barra/usfast/backup/test_estu/sss.txt

[vibhansa-msft]

If doing 1sudo sysctl -w vm.drop_caches=3` solves the issue for you then it means kernel has cached the contents and requests are not even reaching to blobfuse. To configure blobfuse in a way where kernel is not in caching mode use "-o direct_io" option on your mount cli. But be-aware that this means neither kernel nor Blobfuse will have any cache and may result into higher network calls to backend storage and higher billing due to same.

[surrealmemoir]

The issue is that I'm seeing the file stay stale long after the various timeout parameters are passed. I'm wondering why is the kernel cached "locked".

If we don't use direct_io, do you think the following cache timeout parameters should ideally all match?

libfuse.attribute-expiration-sec
libfuse.entry-expiration-sec
file_cache.timeout-sec
attr_cache.timeout-sec

I'm starting to suspect that if these are not the same, an ls call may trigger a meta update but not a file content update, thus "locking" the file to be a stale state, perhaps with an updated meta file info but outdated file content.

[vibhansa-msft]

None of the cache parameters control how long kernel cache will keep the data, so only way around is to keep manually clearing the kernel cache.

[vibhansa-msft]

Closing this as this is all by design. Feel free to post here in case you have any further queries on this.