Replies: 1 comment 8 replies
-
|
Earlier "direct_io" flag was not being honoured properly by blobfuse. Ideally that flag means all caching at kernel and blobfuse level shall be disabled and all calls shall be forwarded directly to storage and that's what its doing now. If there is no reason for you to use that flag, kindly remove that from your config and it shall work fine. Assuming it was anyway not working properly it might be good idea to remove it and may be configure file-cache-timeout to 0 so that contents are not cached by blobfuse post you close the file. Linux kernel may still cache the contents and keep serving you from its own cache. |
Beta Was this translation helpful? Give feedback.
-
|
what makes me think that it is not really blobfuse side, when mounted with direct-io enabled:
|
Beta Was this translation helpful? Give feedback.
-
|
Generally when kernel lists we provide the list of files with few attributes. Kernel can optionally request properties of each file (one call per file) as metadata cache is also disabled with direct_io, each such call will result into a storage call, hence higher number of calls to Azure. |
Beta Was this translation helpful? Give feedback.
-
|
Again, take a step back and first evaluate whether you really need the direct_io flag or not. |
Beta Was this translation helpful? Give feedback.
-
|
hey again, so whether I need direct_io or not is one question, that I don't think I can answer per say, what I need is the ability to change files mounted via blobfuse and reflect those changes in other clients (including other blobfuses, but also rclone and azure storage explorer). This part worked fine withoout the direct_io and with a sensible cache limit (I don't think we are looking for hard "imediat" but good enough - we opted for 10 seconds timeouts for cacehs and attributes both on the blobfuse level as well as the kernel level The issue was when writing data from other clients for a file, those would never be reflecte in a file that was prior mounted with blobfuse after removing the direct_io flag, even whem setting any of the timeouts to 0. What seems to have fixed the isse is enabling the disable-writeback-cache flag. No, I am very much at the limit of my knowledge in regards to how the kernel works, but I cannot exactly yet pinpoint why it works, maybe you could explain that a bit? I can see the same-sh beahviour in this issue, where you recommend the same approach: Now what I am interested is not an explanation of what happens in the libfuse when that flag is set to true - while I can bet it would be a nice reading, most probably you have better things to do, but rather on what are some potential gotchas and consequeses that we might encounter by enabling this flag? |
Beta Was this translation helpful? Give feedback.
-
|
direct_io flag is one that configures kernel to not cache any data or meta for the files coming from a given mount path. This means for every call it needs to talk to backend file-system driver (in this case its blobfuse). |
Beta Was this translation helpful? Give feedback.
-
Hi, I am quite sure that this is not where the issue is, but I will still give it a shot.
We have a setup where we use blobfuse2 (via blobfuse-drive-csi) to access data in azure blob containers (flat) via jupyter notebooks.
For the past 2-3 years it has been working fine, for us it is important to reflect changes quickly, and from different clients, thus our mount options look like so:
which was working well for us.
Recently we started seeing quite a performance hit, very noticeable, and a siginificant increase in transactions (without increasing the number of blobs per say).
What we noticed is the following, when looking at the logs while accessing a "folder" we can see 25+ requests for a given item to list:
example trace:
example debug:
is this normal or something wrong? And what can we investigate more - we have not touched the jupyter side of the setup, and it is controlled via ArgoCD so no changes should have happened there. We were using the managed service for the blobfuse from AKS, since the issue occured we changed to the helm deployment so we can control better the versions and upgraded to the newset chart, and blobfuse 2.4.1 with the issue persisting.
Tried reversing back both on the blobfuse-csi and blobfguse itself (all the way back to 2.2.x) without any changes.
Any suggestion (gut feelings etc.) are welcomed!
Beta Was this translation helpful? Give feedback.
All reactions