WebDAV upload produces empty files for one provider

[albrechtd]

I use DAVx5 v. 4.3.2-ose on the current GrapheneOS, with inter alia two WebDAV accounts. For one provider (NetCologne.de), this works perfectly, for both downloading and uploading files.

The other account is a free one from Deutsche Telekom (magentacloud.de). Here, I can download files to the mobile phone without problems. However, all uploaded files end up with 0 bytes in the cloud folder. I am quite sure that data has actually been uploaded, as the operation took the “usual” time and the progress bar indicated activity. In principle, upload does work for this provider, e.g. from a Linux XFCE desktop using the standard Thunar file manager. I guess this is a bug in the Telekom cloud (it doesn't implement the RFC's properly for CardDAV and CalDAV either), but as Thunar is able to upload files properly, it seems that it is “somehow” possible to work around them. Any idea how this issue could be fixed? Any input I could provide for help?

Thanks in advance, Albrecht.

[rfc2822]

Hi,

Can you maybe a test account (maybe app-password or set a password for us and then reset it)? So that we can have a look at it.

[albrechtd]

Hi,

thanks a lot for your response! There are two options:

1. As the Telekom Email and Cloud is a free service, you could create a test account yourself here. Note that you have to activate WebDAV separately using the web mailer, and you'll get a different password for the DAV services. It's in German, though, no idea if/how you could switch to a different locale…
2. If you really want to test with my account, please tell me an Email address (and a GPG public key if possible) to which I could send my credentials (not a good idea to post them here IMHO…).

Thanks, Albrecht.

[rfc2822]

I had a look and could reproduce your problem.

Unfortunately, DAVx5 doesn't know the file size in prior of the upload. So okhttp (the HTTP library that DAVx5 uses) uploads the file without naming a file size and using chunked transfer encoding (or its HTTP/2 equivalent: data frames). This is correct and must be supported by servers.

The problem can be reproduced with curl:

curl -H 'Transfer-Encoding: chunked' -H 'Content-Type: image/png' --compressed -vu [email protected]:webdav-app-pw -T some-image.png https://magentacloud.de/remote.php/webdav/some-image.png      # note that Transfer-Encoding: chunked is not actually sent for HTTP/2
> PUT /remote.php/webdav/some-image.png
…
< HTTP/2 204 

(Or for HTTP/1.1 with the additional argument --http1.1, which then actually sends Transfer-Encoding: chunked)

Note that 204 No content is returned instead of 200 OK, and then the file has no content.

The MagentaCloud server seems to have a problem with Content-Transfer: chunked and creates a 0-byte sized file in this case. This is a server-side problem and unfortunately we can't easily work around it because we really don't know how many bytes will be fed into our upload stream by the uploading Android app (basically the app you use to upload the file, for instance a file manager, but could be another app, too).

Can you please report that to MagentaCloud?

[albrechtd]

Thank you so much for your explanation! I fully agree with you that this behavior is a violation of RFC 9112, Sect. 6.2 and 7.1. As I mentioned above, that server has more flaws. E.g. it explicitly advertises support for the addressbook-query (RFC 6352) REPORT, but always returns an empty result with status 207, no matter which filters I add, or even without any filters. Otoh, addressbook-multiget does work… (the not very obvious URI for CardDAV and CalDAV is https://spica.t-online.de, btw., if you want to try yourself).

Unfortunately (and maybe because it's for free), Deutsche Telekom has no means to report bugs, just a “Support Forum”. I posted the issue there (in German, sorry), and keep you updated if I receive any reply.

[albrechtd]

Update: two users in the support forum mentioned that the MagentaCloud would be based on NextCloud. As I don't have a different NextCloud account (afaik), I installed the latest version (26.0.1) locally in a VM. Uploads to it using DAVx5 work flawlessly, whilst one user could confirm the issue with MagentaCloud. Thus, iff it is really based on NextCloud, the Telekom either modified the code and/or uses an outdated or buggy version.

[rfc2822]

I think it could be related to Web server/intermediate proxy servers and/or their settings.

[muelleel]

Hi,
running into the same problem - just with an different nextcloud instance (also most likely an proxy server problem).

There must be a working workaround for this since other webdav clients just work fine. Does anybody know how theese working clients operate under the hood? For example what is the Thunar file manager sending behind the scenes that the upload is working correctly?

[Kalawit]

Good day Sir/Mesdames! My DavX5 stooped working. Im using it to sync contacts & calendar from my huawei nova 7 to google . I hope you could help me... i used the https://www.google.com/calendar/dav/my google account/events asmy Base URL

[exstntlmsnthrp]

I think this is related to #337.

[dktzde]

Unfortunately, the problem persists and it seems that it will take some time before Telekom may (is willing to) find a solution.

Hence the following suggestion:

• Point out on the website: https://www.davx5.com/tested-with/t-online that there is a problem with WebDAV.
• Display a warning - or prevent the upload when a mangentacloud.de account is added.

PS: I have had to accept a loss of data myself due to this error. See there: seedvault-app/seedvault#622

[albrechtd]

Unfortunately, the problem persists and it seems that it will take some time before Telekom may (is willing to) find a solution.

To be honest, I seriously doubt that they are willing to fix the bug. The service is free, and as long as it works with other WebDAV software (which may or may not be the case, but I presume that more users would complain if this wouldn't be the case; e.g. my XFCE Linux Desktop using the gvfs backend does create proper uploads), the pressure for investing any time (=€€€) into this is low at best.

You can try to either add your remarks to my issue on their (pardon my French) crappy „Help Desk” site, or open a new one. I'm afraid the result will be frustrating, though.

Point out on the website: https://www.davx5.com/tested-with/t-online that there is a problem with WebDAV.

Yes, see my comment above!

Display a warning - or prevent the upload when a mangentacloud.de account is added.

As the Telekom probably uses some (half-broken) off-the-shelf solution, there might be other providers with the same issue. I'm not familiar with RFC 4918, but it might be possible to read back the size of the file created on the remote server (which shouldn't be too costly). If they differ, the upload failed, and at least at this point an error message could be produced (i.e. you wouldn't get the impression that your backup has been successful).

BTW, the Telekom server has other bugs too – for CardDAV (RFC 6352), it claims to support addressbook-query requests, but doesn't return anything, i.e. the (more costly) addressbook-multiget request must always be used instead.

[rfc2822]

https://www.davx5.com/tested-with/t-online updated

[gnoutchd]

I think I just discovered this issue on my own personal, self-hosted NextCloud server (v27.1.17). Uploading large-ish (~200KiB) files via a davx5 WebDAV mount results in zero-length files on the server. No errors are reported on the client when this happens. Both SeedVault and the Android file manager can trigger this bug.

This is pretty bad, my SeedVault backups were destroyed by this. I think we have to assume that we cannot safely write files via a davx5 WebDav mount to any NextCloud or NextCloud-based server.

[gnoutchd]

Ah, I see this is indeed a longstanding NextCloud issue: nextcloud/server#7995

Some deployments have workarounds in place, but I'm using the Snap package which presumably does not: nextcloud-snap/nextcloud-snap#365

[tcecyk]

just a heads up: rclone wrappers like RoundSync and RSAF do the webdav-mount writes fully for Nextclouds in nginx-fastcgi-php setups (vs Apache, that can be made to handle the chunked transfers).

RS will ask for full file access permissions, RSAF creates a vfs for rclone using a Go-written rcbridge, but also see its Limitations. Seedvault can now use RS as backend.

By my laymans reading they work because they don't do streaming writes vs. Davx5 that does if a few conditions aren't met: as for new writes ETag/last-modified is unknown, they'll always fall into streamingFileDescriptorFactory instead of openProxyFileDescriptor. Not sure if there's room for flexibility or my take is misguided.