support for chunked file transfers

[Vssblt]

What happened:

$ /usr/bin/python3 /usr/bin/xpra control :1025 send-file ./test
Warning: XDG_RUNTIME_DIR is not defined
 and '/run/user/1000' does not exist
 using '/tmp'
server returned error code 127
 error processing control command: uncompressed data is too large: 1000MB, limit is 256MB

I use the html5 client and use commands to transfer files.
xpra told me that the file transfer size limit was exceeded.
The file transfer size limit configuration only working less than 256MB.
How can I set the limit to be more than 256MB?

Information:

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.3 LTS
Release:	20.04
Codename:	focal

$ xpra --version
xpra v4.2.2-r0

$ xpra info :1025 | grep transfer
Warning: XDG_RUNTIME_DIR is not defined
 and '/run/user/1000' does not exist
 using '/tmp'
client.file-transfers.ask=False
client.file-transfers.ask-timeout=3600
client.file-transfers.chunks=65536
client.file-transfers.enabled=True
client.file-transfers.open=True
client.file-transfers.open-ask=False
client.file-transfers.open-url=True
client.file-transfers.open-url-ask=False
client.file-transfers.printing=False
client.file-transfers.printing-ask=False
client.file-transfers.remote.file-ask-timeout=0
client.file-transfers.remote.file-chunks=0
client.file-transfers.remote.file-size-limit=10485760000000
client.file-transfers.remote.file-transfer=True
client.file-transfers.remote.file-transfer-ask=False
client.file-transfers.remote.open-files=False
client.file-transfers.remote.open-files-ask=False
client.file-transfers.remote.open-url=True
client.file-transfers.remote.open-url-ask=False
client.file-transfers.remote.printing=True
client.file-transfers.remote.printing-ask=False
client.file-transfers.size-limit=10000000000
features.file-transfer=True
features.file-transfer-ask=False

$ ps aux | grep "xpra start :1025" | grep -v grep
user       50401  1.7  0.3 2835772 216664 ?      SLl  03:23   0:15 /usr/bin/python3 /usr/bin/xpra start :1025 --tcp-auth=password:value=password --daemon=no --start-child=application --exit-with-children

$ cat /etc/xpra/xpra.conf 
tcp-auth=allow
bandwidth-limit=3000Mbps
opengl=no
speaker=off
encoding=jpeg
notifications=no 
system-tray=no 
webcam=no 
mousewheel=on 
min-speed=100
min-quality=88
auto-refresh-delay=0 
title="title" 
mdns=no 
exit-with-children=yes 
video-scaling=off
exec-wrapper=/opt/VirtualGL/bin/vglrun -d :5 --
env=XPRA_CLOSE_GTK_DISPLAY=0 XDG_RUNTIME_DIR=/m/runtime/xdg-start
sharing=yes
server-idle-timeout=30
idle-timeout=10800
terminate-children=yes
dbus-launch=no
bandwidth-detection=no
file-size-limit=10000M
clipboard=default
printing=no

[totaam]

Just tested with git master and a 280MB file.
With the default settings, it is rejected.
Starting my server with:

xpra start --start=xterm  -d file --file-size-limit=500M

Then the client with:

xpra attach --file-size-limit=500M

The transfer succeeds.

---

Trying with the html5 client: the default limit is 10MB there. How did you change that?
What else did you modify?

Bumping this value to 500 helps, but then we hit a different issue:

2021-12-01 12:20:29,668 error processing control command 'send-file'
Traceback (most recent call last):
  File "/usr/lib64/python3.10/site-packages/xpra/server/server_core.py", line 712, in process_control_command
    v = command.run(*args[1:])
  File "/usr/lib64/python3.10/site-packages/xpra/server/control_command.py", line 68, in run
    return super().run(*args)
  File "/usr/lib64/python3.10/site-packages/xpra/server/control_command.py", line 32, in run
    return self.do_run(*args)
  File "/usr/lib64/python3.10/site-packages/xpra/server/mixins/server_base_controlcommands.py", line 295, in control_command_send_file
    return self.do_control_file_command("send file", client_uuids, filename, "file_transfer", (False, openit))
  File "/usr/lib64/python3.10/site-packages/xpra/server/mixins/server_base_controlcommands.py", line 351, in do_control_file_command
    ss.send_file(filename, "", data, file_size, *send_file_args)
  File "/usr/lib64/python3.10/site-packages/xpra/net/file_transfer.py", line 724, in send_file
    self.do_send_file(filename, mimetype, data, filesize, printit, openit, options)
  File "/usr/lib64/python3.10/site-packages/xpra/net/file_transfer.py", line 885, in do_send_file
    cdata = self.compressed_wrapper("file-data", data)
  File "/usr/lib64/python3.10/site-packages/xpra/server/source/client_connection.py", line 147, in compressed_wrapper
    return compressed_wrapper(datatype, data, can_inline=False, **kw)
  File "/usr/lib64/python3.10/site-packages/xpra/net/compression.py", line 196, in compressed_wrapper
    raise Exception("uncompressed data is too large: %iMB, limit is %iMB" % (sizemb, maxmb))
Exception: uncompressed data is too large: 269MB, limit is 256MB

That's because the lz4 encoder has a hard-coded limit and it should not be changed.
The html5 client needs to implement chunked file transfers instead: Xpra-org/xpra#1026 (comment)

[Vssblt]

Sorry, I forgot that I changed the maximum file transfer size of the HTML5 client and set a very large number. You reproduced the problem correctly.
Does new version support for chunked file transfers? I used a very old xpra html5 version. v4.1

[totaam]

Does new version support for chunked file transfers?

No. This needs to be implemented.

[totaam]

The client can now send and receive chunked files of arbitrary size without causing connection issues.

Still TODO:

• make it faster - somehow the html5 client takes too long to parse those data packets - the python client is faster, not sure why
• increase the chunk size? (already 64KB) allow the client to choose bigger chunks? only for bigger files? (supporting multiple chunks in flight would be much harder)
• using zero-copy buffer transfers from the network worker would help a bit
• sha256 checksums:

x = forge.md.sha256.create()
x.update("abc")
x.digest().data

• use a dedicated logging category for debugging
• add a progress indicator somewhere

[totaam]

Now using StreamSaver.js!

As per the docs, we should probably register an onunload and / or onbeforeunload handlers.