zpaq over ssh?

[HolyOne]

Hello, first of all this is an awesome fork.
I am searching for a solution to create zpaq archive on my local PC containing the the folders on remote PCs. (I dont want to create archives on remote pcs and download them, since it would take long time and I cannot guarantee remote disk sizes)
Is there a way I can target a remote ssh/powershell directory with zpaqfranz ?

or is there a better suggested way for it?

Thanks

[fcorbelli]

It is possible to operate SMB over ssh to store data between two Windows machines
But it's essentially unrealistic (unless you have very fast internet connections)

The way I recommend to transfer a (large) backup between two workstations over the internet, with a speed of a few MB / s, is to store the .zpaq file locally, and then transfer it

It's a practice I've been doing for years, it works

There are various configurations for rsync

Source PC: Windows
PC Backup: Windows

Source PC: *nix
PC Backup: Windows

Source PC: Windows
PC Backup: *nix

Source PC: *nix
PC Backup: *nix

I can give you instructions for each of these

Recap: on the source PC you archive data with zpaqfranz locally (or mounted, for *nix)
Then you can SEND (push) the data to "something else" (aka: starting from source=>backup) or GET (pull) data (backup=>source)
Typically for crontabbed executions

This is an example of Windows GETTING data from a *NIX server
In this case
1.2.3.4 is the IP of the *nix (source PC)
23456 is the ssh port (default: 22) of *nix
franco is the *nix user
/monta/nas1_nfs/thepaqfolder is the *nix folder with the .zpaq(s) to be copied

c:\cloud\thekey.rsa is the RSA key of the *nix PC stored in the Windows PC
h:\mylocalbackup is the Windows PC destination folder

@echo off
SETLOCAL
SET CWRSYNCHOME=c:\cloud
SET PATH=%CWRSYNCHOME%\BIN;%PATH%

rsync -e "ssh -p 23456 -i /cygdrive/c/cloud/thekey.rsa" -r --append --no-perms --no-owner --no-group --progress -vv --delete franco@1.2.3.4:/monta/nas1_nfs/thepaqfolder /cygdrive/h/mylocalbackup

[fcorbelli]

Windows PUTTING (sending) backup to a *nix, over ssh
In this example

Windows
k:\test\cloud\vari.zpaq is the local archive (with mygoodpassword)
"Things" to be copied are c:\nz c:\cloud c:\dropbox c:\zpaqfranz*
In c:\cloud\backup_franco.rsa there is the RSA *nix key (of the franco user)

Nix
The *nix user is franco
franco@backup.somewhere.com

The *nix (the backup machine) is (better choose numeric IP)
franco@backup.somewhere.com

In this case default port (22)

The destination folder (on the *nix) is
/home/franco/copie

c:\cloud\zpaqfranz a k:\test\cloud\vari.zpaq c:\nz c:\cloud c:\dropbox c:\zpaqfranz\* -key mygoodpassword 
rsync -e "c:\cloud\ssh -i /cygdrive/c/cloud/backup_franco.rsa"      -I  --append --partial -r --progress -vv --chmod=a=rwx,Da+x --delete /cygdrive/k/test/cloud franco@backup.somewhere.com:/home/franco/copie

[fcorbelli]

*Nix (FreeBSD) putting to *Nix (via internet or LAN, ex. a Linux-based NAS)

Source PC
Local zpaq archive
/zroot/ssd/copie/paqqa/thearchive.zpaq
"things" to be copied /tank/dati

Destination "thing" (*nix)

/home/mybackup/copia (destination folder)
port, user, server and key in the example script

/usr/local/bin/zpaqfranz a /zroot/ssd/copie/paqqa/thearchive.zpaq /tank/dati  

PORTA=22
UTENTE=franco
SERVER=1.2.3.4
CHIAVE=/root/script/backup_franco.rsa

/usr/local/bin/rsync  -I --append --omit-dir-times --no-owner --no-perms --partial --progress -e "/usr/bin/ssh -p $PORTA -i $CHIAVE "  -rlt  --delete "/zroot/ssd/copie/paqqa" "$UTENTE@$SERVER:/home/mybackup/copia"

I hope I didn't make any mistakes :)
However, this is how it works

Then there is the version with zfs replicas, but I don't think they interest you

[fcorbelli]

It is quite common (in an enterprise environment) to have TWO different connections, one for internet use, one for backups.
In this case, specific outbound routes are used

In the home environment, on the other hand, it is common to limit (with --bwlimit = something for example) the upload bandwidth in order not to excessively slow down the use of the internet surfing.

In real-world scenario it is common to deal with (as in this example) 380GB taking the last day delta in about 3-5 minutes (at 2.40MB/s upload bandwidth, a cheap VDSL)

C:\cloud>prendicorsa
opening connection using: ssh XXX
receiving incremental file list
delta-transmission enabled
paqqa/copiacorsa.zpaq
382417454978  99%    2.40MB/s    0:03:02

Of course the faster the internet connection, the better

The "real world" example archive about 18 months of daily backups

copiacorsa.zpaq: 
577 versions, 621.783 files, 5.929.571 fragments, 46.563 blocks, 346.029.121.619 bytes (322.26 GB)
-------------------------------------------------------------------------
< Ver  > <  date  > < time >  < added > <removed>    <    bytes added   >
-------------------------------------------------------------------------
00000001 2020-03-01 14:39:07  +00113826 -00000000 ->      102.924.764.088
00000002 2020-03-02 04:12:22  +00000043 -00000000 ->            8.937.922
(...)
00000576 2022-06-21 19:41:27  +00001188 -00000355 ->          393.696.536
00000577 2022-06-22 19:41:23  +00000629 -00000015 ->          272.684.238

27.922 seconds (000:00:27)  (all OK)

[fcorbelli]

In the attached file an example of 1-hour backup sended with ssh of a mid-sized fileserver
The more frequently you send the data, the faster you will complete (because the data is scarce)
1.txt

Yesterday : ransom safe @ 1 hour interval

00003192 2022-10-12 09:32:01  +00000572 -00000001 ->           64.495.693
00003193 2022-10-12 10:32:00  +00000561 -00000000 ->           28.853.453
00003194 2022-10-12 11:32:02  +00000513 -00000077 ->           53.710.535
00003195 2022-10-12 12:32:00  +00000620 -00000000 ->          119.186.488
00003196 2022-10-12 13:31:59  +00000421 -00000000 ->            1.689.430
00003197 2022-10-12 14:31:59  +00000431 -00000001 ->            2.853.120
00003198 2022-10-12 15:31:59  +00000569 -00000006 ->           39.825.057
00003199 2022-10-12 16:32:01  +00000501 -00000009 ->           29.757.724
00003200 2022-10-12 17:32:03  +00000521 -00000002 ->           30.779.030
00003201 2022-10-12 18:32:02  +00000511 -00000003 ->           23.649.516
00003202 2022-10-12 19:32:14  +00000458 -00000004 ->            6.382.318
00003203 2022-10-12 20:32:23  +00000418 -00000000 ->              968.181

[HolyOne]

Thank you for the detailed answer!

They are good solutions with what are available but

1. I cannot guarantee there is space in remote PC
2. I lose a lot of time by that way
3. In case of I need an encrypted zpaq, I may have to transfer the password to remote pc

Wouldnt be easier and more efficient if zpaqfranz had an additional parameter or support destination folder paths like ftp://domain.com/folder or ssh://user@domain.com/folder ?

[fcorbelli]

1. This in fact is mandatory
2. No, you run much faster (this way)
3. In fact no

"be easier": no, it is much harder

It is possible to mount something (even Windows SMB share) over ssh
I am thinking on a full client-server implementation, but requires some work

[fcorbelli]

It is much harder than expected (nothing is easy with Mahoney's code, afterall :)
The very first zpaqfranz-over-socket
Do not support encrypted (yet) and indexes (yet), neither versions (yes)

C:\zpaqfranz>zpaqfranz a z:\1.zpaq c:\dropbox\Dropbox c:\nz -server

become...

zpaqfranz server v0.01
00089: Using Winsocket version 2.2.

0113: My listening infos...
Way # 1:
        IPv4
        Protocol:       TCP
        Sockettype:     Stream
05921: Socket:          OK
05929: Socket bind:     OK
05938: listening:       OK

Waiting connections...
05965: ACCEPTED!
k1
k2
k3 0
New bytes read
k3 0
Ricostruzione 82
New bytes read
IL packedsise vale 82
05978: RECEIVED FIRST PACKED with bytesreceived 80
----------- inizio decode
Decodifico il buffersize da headerino letto 80
OK hash 777192
vediamo 777192
bufsize 80
bufhash                         E07F950C
bytes   10
Char getted  N
Command N: new file |otherthings|
06013: Exit from First Contact
06016: Rebuilding for flux
06023: Accepting the flux
Getted so far 000:00:01             6.913.279 @    6.59 MB/s
Getted so far 000:00:02           317.488.383 @  151.39 MB/s
Getted so far 000:00:03           543.752.192 @  172.85 MB/s
Getted so far 000:00:04           763.198.719 @  181.96 MB/s
Getted so far 000:00:05           980.647.167 @  187.04 MB/s
(...)
Getted so far 000:00:50         7.860.747.519 @  149.93 MB/s
Fine scrittura
DImensione 7861808028
farei qualcosa start$fix;0;20221023102121;7856460102;1;7861807748;7861807748;
000  |start$fix|
001  |0|
002  |20221023102121|
003  |7856460102|
004  |1|
005  |7861807748|
006  |7861807748|
headerpos    0
datre        20221023102121
cdatasize    7856460102
htsize       1
archive_end  7861807748
archive_size 7861807748
jidac len   104
Jidaclen OK, fixing jidac stuff
Opening in WB -seek and overwrite
Written 104
06148: truncated 0 at 7.861.807.748!
Closing now

C:\zpaqfranz\zcloud>fc z:\1.zpaq z:\3.zpaq /b
Confronto in corso dei file Z:\1.zpaq e Z:\3.ZPAQ
FC: nessuna differenza riscontrata

Definitely more difficult than expected, but I think feasible in the medium term.
I'm forging it as an anti-ransomware system, it might have some use

[fcorbelli]

In other words zpaqfranz will be able to send, through a socket and then over IP, the data to a server (always working on a socket) which will rebuild the file (obviously better to tunnel on ssh, but it's trivial). Any ransom, or even an evil user, will not be able to overwrite the data already secured in the server, in any way.

[fcorbelli]

Work in progress...

[HolyOne]

Thats greatt! Exactly what I am looking for =)

Please make sure if the connection is disconnected I wont lose the existing zpaq archive =)

[fcorbelli]

You can find the very first pre-release here
https://encode.su/threads/3961-Client-server-zpaq-is-it-worth-it?p=76772&viewfull=1#post76772

Please make sure if the connection is disconnected I wont lose the existing zpaq archive
This will require storing two copies. One on the client, and one on the server. Work in progress...

[HolyOne]

Hello, I could not find cloudpaq in your github, is there somewhere I can download the latest version?

…

On Sat, Oct 29, 2022 at 7:54 PM Franco Corbelli ***@***.***> wrote: You can find the very first pre-release here https://encode.su/threads/3961-Client-server-zpaq-is-it-worth-it?p=76772&viewfull=1#post76772 — Reply to this email directly, view it on GitHub <#34 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACHVYOJX5SOKA2EX5KCJE3WFVJCTANCNFSM6AAAAAARD3FR4M> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[fcorbelli]

No, because the development focused on zfs refactoring and backup. The client-server version is suspended for now (much more difficult than expected), after however having demonstrated its feasibility

[kwag]

Hi fcorbelli and HolyOne,

Great OP question and super great and detailed replies, fcorbelli!

I do have a reply to the original question by HolyOne, which is what can easily be done and what I do.
The zpaq archives will be created on the local PC and not on the remote PCs ( a PULL operation )

First, on the local PC, mount all remote PCs with sshfs like the following examples:

sshfs userA@remotehostA:/home/userA/ /mnt/remotehostA -o delay_connect -o reconnect -o user -o allow_other
sshfs userB@remotehostB:/home/userB/ /mnt/remotehostB -o delay_connect -o reconnect -o user -o allow_other
sshfs userC@remotehostC:/home/userC/ /mnt/remotehostC -o delay_connect -o reconnect -o user -o allow_other
sshfs userD@remotehostD:/home/userD/ /mnt/remotehostD -o delay_connect -o reconnect -o user -o allow_other
etc.

Make sure you have done a mount point: "mkdir /mnt/remotehostX" for all remotes.

Then just test like: ls -la /mnt/remotehostA
Which should list all files on the remote host A and also the others.

Then, the backup is as simple as this:

zpaqfranz a remotehosts-A-to-D.zpaq /mnt/remotehostA /mnt/remotehostB /mnt/remotehostC /mnt/remotehostD

That is, if you want to pack all remotes on a single local zpaq archive.
If not, then do an individual backup, like this:

zpaqfranz a remotehostA.zpaq /mnt/remotehostA
zpaqfranz a remotehostB.zpaq /mnt/remotehostB
zpaqfranz a remotehostC.zpaq /mnt/remotehostC
zpaqfranz a remotehostD.zpaq /mnt/remotehostD

Hope this makes sense as I find it very easy to do with sshfs and never fails!
And that's all.

[fcorbelli]

Thank you for your suggestion, I will include it in the official documentation.
The zpaq-over-TCP project (which I work on very little) has a different goal, namely to also run on Windows (not just *nix) and be ransomware resistant.
It's a big job to do, and since I have no requests from users, I work on it once a month, when I happen to have free time.

[fcorbelli]

Update: developing underway
Very, very complex, but now seems doable a "real" zpaqfranz-over-TCP
That is, locally the file with the data is NOT stored.
Probably after the summer

[HolyOne]

I think it would change how backup is done!
but only if it is safe against connection drops, changes in source after those connection drops, and not very slow

[fcorbelli]

I'll try my best :-D

[mirogeorg]

zpaq-over-TCP project is a great idea!!!
thank you for your efforts!

[fcorbelli]

There are quite a few special cases in zpaq that make implementation difficult.
Actually, after a first failed attempt, text file exposure of CRC-32s seems to work well (which means that the -over-TCP version would work too).
During the summer holidays I dedicated myself to the "packaging" of zpaqfranz, maybe next year I'll finish the server :)

What I will implement will be a service that is 100% immune to ransomware (obviously if the server is not compromised)