The corret command to use with ngPost

[Abu3safeer]

Hello, I am using parpar with ngPost and I am happy with it, but I have one issue there, when I try to upload larger files, I get issue with something related to auto-slice, I don't remember what was the error message, but I know it's only showing with large files

I use the default command in ngPost which is:
PAR2_ARGS = -s5M -r1n*0.6 -m2048M -p1l --progress stdout -q

I also notice that in par2cmdline-turbo There is a note:

features unique to ParPar, such as auto-slice size scaling

So I thought this might be the issue I am having

Finally, what is the best command for parpar?

[Squiggles4321]

Hi,

Your issue is relating to the slice size of 5M. There is an upper limit on the number of slices that can be created from the content you are trying to create recovery data for, 32768. At 5MB slices it cannot put all your input data into 32768 slices.

I haven't fully explored the auto-slicing options myself. I have a packing wrapper script that evaluates how much data I'm packing. It defaults to 3.8MB, and if its packing more that 100GB it doubles to 7.6MB slices.

-s2M -r 10% -d pow2 -p 64 -m4096M
Too many input slices: 47469 exceeds maximum of 32768. Please consider increasing the slice size, reducing the amount of input data, or using the --auto-slice-size option
In this example my input is 99549708288 bytes, with 1048576 byte slices.
99549708288 / 2097152 = 47469 slices

-s1M -r 10% -d pow2 -p 64 -m4096M
Too many input slices: 94938 exceeds maximum of 32768. Please consider increasing the slice size, reducing the amount of input data, or using the --auto-slice-size option
In this example my input is 99549708288 bytes, with 1048576 byte slices.
99549708288 / 1048576 = 94938 slices

-s 5M -r 10% -d pow2 -p 64 -m4096M
Input data : 92.71 GiB (18988 slices from 1 file)
Recovery data : 9495 MiB (1899 * 5120 KiB slices)

For my release a 5MB slice works.
99549708288 / 5242880 =18987.6 slices (rounded up by parpar to 18988).

You can either choose a larger slice, or let parpar choose itself. If you are trying to align it to usenet article sizes, you might want to try multiples of recommended article sizes, so you don't have a dangling article of a couple of kilobytes for the rounding error. Otherwise you can pass -S with number of slices for auto calculation.

-s 10000 -S -r 10% -d pow2 -p 64 -m4096M
Input data : 92.71 GiB (10000 slices from 1 file)
Recovery data : 9493.8 MiB (1000 * 9721.65 KiB slices)

parpar decided on 9.95MB chunks.
99549708288 / 10000 slices = 9954970.8288 per slice.

-s 5 -S -r 10% -d pow2 -p 64 -m4096M
Input data : 92.71 GiB (5 slices from 1 file)
Recovery data : 18.54 GiB (1 * 18.54 GiB slice)

parpar decided on 18.5GB slices. Probably not a good idea.

Squigs

[Abu3safeer]

Thanks @Squiggles4321 for your detailed answer.

Can I assume that letting parpar decided it bad idea?
I didn't change any options before, I just use the default options.
But seeing the error while uploading big files made me want to know if it was my mistake, or there is internal issue with parpar.

from your answer I see no way to get a command to cover "All cases", is that because of how par2 files work? or just an issue with parpar?

Or if there default options like just give parpar 10% and it do all of it's magic automatically?

[animetosho]

For Usenet posting, you should be able to do something like --slice-size-multiple [article size] --auto-slice-size and it'll scale the slice size automatically.

But seeing the error while uploading big files made me want to know if it was my mistake, or there is internal issue with parpar.

It's likely a mistake with the ngPost default options, as it doesn't enable auto scaling, so will break on large enough uploads.

is that because of how par2 files work? or just an issue with parpar?

PAR2 spec only allows 32,768 input slices, so it's not an issue with ParPar.

[Abu3safeer]

What is the preferred [article size]?
isn't there one size that can cover all cases? or recommended one?

[animetosho]

-s 10000 -S -r 10% -d pow2 -p 64 -m4096M

Specifying -s 10000 -S doesn't make much sense because you've specified the count directly. Auto sizing is for when you've specified a slice size, which needs to be adjusted because it would exceed a limit.
If you specify the slice count, it, by definition, cannot exceed the limit, so the -S doesn't do anything.

[Abu3safeer]

To be clear, I use 10% of the file size, it has separate config var in ngPost, I am not sure if it's a part of parpar command I provided before.

[animetosho]

What is the preferred [article size]?

It's what you've set in the uploader.
From memory, ngPost uses 700K by default. (if this is the case, the default choice of 5M slice size is likely suboptimal)

isn't there one size that can cover all cases? or recommended one?

No, otherwise it'd be set as the default.
Generally you want it to be a multiple of the article size. Otherwise, it depends on how much recoverability you want, relative to performance.

[Abu3safeer]

I get it now, you mean the article size for the posted file to usenet, I thought there was another article size for parpar.

so the command will look like this:

PAR2_ARGS = -s5M -r1n*0.6 -m2048M -p1l --progress stdout -q --slice-size-multiple 700K --auto-slice-size

[duhagppn]

For Usenet posting, you should be able to do something like --slice-size-multiple [article size] --auto-slice-size and it'll scale the slice size automatically.

could you explain what effect (if any) scaling has on the recoverability of missing articles?

if alignment of slice size to article size is optimal and each increase in slice size reduces that, should we be looking at increasing the recovery percentage as the slice size increases?

5% @ 716800b = usenet recovery 5% (1 slice = 1 article)
5% @ 1433600b =usenet recovery 2.5% (1 slice = 2 articles)
5% @ 5017600b = usenet recovery 0.714% (1 slice = 7 articles)

so to maintain a "real" usenet message recovery rate of 5%, 1433600b should actually use 10% recovery, and 5017600b use 35%?

[animetosho]

It depends on how you're modelling loss. If you're assuming articles go missing in a completely random fashion, your reasoning makes sense.

If an article is missing, the slice that encompasses it is considered invalid. If the slice size is a multiple of the article size, it ensures that a missing article won't straddle two slices (which would cause those two slices to be invalid).
Thus larger slices means that more data is considered bad whenever an article is missing. If multiple articles from the same slice are missing, it won't increase the number of bad slices.

[duhagppn]

interesting...checked a recent failed download to get an idea of the distribution and got this:

https://privatebin.net/?bced75e7bd9762d4#5kMSqVn784ZYGFsrtsj47cG7Z11sENsSWnp6VKY6LVBP

missing articles are random and typically number very few per file (1-4). I'll keep monitoring failed downloads and see what other results come about