-
|
Hello all, I have been using PHP to generate a streamed zip-archive back-up of a nested folder tree. When I run this code locally (on my development machine), all is well and the complete folder (approx. 2,4GB) is zipped and streamed to the browser. When I run that same code (with the same data folder and database records) on the shared hosting webspace, the stream gets aborted after approx. 600MB. The browser shows HTTP-status 200, but the streamed ZIP-file is damaged/corrupt, incomplete. This is the current situation:
I have tried to narrow down the problem the following ways:
however
Can anyone help me solve this issue? How can I debug the problem any further, when the webserver returns HTTP 200 but still serves an incomplete (aborted) stream? I have asked the hosting company for help, but they say they don't support custom-built scripts. I'd have to run a (virtual) private server to get full control of the webserver. Thanks, kind regards, |
Beta Was this translation helpful? Give feedback.
Replies: 7 comments 12 replies
-
|
Can you check if there’s an error string at the end of the zip file? Since the file is streamed, the status is set to 200 at the start of the stream and it can’t be changed later even if an error occurs. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for tour quick response. There seems to be no visible error at the end of the stream. To check, I added the following statements at the top op my script: error_reporting(E_ALL);
ini_set('display_errors', '1');If I examine the end of the corrupt zip-file using a hex-editor, I see the last filename, followed by several bytes: This is the same hex-editor tail, but from the non-corrupt zip-file generated on Yes, a different last filename, but also (I believe) a valid zip-footer ( How can I debug this issue any further? Thanks, |
Beta Was this translation helpful? Give feedback.
-
|
How are you trying to unpack the file? Is there a more specific error message? |
Beta Was this translation helpful? Give feedback.
-
|
Even just listing the files using Meanwhile, I've added a logfile to the export process. It writes a single line when a file was successfully added to the zip. It has between 275 and 277 lines (thus files). Then the mysterious 600MB limit is reached and the export stops. |
Beta Was this translation helpful? Give feedback.
-
|
Hm, it shouldn’t just silently stop. I would expect at least an error in the zip file or php error log. Getting nothing at all could point to an even more severe error. For example a SEGFAULT that ends the execution. Can you make sure to look into any Webserver / PHP logfiles to make sure that this is not the case? If you could share the file (possibly via email if you can’t share it publicly), I can also try to see what exactly is wrong with it. |
Beta Was this translation helpful? Give feedback.
-
The error logs on the shared hosting are empty. I will try to replicate the issue with dummy files, then perhaps I can share more info. To be continued... |
Beta Was this translation helpful? Give feedback.
-
|
One more frown upon my face... I created 1200 dummy files of 2MB each, and put them in the for i in $( seq 1 1 1200 ); do
dd if=/dev/zero of="$( printf '%04d' ${i} ).dummy" bs=2M count=1;
doneI then created a simple PHP-script in the hosting's root folder: <?php
require 'vendor/autoload.php';
$zip = new ZipStream\ZipStream(
outputName: 'example.zip',
sendHttpHeaders: true,
);
$zip->addFile(
fileName: 'hello.txt',
data: 'This is the contents of hello.txt',
);
for ($i = 1; $i <= 1200; $i++) {
$iformatted = sprintf("%04d.dummy", $i);
$path = __DIR__.'/dummy/'.$iformatted;
$zip->addFileFromPath(
fileName: $iformatted,
path: $path
);
}
$zip->finish();This results in a perfectly streamed zipfile (2,6MB in size because of all the zeros), including 1200 2MB dummy files and What we learn: it isn't the amount or total size of uncompressed files that causes the issue on this shared hosting. I will investigate further and report back. |
Beta Was this translation helpful? Give feedback.
-
I have retried this step, and it fails - every time. Now, the 'magic' threshold is around 750MB. I uploaded a large file (Fedora Workstation installation ISO, 2.4GB) to the hosting's root folder. Then commented out the $zip->addFileFromPath(fileName:'f.iso', path:__DIR__.'/f.iso');Still, the webserver logs are empty and the zip stream is not complete. Will experiment with different compression settings (ZipStream constructor parameters). |
Beta Was this translation helpful? Give feedback.
-
|
Found it! When I either set Both the dummy-script and my ArchiveExport script now work as expected. For me, the issue is solved. But I remain curious as to why on this particular shared hosting, Thanks for your wonderful work, reading and thinking along in solving this! |
Beta Was this translation helpful? Give feedback.
-
|
@fwiep Nice that you found a workaround 😊 Deflate however should not pose any issues. If you can replicate it with a file that you can share, I’d love to take a look and make sure it works properly. |
Beta Was this translation helpful? Give feedback.
-
|
Perhaps it's related to the different On
On the shared hosting:
|
Beta Was this translation helpful? Give feedback.
-
|
That should not make a difference. Most zlib features we use are already around since at least PHP 5 days… Are both systems 64bit? |
Beta Was this translation helpful? Give feedback.
-
Yes. On my
On the shared hosting:
|
Beta Was this translation helpful? Give feedback.
-
|
Hm, that's not it either then. For your Hex Dump I can see that the last entry in the file is a This means that Assuming that you correctly checked and there's no error which were overlooked, the only thing that makes sense to me is time / resources. Deflating takes more time and consumes more CPU / Memory. It is possible that the process is OOM killed or that a reverse proxy closes the connection after a certain amount of time causing PHP to stop the process. Can you try to intentionally make the non-deflate version slower with |
Beta Was this translation helpful? Give feedback.
-
Good guess! This is a hexdump of the tail of the broken zip file: Is this helpful? Kind regards, |
Beta Was this translation helpful? Give feedback.
-
|
Nice, we’re getting closer. Is there a loadbalancer, proxy or something relaying the request? If yes: Can you make sure there’s none or a long enough maximum request time? I assume that a proxy terminates the request which leads to PHP terminating the process. That would lead to a partial file. |
Beta Was this translation helpful? Give feedback.
-
|
Sadly, this is a shared hosting environment. I have no control or any insight into the infrastructure 'behind the scenes'. I guess, we'll never know exactly where the request gets killed; at least not without the hosting company's support. They write:
I'm sorry. Kind regards, |
Beta Was this translation helpful? Give feedback.
-
|
May I ask why you would use a shared hosting? A VPS where you are root is like 5$/month. I'm not sure many providers still provide shared hosting, as it's definitely a thing of the past. Now that we have containers, I can't imagine having to deal with a php configuration that I cannot change myself... |
Beta Was this translation helpful? Give feedback.
-
|
Good point... the project using ZipStream-PHP is not actually mine; I develop and maintain all things digital for this choir, including a digital music archive which is streamed from the PHP webapplication. Personally, I do have a VPS up and running (with that same hosting company) and am VERY happy running all stuff in Docker containers. Perhaps I should have a talk with them... :) Thanks for the food for thought, kind regards, |
Beta Was this translation helpful? Give feedback.
Found it!
When I either set
$defaultCompressionMethodto::STOREor (keep it::DEFLATEand set$defaultDeflateLevelto0), the zip successfully streams in its entirety.Both the dummy-script and my ArchiveExport script now work as expected. For me, the issue is solved.
But I remain curious as to why on this particular shared hosting,
deflate-ing with level-1or1-9lets the script terminate without any sign of trouble. Oh well...Thanks for your wonderful work, reading and thinking along in solving this!
FWieP