Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix file size computation on 32bit platforms #23257

Merged
merged 1 commit into from
Dec 2, 2020
Merged

Fix file size computation on 32bit platforms #23257

merged 1 commit into from
Dec 2, 2020

Conversation

aler9
Copy link
Contributor

@aler9 aler9 commented Oct 7, 2020

This fixes #5031 (or at least, the comments referring to versions >= 19.0.0RC1)

Steps to reproduce

Run ./occ files:scan --all on a server running on a 32bit platform (in my case, a Raspberry Pi 4 with 32bit Raspberry Pi OS)

Expected behavior

The size of files bigger than 4GB, displayed in the UI and saved in the DB, should correspond to the actual file size.

Actual behaviour

the size of files bigger than 4GB gets replaced with an apparently random value, making impossible to download the file correctly.

Reason

This is caused by a recent optimization, included in all releases starting from v19.0.0RC1, that reintroduced the native PHP functionstat() for computing file metadatas. The filesize component of the array returned bystat() is broken on 32bit platforms, like the Raspberry Pi, and contains a wrong value. This behavior is well known and was already addressed in the past, with the introduction of a dedicated method that fixes the output of stat().

Fix

This patch replace stat() with the already-developed stat() workaround, restoring the correct behavior.

@gary-kim gary-kim added 3. to review Waiting for reviews bug labels Oct 7, 2020
@gary-kim gary-kim added this to the Nextcloud 21 milestone Oct 7, 2020
Copy link
Member

@MorrisJobke MorrisJobke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense 👍

@kesselb
Copy link
Contributor

kesselb commented Oct 7, 2020

As reference: #16636, #17609.

I'm not a fan. Of course it will fix a problem but more and more code is type hinted anyway. That change (together with the others) is only playing for time.

@aler9
Copy link
Contributor Author

aler9 commented Oct 7, 2020

@kesselb i am aware of the issue you are referencing to, but it's unrelated to this.
Of course i'd like to have that fixed too.

@dasaweb
Copy link

dasaweb commented Oct 15, 2020

I am desperately waiting for a fix of this problem, because my Nextcloud on a 32bit system (ODROID-HC2) is not really working anymore due to this problem. Furthermore I use a snap installation and therefore I can't change the few lines of code by myself but have to wait until the fix is first officially implemented in the Nextcloud and secondly in the snap...

Thanks to everyone who contributes to the solution here!

@aler9
Copy link
Contributor Author

aler9 commented Oct 21, 2020

@rullzer @ChristophWurst any chance of merging?

Copy link

@PeeK1e PeeK1e left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My files are now shown with the right size on my 32bit system

@Skyhooker
Copy link

I am desperately waiting for a fix of this problem, because my Nextcloud on a 32bit system (ODROID-HC2) is not really working anymore due to this problem. Furthermore I use a snap installation and therefore I can't change the few lines of code by myself but have to wait until the fix is first officially implemented in the Nextcloud and secondly in the snap...

Thanks to everyone who contributes to the solution here!

I'm in the same boat, or one boat over - I'm running Nextcloudpi on an Odroid-HC2, and I have many large files I cannot add until it's fixed. I don't think I have the expertise to edit that file, so I'm stuck waiting for the patch to be merged, and then for the folks at Nextcloudpi to test and approve the patched version.

I echo your thanks to all who are working on this!

@MorrisJobke
Copy link
Member

Unfortunately some tests broke:

There were 23 errors:
94	
95	1) Test\Files\Cache\ScannerTest::testScanRemovedFile
96	stat(): stat failed for /tmp/oc_tmp_zMCbUC-folder/folder/bar.txt
97	
98	/drone/src/lib/private/Files/Storage/Local.php:149
99	/drone/src/lib/private/Files/Storage/Local.php:162
100	/drone/src/lib/private/Files/Cache/Scanner.php:114
101	/drone/src/lib/private/Files/Cache/Scanner.php:152
102	/drone/src/tests/lib/Files/Cache/ScannerTest.php:310
...

See https://drone.nextcloud.com/nextcloud/server/33800/11/4

@MorrisJobke MorrisJobke added 2. developing Work in progress and removed 3. to review Waiting for reviews labels Oct 25, 2020
@Bolkarr

This comment has been minimized.

@dasaweb

This comment has been minimized.

@dl6dbh
Copy link

dl6dbh commented Nov 14, 2020

i use NextCloudPi 19.0.4
after patch its running all ok
thanks

@aler9
Copy link
Contributor Author

aler9 commented Dec 1, 2020

@MorrisJobke @rullzer @ChristophWurst hello, i finally got time to find out why tests failed and fix the issue.
Now the patch is compatible with existing tests, and CI seems to be ok.
Please tell me if there's anything else that must be done (i receive private mails about this PR every week).

Thanks for your work.

@Bolkarr
Copy link

Bolkarr commented Dec 1, 2020

Appreciate the work. There are many of us desperately waiting for this patch to finally upgrade from v18. I hope the developers (@MorrisJobke @rullzer @ChristophWurst ) will take a look and merge this if all is good.

Signed-off-by: aler9 <46489434+aler9@users.noreply.github.com>
@aler9
Copy link
Contributor Author

aler9 commented Dec 2, 2020

I waited for the Drone server to be fixed, then i force-pulled a couple of times, and finally all tests passed.
BTW the number of parallel builds of the Drone server should be lowered to 2 (or even 1), there are too many false-negatives caused by parallel builds and high server load.

@MorrisJobke
Copy link
Member

there are too many false-negatives caused by parallel builds and high server load.

They are related to other stuff - we are already looking into this. But thanks for this.

And also thanks for the contribution :)

@MorrisJobke MorrisJobke merged commit e580f91 into nextcloud:master Dec 2, 2020
@welcome
Copy link

welcome bot commented Dec 2, 2020

Thanks for your first pull request and welcome to the community! Feel free to keep them coming! If you are looking for issues to tackle then have a look at this selection: https://github.com/nextcloud/server/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22

@MorrisJobke
Copy link
Member

/backport to stable20

@MorrisJobke
Copy link
Member

/backport to stable19

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2. developing Work in progress bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Wrong or no sizes of files/folders