Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

support symlinks #93

Open
2 of 6 tasks
Trass3r opened this issue Jul 8, 2020 · 15 comments · May be fixed by #201
Open
2 of 6 tasks

support symlinks #93

Trass3r opened this issue Jul 8, 2020 · 15 comments · May be fixed by #201

Comments

@Trass3r
Copy link

Trass3r commented Jul 8, 2020 •
edited
Loading

Typical native Linux builds contain a lot of symlinks. That inflates the upload times and massively increases the artifact sizes.

##[debug]File: 3393/3605. /opt/clang/lib/libclang-cpp.so.11git took 81461.559 milliseconds to finish upload
##[debug]File: 3394/3605. /opt/clang/lib/libclang-cpp.so took 81832.023 milliseconds to finish upload
##[debug]File: 3395/3605. /opt/clang/lib/libclang.so took 983.696 milliseconds to finish upload
##[debug]File: 3396/3605. /opt/clang/lib/libclang.so.11 took 978.577 milliseconds to finish upload
##[debug]File: 3397/3605. /opt/clang/lib/libclang.so.11git took 993.718 milliseconds to finish upload

Version

  • V1
  • V2

Environment

  • self-hosted
  • Linux
  • Windows
  • Mac

Run/Repo Url
https://github.com/Trass3r/llvm-project/runs/849655143
@Trass3r Trass3r added the bug Something isn't working label Jul 8, 2020
@konradpabjan konradpabjan removed the bug Something isn't working label Jul 8, 2020
@konradpabjan
Copy link
Collaborator

I don't quite understand the problem 🤔 Could you elaborate a little more on what the expected behavior should be?

Looking at your run, the two files libclang-cpp.so.11git and libclang-cpp.so had very very large upload times because they were over 4 MB in size so they were uploaded in chunks (this is normal for large files). They have slightly different logs/output if you look at the output just above your screenshot.

image

The time that you see is the sum of all the calls to upload the individual files.

@Trass3r
Copy link
Author

Trass3r commented Jul 8, 2020

libclang-cpp.so is a symlink to libclang-cpp.so.11git as usual.

@freak4pc freak4pc mentioned this issue Oct 3, 2020
Closed
6 tasks
@mossheim mossheim mentioned this issue Nov 25, 2020
Closed
@stefanluptak
Copy link

stefanluptak commented Feb 17, 2021 •
edited
Loading

I am running into this too. Is any ETA available, please?

@p--b
Copy link

p--b commented May 10, 2021

As far as I can see, this is because of

upload-artifact/src/search.ts

Line 16 in ee69f02

followSymbolicLinks: true,
with no available mechanism to override.

Would you accept a small PR to provide a follow-symlinks opt-out as a boolean input, as recommended by https://www.npmjs.com/package/@actions/glob#recommended-action-inputs ?

p--b added a commit to fourieraudio/upload-artifact that referenced this issue May 10, 2021
@p--b
Implement follow-symlinks boolean action input
92b9156 
This commit implements a new boolean input to the action,
`follow-symlinks`. This option configures whether the glob expansion
will follow any symlinks it finds when determining the set of of files
to be archived into the artifact.

The default value of the option, which preserves the existing behaviour,
is `true`. When set to true, symbolic links will be be followed and expanded
If `false`, symbolic links will be included in the archived artifact verbatim.

Users may wish to set this option to false if their artifact contains
internally-referencing symlinks which would result in significant bloat
(and semantic change!) in the source files when the artifact is created.

Resolves: actions#93.
@p--b p--b linked a pull request May 10, 2021 that will close this issue
Draft
beraldoleal added a commit to beraldoleal/avocado that referenced this issue Aug 28, 2021
@beraldoleal
prerelease: fix home dir
2e4a4dd 
Looks like GH is not accepting symlinks. Lets give this a try.

actions/upload-artifact#93

Signed-off-by: Beraldo Leal <bleal@redhat.com>
beraldoleal added a commit to beraldoleal/avocado that referenced this issue Aug 28, 2021
@beraldoleal
prerelease: fix home dir
f40a876 
Looks like GH is not accepting symlinks. Lets give this a try.

actions/upload-artifact#93

Signed-off-by: Beraldo Leal <bleal@redhat.com>
@borrrden borrrden mentioned this issue Mar 2, 2022
Closed
@staronline007
Copy link

Ok, it urgent call for the MTI

@shakfu
Copy link

shakfu commented Apr 4, 2022

I'm not sure it's the same problem but I am finding in v2 that is I have uploaded a macos .bundle which itself contains a .framework bundle and the latter has internal symlinks (Header, Version, etc..), after upload the symlinks are gone and have become duplicate folders of what linked to.

Can someone confirm that this is the same or a similar issue in this case, please?

@antonpirker
Copy link

@shakfu your stated scenario is the same issue.

@DeBlister DeBlister mentioned this issue May 29, 2022
Merged
@dfornika dfornika mentioned this issue Sep 22, 2022
Merged
@jrandolf jrandolf mentioned this issue Oct 3, 2022
Merged
@RebeccaMahany RebeccaMahany mentioned this issue Nov 7, 2022
Merged
@DBraun DBraun mentioned this issue Mar 2, 2023
Closed
@maartenbreddels maartenbreddels mentioned this issue Jun 13, 2023
Merged
@wxtim
Copy link

wxtim commented Jul 25, 2023

I've just come across this issue trying to use this action.

@shakfu
Copy link

shakfu commented Jul 25, 2023

@antonpirker

I find this occurs when I use cp -r <src> <dst> instead of cp -a <src> <dst>

@SteveDevOps
Copy link

SteveDevOps commented Oct 6, 2023 •
edited
Loading

This happened to us on nextjs project / next bin path which is a symlink from .bin. By the time build makes it to azure linux appservice, the app can't spin up its container correctly / can't find module because symlink is missing. Background: https://azureossd.github.io/2022/10/18/NextJS-deployment-on-App-Service-Linux/#note-about-symlinks-why-next-start-may-fail-in-certain-deployment-scenarios -- We fixed by putting full path in app startup cmd on appservice. Other way is full path in package.json (instead of just: yarn start or next start)

wato-github-automation bot pushed a commit to WATonomous/watcloud-website that referenced this issue Nov 5, 2023
@ben-z
Fix gh-pages symlink (#1625)
4c6815c 
It turns out that upload-artifacts strips broken symlinks:
actions/upload-artifact#93, so our approach of
generating a broken symlink that gets linked when it gets to gh-pages
doesn't work. We can work around this by using tarring the output before
uploading the artifacts, but it turns out that the "pages build and
deployment" action doesn't support broken symlinks either, because it
uses `--dereference` in its tar command:

<img width="680" alt="image"
src="https://github.com/WATonomous/infra-config/assets/5977478/c78c2152-f989-4dbc-84c2-daf9e4a89347">

The new strategy in this PR is to set up the symlinks only on master and
only in CI.
@soundasleep soundasleep mentioned this issue Jan 27, 2024
Closed
@soundasleep
Copy link

In #508 the lack of symlink support seems to break MacOS code signatures/notarization. Using tar to compress the folder first seems to work.

@PepeANZ
Copy link

PepeANZ commented Mar 17, 2024

I have opened up a PR to fix this #544

@sanjacob
Copy link

Why not support this as an opt-in? Is there a reason on github's side?

This was referenced Aug 15, 2024
Closed
Closed
eXhumer added a commit to eXhumer/eXViewer that referenced this issue Aug 22, 2024
@eXhumer
chore: Small changes
7d909ac 
* Update README
* use actions/setup-python for Windows as well
* Remove unnecessary end of options for yarn commands
* Use custom upload-artifact to fix uploading macOS builds without needing to double ZIP artifacts to preserve symlinks (actions/upload-artifact#93)

Signed-off-by: eXhumer <exhumer@exhumer.cc>
@eXhumer eXhumer mentioned this issue Aug 22, 2024
Open
@eXhumer
Copy link

eXhumer commented Aug 22, 2024

If anyone wants to use preserve links instead of following them in artifacts during upload, you can use eXhumer/upload-artifact@0b7d5f5684d3f642f978d2faad9ade64f5b4dd57. It is based on the latest upload-artifact version 4.3.6.

https://github.com/eXhumer/eXViewer/blob/119ec1b0081eeeadf1b371dd873d585ef22f0b32/.github/workflows/build.yml#L104-L112

I currently use it to build my macOS application package ZIP and it no longer breaks the application without needing to pre-ZIP before using upload-artifact

@TechnicJelle
Copy link

TechnicJelle commented Oct 18, 2024 •
edited
Loading

I have this issue as well.
Specifically, with my libSDL3.so files.
Normally, there is one "real" libSDL3.so file compiled, and then some symlinks are created for libSDL3.so.0 and libSDL3.so.0.1.5.
I just noticed that the uploaded artifact is very large, because it un-symlinked them, and made them into full copies instead!
Please add a way to make it not do that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Implement follow-symlinks boolean action input fourieraudio/upload-artifact