Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

makefiles: Sort >/dev/null and 2>&1 #16775

Merged
merged 1 commit into from
Aug 27, 2021

Conversation

chrysn
Copy link
Member

@chrysn chrysn commented Aug 24, 2021

Contribution description

Redirecting 2>&1 >/dev/null moves stderr to stdout first and then
stdout to /dev/null; when checking for command existence or otherwise
silencing output, this is usually not desired (but only starts producing
errors when the actual command fails, which is often not tested).

In particular cases, the redirect made things even worse: while stderr would have gone to the user's screen otherwise, the 2>&1 >/dev/null means that warnings raised now get mingled in the later echo $? and thus make the equality check for 0 fail -- tools are then not detected any more.

Testing procedure

Try building anything on a recent Debian sid system; it will fail before with

.../Makefile.include:340: *** Neither wget nor curl is installed!.  Stop.

and now run through.

Issues/PRs references

debianutils' which, in its changelog, states:

  * The 'which' utility will be removed in the future.  Shell scripts
    often use it to check whether a command is available.  A more
    standard way to do this is with 'command -v'; for example:
      if command -v update-icon-caches >/dev/null; then
        update-icon-caches /usr/share/icons/...
      fi
    '2>/dev/null' is unnecessary when using 'command': POSIX says "no
    output shall be written" if the command isn't found.  It's also
    unnecessary for the debianutils version of 'which', and hides the
    deprecation warning.

Note that this PR does not change which to command, it just fixes the fallout of ill-sequenced redirects. That change is probably good to do, but right now this should be minimal enough to get merged quickly, and I don't want it held up in the question of whether everyone and their dog already implement POSIX command.

Redirecting `2>&1 >/dev/null` moves stderr to stdout first and then
stdout to /dev/null; when checking for command existence or otherwise
silencing output, this is usually not desired (but only starts producing
errors when the actual command fails, which is often not tested).
@github-actions github-actions bot added Area: build system Area: Build system Area: pkg Area: External package ports Area: tools Area: Supplementary tools labels Aug 24, 2021
@chrysn chrysn added Impact: minor The PR is small in size and might only require a quick look of a knowledgeable reviewer Type: bug The issue reports a bug / The PR fixes a bug (including spelling errors) CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR CI: skip compile test If set, CI server will run only non-compile jobs, but no compile jobs or their dependent jobs and removed Area: pkg Area: External package ports Area: tools Area: Supplementary tools labels Aug 24, 2021
@chrysn
Copy link
Member Author

chrysn commented Aug 24, 2021

I've set this to skip the compile tests, based on this only affecting tool detection lines; reviewers please remove that flag if you disagree.

@fjmolinas fjmolinas requested a review from benpicco August 24, 2021 13:03
@fjmolinas
Copy link
Contributor

'2>/dev/null' is unnecessary when using 'command': POSIX says "no
output shall be written" if the command isn't found.  It's also
unnecessary for the debianutils version of 'which', and hides the
deprecation warning.

Might this be an issue for other versions onf which then? Why not replace by command -v right away?

@fjmolinas
Copy link
Contributor

Regarding testing, the behaviour is still the same but I'm using the debianutils version of which...

@chrysn
Copy link
Member Author

chrysn commented Aug 24, 2021

I held off on replacing with command because that might spark discussion on whether that's really universally available.

Having found it used in the makefiles since, I'm putting the later changes on top of this.

@@ -327,12 +327,12 @@ APPLICATION := $(strip $(APPLICATION))

ifeq (,$(and $(DOWNLOAD_TO_STDOUT),$(DOWNLOAD_TO_FILE)))
ifeq (,$(WGET))
ifeq (0,$(shell which wget 2>&1 > /dev/null ; echo $$?))
ifeq (0,$(shell which wget > /dev/null 2>&1 ; echo $$?))
Copy link
Contributor

@benpicco benpicco Aug 24, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it even necessary to redirect stderr to stdout at this point? It will be printed either way.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was trying for minimal changes here; as that's apparently not called for (cf. #16775 (comment)), the latest additions do away with this.

@github-actions github-actions bot added Area: CI Area: Continuous Integration of RIOT components Area: pkg Area: External package ports Area: tools Area: Supplementary tools Platform: RISC-V Platform: This PR/issue effects RISC-V-based platforms labels Aug 24, 2021
@chrysn
Copy link
Member Author

chrysn commented Aug 24, 2021

Regarding testing, the behaviour is still the same but I'm using the debianutils version of which...

As in "it still fails" or in "it didn't fail before"? Which command were you testing with? (My current go-to is BOARD=microbit-v2 make -C examples/saul all flash term)

@chrysn
Copy link
Member Author

chrysn commented Aug 24, 2021

Seems my worries were warranted -- one of the build environments doesn't even know command.

No clue why (the python-tests job of .github/workflows/tools-test.yml is supposed to be run on ubuntu-latest whose shells do know command), but I'll revert this to only being the first commit which is minimally fixing things, and see to the rest in a follow-up.

@github-actions github-actions bot removed Platform: RISC-V Platform: This PR/issue effects RISC-V-based platforms Area: CI Area: Continuous Integration of RIOT components labels Aug 24, 2021
@chrysn chrysn removed Area: pkg Area: External package ports Area: tools Area: Supplementary tools labels Aug 24, 2021
chrysn added a commit to chrysn-pull-requests/RIOT that referenced this pull request Aug 24, 2021
While this could theoretically be desired, it's usually just a mishap.
It is unlikely that legitimate cases will be needed in the build system;
if so, they can exclude themselves.

See-Also: RIOT-OS#16775
Copy link
Contributor

@fjmolinas fjmolinas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK, small enough change. Locally behaviour is the same.

@chrysn chrysn merged commit de768b5 into RIOT-OS:master Aug 27, 2021
@chrysn chrysn deleted the shell-make-fixes branch August 27, 2021 08:38
chrysn added a commit to chrysn-pull-requests/RIOT that referenced this pull request Aug 27, 2021
While this could theoretically be desired, it's usually just a mishap.
It is unlikely that legitimate cases will be needed in the build system;
if so, they can exclude themselves.

See-Also: RIOT-OS#16775
chrysn added a commit to chrysn-pull-requests/RIOT that referenced this pull request Sep 2, 2021
While this could theoretically be desired, it's usually just a mishap.
It is unlikely that legitimate cases will be needed in the build system;
if so, they can exclude themselves.

See-Also: RIOT-OS#16775
chrysn added a commit to chrysn-pull-requests/RIOT that referenced this pull request Sep 3, 2021
While this could theoretically be desired, it's usually just a mishap.
It is unlikely that legitimate cases will be needed in the build system;
if so, they can exclude themselves.

See-Also: RIOT-OS#16775
chrysn added a commit to chrysn-pull-requests/RIOT that referenced this pull request Sep 3, 2021
This is a follow-up for [16775], and was not caught there as that search
was limited to `which`.

Note that while this line can be simplified, the redirects ensure that
GNU Make < 4.3 will not optimize it into its own built-in shell that
does not know `command`.

[16775]: RIOT-OS#16775
@benpicco benpicco added this to the Release 2021.10 milestone Oct 20, 2021
chrysn added a commit to chrysn-pull-requests/RIOT that referenced this pull request Apr 16, 2022
While this could theoretically be desired, it's usually just a mishap.
It is unlikely that legitimate cases will be needed in the build system;
if so, they can exclude themselves.

See-Also: RIOT-OS#16775
chrysn added a commit to chrysn-pull-requests/RIOT that referenced this pull request Apr 16, 2022
This is a follow-up for [16775], and was not caught there as that search
was limited to `which`.

Note that while this line can be simplified, the redirects ensure that
GNU Make < 4.3 will not optimize it into its own built-in shell that
does not know `command`.

[16775]: RIOT-OS#16775
chrysn added a commit to chrysn-pull-requests/RIOT that referenced this pull request Apr 16, 2022
While this could theoretically be desired, it's usually just a mishap.
It is unlikely that legitimate cases will be needed in the build system;
if so, they can exclude themselves.

See-Also: RIOT-OS#16775
chrysn added a commit to chrysn-pull-requests/RIOT that referenced this pull request Apr 16, 2022
This is a follow-up for [16775], and was not caught there as that search
was limited to `which`.

Note that while this line can be simplified, the redirects ensure that
GNU Make < 4.3 will not optimize it into its own built-in shell that
does not know `command`.

[16775]: RIOT-OS#16775
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: build system Area: Build system CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR CI: skip compile test If set, CI server will run only non-compile jobs, but no compile jobs or their dependent jobs Impact: minor The PR is small in size and might only require a quick look of a knowledgeable reviewer Type: bug The issue reports a bug / The PR fixes a bug (including spelling errors)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants