Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

make: centralize wget/curl & unzip/7z feature test #1431

Merged
merged 1 commit into from
Aug 21, 2014
Merged

make: centralize wget/curl & unzip/7z feature test #1431

merged 1 commit into from
Aug 21, 2014

Conversation

Kijewski
Copy link
Contributor

With many open PRs that could benefit from loading SDKs when needed,
instead adding vast amounts of code to RIOTs master, this PR provides
the "functions" $(DOWNLOAD_TO_STDOUT), $(DOWNLOAD_TO_FILE), and
$(UNZIP_HERE).

The first "function" takes one argument, the URL from where to download
the content. It is then piped to stdout. To be used e.g. with tar xz.

The second "function" taken two arguments, the destination file name,
and the source URL. If the previous invocation was interrupted, then the
download gets continued, if possible.

The last "function" takes one argument, the source ZIP file. The file
gets extracted into the cwd, so best use this "function" with
cd $(SOME_WHERE) &&.

The clumsy name $(UNZIP_HERE) is taken because the program "unzip"
takes the environment variable UNZIP as the source file, even if
another file name was given on the command line. The rationale for that
is that the hackers of "unzip" hate their users. Also they sacrifice
hamsters to Satan.

@Kijewski Kijewski added this to the Release NEXT MAJOR milestone Jul 15, 2014
@LudwigKnuepfer
Copy link
Member

What kind of hamsters?

@Kijewski
Copy link
Contributor Author

The cute kind of hamsters, these devils!

@LudwigKnuepfer
Copy link
Member

Whaaat!?! That's too cute to bear.
Regarding the broken treatment of environment variables.. I assume they are unwilling to change their evil ways due to backwards compatibility, but did you try opening a ticket with them nonetheless? (I know this wont change the PR at hand.)

@Kijewski
Copy link
Contributor Author

The behavior is kinda specified in the man page: http://www.unix.com/man-page/opensolaris/0/UNZIP/ (section "ENVIRONMENT OPTIONS")

Environment options are, in effect, considered to be just like any other command-line options, except that they are effectively the first options on the command line.

I guess change this behavior now would cause the maintainers too much trouble.

At least FreeBSD does not mention the environment variable in it's man page, but Linux, Solaris and DragonflyBSD do.

@LudwigKnuepfer
Copy link
Member

OT: They could change the "API" to make it saner, that's what major releases are for ;-)

@Kijewski
Copy link
Contributor Author

So … ACK?

@Kijewski
Copy link
Contributor Author

Ping.

DOWNLOAD_TO_STDOUT := $(if $(CURL),$(CURL) -s,$(WGET) -q -O-)
endif
ifeq (,$(DOWNLOAD_TO_FILE))
ifneq (,$(AXEL))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why isn't axel treated in the same manner as wget and curl (or the other way around)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

axel can't output to stdout.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking more about the if/else construct, but you could still give /dev/stdout as a file.. as long as axel does not spam around that is.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope, you can't. While downloading axel always creates a file with the same name + some extension, so you can continue an interrupted download.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or do you mean

DOWNLOAD_TO_FILE := $(if $(AXEL),$(AXEL) -n 4 -q -a -o,$(if $(WGET),$(WGET) -nv -c -O,$(CURL) -s -o))

I think this is unreadable. But maybe

$(strip $(if $(AXEL), $(AXEL) -n 4 -q -a -o,
        $(if $(WGET), $(WGET) -nv -c -O,
                      $(CURL) -s -o)))

?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The last one reflects the logic better I'd say.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While downloading axel always creates a file with the same name + some extension, so you can continue an interrupted download.

OK then.

@Kijewski
Copy link
Contributor Author

@LudwigOrtmann please see my last commit

@Kijewski
Copy link
Contributor Author

Nope, nesting $(if)s does not reliably work. See the history towards 887ed57 if you are interested in what I've tried …


$(CURDIR)/$(PKG_NAME)-$(PKG_VERSION).$(PKG_EXT):
# Get PKG_VERSION of package from PKG_URL
$(AD)$(FETCH) $(FETCH_FLAGS) $@ $(PKG_URL)/$(PKG_EXT)/$(PKG_NAME)-$(PKG_VERSION) || true
$(AD)$(FETCH) $(FETCH_FLAGS) $@ $(PKG_URL)/$(PKG_EXT)/$(PKG_NAME)-$(PKG_VERSION)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FETCH is not defined?

@LudwigKnuepfer
Copy link
Member

I suggest you change the documentation around this along with the PR (i.e. pkg/Makefile.http).

@Kijewski
Copy link
Contributor Author

Hm, I think I'll rewrite the pkg'ing description in the wiki soonish. I stumbled upon some gotchas that are not addressed in the examples.

@LudwigKnuepfer
Copy link
Member

Is there some known test application for OpenWSN?

@Kijewski
Copy link
Contributor Author

Nope, it's an outstanding issue: #1231.

@Kijewski
Copy link
Contributor Author

I don't think we should wait for #1231 to be resolved. :)
@thomaseichinger, could you test this PR, please?

@Kijewski
Copy link
Contributor Author

Ping. This PR was meant to make the work easier for the SWP people. I don't know if this is needed anymore after the month delay.

@LudwigKnuepfer
Copy link
Member

@thomaseichinger ping

@thomaseichinger
Copy link
Member

I know, I'm on it, but I'll have to "undo" all my additional setups to test this.

$(AD)$(UNPACK) -q $< -d $(PKG_NAME)-$(PKG_VERSION)
$(AD)cd $@ && sh ../structure_changes.sh
$(AD)cd $@ && sh ../apply_patches.sh
$(AD)cd $@ && $(UNZIP_HERE) $<
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line doesn't work, please change to

$(AD)$(UNZIP_HERE) $< -d $(PKG_NAME)-$(PKG_VERSION)

It creates $@

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$(AD)cd $(CURDIR) && $(UNZIP_HERE) $<? I intended this command not to have any further parameters. E.g. -d is not understood by 7z.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The archive extracts to a not "pretty"/long name and if I use -d option in the package Makefile it's just an extension to your command and I don't need to change the accompanying scripts :-)

@thomaseichinger
Copy link
Member

Tested for OS X and FreeBSD with wget/curl & unzip. I don't have any 7z installed.
ACK when above comment is addressed.

With many open PRs that could benefit from loading SDKs when needed,
instead adding vast amounts of code to RIOTs master, this PR provides
the "functions" `$(DOWNLOAD_TO_STDOUT)`, `$(DOWNLOAD_TO_FILE)`, and
`$(UNZIP_HERE)`.

The first "function" takes one argument, the URL from where to download
the content. It is then piped to stdout. To be used e.g. with `tar xz`.

The second "function" taken two arguments, the destination file name,
and the source URL. If the previous invocation was interrupted, then the
download gets continued, if possible.

The last "function" takes one argument, the source ZIP file. The file
gets extracted into the cwd, so best use this "function" with
`cd $(SOME_WHERE) &&`.

The clumsy name `$(UNZIP_HERE)` is taken because the program "unzip"
takes the environment variable `UNZIP` as the source file, even if
another file name was given on the command line. The rationale for that
is that the hackers of "unzip" hate their users. Also they sacrifice
hamsters to Satan.
@Kijewski
Copy link
Contributor Author

@thomaseichinger ping.

@thomaseichinger
Copy link
Member

ACK

Kijewski added a commit that referenced this pull request Aug 21, 2014
make: centralize wget/curl & unzip/7z feature test
@Kijewski Kijewski merged commit b22c2f4 into RIOT-OS:master Aug 21, 2014
@Kijewski Kijewski deleted the wget-for-everyone-woooohooooo branch August 21, 2014 19:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: build system Area: Build system Type: enhancement The issue suggests enhanceable parts / The PR enhances parts of the codebase / documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants