Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Yocto, devtool modify #13554

Closed
hoinmic opened this issue Feb 15, 2020 · 28 comments
Closed

Yocto, devtool modify #13554

hoinmic opened this issue Feb 15, 2020 · 28 comments
Labels
question Further information is requested

Comments

@hoinmic
Copy link
Contributor

hoinmic commented Feb 15, 2020

Cockpit version: 211
OS: Yocto 3.0
Page: shell, .... (WEBPACK_PACKAGES)

When I compile Cockpit with yocto (devtool modify) the plugins are installed in the wrong place.

Plugins like shell, ... are placed in the wrong folder (/usr/share/cockpit/sources/cockpit/dist). On the webserver only the login is displayed.

When I move the files in the recipe everything works:
cp -al ${D}${pkgdatadir}/sources/cockpit/dist/* ${D}/${pkgdatadir}
rm -rf ${D}${pkgdatadir}/sources/cockpit/dist

I could not find the cause in the Make File. Any ideas?

@hoinmic
Copy link
Contributor Author

hoinmic commented Feb 15, 2020

The same cockpit problem occurs in meta-intel-aero: https://layers.openembedded.org/layerindex/recipe/89197/

recipe:
https://github.com/intel-aero/meta-intel-aero/blob/master/recipes-cockpit/cockpit/cockpit_145.bb

lines 76 - 78:
# fix up install location of these files
cp -al ${D}${pkgdatadir}/dist/* ${D}/${pkgdatadir}
rm -rf ${D}${pkgdatadir}/dist

@hoinmic
Copy link
Contributor Author

hoinmic commented Feb 17, 2020

The TAR.XZ file of version 211 is taken as source. The following files are installed to the wrong path:

/usr/share/cockpit/sources/cockpit/dist/apps/default.png
/usr/share/cockpit/sources/cockpit/dist/playground/hammer.gif
/usr/share/cockpit/sources/cockpit/dist/shell/images/server-error.png
/usr/share/cockpit/sources/cockpit/dist/shell/images/server-large.png
/usr/share/cockpit/sources/cockpit/dist/shell/images/server-small.png
/usr/share/cockpit/sources/cockpit/dist/shell/index.html
/usr/share/cockpit/sources/cockpit/dist/sosreport/sosreport.png
/usr/share/cockpit/sources/cockpit/dist/storaged/images/storage-array.png
/usr/share/cockpit/sources/cockpit/dist/storaged/images/storage-disk.png

All other files are installed correctly. I can't see the connection of the wrong files. Thank you for every hint.

@martinpitt
Copy link
Member

I'm afraid I can't follow. I don't know about yocto, but the given recipe file does not have any configure arguments or otherwise that look weird/wrong at first sight. The bit that I can't see is where it actually calls make install DESTDIR=..., is that implicit somewhere?

I. e. can you give some more information about how the build process looks like, if the installed tree in $DESTDIR is right or wrong, or more generally at which step there's something wrong?

@martinpitt martinpitt added the question Further information is requested label Feb 18, 2020
@hoinmic
Copy link
Contributor Author

hoinmic commented Feb 19, 2020

Yocto takes some getting used to :-)
In the Yocto recipe autotools is inherited. Baking the recipe triggers "make". Extract from "log.task_order":

....
do_configure (4924): log.do_configure.4924
do_compile (9003): log.do_compile.9003
do_install (11266): log.do_install.11266
do_package (12834): log.do_package.12834
....

Extract from the environment:

...
autotools_do_install() {
        oe_runmake 'DESTDIR=/home/hoim/meta-horizon/build/tmp/work/aarch64-ccp-linux/cockpit/211-r0/image' install
....

Extract from "log.do_configure":

....
               Cockpit 211
             ================

        prefix:                     /usr
        exec_prefix:                /usr
        libdir:                     /usr/lib
        libexecdir:                 /usr/libexec
        bindir:                     /usr/bin
        sbindir:                    /usr/sbin
        datarootdir:                ${prefix}/share
        datadir:                    /usr/share
        sysconfdir:                 /etc
        localstatedir:              /var
        pamdir:                     ${libdir}/security
        systemd unit dir:           /lib/systemd/system

        compiler:                   aarch64-ccp-linux-gcc  --sysroot=/home/hoim/meta-horizon/build/tmp/work/aarch64-ccp-linux/cockpit/211-r0/recipe-sysroot
        cflags:                     -Wall           -Werror=strict-prototypes -Werror=missing-prototypes           -Werror=implicit-function-declaration           -Werror=pointer-arith -Werror=init-self           -Werror=format=2           -Werror=return-type           -Werror=missing-include-dirs  -O2 -pipe -g -feliminate-unused-debug-types -fdebug-prefix-map=/home/hoim/meta-horizon/build/tmp/work/aarch64-ccp-linux/cockpit/211-r0=/usr/src/debug/cockpit/211-r0 -fdebug-prefix-map=/home/hoim/meta-horizon/build/tmp/work/aarch64-ccp-linux/cockpit/211-r0/recipe-sysroot= -fdebug-prefix-map=/home/hoim/meta-horizon/build/tmp/work/aarch64-ccp-linux/cockpit/211-r0/recipe-sysroot-native=  -g
        cppflags:                   

        cockpit-ws user:            root
        cockpit-ws group:           root
        cockpit-ws instance user:   root
        cockpit-ws instance group:  
        selinux config type:        

        Maintainer mode:            yes
        Building docs:              no
        Debug mode:                 default
        Node environment:           production
        With coverage:              no
        With address sanitizer:     no
        With PCP:                   no
        Branding:                   

        cockpit-ssh:                yes
        Supports key auth:          

        pkexec:                     /usr/bin/pkexec
        ssh-add:                    /usr/bin/ssh-add
        ssh-agent:                  /usr/bin/ssh-agent
        sudo:                       /home/hoim/meta-horizon/build/tmp/hosttools/sudo
        usermod:                    /home/hoim/meta-horizon/build/tmp/work/aarch64-ccp-linux/cockpit/211-r0/recipe-sysroot-native/usr/sbin/usermod
        chcon:                      /usr/bin/chcon
...

I was able to narrow down the misconduct to "make install". At this step the wrong paths occur with the files listed above (previous message).

@jsolla
Copy link

jsolla commented Mar 3, 2020

Hi,
Yocto user here, just bumped my version to 211 and it builds fine when using "bitbake cockpit" having the following lines commented in the recipe:
# fix up install location of these files
#cp -al ${D}${pkgdatadir}/dist/* ${D}/${pkgdatadir}
#rm -rf ${D}${pkgdatadir}/dist

@hoinmic is your recipe building properly WITHOUT using devtool modify?

Just try:

devtool reset cockpit
bitbake cockpit

@hoinmic
Copy link
Contributor Author

hoinmic commented Mar 4, 2020

Hi
Without the fix, I just see the cockpit login. After that nothing is loaded anymore (independent if devtool modify is used). But no error messages occur during baking.
My current workaround in the recipe (First "if" fixes normal baking, second "if" fixes devtool modify baking):

# fix up install location of these files (also for devtool)
if [ -d "${D}${pkgdatadir}/dist" ]
then
	cp -al ${D}${pkgdatadir}/dist/* ${D}/${pkgdatadir}
	rm -rf ${D}${pkgdatadir}/dist
fi
if [ -d "${D}${pkgdatadir}/sources/cockpit/dist" ]
then
  cp -al ${D}${pkgdatadir}/sources/cockpit/dist/* ${D}/${pkgdatadir}
  rm -rf ${D}${pkgdatadir}/sources/cockpit/dist
fi

@jsolla Could I take a look at your cockpit recipe? Is cockpit starting correctly and all plugins are available?

@jsolla
Copy link

jsolla commented Mar 4, 2020

@hoinmic About the recipe, I tried to contact you by email... no luck.

Anyway, what cockpit packages are you installing?

@hoinmic
Copy link
Contributor Author

hoinmic commented Mar 4, 2020

Thank you. I have just received the e-mail and replied :-)

IMAGE_INSTALL_append = " \
    cockpit \
    cockpit-swupdate \
    cockpit-ws \
    cockpit-systemd \
    cockpit-docker \
    cockpit-users \
    cockpit-shell \
    cockpit-networkmanager \
   "

@hoinmic
Copy link
Contributor Author

hoinmic commented Mar 26, 2020

@martinpitt Do I need to assign the merge request/ qull request to someone or does this happen periodically without my action?

@martinpitt
Copy link
Member

@hoinmic : No need to assign anything, we'll get to it. Sorry, this is rather obscure, and needs some diving into our build system, as well as preparing some intelligent questions why this does not work with Yocto in the first place. I didn't find the time to do that yet.

@hoinmic
Copy link
Contributor Author

hoinmic commented Mar 26, 2020

@martinpitt Sorry, no rush. I'm more familiar with Gitlab (merge requests) and GitHub is not so familiar :-)
The problem is due to an unfortunate coincidence with the path processing in the Makefile. That's why the error doesn't occur under standard circumstances. In an environment where the Makefile steps are independent, it has an effect.
If desired, I can answer further questions.
The commit corrects the Makefile and thus solves the problem for all environments.

@martinpitt
Copy link
Member

The problem is due to an unfortunate coincidence with the path processing in the Makefile. That's why the error doesn't occur under standard circumstances. In an environment where the Makefile steps are independent, it has an effect.

The PR (#13781) too ad-hoc, I'm afraid -- if possible, I'd rather fix this more generically. It means that files installed by webpack.config.js land in the wrong place, and (1) we don't want to duplicate all these in Makefile, and (2) we will break this again in the future as long as we don't run into this problem in our CI and local processes. For example, webpack.config.js also ships docker/images/drive-harddisk-symbolic.svg, and that doesn't appear in your list. Now, maybe you just don't build docker, but it's still completely non-obvious why pkg/shell/index.html needs special treatment, but every other page's index.html doesn't.

What do you mean with "where the Makefile steps are independent"? This comment sounds like yocto would do essentially the same configure/make/make install steps as we do everywhere else.

In short, I'm looking for a series of configure/make invocations that reproduces the bug. Hopefully yocto has some kind of verbose mode where it shows the exact commands that it runs?

Also, perhaps commit fc7734e fixed this as a by-product?

@martinpitt
Copy link
Member

Also, perhaps commit fc7734e fixed this as a by-product?

In other words, is that still a problem with version 215?

@hoinmic
Copy link
Contributor Author

hoinmic commented Apr 1, 2020

To make the bug visible (with the focus on the "Makefile.am" with the lines):

install-data-local:: $(WEBPACK_INSTALL)
	$(MKDIR_P) $(DESTDIR)$(pkgdatadir)
	tar -cf - $^ | tar -C $(DESTDIR)$(pkgdatadir) --strip-components=1 -xvf -

Note the path of file hammer.gif in the verbose outputs

Verbose output with the fix:
tar -cf - dist/apps/manifest.json ..... dist/playground/exception.min.js.gz dist/playground/hammer.gif dist/playground/index.css.gz ..... dist/users/po.zh_TW.js.gz | tar -C /home/hoim/meta-horizon/build/tmp/work/aarch64-ccp-linux/cockpit/214-r0/image/usr/share/cockpit --strip-components=1 -xvf -

Verbose output without the fix:
tar -cf - dist/apps/manifest.json ..... dist/playground/exception.min.js.gz ../../../../../../workspace/sources/cockpit/dist/playground/hammer.gif dist/playground/index.css.gz ..... dist/users/po.zh_TW.js.gz | tar -C /home/hoim/meta-horizon/build/tmp/work/aarch64-ccp-linux/cockpit/214-r0/image/usr/share/cockpit --strip-components=1 -xvf -

I am very sure that the paths in your environment will not be uniform either. But probably --strip-components=1 often covers up the problem.

A generic solution would be great. But currently the COPY rule is designed only for one source and one target.

@hoinmic
Copy link
Contributor Author

hoinmic commented Apr 1, 2020

Currently I am building version 214, I will try to change to version 215.

@hoinmic
Copy link
Contributor Author

hoinmic commented Apr 1, 2020

The mentioned Docker File drive-harddisk-symbolic.svg is different. See the ending in docker_INSTALL (.gz).
conversion:

docker_OUTPUTS = \
....
dist/docker/images/drive-harddisk-symbolic.svg \
.....

docker_INSTALL = \
....
dist/docker/images/drive-harddisk-symbolic.svg.gz \
.....

vs
one to one copy:

playground_OUTPUTS = \
.....
	dist/playground/hammer.gif \
.....

playground_INSTALL = \
.....
	dist/playground/hammer.gif \
.....

@hoinmic
Copy link
Contributor Author

hoinmic commented Apr 1, 2020

@martinpitt I have tested version 215 and 216. The problem is still the same.

@hoinmic
Copy link
Contributor Author

hoinmic commented Apr 12, 2020

Closed with #13781

@hoinmic hoinmic closed this as completed Apr 12, 2020
@martinpitt
Copy link
Member

Closing the PR was ok, but this issue should still be kept open.

@martinpitt martinpitt reopened this Apr 13, 2020
@PMaynard
Copy link

It might be worthwhile merging this recipe into the primary package repository for yocto/openembedded [1] they already have webmin. I feel devices running systemd would benefit having cockpit as an option over webmin.

[1] https://github.com/openembedded/meta-openembedded/tree/master/meta-webserver

@hoinmic
Copy link
Contributor Author

hoinmic commented Apr 22, 2020

@PMaynard I completely agree with you. I just need a little tweaking of the recipe, then I'll try to give it upstream.
I wrote a recipe patch for the corrupt MakeFile of Cockpit, so devtool modify works now.
The versioning problem is solved since Cockpit version 217.
I will also wait for the answers to issue #13947.

@hoinmic
Copy link
Contributor Author

hoinmic commented Apr 29, 2020

@PMaynard Started openembedded pull request for cockpit 218 recipe :-) openembedded/meta-openembedded#218

@PMaynard
Copy link

Great, I will try it out asap.

@hoinmic
Copy link
Contributor Author

hoinmic commented May 8, 2020

@martinpitt Cockpit recipe is now available on the master branch of openembedded
https://github.com/openembedded/meta-openembedded/tree/master/meta-webserver/recipes-webadmin/cockpit

@jsolla
Copy link

jsolla commented May 10, 2020

Amazing!
@hoinmic I'm missing in the recipe a dependency on "udisks2" required for storaged package:

RDEPENDS_${PN}-storaged = "udisks2"

Any reason for this?

@hoinmic
Copy link
Contributor Author

hoinmic commented May 11, 2020

@jsolla Thank you :-)
I am not currently using storaged. Therefore, it is quite possible that a dependency is missing.

Side note: udisks2 has a dependence on Polkit

@jsolla
Copy link

jsolla commented May 14, 2020

@hoinmic I submited a PR to the openembedded repo, is now merged :)

@martinpitt
Copy link
Member

Is there still anything left to do here? Current cockpit got a lot of build system fixes/cleanups, the docker page is gone entirely, etc. The above few comments sound like it's working now?

I am closing this issue as I don't see anything actionable from our side. Please shout if there is still something missing. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants