Yocto, devtool modify

[hoinmic]

Cockpit version: 211
OS: Yocto 3.0
Page: shell, .... (WEBPACK_PACKAGES)

When I compile Cockpit with yocto (devtool modify) the plugins are installed in the wrong place.

Plugins like shell, ... are placed in the wrong folder (/usr/share/cockpit/sources/cockpit/dist). On the webserver only the login is displayed.

When I move the files in the recipe everything works:
cp -al ${D}${pkgdatadir}/sources/cockpit/dist/* ${D}/${pkgdatadir}
rm -rf ${D}${pkgdatadir}/sources/cockpit/dist

I could not find the cause in the Make File. Any ideas?

[hoinmic]

The same cockpit problem occurs in meta-intel-aero: https://layers.openembedded.org/layerindex/recipe/89197/

recipe:
https://github.com/intel-aero/meta-intel-aero/blob/master/recipes-cockpit/cockpit/cockpit_145.bb

lines 76 - 78:
# fix up install location of these files
cp -al ${D}${pkgdatadir}/dist/* ${D}/${pkgdatadir}
rm -rf ${D}${pkgdatadir}/dist

[hoinmic]

The TAR.XZ file of version 211 is taken as source. The following files are installed to the wrong path:

/usr/share/cockpit/sources/cockpit/dist/apps/default.png
/usr/share/cockpit/sources/cockpit/dist/playground/hammer.gif
/usr/share/cockpit/sources/cockpit/dist/shell/images/server-error.png
/usr/share/cockpit/sources/cockpit/dist/shell/images/server-large.png
/usr/share/cockpit/sources/cockpit/dist/shell/images/server-small.png
/usr/share/cockpit/sources/cockpit/dist/shell/index.html
/usr/share/cockpit/sources/cockpit/dist/sosreport/sosreport.png
/usr/share/cockpit/sources/cockpit/dist/storaged/images/storage-array.png
/usr/share/cockpit/sources/cockpit/dist/storaged/images/storage-disk.png

All other files are installed correctly. I can't see the connection of the wrong files. Thank you for every hint.

[martinpitt]

I'm afraid I can't follow. I don't know about yocto, but the given recipe file does not have any configure arguments or otherwise that look weird/wrong at first sight. The bit that I can't see is where it actually calls make install DESTDIR=..., is that implicit somewhere?

I. e. can you give some more information about how the build process looks like, if the installed tree in $DESTDIR is right or wrong, or more generally at which step there's something wrong?

[hoinmic]

Yocto takes some getting used to :-)
In the Yocto recipe autotools is inherited. Baking the recipe triggers "make". Extract from "log.task_order":

....
do_configure (4924): log.do_configure.4924
do_compile (9003): log.do_compile.9003
do_install (11266): log.do_install.11266
do_package (12834): log.do_package.12834
....

Extract from the environment:

...
autotools_do_install() {
        oe_runmake 'DESTDIR=/home/hoim/meta-horizon/build/tmp/work/aarch64-ccp-linux/cockpit/211-r0/image' install
....

Extract from "log.do_configure":

....
               Cockpit 211
             ================

        prefix:                     /usr
        exec_prefix:                /usr
        libdir:                     /usr/lib
        libexecdir:                 /usr/libexec
        bindir:                     /usr/bin
        sbindir:                    /usr/sbin
        datarootdir:                ${prefix}/share
        datadir:                    /usr/share
        sysconfdir:                 /etc
        localstatedir:              /var
        pamdir:                     ${libdir}/security
        systemd unit dir:           /lib/systemd/system

        compiler:                   aarch64-ccp-linux-gcc  --sysroot=/home/hoim/meta-horizon/build/tmp/work/aarch64-ccp-linux/cockpit/211-r0/recipe-sysroot
        cflags:                     -Wall           -Werror=strict-prototypes -Werror=missing-prototypes           -Werror=implicit-function-declaration           -Werror=pointer-arith -Werror=init-self           -Werror=format=2           -Werror=return-type           -Werror=missing-include-dirs  -O2 -pipe -g -feliminate-unused-debug-types -fdebug-prefix-map=/home/hoim/meta-horizon/build/tmp/work/aarch64-ccp-linux/cockpit/211-r0=/usr/src/debug/cockpit/211-r0 -fdebug-prefix-map=/home/hoim/meta-horizon/build/tmp/work/aarch64-ccp-linux/cockpit/211-r0/recipe-sysroot= -fdebug-prefix-map=/home/hoim/meta-horizon/build/tmp/work/aarch64-ccp-linux/cockpit/211-r0/recipe-sysroot-native=  -g
        cppflags:                   

        cockpit-ws user:            root
        cockpit-ws group:           root
        cockpit-ws instance user:   root
        cockpit-ws instance group:  
        selinux config type:        

        Maintainer mode:            yes
        Building docs:              no
        Debug mode:                 default
        Node environment:           production
        With coverage:              no
        With address sanitizer:     no
        With PCP:                   no
        Branding:                   

        cockpit-ssh:                yes
        Supports key auth:          

        pkexec:                     /usr/bin/pkexec
        ssh-add:                    /usr/bin/ssh-add
        ssh-agent:                  /usr/bin/ssh-agent
        sudo:                       /home/hoim/meta-horizon/build/tmp/hosttools/sudo
        usermod:                    /home/hoim/meta-horizon/build/tmp/work/aarch64-ccp-linux/cockpit/211-r0/recipe-sysroot-native/usr/sbin/usermod
        chcon:                      /usr/bin/chcon
...

I was able to narrow down the misconduct to "make install". At this step the wrong paths occur with the files listed above (previous message).

[jsolla]

Hi,
Yocto user here, just bumped my version to 211 and it builds fine when using "bitbake cockpit" having the following lines commented in the recipe:
# fix up install location of these files
#cp -al ${D}${pkgdatadir}/dist/* ${D}/${pkgdatadir}
#rm -rf ${D}${pkgdatadir}/dist

@hoinmic is your recipe building properly WITHOUT using devtool modify?

Just try:

devtool reset cockpit
bitbake cockpit

[hoinmic]

Hi
Without the fix, I just see the cockpit login. After that nothing is loaded anymore (independent if devtool modify is used). But no error messages occur during baking.
My current workaround in the recipe (First "if" fixes normal baking, second "if" fixes devtool modify baking):

# fix up install location of these files (also for devtool)
if [ -d "${D}${pkgdatadir}/dist" ]
then
	cp -al ${D}${pkgdatadir}/dist/* ${D}/${pkgdatadir}
	rm -rf ${D}${pkgdatadir}/dist
fi
if [ -d "${D}${pkgdatadir}/sources/cockpit/dist" ]
then
  cp -al ${D}${pkgdatadir}/sources/cockpit/dist/* ${D}/${pkgdatadir}
  rm -rf ${D}${pkgdatadir}/sources/cockpit/dist
fi

@jsolla Could I take a look at your cockpit recipe? Is cockpit starting correctly and all plugins are available?

[jsolla]

@hoinmic About the recipe, I tried to contact you by email... no luck.

Anyway, what cockpit packages are you installing?

[hoinmic]

Thank you. I have just received the e-mail and replied :-)

IMAGE_INSTALL_append = " \
    cockpit \
    cockpit-swupdate \
    cockpit-ws \
    cockpit-systemd \
    cockpit-docker \
    cockpit-users \
    cockpit-shell \
    cockpit-networkmanager \
   "

[hoinmic]

@martinpitt Do I need to assign the merge request/ qull request to someone or does this happen periodically without my action?

[martinpitt]

@hoinmic : No need to assign anything, we'll get to it. Sorry, this is rather obscure, and needs some diving into our build system, as well as preparing some intelligent questions why this does not work with Yocto in the first place. I didn't find the time to do that yet.

[hoinmic]

@martinpitt Sorry, no rush. I'm more familiar with Gitlab (merge requests) and GitHub is not so familiar :-)
The problem is due to an unfortunate coincidence with the path processing in the Makefile. That's why the error doesn't occur under standard circumstances. In an environment where the Makefile steps are independent, it has an effect.
If desired, I can answer further questions.
The commit corrects the Makefile and thus solves the problem for all environments.

[martinpitt]

The problem is due to an unfortunate coincidence with the path processing in the Makefile. That's why the error doesn't occur under standard circumstances. In an environment where the Makefile steps are independent, it has an effect.

The PR (#13781) too ad-hoc, I'm afraid -- if possible, I'd rather fix this more generically. It means that files installed by webpack.config.js land in the wrong place, and (1) we don't want to duplicate all these in Makefile, and (2) we will break this again in the future as long as we don't run into this problem in our CI and local processes. For example, webpack.config.js also ships docker/images/drive-harddisk-symbolic.svg, and that doesn't appear in your list. Now, maybe you just don't build docker, but it's still completely non-obvious why pkg/shell/index.html needs special treatment, but every other page's index.html doesn't.

What do you mean with "where the Makefile steps are independent"? This comment sounds like yocto would do essentially the same configure/make/make install steps as we do everywhere else.

In short, I'm looking for a series of configure/make invocations that reproduces the bug. Hopefully yocto has some kind of verbose mode where it shows the exact commands that it runs?

Also, perhaps commit fc7734e fixed this as a by-product?

[martinpitt]

Also, perhaps commit fc7734e fixed this as a by-product?

In other words, is that still a problem with version 215?

[hoinmic]

To make the bug visible (with the focus on the "Makefile.am" with the lines):

install-data-local:: $(WEBPACK_INSTALL)
	$(MKDIR_P) $(DESTDIR)$(pkgdatadir)
	tar -cf - $^ | tar -C $(DESTDIR)$(pkgdatadir) --strip-components=1 -xvf -

Note the path of file hammer.gif in the verbose outputs

Verbose output with the fix:
tar -cf - dist/apps/manifest.json ..... dist/playground/exception.min.js.gz dist/playground/hammer.gif dist/playground/index.css.gz ..... dist/users/po.zh_TW.js.gz | tar -C /home/hoim/meta-horizon/build/tmp/work/aarch64-ccp-linux/cockpit/214-r0/image/usr/share/cockpit --strip-components=1 -xvf -

Verbose output without the fix:
tar -cf - dist/apps/manifest.json ..... dist/playground/exception.min.js.gz ../../../../../../workspace/sources/cockpit/dist/playground/hammer.gif dist/playground/index.css.gz ..... dist/users/po.zh_TW.js.gz | tar -C /home/hoim/meta-horizon/build/tmp/work/aarch64-ccp-linux/cockpit/214-r0/image/usr/share/cockpit --strip-components=1 -xvf -

I am very sure that the paths in your environment will not be uniform either. But probably --strip-components=1 often covers up the problem.

A generic solution would be great. But currently the COPY rule is designed only for one source and one target.

[hoinmic]

Currently I am building version 214, I will try to change to version 215.

[hoinmic]

The mentioned Docker File drive-harddisk-symbolic.svg is different. See the ending in docker_INSTALL (.gz).
conversion:

docker_OUTPUTS = \
....
dist/docker/images/drive-harddisk-symbolic.svg \
.....

docker_INSTALL = \
....
dist/docker/images/drive-harddisk-symbolic.svg.gz \
.....

vs
one to one copy:

playground_OUTPUTS = \
.....
	dist/playground/hammer.gif \
.....

playground_INSTALL = \
.....
	dist/playground/hammer.gif \
.....

[hoinmic]

@martinpitt I have tested version 215 and 216. The problem is still the same.

[hoinmic]

Closed with #13781

[martinpitt]

Closing the PR was ok, but this issue should still be kept open.

[PMaynard]

It might be worthwhile merging this recipe into the primary package repository for yocto/openembedded [1] they already have webmin. I feel devices running systemd would benefit having cockpit as an option over webmin.

[1] https://github.com/openembedded/meta-openembedded/tree/master/meta-webserver

[hoinmic]

@PMaynard I completely agree with you. I just need a little tweaking of the recipe, then I'll try to give it upstream.
I wrote a recipe patch for the corrupt MakeFile of Cockpit, so devtool modify works now.
The versioning problem is solved since Cockpit version 217.
I will also wait for the answers to issue #13947.

[hoinmic]

@PMaynard Started openembedded pull request for cockpit 218 recipe :-) openembedded/meta-openembedded#218

[PMaynard]

Great, I will try it out asap.

[hoinmic]

@martinpitt Cockpit recipe is now available on the master branch of openembedded
https://github.com/openembedded/meta-openembedded/tree/master/meta-webserver/recipes-webadmin/cockpit

[jsolla]

Amazing!
@hoinmic I'm missing in the recipe a dependency on "udisks2" required for storaged package:

RDEPENDS_${PN}-storaged = "udisks2"

Any reason for this?

[hoinmic]

@jsolla Thank you :-)
I am not currently using storaged. Therefore, it is quite possible that a dependency is missing.

Side note: udisks2 has a dependence on Polkit

[jsolla]

@hoinmic I submited a PR to the openembedded repo, is now merged :)

[martinpitt]

Is there still anything left to do here? Current cockpit got a lot of build system fixes/cleanups, the docker page is gone entirely, etc. The above few comments sound like it's working now?

I am closing this issue as I don't see anything actionable from our side. Please shout if there is still something missing. Thanks!