Docker on steroids

[dokterbob]

This builds on from #1672 and will show up clean once that's been merged.

It features:

• More strongly layered Docker build enabling MUCH more aggressive caching. Last (warm) build was running in 1s(!)
• Additional space savings, we're now at 1.65 GB.
• Runs as a non-privileged user.
• Compiles Python into byte-code for quicker starts.
• Includes a fix for JS comment lingering in Python file.
• Expect huge speedups with cache on GH actions enabled.

[pierotofy]

Runs as a non-privileged user.

Be very careful here, we've been running the images as root up until now, so if you switch user now, you will probably get a bunch of permission read/write errors because files created from previous images are owned by root (for existing users, that is).

[dokterbob]

Runs as a non-privileged user.

Be very careful here, we've been running the images as root up until now, so if you switch user now, you will probably get a bunch of permission read/write errors because files created from previous images are owned by root (for existing users, that is).

Good point. What do you think about chmod + chown from docker-compose on start?

Somethinglike

Perhaps on that note, perhaps it makes sense having a 'legacy' environment and a 'modern' environment, at some point (soon)? Allowing new users to use things like updated Postgres and also this?

Another thought: detecting and automatically upgrading these things on first start.

Just testing the waters here.

[dokterbob]

@pierotofy I removed the unprivileged patch as I noticed more side effects, see WIP in #1685. Let me know if there's anything else you find here.

[dokterbob]

I've noticed there's still Python3.10 being installed as a result of the Ubuntu certbot package. Probably ~100 MB wasted.

Either we could globally pip install it or we go straight for getting nginx out altogether (perhaps as part of #1685). Happy to hear your thoughts.

[dokterbob]

I've noticed there's still Python3.10 being installed as a result of the Ubuntu certbot package. Probably ~100 MB wasted.

Either we could globally pip install it or we go straight for getting nginx out altogether (perhaps as part of #1685). Happy to hear your thoughts.

Turns out GDAL also requires python3 and so it's not so trivial. Leaving this for later. ;)

[pierotofy]

detecting and automatically upgrading these things on first start.

This would probably be the way to go, changing the permissions once, definitely not at every start because some installations have a lot of files / folders.

[dokterbob]

detecting and automatically upgrading these things on first start.

This would probably be the way to go, changing the permissions once, definitely not at every start because some installations have a lot of files / folders.

Yes, same for DB. To reduce complication I removed the unprivileged user patch from this PR. I think it should be ready to go now, but do let me know if you have additional feedback (also, anyone doing manual tests would be great - @Saijin-Naib ?).

[pierotofy]

Nice, this seems to be speeding up the build by quite a bit. Something must be missing from the build because I'm seeing these messages at startup regarding the plugins not being built (which should be built during the build):

webapp  | Trying to establish communication...
webapp  | INFO Booting WebODM 2.8.4
webapp  | INFO Running webpack for measure

[dokterbob]

Nice, this seems to be speeding up the build by quite a bit. Something must be missing from the build because I'm seeing these messages at startup regarding the plugins not being built (which should be built during the build):

webapp  | Trying to establish communication...
webapp  | INFO Booting WebODM 2.8.4
webapp  | INFO Running webpack for measure

Maybe they're not included/copied properly?

Looking at the Dockerfile I see no immediate issues.

But also, I noticed we're not checking the return status on plugin webpack build commands. Is that intentional?

[pierotofy]

we're not checking the return status on plugin webpack build commands. Is that intentional?

Yes. Some plugins can and sometimes do break, but because they are plugins, they shouldn't stop the core software from working.

[dokterbob]

we're not checking the return status on plugin webpack build commands. Is that intentional?

Yes. Some plugins can and sometimes do break, but because they are plugins, they shouldn't stop the core software from working.

But these are core plugins, no? 😉
Ehehe sorry long day bit of a smartass. ;)
It would be good UX perhaps to at least give a notification, some user feedback, like setting a status on the model with some kind of error message, then displaying it in the UI. Or at least a BIG FAT WARNING in the errors. What do you think, worth an issue?

On the specific issue, you know much, much more of what a successfull build should look like. Would you care to read through the logs of the Docker build to see if perhaps there's some obvious errors I've missed (I didn't notice any)?

Thanks!

[pierotofy]

The log will already show if a plugin's build fails. So I don't think it's super-important, but feel free to improve.

At a quick glance I'm not sure what's causing the plugins to be rebuilt at container start due to the new changes.

[dokterbob]

The log will already show if a plugin's build fails. So I don't think it's super-important, but feel free to improve.

At a quick glance I'm not sure what's causing the plugins to be rebuilt at container start due to the new changes.

If you could point me at what to look for in the logs I'm happy to give it an extra look.

Apart from that, is there anything that's blocking this? I feel it would greatly facilitate contribution (as it shortens the feedback cycle).

[pierotofy]

The plugin build issue is the only problem I've found thus far and definitely should be fixed before merging.

Try to launch the app; you'll notice that webpack is being called to rebuild the plugins at startup (there will be several lines with webpack's output).