Building outside of .west workspace with ZEPHYR_BASE not properly documented?

[drojf]

According to the "Troubleshooting West" documentation, there is only one way to build outside of a .west workspace: using the ZEPHYR_BASE environment variable.

However, the "Getting Started Guide" never mentions this variable, and the "Application Development" page doesn't specify that this variable is required (it is listed on that page, but not stated that it's required for the instructions on that page to work). If you follow the instructions on the "Application Development" without setting ZEPHYR_BASE, the instructions will fail with the error mentioned in the above "Troubleshooting West" link.

I don't think either page talks about how the west workspaces work either. I feel like the information regarding west workspaces in "Troubleshooting West" should be part of the "beginners information" on using west.

I would expect that on the "Application Development" page, there would be a step or blue note box stating "You must set the ZEPHYR_BASE for the below instructions to work".

When I was initially starting Zephr I was pretty confused about how west was meant to work (I thought it would remember where your main "install" was automatically, since I didn't see the docs state you need to specify the ZEPHYR_BASE environment variable)

The reason I'm raising this as a question instead of an issue/PR is I feel like I'm missing something here (or missed part of the documentation). I'm also not sure if zephyr is meant to set the ZEPHYR_BASE variable automatically and I've messed something up when installing. If it really is the documentation missing these instructions, I can raise a PR with the appropriate changes.

---

Edit 1:

Interestingly, 2.2.0 versions of the docs told you to run the zephyr-env.sh (Build the Blinky, Step 6), which sets the ZEPHYR_BASE variable (although it didn't say what running the script did on that page).

The old "Application" page also briefly mentions you should run the zephyr-env.sh in the "Important Build System Variables" section, explaining that it sets ZEPHYR_BASE

According to the 2.3.0 release notes in this commit, the zephyr-env.sh is no longer needed due to including the Zephyr CMake package (although that file is still present in master).

I did try re-running west zephyr-export (which I might have missed when I was following the Blinky tutorial), which does create a package in ~/.cmake/package/Zephyr, but that still doesn't let me use west from a Zephyr freestanding application (an app not in the main west workspace)

---

Edit 2:

Building without west (using cmake) after the Zephyr CMake package is installed seems to work as expected, even if ZEPHYR_BASE is not set.

[carlescufi]

@mbolivar-nordic and @tejlmand FYI

[drojf]

Forgot to mention that I opened a PR already, which adds a blue info box to the Application page : #33537

[tejlmand]

Hi @drojf

You're absolute correct in your observations.
Part of this is related to the fact that west originated when ZEPHYR_BASE was still a requirement.
As you have discovered, when using plain CMake, then cmake is able to look up the Zephyr repo, but this was introduced later.

This is a bit trickier with west, because west build is an extension command, so before west knows what build means, it must know which workspace to use because the build extension command is defined in the workspace.
But CMake always knows what cmake means, so therefore there is not the same dependency here.

The current plan is to extend west to support an argument so that users can tell which workspace to use, like:
west --use-this-workspace /path/to/workspace build
which of course should also support this as config setting, like west config --global default.workspace /path/to/workspace so that you can avoid passing this command if you only has a single workspace.

[drojf]

Ah, good to know that I was on the right track.

Just now I raised this PR on the west repo which improves the error message if you type an unknown command, which should provide enough info for users to fix the problem. I'm not sure if you want to accept it, but thought it would be worth bringing up anyway via a PR: zephyrproject-rtos/west#493

[mbolivar-nordic]

You are basically running into disagreement between maintainers that never got resolved, unfortunately.

My original design for west envisioned that everybody would put their applications inside the workspace, as part of a general process of removing zephyr-env.sh. Others objected to this and I added the ZEPHYR_BASE hack to west in a hurry to smooth things over. Then we never really got around to resolving the situation.

Sorry for the inconvenience and thanks for your west PR.

[marc-hb]

Slightly longer explanation at

• west init fails if ZEPHYR_BASE is set west#285

[marc-hb]

(I thought it would remember where your main "install" was automatically, ...

I don't remember seeing any concept of a "main" workspace in the documentation, so no one should expect that. On the other hand, I suspect some (mostly Windows) users could expect west to remember where their "only" workspace is. Other (mostly Unix) users would take for granted the ability to have multiple workspaces and to switch between them with a mere cd command.

My original design for west envisioned that everybody would put their applications inside the workspace...

... which of course feels like an unnecessary constraint when you restrict yourself to a single workspace.

I feel like the information regarding west workspaces in "Troubleshooting West" should be part of the "beginners information" on using west.

+1

[drojf]

(I thought it would remember where your main "install" was automatically, since I didn't see the docs state you need to specify the ZEPHYR_BASE environment variable)

I don't remember seeing any concept of a "main" workspace in the documentation, so no one should expect that. On the other hand, I suspect some (mostly Windows) users could expect west to remember where their "only" workspace is. Other (mostly Unix) users would take for granted the ability to have multiple workspaces and to switch between them with a mere cd command.

I am indeed a mainly Windows user :)

I was more thinking, "because the Application Development page does not tell me to set a west workspace, it must happen automatically or by some means I don't understand", rather than, as you mentioned, the documentation directly telling me there is a 'main workspace'.

Within a west workspace, west does "remember" certain settings, like the last board you used when you do west build, so it isn't entirely unexpected that it might remember the last west workspace you used if not specified (for a beginner), although in hindsight it doesn't make much sense. I guess it's because certain systems have a "central store" like pip and npm, so I'm used to thinking that way.

[marc-hb]

so it isn't entirely unexpected that it might remember the last west workspace you used if not specified

Because a number of Unix tools work like this, I think many Unix users have this ingrained assumption that they can have any number of "workspaces" and that the current "workspace" is automatically defined by the current directory and only by the current directory, never remembered nor hardcoded anywhere. Another, similar, big and related cultural difference is environment variables (hi ZEPHYR_BASE!) in Unix they affect at most new child processes. All these concepts are local and when they have 5 or 10 terminals open no one wants to have to remember which workspace was used last in which terminal, that would be hell.

Of course git originated from the Unix side of the world (understatement :-) and it works exactly like this[*] - even on Windows. west being a "meta-git" tool it must work like git (try: west topdir) too much dissonance otherwise. It would be super weird if a tool designed to manage multiple git repos made it difficult to have multiple west workspaces or required them to be managed differently.

I think Windows is different because Microsoft tried for years to kill the command line and I don't think the concept of a current directory works without a command line, hence the very frequent practice of hardcoding paths, "remembering" the last one used or even simpler: not even allowing multiple instances at all for some tools.

Maybe mentioning these cultural differences in some carefully chosen documentation page would help avoid some confusions.

New, smart Windows people in charge now understand that graphical interfaces and command lines are not mutually exclusive:
https://www.howtogeek.com/673729/heres-why-the-new-windows-10-terminal-is-amazing/
https://en.wikipedia.org/wiki/PowerShell
https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux
https://cli.github.com/

[*] git has an explicit (and useful) -C dir option but no "last one used" memory or concept of any default project.

[aborisovich]

I agree @marc-hb on what you said about Windows, however please note that west tool does already have a feature that works without operating in the "current directory".
Check zephyr_cmake_package and we even have it in the getting_started guide .
Command west-export causes CMake to store a location of the Zephyr project in the "Cmake export package registry" (see: CMake command export. This "registry" is OS dependant. On Windows it uses system registry entries, on linux it is some file on the filesystem (don't remember which sorry).
Every Zephyr application (ok, 90%+ of applications) start their CMake scripts with:
find_package(Zephyr HINTS $ENV{ZEPHYR_BASE}).
But in this find_package command ZEPHYR_BASE variable is only a "hint" on where to search.
Standard command usage is to read that "Cmake export package registry" for the directory where Zephyr project is.

In my opinion we should stick with what we already have and also try to be harmonic to CMake which is our primary tool.
So basically we can remove "ZEPHYR_BASE" variable anywhere and implement west init command so it makes a call to west-export and does not set a ZEPHYR_BASE variable. Yeah I love git too but west is not git specific right?

[marc-hb]

I think Windows is different because Microsoft tried for years to kill the command line

Wow, I had heard things and heard rumours but I had no idea how accurate that was:

https://www.theregister.com/AMP/2022/06/29/snover_leaves_micrososft/
https://youtu.be/nKyF8fzed0w?t=957

"'Admins don't want command line interfaces,'" or "'exactly what part of effing Windows is confusing you?'"

[mbolivar-nordic]

Yeah I love git too but west is not git specific right?

It is, actually

[o7-machinehum]

What's the status of freestanding projects? These are very useful if you don't want dozens of checkouts of zephyr (and you can't put all your code into one zephyr fork). This seems to work fine on my Linux machine, but am having issues on my osx machine.

machinehum@walkers-MacBook-Pro pl02-firmware % source ~/zephyr/zephyr-env.sh                        
machinehum@walkers-MacBook-Pro pl02-firmware % west build -b esp32 esp32-app --build-dir build/esp32
usage: west [-h] [-z ZEPHYR_BASE] [-v] [-V] <command> ...
west: unknown command "build"; do you need to run this inside a workspace?
machinehum@walkers-MacBook-Pro pl02-firmware % 

[pamolloy]

I receive the same error on Fedora 39 with west 1.2.0 as well as a Ubuntu 22.04.1 container.

[pamolloy]

Since I couldn't get the above to work it was recommended to use one Zephyr project and then git worktree inside the Zephyr directory/repo:

west init zephyrproject
cd zephyrproject/zephyr
west update
source zephyr-env.sh
west build -b qemu_riscv32 samples/hello_world/
# Switch to another development branch or tag
git worktree add ../zephyr-v3.5.0 v3.5.0
cd ../zephyr-v3.3.0
west update
source zephyr-env.sh
west build -b qemu_riscv32 samples/hello_world/

I tried several versions of the above, but also couldn't get it to work.

[marc-hb]

@pamolloy this discussion is about building OUTSIDE a west workspace. But the steps you gave add a git worktree INSIDE your unique west workspace. I don't know what specific problem you're trying to solve (you didn't give any error message) but it seems unrelated to this discussion.

One of the things people don't like about being forced to work inside west workspace(s) is the large duplication of identical dependencies and git checkouts across their multiple west workspaces (one disk-consuming workspace for each application).

I'm speculating from the very little information your shared that the git worktree recommendation you received was aimed at that problem: reducing duplication across multiple workspaces (cause that's the main purpose of git worktree). But that's not what your steps above do. I can't understand what they're trying to achieve.

[pamolloy]

My comment is a reply to the previous comment where the author wrote the following:

These are very useful if you don't want dozens of checkouts of zephyr (and you can't put all your code into one zephyr fork).

The worktree example apparently used to work according to a colleague of mine, but no longer does with the latest version of Zephyr.

The goal isn't to reduce duplication across workspaces, but to prevent "dozens of checkouts of Zephyr" or perhaps more accurately, clones of Zephyr.

[pamolloy]

I'd be happy to explain git worktree, but perhaps #build-system on Discord is a better place?