Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add build sdkconfig support #870

Merged
merged 3 commits into from
Jan 20, 2023
Merged

add build sdkconfig support #870

merged 3 commits into from
Jan 20, 2023

Conversation

brianignacio5
Copy link
Collaborator

@brianignacio5 brianignacio5 commented Jan 9, 2023

Description

Add support for multiple build configuration as done in esp-idf cmake multi_config example.

Load sdkconfig path as follows:

  1. From current workspace folder root directory CMakeLists.txt SDKCONFIG variable else
  2. From SDKCONFIG environment variable (either system or extension defined in idf.customExtraVars setting) else
  3. From current workspace folder root directory sdkconfig file.

Also add idf.sdkconfigDefaults to define the set of SDKCONFIG_DEFAULTS to use to initially create the sdkconfig.

Fix #855

Type of change

Please delete options that are not relevant.

  • New feature (non-breaking change which adds functionality)

How has this been tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

  • Manual testing building and running SDK Configuration editor.

Test Configuration:

  • ESP-IDF Version: 4.4. 5.1
  • OS (Windows,Linux and macOS): macOS

Checklist

  • PR Self Reviewed
  • Applied Code formatting
  • Added Documentation
  • Added Unit Test
  • Verified on all platforms - Windows,Linux and macOS

@brianignacio5 brianignacio5 merged commit 3c37bfa into master Jan 20, 2023
@brianignacio5 brianignacio5 deleted the enhance/build-sdkconfig branch January 20, 2023 06:29
@chipweinberger
Copy link

Thanks for adding this feature Brian!

Load sdkconfig path as follows:
From current workspace folder root directory CMakeLists.txt SDKCONFIG variable else

I am trying to understand, how does this support "multiple" build configuration? It seems like this PR is more about supporting a custom SDKCONFIG path?

Is the idea that I am supposed to change the CMakeLists.txt SDKCONFIG variable myself? i.e. without a UI

@chipweinberger
Copy link

chipweinberger commented Apr 7, 2023

I just found and read your documentation (thanks)!

To answer my own question, there is no UI for switching multiple build configs at this time. This PR is for supporting multiple heirarchical sdkconfigs, such as: sdkconfig.common + sdkconfig.prod combined together. If you have a separate sdkconfig.dev, you need to change the SDKCONFIG var yourself (no UI).

Edit: actually I think I can use this UI to switch between sdkconfig.dev vs sdkconfig.prod: #882

Thanks for this great feature.

Screen Shot 2023-04-07 at 2 09 34 PM

@brianignacio5
Copy link
Collaborator Author

Thanks for your feedback @chipweinberger Still need to have more links to docs in the main readme to make it more visible.

There is also a tutorial for the Project Configuration Editor in here about using this UI you mentioned. If you have any feedback let us know !

@chipweinberger
Copy link

chipweinberger commented Apr 26, 2023

I'm still confused by how this should work.

I've set my sdkconfig vars as such:

Screenshot 2023-04-26 at 4 02 00 PM

But I still don't understand, which file does the SDK Config Editor load and save to?

Screenshot 2023-04-26 at 4 01 42 PM

Is there any way to Edit my sdkconfig.defaults or sdkconfig.prod file using this UI?

@chipweinberger
Copy link

chipweinberger commented Apr 26, 2023

IMO, the "SDK Configuration Editor" should be per file.

Instead of UI title saying "SDK Configuration Editor"

Screenshot 2023-04-26 at 10 14 34 PM

^ it should say "sdkconfig.default" or "sdkconfig.prod" depending on the file you are currently editing.


Also the idf.sdkconfigDefaults variable does not make sense to me.

It should support multiple configurations.

i.e.

idf.sdkconfigDefaults : {
     "dev: [
        "/Volumes/User/MBP-Google-Drive/jamcorder/firmware/jamcorder-firmware/jamcorder_app/sdkconfig.defaults"
    ]
     "prod: [
        "/Volumes/User/MBP-Google-Drive/jamcorder/firmware/jamcorder-firmware/jamcorder_app/sdkconfig.defaults"
        "/Volumes/User/MBP-Google-Drive/jamcorder/firmware/jamcorder-firmware/jamcorder_app/sdkconfig.prod"
    ]
}

Next to the Gear Icon Screenshot 2023-04-26 at 10 16 30 PM should be a label for the currently active config (in this case it would say "dev" or "prod" since that is what I've named my 2 configurations above^^

In the "SDK Configuration Editor":

  1. the title should say which file you are currently editing, as I mentioned.
  2. there should be another row of tabs in the "SDK Configuration Editor" to switch between the files in the configuration.
    Like this (nested tabs):

Screenshot 2023-04-26 at 5 19 52 PM

@brianignacio5
Copy link
Collaborator Author

Hi @chipweinberger so the idea of sdkconfig.defaults is to build an sdkconfig file.

Let's say you have 2 products, prod1 and prod2. Each product has a single resulting sdkconfig file in ${workspaceFolder}/build_prod1/sdkconfig and ${workspaceFolder}/build_prod2/sdkconfig.

  1. ${workspaceFolder}/build_prod1/sdkconfig is built from sdkconfig.defaults; [sdkconfig.common, sdkconfig.prod1]. See how sdkconfig.common just list a set of common settings for prod1 and prod2 while sdkconfig.prod1 have product 1 specific settings.

  2. build_prod2/sdkconfig is built from sdkconfig.defaults; [sdkconfig.common, sdkconfig.prod2]. See how sdkconfig.common just list a set of common settings for prod1 and prod2 while sdkconfig.prod2 have product 2 specific settings.

  3. For the tutorial example, the location of sdkconfig file is determine in the CMakeLists.txt SDKCONFIG variable.

  4. When you choose a profile, say prod1, you are setting the build directory to build_prod1 so when you try to build or use the SDK Configuration Editor it will use the build_prod1/sdkconfig to save settings and build your binaries in ${workspaceFolder}/build_prod1.

The currently selected profile is shown in the status bar at the bottom of the Visual Studio Code.

Does this help @chipweinberger ? We should clarify a bit more of this in the documentation and tutorials.

@brianignacio5
Copy link
Collaborator Author

By the way, We are also adding the sdkconfig file location and IDF_TARGET to such profile configuration. So will try to explain better the difference between sdkconfig and sdkconfig.defaults

@chipweinberger
Copy link

chipweinberger commented Apr 27, 2023

question 1

Does this help @chipweinberger ?

Yes, it mostly makes sense to me the build process you are describing.

But I want to know about the editing workflow.

How do I edit a specific sdkconfig file using the "SDK Config Editor"?

question 2

You say "sdkconfig.common". How do I set the name of "sdkconfig.common"? Does it have to be called "sdkconfig.common" or can I call it something else?

question 3

The json format I describe makes more intuitive sense to me. What settings should I set in the settings you describe to achieve the equivalent setup?

idf.sdkconfigDefaults : {
     "dev: [
        "/Volumes/User/MBP-Google-Drive/jamcorder/firmware/jamcorder-firmware/jamcorder_app/sdkconfig.defaults"
    ]
     "prod: [
        "/Volumes/User/MBP-Google-Drive/jamcorder/firmware/jamcorder-firmware/jamcorder_app/sdkconfig.defaults"
        "/Volumes/User/MBP-Google-Drive/jamcorder/firmware/jamcorder-firmware/jamcorder_app/sdkconfig.prod"
    ]
}

question 4

Does the json description format I am describing above ^^ make sense to you?

question 5

Maybe you can consider if the UI could be improved, to be simpler to understand?


Thank you for working on this. I hope my feedback & questions are useful.

@brianignacio5
Copy link
Collaborator Author

For the current behavior, 1) Select a profile prod1 which has a build_prod1 build directory. The sdconfig path will be resolved by replacing CMAKE_BINARY_DIR in ${CMAKE_BINARY_DIR}/sdkconfig from CMakeLists.txt with the current profile build directory path. Any changes you make in the SDK Configuration editor will be saved in build_prod1/sdconfig.

NOTE: Will add the sdkconfig as another setting in the project configuration wizard so the user can manually define the sdkconfig used for development.

  1. sdkconfig.common is just the name of the file listed in sdkconfigDefaults array. It can be any name. The way sdkconfig is generated from these defaults is resolving these sdkconfig files in order. So

sdkconfigDefaults; [sdkconfig.common, sdkconfig.prod1] will use values from sdkconfig.common (which a typical use case will be to put common settings for all profiles here) and then load on top values of sdkconfig.prod1 (which would have a specific product setting).

The example is just to illustrate why to use sdkconfigDefaults. You would define these values when you want to distribute the source code in a cleaner way without ESP-IDF generated values in resulting sdkconfig. If you look at the Multiple configuration example the sdkconfig.prod1 and sdkconfig.prod2 specify unique values but resulting sdkconfig files include ESP-IDF values.

  1. I think it might be better to think that the use of sdkconfigDefaults for production only and use sdkconfig for development purposes. Usually the sdkconfig is generated by the sdk configuration editor and build process. You modify sdkconfig and build again.

JSON would be

"product1": {
   "idf.sdkconfigDefaults" : {
      "${workspaceFolder}/sdkconfig.common"
      "${workspaceFolder}/sdkconfig.prod1"
   },
   "idf.sdkconfig": "${workspaceFolder}/sdkconfig.product1",
   "idf.buildDirectoryPath": "${workspaceFolder}/product1_build"
},
"product2": {
   "idf.sdkconfigDefaults" : {
      "${workspaceFolder}/sdkconfig.common"
      "${workspaceFolder}/sdkconfig.prod2"
   },
   "idf.sdkconfig": "${workspaceFolder}/sdkconfig.product2",
   "idf.buildDirectoryPath": "${workspaceFolder}/product2_build"
}
  1. I understand the idea you are referring. But the purposes of these files is more for production while sdkconfig is more for development process.

  2. This files are used in many CMake projects so in terminology it would be best to keep them close as original intent. We do need to make it easier to understand what are there for better though.

@chipweinberger
Copy link

chipweinberger commented Apr 27, 2023

I still do not really understand very well.

Question 1

For the current behavior, 1) Select a profile prod1 which has a build_prod1 build directory. The sdkconfig path will be resolved by replacing CMAKE_BINARY_DIR in ${CMAKE_BINARY_DIR}/sdkconfig from CMakeLists.txt with the current profile build directory path. Any changes you make in the SDK Configuration editor will be saved in build_prod1/sdkconfig.

"Any changes you make in the SDK Configuration editor will be saved in build_prod1/sdkconfig.",

That does not make sense to me. The build folder is ephemeral, and gets deleted on full clean. If you "save" to the build folder, then it is not really saved at all. It will be deleted! Am I supposed to add the build folder to my git repo? Or am I suppose to manually copy the settings from the build folder to my repo? Very frustrating user experience!

It should save to our workspace folder! "git status" should show the changes I make, so that I can commit it!

Without fixing this issue, "SDK Configuration editor" is useless. Now I have to manually edit the files every time! So frustrating! What is the point of "SDK Configuration editor" now?

question 3

I think it might be better to think that the use of sdkconfigDefaults for production only and use sdkconfig for development purposes. Usually the sdkconfig is generated by the sdk configuration editor and build process. You modify sdkconfig and build again.

This seems to be the biggest misunderstanding. I really don't understand what sdkconfigDefaults does. I don't understand how it affects building. I dont understand how it affects the "SDK Config Editor". I don't understand how to switch between multiple configs.

summary

Overall, I just don't really get it. I don't understand how it is "supposed" to work. The PR made things more complicated, and AFAICT, made things worse for me.

Before SDK Config Editor created a sdkconfig file in my workspace. Things were simple. I understood it.

After I switched my product to multi config, the VSC extension did not support it at first, so I continued to also use a sdkconfig file in my workspace. I would manually copy settings to me sdkconfig.defaults and sdkconfig.prod as needed. Things were simple. I understood it.

AFAICT, the PR makes things worse. The sdkconfig file is no longer created in my workspace. "git status" no longer works after I save a change, forcing me to manually copy settings from my build folder?

my opinion

My opinion: this pr should be reverted. I've read the documentation, and I still don't get it.

I am an an "advanced" user. If an "advanced" user does not understand it, then very few developers are going to understand it. This is a big issue.

I just want the old behavior back...

IMO this feature needs to be re-designed.

@brianignacio5
Copy link
Collaborator Author

brianignacio5 commented Apr 27, 2023

This multi config is not the default use case of ESP-IDF. Not at all. In fact is a very specific use case. You don't need to use the project configuration editor at all. Previous behavior still works well.

I just trying to explain to you that sdkconfig is a file generated by the build system. See the esp-idf examples and you will notice that sdkconfig file doesn't exist. it is generated after running idf.py menuconfig. There is a sdkconfig.ci which is used to "generate" the sdkconfig to be used later in the build process.

This example (and the idea of project configuration in general) is to create multiple projects with a single source code. Let me give you an example use case. Say you have project using multiple tft screens. You want product 1 to use a screen A while product 2 to use a screen B. In your code you write a Kconfig file to choose between screen based on a sdkconfig value say CONFIG_SCREEN_TO_USE.

In the ESP-IDF documentation about sdkconfig defaults

Basically use sdkconfig.defaults as generators to have multiple sdkconfig files.

[sdkconfig.common, sdkconfig.prod1] -> ${workspaceFolder}/sdkconfig.product1

[sdkconfig.common, sdkconfig.prod2] -> ${workspaceFolder}/sdkconfig.product2

This feature is not intended at all to be used for most ESP-IDF projects.

That why I mentioned that think of sdkconfig.defaults use more for a production use case to build multiple binaries from a single source at once than from a development point of view. In development point of view you get the sdkconfig working to your needs and when you want to release you would like to have a minimal set of sdkconfig values that you could, for example, share as open source and is not limited to specific ESP-IDF version since other ESP-IDF sdkconfig values.

@chipweinberger
Copy link

chipweinberger commented Apr 27, 2023

Thanks for your comments Brian.

This feature is not intended at all to be used for most ESP-IDF projects.

Having a "dev" and "release" mode is common use case. Most products need these. This is all I want.

I just trying to explain to you that sdkconfig is a file generated by the build system.

Yes I do understand that. ${workspacefolder}/build/sdkconfig.

But, AFAIK, ${workspacefolder}/sdkconfig was always created the first time you opened the "gear" icon. And updated when you clicked "save" in the SDK Config Editor. This is what I don't know how to do anymore.

Previous behavior still works well.

Before the sdkconfig file was created & edited in the workspace. Now it no longer does that anymore. Am I wrong?

@chipweinberger
Copy link

chipweinberger commented Apr 27, 2023

Idea: Perhaps you should create an example that has "dev" and "release" modes.

Show:

  1. how to setup "dev" and "release" in the settings
  2. how to edit "dev" using the "SDK Config Editor" & save changes to your repo (not build folder)
  3. how to edit "release" using the "SDK Config Editor" & save changes to your repo (not build folder)
  4. how to build & flash "dev"
  5. how to build & flash "release"

Note: I know how to do all of this manually using idf.py. But I do not know how to do this in VSC Extension!

@brianignacio5
Copy link
Collaborator Author

Before the sdkconfig file was created & edited in the workspace. Now it no longer does that anymore. Am I wrong?

The sdkconfig file is created and edited in the workspace by default in CMake projects and most ESP-IDF projects.

Here is extending the use of a single sdkconfig to use many sdkconfig files for a single code. You can switch between one and another switch configuration profile. For this specific esp-idf example, build system multiple configuration example, the sdkconfig default path is modified in here

@brianignacio5
Copy link
Collaborator Author

Thanks @chipweinberger for this conversation. It really helps us to improve the intent of the feature and write better documentation.

By the way there is a idf.py save-defconfig to save sdkconfig -> sdkconfig.defaults which could help with this tutorials.

@brianignacio5
Copy link
Collaborator Author

brianignacio5 commented Apr 27, 2023

I just want to add that the original idea of the feature is for make easier scripting of these idf.py build and other commands.

While sdkconfig and sdkconfig defaults is one of many things we can do with idf.py -B build_path -DSDKCONFIG=sdkconfig_path -DSDKCONFIG_DEFAULTS=[common,prod1] command we can do many other things based on the arguments here. I think we really need to clarify how all these translate to CLI commands.

@chipweinberger
Copy link

Any progress on #870 (comment)?

@brianignacio5
Copy link
Collaborator Author

Sorry for late reply, I haven't had bandwidth to continue writing these docs but for sure I'll make time before next release and I'll tag you in the MR .

@chipweinberger
Copy link

I appreciate your communications! Thanks Brian.

@chipweinberger
Copy link

chipweinberger commented May 27, 2023

Just want to bump this thread again =) Looking forward to the docs for #870 (comment)

& save changes to your repo (not build folder)

This is the part I most want to know how to do. As well as how to switch between dev and release quickly.

Ideally VSC would support sdkconfig.defaults + sdkconfig.prod, where .prod is just the diff from defaults. I think it would work like this:

  • Opening sdkconfig.defaults : just load as normal
  • Saving sdkconfig.defaults : just save as normal
  • Opening sdkconfig.prod : When you open .prod it would first load .defaults, and then apply .prod on top of if
  • Saving sdkconfig.prod : When you save .prod, it will diff the changes from .defaults and save just the diff to.prod

The only simple way that I can think of to support this, is to have a switcher in the UI. Clicking .prod would allow you to switch between .defaults and .prod.
Screenshot 2023-05-26 at 7 57 00 PM

If that is too difficult to support, even knowing how to do a fully separate sdkconfig.dev & sdkconfig.release would be useful too.

@brianignacio5
Copy link
Collaborator Author

Could you check this PR with the docs ? #983

The idf.py save-defconfig works on ESP-IDF v5.0

@chipweinberger
Copy link

overall looks good!

i left some comments and suggestions!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature Request]: support sdkconfig.defaults instead of sdkconfig (VSC-1019)
2 participants