Skip to content

Commit

Permalink
Compile time code generation - add relevant code to all example build…
Browse files Browse the repository at this point in the history
…s (plus one sample codegen moved) (#23247)

* Compile time code generation.

Picked PluginApplicationCallbacks because it is simple and added
it to a compile-time generated list.

The main exercise for this is to make sure we can inject the
appropriate calls to compile time generation to all examples.
It is not yet perfect as it only involves include directories,
however it does a "if it compiles it works, otherwise it fails
to compile" approach.

Uses codegen.py rather than ZAP for this particular file. For
other files, we may chose to use ZAP however for that we have
to:
  - ensure it is stand-alone runnable (likely as an installable app)
  - figure out any multi-processing conflicts as cmake/gn WILL run
    build steps in parallel and zap shares a common DB.

* Remove PluginApplicationCallbacks.zapt

* Restyle

* Add idl to build instructions

* Disable pylint for now. It looks like pylint is a lot more strict than expected and this was triggered only if we have dependencies

* Ensure we run codegen so sanitizer can find generated files

* Revert telink changes after master merge: no need for separate codegen anymore as telink now uses standard data model methods

* Ensure all needed codegen is done (for clang tidy at least). Tested in linux, may need to adjust darwin

* Code review comment: centralize the generator/string for known generators

* Undo undesired file creation

* Adjust the run_codegen_targets a bit

* Undo shellharden: it breaks the script

* Add exception on shellharden on run_codegen_targets

* Fix clang tidy location for darwin: it uses out/default not out/sanitizers

* Add a server cluster to the several_clusters unit test, to validate that we generate server callbacks to

* Make bridge generate ONLY client clusters as it reuses the same name of generation

* Separate client and server headers for bridge codegen

* Resolve bridge: clients have default names, Server headers are separate. This is until we figure out real names

* Fix build rules

* Do not lint test matter files for spec compliance

* Github runners do not use bash by default. Make the conditionals different for lint skipping

* Fix operator for lint exception check

* Undo typo
  • Loading branch information
andy31415 authored and pull[bot] committed Nov 28, 2023
1 parent c0b9f8b commit 1555656
Show file tree
Hide file tree
Showing 84 changed files with 548 additions and 2,301 deletions.
8 changes: 8 additions & 0 deletions .github/workflows/build.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -210,6 +210,10 @@ jobs:
BUILD_TYPE=sanitizers scripts/build/gn_gen.sh --args="$GN_ARGS" --export-compile-commands
BUILD_TYPE=sanitizers scripts/tests/gn_tests.sh
done
- name: Ensure codegen is done for sanitize
timeout-minutes: 45
run: |
./scripts/run_in_build_env.sh "./scripts/run_codegen_targets.sh out/sanitizers"
- name: Clang-tidy validation
timeout-minutes: 45
run: |
Expand Down Expand Up @@ -401,6 +405,10 @@ jobs:
scripts/run_in_build_env.sh "ninja -C ./out/$BUILD_TYPE"
BUILD_TYPE=$BUILD_TYPE scripts/tests/gn_tests.sh
done
- name: Ensure codegen is done for sanitize
timeout-minutes: 45
run: |
./scripts/run_in_build_env.sh "./scripts/run_codegen_targets.sh out/default"
- name: Clang-tidy validation
timeout-minutes: 45
run: |
Expand Down
7 changes: 7 additions & 0 deletions .github/workflows/lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,13 @@ jobs:
if [ "$idl_file" = './examples/thermostat/thermostat-common/thermostat.matter' ]; then continue; fi
if [ "$idl_file" = './examples/window-app/common/window-app.matter' ]; then continue; fi
# Test files are intentionally small and not spec-compilant, just parse-compliant
if [ "$idl_file" = "./scripts/idl/tests/inputs/cluster_struct_attribute.matter" ]; then continue; fi
if [ "$idl_file" = "./scripts/idl/tests/inputs/global_struct_attribute.matter" ]; then continue; fi
if [ "$idl_file" = "./scripts/idl/tests/inputs/optional_argument.matter" ]; then continue; fi
if [ "$idl_file" = "./scripts/idl/tests/inputs/several_clusters.matter" ]; then continue; fi
if [ "$idl_file" = "./scripts/idl/tests/inputs/simple_attribute.matter" ]; then continue; fi
./scripts/run_in_build_env.sh "./scripts/idl_lint.py --log-level warn $idl_file" >/dev/null || exit 1
done
Expand Down
1 change: 1 addition & 0 deletions .restyled.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ exclude:
- "examples/chef/zzz_generated/**/*"
- "examples/platform/nxp/k32w/k32w0/scripts/demo_generated_certs/**/*"
- "integrations/cloudbuild/*.yaml" # uglier long command line content
- "scripts/run_codegen_targets.sh" # shellharden breaks for loops over command outputs


changed_paths:
Expand Down
70 changes: 70 additions & 0 deletions build/chip/chip_codegen.cmake
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# Run chip code generation.
#
# Example usage:
# chip_codegen("app"
# INPUT "some_file.matter"
# GENERATOR "bridge"
# OUTPUTS
# "bridge/OnOff.h"
# "bridge/LevelControl.h"
# "bridge/Switch.h"
# # ... more outputs
# OUTPUT_PATH DIR_NAME_VAR
# OUTPUT_FILES FILE_NAMES_VAR
# )
#
# Arguments:
# INPUT - the name of the ".matter" file to use for generation
# GENERATOR - generator to use for codegen.py
# OUTPUTS - EXPECTED output names. MUST match actual outputs
#
# OUTPUT_PATH - [OUT] output variable will contain the directory where the
# files will be generated
# OUTPUT_FILES - [OUT] output variable will contain the path of generated files.
# suitable to be added within a build target
#
function(chip_codegen TARGET_NAME)
cmake_parse_arguments(ARG
""
"INPUT;GENERATOR;OUTPUT_PATH;OUTPUT_FILES"
"OUTPUTS"
${ARGN}
)

set(GEN_FOLDER "${CMAKE_BINARY_DIR}/gen/${TARGET_NAME}/${ARG_GENERATOR}")

string(REPLACE ";" "\n" OUTPUT_AS_NEWLINES "${ARG_OUTPUTS}")

file(MAKE_DIRECTORY "${GEN_FOLDER}")
file(GENERATE
OUTPUT "${GEN_FOLDER}/expected.outputs"
CONTENT "${OUTPUT_AS_NEWLINES}"
)


set(OUT_NAMES)
foreach(NAME IN LISTS ARG_OUTPUTS)
list(APPEND OUT_NAMES "${GEN_FOLDER}/${NAME}")
endforeach()

# Python is expected to be in the path
#
# find_package(Python3 REQUIRED)
add_custom_command(
OUTPUT "${OUT_NAMES}"
COMMAND "${CHIP_ROOT}/scripts/codegen.py"
ARGS "--generator" "${ARG_GENERATOR}"
"--output-dir" "${GEN_FOLDER}"
"--expected-outputs" "${GEN_FOLDER}/expected.outputs"
"${ARG_INPUT}"
DEPENDS
"${ARG_INPUT}"
VERBATIM
)

add_custom_target(${TARGET_NAME} DEPENDS "${OUT_NAMES}")

# Forward outputs to the parent
set(${ARG_OUTPUT_FILES} "${OUT_NAMES}" PARENT_SCOPE)
set(${ARG_OUTPUT_PATH} "${GEN_FOLDER}" PARENT_SCOPE)
endfunction()
45 changes: 45 additions & 0 deletions build/chip/esp32/esp32_codegen.cmake
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
#
# Copyright (c) 2022 Project CHIP Authors
# All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#



macro(chip_app_component_codegen IDL_NAME)
include("${CHIP_ROOT}/build/chip/chip_codegen.cmake")

# The IDF build system performs a two-pass expansion to determine
# component expansion. The first pass runs in script-mode
# to determine idf_component_register REQUIRES and PRIV_REQUIRES.
#
# We can only set up code generation during the 2nd pass
#
# see https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-guides/build-system.html
if (NOT CMAKE_BUILD_EARLY_EXPANSION)

chip_codegen(app-codegen
INPUT "${IDL_NAME}"
GENERATOR "cpp-app"
OUTPUTS
"app/PluginApplicationCallbacks.h"
OUTPUT_PATH APP_GEN_DIR
OUTPUT_FILES APP_GEN_FILES
)

target_include_directories(${COMPONENT_LIB} PUBLIC "${APP_GEN_DIR}")

add_dependencies(${COMPONENT_LIB} app-codegen)
endif()
endmacro()
8 changes: 6 additions & 2 deletions examples/all-clusters-app/esp32/main/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#

# (Uses default behaviour of compiling all source files in directory, adding 'include' to include path.)
# The list of src and include dirs must be in sync with that in all-clusters-app/esp32/main/component.mk
set(PRIV_INCLUDE_DIRS_LIST
Expand Down Expand Up @@ -137,6 +136,12 @@ idf_component_register(PRIV_INCLUDE_DIRS ${PRIV_INCLUDE_DIRS_LIST}
EXCLUDE_SRCS ${EXCLUDE_SRCS_LIST}
PRIV_REQUIRES ${PRIV_REQUIRES_LIST})

get_filename_component(CHIP_ROOT ${CMAKE_SOURCE_DIR}/third_party/connectedhomeip REALPATH)

include("${CHIP_ROOT}/build/chip/esp32/esp32_codegen.cmake")

chip_app_component_codegen("${CHIP_ROOT}/examples/all-clusters-app/all-clusters-common/all-clusters-app.matter")

set_property(TARGET ${COMPONENT_LIB} PROPERTY CXX_STANDARD 17)
target_compile_options(${COMPONENT_LIB} PRIVATE "-DCHIP_HAVE_CONFIG_H")
target_compile_options(${COMPONENT_LIB} PUBLIC
Expand All @@ -145,7 +150,6 @@ target_compile_options(${COMPONENT_LIB} PUBLIC

if (CONFIG_ENABLE_PW_RPC)

get_filename_component(CHIP_ROOT ${CMAKE_SOURCE_DIR}/third_party/connectedhomeip REALPATH)

set(PIGWEED_ROOT "${CHIP_ROOT}/third_party/pigweed/repo")
include(${PIGWEED_ROOT}/pw_build/pigweed.cmake)
Expand Down
8 changes: 6 additions & 2 deletions examples/all-clusters-minimal-app/esp32/main/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,12 @@ idf_component_register(PRIV_INCLUDE_DIRS ${PRIV_INCLUDE_DIRS_LIST}
SRC_DIRS ${SRC_DIRS_LIST}
PRIV_REQUIRES ${PRIV_REQUIRES_LIST})

get_filename_component(CHIP_ROOT ${CMAKE_SOURCE_DIR}/third_party/connectedhomeip REALPATH)

include("${CHIP_ROOT}/build/chip/esp32/esp32_codegen.cmake")

chip_app_component_codegen("${CHIP_ROOT}/examples/all-clusters-minimal-app/all-clusters-common/all-clusters-minimal-app.matter")

set_property(TARGET ${COMPONENT_LIB} PROPERTY CXX_STANDARD 17)
target_compile_options(${COMPONENT_LIB} PRIVATE "-DCHIP_HAVE_CONFIG_H")
target_compile_options(${COMPONENT_LIB} PUBLIC
Expand All @@ -136,8 +142,6 @@ target_compile_options(${COMPONENT_LIB} PUBLIC

if (CONFIG_ENABLE_PW_RPC)

get_filename_component(CHIP_ROOT ${CMAKE_SOURCE_DIR}/third_party/connectedhomeip REALPATH)

set(PIGWEED_ROOT "${CHIP_ROOT}/third_party/pigweed/repo")
include(${PIGWEED_ROOT}/pw_build/pigweed.cmake)
include(${PIGWEED_ROOT}/pw_protobuf_compiler/proto.cmake)
Expand Down
6 changes: 6 additions & 0 deletions examples/bridge-app/esp32/main/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,12 @@ idf_component_register(PRIV_INCLUDE_DIRS
"${CMAKE_SOURCE_DIR}/third_party/connectedhomeip/examples/providers"
PRIV_REQUIRES chip QRCode bt)

get_filename_component(CHIP_ROOT ${CMAKE_SOURCE_DIR}/third_party/connectedhomeip REALPATH)

include("${CHIP_ROOT}/build/chip/esp32/esp32_codegen.cmake")

chip_app_component_codegen("${CHIP_ROOT}/examples/bridge-app/bridge-common/bridge-app.matter")

set_property(TARGET ${COMPONENT_LIB} PROPERTY CXX_STANDARD 14)
target_compile_options(${COMPONENT_LIB} PRIVATE "-DCHIP_HAVE_CONFIG_H")
target_compile_options(${COMPONENT_LIB} PUBLIC
Expand Down
7 changes: 4 additions & 3 deletions examples/chef/esp32/main/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
#
# (Uses default behaviour of compiling all source files in directory, adding 'include' to include path.)

get_filename_component(CHIP_ROOT ${CMAKE_CURRENT_LIST_DIR}/../../../.. REALPATH)
get_filename_component(CHIP_ROOT ../third_party/connectedhomeip REALPATH)
get_filename_component(CHEF ${CMAKE_CURRENT_SOURCE_DIR}/../../ REALPATH)
get_filename_component(GEN_DIR ${CHEF}/out/${SAMPLE_NAME}/zap-generated REALPATH)

Expand Down Expand Up @@ -107,6 +107,9 @@ idf_component_register(PRIV_INCLUDE_DIRS
PRIV_REQUIRES chip nvs_flash bt console esp32_mbedtls QRCode tft screen-framework spidriver
SRC_DIRS ${SRC_DIRS_LIST})

include("${CHIP_ROOT}/build/chip/esp32/esp32_codegen.cmake")
chip_app_component_codegen("${CHEF}/devices/${SAMPLE_NAME}.matter")

set_property(TARGET ${COMPONENT_LIB} PROPERTY CXX_STANDARD 17)
target_compile_options(${COMPONENT_LIB} PRIVATE "-DCHIP_HAVE_CONFIG_H")
target_compile_options(${COMPONENT_LIB} PUBLIC
Expand All @@ -115,8 +118,6 @@ target_compile_options(${COMPONENT_LIB} PUBLIC

if (CONFIG_ENABLE_PW_RPC)

get_filename_component(CHIP_ROOT ../third_party/connectedhomeip REALPATH)

set(PIGWEED_ROOT "${CHIP_ROOT}/third_party/pigweed/repo")
include(${PIGWEED_ROOT}/pw_build/pigweed.cmake)
include(${PIGWEED_ROOT}/pw_protobuf_compiler/proto.cmake)
Expand Down
3 changes: 2 additions & 1 deletion examples/dynamic-bridge-app/bridge-common/BUILD.gn
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ chip_data_model("dynamic-bridge-common") {

zap_pregenerated_dir =
"${chip_root}/zzz_generated/dynamic-bridge-app/zap-generated"

is_server = true

# TODO: the definition of DYNAMIC_ENDPOINT_COUNT needs find a common home!
Expand All @@ -30,7 +31,7 @@ chip_data_model("dynamic-bridge-common") {

if (chip_enable_pw_rpc) {
import("//build_overrides/pigweed.gni")
import("$dir_pw_protobuf_compiler/proto.gni")
IMPORT("$dir_pw_protobuf_compiler/proto.gni")

pw_proto_library("bridge_service") {
sources = [ "protos/bridge_service.proto" ]
Expand Down
7 changes: 6 additions & 1 deletion examples/light-switch-app/esp32/main/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
# (Uses default behaviour of compiling all source files in directory, adding 'include' to include path.)

# (Uses default behaviour of compiling all source files in directory, adding 'include' to include path.)
idf_component_register(PRIV_INCLUDE_DIRS
"${CMAKE_SOURCE_DIR}/third_party/connectedhomeip/zzz_generated/light-switch-app"
"${CMAKE_CURRENT_LIST_DIR}/include"
Expand Down Expand Up @@ -61,6 +61,11 @@ idf_component_register(PRIV_INCLUDE_DIRS
"${CMAKE_SOURCE_DIR}/third_party/connectedhomeip/src/app/clusters/groups-server"
"${CMAKE_SOURCE_DIR}/third_party/connectedhomeip/src/app/clusters/group-key-mgmt-server"
PRIV_REQUIRES chip QRCode bt app_update)
get_filename_component(CHIP_ROOT ${CMAKE_SOURCE_DIR}/third_party/connectedhomeip REALPATH)

include("${CHIP_ROOT}/build/chip/esp32/esp32_codegen.cmake")

chip_app_component_codegen("${CHIP_ROOT}/examples/light-switch-app/light-switch-common/light-switch-app.matter")

set_property(TARGET ${COMPONENT_LIB} PROPERTY CXX_STANDARD 17)
target_compile_options(${COMPONENT_LIB} PRIVATE "-DCHIP_HAVE_CONFIG_H")
Expand Down
8 changes: 7 additions & 1 deletion examples/lighting-app/esp32/main/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,13 @@
# (Uses default behaviour of compiling all source files in directory, adding 'include' to include path.)

set(PRIV_INCLUDE_DIRS_LIST
"${APP_GEN_DIR}"
"${CMAKE_SOURCE_DIR}/third_party/connectedhomeip/zzz_generated/lighting-app"
"${CMAKE_SOURCE_DIR}/third_party/connectedhomeip/examples/providers"
"${CMAKE_SOURCE_DIR}/third_party/connectedhomeip/examples/lighting-app/lighting-common/include"
"${CMAKE_CURRENT_LIST_DIR}/include"
"${CMAKE_SOURCE_DIR}/third_party/connectedhomeip/examples/platform/esp32"
)
)
set(SRC_DIRS_LIST
"${CMAKE_CURRENT_LIST_DIR}"
"${CMAKE_SOURCE_DIR}/third_party/connectedhomeip/zzz_generated/app-common/app-common/zap-generated/attributes"
Expand Down Expand Up @@ -90,6 +91,11 @@ endif (CONFIG_ENABLE_PW_RPC)
idf_component_register(PRIV_INCLUDE_DIRS ${PRIV_INCLUDE_DIRS_LIST}
SRC_DIRS ${SRC_DIRS_LIST}
PRIV_REQUIRES ${PRIV_REQUIRES_LIST})
get_filename_component(CHIP_ROOT ${CMAKE_SOURCE_DIR}/third_party/connectedhomeip REALPATH)

include("${CHIP_ROOT}/build/chip/esp32/esp32_codegen.cmake")

chip_app_component_codegen("${CHIP_ROOT}/examples/lighting-app/lighting-common/lighting-app.matter")

set_property(TARGET ${COMPONENT_LIB} PROPERTY CXX_STANDARD 17)
target_compile_options(${COMPONENT_LIB} PRIVATE "-DCHIP_HAVE_CONFIG_H")
Expand Down
13 changes: 11 additions & 2 deletions examples/lock-app/esp32/main/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@
#
# (Uses default behaviour of compiling all source files in directory, adding 'include' to include path.)

get_filename_component(CHIP_ROOT ${CMAKE_SOURCE_DIR}/third_party/connectedhomeip REALPATH)

if (CONFIG_ENABLE_PW_RPC)
idf_component_register(INCLUDE_DIRS
"${CMAKE_CURRENT_LIST_DIR}"
Expand Down Expand Up @@ -67,8 +69,7 @@ idf_component_register(INCLUDE_DIRS
"${CMAKE_SOURCE_DIR}/third_party/connectedhomeip/src/app/clusters/identify-server"
"${CMAKE_SOURCE_DIR}/third_party/connectedhomeip/src/app/clusters/groups-server"
PRIV_REQUIRES bt chip QRCode)

get_filename_component(CHIP_ROOT ${CMAKE_SOURCE_DIR}/third_party/connectedhomeip REALPATH)
add_dependencies(${COMPONENT_LIB} app-codegen)

set(PIGWEED_ROOT "${CHIP_ROOT}/third_party/pigweed/repo")
include(${PIGWEED_ROOT}/pw_build/pigweed.cmake)
Expand Down Expand Up @@ -186,10 +187,18 @@ idf_component_register(PRIV_INCLUDE_DIRS
"${CMAKE_SOURCE_DIR}/third_party/connectedhomeip/src/app/clusters/groups-server"
PRIV_REQUIRES chip QRCode bt)

add_dependencies(${COMPONENT_LIB} app-codegen)

set_property(TARGET ${COMPONENT_LIB} PROPERTY CXX_STANDARD 17)
target_compile_options(${COMPONENT_LIB} PRIVATE "-DCHIP_HAVE_CONFIG_H")
target_compile_options(${COMPONENT_LIB} PUBLIC
"-DCHIP_ADDRESS_RESOLVE_IMPL_INCLUDE_HEADER=<lib/address_resolve/AddressResolve_DefaultImpl.h>"
)

endif (CONFIG_ENABLE_PW_RPC)

get_filename_component(CHIP_ROOT ${CMAKE_SOURCE_DIR}/third_party/connectedhomeip REALPATH)

include("${CHIP_ROOT}/build/chip/esp32/esp32_codegen.cmake")

chip_app_component_codegen("${CHIP_ROOT}/examples/lock-app/lock-common/lock-app.matter")
6 changes: 5 additions & 1 deletion examples/ota-provider-app/esp32/main/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
#
#
# (Uses default behaviour of compiling all source files in directory, adding 'include' to include path.)
idf_component_register(PRIV_INCLUDE_DIRS
idf_component_register(PRIV_INCLUDE_DIRS
"${CMAKE_CURRENT_LIST_DIR}/include"
"${CMAKE_SOURCE_DIR}/third_party/connectedhomeip/zzz_generated/ota-provider-app/"
"${CMAKE_SOURCE_DIR}/third_party/connectedhomeip/examples/ota-provider-app"
Expand Down Expand Up @@ -59,6 +59,10 @@ idf_component_register(PRIV_INCLUDE_DIRS
"${CMAKE_SOURCE_DIR}/third_party/connectedhomeip/examples/ota-provider-app/ota-provider-common/BdxOtaSender.cpp"
PRIV_REQUIRES chip QRCode bt console spiffs)

get_filename_component(CHIP_ROOT ${CMAKE_SOURCE_DIR}/third_party/connectedhomeip REALPATH)
include("${CHIP_ROOT}/build/chip/esp32/esp32_codegen.cmake")
chip_app_component_codegen("${CHIP_ROOT}/examples/ota-provider-app/ota-provider-common/ota-provider-app.matter")

spiffs_create_partition_image(img_storage ${CMAKE_SOURCE_DIR}/spiffs_image FLASH_IN_PROJECT)
set_property(TARGET ${COMPONENT_LIB} PROPERTY CXX_STANDARD 14)
target_compile_options(${COMPONENT_LIB} PRIVATE "-DCHIP_HAVE_CONFIG_H")
Expand Down
Loading

0 comments on commit 1555656

Please sign in to comment.