"mbed compile" error with Python3 and mbed-cli 1.9.1

[bogdanm]

Trying to use mbed-cli 1.9.1 with Mbed OS 5.11.4 results in an error (AttributeError: 'str' object has no attribute 'decode') on any project that I tried to compile:

🤘 [bogdanm:/hdd/bogdanm/work] $ mkdir temp && cd temp
🤘 [bogdanm:/hdd/bogdanm/work/temp] $ mbed new .
[mbed] WARNING: If you're using Python 3 with Mbed OS 5.8 and earlier versions, Python errors will occur when compiling, testing and exporting
---
[mbed] Working path "/hdd/bogdanm/work/temp" (directory)
[mbed] Creating new program "temp" (git)
[mbed] Adding library "mbed-os" from "https://github.com/ARMmbed/mbed-os" at branch/tag "latest"
[mbed] Updating reference "mbed-os" -> "https://github.com/ARMmbed/mbed-os/#ecb3c8c837162c73537bd0f3592c6e2a42994045"
🤘 [bogdanm:/hdd/bogdanm/work/temp] $ cd mbed-os
🤘 [bogdanm:/hdd/bogdanm/work/temp/mbed-os] ecb3c8c 128 ± git show HEAD
commit ecb3c8c837162c73537bd0f3592c6e2a42994045
Merge: a8d1d26 33d779f
Author: Martin Kojtal <martin.kojtal@arm.com>
Date:   Mon Feb 11 11:56:11 2019 +0100

    Merge pull request #9646 from ARMmbed/release-candidate
    
    Release candidate for mbed-os-5.11.4

🤘 [bogdanm:/hdd/bogdanm/work/temp/mbed-os] ecb3c8c ± cd ..
🤘 [bogdanm:/hdd/bogdanm/work/temp] master* 1 ± mbed compile -t GCC_ARM -m K64F -v
[mbed] WARNING: If you're using Python 3 with Mbed OS 5.8 and earlier versions, Python errors will occur when compiling, testing and exporting
---
[mbed] Working path "/hdd/bogdanm/work/temp" (program)
[mbed] Exec "/usr/bin/python3 -u /hdd/bogdanm/work/temp/mbed-os/tools/make.py -t GCC_ARM -m K64F --source . --build ./BUILD/K64F/GCC_ARM -v" in "/hdd/bogdanm/work/temp"
Building project temp (K64F, GCC_ARM)
Scan: temp
Macros: -DDEVICE_SERIAL_ASYNCH=1 -DFEATURE_STORAGE=1 -D__MBED__=1 -D__MBED_CMSIS_RTOS_CM -DTARGET_M4 -DCOMPONENT_PSA_SRV_EMUL=1 -DTARGET_Freescale_EMAC -DDEVICE_RTC=1 -DDEVICE_STDIO_MESSAGES=1 -DDEVICE_USTICKER=1 -DTARGET_RTOS_M4_M7 -DTARGET_CORTEX_M -DDEVICE_ANALOGOUT=1 -DTARGET_FF_ARDUINO -DTOOLCHAIN_GCC_ARM -DTARGET_Freescale -DDEVICE_SPISLAVE=1 -DDEVICE_PORTIN=1 -DDEVICE_LPTICKER=1 -D__FPU_PRESENT=1 -D__CMSIS_RTOS -DMBEDTLS_PSA_CRYPTO_C -DTARGET_K64F -DTARGET_KSDK2_MCUS -DTARGET_LIKE_MBED -DTARGET_MCUXpresso_MCUS -D__CORTEX_M4 -DTARGET_FRDM -DCPU_MK64FN1M0VMD12 -DTARGET_KPSDK_MCUS -DDEVICE_EMAC=1 -DDEVICE_INTERRUPTIN=1 -DDEVICE_I2CSLAVE=1 -DTARGET_LIKE_CORTEX_M4 -DMBED_BUILD_TIMESTAMP=1550439801.5426726 -DDEVICE_I2C=1 -DCOMPONENT_NSPE=1 -DTARGET_KPSDK_CODE -DFSL_RTOS_MBED -DDEVICE_CRC=1 -DDEVICE_PORTOUT=1 -DDEVICE_PWMOUT=1 -DCOMPONENT_SD=1 -DTARGET_PSA -DTARGET_CORTEX -DDEVICE_SLEEP=1 -DDEVICE_PORTINOUT=1 -DDEVICE_FLASH=1 -DDEVICE_SPI=1 -DCOMPONENT_FLASHIAP=1 -DDEVICE_SERIAL=1 -DARM_MATH_CM4 -DDEVICE_SPI_ASYNCH=1 -DDEVICE_STORAGE=1 -DDEVICE_ANALOGIN=1 -DCOMPONENT_PSA_SRV_IMPL=1 -DDEVICE_SERIAL_FC=1 -DTOOLCHAIN_GCC -DDEVICE_TRNG=1 -DTARGET_RELEASE -DTARGET_MCU_K64F
Compile [  1.3%]: psa_prot_internal_storage.cpp
Compile: arm-none-eabi-g++ -std=gnu++98 -fno-rtti -Wvla -c -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -fmessage-length=0 -fno-exceptions -fno-builtin -ffunction-sections -fdata-sections -funsigned-char -MMD -fno-delete-null-pointer-checks -fomit-frame-pointer -Os -g1 -mcpu=cortex-m4 -mthumb -mfpu=fpv4-sp-d16 -mfloat-abi=softfp -DMBED_ROM_START=0x0 -DMBED_ROM_SIZE=0x100000 -DMBED_RAM_START=0x20000000 -DMBED_RAM_SIZE=0x30000 -DMBED_RAM1_START=0x1fff0000 -DMBED_RAM1_SIZE=0x10000 -DDEVICE_SERIAL_ASYNCH=1 -DFEATURE_STORAGE=1 -D__MBED__=1 -D__MBED_CMSIS_RTOS_CM -DTARGET_M4 -DCOMPONENT_PSA_SRV_EMUL=1 -DTARGET_Freescale_EMAC -DDEVICE_RTC=1 -DDEVICE_STDIO_MESSAGES=1 -DDEVICE_USTICKER=1 -DTARGET_RTOS_M4_M7 -DTARGET_CORTEX_M -DDEVICE_ANALOGOUT=1 -DTARGET_FF_ARDUINO -DTOOLCHAIN_GCC_ARM -DTARGET_Freescale -DDEVICE_SPISLAVE=1 -DDEVICE_PORTIN=1 -DDEVICE_LPTICKER=1 -D__FPU_PRESENT=1 -D__CMSIS_RTOS -DMBEDTLS_PSA_CRYPTO_C -DTARGET_K64F -DTARGET_KSDK2_MCUS -DTARGET_LIKE_MBED -DTARGET_MCUXpresso_MCUS -D__CORTEX_M4 -DTARGET_FRDM -DCPU_MK64FN1M0VMD12 -DTARGET_KPSDK_MCUS -DDEVICE_EMAC=1 -DDEVICE_INTERRUPTIN=1 -DDEVICE_I2CSLAVE=1 -DTARGET_LIKE_CORTEX_M4 -DMBED_BUILD_TIMESTAMP=1550439801.5426726 -DDEVICE_I2C=1 -DCOMPONENT_NSPE=1 -DTARGET_KPSDK_CODE -DFSL_RTOS_MBED -DDEVICE_CRC=1 -DDEVICE_PORTOUT=1 -DDEVICE_PWMOUT=1 -DCOMPONENT_SD=1 -DTARGET_PSA -DTARGET_CORTEX -DDEVICE_SLEEP=1 -DDEVICE_PORTINOUT=1 -DDEVICE_FLASH=1 -DDEVICE_SPI=1 -DCOMPONENT_FLASHIAP=1 -DDEVICE_SERIAL=1 -DARM_MATH_CM4 -DDEVICE_SPI_ASYNCH=1 -DDEVICE_STORAGE=1 -DDEVICE_ANALOGIN=1 -DCOMPONENT_PSA_SRV_IMPL=1 -DDEVICE_SERIAL_FC=1 -DTOOLCHAIN_GCC -DDEVICE_TRNG=1 -DTARGET_RELEASE -DTARGET_MCU_K64F @./BUILD/K64F/GCC_ARM/.includes_0d57965f5bbb0584c468c8db7ac27e8d.txt -include ./BUILD/K64F/GCC_ARM/mbed_config.h -MD -MF BUILD/K64F/GCC_ARM/mbed-os/components/TARGET_PSA/services/psa_prot_internal_storage/COMPONENT_PSA_SRV_EMUL/psa_prot_internal_storage.d -o BUILD/K64F/GCC_ARM/mbed-os/components/TARGET_PSA/services/psa_prot_internal_storage/COMPONENT_PSA_SRV_EMUL/psa_prot_internal_storage.o ./mbed-os/components/TARGET_PSA/services/psa_prot_internal_storage/COMPONENT_PSA_SRV_EMUL/psa_prot_internal_storage.cpp
Traceback (most recent call last):
  File "/hdd/bogdanm/work/temp/mbed-os/tools/make.py", line 73, in wrapped_build_project
    src_dir, build_dir, mcu, *args, **kwargs
  File "/hdd/bogdanm/work/temp/mbed-os/tools/build_api.py", line 554, in build_project
    objects = toolchain.compile_sources(resources, sorted(resources.get_file_paths(FileType.INC_DIR)))
  File "/hdd/bogdanm/work/temp/mbed-os/tools/toolchains/__init__.py", line 416, in compile_sources
    return self.compile_queue(queue, objects)
  File "/hdd/bogdanm/work/temp/mbed-os/tools/toolchains/__init__.py", line 470, in compile_queue
    res['command']
  File "/hdd/bogdanm/work/temp/mbed-os/tools/toolchains/__init__.py", line 574, in compile_output
    _stderr = output[1].decode("utf-8")
AttributeError: 'str' object has no attribute 'decode'
[mbed] ERROR: "/usr/bin/python3" returned error.
       Code: 1
       Path: "/hdd/bogdanm/work/temp"
       Command: "/usr/bin/python3 -u /hdd/bogdanm/work/temp/mbed-os/tools/make.py -t GCC_ARM -m K64F --source . --build ./BUILD/K64F/GCC_ARM -v"
       Tip: You could retry the last command with "-v" flag for verbose output
---
[mbed] WARNING: Using Python 3 with Mbed OS 5.8 and earlier can cause errors with compiling, testing and exporting
---
🤘 [bogdanm:/hdd/bogdanm/work] $ mbed --version
1.9.1
🤘 [bogdanm:/hdd/bogdanm/work] $ python3 --version
Python 3.5.2
🤘 [bogdanm:/hdd/bogdanm/work] $ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.5 LTS
Release:	16.04
Codename:	xenial

The same thing happens if I create a Python3 venv for mbed-cli.

[ciarmcom]

Internal Jira reference: https://jira.arm.com/browse/MBOCUSTRIA-887

[yogpan01]

I am also having the same issue.

[bogdanm]

This might be an issue with Mbed OS, not mbed-cli. In any case, it seems to work fine with Python 2.

[naveenkaje]

Hi @yogpan01 , @bogdanm
Thanks for your input. We will investigate this issue.

I tried the same steps with Mbed CLI version 1.8.3

$ mbed --version
1.8.3

It fails to find main.cpp and fails. This is expected behavior. As reported, there may be a in issue with newer mbed cli or tools.

<snip>
MCUS/fsl_common.c
[DEBUG] Return: 0
Link: custria-887
Preproc: /usr/local/gcc-arm-none-eabi-8-2018-q4-major/bin/arm-none-eabi-cpp -E -P ./mbed-os/targets/TARGET_Freescale/TARGET_MCUXpresso_MCUS/TARGET_MCU_K64F/device/TOOLCHAIN_GCC_ARM/MK64FN1M0xxx12.ld -Wl,--gc-sections -Wl,--wrap,main -Wl,--wrap,_malloc_r -Wl,--wrap,_free_r -Wl,--wrap,_realloc_r -Wl,--wrap,_memalign_r -Wl,--wrap,_calloc_r -Wl,--wrap,exit -Wl,--wrap,atexit -Wl,-n -mcpu=cortex-m4 -mthumb -mfpu=fpv4-sp-d16 -mfloat-abi=softfp -DMBED_ROM_START=0x0 -DMBED_ROM_SIZE=0x100000 -DMBED_RAM_START=0x20000000 -DMBED_RAM_SIZE=0x30000 -DMBED_RAM1_START=0x1fff0000 -DMBED_RAM1_SIZE=0x10000 -DMBED_BOOT_STACK_SIZE=1024 -o ./BUILD/K64F/GCC_ARM/.link_script.ld
[DEBUG] Return: 0
Link: /usr/local/gcc-arm-none-eabi-8-2018-q4-major/bin/arm-none-eabi-gcc @./BUILD/K64F/GCC_ARM/.link_options.txt
[DEBUG] Return: 1
[DEBUG] Errors: /usr/local/gcc-arm-none-eabi-8-2018-q4-major/bin/../lib/gcc/arm-none-eabi/8.2.1/../../../../arm-none-eabi/bin/ld: BUILD/K64F/GCC_ARM/mbed-os/rtos/TARGET_CORTEX/TOOLCHAIN_GCC_ARM/mbed_boot_gcc_arm.o: in function `__wrap_main':
[DEBUG] Errors: /home/navkaj01/code/tmp/custria-887/./mbed-os/rtos/TARGET_CORTEX/TOOLCHAIN_GCC_ARM/mbed_boot_gcc_arm.c:93: undefined reference to `main'
[DEBUG] Errors: collect2: error: ld returned 1 exit status
Traceback (most recent call last):
  File "/home/navkaj01/code/tmp/custria-887/mbed-os/tools/make.py", line 73, in wrapped_build_project
    src_dir, build_dir, mcu, *args, **kwargs
  File "/home/navkaj01/code/tmp/custria-887/mbed-os/tools/build_api.py", line 576, in build_project
    res, _ = toolchain.link_program(resources, build_path, name)
  File "/home/navkaj01/code/tmp/custria-887/mbed-os/tools/toolchains/__init__.py", line 647, in link_program
    self.link(elf, objects, libraries, lib_dirs, linker_script)
  File "/home/navkaj01/code/tmp/custria-887/mbed-os/tools/hooks.py", line 34, in wrapper
    res = function(t_self, *args, **kwargs)
  File "/home/navkaj01/code/tmp/custria-887/mbed-os/tools/toolchains/gcc.py", line 270, in link
    self.default_cmd(cmd)
  File "/home/navkaj01/code/tmp/custria-887/mbed-os/tools/toolchains/__init__.py", line 676, in default_cmd
    raise ToolException(_stderr)
ToolException: /usr/local/gcc-arm-none-eabi-8-2018-q4-major/bin/../lib/gcc/arm-none-eabi/8.2.1/../../../../arm-none-eabi/bin/ld: BUILD/K64F/GCC_ARM/mbed-os/rtos/TARGET_CORTEX/TOOLCHAIN_GCC_ARM/mbed_boot_gcc_arm.o: in function `__wrap_main':
/home/navkaj01/code/tmp/custria-887/./mbed-os/rtos/TARGET_CORTEX/TOOLCHAIN_GCC_ARM/mbed_boot_gcc_arm.c:93: undefined reference to `main'
collect2: error: ld returned 1 exit status

[mbed] Working path "/home/navkaj01/code/tmp/custria-887" (program)
[mbed] Exec "/usr/bin/python -u /home/navkaj01/code/tmp/custria-887/mbed-os/tools/make.py -t GCC_ARM -m K64F --source . --build ./BUILD/K64F/GCC_ARM -v" in "/home/navkaj01/code/tmp/custria-887"

[bridadan]

There was a regression in Mbed OS 5.11.4 for compiling with Python 3. There is a fix for this that will be released with 5.11.5: ARMmbed/mbed-os#9673

I'm closing this since the fix is related to the scripts within the mbed-os repository.

So far the only workarounds I'm aware of is to use Python 2 to build 5.11.4 or pick a previous release. Sorry for the inconvenience, I believe a Python 3 build check is being added to the Mbed OS CI system to prevent this failure from happening again.