mbed compile doesn't work with Python 3.7

[vmedcy]

Description

mbed compile doesn't with Python3.7 since 149d280

Errors are likely caused by #9521:

File "C:\mbed\mbed-os-example-wifi\mbed-os\tools\toolchains_init_.py", line 576, in compile_output
_stderr = output[1].decode("utf-8")
AttributeError: 'str' object has no attribute 'decode'

File "C:\mbed\mbed-os-example-wifi\mbed-os\tools\toolchains\gcc.py", line 136, in version_check
match = self.GCC_VERSION_RE.search(stdout)
TypeError: cannot use a bytes pattern on a string-like object

Steps to reproduce:

• Install Python 3.7 on Windows 10, open MINGW64 shell (Git for Windows)

python --version
Python 3.7.1
which pip
/c/Users/vmed/AppData/Local/Programs/Python/Python37/Scripts/pip

• Install mbed module

pip install mbed-cli
mbed --version
1.8.3
which mbed
/c/Users/vmed/AppData/Local/Programs/Python/Python37/Scripts/mbed

• Import code example, checkout 149d280

mbed import mbed-os-example-wifi
cd mbed-os-example-wifi/mbed-os
git fetch
git checkout 149d280e7a0bb8d50f2def942892c80ad504ab2e
cd ..

• Try to compile

mbed compile -t GCC_ARM -m K64F -v
[mbed] WARNING: If you're using Python 3 with Mbed OS 5.8 and earlier versions, Python errors will occur when compiling, testing and exporting
---
[mbed] Working path "C:\mbed\mbed-os-example-wifi\mbed-os" (library)
[mbed] Program path "C:\mbed\mbed-os-example-wifi"
[mbed] Exec "c:\users\vmed\appdata\local\programs\python\python37\python.exe -u C:\mbed\mbed-os-example-wifi\mbed-os\tools\make.py -t GCC_ARM -m K64F --source .. --build ..\BUILD\K64F\GCC_ARM -v" in "C:\mbed\mbed-os-example-wifi\mbed-os"
Building project mbed-os-example-wifi (K64F, GCC_ARM)
Scan: mbed-os-example-wifi
Macros: -DTARGET_K64F -DTARGET_KPSDK_MCUS -DTARGET_RTOS_M4_M7 -DTARGET_MCU_K64F -D__CORTEX_M4 -DTARGET_MCUXpresso_MCUS -D__MBED_CMSIS_RTOS_CM -DDEVICE_SPI_ASYNCH=1 -DDEVICE_SPISLAVE=1 -DDEVICE_USTICKER=1 -DDEVICE_SERIAL_ASYNCH=1 -DDEVICE_PORTOUT=1 -DDEVICE_ANALOGIN=1 -DCOMPONENT_PSA_SRV_EMUL=1 -DDEVICE_STORAGE=1 -DDEVICE_CRC=1 -DCOMPONENT_PSA_SRV_IMPL=1 -DDEVICE_LPTICKER=1 -DFSL_RTOS_MBED -DDEVICE_FLASH=1 -DTOOLCHAIN_GCC_ARM -DDEVICE_PORTIN=1 -DDEVICE_I2CSLAVE=1 -DCOMPONENT_SD=1 -DCOMPONENT_NSPE=1 -DTARGET_FRDM -DDEVICE_EMAC=1 -D__FPU_PRESENT=1 -DTARGET_KPSDK_CODE -DMBEDTLS_PSA_CRYPTO_C -DDEVICE_PWMOUT=1 -DTARGET_KSDK2_MCUS -DTARGET_RELEASE -DTARGET_M4 -DDEVICE_STDIO_MESSAGES=1 -DMBED_BUILD_TIMESTAMP=1549377243.4695053 -DTARGET_CORTEX_M -DCOMPONENT_FLASHIAP=1 -DARM_MATH_CM4 -DTARGET_LIKE_MBED -DDEVICE_ANALOGOUT=1 -DDEVICE_SERIAL=1 -DDEVICE_SERIAL_FC=1 -D__MBED__=1 -DTOOLCHAIN_GCC -DDEVICE_INTERRUPTIN=1 -DTARGET_LIKE_CORTEX_M4 -DTARGET_CORTEX -DDEVICE_TRNG=1 -DTARGET_Freescale -DCPU_MK64FN1M0VMD12 -DDEVICE_RTC=1 -DTARGET_Freescale_EMAC -DTARGET_FF_ARDUINO -DDEVICE_SLEEP=1 -D__CMSIS_RTOS -DTARGET_NAME=K64F -DDEVICE_PORTINOUT=1 -DDEVICE_I2C=1 -DDEVICE_SPI=1 -DTARGET_PSA -DFEATURE_STORAGE=1
Compile [  0.8%]: MCR20Drv.c
Compile: C:\Program Files (x86)\GNU Tools ARM Embedded\6 2017-q2-update\bin\arm-none-eabi-gcc -std=gnu99 -c -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -fmessage-length=0 -fno-exceptions -fno-builtin -ffunction-sections -fdata-sections -funsigned-char -MMD -fno-delete-null-pointer-checks -fomit-frame-pointer -Os -g1 -mcpu=cortex-m4 -mthumb -mfpu=fpv4-sp-d16 -mfloat-abi=softfp -DMBED_ROM_START=0x0 -DMBED_ROM_SIZE=0x100000 -DMBED_RAM_START=0x20000000 -DMBED_RAM_SIZE=0x30000 -DMBED_RAM1_START=0x1fff0000 -DMBED_RAM1_SIZE=0x10000 -DTARGET_K64F -DTARGET_KPSDK_MCUS -DTARGET_RTOS_M4_M7 -DTARGET_MCU_K64F -D__CORTEX_M4 -DTARGET_MCUXpresso_MCUS -D__MBED_CMSIS_RTOS_CM -DDEVICE_SPI_ASYNCH=1 -DDEVICE_SPISLAVE=1 -DDEVICE_USTICKER=1 -DDEVICE_SERIAL_ASYNCH=1 -DDEVICE_PORTOUT=1 -DDEVICE_ANALOGIN=1 -DCOMPONENT_PSA_SRV_EMUL=1 -DDEVICE_STORAGE=1 -DDEVICE_CRC=1 -DCOMPONENT_PSA_SRV_IMPL=1 -DDEVICE_LPTICKER=1 -DFSL_RTOS_MBED -DDEVICE_FLASH=1 -DTOOLCHAIN_GCC_ARM -DDEVICE_PORTIN=1 -DDEVICE_I2CSLAVE=1 -DCOMPONENT_SD=1 -DCOMPONENT_NSPE=1 -DTARGET_FRDM -DDEVICE_EMAC=1 -D__FPU_PRESENT=1 -DTARGET_KPSDK_CODE -DMBEDTLS_PSA_CRYPTO_C -DDEVICE_PWMOUT=1 -DTARGET_KSDK2_MCUS -DTARGET_RELEASE -DTARGET_M4 -DDEVICE_STDIO_MESSAGES=1 -DMBED_BUILD_TIMESTAMP=1549377243.4695053 -DTARGET_CORTEX_M -DCOMPONENT_FLASHIAP=1 -DARM_MATH_CM4 -DTARGET_LIKE_MBED -DDEVICE_ANALOGOUT=1 -DDEVICE_SERIAL=1 -DDEVICE_SERIAL_FC=1 -D__MBED__=1 -DTOOLCHAIN_GCC -DDEVICE_INTERRUPTIN=1 -DTARGET_LIKE_CORTEX_M4 -DTARGET_CORTEX -DDEVICE_TRNG=1 -DTARGET_Freescale -DCPU_MK64FN1M0VMD12 -DDEVICE_RTC=1 -DTARGET_Freescale_EMAC -DTARGET_FF_ARDUINO -DDEVICE_SLEEP=1 -D__CMSIS_RTOS -DTARGET_NAME=K64F -DDEVICE_PORTINOUT=1 -DDEVICE_I2C=1 -DDEVICE_SPI=1 -DTARGET_PSA -DFEATURE_STORAGE=1 @..\BUILD\K64F\GCC_ARM\.includes_ca98ae87ad15d9ebab9059fb9a91e501.txt -include ..\BUILD\K64F\GCC_ARM\mbed_config.h -MD -MF ..\BUILD\K64F\GCC_ARM\mbed-os\components\802.15.4_RF\mcr20a-rf-driver\source\MCR20Drv.d -o ..\BUILD\K64F\GCC_ARM\mbed-os\components\802.15.4_RF\mcr20a-rf-driver\source\MCR20Drv.o ..\mbed-os\components\802.15.4_RF\mcr20a-rf-driver\source\MCR20Drv.c
Traceback (most recent call last):
  File "C:\mbed\mbed-os-example-wifi\mbed-os\tools\make.py", line 73, in wrapped_build_project
    src_dir, build_dir, mcu, *args, **kwargs
  File "C:\mbed\mbed-os-example-wifi\mbed-os\tools\build_api.py", line 547, in build_project
    objects = toolchain.compile_sources(resources, sorted(resources.get_file_paths(FileType.INC_DIR)))
  File "C:\mbed\mbed-os-example-wifi\mbed-os\tools\toolchains\__init__.py", line 418, in compile_sources
    return self.compile_queue(queue, objects)
  File "C:\mbed\mbed-os-example-wifi\mbed-os\tools\toolchains\__init__.py", line 472, in compile_queue
    res['command']
  File "C:\mbed\mbed-os-example-wifi\mbed-os\tools\toolchains\__init__.py", line 576, in compile_output
    _stderr = output[1].decode("utf-8")
AttributeError: 'str' object has no attribute 'decode'
[mbed] ERROR: "c:\users\vmed\appdata\local\programs\python\python37\python.exe" returned error.
       Code: 1
       Path: "C:\mbed\mbed-os-example-wifi\mbed-os"
       Command: "c:\users\vmed\appdata\local\programs\python\python37\python.exe -u C:\mbed\mbed-os-example-wifi\mbed-os\tools\make.py -t GCC_ARM -m K64F --source .. --build ..\BUILD\K64F\GCC_ARM -v"
       Tip: You could retry the last command with "-v" flag for verbose output
---
[mbed] WARNING: Using Python 3 with Mbed OS 5.8 and earlier can cause errors with compiling, testing and exporting
---

• Revert to previous revision

cd mbed-os
git checkout HEAD~
cd ..

• Try to compile, error is different:

mbed compile -t GCC_ARM -m K64F -v
[mbed] WARNING: If you're using Python 3 with Mbed OS 5.8 and earlier versions, Python errors will occur when compiling, testing and exporting
---
[mbed] Working path "C:\mbed\mbed-os-example-wifi" (program)
[mbed] Exec "c:\users\vmed\appdata\local\programs\python\python37\python.exe -u C:\mbed\mbed-os-example-wifi\mbed-os\tools\make.py -t GCC_ARM -m K64F --source . --build .\BUILD\K64F\GCC_ARM -v" in "C:\mbed\mbed-os-example-wifi"
Traceback (most recent call last):
  File "C:\mbed\mbed-os-example-wifi\mbed-os\tools\make.py", line 73, in wrapped_build_project
    src_dir, build_dir, mcu, *args, **kwargs
  File "C:\mbed\mbed-os-example-wifi\mbed-os\tools\build_api.py", line 513, in build_project
    toolchain.version_check()
  File "C:\mbed\mbed-os-example-wifi\mbed-os\tools\toolchains\gcc.py", line 136, in version_check
    match = self.GCC_VERSION_RE.search(stdout)
TypeError: cannot use a bytes pattern on a string-like object
[mbed] ERROR: "c:\users\vmed\appdata\local\programs\python\python37\python.exe" returned error.
       Code: 1
       Path: "C:\mbed\mbed-os-example-wifi"
       Command: "c:\users\vmed\appdata\local\programs\python\python37\python.exe -u C:\mbed\mbed-os-example-wifi\mbed-os\tools\make.py -t GCC_ARM -m K64F --source . --build .\BUILD\K64F\GCC_ARM -v"
       Tip: You could retry the last command with "-v" flag for verbose output
---
[mbed] WARNING: Using Python 3 with Mbed OS 5.8 and earlier can cause errors with compiling, testing and exporting
---

• Revert to the latest release

cd mbed-os
git checkout mbed-os-5.11.3
cd ..
mbed compile -t GCC_ARM -m K64F -v
...
(compilation continues without python errors)

Issue request type

[ ] Question
[ ] Enhancement
[X] Bug

[ciarmcom]

Internal Jira reference: https://jira.arm.com/browse/MBOCUSTRIA-835

[cmonr]

@vmedcy Could you do a git bisect to confirm that one of those two PR caused the issue?
Also, what version of GCC are you using?

You might also want to check that 5.11.4 fails. The RCp R was just merged and passed (https://travis-ci.org/ARMmbed/mbed-os/jobs/490642405), which is strange since this would suggest that the unit-test update was not complete.

[vmedcy]

I can reproduce this with 5.11.4 release with vanilla Python3.7, mbed 1.8.3 and GCC 6 2017-q2-update.
I executed the following in Windows Command Prompt:

> set PATH=C:\Users\vmed\AppData\Local\Programs\Python\Python37;C:\Users\vmed\AppData\Local\Programs\Python\Python37\Scripts;%PATH%
> where mbed
C:\Users\vmed\AppData\Local\Programs\Python\Python37\Scripts\mbed.exe
C:\Python27\Scripts\mbed.exe
> mbed import mbed-os-example-blinky
> cd mbed-os-example-blinky
> mbed config -G -L
[mbed] WARNING: If you're using Python 3 with Mbed OS 5.8 and earlier versions, Python errors will occur when compiling, testing and exporting
---
[mbed] Working path "C:\Users\vmed\mbed-os-example-blinky" (program)
[mbed] Global config:
ARMC6_PATH=C:\Keil_v5\ARM\ARMCLANG\bin
ARM_PATH=C:\Keil_v5\ARM\ARMCC
GCC_ARM_PATH=C:\Program Files (x86)\GNU Tools ARM Embedded\6 2017-q2-update\bin
CACHE=enabled
IAR_PATH=C:\Program Files (x86)\IAR Systems\Embedded Workbench 7.5\arm

[mbed] Local config (C:\Users\vmed\mbed-os-example-blinky):
No local configuration is set

> mbed compile -t GCC_ARM -m K64F
[mbed] WARNING: If you're using Python 3 with Mbed OS 5.8 and earlier versions, Python errors will occur when compiling, testing and exporting
---
[mbed] Working path "C:\Users\vmed\mbed-os-example-blinky" (program)
Building project mbed-os-example-blinky (K64F, GCC_ARM)
Scan: mbed-os-example-blinky
Compile [  0.1%]: except.S
[ERROR] 'str' object has no attribute 'decode'
[mbed] ERROR: "c:\users\vmed\appdata\local\programs\python\python37\python.exe" returned error.
       Code: 1
       Path: "C:\Users\vmed\mbed-os-example-blinky"
       Command: "c:\users\vmed\appdata\local\programs\python\python37\python.exe -u C:\Users\vmed\mbed-os-example-blinky\mbed-os\tools\make.py -t GCC_ARM -m K64F --source . --build .\BUILD\K64F\GCC_ARM"
       Tip: You could retry the last command with "-v" flag for verbose output
---
[mbed] WARNING: Using Python 3 with Mbed OS 5.8 and earlier can cause errors with compiling, testing and exporting

After I edit line 574 in mbed-os\tools\toolchains_init_.py from:
_stderr = output[1].decode("utf-8")
to:
_stderr = output[1]

the python error goes away.
Moreover, I am able to compile with C:\Python27\Scripts\mbed.exe after the fix.

"'str' object has no attribute 'decode'" seems to be a valid problem in python code: https://stackoverflow.com/a/28583969

[vmedcy]

@cmonr: I verified that #9673 resolves the issue