UnicodeEncodeError: 'charmap' codec can't encode characters in position

[fluppie]

Anyone an idea why I get this on Home 2.3.3 and Core 4.0.3 I switched to Visual Studio Code (was using Atom the last 2 years but the performance is horrible, claiming the CPU all the time).

UnicodeEncodeError: 'charmap' codec can't encode characters in position 1653755-1653756: character maps to <undefined>:
  File "C:\Users\Philippe\.platformio\penv\lib\site-packages\platformio\builder\main.py", line 126:
    env.SConscript("$BUILD_SCRIPT")
  File "C:\Users\Philippe\.platformio\packages\tool-scons\script\..\engine\SCons\Script\SConscript.py", line 605:
    return _SConscript(self.fs, *files, **subst_kw)
  File "C:\Users\Philippe\.platformio\packages\tool-scons\script\..\engine\SCons\Script\SConscript.py", line 286:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "C:\Users\Philippe\.platformio\platforms\espressif8266@2.2.2\builder\main.py", line 212:
    target_elf = env.BuildProgram()
  File "C:\Users\Philippe\.platformio\packages\tool-scons\script\..\engine\SCons\Environment.py", line 224:
    return self.method(*nargs, **kwargs)
  File "C:\Users\Philippe\.platformio\penv\lib\site-packages\platformio\builder\tools\platformio.py", line 110:
    env.BuildFrameworks(env.get("PIOFRAMEWORK"))
  File "C:\Users\Philippe\.platformio\packages\tool-scons\script\..\engine\SCons\Environment.py", line 224:
    return self.method(*nargs, **kwargs)
  File "C:\Users\Philippe\.platformio\penv\lib\site-packages\platformio\builder\tools\platformio.py", line 288:
    env.ConvertInoToCpp()
  File "C:\Users\Philippe\.platformio\packages\tool-scons\script\..\engine\SCons\Environment.py", line 224:
    return self.method(*nargs, **kwargs)
  File "C:\Users\Philippe\.platformio\penv\lib\site-packages\platformio\builder\tools\piomisc.py", line 198:
    out_file = c.convert(ino_nodes)
  File "C:\Users\Philippe\.platformio\penv\lib\site-packages\platformio\builder\tools\piomisc.py", line 57:
    return self.process(contents)
  File "C:\Users\Philippe\.platformio\penv\lib\site-packages\platformio\builder\tools\piomisc.py", line 80:
    assert self._gcc_preprocess(contents, out_file)
  File "C:\Users\Philippe\.platformio\penv\lib\site-packages\platformio\builder\tools\piomisc.py", line 90:
    fp.write(contents)
  File "C:\Users\Philippe\AppData\Local\Programs\Python\Python37\lib\encodings\cp1252.py", line 19:
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]

We are also discussing it here: letscontrolit/ESPEasy#2593

[emieldejong]

This line in _P052_SenseAir.ino was the problem for me:
#define P052_EEPROM_ADDR_LOGGER_STRUCTURE_ADDRESS 0x200 // 16b Described in “BLG_ELG Logger Structure”
Removing/changing the quotes in the comment resolved the issue.

[fmuntean]

having the same issue.
We can't just start changing all comments just to get it compile.
We need the Platform IO to correctly handle the special characters in comments.
This was working fine before.

[rominator1983]

I have the same issue trying to build tasmota in VS code. I have the latest versions (Code 1.39.2, PlatformIO 2.3.3 PlatformIO-Core 4.0.3) just as @fluppie.
@fluppie: Did you manage to get a workaround for this? I seem to be stuck here :-(

edit:
I forgot to mention that the code was compiling before VS code decided to upgrade my extensions (and somehow platformio core) on its own.
I finally managed to work-around this issue. The specific error I got was: UnicodeEncodeError: 'charmap' codec can't encode character '\x81' in position 229617: character maps to

So I did me a PowerShell script (my weapon of choice) to find a character 0x81 (decimal 129) in ino files:
gci -Recurse . *.ino | Where {-not $_.PsIsContainer} |% {$content =Get-Content $_.FullName -Encoding Byte; $i =0; foreach ($byte in $content) { if ($byte -eq 129) {Write-Host ("aaaah @ $($_.FullName),$i")} $i=$i+1 } }
This told me: aaaah @ C:\dev\tasmota\sonoff\support.ino,3837.
There I found a comment:
// Locate the first occurrence in the string pointed to by s1 of any character from the string pointed to by s2
Only the "fi" of "first" being a ligature (a combination of two chracters into one) of f and i. See https://en.wikipedia.org/wiki/Orthographic_ligature

By replacing that character with normal f and i I got the code compiling again.

edit: When trying to do a pull request for tasmota I saw that someone had obviously already fixed that issue on the develop-branch.

[fluppie]

I tried again (no changes, only pulling the ESP Easy repo from GIT)

`Processing normal_ESP8266_4M1M (platform: espressif8266@2.2.3; board: esp12e; framework: arduino)

Verbose mode can be enabled via -v, --verbose option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif8266/esp12e.html
PLATFORM: Espressif 8266 2.2.3 > Espressif ESP8266 ESP-12E
HARDWARE: ESP8266 80MHz, 80KB RAM, 4MB Flash
PACKAGES: toolchain-xtensa 2.40802.190218 (4.8.2), framework-arduinoespressif8266 2.20502.0 (2.5.2), tool-esptool 1.413.0 (4.13), tool-esptoolpy 1.20600.0 (2.6.0)
UnicodeEncodeError: 'charmap' codec can't encode characters in position 1666132-1666133: character maps to :
File "C:\Users\Philippe.platformio\penv\lib\site-packages\platformio\builder\main.py", line 126:
env.SConscript("$BUILD_SCRIPT")
File "C:\Users\Philippe.platformio\packages\tool-scons\script..\engine\SCons\Script\SConscript.py", line 605:
return _SConscript(self.fs, *files, **subst_kw)
File "C:\Users\Philippe.platformio\packages\tool-scons\script..\engine\SCons\Script\SConscript.py", line 286:
exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
File "C:\Users\Philippe.platformio\platforms\espressif8266\builder\main.py", line 203:
target_elf = env.BuildProgram()
File "C:\Users\Philippe.platformio\packages\tool-scons\script..\engine\SCons\Environment.py", line 224:
return self.method(*nargs, **kwargs)
File "C:\Users\Philippe.platformio\penv\lib\site-packages\platformio\builder\tools\platformio.py", line 110:
env.BuildFrameworks(env.get("PIOFRAMEWORK"))
File "C:\Users\Philippe.platformio\packages\tool-scons\script..\engine\SCons\Environment.py", line 224:
return self.method(*nargs, **kwargs)
File "C:\Users\Philippe.platformio\penv\lib\site-packages\platformio\builder\tools\platformio.py", line 288:
env.ConvertInoToCpp()
File "C:\Users\Philippe.platformio\packages\tool-scons\script..\engine\SCons\Environment.py", line 224:
return self.method(*nargs, **kwargs)
File "C:\Users\Philippe.platformio\penv\lib\site-packages\platformio\builder\tools\piomisc.py", line 198:
out_file = c.convert(ino_nodes)
File "C:\Users\Philippe.platformio\penv\lib\site-packages\platformio\builder\tools\piomisc.py", line 57:
return self.process(contents)
File "C:\Users\Philippe.platformio\penv\lib\site-packages\platformio\builder\tools\piomisc.py", line 80:
assert self._gcc_preprocess(contents, out_file)
File "C:\Users\Philippe.platformio\penv\lib\site-packages\platformio\builder\tools\piomisc.py", line 90:
fp.write(contents)
File "C:\Users\Philippe\AppData\Local\Programs\Python\Python37\lib\encodings\cp1252.py", line 19:
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
=============================================================================== [FAILED] Took 0.85 seconds ===============================================================================`

[rominator1983]

@fluppie: I changed the faulting lines in cp1252.py to:
class IncrementalEncoder(codecs.IncrementalEncoder): def encode(self, input, final=False): try: return codecs.charmap_encode(input,self.errors,encoding_table)[0] except Exception as e: print ("aaaaahhh!") print(e) raise

And added the file that is processed (out_file) to the output in piomics.py:
def process(self, contents): out_file = self._main_ino + ".cpp" print(out_file) assert self._gcc_preprocess(contents, out_file) contents = get_file_contents(out_file) contents = self._join_multiline_strings(contents) with open(out_file, "w") as fp: fp.write(self.append_prototypes(contents)) return out_file

The comment window distorted my line breaks. But you should be able to see what I changed.

maybe that gives you a clue on how to proceedee? You are getting a different message without a hint to a character that is causing that problem so my PowerShell-script won't be of any help to you I guess.