UnicodeEncodeError: 'charmap' codec can't encode character '\u03a9' in position 3361: character maps to <undefined>

[laptoperist]

Here is a code:

import get_bulk_patent_data functionality

from patentpy.acquire import get_bulk_patent_data

download patents from the first week of 1976 and get data frame

patent_data = get_bulk_patent_data(year = 2021, week = 1, output_file = "patent-data-1.csv")

Which results:

Traceback (most recent call last):
File "C:\Users\XXXXXXXXXX\patentpy\patentpy\convert_xml2.py", line 229, in convert_xml2_to_df
xml2_to_df(curr_file, output_file if output_file is not None else temp_output_file, append, header)
UnicodeEncodeError: 'charmap' codec can't encode character '\u03a9' in position 3361: character maps to

[JYProjs]

@laptoperist Thanks for pointing this out! I can't replicate this on my local linux or mac. What OS, python and patentpy version are you using?

[chris-swenson]

I'm having the same problem on Windows 10 with Python 3.8 and the latest patentpy version. I believe the issue has to do with the output and temporary file encoding. Windows 10 may be defaulting to something that isn't compatible. Using the encoding='utf-8' may solve this issue, but I'm not sure. I may try to patch this to see if I can fix it. It may be necessary to exposing a file encoding option to prevent this issue.

[chris-swenson]

Hello. I tested modifying the code - I'm not exactly familiar with the process of forking and syncing with github and all that. But this change appeared to fix the issue for Windows:

def xml2_to_df(input_file, output_file, append, header):
write_mode = 'a' if append else 'w'
with open(input_file, 'rb') as f1, open (output_file, write_mode, encoding='utf-16') as f2:

I'm not sure how you'd patch this just for Windows.

Edit: How on earth do you get the formatting right for Python... anyway. Sorry about that.

[JYProjs]

@chris-swenson Thank you for pointing this out and providing a working solution. I will see what I can do to update the pkg once I can switch from Travis CI (ran out of build time and requesting more for open source is tedious). Thanks!

[JYProjs]

Leaving this open so anyone else who runs into this issue can see.

[JYProjs]

A better solution to this issue is to enable utf-8 mode when running python on Windows, i.e. python -X utf8. Everything should then work as expected. This error likely arises from Windows using ANSI and python modules using defaulting to it.

Note: This requires python 3.7 or newer
See link for more information: https://docs.python.org/3.7/using/windows.html

[chris-swenson]

I'd have to check, but in systems like Databricks, I don't think I can specify the python launch options. I think a more robust option is to allow specification of the encoding.

[JYProjs]

Okay, I will just edit the code to specify the encoding as utf-8, since it seems like the error is from not being able to encode a specific unicode char as Windows default encoding, mbcs/ansi, and since original uspto files are utf-8

[NJStudent1]

Hi,

I am getting a similar error while running it on Python3.10.4 on windows. I also tried "python -X utf8" in the cmd prompt. Could you please tell how to rectify the issue.

Thanks.