Cedict reader #138
Cedict reader #138
Conversation
conceptnet5/readers/cc_cedict.py
Outdated
| DATE_RANGE_REGEX = re.compile(r'(.+?)\s\(.+\d.+\),') # date range | ||
| PAREN_REGEX = re.compile(r'\(.+?\)') # parenthesis | ||
| CHINESE_CHAR_REGEX = re.compile(r'([\u4e00-\u9fff]+[\|·]?)+') # Chinese characters | ||
| BRACKETS_REGEX = re.compile(r'\[.+\]') # pronunciation |
There was a problem hiding this comment.
I think this regex is too greedy and should have a ? on it like PAREN_REGEX does.
I believe that in this definition:
一甲 一甲 [yi1 jia3] /1st rank or top three candidates who passed the imperial examination (i.e. 狀元|状元[zhuang4 yuan2], 榜眼[bang3 yan3], and 探花[tan4 hua1], respectively)/
it will match this text:
[zhuang4 yuan2], 榜眼[bang3 yan3], and 探花[tan4 hua1]
There was a problem hiding this comment.
This particular definition would actually have everything inside the parentheses removed before matching the brackets, but the problem was true for a couple of other definitions, so I changed it.
conceptnet5/readers/cc_cedict.py
Outdated
| LINE_REGEX = re.compile(r'(.+)\s(.+)\[.+\]\s/(.+)/') # separate traditional and simplified words | ||
| DATE_RANGE_REGEX = re.compile(r'(.+?)\s\(.+\d.+\),') # date range | ||
| PAREN_REGEX = re.compile(r'\(.+?\)') # parenthesis | ||
| CHINESE_CHAR_REGEX = re.compile(r'([\u4e00-\u9fff]+[\|·]?)+') # Chinese characters |
There was a problem hiding this comment.
This appears to exclude the range from U+3400..U+4DBF, which appears in definitions such as:
㐌 㐌 [ta1] /variant of 它[ta1]/
There was a problem hiding this comment.
For what it's worth, it also excludes the other CJK extensions with codepoints U+20000 and up, but CEDICT never uses those anyway.
There was a problem hiding this comment.
That was correct for a couple of definitions, so I ended up switching to regex.compile('([\p{IsIdeo}]+[\|·]?)+'), per your suggestion.
No description provided.