Fixes to the ConceptNet build process and some readers #148
Conversation
Snakefile
Outdated
| CORE_DATASET_NAMES += ["emoji/{}".format(lang) for lang in EMOJI_LANGUAGES] | ||
|
|
||
|
|
||
| DATASET_NAMES = CORE_DATASET_NAMES + ["dbpedia/dbpedia_en"] |
There was a problem hiding this comment.
In the code-review-fixes-20171117 branch, I moved this part to after setting TESTMODE variables, to ensure that in the test mode CORE_DATASET_NAMES += ["emoji/{}".format(lang) for lang in EMOJI_LANGUAGES] is being set with EMOJI_LANGUAGES equal to ['en', 'en_001'] as opposed to all languages.
There was a problem hiding this comment.
Oh whoops, I should have paid more attention to the merge conflict.
Snakefile
Outdated
| CORE_DATASET_NAMES += ["conceptnet4/conceptnet4_flat_{}".format(num) for num in range(10)] | ||
| CORE_DATASET_NAMES += ["ptt_petgame/part{}".format(num) for num in range(1, 13)] | ||
| CORE_DATASET_NAMES += ["wiktionary/{}".format(lang) for lang in WIKTIONARY_LANGUAGES] | ||
| CORE_DATASET_NAMES += ["emoji/{}".format(lang) for lang in EMOJI_LANGUAGES] |
There was a problem hiding this comment.
I think this is still incorrect, because CORE_DATASET_NAMES gets updated with emoji files before EMOJI_LANGUAGES is overwritten in line 107.
| output: | ||
| DATA + "/raw/{dirname}/{filename}" | ||
| shell: | ||
| "wget -nv {RAW_DATA_URL}/{wildcards.dirname}/{wildcards.filename} -O {output}" | ||
| "unzip {input} raw/{wildcards.dirname}/{wildcards.filename} -d {DATA}" |
There was a problem hiding this comment.
It looks like every time a single new file is added to the process, one would have to re-download the entire conceptnet-raw-data-5.5.zip package. Would it be possible to unbundle it?
/r/MannerOf, not/r/IsA