sample data, pre-trained word embedding #3

hpduong · 2017-07-20T23:57:10Z

I'm getting this issue when I run training on a08 entity network and a06 seq2seq models.

Can I get or train this file?

zhihu-word2vec-title-desc.bin-100

Also, do you have sample datasets compatible with these models?

brightmart · 2017-07-25T08:02:26Z

for zhihu-word2vec-title-desc.bin-100, please find it in:
https://pan.baidu.com/s/1jIP9e6q
[OLD, TO BE DELETE]
for sample data (multi-label. file name:test-zhihu6-title-desc.txt), you can find it in:
https://pan.baidu.com/s/1gf49auB
[OLD, TO BE DELETE]
train-zhihu4-only-title-all.txt (single-label):
https://pan.baidu.com/s/1jI7R4X4
[NEW, try use this, update 2018-08-12]

zhihu-title-desc-multiple-label-v6.txt.zip, it contains three files:

https://pan.baidu.com/s/1mHgELJUHewQZ9zHDo_uhmA

train-zhihu-title-desc-multiple-label-v6.txt(around 3 million training data, multiple labels)
test-zhihu-title-desc-multiple-label-v6.txt(around 70k validation/test data,multiple labels)
train-zhihu-title-desc-multiple-label-200k-v6.txt( 200k training data,multiple labels, a subset of file one.)

ghost · 2017-07-25T16:21:32Z

Could you please upload it in a different service? I am having a hard time downloading the data and word2vec model. Thanks!

brightmart · 2017-07-28T01:36:36Z

you can follow two steps to get the file.
step1: click the place for download

step2: download the file

Cauchyzhou · 2017-09-05T03:59:50Z

When I run TextRNN model. Terminal reports :
IOError: [Errno 2] No such file or directory: '../zhihu-word2vec.bin-100'
There is no link about zhihu-word2vec.bin-100.

brightmart · 2017-09-05T05:42:49Z

you may use zhihu-word2vec-title-desc.bin-100 or file from your own.
.bin file is just a word embedding file trained from word2vec.

jacky20172017 · 2017-11-10T06:20:30Z

where is the file: /test-zhihu-forpredict-title-desc-v6.txt
when running the a8_predict.py:
IOError: [Errno 2] No such file or directory: '../test-zhihu-forpredict-title-desc-v6.txt'

jacky20172017 · 2017-11-10T06:21:33Z

and also: train-zhihu6-title-desc.txt

jacky20172017 · 2017-11-10T09:59:11Z

there're 2 data_util_zhihu.py in folders aa1_data_util and a07_Transformer
which one to import?

jacky20172017 · 2017-11-10T13:32:46Z

what's the data_type ?

train, test, _ = load_data(vocabulary_word2index, vocabulary_word2index_label,data_type='train')
TypeError: load_data() got an unexpected keyword argument 'data_type'

pmahend1 · 2017-11-30T22:37:29Z

I cant see files there in the links mentioned. They give below error in Chinese

Oh, the page you visit does not exist.

possible reason:

Enter the wrong address in the address bar.
A link you clicked has expired.

#########################

for zhihu-word2vec-title-desc.bin-100, please find it in:
https://pan.baidu.com/s/1kVgdDD9

for sample data (multi-label. file name:test-zhihu6-title-desc.txt), you can find it in:
https://pan.baidu.com/s/1gf49auB

train-zhihu4-only-title-all.txt (single-label):
https://pan.baidu.com/s/1jI7R4X4

#########################################

Could you please upload them to https://github.com/brightmart/text_classification in sample data folder?

pmahend1 · 2017-11-30T22:42:17Z

Nevermind. After few tries the above links worked.

But bin100 is not downloading for some reason/

deatherving · 2017-12-04T03:27:17Z

@pmahend1 Same. The bin100 is not downloading even in China.

brightmart · 2017-12-04T16:06:57Z

@pmahend1 @deatherving
for zhihu-word2vec-title-desc.bin-100, please use this:
https://pan.baidu.com/s/1jIP9e6q

deatherving · 2017-12-06T01:49:18Z

@brightmart Thanks. The file is accessible.

searchlink · 2017-12-06T05:57:01Z

@brightmart
when I run p5_fastTextB_predict.py ,where come with the error below:
FileNotFoundError: [Errno 2] No such file or directory: 'test-zhihu-forpredict-v4only-title.txt'
In addition,where is zhihu-word2vec-multilabel.bin-100?

brightmart · 2017-12-06T11:12:14Z

Hi, You can download some test data ,find it from closed issue. Bright

…

________________________________ 发件人: sky_shine <notifications@github.com> 发送时间: 2017年12月6日 13:57 收件人: brightmart/text_classification 抄送: brightmart; Mention 主题: Re: [brightmart/text_classification] sample data, pre-trained word embedding (#3) when I run p5_fastTextB_predict.py ,where come with the error below: FileNotFoundError: [Errno 2] No such file or directory: 'test-zhihu-forpredict-v4only-title.txt' In addition,where is zhihu-word2vec-multilabel.bin-100? ― You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#3 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ASuYMKE4FWVVhDhdOPqUAKMvXC24J5gMks5s9iyvgaJpZM4Oezj1>.

pmahend1 · 2017-12-07T16:42:38Z

@brightmart I could download the file now. Thanks 👍

behnazeslami · 2017-12-10T10:11:25Z

@pmahend1
@brightmart Hi,
After running p8_TextRNN_train.py 've got this error :

File "./p8_TextRNN_train.py", line 117, in main
    test_loss, test_acc = do_eval(sess, textRNN, testX, testY, batch_size,vocabulary_index2word_label)
  File "./p8_TextRNN_train.py", line 167, in do_eval
    return eval_loss/float(eval_counter),eval_acc/float(eval_counter)
ZeroDivisionError: float division by zero

How can I solve this issue?
I checked do_eval function in train, there is a [for loop] that it doesn't work.
I attached images as below :

iqrasafder · 2018-01-15T10:29:29Z

above links to download "zhihu-word2vec-title-desc.bin-100' are not working.
please share some working links to download data.

Thanks

iqrasafder · 2018-01-15T10:32:38Z

@brightmart please share links to download dataset.

arunarn2 · 2018-01-17T20:06:26Z

@brightmart Do I need an account on pan.baidu.com to download the dataset. Can you please upload the data to the repo?

brightmart · 2018-02-26T14:43:55Z

no need account.

parahaoer · 2018-03-26T13:51:31Z

what directory should I put the file ‘zhihu-word2vec-title-desc.bin-100’？
Thanks！

liangtianxin · 2018-04-06T16:19:17Z

thank you so much

JaeZheng · 2018-04-12T07:16:05Z

@parahaoer I think put it in the same directory with the "model_train.py". For example, when you use TextCNN in the directory"a02_TextCNN", you process p7_TextCNN_train.py to train the model, at this time you should put the file 'zhihu-word2vec-title-desc.bin-100’ in the same directory.

harirajeev · 2018-04-14T02:34:11Z

Could you please upload data 'zhihu-word2vec-title-desc.bin-100’ as well. The links do not work. Appreciate any quick response.

kevinsay · 2018-07-27T02:31:43Z

@brightmart I load the zhihu-word2vec-title-desc.bin-100 as the wordvector file,train-zhihu4-only-title-all.txt as the trainning file,set multi_label_flag=false,use_embedding=true,
a01_FastText，a03_TextRNN，a04_TextRCNN，a05_HierarchicalAttentionNetwork，a06_Seq2seqWithAttention，these models can run，but the accuracy is very low，i don't know why.
and predict，also set multi_label_flag=false,use_embedding=true,there will be more than one prediction label，i need you help.thanks.

brightmart · 2018-08-12T08:30:33Z

hi. thanks for your feedback. as long as you can see that training and validation loss during training process is decreasing, it will be fine. the previous reported f1 score is not a right indicator of accuracy. I am updating the way of how to compute f1 score today.

it is good to see that you can make it work for these several models. can you commit your version to this repository as a new branch?

hlshao · 2018-08-24T06:26:00Z

Respect Sir:

It is a good project!
Could you please provide the file "zhihu-word2vec-title-desc.bin-100" in some where?
and the link below is out of date too...
Many thanks if you can help.

[NEW, try use this, update 2018-08-12]
zhihu-title-desc-multiple-label-v6.txt.zip, it contains three files:
https://pan.baidu.com/s/1mHgELJUHewQZ9zHDo_uhmA
train-zhihu-title-desc-multiple-label-v6.txt(around 3 million training data, multiple labels)
test-zhihu-title-desc-multiple-label-v6.txt(around 70k validation/test data,multiple labels)
train-zhihu-title-desc-multiple-label-200k-v6.txt( 200k training data,multiple labels, a subset of file one.)

brightmart · 2018-08-24T07:25:23Z

set hyper parameter of use pretrain word embedding to false, you will no need to have 'zhihu-word2vec-title-desc.bin-100.

…

________________________________ 发件人: hlshao <notifications@github.com> 发送时间: 2018年8月24日 14:26 收件人: brightmart/text_classification 抄送: brightmart; State change 主题: Re: [brightmart/text_classification] sample data, pre-trained word embedding (#3) Respect Sir: It is a good project! Could you please provide the file "zhihu-word2vec-title-desc.bin-100" in some where? and the link below is out of date too... Many thanks if you can help. [NEW, try use this, update 2018-08-12] zhihu-title-desc-multiple-label-v6.txt.zip, it contains three files: https://pan.baidu.com/s/1mHgELJUHewQZ9zHDo_uhmA train-zhihu-title-desc-multiple-label-v6.txt(around 3 million training data, multiple labels) test-zhihu-title-desc-multiple-label-v6.txt(around 70k validation/test data,multiple labels) train-zhihu-title-desc-multiple-label-200k-v6.txt( 200k training data,multiple labels, a subset of file one.) ― You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub<#3 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ASuYMCMg3LAtterhP33hDleahzuoJnmsks5uT5x-gaJpZM4Oezj1>.

peterHeuz · 2018-08-28T08:11:58Z

I am using the TextRNN (a03) and cannot find this flag. Neither the downloads are working.

I have changed following in p8_TextRNN_train.py:
tf.app.flags.DEFINE_boolean("use_embedding", False, "whether to use embedding or not.")

but the error is still the same:
IOError: [Errno 2] No such file or directory: 'zhihu-word2vec.bin-100'

I have also changed this flag in a02_TextCNN since the TextRNN uses code (data_util_zhihu.py) from this part. The error is still the same.

Can you please share the pretrained embeddings or point me to the right place?

edit: This one seems to be up to date: https://pan.baidu.com/s/1jIP9e6q. I am using your instructions to download @brightmart . After step 1 I get a windows that tells me to download the netdisk client from Baidu:

The installer is in Chinese, which I don't speak, and there is no english version.
Could anyone who has it please upload with another service? Like google, dropbox, onedrive? @deatherving @liangtianxin and anyone else who has it. Would be appreciated.

Gunjitbedi · 2018-09-06T12:48:28Z

@brightmart Cant down load file 'test-zhihu-forpredict-title-desc-v6.txt', please share it on any other platform

brightmart · 2018-11-17T19:17:10Z

re-generated data, and save as cached file, available to download. check this session in README.md:

#Sample data: cached file

wuliuyuedetian · 2019-03-21T03:11:57Z

where is the file: /test-zhihu-forpredict-title-desc-v6.txt
when running the a8_predict.py:
IOError: [Errno 2] No such file or directory: '../test-zhihu-forpredict-title-desc-v6.txt'
Me too

Aivi001 · 2020-02-14T13:30:58Z

I am using the TextRNN (a03) and cannot find this flag. Neither the downloads are working.

I have changed following in p8_TextRNN_train.py:
tf.app.flags.DEFINE_boolean("use_embedding", False, "whether to use embedding or not.")

but the error is still the same:
IOError: [Errno 2] No such file or directory: 'zhihu-word2vec.bin-100'

I have also changed this flag in a02_TextCNN since the TextRNN uses code (data_util_zhihu.py) from this part. The error is still the same.

Can you please share the pretrained embeddings or point me to the right place?

edit: This one seems to be up to date: https://pan.baidu.com/s/1jIP9e6q. I am using your instructions to download @brightmart . After step 1 I get a windows that tells me to download the netdisk client from Baidu:

The installer is in Chinese, which I don't speak, and there is no english version.
Could anyone who has it please upload with another service? Like google, dropbox, onedrive? @deatherving @liangtianxin and anyone else who has it. Would be appreciated.

Hi, I'm facing the same problem, how did you solve it? Thanks in advance

litomvv · 2020-08-18T02:11:31Z

hi.where is the file: /test-zhihu-forpredict-title-desc-v6.txt

brightmart closed this as completed Jul 28, 2017

brightmart changed the title ~~ImportError: No module named data_util_zhihu~~ sample data, pre-trained word embedding Aug 3, 2017

Repository owner deleted a comment from yinjianhong Aug 7, 2017

brightmart mentioned this issue Aug 22, 2017

not found data file #15

Closed

brightmart mentioned this issue Sep 17, 2017

你好，你的项目是不是没有数据文件啊 #20

Closed

wyl6 mentioned this issue Apr 7, 2018

Errors when debugging #42

Closed

brightmart reopened this Aug 12, 2018

peterHeuz mentioned this issue Aug 28, 2018

ModuleNotFoundError: No module named 'data_util_zhihu' #59

Closed

brightmart closed this as completed Nov 17, 2018

sample data, pre-trained word embedding #3

sample data, pre-trained word embedding #3

Comments

hpduong commented Jul 20, 2017 • edited Loading

brightmart commented Jul 25, 2017 • edited Loading

ghost commented Jul 25, 2017

brightmart commented Jul 28, 2017

Cauchyzhou commented Sep 5, 2017

brightmart commented Sep 5, 2017 • edited Loading

jacky20172017 commented Nov 10, 2017 • edited Loading

jacky20172017 commented Nov 10, 2017

jacky20172017 commented Nov 10, 2017

jacky20172017 commented Nov 10, 2017

pmahend1 commented Nov 30, 2017 • edited Loading

pmahend1 commented Nov 30, 2017 • edited Loading

deatherving commented Dec 4, 2017

brightmart commented Dec 4, 2017

deatherving commented Dec 6, 2017

searchlink commented Dec 6, 2017 • edited Loading

brightmart commented Dec 6, 2017 via email

pmahend1 commented Dec 7, 2017

behnazeslami commented Dec 10, 2017 • edited Loading

iqrasafder commented Jan 15, 2018

iqrasafder commented Jan 15, 2018

arunarn2 commented Jan 17, 2018

brightmart commented Feb 26, 2018

parahaoer commented Mar 26, 2018

liangtianxin commented Apr 6, 2018

JaeZheng commented Apr 12, 2018

harirajeev commented Apr 14, 2018

kevinsay commented Jul 27, 2018 • edited Loading

brightmart commented Aug 12, 2018 • edited Loading

hlshao commented Aug 24, 2018

brightmart commented Aug 24, 2018 via email

peterHeuz commented Aug 28, 2018 • edited Loading

Gunjitbedi commented Sep 6, 2018 • edited Loading

brightmart commented Nov 17, 2018

wuliuyuedetian commented Mar 21, 2019

Aivi001 commented Feb 14, 2020

litomvv commented Aug 18, 2020

hpduong commented Jul 20, 2017 •

edited

Loading

brightmart commented Jul 25, 2017 •

edited

Loading

brightmart commented Sep 5, 2017 •

edited

Loading

jacky20172017 commented Nov 10, 2017 •

edited

Loading

pmahend1 commented Nov 30, 2017 •

edited

Loading

pmahend1 commented Nov 30, 2017 •

edited

Loading

searchlink commented Dec 6, 2017 •

edited

Loading

behnazeslami commented Dec 10, 2017 •

edited

Loading

kevinsay commented Jul 27, 2018 •

edited

Loading

brightmart commented Aug 12, 2018 •

edited

Loading

peterHeuz commented Aug 28, 2018 •

edited

Loading

Gunjitbedi commented Sep 6, 2018 •

edited

Loading