Skip to content

Commit 55f8eb6

Browse files
authored
np.fromfile instead of np.fromstring is shorter and doesn't throw an assertion in python 2.7.
Original code, in python 2.7 was resulting in: python train.py ('opening:', '/home/dchichkov/n/wikitext-103-raw') Traceback (most recent call last): File "train.py", line 183, in <module> trX, vaX, teX = wiki3(path=args.data_file) File "/home/dchichkov/n/blocksparse/examples/lstm/utils.py", line 48, in wiki3 tr_text = text_to_npy(os.path.join(path, "wiki.train.raw")) File "/home/dchichkov/n/blocksparse/examples/lstm/utils.py", line 42, in text_to_npy text = open(path).read(nbytes).encode() UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 39: ordinal not in range(128)
1 parent b8ab380 commit 55f8eb6

File tree

1 file changed

+1
-2
lines changed

1 file changed

+1
-2
lines changed

examples/lstm/utils.py

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -39,8 +39,7 @@ def text8_stream(text, nbatch, nsteps, maxbatches=None):
3939

4040

4141
def text_to_npy(path, nbytes=-1):
42-
text = open(path).read(nbytes).encode()
43-
text = np.fromstring(text, dtype=np.uint8)
42+
text = np.fromfile(path, dtype=np.uint8, count=nbytes)
4443
return text
4544

4645
def wiki3(path):

0 commit comments

Comments
 (0)