AttributeError: 'SegDirectoryIterator' object has no attribute 'next'

[simonsayshi]

@ahundt @aurora95 need help here!!

I met this issue when I run train.py

Exception in thread Thread-2:
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\lib\threading.py", line 916, in _bootstrap_inner
self.run()
File "C:\ProgramData\Anaconda3\lib\threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\keras\utils\data_utils.py", line 568, in data_generator_task
generator_output = next(self._generator)
File "C:\ProgramData\Anaconda3\lib\site-packages\keras\preprocessing\image.py", line 737, in next
return self.next(*args, **kwargs)
AttributeError: 'SegDirectoryIterator' object has no attribute 'next'
lr: 0.010000

Epoch 1/250
Traceback (most recent call last):
File "train.py", line 233, in
label_suffix=label_suffix, ignore_label=ignore_label, label_cval=label_cval)
File "train.py", line 163, in train
class_weight=class_weight
File "C:\ProgramData\Anaconda3\lib\site-packages\keras\legacy\interfaces.py", line 87, in wrapper
return func(*args, **kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\training.py", line 2011, in fit_generator
generator_output = next(output_generator)
StopIteration

I debugged for all day and have no idea why is that happening :(
my sys Env is win10 python3.6 keras 2.0.8 tensorflow 1.2.1 any suggestion？

Thanks a lot!!!!

[simonsayshi]

@unixnme @ahundt @PavlosMelissinos @aurora95 please！

[zy937085318]

hey, do u handle this issue? I meet the question too.

[cemsaz]

@simonsayshi, @zy937085318 I have the following solution:
Strangely, this repository is using Python 2 with Tensorflow. If you are using Python 3, you will experience this issue. One workaround is defining your own 'next' function.
You should replace '_get_batches_of_transformed_samples(self, index_array):' with 'next(self):'

You should also add the following in your new next function:

with self.lock:
     index_array, current_index, current_batch_size = next(self.index_generator)

[chinmayshah99]

@cemsaz I'm running it on python3 and it works fine. Doesn't seem to be problem in python3 as in general.

[F951]

@cemsaz I have the same problem. Could you please specify in which script and in which lines I should do the replacement?
(Just in case, I'm using CUDA v8.0, cudnn 6.0, python 3.6 and Keras 2.0.8)
Thanks in advance.

[drmohler]

Would someone be willing to explicitly post the solution to this issue? I am having a hard time getting around it. Thank you!!

[simonsayshi]

@cemsaz you got the point but my solution is different. @drmohler try update your python version to 3.6.x
the scripts were written by python2 so there were few differences. follow the error messages and debug one by one you will make it work

[daodaoawaker]

@simonsayshi Could you tell me your final solution? Thank U very much

[simonsayshi]

@daodao1991 plz follow the instruction above..

[daodaoawaker]

@simonsayshi I run it on Ubuntu16.04 ，and try tensorflow-gpu 1.3.0/1.2.1 with keras 2.0.8/2.1.2，and Python3.6/2.7 have been used. I just fail again and again，which drives me crazy. Please help me!!
I know this repository is good, but I cann't run it. So frustrated!

[simonsayshi]

attach your full log maybe？dont know how to run in ubuntu but logs may help a lot

…

On Wed, Aug 21, 2019 at 21:37 daodao1991 ***@***.***> wrote: @simonsayshi <https://github.com/simonsayshi> I run it on Ubuntu16.04 ，and try tensorflow-gpu 1.3.0/1.2.1 with keras 2.0.8/2.1.2，and Python3.6/2.7 have been used. I just fail again and again，which drives me crazy. Please help me!! I know this repository is good, but I cann't run it. So frustrated! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#68?email_source=notifications&email_token=AHQ4RSTLT2GG63MQUF4ZPTTQFYJWPA5CNFSM4ESIT7P2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD433HBY#issuecomment-523744135>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHQ4RSSAVJSAHDR47NZZOALQFYJWPANCNFSM4ESIT7PQ> .

[daodaoawaker]

@simonsayshi Thank u for your reply, I change to win 10, and install tensorflow-gpu 1.4.0 && keras 2.1.2 && python 3.5 , but also failed. This is error logs:

(tf_1_4) F:\Pycharm_Projects\FCN\aurora95\Keras-FCN>python train.py
Using TensorFlow backend.
D:\Anaconda3\envs\tf_1_4\lib\site-packages\scipy_init_.py:115: UserWarning: Numpy 1.13.3 or above is required for this version of scipy (detected version 1.13.1)
UserWarning)
Traceback (most recent call last):
File "train.py", line 16, in
from models import *
File "F:\Pycharm_Projects\FCN\aurora95\Keras-FCN\models.py", line 6, in
from keras_contrib.applications import densenet
File "D:\Anaconda3\envs\tf_1_4\lib\site-packages\keras_contrib-1.2.1-py3.5.egg\keras_contrib_init_.py", line 2, in
File "D:\Anaconda3\envs\tf_1_4\lib\site-packages\keras_contrib-1.2.1-py3.5.egg\keras_contrib\backend_init_.py", line 14, in
File "D:\Anaconda3\envs\tf_1_4\lib\site-packages\keras_contrib-1.2.1-py3.5.egg\keras_contrib\backend\tensorflow_backend.py", line 10, in
ImportError: cannot import name '_postprocess_conv3d_output'

I change it to keras 2.0.8, it will go back to the old error,

[daodaoawaker]

@simonsayshi Hi, I fix the above problem by copying the revalent from a previous version of keras which included it. But ....
Then, python train.py, it seems to be running like this ：

Trainable params: 134,346,581
Non-trainable params: 0
lr: 0.000625
Epoch 1/250
2019-08-24 16:47:49.694895: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\bfc_allocator.cc:217] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.14GiB
. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2019-08-24 16:47:50.586328: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\bfc_allocator.cc:217] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.14GiB
. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2019-08-24 16:47:50.594907: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\bfc_allocator.cc:217] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.07GiB
. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2019-08-24 16:47:50.610010: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\bfc_allocator.cc:217] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.13GiB
. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2019-08-24 16:47:50.681211: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\bfc_allocator.cc:217] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.07GiB
. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
27/11127 [..............................] - ETA: 3:17:14 - loss: 1.7019 - sparse_accuracy_ignoring_last_label: 0.6480

My new question is, the following error occurs when the program reads an image, it seems like GPU resource exhausted :

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[7,7,512,4096]
[[Node: training/SGD/gradients/fc1/Square_grad/mul = Mul[T=DT_FLOAT, _class=["loc:@fc1/Square"], _device="/job:localhost/replica:0/task:0/device:GPU:0"](training/SGD/gradients/fc1/Square_grad/mul/x
, fc1/kernel/read)]]
[[Node: metrics/sparse_accuracy_ignoring_last_label/Reshape/_277 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/t
ask:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_471_metrics/sparse_accuracy_ignoring_last_label/Reshape", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

The strange thing is that my program has the following code:
config = tf.ConfigProto()
config.gpu_options.allocator_type = 'BFC'
config.gpu_options.per_process_gpu_memory_fraction = 0.80
session = tf.Session(config=config)
K.set_session(session)
Now I'm stuck with this bug and I don't know how to fix it. Help me !! I'm gratitude to you for helping me !!

[luyu-wang]

https://github.com/aurora95/Keras-FCN/pull/56/files#diff-884514ab248f7857177828d3a62cbdc5R194 shows the changed made for keras==2.1.x, as i use keras==2.0.8, i go back to the origin code, the code is of utils/SegGenerator.py.
then i can run the train.py