ValueError: Input contains NaN, infinity or a value too large for dtype('float32')

[GraceBoston]

Hello,

I followed the instruction, but i stuck with this error. Any one help me? I used 5 steps per epoch just for checking if it is working without error. But I found ValueError after one epoch.

CUDA v9.0.176
tensorflow-gpu 1.11.0
Keras2.2.4

** starting learning_rate is 0.001**
** training start with parameters: **
	verbose: 1
	max_queue_size: 32
	workers: 32
	epochs: 30
	use_multiprocessing: True
	shuffle: False
	steps_per_epoch: 5
	generator: <app.datasets.dataset_utility.DataSequence object at 0x7f0c24034550>
	validation_data: <app.datasets.dataset_utility.DataSequence object at 0x7f0c24034898>
	callbacks: [<keras.callbacks.ModelCheckpoint object at 0x7f0c24027f28>, <keras.callbacks.TensorBoard object at 0x7f06541db390>, <keras.callbacks.ReduceLROnPlateau object at 0x7f06541db160>, <app.callback.MultipleClassAUROC object at 0x7f0654271cc0>, <app.callback.SaveBaseModel object at 0x7f06541db0f0>, <app.callback.ShuffleGenerator object at 0x7f0654ef7780>]
** MultipleClassAUROC callback is ready
** SaveBaseModel callback is ready
Epoch 1/30
5/5 [==============================] - 42s 8s/step - loss: nan - val_loss: nan
current learning rate: 0.0010000000474974513
*** epoch#1 dev auroc ***
103/103 [==============================] - 40s 390ms/step
*** dev auroc ***
y = (3272, 14)
y_hat = (3272, 14)
Traceback (most recent call last):
  File "./kmi_train", line 22, in <module>
    main(config_file=args.config)
  File "./kmi_train", line 9, in main
    tr.train()
  File "/home/grace/Keras_MedicalImgAI/app/main/Trainer.py", line 164, in train
    self.history = self.model_train.fit_generator(**self.fitter_kwargs)
  File "/home/grace/Keras_MedicalImgAI/test/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/home/grace/Keras_MedicalImgAI/test/lib/python3.6/site-packages/keras/engine/training.py", line 1418, in fit_generator
    initial_epoch=initial_epoch)
  File "/home/grace/Keras_MedicalImgAI/test/lib/python3.6/site-packages/keras/engine/training_generator.py", line 251, in fit_generator
    callbacks.on_epoch_end(epoch, epoch_logs)
  File "/home/grace/Keras_MedicalImgAI/test/lib/python3.6/site-packages/keras/callbacks.py", line 79, in on_epoch_end
    callback.on_epoch_end(epoch, logs)
  File "/home/grace/Keras_MedicalImgAI/app/callback.py", line 132, in on_epoch_end
    _, mean_auroc, _, _ = metrics.compute_auroc(self.model, self.generator, self.class_mode, self.class_names)
  File "/home/grace/Keras_MedicalImgAI/app/utilities/metrics.py", line 21, in compute_auroc
    current_auroc = roc_auc_score(y, y_hat, average=None)
  File "/home/grace/Keras_MedicalImgAI/test/lib/python3.6/site-packages/sklearn/metrics/ranking.py", line 277, in roc_auc_score
    sample_weight=sample_weight)
  File "/home/grace/Keras_MedicalImgAI/test/lib/python3.6/site-packages/sklearn/metrics/base.py", line 79, in _average_binary_score
    y_score = check_array(y_score)
  File "/home/grace/Keras_MedicalImgAI/test/lib/python3.6/site-packages/sklearn/utils/validation.py", line 453, in check_array
    _assert_all_finite(array)
  File "/home/grace/Keras_MedicalImgAI/test/lib/python3.6/site-packages/sklearn/utils/validation.py", line 44, in _assert_all_finite
    " or a value too large for %r." % X.dtype)
ValueError: Input contains NaN, infinity or a value too large for dtype('float32').