You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi
When I start training, time of train of any step is good and it uses GPU but after 32 steps (if _batches_per_group=32 in datafeeder) GPU utilization going to 0 and after many seconds the queue of data becomes ready and again training starts.
I saw datafeeder.py and it uses threading for queue of data but why it stops training? How can I increase GPU utilization?
I increased _batches_per_group in datafeeder.py but it causes training stops more seconds. Following picture show that if I set _batches_per_group=128 training stops for 89 seconds!
The text was updated successfully, but these errors were encountered:
Hi
When I start training, time of train of any step is good and it uses GPU but after 32 steps (if
_batches_per_group=32
in datafeeder) GPU utilization going to 0 and after many seconds the queue of data becomes ready and again training starts.I saw datafeeder.py and it uses threading for queue of data but why it stops training? How can I increase GPU utilization?
I increased
_batches_per_group
in datafeeder.py but it causes training stops more seconds. Following picture show that if I set_batches_per_group=128
training stops for 89 seconds!The text was updated successfully, but these errors were encountered: