Job failed - NVIDIA TAO

[karkapur]

Hey there,

I was trying to train TAO YOLOv4/YOLOv3 model and the job keeps failing or different reasons:

For 200 epochs

For 100 epochs:

Here’s the project link: mouse vs cup - Dashboard - Edge Impulse

Could you help me with this? TIA

[louis]

Hello @karkapur,

The first error should be fixed very soon, @cward has been working on a fix.
For your second error, we are still investigating but we were not able to reproduce it yet. We’ll let you know when we found a fix.

Best,

Louis

[karkapur]

Hey @louis

Thanks for letting me know! @cward could you update me when it’s fixed?

Cheers,
Karan

[cward]

Hey @karkapur,

I will let you know once the fix it live.

Thanks
Carl

[cward]

Hi @karkapur, the first error should be fixed now. The second one we are still working on as it it is quite a rare error.
Thanks
Carl

[karkapur]

Thanks for the update!