Job failed from running out of memory

Report bugs
#1

For some reason every time I try to build a model from the dataset I uploaded, the job fails because it keeps running out of memory. I don’t think it is because I am using more memory then I am allowed on the free version because I have set every possible setting to the lowest to train the model e.g. images processed in greyscale, using smallest version of MobilenetV1, using a batch size of 8. So I have no clue what could be causing this.

I would very much appreciate any help on this, thanks!

Also, I am new to using EdgeImpulse, so sorry if this isn’t a very big problem and I’ve just done something silly.

Project ID: 1088132

This problem happens when I am in Impulse #2 and am trying to train the model with transfer learning tab or the retrain model tab. The model trains all the way to the last epoch, and then fails and says it is out of memory

Reproducibility:

  • [ x ] Always
  • [ ] Sometimes
  • [ ] Rarely
#2

Hey @danjo, apologies for the poor experience. We’re looking into the issue and we’ll let you know as soon as we have an explanation and a way forward. Thanks for your patience.

1 Like
#3

H @danjo, we’ve had a look at your project and there’s nothing that looks problematic with your current settings of impule #2. Though it looks like the impulse started having issues since it was first attempted with unsupported featuers, e.g. size 170x75. This might be due to cached values on our side, and as we’ve landed fixes around related things lately, it would be helpful to hear if this problem remains if you tried training in a new impulse /or new project. Thanks!

1 Like
#4

Thanks @sara.olsson.ei! That worked fine yep