Failure to during Training on Quantization

#1

I have a dataset of audio that is a mix of 2 second clips and longer ones that are a couple minutes. I am able to train and build a model fine when it the impulse is configured with a Window Size of 2000. However when I change the Window Size to 1000, I get a failure during the NN training. I pops up after all the steps of training occur:

Saving model...
Converting TensorFlow Lite float32 model...
Converting TensorFlow Lite int8 quantized model with float32 input and output...
Unable to convert and save TensorFlow Lite int8 quantized model:
Model resulted in Nan value during calibration. Please make sure model results in all real-values during inference with provided dataset.Node number 0 (RESHAPE) failed to invoke.

Converting TensorFlow Lite int8 quantized model with int8 input and output...
Unable to convert and save TensorFlow Lite int8 quantized model:
Model resulted in Nan value during calibration. Please make sure model results in all real-values during inference with provided dataset.Node number 0 (RESHAPE) failed to invoke.

Calculating performance metrics...
Traceback (most recent call last):
  File "/home/train.py", line 651, in <module>
    main_function()
  File "/home/train.py", line 638, in main_function
    model_int8=tflite_quant_model,
UnboundLocalError: local variable 'tflite_quant_model' referenced before assignment

Application exited with code 1 (Error)

Job failed (see above)

I tried removing the existing MFE block and NN and re-adding them, so the default values are there. I still ended up with the same error.

1 Like
#2

Hi Luke,

Hope you’re doing well! I saw you have a few projects under your account—let me know the project ID where you’re seeing this happening and I’ll take a closer look.

Warmly,
Dan

#3

Thanks Dan! I hope you are doing well too.
It is project 3620 called Urban Sound. I configured it so it currently fails when when training is run. I am betting there is a bad .wav sample in there some where, so when it is going to run the quantization profiling, it is getting a 0 length wav.

#4

I’m also getting this error. Uploaded the yes/no dataset and did an MFE block -> NN (keeping default parameters in both). Please let me know if you’re able to solve it and what steps I can take!

ei_nn_error
ei_nn_error1646×1080 160 KB

#5

Thank you both for the reports, I’m looking into this right now.

#6

We’ve identified an issue on our side and will have a fix deployed soon! Thanks again for the detailed reports, they were very helpful in tracking down the issue.

Once the fix arrives, your MFE blocks will still have the bad data cached. To get things working again you’ll need to either delete and recreate the MFE block, or change one of the settings to invalidate the cache (e.g. by setting the window increase 1ms higher).

I’ll post in here again once the fix has been deployed. Have a great weekend!

Warmly,
Dan

2 Likes
#7

The fix is now live, but let us know if you run into any more issues!

Warmly,
Dan

1 Like
#8

That did the trick!! Thanks @dansitu and team. I have never seen such quick turn around on fixes. Y’all are doing impressive work!

3 Likes
#9

Thank you for the kind words :slight_smile: We love having such an active and highly technical community, it makes our work a lot easier!