Intuitive User-Settable FOMO Object Detection Threshold

When the Build button on the EI Studio Deployment page is clicked, popup a form with defaults populated. At a minimum one form field is required entittled Object Detection Threshold with a default value of 0.0000000000000000000001.

I understand that the EI Studio Model Testing page has a menu item Set Confidence Thresholds that actually controls the parameter, EI_CLASSIFIER_OBJECT_DETECTION_THRESHOLD, and is saved into model_metadata.h. But how is this EI Studio setting in anyway obvious that it will control the &results of an inference via a call to run_classifier()?


Precieved FOMO Instability

When I would deploy a FOMO model to a Sony Spresense, the model would identify objects sometimes but usually not. There was no ryme or reason as to why the model would correctly predit a FOMO object was present. I’ve put in 19 days on this discerning issue. I tried everything to make the FOMO model more reliable. Hooking up to Live Classification worked well, usually giving 90% to 100% accuracy. I would keep the camera and lighting the same. I did not move the object to be FOMOed. I would then deploy my Arduino model to the device and FOMO would report No Objects Found. What? Why?

I studied the EI Spresense Firmware that uses complex methods of capturing camera data and converting data. I simply used the Sony Spresense SDK built-in functions for such conversions. Could these FrameBuffer manipulation routines be so different to cause FOMO to act un-reliably? That is, I thought maybe using the EI Spresense Firmware to collect data with the EI Studio, but then use the Sony SDK routines in the Arduino code might be the cause of the FOMO unsteadiness.

I tried this, that and the other thing. I made new models. I created new data on device saving to SD card then uploaded to EI Studio. I deleted the bounding boxes file and re-labeld all the images. Nothing worked. I was about to cry Uncle until I found this gem of a meatball: EI_CLASSIFIER_OBJECT_DETECTION_THRESHOLD

Changing EI_CLASSIFIER_OBJECT_DETECTION_THRESHOLD from 0.500 to 0.050 allowed me to see the FOMO results when the object was in view of the camera. When I removed the object from the FOV, FOMO correctly reported No Object Found.

So please ML Model creators, allow me, the master of my domain, to interpret, translate, decipher, exploit and act on the Model results. The ML Model has no context as to the solution it has been deployed into, so do not hide results from the developer. I am yet to come across Edge Impulse documentation that says, "oh, yeah, you need to edit model_metadata.h".

Perhaps Performance Calibration will make this point all moot in the future. But I wonder how many developers came to the platform and found it very easy to build, train, and deploy a model to a device only to find that FOMO declared it could not find any objects (when running on device). Naturally, they would quickly move on to other platforms, not willing to dig into some very obscrue settings within EI Studio.

!Mamma Mia


Hi @MMarcial

Let me check it out it may take a bit of digging to get an answer on wether this can be covered by performance calibration.

On the docs side, there is a page on model_metadata.h, did you see this?

This doc notes that this file should not be edited. As it is generated by studio, so that parameter should be configured prior to generation. Let me do some testing locally and get back to you.



I suppose the simple path forward is to change this line in the FOMO example code from:

  • ei_printf("No objects found.\n");
  • ei_printf("No objects found with a prediction value greater than %f.\n", EI_CLASSIFIER_OBJECT_DETECTION_THRESHOLD);

@Eoin I looked at the link you provided where therein it reads,

If `EI_CLASSIFIER_OBJECT_DETECTION` is set to 1, this macro is defined. Only bounding boxes with confidence scores equal to or above this value will be returned from inference.

I don’t understand If EI_CLASSIFIER_OBJECT_DETECTION is set to 1, this macro is defined.

So if I #define EI_CLASSIFIER_OBJECT_DETECTION 0.500 then is the macro defined or not?

I think the definition should read:

Only bounding boxes with confidence scores equal to or greater than this value will be returned from an inference.

I understand one should not edit model_metadata.h.

But as I stated in the original post how is one to know that EI_CLASSIFIER_OBJECT_DETECTION_THRESHOLD is controlled by the EI Studio Model Testing page menu item Set Confidence Thresholds parameter?

Hi @MMarcial

The default value in the spresense firmware may need to be updated to reflect your findings, I’ve got a spresense to test with now, can you give me the project id or invite me to collaborate so can try reproducing? The firmware will then be updated if i can verify here firmware-sony-spresense/model_metadata.h at 6b9b56b5d255c99d1f8546109ead2851936ab76d · edgeimpulse/firmware-sony-spresense · GitHub


Also thanks for the suggestion to improve our docs, I will put together a pr for that now :smiley:

@Eion this is not a Spresense issue (I just happen to find it while developing with a Spresense.)

The issue happens when deploying the FOMO Arduino library. I have not tested other deployment libraries.

Basically is boils down to this screen in EI Studio:

The last sentence:

This affects both live classification and model testing.

needs to be modified as:

This setting affects the results shown on the live classification page, the model testing page, and the deployed Arduino library.

As I already stated, "...the deployed Arduino library" may need to be replaced with "...the deployed library or model".

The setting in the “Set Confidence Thresholds” does not effect an Image Classification model or a Keyword detection model. Only the FOMO model is affected by the EI Studio GUI entry.

1 Like

Ok I’ve got a pr under review to fix that wording @MMarcial will let you know when its live :smiley:

BTW are you on our expert calls, will you make the next one? Maybe we can take some time to go through all of your issues with the guys at the end of the call. Perhaps there are a few we could work through faster in a discussion.



Also the model_metadata.h doc will need input from the embedded team and studio to get it linked to the studio ui as you suggested.

This one will take a bit more time to put together. Thanks again for the valued suggestions