Experienced person to help with the Edge Impulse model in an ESP32-CAM

Hello,
I want to use a ESP32-CAM to make a color sensor. There will be one cylinder that can be green or red in front of the camera and I want to classify it.

I did two steps and it doesn’t work yet.

  • Step 1. I used the mobile phone to take 25 pictures green and 25 pictures red. I trained the edge impulse with them and obtained 100% of correct classification in the edge impulse. After:

    • I deployed the edge impulse in the mobile phone using the QR code. The classification works very well with 0.90 of confidence.
    • I deployed the edge impulse in a ESP32-CAM. It doesn’t work properly and the confidence is very low.
  • Step 2. I used the ESP32-CAM to take 50 pictures green and 50 pictures red. I trained the edge impulse with them and obtained 91% of correct classification in the edge impulse. After:

    • I deployed the edge impulse in the mobile phone using the QR code. The classification works amazingly well with 1.00 of confidence.
    • I deployed the edge impulse in a ESP32-CAM. It doesn’t work properly and the confidence is very low.

My thoughts:

  • The edge impulse is well trained because it works perfectly with the mobile.
  • I used the ESP32-CAM to take the pictures to train the model. The pictures used for the classification at the deployed ESP32-CAM are very similar to the ones used for the training (light, angle…).

I need help with:

  • I don’t know how to improve the performance of the model in the ESP32-CAM.
  • I guess the camera of the ESP32-CAM is a low quality one. Can it be improved?

Thank you!!

What does it doesn't work properly mean in this context?

Just because the model is making predictions with a low confidence value doesn’t mean the model is performing poorly. If your False Positive Rate and False Negative Rate are zero, then a prediction with 0.01 confidence on a True Positive is perfectly fine.

What are your False Positive Rate and False Negative Rate values?

Thank you MMarcial!
The issue I have it is not the confidence it is the False Positives.
When I use the mobile phone it detects the colors perfectly (very near to 100% of true positives) but when I use the ESP-32 it clasifies allmost all the photos as GREEN. I would say that it classifies the 75% of RED photos as GREEN.

I don’t think the resolution of the camera matters in your use case of classifying different colored cylinders. If the Edge Impulse Studio Feature Explorer chart is showing good separation of classes, then the model should work well when deployed to the ESP hardware.

So at this point I cannot offer any further advice :frowning:

Hello @Mikel,

Could you share here an image example from your training dataset and one taken from the ESP32?
Maybe the angles, brightness, shadows, colors, etc… are different?

Could you use pictures taken from the esp32 device to train your model and see if there is any improvements?

Best,

Louis

Hello @louis and @MMarcial ,
I took 50 green pictures and 50 red pictures using the ESP32-CAM (different angles, different distances…) and I started from the beginning again. The truth is that there is not a big improvement and the ESP32-CAM classifies as Green a lot of the Red pictures. When I deploy it in the moblie the classification is almost perfect.

The pictures taken for the training and the ones used for the classification are as similar as I can (same light, similar distances, similar position…). Looking to the pictures taken by the ESP32-CAM when the picture is taken very near the color quality is not very good, but the classification of the ESP32-CAM it is bad regardless the distance.

When they are delivered to me I will use a OV5640 and a OV5640AF cameras to see if there is any improvement.

Thank you very much for your help.

@Mikel,

Can you share your project ID so I can have a look and check if the project is properly setup to maximize the inference results please?

Best.

Louis

Hello @louis ,

Thank you for your answer. The ID is: 156730

Best regards,

Mikel

Hello @Mikel,

Thanks I just had a look at your project and indeed nothing seems wrong. You might want to add some more data over time to improve the accuracy but it’s not a big deal for now.

I believe the issue comes from the embedded part. How did you deployed the firmware?
Did you use the binary that you recompiled using ESP-IDF?
Did you use the Arduino Library?
Did you use something custom?

Are you able to share your code so I can have a quick look?

Best,

Louis

Hello @louis,

I make the prototype described in here: GitHub - alankrantas/edge-impulse-esp32-cam-image-classification: Live Image Classification on ESP32-CAM and TFT with MobileNet v1 from Edge Impulse (TinyML). So, I am able to see at the TFT the picture taken by the camera.

I download the arduino library from Edge Impulse and chage the inferencing “xxx.h” at the original code from that Github.

If you think it is neccessary I can share the code (I don’t have it with me while I am writing this message). By the way, if you want me to share the code, which one is the best way??

Thank you!!

Hello @Mikel,

I’d suggest to also have a look at ESP32 example that is now directly available from the library examples.

We have a different way of resizing the image than in this repo: GitHub - edgeimpulse/example-esp32-cam: Builds and runs an exported image classification impulse on ESP32 Cam

See here to setup the board parameters if needed:

Best,

Louis

Hello @louis,

Thank you for your help.

After a time I could return to this project and these are my comments:

  • I used that Github because I want to start the inference when I push a button and it has an output. Those characteristics are very useful for me. Right now I don’t have time/knowledge to use another one.
  • I tried these cameras: OV5640 and a OV5640AF and the quality of the initial picture is better, but the classification remains as bad as before.
  • There was a really big improvement in the classification when in the “EON Tuner” at “View” I changed the selection from “Validation” to “Test”. After I select the model that has the best score with the “Test” samples. I guess that could mean it is overfitted.

Thank you!!