Hello @edghba,
I tried with the default example, I just changed the
#define CAMERA_MODEL_AI_THINKER.
Please note that it was not working perfectly, I had like 10% of the frames that contained a bounding box.
@matkelcey, have you encountered this kind of color limitation with FOMO? Is there any tip to make the model better and separate better on colors?
Best,
Louis
Hello Louis,
Thank you for the information. Then we see the same issue.
I think the problem is at the quantized, Tensorflow Lite model created by Edge Impulse because the neural net works on PC in Edge Impulse well. It detects the objects different only in colors perfectly.
Some accuracy degradation should be on ESP because of the int instead of float etc. but actually unfortunately it does not work on ESP.
Best regards,
Lehel
Hello Louis,
Short summary and a question.
After lot of different training (RGB, grey-scale, with more, less training data, at different light conditions and in different places, with MobileNetV2 0.1/0.35) and testing on ESP-EYE and ESP32 Ai-Thinker I found that:
The FOMO (Faster Objects, More Objects) MobileNetV2 0.35, so when the alpha = 0.35, works in grey-scale very robustly when I train it with lot of data (for 2 different objects with a total of 800 training images in different positions).
ESP-EYE detects the object very reliably but since it is trained in grey-scale it can not distinguish among the objects of exact same shape but different colors.
I have found in the Edge Impulse description also:
FOMO (Faster Objects, More Objects) MobileNetV2 0.35 → These models are designed to be <100KB in size and support a grayscale input at any resolution.
I have not found such neural net on Edge Impulse site which works with colorful objects.
The MobileNetV2 algorithm and your FOMO (Faster Objects, More Objects) MobileNetV2 0.35 algorithm works for colorful objects also. Edge Impulse proves also because it works on PC or on mobile phone perfectly on your site in the Launch in Browser section.
I wonder why it can not work after the quantization at all?
I am thinking about that it should work after quantization also. I think it is not issue of the light conditions and of the camera settings, because I also tested at different conditions, and because the quantized grey-scale model works in different light conditions also.
Could you investigate/look at, please why we have this issue for colorful objects but not for grey-scaled objects?
The MobileNetV2 object detection algorithm is good for colorful objects also.
If this issue could be clarified and solved then it would be really great for me.
Project Id for this entry: 288939
Thank you & Best regards,
Lehel
Hello Louis,
Have you had time to ask within the team?
I am thinking about what the problem can be. I suspect that the FOMO is trained by color images (RGB, 3 channels) but during the object detection the input image is read as grey-scale (1 channel) image. So the FOMO trained on colored image tries to detect from grey-scale input image and therefore it does not work or very-very rarely it can detect from grey-scale input image.
You handle inside the lib if the input image is grey-scaled (1 channel) or RGB with 3 channels.
I can only see this line converted = fmt2rgb888(fb->buf, fb->len, PIXFORMAT_JPEG, snapshot_buf); in your ei_camera_capture() where you read the captured jpg image and convert to RGB format.
I do not know what you do with the RGB input (3 channels) image in the lib later.
I suspect you convert it to grey-scale image when the FOMO is set to grey-scale but what happens if the FOMO is trained in RGB…
Best regards,
Lehel
@edghba I have tested this project with EON Compiler + Int8 quantization in RGB on a couple of other pieces of hardware like the Nicla Vision and it works as expected, I’ve got an ESP-EYE on order so will take a look at the hardware side later this week.
@edghba I have now thoroughly tested this deployment with int8+EON compiler and it does work as expected on the ESP-EYE. The grayscale model you built has a much larger dataset so does perform better after being quantized compared to the RGB model, but I saw no real difference between my ESP-EYE and Nicla Vision in both Arduino Library Output and EI Firmware modes for your RGB model. I was testing by pointing the cameras at some of your test samples on my screen.
One thing to note is that with a smaller dataset using a different camera for acquisition and inferencing can give poor results because of the different exposure/iso of each sensor. A larger dataset should mitigate this somewhat.
For reference these were my board selection options for flashing the ESP-EYE in the Arduino IDE. Key bits being the PSRAM enabled
Let me know how you get on, hope this helps
Hello Jim,
Thank you for your testing and supporting. I will compare the board selection options. Thank you.
I have created lot of datasets in the Edge Impulse and I also tried settings such as grey-scale and RGB and combinations of these to solve the issue.
(You may not see my best setting only my last trial. I do not remember how I left there earlier but that does not matter now.)
Could you give me name of the dataset which is working at you, please?
I would like to follow your settings exactly to be on the same page.
How many objects were detected correctly or detected at all?
For example from 50 images?
Yes, I know it can results worse results when I train with mobile phone images but I use the ESP-EYE or the ESP32-CAM for inference and it would be the ideal to use the same camera for training and inference. This is the natural method.
But it is also not so good that the image quality of the training set created by ESP-EYE was bad when I used. Therefore I try out more possibilities.
Additionally when I used the option “Connect your device or development board” to take photos with EI then quality of the training images were poor (size were also low).
See my ObjDetnRedBlackESPImages dataset, please.
(I have now a new idea for this namely I create the images with ESP-EYE and I upload it to EI manually. This is also a plan which I would like to try out later.)
But now the most important that I can see the same results as you where it works. 
Best regards,
Lehel
Hi Lehel,
It is worth remembering that:
Hi Jim,
Yes, it is ok that the generated SW works. It was not my question. Sorry, maybe the title of the topic is a little misleading.
My goal is “simple”:
The problem is that (see my entry from Oct 17 also, please):
You can try with ObjDetnRedBlackCylinder also if you set to RGB where there are lot of images.
I already tried out many options.
My opinion is that the results for RGB mode so bad (actually nothing) that it can not come from the image quality of the ESP. The neural nets are much more robust if it can not see the object exactly in the same light condition in the same environment which was it during the training set than it can not detect the object. It can. I also tried the detection by creating the same place and the same lighting conditions. It did not work. (I suspect the problem is, that in RGB mode the captured image will be grey-scaled transformed which is not good because the FOMO is trained on RGB images. Only idea.)
From the moving images the EI SW capture photos and anyway during the testing phase I do not move the camera. So it can not be the problem.
For me it is enough at first if you share a sample EI project example with me which proves that RGB+ESP32 works. I mean that the object detection works.
I would like to test if it can really detect the objects e.g. ca. 8 from 10.
Thank you for your support.
Best regards,
Lehel
Hi Jim,
I will try with the following training set also:
Best regards,
Lehel
Hi Lehel,
Please check your forum inbox, I have send over a short recording of my setup showing reliable detection in RGB for the ObjDetnRedBlackCylinder1 project. I will clone the ObjDetnRedBlackCylinder project and train it with RGB to verify. I hope this is helpful to debug.
We do have YoloV5 available as a Community block under Object Detection, though you may struggle to fit this onto MCU hardware and it is not available in conjunction with FOMO.
Hi Jim,
Do you mean this screenshot? Or what does the forum inbox mean?
…
I will check, thank you.
Do you mean that it detects not only that there is an object but right according to the type also? For example red-black?
With grey-scaled training it detects the objects but the type, which depends only on the colors, are not correct in the most cases. This is understandable because in a grey-scaled image for example the red and the black are very similar especially if the lighting conditions are not already good.
I agree if the color does not matter and the objects can be distinguished by shape then it works because the grey-scaled training can be used.
I can see the problem only with the color case when I have to train in RGB mode instead of grey-scaled mode.
Best regards,
Lehel
Hi Lehel,
Your forum inbox can be found up in the top right here:
There should be a private message from me with a link to a screen recording- I didn’t want to share your project details publicly.
The deployment was detecting between red-red & red-black reliably yes, I have also tested your ObjDetnRedBlackCylinder model with the full dataset by retraining it as an RGB model and it too works as expected- detecting reliably between all the cylinder types on the ESP-EYE. I’ve sent screen recordings of both via the private message. I hope this is helpful!
Many thanks,
Jim
Hi Jim,
Thank you very much for your tests. I checked your videos.
Yes, everything works correctly if you use the Launch in Browser for testing. I know it.
But build the Arduino Library and put into the Arduino IDE, please start your framework program and check the Serial Port when you show the picture for ESP-EYE. On the Serial Port you will see that it does not detect any objects in RGB case.
There you will see my issue.
That was also suspicious for me on Edge Impulse site that Launching from Browser everything is working. I used with Webcamera.
I think it runs the RGB float model and not the quantized/int version therefore everything is so perfect.
Anyway how did you connect your ESP-EYE to the Launch in Browser?
Best regards,
Lehel
Hi Lehel,
These screen recordings are not running the model in my browser, they are running on my ESP-EYE. Once you have downloaded the ESP-EYE firmware option from the Deployment page of the model:
You can then use our edge-impulse-cli tool to connect to the ESP-EYE and view the model output via your browser, this lets you see how the model is performing directly on the ESP-EYE. Once the CLI tool is installed use edge-impulse-runner --debug in your terminal and open the link that is displayed to see the model running. This runs on the ESP-EYE itself, not in your browser. The results should be the same as when building via Arduino. I am going on leave tomorrow until the start of January but will be happy to share another screen recording of the model running via Arduino in the new year.
Many thanks,
Jim
Hi Jim,
Happy New Year!
Thank you for your videos and for your description very much.
You test it differently like me. I have not used this method. I will try it.
Could you test the following method also, please:
Sometimes there are issues with Arduino IDE (too slow compiling, go Offline etc.), I do not know why and annoying. If you have such issue then you can use the Visual Studio Code also.
So without any changes in your Edge Impulse code/library it does not detect anything in RGB mode at me.
I look forward to seeing your results.
Best regards,
Lehel
@edghba I have recorded myself testing with the above settings (no changes) and I am seeing detection as expected. You’ll find the video in your private forum messages as before. Please review and see if there are any differences. Many thanks,
Jim
Hi Jim,
Thank you for your video and for retesting. According to your video you executed the same as me.
Currently, I do not know what the difference can be in my case because I do the same as you.
I check and investigate it again and I will inform you.
Best regards & thanks,
Lehel
Hi Jim,
I noticed in your video that in your generated Sketch you have: v1
#include <ObjDetnRedBlackCylinder_v1_inferencing.h>
In my case it is without v1:
#include <ObjDetnRedBlackCylinder_inferencing.h>
Why do I have this difference when we created the .zip in the same way? Can it be problem?
I retested applying exactly the same settings which you proposed on December 7, 2023 and which I saw in your video.
Best regards,
Lehel
