@aurel thanks let me try it
I confirmed its using 48x48. This is the error iam getting in serial monitor!
Converting to RGB888...
Done in 162ms
Resizing the frame buffer...
Done in 20ms
Getting signal...
Run classifier...
ERR: failed to allocate tensor arena
Failed to allocate TFLite arena (error code 1)
run_classifier returned: -6
JPG: 1872B 241ms
{"success": false,"timing": {"timing_dsp":0,"timing_classification":0,"timing_anomaly":0}}
{"success": false,"timing": {"timing_dsp":0,"timing_classification":0,"timing_anomaly":0}}This is a memory issue, could you try retraining your model with a 96x96 image and chosing a MobileNetV1 (they have a smaller footprint).
Regards,
Louis
I can see on your account that you have two projects.
One name test and another one call test2.
Your test2 model is too big (96x96 images with MobileNetv2 0.05) but your test project should be fine I think. Can you make sure you are using the right project?
Also can you make sure you selected the quantized version (int8) of the model on the deployment tab please?
Regards,
Louis
I am trying to detect a currency with Esp32-Cam. For this i follow the tutorial of EDGE IMPLUSE YouTube video which show how to detect something with Esp32-Cam.
First of take some picture and trained the as follows.
And get ACCURACY 100.0%
I can also classify all data in your website but not in my Esp-32 cam,
Then download the library and include it in my Arduino IDE. After that i use example code of your github by changing this
// Modify the following line according to your project name
// Do not forget to import the library using "Sketch">"Include Library">"Add .ZIP Library..."
#include <esp32Cam10taaka_inferencing.h>
I have this error in serial monitor
And in Edge Impulse on ESP32 Cam browser:
{
"success": false,
"timing": {
"timing_dsp": 0,
"timing_classification": 0,
"timing_anomaly": 0
}
}
What’s going wrong?
@muhit313 See the line that starts with ERR: - you’re running out of memory here. As @louis mentioned above try:
This is a memory issue, could you try retraining your model with a 96x96 image and chosing a MobileNetV1 (they have a smaller footprint).
2 Likes
I follow this again… But no luck !
Same result as before.
Is it caused for clone Esp32-Cam?
My device is not original Ai-thinker. May be its Chinese clone (Not Sure).
Hello @muhit313
So I took some time today to have a deeper look at the ESP recurring issues.
I adapted one arduino sketch from this link to include Edge Impulse inference: https://randomnerdtutorials.com/esp32-cam-take-photo-save-microsd-card/
Note that you can view the orginal file if you insert a fat32 formated SD Card in the ESP32 Cam.
Here is the code (I tried with your project model to make sure it worked):
You’ll need to modify the PIN definition for your esp camera model but I’m sure you’ll be able to find out 
#include "esp_camera.h"
#include "esp_timer.h"
#include "Arduino.h"
#include "img_converters.h"
#include "image_util.h"
#include "FS.h" // SD Card ESP32
#include "SD_MMC.h" // SD Card ESP32
#include "soc/soc.h" // Disable brownour problems
#include "soc/rtc_cntl_reg.h" // Disable brownour problems
#include "driver/rtc_io.h"
#include <EEPROM.h> // read and write from flash memory
// Modify the following line according to your project name
// Do not forget to import the library using "Sketch">"Include Library">"Add .ZIP Library..."
#include <esp32Cam10taaka_inferencing.h>
// define the number of bytes you want to access
#define EEPROM_SIZE 1
// Pin definition for CAMERA_MODEL_AI_THINKER
#define PWDN_GPIO_NUM 32
#define RESET_GPIO_NUM -1
#define XCLK_GPIO_NUM 0
#define SIOD_GPIO_NUM 26
#define SIOC_GPIO_NUM 27
#define Y9_GPIO_NUM 35
#define Y8_GPIO_NUM 34
#define Y7_GPIO_NUM 39
#define Y6_GPIO_NUM 36
#define Y5_GPIO_NUM 21
#define Y4_GPIO_NUM 19
#define Y3_GPIO_NUM 18
#define Y2_GPIO_NUM 5
#define VSYNC_GPIO_NUM 25
#define HREF_GPIO_NUM 23
#define PCLK_GPIO_NUM 22
int pictureNumber = 0;
uint8_t *ei_buf;
ei_impulse_result_t result = {0};
bool sd_card = true;
void setup() {
WRITE_PERI_REG(RTC_CNTL_BROWN_OUT_REG, 0); //disable brownout detector
Serial.begin(115200);
//Serial.setDebugOutput(true);
//Serial.println();
camera_config_t config;
config.ledc_channel = LEDC_CHANNEL_0;
config.ledc_timer = LEDC_TIMER_0;
config.pin_d0 = Y2_GPIO_NUM;
config.pin_d1 = Y3_GPIO_NUM;
config.pin_d2 = Y4_GPIO_NUM;
config.pin_d3 = Y5_GPIO_NUM;
config.pin_d4 = Y6_GPIO_NUM;
config.pin_d5 = Y7_GPIO_NUM;
config.pin_d6 = Y8_GPIO_NUM;
config.pin_d7 = Y9_GPIO_NUM;
config.pin_xclk = XCLK_GPIO_NUM;
config.pin_pclk = PCLK_GPIO_NUM;
config.pin_vsync = VSYNC_GPIO_NUM;
config.pin_href = HREF_GPIO_NUM;
config.pin_sscb_sda = SIOD_GPIO_NUM;
config.pin_sscb_scl = SIOC_GPIO_NUM;
config.pin_pwdn = PWDN_GPIO_NUM;
config.pin_reset = RESET_GPIO_NUM;
config.xclk_freq_hz = 20000000;
config.pixel_format = PIXFORMAT_JPEG;
if (psramFound()) {
config.frame_size = FRAMESIZE_QVGA; // FRAMESIZE_ + QVGA|CIF|VGA|SVGA|XGA|SXGA|UXGA
config.jpeg_quality = 10;
config.fb_count = 2;
} else {
config.frame_size = FRAMESIZE_QVGA;
config.jpeg_quality = 12;
config.fb_count = 1;
}
// Init Camera
esp_err_t err = esp_camera_init(&config);
if (err != ESP_OK) {
Serial.printf("Camera init failed with error 0x%x", err);
return;
}
//Serial.println("Starting SD Card");
if (!SD_MMC.begin()) {
Serial.println("SD Card Mount Failed");
sd_card = false;
}
uint8_t cardType = SD_MMC.cardType();
if (cardType == CARD_NONE) {
Serial.println("No SD Card attached");
sd_card = false;
}
}
void take_picture()
{
camera_fb_t * fb = NULL;
uint8_t *rgb888_buf = NULL;
size_t out_len;
bool s;
// Take Picture ----------------------------------
int64_t time_start = esp_timer_get_time();
fb = esp_camera_fb_get();
if (!fb) {
Serial.println("Camera capture failed");
return;
}
int64_t time_end = esp_timer_get_time();
Serial.printf("JPG: %uB %ums\n", (uint32_t)(fb->len), (uint32_t)((time_end - time_start) / 1000));
//Save Orginal Picture ------------------
if (sd_card) {
// initialize EEPROM with predefined size
EEPROM.begin(EEPROM_SIZE);
pictureNumber = EEPROM.read(0) + 1;
// Path where new picture will be saved in SD Card
String path = "/original" + String(pictureNumber) + ".jpg";
fs::FS &fs = SD_MMC;
Serial.printf("Picture file name: %s\n", path.c_str());
File file = fs.open(path.c_str(), FILE_WRITE);
if (!file) {
Serial.println("Failed to open file in writing mode");
}
else {
file.write(fb->buf, fb->len); // payload (image), payload length
Serial.printf("Saved file to path: %s\n", path.c_str());
EEPROM.write(0, pictureNumber);
EEPROM.commit();
}
file.close();
}
// Process the image before inferencing --------------------
// Allocate ei_matrix buffer
dl_matrix3du_t *rgb888_matrix = dl_matrix3du_alloc(1, fb->width, fb->height, 3);
//Resize Image
rgb888_buf = rgb888_matrix->item;
out_len = EI_CLASSIFIER_INPUT_WIDTH * EI_CLASSIFIER_INPUT_HEIGHT * 3;
// Convert frame to RGB888
Serial.println("Converting to RGB888...");
time_start = esp_timer_get_time();
s = fmt2rgb888(fb->buf, fb->len, fb->format, rgb888_buf);
if (!s)
{
dl_matrix3du_free(rgb888_matrix);
Serial.println("to rgb888 failed");
return;
}
time_end = esp_timer_get_time();
Serial.printf("Done in %ums\n", (uint32_t)((time_end - time_start) / 1000));
Serial.println("Resizing the frame buffer...");
dl_matrix3du_t *resized_matrix = dl_matrix3du_alloc(1, EI_CLASSIFIER_INPUT_WIDTH, EI_CLASSIFIER_INPUT_HEIGHT, 3);
ei_buf = resized_matrix->item;
time_start = esp_timer_get_time();
image_resize_linear(ei_buf, rgb888_buf, EI_CLASSIFIER_INPUT_WIDTH, EI_CLASSIFIER_INPUT_HEIGHT, 3, fb->width, fb->height);
time_end = esp_timer_get_time();
Serial.printf("Done in %ums\n", (uint32_t)((time_end - time_start) / 1000));
dl_matrix3du_free(rgb888_matrix);
dl_matrix3du_free(resized_matrix);
esp_camera_fb_return(fb);
// Run inference ---------------------
classify();
// Turns off the ESP32-CAM white on-board LED (flash) connected to GPIO 4
pinMode(4, OUTPUT);
digitalWrite(4, LOW);
rtc_gpio_hold_en(GPIO_NUM_4);
delay(2000);
Serial.println("Going to sleep now");
delay(2000);
esp_deep_sleep_start();
Serial.println("This will never be printed");
}
int raw_feature_get_data(size_t offset, size_t length, float *signal_ptr)
{
memcpy(signal_ptr, ei_buf + offset, length * sizeof(float));
return 0;
}
void classify()
{
Serial.println("Getting signal...");
// Set up pointer to look after data, crop it and convert it to RGB888
signal_t signal;
signal.total_length = EI_CLASSIFIER_INPUT_WIDTH * EI_CLASSIFIER_INPUT_WIDTH;
signal.get_data = &raw_feature_get_data;
Serial.println("Run classifier...");
// Feed signal to the classifier
EI_IMPULSE_ERROR res = run_classifier(&signal, &result, false /* debug */);
// Returned error variable "res" while data object.array in "result"
ei_printf("run_classifier returned: %d\n", res);
if (res != 0)
return;
// print the predictions
ei_printf("Predictions (DSP: %d ms., Classification: %d ms., Anomaly: %d ms.): \n",
result.timing.dsp, result.timing.classification, result.timing.anomaly);
for (size_t ix = 0; ix < EI_CLASSIFIER_LABEL_COUNT; ix++) {
ei_printf(" %s: \t%f\r\n", result.classification[ix].label, result.classification[ix].value);
}
#if EI_CLASSIFIER_HAS_ANOMALY == 1
ei_printf(" anomaly score: %f\r\n", result.anomaly);
#endif
}
void loop() {
take_picture();
}
And as a return in the serial console you should see something like this:
Picture file name: /original1.jpg
Saved file to path: /original1.jpg
Converting to RGB888...
Done in 100ms
Resizing the frame buffer...
Done in 55ms
Getting signal...
Run classifier...
run_classifier returned: 0
Predictions (DSP: 4 ms., Classification: 583 ms., Anomaly: 0 ms.):
10taka: 0.367188
unknown: 0.632812
Going to sleep now
If this works for you I’ll update the Github repository to replace the basic example with this one.
Loading a web interface over wifi might be too greedy on ressources for some use cases.
Regards,
Louis
1 Like
First of all i wants to thank you @louis and your friend/team member @janjongboom .
Because you and your team have a good mentality to help other people like me who are not so much professional.
Also thanks for having a deeper look at the ESP recurring issues. 
Yes, it is work. Means after upload the code i am able take picture and save it into SD. And give some result.
But the Predictions result is fixed for all picture.
10taka: 0.367188 unknown: 0.632812
But it should give me different Predictions value for deferent picture.
Am I right ?
If yes what is the problem now?
Note: As I am a beginner not a professional it may take to much time to find out the problem. But I try to my best.
If you have any idea please share with me.
So there was an issue with the raw_feature_get_data function. I also cleaned the code so it is easier to read and to understand even for non-professional And most important, I removed duplicated buffers and free the memory when needed to give as much space as possible for the edge-impulse models.
Here is the full code:
#include "esp_camera.h"
#include "esp_timer.h"
#include "Arduino.h"
#include "img_converters.h"
#include "image_util.h"
#include "FS.h" // SD Card ESP32
#include "SD_MMC.h" // SD Card ESP32
#include "soc/soc.h" // Disable brownour problems
#include "soc/rtc_cntl_reg.h" // Disable brownour problems
#include "driver/rtc_io.h"
#include <EEPROM.h> // read and write from flash memory
/* Modify the following line according to your project name
Do not forget to import the library using "Sketch">"Include Library">"Add .ZIP Library..."
*/
#include <Car_Detection_inferencing.h>
// define the number of bytes you want to access
#define EEPROM_SIZE 1
// Pin definition for CAMERA_MODEL_AI_THINKER
#define PWDN_GPIO_NUM 32
#define RESET_GPIO_NUM -1
#define XCLK_GPIO_NUM 0
#define SIOD_GPIO_NUM 26
#define SIOC_GPIO_NUM 27
#define Y9_GPIO_NUM 35
#define Y8_GPIO_NUM 34
#define Y7_GPIO_NUM 39
#define Y6_GPIO_NUM 36
#define Y5_GPIO_NUM 21
#define Y4_GPIO_NUM 19
#define Y3_GPIO_NUM 18
#define Y2_GPIO_NUM 5
#define VSYNC_GPIO_NUM 25
#define HREF_GPIO_NUM 23
#define PCLK_GPIO_NUM 22
int pictureNumber = 0;
dl_matrix3du_t *resized_matrix = NULL;
//size_t ei_len = 0;
size_t out_len = EI_CLASSIFIER_INPUT_WIDTH * EI_CLASSIFIER_INPUT_HEIGHT;
ei_impulse_result_t result = {0};
bool sd_card = true;
void setup() {
WRITE_PERI_REG(RTC_CNTL_BROWN_OUT_REG, 0); //disable brownout detector
Serial.begin(115200);
camera_config_t config;
config.ledc_channel = LEDC_CHANNEL_0;
config.ledc_timer = LEDC_TIMER_0;
config.pin_d0 = Y2_GPIO_NUM;
config.pin_d1 = Y3_GPIO_NUM;
config.pin_d2 = Y4_GPIO_NUM;
config.pin_d3 = Y5_GPIO_NUM;
config.pin_d4 = Y6_GPIO_NUM;
config.pin_d5 = Y7_GPIO_NUM;
config.pin_d6 = Y8_GPIO_NUM;
config.pin_d7 = Y9_GPIO_NUM;
config.pin_xclk = XCLK_GPIO_NUM;
config.pin_pclk = PCLK_GPIO_NUM;
config.pin_vsync = VSYNC_GPIO_NUM;
config.pin_href = HREF_GPIO_NUM;
config.pin_sscb_sda = SIOD_GPIO_NUM;
config.pin_sscb_scl = SIOC_GPIO_NUM;
config.pin_pwdn = PWDN_GPIO_NUM;
config.pin_reset = RESET_GPIO_NUM;
config.xclk_freq_hz = 20000000;
config.pixel_format = PIXFORMAT_JPEG;
if (psramFound()) {
config.frame_size = FRAMESIZE_QVGA; // FRAMESIZE_ + QVGA|CIF|VGA|SVGA|XGA|SXGA|UXGA
config.jpeg_quality = 10;
config.fb_count = 2;
} else {
config.frame_size = FRAMESIZE_QVGA;
config.jpeg_quality = 12;
config.fb_count = 1;
}
// Init Camera
esp_err_t err = esp_camera_init(&config);
if (err != ESP_OK) {
Serial.printf("Camera init failed with error 0x%x", err);
return;
}
//Serial.println("Starting SD Card");
if (!SD_MMC.begin()) {
Serial.println("SD Card Mount Failed");
sd_card = false;
}
uint8_t cardType = SD_MMC.cardType();
if (cardType == CARD_NONE) {
Serial.println("No SD Card attached");
sd_card = false;
}
}
void take_picture()
{
camera_fb_t * fb = NULL;
bool s;
// --- Taking picture ---
int64_t time_start = esp_timer_get_time();
fb = esp_camera_fb_get();
if (!fb) {
Serial.println("Camera capture failed");
return;
}
int64_t time_end = esp_timer_get_time();
Serial.printf("JPG: %uB %ums\n", (uint32_t)(fb->len), (uint32_t)((time_end - time_start) / 1000));
// --- Save Orginal Picture ---
if (sd_card) {
// initialize EEPROM with predefined size
EEPROM.begin(EEPROM_SIZE);
pictureNumber = EEPROM.read(0) + 1;
// Path where new picture will be saved in SD Card
String path = "/" + String(pictureNumber) + "_original" + ".jpg";
fs::FS &fs = SD_MMC;
Serial.printf("Picture file name: %s\n", path.c_str());
File file = fs.open(path.c_str(), FILE_WRITE);
if (!file) {
Serial.println("Failed to open file in writing mode");
}
else {
file.write(fb->buf, fb->len); // payload (image), payload length
Serial.printf("Saved file to path: %s\n", path.c_str());
EEPROM.write(0, pictureNumber);
EEPROM.commit();
}
file.close();
}
// --- Convert frame to RGB888 ---
Serial.println("Converting to RGB888...");
time_start = esp_timer_get_time();
// Allocate rgb888_matrix buffer
dl_matrix3du_t *rgb888_matrix = dl_matrix3du_alloc(1, fb->width, fb->height, 3);
s = fmt2rgb888(fb->buf, fb->len, fb->format, rgb888_matrix->item);
time_end = esp_timer_get_time();
Serial.printf("Done in %ums\n", (uint32_t)((time_end - time_start) / 1000));
// --- Resize the RGB888 frame to 96x96 in this example ---
Serial.println("Resizing the frame buffer...");
time_start = esp_timer_get_time();
resized_matrix = dl_matrix3du_alloc(1, EI_CLASSIFIER_INPUT_WIDTH, EI_CLASSIFIER_INPUT_HEIGHT, 3);
image_resize_linear(resized_matrix->item, rgb888_matrix->item, EI_CLASSIFIER_INPUT_WIDTH, EI_CLASSIFIER_INPUT_HEIGHT, 3, fb->width, fb->height);
time_end = esp_timer_get_time();
Serial.printf("Done in %ums\n", (uint32_t)((time_end - time_start) / 1000));
// --- Free memory ---
dl_matrix3du_free(rgb888_matrix);
esp_camera_fb_return(fb);
// --- Call classifier function ---
classify();
// --- Convert back the resized RGB888 frame to JPG to save on SD card ---
Serial.println("Converting ei_buf to JPG...");
time_start = esp_timer_get_time();
// Allocate out_matrix buffer
dl_matrix3du_t *jpg_out_matrix = dl_matrix3du_alloc(1, fb->width, fb->height, 3);
s = fmt2jpg(resized_matrix->item, out_len, EI_CLASSIFIER_INPUT_WIDTH, EI_CLASSIFIER_INPUT_HEIGHT, PIXFORMAT_RGB888, 10, &jpg_out_matrix->item, &out_len);
time_end = esp_timer_get_time();
Serial.printf("Done in %ums\n", (uint32_t)((time_end - time_start) / 1000));
// --- Free memory ---
dl_matrix3du_free(resized_matrix);
//--- Save resized Picture ---
if (sd_card) {
// Path where new picture will be saved in SD Card
String path = "/" + String(pictureNumber) + "_resized" + ".jpg";
fs::FS &fs = SD_MMC;
Serial.printf("Picture file name: %s\n", path.c_str());
File file = fs.open(path.c_str(), FILE_WRITE);
if (!file) {
Serial.println("Failed to open file in writing mode");
}
else {
file.write(jpg_out_matrix->item, out_len); // payload (image), payload length
Serial.printf("Saved file to path: %s\n", path.c_str());
}
file.close();
}
// --- Turns off the ESP32-CAM white on-board LED (flash) connected to GPIO 4 ---
pinMode(4, OUTPUT);
digitalWrite(4, LOW);
rtc_gpio_hold_en(GPIO_NUM_4);
delay(2000);
// --- Deep sleep ---
Serial.println("Going to sleep now");
delay(2000);
esp_deep_sleep_start();
Serial.println("This will never be printed");
}
int raw_feature_get_data(size_t offset, size_t out_len, float *signal_ptr)
{
size_t pixel_ix = offset * 3;
size_t bytes_left = out_len;
size_t out_ptr_ix = 0;
// read byte for byte
while (bytes_left != 0) {
// grab the values and convert to r/g/b
uint8_t r, g, b;
r = resized_matrix->item[pixel_ix];
g = resized_matrix->item[pixel_ix + 1];
b = resized_matrix->item[pixel_ix + 2];
// then convert to out_ptr format
float pixel_f = (r << 16) + (g << 8) + b;
signal_ptr[out_ptr_ix] = pixel_f;
// and go to the next pixel
out_ptr_ix++;
pixel_ix += 3;
bytes_left--;
}
return 0;
}
void classify()
{
Serial.println("Getting signal...");
signal_t signal;
signal.total_length = EI_CLASSIFIER_INPUT_WIDTH * EI_CLASSIFIER_INPUT_WIDTH;
signal.get_data = &raw_feature_get_data;
Serial.println("Run classifier...");
// Feed signal to the classifier
EI_IMPULSE_ERROR res = run_classifier(&signal, &result, false /* debug */);
// Returned error variable "res" while data object.array in "result"
ei_printf("run_classifier returned: %d\n", res);
if (res != 0)
return;
// print the predictions
ei_printf("Predictions (DSP: %d ms., Classification: %d ms., Anomaly: %d ms.): \n",
result.timing.dsp, result.timing.classification, result.timing.anomaly);
for (size_t ix = 0; ix < EI_CLASSIFIER_LABEL_COUNT; ix++) {
ei_printf(" %s: \t%f\r\n", result.classification[ix].label, result.classification[ix].value);
}
#if EI_CLASSIFIER_HAS_ANOMALY == 1
ei_printf(" anomaly score: %f\r\n", result.anomaly);
#endif
}
void loop() {
take_picture();
}
Let me know if it works.
Note that I tried on this project and it worked well: https://studio.edgeimpulse.com/public/28056/latest
Regards,
Louis
1 Like
Many thanks for your help. This one and also the previous one is working now. But I think I need to add huge data for more accuracy.
1 Like
Hey I followed the tutorial as well as played with the code. After a few hours of playing around, I just cant seem to get anything working. I think there has been so many updates to the libraries that the code is now outdated. Is there any way I could receive some help getting this going on a ESP32 Cam? I have some projects I would love to integrate Edge into.
Hello @Galbraithmedia,
The ESP-EYE is now officially supported directly by Edge Impulse. You can also use other ESP32 boards by changing the pin headers for the camera for example): On your Espressif ESP-EYE (ESP32) development board - Edge Impulse Documentation
Good new is that we take advantage of the ESP-NN hardware acceleration. However, you won’t be able to use that with Arduino IDE, you will need to use ESP-IDF to compile your code (ESP32 boards on Arduino IDE don’t support the neural network hardware acceleration).
Regards,
Louis
Great thanks! I ended up buying an ESP32 eye. Quick question with it, I have it all set up and its taking a photo every 2 seconds to analyze the object. Is there a way I can see what the camera sees while it tracks the object?
Hello @Galbraithmedia,
I believe you can use the --debug flag when you run the edge impulse daemon.
edge-impulse-daemon --debug
But I have to admit that I haven’t tested it on the esp32 
Regards,
Louis
hello @louis , I’m a newbie about this. I want to know, can the esp32cam do inferencing every 2 or 3 seconds?
Hello @Rigraphic,
Yes it can! I can have something around 600ms on 64x64 or 96x96 images.
Best,
Louis
