Deploy Edge-Impulse model to ESP-IDF ESP32s3

Question/Issue:
[Describe the question or issue in detail]
The build fails when trying to deploy Edge-impulse model to ESP32s3 based custom PCB.
Project ID:
[793647]

Context/Use case:
Use case is using Microphone to sample and train data and then deploy the model

Summary:
[Provide a concise summary of the bug]
I have copied the standalone application template for esp32 idf environment. The issue that I am running into is during the build process of the application.
The project fails to build
C:/Users/ujjwa/try_mic_espc6/edge-impulse-sdk/dsp/numpy_types.h:693:5: error: expected specifier-qualifier-list before ‘std’
693 | std::function<int(size_t offset, size_t length, float *out_ptr)> get_data;
| ^~~
In file included from C:/Users/ujjwa/try_mic_espc6/edge-impulse-sdk/classifier/ei_model_types.h:41:
C:/Users/ujjwa/try_mic_espc6/edge-impulse-sdk/dsp/ei_dsp_handle.h:38:1: error: unknown type name ‘class’
38 | class DspHandle {
| ^~~~~
C:/Users/ujjwa/try_mic_espc6/edge-impulse-sdk/dsp/ei_dsp_handle.h:38:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘attribute’ before ‘{’ token
38 | class DspHandle {
| ^
In file included from C:/Users/ujjwa/try_mic_espc6/edge-impulse-sdk/classifier/ei_model_types.h:42:
C:/Users/ujjwa/try_mic_espc6/edge-impulse-sdk/dsp/numpy.hpp:54:10: fatal error: cfloat: No such file or directory
54 | #include
| ^~~~~~~~
compilation terminated.
[1066/1342] Building CXX object esp-idf/main/CMakeFiles/idf_main.dir//edge-impulse-sdk/dsp/memory.cpp.obj
ninja: build stopped: subcommand failed.
Steps to Reproduce:

  1. [Step 1]
    Create a model on Edge impulse and train it. Download the keyword spotting model.
  2. [Step 2]
    Download the standalone app for ESP-IDF dev. Enviromint.
  3. [Step 3]
    Build the application code along with main() application.

Expected Results:
[Describe what you expected to happen]
Build normally
Actual Results:
[Describe what actually happened]

Reproducibility:

  • [ *] Always
  • [ ] Sometimes
  • [ ] Rarely

Environment:

  • Platform: [ESP32s3-ESP-IDF.]

  • Build Environment Details: [ ESP-IDF]

  • OS Version: [ Windows 11]

  • Edge Impulse Version (Firmware): [e.g., 1.2.3]
    To find out Edge Impulse Version:

  • Edge Impulse CLI Version: [e.g., 1.5.0]

  • Project Version: [e.g., 1.0.0]

  • Custom Blocks / Impulse Configuration: [Describe custom blocks used or impulse configuration]
    Logs/Attachments:
    [Include any logs or screenshots that may help in diagnosing the issue]

Logs/Attachments:

Additional Information:
I have a model trained on the studio. The model is keyword detection model using Microphone.

1 Like

Hi, @ujjwalrathod007 !
Please let us know more details about your environment. Use the template for that, thank you.

Environment:

  • Platform: [ESP32s3-ESP-IDF.]
  • Build Environment Details: [ ESP-IDF]
  • OS Version: [ Windows 11]
  • Edge Impulse Version (Firmware): [e.g., 1.2.3]
    To find out Edge Impulse Version:
  • Edge Impulse CLI Version: [e.g., 1.5.0]
  • Project Version: [e.g., 1.0.0]
  • Custom Blocks / Impulse Configuration: [Describe custom blocks used or impulse configuration]
    Logs/Attachments:
    [Include any logs or screenshots that may help in diagnosing the issue]
2 Likes

Hi @AIWintermuteAI

Environment:

  • Platform: [ESP32s3-ESP-IDF.]
    I am using a custom board with ESP32s3 chip. I am using ESP-IDF as build environment.
  • Build Environment Details: [ ESP-IDF]
    I am using ESP-IDF version 5.5.0
  • OS Version: [ Windows 11]
    I am using Windows-11 64-bits
  • Edge Impulse Version (Firmware): [e.g., 1.2.3]
    To find out Edge Impulse Version:
  • Edge Impulse CLI Version: [e.g., 1.5.0]
    I am not using CLI at all
  • Project Version: [e.g., 1.0.0]
  • Custom Blocks / Impulse Configuration: [Describe custom blocks used or impulse configuration]
    I am not using any custom blocks or anything similar to that. I have created an impulse online. The following image shows the configurations.
    image

Logs/Attachments:

I get the following error message when trying to build the project. The error is when trying to include the following file. It seems that the #include<functional> is not included as the file might not be treated as C++ file.

#include "edge-impulse-sdk/classifier/ei_run_classifier.h"

                 from C:/Users/ujjwa/try_mic_espc6/edge-impulse-sdk/classifier/ei_model_types.h:40,
                 from C:/Users/ujjwa/try_mic_espc6/edge-impulse-sdk/classifier/ei_run_classifier.h:38,
                 from C:/Users/ujjwa/try_mic_espc6/main/hello_world_main.c:31:
C:/Users/ujjwa/try_mic_espc6/edge-impulse-sdk/dsp/numpy_types.h:699:5: error: expected specifier-qualifier-list before 'std'
  699 |     std::function<int(size_t offset, size_t length, float *out_ptr)> get_data;
      |     ^~~
In file included from C:/Users/ujjwa/try_mic_espc6/edge-impulse-sdk/classifier/ei_model_types.h:41:
C:/Users/ujjwa/try_mic_espc6/edge-impulse-sdk/dsp/ei_dsp_handle.h:42:1: error: unknown type name 'class'
   42 | class DspHandle {
      | ^~~~~
C:/Users/ujjwa/try_mic_espc6/edge-impulse-sdk/dsp/ei_dsp_handle.h:42:17: error: expected '=', ',', ';', 'asm' or '__attribute__' before '{' token
   42 | class DspHandle {
      |                 ^
In file included from C:/Users/ujjwa/try_mic_espc6/edge-impulse-sdk/classifier/ei_model_types.h:42:
C:/Users/ujjwa/try_mic_espc6/edge-impulse-sdk/dsp/numpy.hpp:54:10: fatal error: cfloat: No such file or directory
   54 | #include <cfloat>
      |          ^~~~~~~~

If you have any idea about what might be wrong, let me know. I will try to correct the errors

1 Like

Thanks for the information, @ujjwalrathod007 !
Two thoughts:

  1. You are using ESP IDF 5.5.0, while we created the example and tested with 5.1.0 (see here GitHub - edgeimpulse/example-standalone-inferencing-espressif-esp32: Builds and runs an exported impulse locally (ESP IDF)). Probably not the source of the current issue, but something to keep in mind!
  2. More importantly you are using Windows… We test building on MacOS / Linux, since compiling C/C++ applications on Windows is … not as convenient as in MacOS / Linux. In my experience there is a lot of intricacies with setting up the right toolchain on Win - and the issue in your example is clearly with your C/C++ setup, as we have it tested every day in CI
    My advice would be to install WSL and work with ESP IDF there:
    Using WSL in Windows - - — ESP-IDF Extension for VSCode latest documentation

Ok, I have another PC with Linux and esp-idf. I will try that and let you know.

Hello,

I have tried this sample on Linux PC and I am getting some linking errors as follows. At first, I got some errors including files such as dsp_err.h but after copy paste those files I got the following errors.!

/home/ujjval/.espressif/tools/xtensa-esp-elf/esp-14.2.0_20241119/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/14.2.0/../../../../xtensa-esp-elf/bin/ld: esp-idf/main/libmain.a(main.cpp.obj):(.literal._ZN2ei3fftL10hw_r2c_fftEPKfPNS_13fft_complex_tEj+0x1c): undefined reference to `dsps_fft_w_table_fc32'
/home/ujjval/.espressif/tools/xtensa-esp-elf/esp-14.2.0_20241119/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/14.2.0/../../../../xtensa-esp-elf/bin/ld: esp-idf/main/libmain.a(main.cpp.obj):(.literal._ZN2ei3fftL10hw_r2c_fftEPKfPNS_13fft_complex_tEj+0x24): undefined reference to `dsps_fft2r_init_fc32'
/home/ujjval/.espressif/tools/xtensa-esp-elf/esp-14.2.0_20241119/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/14.2.0/../../../../xtensa-esp-elf/bin/ld: esp-idf/main/libmain.a(main.cpp.obj):(.literal._ZN2ei3fftL10hw_r2c_fftEPKfPNS_13fft_complex_tEj+0x28): undefined reference to `dsps_fft2r_fc32_ansi_'
/home/ujjval/.espressif/tools/xtensa-esp-elf/esp-14.2.0_20241119/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/14.2.0/../../../../xtensa-esp-elf/bin/ld: esp-idf/main/libmain.a(main.cpp.obj):(.literal._ZN2ei3fftL10hw_r2c_fftEPKfPNS_13fft_complex_tEj+0x2c): undefined reference to `dsps_bit_rev_fc32_ansi'
/home/ujjval/.espressif/tools/xtensa-esp-elf/esp-14.2.0_20241119/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/14.2.0/../../../../xtensa-esp-elf/bin/ld: esp-idf/main/libmain.a(main.cpp.obj): in function `_ZN2ei3fftL10hw_r2c_fftEPKfPNS_13fft_complex_tEj':
/home/ujjval/esp/esp-idf/example-standalone-inferencing-espressif-esp32/edge-impulse-sdk/dsp/dsp_engines/ei_esp_dsp.h:37:(.text._ZN2ei3fftL10hw_r2c_fftEPKfPNS_13fft_complex_tEj+0x2c): undefined reference to `dsps_fft2r_init_fc32'
/home/ujjval/.espressif/tools/xtensa-esp-elf/esp-14.2.0_20241119/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/14.2.0/../../../../xtensa-esp-elf/bin/ld: /home/ujjval/esp/esp-idf/example-standalone-inferencing-espressif-esp32/edge-impulse-sdk/dsp/dsp_engines/ei_esp_dsp.h:67:(.text._ZN2ei3fftL10hw_r2c_fftEPKfPNS_13fft_complex_tEj+0xcd): undefined reference to `dsps_fft2r_fc32_ansi_'
/home/ujjval/.espressif/tools/xtensa-esp-elf/esp-14.2.0_20241119/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/14.2.0/../../../../xtensa-esp-elf/bin/ld: /home/ujjval/esp/esp-idf/example-standalone-inferencing-espressif-esp32/edge-impulse-sdk/dsp/dsp_engines/ei_esp_dsp.h:73:(.text._ZN2ei3fftL10hw_r2c_fftEPKfPNS_13fft_complex_tEj+0xf0): undefined reference to `dsps_bit_rev_fc32_ansi'
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
ninja failed with exit code 1, output of the command is in the /home/ujjval/esp/esp-idf/example-standalone-inferencing-espressif-esp32/build/log/idf_py_stderr_output_21400 and /home/ujjval/esp/esp-idf/example-standalone-inferencing-espressif-esp32/build/log/idf_py_stdout_output_21400
1 Like

@ujjwalrathod007 ,
I fixed some issues around ESP DSP in standalone example - please do
git pull
and then
idf.py fullclean

Then build again.

1 Like

@AIWintermuteAI
Thanks,
Now I will test if it can run live inference …

Basic code compiles ok.

I tried to run the sample on the device using Live Classifications.

The results are not accurate on the device, and I may need to change my model and or training method.

On the studio it looks quite good but in reality, it is not.

So, I go to Live classification and load the sample. The result on the studio is right but after copying the raw features and running the impulse on it the result is always like the following.

The keyword “red” is most of the time. Why it is like that? Am I doing something wrong?

I tried to clean the project and build it again. I also tried to load different samples.

Timing: DSP 317 ms, inference 2 ms, anomaly 0 ms
Predictions:
  blue: 0.00781
  green: 0.02734
  red: 0.96484
Timing: DSP 317 ms, inference 2 ms, anomaly 0 ms
Predictions:
  blue: 0.00781
  green: 0.02734
  red: 0.96484

Can changing the target to download the C++ library would help? I have selected ESP-EYE.
I am using custom made board with ESP32s3 chip and stereo i2s mics.

I have made my project public for now.

@ujjwalrathod007
I think I understand what is going on there.
The ESP32 ESP IDF example is just static buffer example - you copy features from Studio and the device runs inference on those features. It is not meant for usage, only for demonstration on how to run inference on ESP32.

What you need is to collect the data (in your case sound from your microphone) and pass THAT data into run_classifier. We cannot know the exact specifics of your board, so that part is left to you.

However, there is a reference firmware for ESP32, where you can see how audio collection with I2S microphone is implemented:

Keep in mind that firmware uses ESP IDF 4.4, the code might differ from ESP IDF 5.5. Also take a look at the Arduino example for ESP32 - you can get it by downloading Arduino library.

Good luck!

I tried in different way. I just have a simple code to do the above task. But I got the following error. At first, I had tried Arduino and I remember that I got similar error message there too.
If you have idea let me know.!

ERR: frame_length (551) cannot be larger than signal's total length (4) for continuous classification
ERR: Failed to run DSP process (-1008)
ERR: Failed to run classifier (-5)
I (498) main_task: Returned from app_main()
/* Edge Impulse Espressif ESP32 Standalone Inference ESP IDF Example
 * Copyright (c) 2022 EdgeImpulse Inc.
 *
 * Permission is hereby granted, free of charge, to any person obtaining a copy
 * of this software and associated documentation files (the "Software"), to deal
 * in the Software without restriction, including without limitation the rights
 * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 * copies of the Software, and to permit persons to whom the Software is
 * furnished to do so, subject to the following conditions:
 *
 * The above copyright notice and this permission notice shall be included in
 * all copies or substantial portions of the Software.
 *
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 * SOFTWARE.
 */

/* Include ----------------------------------------------------------------- */
#include <stdio.h>

#include "edge-impulse-sdk/classifier/ei_run_classifier.h"
#include "edge-impulse-sdk/dsp/numpy.hpp"

#include "driver/gpio.h"
#include "sdkconfig.h"
#include "esp_idf_version.h"

#include "driver/i2s_pdm.h"
#include "driver/i2s_std.h"
#include "driver/gpio.h"

#define BUF_SIZE    22100
#define USE_DUPLEX  1

#define BCLK_IO     GPIO_NUM_36
#define WS_IO       GPIO_NUM_37
#define DS_IO       GPIO_NUM_35

static i2s_chan_handle_t tx_chan;
static i2s_chan_handle_t rx_chan;

#define LED_PIN GPIO_NUM_21

const float features[22050] = {
    // copy raw features here (for example from the 'Live classification' page)
};


int raw_feature_get_data(size_t offset, size_t length, float *out_ptr) {
  memcpy(out_ptr, features + offset, length * sizeof(float));
  return 0;
}

void print_inference_result(ei_impulse_result_t result) {

    // Print how long it took to perform inference
    ei_printf("Timing: DSP %d ms, inference %d ms, anomaly %d ms\r\n",
            result.timing.dsp,
            result.timing.classification,
            result.timing.anomaly);

    // Print the prediction results (object detection)
#if EI_CLASSIFIER_OBJECT_DETECTION == 1
    ei_printf("Object detection bounding boxes:\r\n");
    for (uint32_t i = 0; i < result.bounding_boxes_count; i++) {
        ei_impulse_result_bounding_box_t bb = result.bounding_boxes[i];
        if (bb.value == 0) {
            continue;
        }
        ei_printf("  %s (%f) [ x: %u, y: %u, width: %u, height: %u ]\r\n",
                bb.label,
                bb.value,
                bb.x,
                bb.y,
                bb.width,
                bb.height);
    }

    // Print the prediction results (classification)
#else
    ei_printf("Predictions:\r\n");
    for (uint16_t i = 0; i < EI_CLASSIFIER_LABEL_COUNT; i++) {
        ei_printf("  %s: ", ei_classifier_inferencing_categories[i]);
        ei_printf("%.5f\r\n", result.classification[i].value);
    }
#endif

    // Print anomaly result (if it exists)
#if EI_CLASSIFIER_HAS_ANOMALY == 1
    ei_printf("Anomaly prediction: %.3f\r\n", result.anomaly);
#endif

}

extern "C" int app_main()
{
    esp_err_t err;

    i2s_chan_config_t rx_chan_cfg = I2S_CHANNEL_DEFAULT_CONFIG(I2S_NUM_AUTO, I2S_ROLE_MASTER);
    ESP_ERROR_CHECK(i2s_new_channel(&rx_chan_cfg, NULL, &rx_chan));

    i2s_std_config_t rx_std_cfg = {
        .clk_cfg = I2S_STD_CLK_DEFAULT_CONFIG(44100),
        .slot_cfg = I2S_STD_MSB_SLOT_DEFAULT_CONFIG(I2S_DATA_BIT_WIDTH_16BIT, I2S_SLOT_MODE_STEREO),
        .gpio_cfg = {
            .bclk = BCLK_IO,
            .ws = WS_IO,
            .din = DS_IO,
            .invert_flags = {
                .mclk_inv = false,          
                .bclk_inv = false,
                .ws_inv = false,              
            },
        },
    };

    rx_std_cfg.slot_cfg.slot_mask = I2S_STD_SLOT_BOTH;

    ESP_ERROR_CHECK(i2s_channel_init_std_mode(rx_chan, &rx_std_cfg));

    float *r_buf = (float *)calloc(1, BUF_SIZE);
    assert(r_buf);
    size_t r_bytes = 0;

    ESP_ERROR_CHECK(i2s_channel_enable(rx_chan));

    ei_sleep(100);

    ei_impulse_result_t result = { nullptr };

    ei_printf("Edge Impulse standalone inferencing (Espressif ESP32)\n");

    if (sizeof(features) / sizeof(float) != EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE)
    {
        ei_printf("The size of your 'features' array is not correct. Expected %d items, but had %u\n",
                EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE, sizeof(features) / sizeof(float));
        return 1;
    }

    while (true)
    {
       
        if (i2s_channel_read(rx_chan, r_buf, BUF_SIZE, NULL, 1000) == ESP_OK) {
            printf("Read Task: i2s read %d bytes\n-----------------------------------\n", r_bytes);
            printf("[0] %f [1] %f [2] %f [3] %f\n[4] %f [5] %f [6] %f [7] %f\n\n",
                   r_buf[0], r_buf[1], r_buf[2], r_buf[3], r_buf[4], r_buf[5], r_buf[6], r_buf[7]);
            
        } else {
            printf("Read Task: i2s read failed\n");
        }
        
        signal_t features_signal;
        features_signal.total_length = sizeof(features) / sizeof(features[0]);
        numpy::signal_from_buffer(r_buf, sizeof(r_buf), &features_signal);

        run_classifier_init();

        EI_IMPULSE_ERROR res = run_classifier_continuous(&features_signal, &result, false, true);
        if (res != EI_IMPULSE_OK) {
            ei_printf("ERR: Failed to run classifier (%d)\n", res);
            return res;
        }

        print_inference_result(result);
        gpio_set_level(LED_PIN, 0);
        ei_sleep(1000);
    }    
}

There is an issue with your application code. We don’t provide forum support for debugging user applications. Try using generative AI or ask for mentorship :slight_smile:

Hi,
I am not an expert. Specially in ML. But I will try to figure it out.

BTW, I have just tried the Arduino version of the same model. And here is the output. In that I just have changed the I2S pins as per my board. I am using esp32_microphone_continuous.ino code.

ERR: Failed to run classifier (-5)

ERR: MFCC failed (-1002)

ERR: Failed to run DSP process (-1002)

ERR: Failed to run classifier (-5)

ERR: MFCC failed (-1002)

ERR: Failed to run DSP process (-1002)

ERR: Failed to run classifier (-5)

ERR: MFCC failed (-1002)

It is not related to ML, just embedded C/C++ and code comprehension.
First thing to try always is to run static_buffer example with features from one of the Studio samples. You can find static buffer example in Arduino deployment examples.
If that works correctly, you can go on making it work with your sensor.