ESP-EYE MEMS PDM MIC - Export Arduino Library NOT WORKING...

By looking at the following project, and related tutorial,
[Project]

[Tutorial]

I was able to build a EI project using data for “no”,“yes” and “noise”, then export as binary firmware to ESP-EYE and upload it to the device using the script…
WORKS FINE!

[Problem when exporting Arduino Library]
Then, I exported the Arduino Lib as .zip file, added it to Arduino IDE…
Compile and upload without problems…
IT DOES NOT WORK…

I was not quite sure why, and after looking around the code and comparing with some working code I do have for the MEMS PDM MIC on the ESP-EYE, I tried some modifications and then…
IT WORKS!

Just change 2 lines for the I2S initialization, as follows:

[Generated by Edge Impulse Studio. NOT working...]
.mode = (i2s_mode_t)(I2S_MODE_MASTER | I2S_MODE_RX | I2S_MODE_TX),
.channel_format = I2S_CHANNEL_FMT_ONLY_RIGHT,

[Changed to... WORKING OK.]
.mode = (i2s_mode_t)(I2S_MODE_MASTER | I2S_MODE_RX),
.channel_format = I2S_CHANNEL_FMT_ONLY_LEFT,

Below, full section of working the configuration…

  i2s_config_t i2s_config = {
      .mode = (i2s_mode_t)(I2S_MODE_MASTER | I2S_MODE_RX),
      .sample_rate = sampling_rate,
      .bits_per_sample = (i2s_bits_per_sample_t)16,
      .channel_format = I2S_CHANNEL_FMT_ONLY_LEFT,
      .communication_format = I2S_COMM_FORMAT_I2S,
      .intr_alloc_flags = 0,
      .dma_buf_count = 8,
      .dma_buf_len = 512,
      .use_apll = false,
      .tx_desc_auto_clear = false,
      .fixed_mclk = -1,  
  };

I have not done any kind of test or checking to see if the data coming from the MIC is consistent, or if the changed configuration is technicaly correct, but… IT IS WORKING and detecting correctly the “no” and “yes”, as well as background noise…

Arduino IDE 1.8.19
esp32 version 2.0.8
Machine: Debian Bullseye 64Bit GNU/Linux

Not sure if we can ask here to have somebody from Edge Impulse to have a look at the code that generates the Binary Firmware (for the ESP-EYE), and see how it is doing the I2S configuration… it may help to clarify/solve the current issue…

Hope it helps ESP-EYE users…
Regards,
Valter

2 Likes

Hi @alkorisedge,

Thank you for pointing this out. I have passed the information on to our engineering team to take a look at it.

1 Like

Hello!
The code that gets compiled into binary is here

Now for channel format, I tested it just now with ESP-EYE I have here (ESP32-EYE v2.1, official board from Espressif) and I2S_CHANNEL_FMT_ONLY_RIGHT works properly, while changing it tot I2S_CHANNEL_FMT_ONLY_LEFT results in inference result being a single class (e.g. for yes-no-unknown keyword spotting, it would always show ‘yes’ with constant probability).
There might be hardware or software discrepancies: I’m using ESP32 Arduino core v2.0.5 - we also tested and developed with 2.0.4.

1 Like

Hi @AIWintermuteAI,
thanks for the input…

I have not yet used the version 2 of the Arduino IDE… I will take a look at it… it is probably something related with different versions of stuff…

I can confirm that the board I have is the Official Expressif ESP-EYE, v2.1

I will take a better look at the issue, it is worth if we can achieve a clear idea about it…

I will report here anything new I can find…

Regards,
Valter

Not the Arduino IDE 2 - I’m referring to the core version

1 Like

… in time… thanks… my fault…
I will check and post any findings…

Valter

Hi, just tried the core 2.0.5 and, YES, it works fine with the “ONLY_RIGHT”, as generated by Edge Impulse Studio…

“ONLY_LEFT”, on the other hand fail…

Compared to core 2.0.8, the behavior is THE OPPOSITE!
as described below…

*NN Functions was disabled.

Arduino Core 2.0.5

[.channel_format = I2S_CHANNEL_FMT_ONLY_LEFT] core 2.0.5
FAIL. Nothing happens…
00:29:05.022 → Predictions (DSP: 87 ms., Classification: 9 ms., Anomaly: 0 ms.):
00:29:05.055 → no: 0.000000
00:29:05.055 → noise: 0.000000
00:29:05.055 → yes: 0.996094
00:29:06.049 → Predictions (DSP: 87 ms., Classification: 9 ms., Anomaly: 0 ms.):
00:29:06.082 → no: 0.000000
00:29:06.082 → noise: 0.000000
00:29:06.082 → yes: 0.996094
00:29:07.010 → Predictions (DSP: 87 ms., Classification: 9 ms., Anomaly: 0 ms.):
00:29:07.043 → no: 0.000000
00:29:07.043 → noise: 0.000000
00:29:07.043 → yes: 0.996094

[.channel_format = I2S_CHANNEL_FMT_ONLY_RIGHT] core 2.0.5
Works OK.

Arduino Core 2.0.8

[.channel_format = I2S_CHANNEL_FMT_ONLY_LEFT] core 2.0.8
Works OK.

[.channel_format = I2S_CHANNEL_FMT_ONLY_RIGHT] core 2.0.8
FAIL. Nothing happens…
01:30:38.198 → Predictions (DSP: 88 ms., Classification: 9 ms., Anomaly: 0 ms.):
01:30:38.198 → no: 0.000000
01:30:38.198 → noise: 0.000000
01:30:38.198 → yes: 0.996094
01:30:39.224 → Predictions (DSP: 88 ms., Classification: 9 ms., Anomaly: 0 ms.):
01:30:39.224 → no: 0.000000
01:30:39.224 → noise: 0.000000
01:30:39.224 → yes: 0.996094
01:30:40.250 → Predictions (DSP: 88 ms., Classification: 9 ms., Anomaly: 0 ms.):
01:30:40.250 → no: 0.000000
01:30:40.250 → noise: 0.000000
01:30:40.250 → yes: 0.996094

I will keep looking at it…
Regards,
Valter

Hi.
I just confirmed the tests I did on Debian Bullseye…
This time on a Ubuntu 22.04…

SAME RESULTS AS BEFORE!
With Core 2.0.5 the Edge Impulse Studio generated code works OK.
With Core 2.0.8 the generated code DOES NOT WORK, need to change to “left”…

With these 2 versions of the ESP32 Cores, the behavior is, again, OPPOSITE to each other… as shown below…

uname -a
Linux ubuntrain-desktop 5.19.0-46-generic #47~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Jun 21 15:35:31 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.2 LTS
Release: 22.04
Codename: jammy

Arduino IDE v1.8.19 (new download)
Running in “portable” mode

ESP32 Arduino Core 2.0.8 ---------------------------

FAIL. DONT WORK 2.0.8
.channel_format = I2S_CHANNEL_FMT_ONLY_RIGHT,

Predictions (DSP: 87 ms., Classification: 3 ms., Anomaly: 0 ms.):
no: 0.000000
noise: 0.000000
yes: 0.996094
Predictions (DSP: 87 ms., Classification: 3 ms., Anomaly: 0 ms.):
no: 0.000000
noise: 0.000000
yes: 0.996094
Predictions (DSP: 87 ms., Classification: 3 ms., Anomaly: 0 ms.):
no: 0.000000
noise: 0.000000
yes: 0.996094

WORK 2.0.8
.channel_format = I2S_CHANNEL_FMT_ONLY_LEFT,

ESP32 Arduino Core 2.0.5 ---------------------------

FAIL. DON’T WORK. 2.0.5
.channel_format = I2S_CHANNEL_FMT_ONLY_LEFT,

Predictions (DSP: 88 ms., Classification: 3 ms., Anomaly: 0 ms.):
no: 0.000000
noise: 0.000000
yes: 0.996094
Predictions (DSP: 88 ms., Classification: 3 ms., Anomaly: 0 ms.):
no: 0.000000
noise: 0.000000
yes: 0.996094
Predictions (DSP: 88 ms., Classification: 3 ms., Anomaly: 0 ms.):
no: 0.000000
noise: 0.000000
yes: 0.996094

WORK. 2.0.5
.channel_format = I2S_CHANNEL_FMT_ONLY_RIGHT,

Because this subject (ESP-EYE + PDM MIC) is of my interest, I will also be looking at Windows and RaspOS in more details, later…
Anyway, I think we have detected an issue here…

XIAO ESP32-S3 SENSE - related remark…
I just want to add some info here that may be of interest…
I had Arduino ESP32 Core version 2.0.8 installed on the IDE because, some weeks ago I was testing a XIAO ESP32S3 Sense, and Seeed Studio wiki page advise to install, at least, that version…

Here is Seeed wiki:

CAUTION
The on-board package for XIAO ESP32S3 requires at least version 2.0.8 to be available.

So, for people interested in ESP32, cameras and pdm mics, and happens to have both, ESP-EYE and XIAO ESP32S3 SENSE, most likely these people will have 2.0.8, or better installed on their IDE…

So, if not yet, telling users that EI Studio code was generated with the Core 2.0.5 in mind may help current/future users…

Specially, if support for the XIAO ESP32S3 Sense is on planning right now…

I will be looking more into this, and report anything interesting…
Regards,
Valter

Hi, I still on Ubuntu here…
Switched to a simple capture/show(plotter) sketch to try more options quickly…

This is a preliminary post, I will update it later… but I just found that if you change:

from (i2s_port_t)1
to (i2s_port_t)0

Both, 2.0.5 and 2.0.8 cores work with the “ONLY_RIGHT” option without change!!!

Again, just a preliminary finding… I need more time to look in details and also test it on the EI AI code, then I will report here…

Not sure if PORT 0 is an option for the EI AI code, or even if it is good to use… but look like it
may be useful info to have…

Regards,
Valter

1 Like

Both, 2.0.5 and 2.0.8 cores work with the “ONLY_RIGHT” option without change!!!

WRONG!
My statement on the previous post is WRONG!
I CHECKED AND RECHECKED, both with a simple “read-mic-plotter-serial” and the Edge Impulse Studio generated code… and, the statement above is NOT CORRECT!

My preliminary hallucination was a result of a DIRTY setup, running multiple IDEs at the same time, installing cores 2.0.5, 2.0.8, back-and-forth, and trying to test stuff…

DOING THE RIGHT WAY!
This time I did 3 copies of the Arduino binary, into 3 different directories, and in each one just one core version is installed (running in PORTABLE MODE).
Also, for testing purpose, only a single version of the IDE/core runs/test at a time, after the test shutdown the IDE and run another IDE/core to run tests…

Choose 3 ESP32 Arduino Cores: 2.0.5, 2.0.8 and 2.0.10 (latest).

Run everything TWICE, and the correct result follows below:

   ONLY_RIGHT      PORT 0     PORT 1   
      2.0.5          OK         OK     
      2.0.8           X          X     
     2.0.10           X          X     
                            
                            
    ONLY_LEFT      PORT 0     PORT 1   
      2.0.5           X          X     
      2.0.8          OK         OK     
     2.0.10          OK         OK

So, changing the I2S Port, DOES NOT AFFECT RESULT!

This portion of the test was done with the following code:
(Open Serial Plotter to “see” the signal)

FOR PORT 0

#include <driver/i2s.h>

#define BUFFER_SIZE 512

#define I2S_MIC_SERIAL_CLOCK      GPIO_NUM_26   //I2S_SCK
#define I2S_MIC_LEFT_RIGHT_CLOCK  GPIO_NUM_32   //I2S_WS
#define I2S_MIC_SERIAL_DATA       GPIO_NUM_33   //I2S_SDO

i2s_config_t i2s_config = {
    .mode                 = (i2s_mode_t)(I2S_MODE_MASTER | I2S_MODE_RX),
    .sample_rate          = 16000U,
    .bits_per_sample      = I2S_BITS_PER_SAMPLE_16BIT,
    .channel_format       = I2S_CHANNEL_FMT_ONLY_RIGHT,
    .communication_format = I2S_COMM_FORMAT_I2S,
    .intr_alloc_flags     = ESP_INTR_FLAG_LEVEL1,
    .dma_buf_count        = 8,
    .dma_buf_len          = 512,
    .use_apll             = false,
    .tx_desc_auto_clear   = false,
    .fixed_mclk           = 0
};

i2s_pin_config_t i2s_mic_pins = {
    .bck_io_num     = I2S_MIC_SERIAL_CLOCK,
    .ws_io_num      = I2S_MIC_LEFT_RIGHT_CLOCK,
    .data_out_num   = I2S_PIN_NO_CHANGE,
    .data_in_num    = I2S_MIC_SERIAL_DATA
};

void setup() {
  Serial.begin(115200);
  
  // start I2S PORT 0
  i2s_driver_install(I2S_NUM_0, &i2s_config, 0, NULL);
  i2s_set_pin(I2S_NUM_0, &i2s_mic_pins);
}

int16_t sample_buffer[BUFFER_SIZE];

void loop() {
  // read I2S
  size_t bytes_read = 0;
  i2s_read(I2S_NUM_0, sample_buffer, sizeof(int16_t) * BUFFER_SIZE, &bytes_read, portMAX_DELAY);
  int samples_read = bytes_read / sizeof(int16_t);
  
  for (int i = 0; i < samples_read; i++)
  {
    Serial.printf("%ld\n", sample_buffer[i]);
  }
}

.
.
FOR PORT 1

#include <driver/i2s.h>

#define BUFFER_SIZE 512

#define I2S_MIC_SERIAL_CLOCK      GPIO_NUM_26   //I2S_SCK
#define I2S_MIC_LEFT_RIGHT_CLOCK  GPIO_NUM_32   //I2S_WS
#define I2S_MIC_SERIAL_DATA       GPIO_NUM_33   //I2S_SDO

i2s_config_t i2s_config = {
    .mode                 = (i2s_mode_t)(I2S_MODE_MASTER | I2S_MODE_RX),
    .sample_rate          = 16000U,
    .bits_per_sample      = I2S_BITS_PER_SAMPLE_16BIT,
    .channel_format       = I2S_CHANNEL_FMT_ONLY_RIGHT,
    .communication_format = I2S_COMM_FORMAT_I2S,
    .intr_alloc_flags     = ESP_INTR_FLAG_LEVEL1,
    .dma_buf_count        = 8,
    .dma_buf_len          = 512,
    .use_apll             = false,
    .tx_desc_auto_clear   = false,
    .fixed_mclk           = 0
};

i2s_pin_config_t i2s_mic_pins = {
    .bck_io_num     = I2S_MIC_SERIAL_CLOCK,
    .ws_io_num      = I2S_MIC_LEFT_RIGHT_CLOCK,
    .data_out_num   = I2S_PIN_NO_CHANGE,
    .data_in_num    = I2S_MIC_SERIAL_DATA
};

void setup() {
  Serial.begin(115200);
  
  // start I2S PORT 1
  i2s_driver_install(I2S_NUM_1, &i2s_config, 0, NULL);
  i2s_set_pin(I2S_NUM_1, &i2s_mic_pins);
}

int16_t sample_buffer[BUFFER_SIZE];

void loop() {
  // read I2S
  size_t bytes_read = 0;
  i2s_read(I2S_NUM_1, sample_buffer, sizeof(int16_t) * BUFFER_SIZE, &bytes_read, portMAX_DELAY);
  int samples_read = bytes_read / sizeof(int16_t);
  
  for (int i = 0; i < samples_read; i++)
  {
    Serial.printf("%ld\n", sample_buffer[i]);
  }
}

.
.
RUNNING ON EDGE IMPULSE STUDIO GENERATED CODE
When testing using the Edge Impulse Studio generated code, the same result as above come out… so, there is NO difference is results using EI Studio code or the simple app listed above!

Tests was done on:
Ubuntu 22.04 64BIT x86
Arduino IDE 1.8.19
ESP32 Cores 2.0.5, 2.0.8, 2.0.10

I will continue to look at the issue, this time onward looking at Espressif repo and searching for similar problems, in the hope to find some clarification about this behavior…

Hope it help somehow,
regards all,
Valter

1 Like

Perhaps this issue will persist for while, if so, as users, having choices to consider is always a good idea…

Maybe, in some cases, detecting the version of the core which is running and them take different actions, is a possibility to consider… in the long term I am not sure how it play out… but, for the moment… Arduino ESP32 allows us to detect, at runtime, the version of the SDK and the version of the ESP32 Core installed… so it may be useful…

// Only works from ESP32 Core 2.0.0 and above...
// https://github.com/espressif/arduino-esp32/blob/master/cores/esp32/esp_arduino_version.h

void setup() {
  Serial.begin(115200);

  // SDK Version
  Serial.println("Using ESP object:");
  Serial.println(ESP.getSdkVersion());
   Serial.println("Using lower level function:");
  Serial.println(esp_get_idf_version());

  // ESP32 Core Version
  Serial.println(ESP_ARDUINO_VERSION_MAJOR);
  Serial.println(ESP_ARDUINO_VERSION_MINOR);
  Serial.println(ESP_ARDUINO_VERSION_PATCH);

  // See the link at top of the file...
  Serial.println(ESP_ARDUINO_VERSION >>16);
  Serial.println((ESP_ARDUINO_VERSION >>8) & 255);
  Serial.println(ESP_ARDUINO_VERSION & 255);
}

void loop() {
}

The ESP32 Core version was only implemented on version 2.x.x and above…
The SDK function works from older versions…

Regards,
Valter

1 Like

Hi:
For what it’s worth, using Arduino 1.8.19 and ESP32 core 2.0.9, the Arduino example programs for the ESP-EYE will only work if ONLY_RIGHT is selected. Looking at the ESP-EYE schematic, I don’t see how to decide which channel the on-board microphone is actually configured for. But, I’ve also used Adafruit’s I2S microphone breakout board (ID 3421) on an ESP32 and even though it is configured for the Left channel, it also needs to be configured for ONLY_RIGHT. So, I believe that Espressif’s driver routines are wrong, in this respect.
Also, I believe that ESP-EYE uses an I2S MEMS microphone- not PDM. On another of my ESP32 boards, the same example code works with my Adafruit I2S MEMS microphone (with ONLY_RIGHT, of course)
This confusion has probably gotten into the Edge Impulse code too.

2 Likes

There are 2 things I like to add, relating to the issue here, because I believe it can help a LOT:

A. Arduino IDE Portable Mode
B. Soft Pedigree/Birth Certificate Declaration

Here is the first one (A): Portable Mode.

Arduino IDE have a mode called “PORTABLE”, which is allows the user to have MANY FULL INDEPENDENT INSTALLATIONS on the same PC without having any kind of conflicts on negative effect (on each other).

Slide 1

1) Just download the normal Arduino binary package,
2) extract it,
3) and, inside the directory which Arduino was extracted, create a folder(directory) with the name “portable” (lower case).
DONE!
Arduino IDE, when executed (run) will run on PORTABLE MODE!

Slide 2

Slide 3

Slide 4

Portable Mode allows users to have many installations, each one a different version of software/libraries, so that, when there is a need to specific version of soft/lib, instead of switching back-and-forth between different soft/libs, just have a FULL INSTALL with specific soft installed, thus allowing a more reliable way to use different versions of software without having to change the setup/configuration every time… (which will lead to mistakes sooner rather than later…)

There are also, many times in which we want to TESTS things, and the way to go is to open/create another copy of the Arduino IDE (directory) and do the tests there, insulated from the main environment/setup.

In fact, the issue we have detected here (in this thread), is one instance where the Arduino IDE Portable Mode can help…
Not only it has helped to investigate the (existence of) issue, it is also a way to OVERCOME it…

This is the SECOND TIME, I am seeing similar issues on ESP32 software, and this is THE SECOND TIME that Arduino IDE Portable Mode can be very helpful to deal with these things…

Besides, THE GREATER ADVANTAGE of the Portable Mode is not to overcome problems, instead, it is a powerful tool to have, as a flexible way to “install” Arduino IDE, without having to lose any setup/config that we already have in use!

The key here is INDEPENDENT INSULATED setup/installation!

I think that this is one feature that users should be aware of, and take advantage of it… and this is the reason I am talking about it here…

Hope it help you, in many situations,
regards,
Valter

2 Likes