Audio Classification with Mbed STM32 IoT Discovery Board (C++ Library)

[Martin_08]

Hi everyone,

I’m in the process of making a basic speech recognition project on the B-L475E-IOT01A board using the C++ library from Edge Impulse and porting it to Mbed.

I’ve run into a few problems though, I want my project to be running audio inference in real-time, but I’m having trouble using the C++ library for this purpose. What I’m ideally trying to do at this point is do the same thing as the Arduino Library “continuous inferencing” example but in Mbed (nothing more). When I tried to compile my first version of my code, I ran into a problem involving a missing file in “EI_IMPULSE_RUN_CLASSFIER.h”:

As far as I’m aware, this file was not included in the default C++ library ZIP file I downloaded from Edge Impulse. Curiously enough, all the C++ libraries I’ve downloaded from Edge Impulse have all had the exact same line in “EI_IMPULSE_RUN_CLASSFIER.h”, calling out ei_sampler.h, but it’s only outputting an error message this time with the audio project.

Aside from my attempt, the closest example project I could find was the firmware provided by Edge Impulse here: GitHub - edgeimpulse/firmware-st-b-l475e-iot01a: Edge Impulse firmware for the ST B-L475E-IOT01A development board

The example application in this GitHub page is amazing, but not what I needed though, as the example code contains many advanced features which might make my end project unable to run. Also, as far as I can tell, it implements RTOS threads to coordinate the execution of functions, and the code is written in such a way that many of these functions are interlinked and make references to each other, so I cannot simply remove the functions which are not needed for this project:

In this GitHub page though, I was able to find an ei_sampler.h file, although I cannot use it as it would require me to use all the other additional firmware included in the example project (the libraries listed in this file make reference to other additional libraries):

Anyways, if anyone can fill me in on where to find a simple version of the ei_sampler.h file, or ideally, an example of a continuous inference mbed OS project using a C++ library (that doesn’t use an I2S mic, as those are pretty well covered) I would be really thankful

Thanks again

[shawn_edgeimpulse]

Hi @Martin_08,

I don’t think you need the ei_sampler.h file for doing audio data (it is only required for doing time series data such as from an accelerometer).

I recommend checking out the following repositories:

For basic mbed: GitHub - edgeimpulse/example-standalone-inferencing-mbed: Builds and runs an exported impulse locally (Mbed OS)

I created a continuous keyword spotting system on the Wio Terminal. Granted, it’s using Arduino, but the concepts should be the same: I set up a timer interrupt to sample from an ADC microphone and then piped the raw data through DMA to fill a buffer (in a double buffer configuration), which was fed to the run_classifier_continuous() function. The code for this can be found here: ei-keyword-spotting/wio-terminal.ino at master · ShawnHymel/ei-keyword-spotting · GitHub

You should not need any multithreading from mbed OS to perform keyword spotting. In fact, it would likely be too slow if your tick interrupts were set to 1 ms.

Hope that helps!

[Martin_08]

Hi Shawn,

I tried your suggestion and it looks like I made a fair bit of progress, however I still currently have two issues in my code.

This is my current output:

Issue 1: The buffer says it’s being overloaded and recommends decreasing the size of EI_CLASSIFIER_SLICES_PER_MODEL_WINDOW, so I changed it to 3 instead of 4, but it still throws out the warning. I’m convinced that this warning gets thrown because of the way I declared my buffers in the code.

Because I’m trying to use the libraries that already exist for this built-in mic (found here: https://github.com/janjongboom/b-l475e-iot01a-audio-mbed), I thought I could declare the buffer size/allocation using the function that already exists (ei_microphone_inference_start) in the Edge Impulse SDK folder of the C++ library:

My assumption is that by using the function ei_microphone_inference_start() I am declaring the buffers in the same way as the Arduino sketch you recommended, but is it correct?

Issue 2: The sketch keeps printing out all the features, making it very hard to read the PuTTY output. I thought that it was happening because the debug flag was set to 1, but I’ve now changed it to 0 and it’s still printing out the whole list of features.

I’ve been able to find that the features are getting printed because the “debug” option seems to have been selected in edge-impulse-sdk/classifier/ei_run_classifier.cpp, but changing the values in my main.cpp haven’t had any effect.

Thanks in advance for your help

[shawn_edgeimpulse]

Hi @Martin_08,

It looks like DSP+Classification time is less than about 80 ms, so (in theory) you should be able to run up to about 12 slices per model window. 3 or 4 is plenty for most keyword spotting applications. The “sample buffer overrun” error is thrown whenever one side of the double buffer fills up before the next inference (run_classifier_continuous()) is called.

I’m not very familiar with how mbed handles printing to the console. My guess is that printing all of the raw features to the console is blocking the run_classifier_continuous() call. As a result, the buffer is overrun, and you get the error. I suspect that if you solve the issue with printing the features, it will solve the buffer overrun issue.

Double-check the debug_option variable before it is sent to the run_classifier_continuous() function. From what I can tell, that is the only way to force the features to be printed (the code for that is here: https://github.com/edgeimpulse/inferencing-sdk-cpp/blob/144730e047d370526ca4b7ecdc247503552f3020/classifier/ei_run_classifier.h#L267). If you’re working in C++, you can try leaving off that argument, as the debug parameter defaults to false.

[handicap]

@Martin_08
Better write ‘false’ than ‘0’ on debug, to do not print matrix features in lines :
…
static bool debug_nn = false;
…
EI_IMPULSE_ERROR r = run_classifier_continuous(&signal, &result, debug_nn = false);
…

I am fighting with it for some time, also got ‘buffor overrun’ error on my Discovery STM32f407VG.
I tried change slices and more, but nothing helps.

[Martin_08]

Hi @handicap,

If you’re still interested, I managed to get a working version of this project, located here: https://github.com/Martin-Biomed/B-L4675-IOT01A-Voice-Keyword-Recognition