Hi all,
I am working on a project that uses an audio model and requires WiFi connectivity.
I’d love to be able to use my Portenta H7 + vision shield - but alas, can’t seem to get the Arduino audio libraries to run properly on it. (If anyone here has had success with this - please let me know! :))
What are my options? I’d love to use Arduino so that I can use the convenient library exports.
Could you use something like an ESP32? I’ve had good luck using the ESP32 boards for WiFi, but I have not used them for ML applications (yet). Alternatively, you might be able to use an ESP8266 as a co-processor for handling your WiFi communications and just use your main board for ML and communicating with the ESP8266 (e.g. through SPI).
For example, I know that the Arduino Nano 33 BLE Sense works well with Edge Impulse libraries, including audio classification. Pairing that with an ESP8266 might work for your needs.
My experience with the Portenta is similar to yours: I find that many of the libraries simply don’t work, and it’s been a very frustrating experience. It’s also a very expensive board, which only adds to that frustration.
Glad I’m not the only one a little let down by the Portenta so far…
I didn’t know whether ESP32 boards could handle EI libraries. Which one(s) do you have? In theory I would need one with a microphone - for instance the ESP32-EYE. Any chance you have one of these?
If so, if you get a chance to try to run an EI library on it, let me know how it goes!
Thanks for the reply.
Aviel
I don’t know any ESP32 boards with a built-in microphone. Personally, I would use an I2S MEMS mic to capture sound. I haven’t tried one with the ESP32 yet, but it should (in theory) work. Here is an article showing how to hook one up: https://diyi0t.com/i2s-sound-tutorial-for-esp32/
If I’m able to get the EI library working on the ESP32, I’ll definitely let you know!
Hi @avielbr,
I have been able to classify images with the ESP32-CAM, so sound classification should run on the ESP32 too.
https://github.com/edgeimpulse/example-esp32-cam
Regards,
Louis
Hi Shawn: The ESP32-EYE (espressif’s own board) has an I2S microphone on it. I have used it successfully in my own Arduino programs.It also works when I use edge impulse Deploy and pick ESP32-EYE as the target, download the generated .bin file and then issue edge-impulse-run-impulse --continuous from the command line. I don’t get very good recognition as I didn’t take that many samples- but the prediction values go from small random values to about 0.2 when I say the keywords.
But when I deploy the same project to an Arduino library, and compile/load one of the examples that gets generated, that program runs but the keyword predictions are always 0.0 and the unknown is always 1.0. I’m pretty certain that this program is not picking up the mic signal right, if at all. In my working Arduino program, the bits per sample has to be 32 to work properly (+ I have to right shift by 8 as the mic is 24-bits, I believe ). The Edge impulse Arduino examples generated using my e.i. application instead use 16-bits per sample. Also the e.i. example uses I2S port 1 where my working Arduino program uses I2S port 0 ( when used, the camera uses I2S port1 I believe).
So however e.i. got the ESP32-EYE to work with the mic. when you Deploy it as an image file does not work when they generate the Arduino source code examples for you to compile.
FYI: I’ve been a loyal D-K customer for 25 years- both at work and home (now retired) . Great company to deal with, and I like your tutorial videos!
Oops, yes it does! I forgot that the ESP-EYE has a mic on it. Thanks!
@bmillier Check out this detailed exploration of this PDM microphone issue created by user alkorisedge.
@alkorisedge perhaps write up a summary of your work and ask Shawn Hymel to add a Troubleshooting section to the ESP page.
@MMarcial, thanks for mentioning here…
I still thinking on the subject, and have, at 2 things to add there/here, on this subject of ESP32 and PDM MIC…
The FIRST thing I think is interesting is the use of Arduino IDE in Portable Mode, it is like a VIRTUAL ENVIRONMENT, making possible to have different “configs/versions” of things, in insulated places, from each other… so users will have more options to do stuff, or just for testing purposes…
The SECOND thing is a thinking that I had (from) some years ago, which I refer to as “source code pedigree, or source code birth certificate”, which is a reference to a simple and informal declaration of data about the environment/conditions that originated such and such source code, in other words, infos about the system where the code/sample was created and was running…
Sometimes, just little details can help save a lot of time, and if we all start to create code with some kind of “birth certificate” included (inline inside the code)…
This second thing/think here is just in thinking stage, I have not developed it to any level, but, many, many times I keep feeling that it may be a good habit to attain…
I will elaborate further on these 2 “things”, in fact I was thinking about placing there…
ESP32, PDM MIC AND EDGE
I have just started with ESP32 and AI, and just started here on Edge Impulse, but, so far I think ESP32 can do a lot on the subject of edge and AI…
There is probably 2 ways to think/do…
The simplest one is some kind of MONOLITHIC EDGE, where ESP32 board itself collect image, sound and do inference…
The other, also valid way I think, is to have some kind of MODULAR EDGE, where one ESP32 capture audio and send to a RaspberryPI Zero 2, and another ESP32 capture image and send to the same RaspberryPI Zero 2, which then will perform inference…
I think that a combination of these 2 “ways” can help make a more robust edge AI, in situations where the monolithic edge approach alone resulted in limitations…
(In fact I am planning to test this scenario using a Zero2 or PI3A+)
I personally will be looking more at the MODULAR EDGE approach, and in this regard, the ESP32 is a thing to consider…
As for inference inside the ESP32 (and S3), my experience is just at the beginning, so, I can’t say much about it…
It would be nice if an EdgeBench table for edge devices exist, so we can consult to see what kind of device I need for the application that I may have in mind… unfortunately I think we don’t have any guidance like this…
Regards,
Valter
@alkorisedge Regarding what hardware you should use in a particular scenario: EI recently release a hardware profiler. You can use the code here and write a loop that will cycle thru any given amount of CPUs, GPUs, and MCUs known to EI. The output of the profiler shows estimated ROM, RAM and inference times.