Running two model on an Spresence board

Question/Issue: I am currently working on a project which aim to recognize several types of animals thanks to a camera connected to the board (Sony Spresence). I would like to setup two models thanks to two different datasets of images: one with pictures taken during the night and the other with pictures taken during the day. My question is how can i generate with edge-impulse a piece of software that can switch between the two models? or if it’s not possible how can i manage (“with the hands”) to do it?

Project ID:
you can found more details about the project here: RAPPORT_FINAL_2022.md · master · WildCount / doc · GitLab
Context/Use case:

Hello @leo_cordier,

Unfortunately this is not something we support off the shelf at the moment on MCUs.

See: Running multiple (two) Edge impulse model simultaneously in a single device

Best,

Louis

@leo_cordier If your MCU CPU has multiple cores and are fast enuf, you can run different Models. Using an Arduino Portenta H7 I ran a keyphrase recognition Model on the M4 240 MHz core and a vision FOMO Model on the M7 480 MHz core. Both Models ran in continuous mode and could notify each other of various ML inferences.

For reference, the Sony Spresense has 6 ARM Cortex-M4F cores with a clock speed of up to 156 MHz.

@louis ok thanks, do you know if this functionality will be available soon?

@MMarcial Hello, the problem is that you can’t select on which core you run your program on the Sony Spresence board using the arduino IDE. And i don’t need to run the two models at the same time i just want the possibility to switch between the two models and load the one that i want.

@leo_cordier I don’t have a Sony Spresence to test with (but am willing to rx a free one from Sony so I can prove myself right or wrong) but it does seem one can target a core with the Aurduino IDE.

I would try:

  • Deploy model_1 to Subcore1 and deploy model_2 to Subcore2.
  • Maincore will read the sensor(s) and feed the desired Model based on inputs to the Maincore.
1 Like

RE: Running 2 Different Vision Models on the Spresense

@leo_cordier I was able to get my hands on a Spresense and do some testing.

  1. The Sony library Camera.h will only run on the MainCore and not on any of the SubCores.
  2. The Sony library Audio.h will only run on the MainCore and not on any of the SubCores.
  3. Presently, one cannot run 2 EI Studio Models on a single core.

Since there are not many good examples on Arduino INO code running on a Spresense with an EI Studio Model, I will publish my Spresense Inferencing Image Classification Model code and my Spresense Inferencing Keyphrase Model code probably to GitHub once I get it cleaned up (there was definitely a learning curve).

Summary
The Spresense can only run one Vision Model or one Audio Model. Although it might be able to host other Models on a SubCore that say run an inference on a stream of pulses or an analog signal on an I/O pin.

By looking at the code it looks like the Accelerometer Model might run on multiple cores since it is just getting data from memory but I don’t have an accelerometer to test with.

Other Possible Solutions
I suggest perhaps the Arduino Portenta H7 since it will run a Vision Model and an Audio Model at the same time.

@Rocksetta did get a Vision Model running on the M4 of the Portenta. So now the question is can the Portenta host a Vision Model on each core at the same time.

2 Likes

Wow good question. Off the top of my head I would say no, but if the camera data was saved to the SDRAM then both cores sequentially should be able to access the data. I think it would be very slow even with FOMO but it would be interesting to test.

If I get around to it, I will post about it.

Can this be done with Nicla Vision board? It has “similar” STM32H747AII6 Dual ARM® Cortex® M7/M4 IC?

It should work as long as the compiler does not put restrictions on the camera. For example a restriction might be, if the camera is being used by Core #1 then Core #2 cannot.

Hello,

I have a similar question and I am looking forward to your answer.

Since Spresense can only run the Audio.h in the main core. My question is: Is it possible to use two channels of audio input to create the model? All examples I found right now only use one channel of audio input.

Thank you in advance for any help.

Best,

Lary

@LarryWang

Yes on the Spresense you can use up to 8 mics. The readframes() is the same no matter how many mics you have. When more than 1 mic is present you need to parse the interleaved mic data. See this link.

Hi @MMarcial

Thank you for your kind reply. I checked your code here and it works fine for one channel: Sony Spresense Arduino Code I tried the continuous mode and it seems working fine (Your comment mentioned there is an issue when using continuous mode. I am not sure whether it is related to the update of base firmware) I also have no problem in using multiple mics for multiple audio inputs without the usage of EI.

I posted my question here: Is it possible to have two channels of audio input? - #6 by LarryWang and got some replies.

Based on some replies in the above post, it seems very challenging to support two channels of audio input with one model. For the processing part of audio input, I hope to find the cross-correlation between two channels, however, the replied answer mentioned two processing blocks in one model which means the processing is for each channel separately. Is it possible to find the correlation between channels for feature extraction?

My last question is: is there any difference in using Arduino code or C++ for using EI on Spresense?

Thank you in advance for any help.

Best,

Larry

For the cross-corr you’ll need to write your own Processing Block. You can start with one of these.

Do you have info/links to how you might do the cross-corr?

I don’t know for sure but I am guessing the Ardy code will include standard Ardy libraries like Wire.h, etc. The C++ library will not know anything about your hardware. As far as using the Spresense with EI start here.

Hi @MMarcial

Thank you so much for your timely reply. I am a beginner in this area so my questions may be stupid.

The reason I need two channels of audio input is that I hope to design a model to achieve the ‘Voice Activity Detection’ algorithm. If using one channel, the classification is inaccurate due to the occurrence of background noise. If using two channels, I hope to compare the MFCC features (or maybe other features) between the two channels. If it is highly similar, then the classification part should be able to determine this. If I hope to design this function, it is possible? Does it mean that I also need a customized classification part?

Another question is : thub.com/edgeimpulse/firmware-sony-spresense our firmware for the Sony Spresense board.
Since dual model/dual channel is not supported out of the box, you can start modifying the base firmware for the Sony Spresense, maybe starting with adding support for dual audio channel (here the audio handling firmware-sony-spresense/ei_microphone.cpp at main · edgeimpulse/firmware-sony-spresense · GitHub ).

Do you have any ideas on how difficult it is to modify the based firmware of Spresense to support two channels? I never modified a base firmware before.

Thank you!

Larry

To compare Features you wil need a Custom Processing Block.

The EI code does not use the Sony AudioClass library so you will be getting deep into the inner-workings of how to interact with the Sony Spresense microphones.

1 Like