Running two model on an Spresence board

Question/Issue: I am currently working on a project which aim to recognize several types of animals thanks to a camera connected to the board (Sony Spresence). I would like to setup two models thanks to two different datasets of images: one with pictures taken during the night and the other with pictures taken during the day. My question is how can i generate with edge-impulse a piece of software that can switch between the two models? or if it’s not possible how can i manage (“with the hands”) to do it?

Project ID:
you can found more details about the project here: RAPPORT_FINAL_2022.md · master · WildCount / doc · GitLab
Context/Use case:

Hello @leo_cordier,

Unfortunately this is not something we support off the shelf at the moment on MCUs.

See: Running multiple (two) Edge impulse model simultaneously in a single device

Best,

Louis

@leo_cordier If your MCU CPU has multiple cores and are fast enuf, you can run different Models. Using an Arduino Portenta H7 I ran a keyphrase recognition Model on the M4 240 MHz core and a vision FOMO Model on the M7 480 MHz core. Both Models ran in continuous mode and could notify each other of various ML inferences.

For reference, the Sony Spresense has 6 ARM Cortex-M4F cores with a clock speed of up to 156 MHz.

@louis ok thanks, do you know if this functionality will be available soon?

@MMarcial Hello, the problem is that you can’t select on which core you run your program on the Sony Spresence board using the arduino IDE. And i don’t need to run the two models at the same time i just want the possibility to switch between the two models and load the one that i want.

@leo_cordier I don’t have a Sony Spresence to test with (but am willing to rx a free one from Sony so I can prove myself right or wrong) but it does seem one can target a core with the Aurduino IDE.

I would try:

  • Deploy model_1 to Subcore1 and deploy model_2 to Subcore2.
  • Maincore will read the sensor(s) and feed the desired Model based on inputs to the Maincore.

RE: Running 2 Different Vision Models on the Spresense

@leo_cordier I was able to get my hands on a Spresense and do some testing.

  1. The Sony library Camera.h will only run on the MainCore and not on any of the SubCores.
  2. The Sony library Audio.h will only run on the MainCore and not on any of the SubCores.
  3. Presently, one cannot run 2 EI Studio Models on a single core.

Since there are not many good examples on Arduino INO code running on a Spresense with an EI Studio Model, I will publish my Spresense Inferencing Image Classification Model code and my Spresense Inferencing Keyphrase Model code probably to GitHub once I get it cleaned up (there was definitely a learning curve).

Summary
The Spresense can only run one Vision Model or one Audio Model. Although it might be able to host other Models on a SubCore that say run an inference on a stream of pulses or an analog signal on an I/O pin.

By looking at the code it looks like the Accelerometer Model might run on multiple cores since it is just getting data from memory but I don’t have an accelerometer to test with.

Other Possible Solutions
I suggest perhaps the Arduino Portenta H7 since it will run a Vision Model and an Audio Model at the same time.

@Rocksetta did get a Vision Model running on the M4 of the Portenta. So now the question is can the Portenta host a Vision Model on each core at the same time.

2 Likes

Wow good question. Off the top of my head I would say no, but if the camera data was saved to the SDRAM then both cores sequentially should be able to access the data. I think it would be very slow even with FOMO but it would be interesting to test.

If I get around to it, I will post about it.