Object Detection on M4 core

Is there a way to run object detection on the M4 core of the Portenta H7?

Hello @ElectricDragon,

I haven’t tried but it should be supported with small images (like 64x64px) and FOMO v0.01.
You’ll probably need to do some workaround in the default example too.

@Rocksetta I know you have been playing a bit with running the inference on the M4 core in the Portenta, maybe you tried with FOMO too and can share a piece of code or how you did it?

Regards,

Louis

Okay…I’ll check it out. Also…what’s the difference between FOMO v0.01 and 0.35?

what’s the difference between FOMO v0.01 and 0.35

It’s related to the pre-trained MobileNetV2 version we are using.
0.01 and 0.35 are the alpha parameter. This parameter controls the width of the network or the weights of the pre-trained model that are being loaded.

Around 6 months ago I did not have much success with anything on the PortentaH7 M4 core. I didn’t have a MKR IMU so could not do acceleration, the PDM (microphone) did not seem to work on M4 and also the Camera had an issue on M4, as well as WiFi/Ethernet, BLE and Serial Print all have issues on M4.

Things have changed, I now have the MKR IMU (bend the RESET pin out of the way!), the camera has had some breaking (for the better) changes. So hopefully if I have time today I will try testing all 4 main models on M4.

@louis

Does this affect the memory footprint of the model on the embedded device?

@Rocksetta Let me know if you have any success running FOMO on the M4 core of the Portenta H7

I will test all 4 main edgeimpulse methods on the Portenta M4 core.

First FOMO:

@louis @ElectricDragon I am trying 64x64 MobileNetV2 0.01 FOMO and my first error is

C:\Arduino-1.8.19\portable\packages\arduino\hardware\mbed_portenta\3.1.1\libraries\Camera\src\camera.cpp: In function 'void HAL_TIM_PWM_MspInit(TIM_HandleTypeDef*)':

C:\Arduino-1.8.19\portable\packages\arduino\hardware\mbed_portenta\3.1.1\libraries\Camera\src\camera.cpp:120:27: error: 'DCMI_TIM' was not declared in this scope
     if (htim->Instance == DCMI_TIM) {

Sound PDM seems to work on both the Portenta M4 and M7 cores. Of course on the M4 core serial print does not work on the M4 and cool tricks using RPC seem to give the red flash of death, but just activating the LED’s red, green, blue: LEDR, LEDG, LEDB seems to work.

Acceleration might work on the M4 core, it compiled and while I was messing with LED’s I lost the serial port. The Portenta just flashes red once on plugin and I can’t seem to connect to it. Possibly bricked another one. :sob:

Next trying classification.

Same issue on the Portenta M4 for Classification as for FOMO

C:\Arduino-1.8.19\portable\packages\arduino\hardware\mbed_portenta\3.1.1\libraries\Camera\src\camera.cpp: In function 'void HAL_TIM_PWM_MspInit(TIM_HandleTypeDef*)':
C:\Arduino-1.8.19\portable\packages\arduino\hardware\mbed_portenta\3.1.1\libraries\Camera\src\camera.cpp:120:27: error: 'DCMI_TIM' was not declared in this scope
     if (htim->Instance == DCMI_TIM) {

@rjames any suggestions? I am going to try a few camera sketches from here on the M4 core.

I filed an issue on the Arduino core MBED site

I successfully got the Arduino example “portenta_h7_microphone_continuous” running on the M4. Debug statements are sent to the M7 via RPC with the M7 doing the Serial.println().

Yes…@rocksetta, I get the same error when I try to upload to the M4 core.

Nice @MMarcial my little RPC trick did not work on the M4 core with sound, but maybe I di something else wrong, might have another look at it.

#include "RPC.h"  // comes with the mbed board installation

#define Serial RPC  // So the M4 regular serial prints go to RPC

@ElectricDragon I havn’t tested an ML model but using my OLED I got the camera working on the PortentaH7 M4 core

My camera.cpp code is here

I cut out a function but not sure if that is a good idea, but it works. Waiting for a reply from Arduino. I will try testing it tonight on a FOMO model.

…

Sad to say my first FOMO attempt compiled but I have no proof if it worked. I will try again tomorrow.

Why are you thinking we should do a smaller model on the M4 core? I thought the PortentaH7 M4 core actually had more space than the M7?

Having said that my 96x96 model is not working so I am now trying your 64x64 model idea.

…

@louis I got your model size working! I just have to improve it a bit, but it looks like M4 is very finicky but can do a FOMO model.

link to post of FOMO working on M4 here

1 Like

I like that trick. It allows keeping one code base depending on the target application.

1 Like

@Rocksetta so would a FOMO model work if the size is 64x64? Or are there a few more steps that need to be taken?

Yeh, took a long time to get to that point, see these deprecated examples (all done 2 years ago!). I am testing if I can get RPC on the M4 working with Edgeimpulse. I assume the problem is the ie_printf() command but on closer inspection of an exported arduino library (…src\edge-impulse-sdk\porting\arduino\ei_classifier_porting.cpp) it does seem to eventually do a Serial.print()

Presently not getting anything from ei_printf() through RPC , perhaps one of the Pro’s could voice an opinion.

I would just get 64x64 FOMO working on M4 and then increase the size. I use an OLED to view my output (Really worth $12.99 USD 1.5inch OLED display Module, 128x128 pixels, 16 gray scale, SPI/I2C interface) so that is probably trashing a fair bit of my memory allocation. I should try a model with just the LED output (as serial through RPC is a pain) and see how big it can go. After 96x96 things on M7 get tricky. I would like 160x160 but that hits a memory limit however I have not tried @louis suggestion of the smaller mobileNet.

I blindly replaced all ei_printf() with equivalent RPC.print(). Also the M4 code would not run if I had RPC.print() in the Arduino setup().