@janjongboom can probably answer better, but my 2 cents are:
An RGB impulse will run fine on both Nano33BleSense with a camera and the Portenta since the Portenta converts GRAYSCALE to a quasi RGB.
instead of RGB888 example
0x1a44cc the Portenta might see
Theoretically a GRAYSCALE impulse could be much smaller than an RGB impulse. So the GRAYSCALE Portenta might be able to load higher resolutions and run them faster. I have been testing bigger models, but my build typically times-out.
The problem with RGB is that I could not get bigger than 48x48 pixels with the Nano33BleSense and the OV7670 cam. And it is cropped not squished so very hard to line up the camera with what you want to see. Are you getting better than that? What camera do you have with your Nano33BleSense?