Arduino PortentaH7 Lite, PortentaH7 and the Vision Shield

Thanks so much @louis, that code is fairly understandable, should be a good base, along with the two versions from Edge Impulse above.

void image_resize_linear(uint8_t *dst_image, uint8_t *src_image, int dst_w, int dst_h, int dst_c, int src_w, int src_h)

Just checking is int dst_c the number of channels meaning RGB is 3 and GRAYSCALE is 1?

Somehow I logged in differently this is still @rocksetta.

This seems very wrong. @louis, I am struggling with how to get the out_ptr into the features_signal structure. This is what I have tried so far, seems very wasteful of memory. It does compile but flashes red on loading to the Portenta.

  int myCamResult =  myCam.grab(sdram_frame_buffer); // myCamResult should be zero 

    // the features are stored into flash, and we don't want to load everything into RAM
    signal_t features_signal;
    features_signal.total_length = CUTOUT_COLS * CUTOUT_ROWS;
    float model_input_buffer[features_signal.total_length];
   
    //features_signal.get_data = &cutout_get_data;   // this activated the old code

    // somehow activate resize???
    // void image_resize_linear(uint8_t *dst_image, uint8_t *src_image, int dst_w, int dst_h, int dst_c, int src_w, int src_h)


    
    uint8_t * tmp_buffer = (uint8_t *) malloc(features_signal.total_length);

    image_resize_linear(  tmp_buffer,  sdram_frame_buffer,  96,96,    1  /* 1 = GRAYSCALE */,    320,320 );
    
// convert int_8 to float
    for (int i=0; i < features_signal.total_length; i++){
       model_input_buffer[i] = tmp_buffer[i] / 255.0f;  
    }

  

    features_signal.get_data((unsigned int)sizeof(uint8_t), (unsigned int)features_signal.total_length, model_input_buffer);
    
    free(tmp_buffer);

Hello @Rocksetta,

I just checked how I did the project using the ESP32 Cam to see if there was any differences, in fact, the way I did allocated the pointer using Espressif SDK was using this function dl_matrix3du_alloc …
This will be different with the Portenta.

Also, to answer you on this parameter: int dst_c it is the channel (so 3 for RGB). In your case, I see that you are using grayscale and from what I see in the header file (https://github.com/espressif/esp-dl/blob/420fc7e219ba98e40a5493b9d4be270db2f2d724/image_util/include/image_util.h)

/**
     * @brief Resize the image in RGB888 format via bilinear interpolation
     * 
     * @param dst_image    The output image
     * @param src_image    Source image
     * @param dst_w        Width of the output image
     * @param dst_h        Height of the output image
     * @param dst_c        Channel of the output image
     * @param src_w        Width of the source image
     * @param src_h        Height of the source image
     */
    void image_resize_linear(uint8_t *dst_image, uint8_t *src_image, int dst_w, int dst_h, int dst_c, int src_w, int src_h);

It has to be in RGB888 format to use this function… I might have misread your initial question.

If needed, here is an arduino sketch I wrote to show how it works with the ESP32 Cam + Edge Impulse

Regards,

Louis

4 Likes

@Rocksetta,

If you want to only resize (and not crop) from 320x320 to 96x96 you can try this (on the Portenta H7):
I’ve used magic numbers just for verbosity.

cam.grab(ei_camera_frame_buffer);

resizeImage(320, 320, // <-- input buf resolution
            ei_camera_frame_buffer, // <-- input buf
            96, 96, // output buf resolutions
            ei_camera_frame_buffer, // <-- output buf can be the same
            8); // <-- bits per pixel
    ei::signal_t signal;
    signal.total_length = 96 * 96; // <-- in pixels 
    signal.get_data = &ei_camera_cutout_get_data;
static inline void mono_to_rgb(uint8_t mono_data, uint8_t *r, uint8_t *g, uint8_t *b) {
    uint8_t v = mono_data;
    *r = *g = *b = v;
}

int ei_camera_cutout_get_data(size_t offset, size_t length, float *out_ptr) {
    size_t bytes_left = length;
    size_t out_ptr_ix = 0;

    // read byte for byte
    while (bytes_left != 0) {

        // grab the value and convert to r/g/b
        uint8_t pixel = ei_camera_frame_buffer[offset];

        uint8_t r, g, b;
        mono_to_rgb(pixel, &r, &g, &b);

        // then convert to out_ptr format
        float pixel_f = (r << 16) + (g << 8) + b;
        out_ptr[out_ptr_ix] = pixel_f;

        // and go to the next pixel
        out_ptr_ix++;
        offset++;
        bytes_left--;
    }

    // and done!
    return 0;
}

2 Likes

Your amazing @rjames, :100: thanks so much for the code examples. This is the first time I have ever had the Arduino Portenta using all 320x320 camera pixels resized to a 96x96 Edge Impulse model, using the SDRAM of the Portenta for the camera buffer.

Look how fast this little beast is processing.

My version of Raul’s code is here .

1 Like

That’s awesome news! :smiley:

// Raul

1 Like

Raul @rjames, any hints about testing multi-object “bounding boxes object detection” instead of what we have working for single object “one label per data” ?

I know from my work with WASM that the differences between the two types of code are very small. I am wondering if this line still grabs the result correctly



    EI_IMPULSE_ERROR res = run_classifier(&signal, &result, false /* debug */);

Also
the result structure has changed for multi-object, but I can’t find any information about the new structure other than the header file ei_classifier_types.h. I think from my WASM the calls look something like this.


result.results.length 

result.results[i].value

result.results[i].label

result.results[i].x, result.results[i].y, result.results[i].width, result.results[i].height   
    
result.anomaly

but from the header file it is probably more like:

result.bounding_boxes[i].x
result.bounding_boxes[i].y
result.bounding_boxes[i].width
result.bounding_boxes[i].height
result.bounding_boxes[i].value
result.bounding_boxes[i].label

result.timing.sampling
result.timing.dsp
result.timing.classification
result.timing.anomaly



result.anomaly





Any confirmation here. Not expecting bounding boxes to work well, just wondering if it works at all on the Portenta. It was very slow on the web browser.

@Rocksetta this won’t work until the new constrained object detection is live.