DSP does not handle grayscale image efficiently

Report bugs
#1

I am capturing single channel grayscale uint8_t data array from the image sensor. I am using quantized model and the run_classifier steps into the code (below) for DSP. Why is it expecting 1-channel data as RGB888? It is a waste of 2x memory size of the grayscale image. We can just use the uint8_t value as gray value.

ei_run_dsp.h: extract_image_features_quantized():
...
       int32_t r = static_cast<int32_t>(pixel >> 16 & 0xff);
       int32_t g = static_cast<int32_t>(pixel >> 8 & 0xff);
       int32_t b = static_cast<int32_t>(pixel & 0xff);
       int32_t gray = (iRedToGray * r) + (iGreenToGray * g) + (iBlueToGray * b);
       gray >>= 16; // scale down to int8_t
       gray += EI_CLASSIFIER_TFLITE_INPUT_ZEROPOINT;
       if (gray < - 128) gray = -128;
       else if (gray > 127) gray = 127;
       output_matrix->buffer[output_ix++] = static_cast<int8_t>(gray);
Grayscale Quantization Explanation
#2

It’s a good question, we’re doing this so we have a consistent signal across all sensor types. The memory overhead is not that bad as you imagine, as you can do this on the fly with the signal_t. Your internal buffer is int8, we then dynamically convert when needed. See e.g. https://github.com/edgeimpulse/example-signal-from-rgb565-frame-buffer which has RGB565 framebuffer and then converts on the fly. We do this on fully supported dev boards already to save RAM.

1 Like
#3

Thanks for the link! Below function converts rgb565 to rgb in the given code:

void r565_to_rgb(uint16_t color, uint8_t *r, uint8_t *g, uint8_t *b) {
    *r = (color & 0xF800) >> 8;
    *g = (color & 0x07E0) >> 3;
    *b = (color & 0x1F) << 3;
}

Although it does not make sense but what would be the equivalent function for grayscale to rgb? Will the function below work?

void grayscale_to_rgb(uint8_t gray, uint32_t *r, uint32_t *g, uint32_t *b) {
    *r = gray<<16;
    *g = gray<<8;
    *b = gray;
}

Or,

void grayscale_to_rgb(uint8_t gray, uint32_t *rgb) {
    *rgb = (gray<<16) | (gray<<8) | gray;
}
#4 
void grayscale_to_rgb(uint8_t gray, uint32_t *r, uint32_t *g, uint32_t *b) {
    *r = gray;
    *g = gray;
    *b = gray;
}

This would work I guess :slight_smile:

#5

OK, let me try it. By the way, is there any example using grayscale camera?

#6

Yeah, see https://github.com/edgeimpulse/firmware-himax-we-i-plus/blob/03847d1f328d14af7f42952d564b9771ea7a6fd6/sensors/ei_camera.cpp#L560 for example.

1 Like