Generate Raw features from an image for a classifier project to run the impulse locally

Help
#1

Question/Issue:
I am trying to run a project locally on Windows 11 using the repository example-standalone-inferencing. I followed all the required steps, and the result was obtained as it should be by pasting the raw features from the edge impulse portal. However, if I want to extract those features on the fly, I need to perform the preprocessing in the code to generate the raw features.

My project is an image binary classifier, so I would receive an RGB .jpg image of any resolution, say (HxWx3). The impulse is trained on a 96x96 (squash resize) - Grayscale input image. So the raw features generated for the train & test data are 9216 features as a 1D array. Now, I need to convert the HxWx3 to a 9216 feature array (in C++) that should be a close enough match to the features generated inside Edge Impulse Studio.

Project ID:
599977

Context/Use case:
To perform a binary classification on a given image.

Steps Taken:
I was able to perform functions in the following order using the CImg C++ library to get a raw feature vector:

  1. Read the image using CImg in RGB format (HxWx3).
  2. Convert the RGB image to YCbCr format and extract the first channel (luminance) to get the grayscale image. (HxWx1)
  3. Resizing the 1 channel image using Nearest Neighbor to 96x96 to get a vector of 9216 features.
  4. Getting 9216 feature vector to pass it to the impulse run_classifier function:
  for (size_t i = 0; i < resizedImage.size() && i < MAX_FEATURES; ++i) {
      uint8_t pixel = resizedImage[i];
      features[i] = static_cast<float>((pixel << 16) | (pixel << 8) | pixel);
  }

Expected Outcome:
The outcome from the script and edge impulse studio should be exactly the same.

Actual Outcome:
Sometimes, the output is not the same, and the confidence score varies by a margin.

Reproducibility:

  • [ ] Always
  • [#] Sometimes
  • [ ] Rarely

Environment:

  • Platform: Local
  • Build Environment Details: Mingw-w64 Cmake.
  • OS Version: Windows 11
  • Edge Impulse Version (Firmware): 1.69.15
  • Project Version: v7
  • Custom Blocks / Impulse Configuration: Image data - 96x96 (Squash) - Image (Grayscale) - Classification (Input features: Image) - Output features: 2

Additional Information:
I am aware that the run_classifier already has a preprocessing block, but this confuses me as to how I can send the 9216 features without doing the preprocessing myself. Also, do you recommend any specific C++ library to perform the preprocessing?

Thanks in advance. Any support is much appreciated.
Cheers,
Garvit

#2

Hi

I’ve tried to formalize my C++ code to run the model locally using this documentation. However, using this, the results from the studio don’t match sometimes. Below is my main.cpp. What could I be doing wrong? All the preprocessing functions are used from the SDK, the only difference now could be image loading. How is it done inside Edge Impulse Studio?

#include <vector>
#include <iostream>
#include <iomanip>
#include <cmath>

#define STB_IMAGE_IMPLEMENTATION
#include "stb_image.h"          // downloaded from https://github.com/nothings/stb/blob/master/stb_image.h

// Edge Impulse SDK headers
#include "edge-impulse-sdk/classifier/ei_run_classifier.h"
#include "edge-impulse-sdk/dsp/image/image.hpp"
#include "model-parameters/model_metadata.h"

#define EI_CLASSIFIER_INPUT_CHANNELS 3
#define EI_CLASSIFIER_FEATURE_COUNT (EI_CLASSIFIER_INPUT_WIDTH * EI_CLASSIFIER_INPUT_HEIGHT)

// Function to preprocess the image and run classification
int classifyImage(const char* imagePath) {
    // Step 1: Load the image
    int width, height, channels;
    unsigned char* image = stbi_load(imagePath, &width, &height, &channels, 3); // Force 3 channels (RGB)
    if (!image) {
        std::cerr << "Error: Could not load image from " << imagePath << "!" << std::endl;
        return 1;
    }

    std::cout << "Loaded image with width: " << width << ", height: " << height << ", channels: " << channels << std::endl;

    // Step 2: Resize the image to 96x96
    std::vector<uint8_t> resizedImage(EI_CLASSIFIER_INPUT_WIDTH * EI_CLASSIFIER_INPUT_HEIGHT * EI_CLASSIFIER_INPUT_CHANNELS);
    int resize_result = ei::image::processing::resize_image_using_mode(
        image, width, height,
        resizedImage.data(), EI_CLASSIFIER_INPUT_WIDTH, EI_CLASSIFIER_INPUT_HEIGHT,
        EI_CLASSIFIER_INPUT_CHANNELS, EI_CLASSIFIER_RESIZE_MODE
    );
    if (resize_result != 0) {
        std::cerr << "Error: Failed to resize image, error code: " << resize_result << std::endl;
        stbi_image_free(image);
        return 1;
    }

    // Free the original image
    stbi_image_free(image);

    // Step 3: Convert the resized image to a flat buffer of uint32_t in 0xRRGGBB format
    std::vector<uint32_t> features(EI_CLASSIFIER_FEATURE_COUNT);
    for (size_t i = 0; i < EI_CLASSIFIER_FEATURE_COUNT; ++i) {
        size_t pixel_idx = i * 3; // Each pixel has 3 channels (R, G, B)
        uint8_t r = resizedImage[pixel_idx];
        uint8_t g = resizedImage[pixel_idx + 1];
        uint8_t b = resizedImage[pixel_idx + 2];
        features[i] = (r << 16) | (g << 8) | b; // Convert to 0xRRGGBB
    }
    
    
    // Debug: Print the first 10 pixels in hex to confirm the format
    std::cout << "First 10 pixels (0xRRGGBB format):\n";
    for (size_t i = 0; i < 10; ++i) {
        std::cout << "Pixel " << i << ": 0x" << std::hex << std::setw(6) << std::setfill('0') << features[i] << std::endl;
    }
    std::cout << std::dec; // Reset to decimal for subsequent output

    // The signal will provide float values, so we need to convert uint32_t to float
    std::vector<float> float_features(EI_CLASSIFIER_FEATURE_COUNT);
    for (size_t i = 0; i < EI_CLASSIFIER_FEATURE_COUNT; ++i) {
        float_features[i] = static_cast<float>(features[i]);
    }

    // Step 4: Set up the signal using numpy::signal_from_buffer
    ei::signal_t signal;
    int signal_result = numpy::signal_from_buffer(float_features.data(), EI_CLASSIFIER_FEATURE_COUNT, &signal);
    if (signal_result != 0) {
        std::cerr << "Error: Failed to create signal from buffer, error code: " << signal_result << std::endl;
        return 1;
    }

    // Step 5: Run the classifier
    ei_impulse_result_t result = { 0 };
    EI_IMPULSE_ERROR res = run_classifier(&signal, &result, false /* debug */);
    if (res != EI_IMPULSE_OK) {
        std::cerr << "Error: Failed to run classifier, error code: " << res << std::endl;
        return 1;
    }

    // Step 6: Print the classification results
    std::cout << "Predictions (DSP: " << result.timing.dsp << " ms, Classification: " << result.timing.classification << " ms, Anomaly: " << result.timing.anomaly << " ms):\n";
    for (size_t ix = 0; ix < EI_CLASSIFIER_LABEL_COUNT; ix++) {
        std::cout << result.classification[ix].label << ": " << std::fixed << std::setprecision(2) << result.classification[ix].value;
        if (ix != EI_CLASSIFIER_LABEL_COUNT - 1) {
            std::cout << ", ";
        }
    }
    std::cout << std::endl;
    return 0;
}

int main(int argc, char* argv[]) {
    if (argc != 2) {
        std::cerr << "Usage: " << argv[0] << " <image_file_path>" << std::endl;
        return 1;
    }

    return classifyImage(argv[1]);
}

Thanks,
Garvit