Gesture detection via IR sensors matrix AG8833

Hi , using the data forwarding I’ve collected data from an Arduino nano ble connected to an AG8833 ( 8x8 IR sensors matrix) and trained a model to detect simple gesture ( up, down , left , right , idle ).Everithing works perfectly. Now I’m looking to deploy on the Arduino so starting from the static_buffer example I’ve modified the sketch to include the sensor reading an test the model, but something is wrong and doesen’t work.In short in the same way I’ve uploaded the data with data forwarding I go to fill the feature vector and the pass the vector to the classifier, but also if the sensor reading works the classification don’t work at alland remain on the idle state ( nothing in fron to the sensor).

Where I’m wrong ?
Is it the right way to proceed ?

/* Includes ---------------------------------------------------------------- */
#include <amg8833_thermal_inference.h>
#include <Wire.h>
#include <Adafruit_AMG88xx.h>
#define FREQUENCY_HZ        28
#define INTERVAL_MS         (1000 / (FREQUENCY_HZ + 1))
Adafruit_AMG88xx amg;

float pixels[AMG88xx_PIXEL_ARRAY_SIZE];
static  float features[896] = {0};   // the 896 was suggested by the static buffer sketch.
static unsigned long last_interval_ms = 0;

/**
 * @brief      Copy raw feature data in out_ptr
 *             Function called by inference library
 *
 * @param[in]  offset   The offset
 * @param[in]  length   The length
 * @param      out_ptr  The out pointer
 *
 * @return     0
 */
int raw_feature_get_data(size_t offset, size_t length, float *out_ptr) {
    memcpy(out_ptr, features + offset, length * sizeof(float));
    return 0;
}

/**
 * @brief      Arduino setup function
 */
void setup() {
    Serial.begin(115200);
    Serial.println(F("AMG88xx pixels"));
    Serial.println("Edge Impulse Inferencing Demo");
    bool status;
    
    // default settings
    status = amg.begin();
    if (!status) {
        Serial.println("Could not find a valid AMG88xx sensor, check wiring!");
        while (1);
    }

}
/**
 * @brief      Arduino main function
 */
void loop()
{
   
   //static float features[]; 
    if (millis() > last_interval_ms + INTERVAL_MS) {
        last_interval_ms = millis();

    //read all the pixels
    amg.readPixels(pixels);

    Serial.print("Sensor reading....");

    //  fill features vector with sensor reading
    
    for(int i=1; i<=AMG88xx_PIXEL_ARRAY_SIZE; i++){
        features[i] = pixels[i-1];
        Serial.print(features[i]);
        Serial.print(",");
        if( i%8 == 0 ) Serial.println();
    }
  

    ei_printf("Edge Impulse standalone inferencing (Arduino)\n");

    if (sizeof(features) / sizeof(float) != EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE) {
        ei_printf("The size of your 'features' array is not correct. Expected %lu items, but had %lu\n",
            EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE, sizeof(features) / sizeof(float));
        delay(1000);
        return;
    }

    ei_impulse_result_t result = { 0 };

    // the features are stored into flash, and we don't want to load everything into RAM
    signal_t features_signal;
    features_signal.total_length = sizeof(features) / sizeof(features[0]);
    features_signal.get_data = &raw_feature_get_data;

    // invoke the impulse
    EI_IMPULSE_ERROR res = run_classifier(&features_signal, &result, false /* debug */);
    ei_printf("run_classifier returned: %d\n", res);

    if (res != 0) return;

    // print the predictions
    ei_printf("Predictions ");
    ei_printf("(DSP: %d ms., Classification: %d ms., Anomaly: %d ms.)",
        result.timing.dsp, result.timing.classification, result.timing.anomaly);
    ei_printf(": \n");
    ei_printf("[");
    for (size_t ix = 0; ix < EI_CLASSIFIER_LABEL_COUNT; ix++) {
        ei_printf("%.5f", result.classification[ix].value);
#if EI_CLASSIFIER_HAS_ANOMALY == 1
        ei_printf(", ");
#else
        if (ix != EI_CLASSIFIER_LABEL_COUNT - 1) {
            ei_printf(", ");
        }
#endif
    }
#if EI_CLASSIFIER_HAS_ANOMALY == 1
    ei_printf("%.3f", result.anomaly);
#endif
    ei_printf("]\n");

    // human-readable predictions
    for (size_t ix = 0; ix < EI_CLASSIFIER_LABEL_COUNT; ix++) {
        ei_printf("    %s: %.5f\n", result.classification[ix].label, result.classification[ix].value);
    }
#if EI_CLASSIFIER_HAS_ANOMALY == 1
    ei_printf("    anomaly score: %.3f\n", result.anomaly);
#endif

    delay(1000);
    }
}
/**
 * @brief      Printf function uses vsnprintf and output using Arduino Serial
 *
 * @param[in]  format     Variable argument list
 */

 void ei_printf(const char *format, ...) {
    static char print_buf[1024] = { 0 };

    va_list args;
    va_start(args, format);
    int r = vsnprintf(print_buf, sizeof(print_buf), format, args);
    va_end(args);

    if (r > 0) {
        Serial.write(print_buf);
    }
}

Hey @bob63 Aren’t you just filling 64 features?

    for(int i=1; i<=AMG88xx_PIXEL_ARRAY_SIZE; i++){
        features[i] = pixels[i-1];
        Serial.print(features[i]);
        Serial.print(",");
        if( i%8 == 0 ) Serial.println();
    }

This seems to be called only once, and thus only the first 64 features are set. I’d expect you to run this multiple times (14 times, with INTERVAL_MS pauses) until the buffer is full.

1 Like

Hi Jan, as suggested I’ve tried to fill all features without success. I’ve tested the static buffer example with raw data from the collected data and it works , but looking the raw data I’ve seen that the raw data aren’t as I was expecting . This an example of raw data (: 19.2500, 0.0000, 19.7500, 0.0000, 19.7500, 0.0000, 20.0000, 0.0000, 20.7500, 0.0000, 21.2500, 0.0000, 21.7500, 0.0000, 24.5000, 0.0000, 19.7500, 0.0000, 20.0000, 0.0000, 20.0000,… ) .The values different from 0.0000 are the temperature read from each element in the matrix , but I don’t understand why are alternated with 0.0000 considering that i sent on the serial alla the single row element reading so 8 value for row. Another thing that I’ve not clear is why the data forward collect the data as 16 row and not 8 as I think to send on the serial.
data forward and here the serial data on com8:

1 Like

Another thing that I’ve not clear is why the data forward collect the data as 16 row and not 8 as I think to send on the serial.

I think you need to get rid of the space between the two numbers, so 27.50,28.25 instead of 27.50, 28.25. I’d guess the data forwarder would do that automatically, but apparently not, will file a bug.

I also think that once that’s fixed this will also fix your on-device impulse. Or alternatively if you don’t want to recollect your data, you could do:

    for (int i=0; i < AMG88xx_PIXEL_ARRAY_SIZE * 2; i += 2) {
        features[i] = pixels[i];
        features[i + 1] = 0;
    }

To test it out quickly ^

edit: Or actually even better, just disable all the odd numbers in Create impulse to remove the 0 channels (disable T1, T3, T5, etc.):

And retrain and export. Then your trained impulse should be in the right format.

1 Like