I want to "Run Inference" when I push a pushbutton in the ESP32-CAM example by luisomoreau

Help
#1

Hello,

I want to classify a photo taken by a ESP32-CAM, but I want to add a physical pushbutton in a input to “Run Inference”

To get there I use the Github shared by luisomoreau: https://github.com/edgeimpulse/example-esp32-cam

I am able to classify a photo, but to do that I have to push “Run Inference” button in the web (I use my mobile for that). After pushing the button I can see the result in the web.

But I want to run the inference without using the web, I want to use a pushbutton to run the inference and some leds to see the result. Right now, I am able to turn on leds depending on the result of the inference. So when it classifies a cat turns on one led and when it clasifies a dog turns on another led.

But my issue is that I am not able to Run Inference when I push a physical pushbutton connected in an input pin of the ESP32-CAM.

To turn on the leds I modified two files:

A - The first one is “Advanced-image-Classification.ino” just to add the two outputs:

void setup() {
...
  pinMode(12, OUTPUT);    // sets the digital pin 12 as output
  pinMode(13, OUTPUT);    // sets the digital pin 13 as output
...
}

B - The second file is “app_httpd.cpp” file I modified one of the functions there (I comment inside the code the modified part)

static esp_err_t inference_results_handler(httpd_req_t *req)
{
  static char json_response[1024] = "";

  char *p = json_response;

  *p++ = '{';

  if (result.classification[0].label)
  {
    p += sprintf(p, "\"success\": true,");
    p += sprintf(p, "\"classification\": [");
    for (size_t ix = 0; ix < EI_CLASSIFIER_LABEL_COUNT; ix++)
    {
      *p++ = '{';
      p += sprintf(p, "\"label\": \"%s\",", result.classification[ix].label);
      p += sprintf(p, "\"value\":%.5f", result.classification[ix].value);
      *p++ = '}';
      if (ix != EI_CLASSIFIER_LABEL_COUNT - 1)
      {
        *p++ = ',';
      }

      //p+=sprintf(p, "\"%s\":%.5f,", result.classification[ix].label, result.classification[ix].value);
    }
    *p++ = ']';
    *p++ = ',';
  }
  else
  {
    p += sprintf(p, "\"success\": false,");
  }
  // human-readable predictions

  p += sprintf(p, "\"timing\": {");
  p += sprintf(p, "\"timing_dsp\":%d,", result.timing.dsp);
  p += sprintf(p, "\"timing_classification\":%d,", result.timing.classification);
  p += sprintf(p, "\"timing_anomaly\":%d", result.timing.anomaly);

  *p++ = '}';
  *p++ = '}';
  *p++ = 0;
  Serial.println(json_response);


//*****************************************************************************************************
//I MODIFIED THIS PART - BEGINNING
//*****************************************************************************************************
  float value_tmp = 0.0;  // to save the porcentage
  String label_tmp;       // to save the name

  if (result.classification[0].label)
  {
    for (size_t ix = 0; ix < EI_CLASSIFIER_LABEL_COUNT; ix++)
    {
      if (result.classification[ix].value > value_tmp) // to choose the class with the biggest value
      {
      value_tmp = result.classification[ix].value;
      label_tmp = result.classification[ix].label;
      }
    }
  }

  if (value_tmp >= 0.85)                  // If the classification porcentage is less than 0.85 we do nothing
  {
    if (label_tmp == "Dog")          // If it classifises a Dog activates a led
    {
      digitalWrite(12, HIGH); // sets the digital pin 12 on
      delay(1000);            // waits for a second
      digitalWrite(12, LOW); // sets the digital pin 12 off
    }
  
    if (label_tmp == "Cat")             // If it classifises a Cat activates a led
    {
      digitalWrite(13, HIGH);  // sets the digital pin 13 on
      delay(1000);            // waits for a second
      digitalWrite(13, LOW); // sets the digital pin 13 off
     } 
  }
//*****************************************************************************************************
//I MODIFIED THIS PART - END
//*****************************************************************************************************

  httpd_resp_set_type(req, "application/json");
  httpd_resp_set_hdr(req, "Access-Control-Allow-Origin", "*");

  ei_impulse_result_t result = {0};

  return httpd_resp_send(req, json_response, strlen(json_response));
}

I HAVE TWO QUESTIONS:

1 - I don’t know where to add and input to Run the Inference to classify a image when I push a pushbutton. I want to know which part of the code I have to change. That way I would be able to Run Inference and see the result without the web, that way the ESP32-CAM would be completely independant.

2 - This is beginner level: the function: esp_err_t inference_results_handler() it is not call anywhere in the code. When it is called? By who?

If I didn’t make myself clear enough, feel free to ask.

Thank you for your time.

#2

Hi @Mikel,

I do not have an example for the ESP32 (@louis would know best how to modify that code). However, here is something I am working on for the Nano 33 BLE Sense. When you press a button, it records the accelerometer for 1 second and then performs inference. Perhaps it will be a useful starting point:

  // Wait for button press
  while (digitalRead(BTN_PIN) == 1);

  // Turn on LED to show we're recording
  digitalWrite(LED_R_PIN, LOW);

  // Record samples in buffer
  for (int i = 0; i < NUM_SAMPLES; i++) {

    // Take timestamp so we can hit our target frequency
    timestamp = millis();

    // Read and convert accelerometer data to m/s^2
    IMU.readAcceleration(x, y, z);
    x *= CONVERT_G_TO_MS2;
    y *= CONVERT_G_TO_MS2;
    z *= CONVERT_G_TO_MS2;

    // Store data in buffer as 1D array [x0, y0, z0, x1, y1, z1, ...]
    raw_buf[(NUM_AXES * i) + 0] = x;
    raw_buf[(NUM_AXES * i) + 1] = y;
    raw_buf[(NUM_AXES * i) + 2] = z;

    // Wait just long enough for our sampling period
    while (millis() < timestamp + SAMPLING_PERIOD_MS);
  }

  // Turn off LED to show we're done recording
  digitalWrite(LED_R_PIN, HIGH);

  // Turn the raw buffer into a signal for inference
  err = numpy::signal_from_buffer(raw_buf, 
                                  EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE, 
                                  &signal);
  if (err != 0) {
    Serial.print("ERROR: Failed to create signal from buffer: ");
    Serial.println(err);
    return;
  }

  // Run the impulse
  err = run_classifier(&signal, &result, false);
  if (err != 0) {
    Serial.print("ERROR: Failed to run classifier: ");
    Serial.println(err);
    return;
  }

  // Find highest predicted class (argmax)
  max_i = 0;
  for (int i = 0; i < EI_CLASSIFIER_LABEL_COUNT; i++) {
    if (result.classification[i].value > max_val) {
      max_i = i;
      max_val = result.classification[i].value;
    }
  }

  // Print highest predicted label and value
  Serial.print("Predicted class: ");
  Serial.print(result.classification[max_i].label);
  Serial.print(" Value: ");
  Serial.println(max_val);```
#3

Hello @Mikel,

If you don’t need the web interface that is pretty greedy, you can have a look at this example that is easier to understand: https://github.com/edgeimpulse/example-esp32-cam/blob/main/Examples/Inference-on-boot/Inference-on-boot.ino

It will take a picture and classify it when the board boots (and save it on the SD card with the detected class as a name, useful if you want to quickly upload them again and increase your dataset size :wink: ).

You can modify a few things:

The loop function:

void loop() {
  take_picture();
}

You will need to attached an interrupt so when you press the button, it will launch the take_picture function.

Also, I can’t remember if the interrupts can exit the deep sleep function that I called line 221:

  // --- Deep sleep ---
  Serial.println("Going to sleep now");
  delay(2000);
  esp_deep_sleep_start();

If not you’ll probably want to add a lighter sleep function at the end of the take_picture function.

I hope that helps,

Regards,

Louis

1 Like