Arduino, turn on and off the EI detection

#1

Hey all,

I have got a nicely running detection, and I want it to run continuously until the target sound is detected. Then I want to handle the actions with my Arduino code. When the arduino is done, I want to go back to inferencing and detecting the thing.

I know how to do this in the Arduino, and it’s set up now. However, when I come back from handling the detection and attempt to restart the inferencing, I immediately get a buffer overrun. I suspect that I did not cleanly end the inferencing function and when it restarts the buffer isn’t prepared to start.

Is there a technique to cleanly end before I launch off to the next thing?

What I have done is just cut and paste the default activities formerly in the loop() and put it into its own function. When my state machine isn’t otherwise doing anything, the inference runs. When the thing is detected, I switch out and stop running that inference function. When nothings happening, I run the function again.

Any tips?

Thanks!

#2

BTW, here’s the function, called “CheckDrinking()”

void CheckDrinking(){

    bool m = microphone_inference_record();
    if (!m) {
        if(Serial){ ei_printf("ERR: Failed to record audio...\n"); }
        return;  }

    signal_t signal;
    signal.total_length = EI_CLASSIFIER_SLICE_SIZE;
    signal.get_data = &microphone_audio_signal_get_data;
    ei_impulse_result_t result = {0};

    EI_IMPULSE_ERROR r = run_classifier_continuous(&signal, &result, debug_nn);
    if (r != EI_IMPULSE_OK) {
        if(Serial){ ei_printf("ERR: Failed to run classifier (%d)\n", r); }
        return;
    }
    if (++print_results >= (EI_CLASSIFIER_SLICES_PER_MODEL_WINDOW)) {
        if(Serial){ 
        for (size_t ix = 0; ix < EI_CLASSIFIER_LABEL_COUNT; ix++) {
            Serial.print(result.classification[ix].label);
            Serial.print(": ");
            Serial.print(result.classification[ix].value);
            Serial.print("  ");
            Serial.println();
        }}
        if(result.classification[0].value > .7 || result.classification[1].value > .7)
             detectDrinking = 1;
        else detectDrinking = 0;
        #if EI_CLASSIFIER_HAS_ANOMALY == 1
           if(Serial) { ei_printf("    anomaly score: %.3f\n", result.anomaly); }
        #endif
        print_results = 0;
    }

}

The only thing that wasn’t here by default is that I’m checking a couple of the classes and if the likelihood is over 70% I set a flag, which is what tips the State machine to do something else and stop running this function. If it doesn’t detect those classes, the flag is set to false.

#3

I’ve found that by running “run_classifier_init();” the model will restart after a couple of cycles of failures. So that’s good - having said that, I notice that after such a restart, it seems much more likely to get a false positive. Could be my imagination or possibly the buffer could be corrupted in some way.

Would really like someone from Edge Impulse to chime in here and recommend how to start and stop inferencing properly.

#4

Hi @braddo,

How are you collecting microphone data? The “buffer overrun” error occurs when you try to write to a full microphone memory buffer before you’ve had a chance to process the data in it with run_classifier_continuous().

My guess is that you have an interrupt service routine (ISR) that you’ve configured somewhere else in the code to continuously write to a buffer (probably microphone_audio_signal_get_data). When you try to process the second part of that state machine, you don’t call run_classifier_continuous() for a while, which means the audio buffer is filling up without being processed. When you return to the first part of the state machine, the buffer has been overwritten or samples have been dropped because you didn’t process them in time.

You have a few options:

  • Disable that ISR when you are in the other part of your state machine so that you’re not collecting audio data
  • Make the other part of your state machine very short (less than probably ~100 ms, depending on your EI_CLASSIFIER_SLICES_PER_MODEL_WINDOW value) so that you don’t miss data as it’s coming in from your microphone
  • Use multiple threads and/or cores so that one thread just handles keyword spotting (with run_classifier_continuous()) and the other thread handles your action whenever it receives notification from the first thread (e.g. pass a notification using something like a queue, mutex, or semaphore)
#5

Hi Shawn, thanks for checking in.

I am collecting the microphone data in the default way that the produced code from EI does. What I did is simply cut all of the EI code from the loop() function and slap it into a routine that is called when “nothing else is happening” i.e. when no buttons have been pressed and no lights are flashing etc.

I don’t have any interrupts, however the simple state machine I’ve made is completely non-blocking, so I suspect that if all of my debugging print statements are turned off, the code will spend the vast majority of time recording/listening/inferencing from the microphone.

But, when a button is pushed, that routine will no longer be called because the state machine is now handling the behaviors, driving motors, flashing lights etc. When those behaviors timeout (using millis() functions throughout) then the state machine will start calling the EI function again.

As in the reply above, I am now calling “run_classifier_init();” when the state machine exits so that in the next cycle through, the EI routine might be ready to start recording/listening again. I don’t know however, if I am properly terminating the EI routine. I simply stop calling it when a button is pressed. It’s possible that I should be clearing a buffer or something else so that, together with the init, it’s fully ready to start fresh again. If you have a suggestion for what to call when exiting the tasks in your default loop() or when re-entering your loop() I would really like to know.

One thing that concerns me is that in the EI generated function called “microphone_inference_record()” there is the dreaded delay, it’s actually “delay(1)”, which seems to be waiting for the buffer to fill up. Despite that it is only delaying for 1 millisecond, that’s like 99.99999% of the cycles available on this 32MHz NRF processor :slight_smile: You suggested I make the other parts of the process short - they are all basically nothing, i.e. 1 or 2 clock cycles, but the EI generated function just mentioned is likely dominating in a bad way, causing button presses to be missed. If you have a suggestion for how to gracefully remove the “delay(1)” from your code I would like to know, and humbly suggest that you replace it in your standard code generation routine so that it becomes non blocking.

Your suggestion about running the classifier on a separate thread probably would work but I don’t have the skill to implement that yet.

#6

Hi @braddo,

There should be an ISR happening on the backend. I’m assuming you’re using the Nano 33 example code, so there’s probably this line:

// configure the data receive callback
PDM.onReceive(&pdm_data_ready_inference_callback);

I haven’t dug into Arduino’s PDM library, but I imagine it sets up PDM to DMA, which automatically fills a buffer every time a piece of new PDM data is captured from the microphone. There’s likely at least one ISR happening there to notify when data is captured or the buffer is full/half-full.

The easiest process is to stop the PDM/DMA process in Arduino and don’t call run_classifier_init(). But, you end up dropping audio data when you do so. You can probably get rid of that delay(1) and just do something like:

while (inference.buf_ready == 0);

Make sure to call microphone_inference_record() just before calling run_classifier_continuous(), as that causes the process to wait until that slice of audio data (probably ~250 ms) is captured before running inference. If your other processes take longer than ~250 ms, you’ll run into the “buffer overrun” problem. You can try disabling audio as mentioned or changing EI_CLASSIFIER_SLICES_PER_MODEL_WINDOW to 2 or 3 (instead of 4) which will buy you more time for your other tasks.

#7

Thanks Shawn - I mean that I am not implementing any ISRs per se. I have no doubt that the EI code and its use of PDM etc have interrupts and ISRs.

I will investigate stopping the PDM process before exiting the “loop” i.e. the routine into which I have pasted the commands that were in your standard auto-generated loop(). Then I guess I just restart the PDM process when the other actions are completed?

I think the while() statement you suggested will also be blocking. The processor will not do anything else until the while is satisfied, and it is likely I want to push a button in that time, and it will be missed. I suppose I could add something like while(inference.buf_ready == 0 and !buttonpushed), but there are lots of conditions I would need to check for that are already checked in the state machine if only your code would allow them to be processed on every turn of the loop.

#8

I posted a new issue, specific to making the default “microphone_inference_record()” routine non-blocking. I think only after resolving that can I try Shawn’s advice above.