Save Inferenced Audio from Arduino Nano BLE 33 Sense to Computer

Help
#1

Hi, I have deployed my model as Arduino Library to my Arduino Nano BLE 33 Sense using this tutorial - https://docs.edgeimpulse.com/docs/running-your-impulse-arduino. I am building on top of the nano_ble33_sense_microphone_continuous example.

What I am trying to achieve is to send the inferenced audio to my computer and save it as .wav file. I am having trouble finding the the binary audio within the loop function. I can see there is a result object that is returned from inference but I don’t know if that object contains the audio. I am also aware of the two buffers which is most likely where the audio I need is located, however I don’t know how to access them within the loop func.

This might be very straight forward but this is my first time coding in C++. Thanks in advance for the help :slight_smile:.

#2

Hi @Ruslan,

You will not be able to access the audio when you’re doing continuous keyword spotting. That’s because each slice of audio (for example, 0.333 s) is converted to a set of features (e.g. MFEs or MFCCs) and stored in a buffer in the background. Each time a new set of features is calculated, inference is performed on the previous 1 second worth of features.

From this, you get a sliding window where inference is performed on 1 second of data every 0.333 s, resulting in 3 inferences every second. However, only 0.333 sec of audio is stored at any given time.

I hope this helps! With the current setup, you can’t record a full window of audio (e.g. 1 second). You would need to store the audio yourself to a separate buffer. However, you’ll probably find that the Arduino Nano BLE 33 Sense will run out of memory, so you’d need a bigger microcontroller if you want to store audio samples.

2 Likes
#3

Hi @shawn_edgeimpulse,
Thanks a lot for the explanation, that makes sense. What about the normal keyword spotting, which doesn’t use buffers and saves 1 second of full audio. Am I able to export that?

#4

Hi @Ruslan,

In theory, yes–you should be able to save it. You will need to write the raw bytes over UART and use something like a Python script to convert it to a .wav file or use a library like this (https://github.com/TMRh20/TMRpcm/wiki/Advanced-Features#recording-audio) to write the bytes to a .wav file to an SD card.

1 Like
#5

hi @shawn_edgeimpulse.
do you have any tutorial continuous audio to control relay in raspberry pi 4?I am using linux sdk…

#6

Hi @desnug,

Which Linux SDK? Python, C++?

For Python, I recommend starting with this example: https://github.com/edgeimpulse/linux-sdk-python/blob/master/examples/audio/classify.py. For C++, this is a good example: https://github.com/edgeimpulse/example-standalone-inferencing-linux/blob/master/source/audio.cpp

I don’t believe either of those uses run_classifier_continuous(), so you’d probably want to read about it here (https://docs.edgeimpulse.com/reference/run_classifier_continuous) and here (https://docs.edgeimpulse.com/reference/continuous-audio-sampling) to set up sampling to a double buffer.

1 Like
#7

hi @shawn_edgeimpulse thanks for replying

I am using python linux sdk on Raspi4. I have done with classifying sound alarm and run perfectly with great accuracy but in the next step I’m having trouble understanding the output of classifying (classifiy.py) which can be used for relay control. I am following your channel on youtube but haven’t found a tutorial about this. Most tutorials that i found on the internet and edge impulse documentation use Arduino nano 33 like this one, none of them using continuous audio or on raspi 4. is it possible to control relay from Linux SDK python output on raspberry pi? or do you have any other suggestions?
or maybe you have planned some tutorial in the near future, it would be appreciated.

regards,

desnug

#8

Hi @desnug,

What you want to care about is this part:

for res, audio in runner.classifier(device_id=selected_device_id):
    print('Result (%d ms.) ' % (res['timing']['dsp'] + res['timing']['classification']), end='')
    for label in labels:
        score = res['result']['classification'][label]
        print('%s: %.2f\t' % (label, score), end='')
    print('', flush=True)

Find the label you want to watch (such as a keyword) and compare it to a threshold:

    for label in labels:
        score = res['result']['classification'][label]
        print('%s: %.2f\t' % (label, score), end='')
        if label == "<my label>" and score >= 0.6:
            # do relay control

We don’t have anything that explicitly performs relay control. You will need to follow a guide such as this one to integrate the above code snippet with relay control.

Good to know there’s interest in better documentation for the Python SDK. I’ll put it on my list of tutorials to work on :slight_smile:

#9

hi @shawn_edgeimpulse
I bought a relay from this using I2C, so I can try your tutorial just now.

bus.write_byte_data(DEVICE_ADDR, RELAY1, 0xFF) #ON relay1
bus.write_byte_data(DEVICE_ADDR, RELAY1, 0x00) #OFF relay1

I work with multiple classifications, here is mine:
for example (I have 5 labels):
if label 1 >= 0.6 turn on relay 1, relay 2,3,4 = OFF
if label 2 >= 0.6 turn on relay 2, relay 1,3,4 = OFF
if label 3 >= 0.6 turn on relay 2, relay 1,3,4 = OFF
if label 4 >= 0.6 turn on relay 2, relay 1,3,4 = OFF
if label 5 >= 0.6 all label turn OFF

               if label =="danger_alarm" and score >= 0.6:
                    bus.write_byte_data(DEVICE_ADDR, RELAY1, 0xFF)  #ON relay1
                    bus.write_byte_data(DEVICE_ADDR, RELAY2, 0x00)  #OFF relay2
                    bus.write_byte_data(DEVICE_ADDR, RELAY3, 0x00)  #OFF relay3
                    bus.write_byte_data(DEVICE_ADDR, RELAY4, 0x00)  #OFF relay3
                    
                elif label =="fire_alarm" and score >= 0.6:
                    bus.write_byte_data(DEVICE_ADDR, RELAY1, 0x00)
                    bus.write_byte_data(DEVICE_ADDR, RELAY2, 0xFF)
                    bus.write_byte_data(DEVICE_ADDR, RELAY3, 0x00)
                    bus.write_byte_data(DEVICE_ADDR, RELAY4, 0x00)
                    
                elif label =="gas_alarm" and score >= 0.6:
                    bus.write_byte_data(DEVICE_ADDR, RELAY1, 0x00)
                    bus.write_byte_data(DEVICE_ADDR, RELAY2, 0x00)
                    bus.write_byte_data(DEVICE_ADDR, RELAY3, 0xFF)
                    bus.write_byte_data(DEVICE_ADDR, RELAY4, 0x00)
                    
                elif label =="tsunami_alarm" and score >= 0.6:
                    bus.write_byte_data(DEVICE_ADDR, RELAY1, 0x00)
                    bus.write_byte_data(DEVICE_ADDR, RELAY2, 0x00)
                    bus.write_byte_data(DEVICE_ADDR, RELAY3, 0x00)
                    bus.write_byte_data(DEVICE_ADDR, RELAY4, 0xFF)
                    
                else:
                    bus.write_byte_data(DEVICE_ADDR, RELAY1, 0x00)
                    bus.write_byte_data(DEVICE_ADDR, RELAY2, 0x00)
                    bus.write_byte_data(DEVICE_ADDR, RELAY3, 0x00)
                    bus.write_byte_data(DEVICE_ADDR, RELAY4, 0x00)

using the code above, for example when detected label fire_alarm relay keeps blinking even though the detected label >= 0.6.
Do you have any suggestions, what should I do?

regards,

#10

hi @desnug.
I think you should change “else” with elif label == “your 5th label”. your relay will not be blinking. although it’s not the best way how to code.

#11

Hi @desnug,

I am not familiar with that relay board, so I’m not sure if the issue is a problem with inference or with the I2C driver/board. I recommend using print statements to make sure that inference is working the way you intend. For example:

if label =="danger_alarm" and score >= 0.6:
    print("danger")
elif label =="fire_alarm" and score >= 0.6:
    print("fire")
elif label =="gas_alarm" and score >= 0.6:
    print("gas")
elif label =="tsunami_alarm" and score >= 0.6:
    print("tsunami")
else:
    print("else")

What do you see when you run that?

#12

hi @ShawnHymel
here is the result when replace with yours. it keeps printing “else”, although “fire_alarm” is detected. so what do you think?

image
image631×512 28.9 KB

I think the best way for now is replace “else” with “elif”, just like @dexvils said…

image
image995×293 36.1 KB

#13

Ah, I see - @dexvils is correct. The ‘else’ case will be triggered each time through the loop. As a result, I recommend finding the maximum score and using that label to create your switch/case statement (or rather, series of if/elif statements, as Python < 3.10 does not have switch/case):

for res, audio in runner.classifier(device_id=selected_device_id):
    #print('Result (%d ms.) ' % (res['timing']['dsp'] + res['timing']['classification']), end='')
    
    # Find argmax(scores)
    max_score = 0.0
    max_label = ''
    for label in labels:
        score = res['result']['classification'][label]
        #print('%s: %.2f\t' % (label, score), end='')
        if score > max_score:
            max_score = score
            max_label = label
        print('', flush=True)
    
    # Perform action if score is over threshold
    if max_score >= 0.6:
        if max_label == 'danger_alarm':
            print("Do danger_alarm action")
        elif max_label == 'fire_alarm':
            print("Do fire_alarm action")
        elif max_label == 'gas_alarm':
            print("Do gas_alarm action")
        elif max_label == 'tsunami_alarm':
            print("Do tsunami_alarm action")
        else:
            print("Turn all off")
    else:
        print("Turn all off")
#14

@shawn_edgeimpulse
I’ve tried your code above, but why are you use else twice?
because after I ran, here is the result. still blinking…
error printing

#15

ah sorry my bad, I put the wrong indentation. since copy-paste directly will destroy indentation so I’ve rewritten the code manually. Look perfect now. Maybe I just do a little bit to modify the code because the gap is too far between one to the other.
image

thanks @shawn_edgeimpulse

regards,

desnug

2 Likes