Nicla Voice + Vision - Integrating I2C ESLOV Comms

Cevec · February 10, 2024, 5:15pm

Beautiful people of Edge Impulse forum,

I am trying to establish communication between Nicla Voice (keyword spotting) and Nicla Vision through I2C with ESLOV.

What I am trying to achieve: on a spotted keyword nicla voice sends a string using I2C to nicla vision.

I have already managed to do it using classic arduino sketch (keyboard commands to control nicla vision using nicla voice), but I just cannot wrap my head around integrating it into your code so I can use NN keyword spotting for the purpose of controlling the other board.

I could really appreciate some help integrating this code:

#include <Wire.h>
#include "Nicla_System.h"

const byte deviceAddress = 0x42; 
String inputString = "";  // A String to hold incoming data

void setup() {
    nicla::begin();
    nicla::leds.begin();
    Wire.begin();
    Serial.begin(9600);  // Start serial communication at 9600 baud to debug
    inputString.reserve(200); // Reserve 200 bytes for the inputString  
    delay(5000);
    Serial.print("\nI2C comms initiated.");
    Serial.print("\nType: 'voice' or 'muted' to communicate with Nicla Voice");
}

void loop() {
    nicla::leds.setColor(green);
    delay(500);
    nicla::leds.setColor(off);
    if (Serial.available()) {
      // Read the incoming data
      inputString = Serial.readStringUntil('\n');
    if (inputString == "voice") {
    Wire.beginTransmission(deviceAddress);
    Wire.write("voice", 5); 
    Serial.print("\nSent: voice");
    nicla::leds.setColor(blue);
    byte error = Wire.endTransmission();
      if (error) {
        Serial.print("\nMessage not received. ");
        Serial.print("\nTransmission error: ");
        Serial.println(error);
        nicla::leds.setColor(red);
      }
      else {
        Serial.print("\nMessage: 'voice' received. ");
      }
      delay(1000);
    }
    if (inputString == "muted") {
    Wire.beginTransmission(deviceAddress);
    Wire.write("muted", 5);
    Serial.print("\nSent: muted"); 
    nicla::leds.setColor(green);
    byte error = Wire.endTransmission();
      if (error) {
        Serial.print("\nMessage not received. ");
        Serial.print("\nTransmission error: ");
        Serial.println(error);
        nicla::leds.setColor(red);
      }
      else {
        Serial.print("\nMessage: 'muted' received. ");
      }
      delay(1000);
    }
  }
}

Into this one:

#include "ei_syntiant_ndp120.h"
#include "Nicla_System.h"
#include "NDP.h"
#include "edge-impulse-sdk/porting/ei_classifier_porting.h"
#include "ingestion-sdk-platform/nicla_syntiant/ei_at_handlers.h"
#include "ingestion-sdk-platform/nicla_syntiant/ei_device_syntiant_nicla.h"
#include "ingestion-sdk-platform/sensors/ei_inertial.h"
#include "inference/ei_run_impulse.h"
#ifdef WITH_IMU
#include "ingestion-sdk-platform/sensors/ei_inertial.h"
#include "model-parameters/model_metadata.h"
#else
#include "ingestion-sdk-platform/sensors/ei_microphone.h"
#endif

#include "rtos.h"
#include "Thread.h"
#include "EventQueue.h"

#define TEST_READ_TANK      0

extern NDPClass NDP;
static ATServer *at;
static bool ndp_is_init;

#if TEST_READ_TANK == 1
static void test_ndp_extract(void);
#endif

static void error_event(void);
static void match_event(char* label);
static void irq_event(void);

static bool _on_match_enabled = false;
static volatile bool got_match = false;
static volatile bool got_event = false;

void ei_setup(char* fw1, char* fw2, char* fw3)
{
    uint8_t valid_synpkg = 0;    
    bool board_flashed = false;
    uint8_t flashed_count = 0;
    EiDeviceSyntiantNicla *dev = static_cast<EiDeviceSyntiantNicla*>(EiDeviceInfo::get_device());
    char* ptr_fw[] = {fw1, fw2, fw3};

    ndp_is_init = false;
    Serial.begin(115200);
    Serial.println("Starting setup...");

    nicla::begin();
    Serial.println("Nicla System initialized.");
    nicla::disableLDO();
    Serial.println("LDO Disabled.");
    nicla::leds.begin();

    while (!Serial) {
        nicla::leds.setColor(red);
        Serial.println("Waiting for Serial...");
    }

    Serial.println("Hello from Edge Impulse on Arduino Nicla Voice\r\n"
                   "Compiled on " __DATE__ " " __TIME__);

    nicla::leds.setColor(green);

    NDP.onEvent(irq_event);
    NDP.onMatch(match_event);

    dev->get_ext_flash()->init_fs();
    Serial.println("External Flash File System Initialized.");

    for (int8_t i = 0; i < 3 ; i++) {
        if (ptr_fw[i] != nullptr){
            if (dev->get_file_exist(ptr_fw[i])){
                Serial.printf("%s exists.\n", ptr_fw[i]);
                valid_synpkg++;
            }
            else{
                Serial.printf("%s not found!\n", ptr_fw[i]);
            }
        }
    }

    if (valid_synpkg == 3){
        Serial.println("All firmware packages are valid.");
        NDP.begin(fw1);
        NDP.load(fw2);
        NDP.load(fw3);
        NDP.getInfo();
        Serial.println("NDP initialized with firmware.");
        ndp_is_init = true;

#ifdef WITH_IMU
        Serial.println("Configuring with IMU.");
        NDP.configureInferenceThreshold(EI_CLASSIFIER_NN_INPUT_FRAME_SIZE);
#else
        Serial.println("Configuring with Microphone.");
        NDP.turnOnMicrophone();
        NDP.getAudioChunkSize();
#endif
        NDP.interrupts();
        Serial.println("NDP interrupts configured.");
        ei_syntiant_set_match();
        nicla::leds.setColor(off);
    }
    else{
        Serial.println("NDP not properly initialized. Missing firmware packages.");
        nicla::leds.setColor(red);
    }

    dev->get_ext_flash()->init_fs();

    at = ei_at_init(dev);
    Serial.println("AT Server initialized.");

    if (ndp_is_init) {
        ei_inertial_init();
        Serial.println("Inertial sensor initialized.");
        ei_run_nn_normal();
        Serial.println("Neural network inference started.");
    }    

    Serial.println("Type AT+HELP to see a list of commands.");
    at->print_prompt();
}

void ei_main(void)
{
    int match = -1;
    char data = Serial.read();

    while (data != 0xFF) {
        at->handle(data);
        Serial.print("Received data: ");
        Serial.println(data, HEX);

        if (ei_run_impulse_is_active() && data == 'b') {
            Serial.println("Stopping impulse due to 'b' command.");
            ei_start_stop_run_impulse(false);
        }

        data = Serial.read();
    }

    if (ei_run_impulse_is_active()) {
        if (got_match) {
            Serial.println("Match detected.");
            got_match = false;
            nicla::leds.setColor(off);
            ThisThread::sleep_for(100);
        }

        if (got_event) {
            Serial.println("Event detected.");
            got_event = false;
            nicla::leds.setColor(green);
            ThisThread::sleep_for(100);
            nicla::leds.setColor(off);
            match = NDP.poll();
        }

        if (match > 0) {
            Serial.printf("Match: %d\n", match);
            match = -1;
        }

#ifdef WITH_IMU
        if (ei_run_impulse_is_active()) {
            ei_run_impulse();
        }
#endif
    }
}

void ei_syntiant_clear_match(void)
{
    _on_match_enabled = false;
}

void ei_syntiant_set_match(void)
{
    _on_match_enabled = true;
}

static void error_event(void)
{
    nicla::leds.begin();
    while (1) {
        nicla::leds.setColor(red);
        Serial.println("Error event triggered.");
        ThisThread::sleep_for(250);
        nicla::leds.setColor(off);
        ThisThread::sleep_for(250);
    }
    nicla::leds.end();
}

static void match_event(char* label)
{
    if (_on_match_enabled) {
        if (strlen(label) > 0) {
            got_match = true;
            Serial.printf("Match: %s\n", label);
        }
    }
}

static void irq_event(void)
{
    if (_on_match_enabled) {
        got_event = true;
        Serial.println("IRQ event triggered.");
    }
}

What I managed to do already is running code when keyword is recognized, but I can’t get I2C to work.

My integration that I tried last time:

/* Include ----------------------------------------------------------------- */
#include "ei_syntiant_ndp120.h"
//--------------------------------------------custom code ----------------------------------
#include <Wire.h>
//----------------------------------------end of custom code--------------------------------------------
#include "Nicla_System.h"
#include "NDP.h"
#include "edge-impulse-sdk/porting/ei_classifier_porting.h"
#include "ingestion-sdk-platform/nicla_syntiant/ei_at_handlers.h"
#include "ingestion-sdk-platform/nicla_syntiant/ei_device_syntiant_nicla.h"
#include "ingestion-sdk-platform/sensors/ei_inertial.h"
#include "inference/ei_run_impulse.h"
#ifdef WITH_IMU
#include "ingestion-sdk-platform/sensors/ei_inertial.h"
#include "model-parameters/model_metadata.h"
#else
#include "ingestion-sdk-platform/sensors/ei_microphone.h"
#endif

#include "rtos.h"
#include "Thread.h"
#include "EventQueue.h"

#define TEST_READ_TANK      0

/* device class */
extern NDPClass NDP;
static ATServer *at;
static bool ndp_is_init;

#if TEST_READ_TANK == 1
static void test_ndp_extract(void);
#endif


static void error_event(void);
static void match_event(char* label);
static void irq_event(void);

static bool _on_match_enabled = false;
static volatile bool got_match = false;
static volatile bool got_event = false;
//--------------------------------------------custom code ----------------------------------
const byte deviceAddress = 0x42;
//----------------------------------------end of custom code--------------------------------------------

/* Public functions -------------------------------------------------------- */
/**
 * @brief 
 * 
 */
void ei_setup(char* fw1, char* fw2, char* fw3)
{
    uint8_t valid_synpkg = 0;    
    bool board_flashed = false;
    uint8_t flashed_count = 0;
    EiDeviceSyntiantNicla *dev = static_cast<EiDeviceSyntiantNicla*>(EiDeviceInfo::get_device());
    char* ptr_fw[] = {fw1, fw2, fw3};

    ndp_is_init = false;
    Serial.begin(115200);
    nicla::begin();
    //--------------------------------------------custom code ----------------------------------
    Wire.begin();
    //----------------------------------------end of custom code--------------------------------------------
    nicla::disableLDO();    // needed 
    nicla::leds.begin();

    while (!Serial) {   /* if Serial not avialable */
        nicla::leds.setColor(red);
    }
    ei_printf("Hello from Edge Impulse on Arduino Nicla Voice\r\n"
            "Compiled on %s %s\r\n",
            __DATE__,
            __TIME__);
    nicla::leds.setColor(green);
    //NDP.onError(error_event);
    NDP.onEvent(irq_event);
    NDP.onMatch(match_event);
    
    dev->get_ext_flash()->init_fs();
    for (int8_t i = 0; i < 3 ; i++) {
        if (ptr_fw[i] != nullptr){                  // nullptr check
            if (dev->get_file_exist(ptr_fw[i])){
                ei_printf("%s exist\n", ptr_fw[i]);
                valid_synpkg++;
            }
            else{
                ei_printf("%s not found!\n", ptr_fw[i]);
            }
        }
    }
    //dev->get_ext_flash()->deinit_fs();  // de init as NDP re init it

    if (valid_synpkg == 3){
        NDP.begin(fw1);
        NDP.load(fw2);
        NDP.load(fw3);
        NDP.getInfo();
        ndp_is_init = true;

#ifdef WITH_IMU
        NDP.configureInferenceThreshold(EI_CLASSIFIER_NN_INPUT_FRAME_SIZE);
#else   
        NDP.turnOnMicrophone();     
        NDP.getAudioChunkSize();    /* otherwise it is not initialized ! */
#endif
        NDP.interrupts();

        ei_syntiant_set_match();
        nicla::leds.setColor(off);
    }
    else{
        ei_printf("NDP not properly initialized\n");
        nicla::leds.setColor(red);
    }
    dev->get_ext_flash()->init_fs();    // NDP probably will de init and unmount

    /* init ar server */
    at = ei_at_init(dev);

    /* start inference */
    if (ndp_is_init == true) {
        /* sensor init */
        ei_inertial_init();
        ei_run_nn_normal();
    }    

    ei_printf("Type AT+HELP to see a list of commands.\r\n");
    at->print_prompt();
}

/**
 * @brief 
 * 
 */
void ei_main(void)
{
    int match = -1;
    /* handle command comming from uart */
    char data = Serial.read();

    while (data != 0xFF) {
        at->handle(data);

        if (ei_run_impulse_is_active() && data == 'b') {
            ei_start_stop_run_impulse(false);
        } 
        
        data = Serial.read();
    }

    if (ei_run_impulse_is_active() ==true) {
        if (got_match == true){
            got_match = false;
            //nicla::leds.setColor(blue);// // => disabled default color
            ThisThread::sleep_for(100);
            nicla::leds.setColor(off);
        }

        if (got_event == true){
            got_event = false;
            nicla::leds.setColor(green);
            ThisThread::sleep_for(100);
            nicla::leds.setColor(off);
            match = NDP.poll();
        }

        if (match > 0) {
            ei_printf("match: %d\n", match);
            match = -1;
        }

#ifdef WITH_IMU
        // for now by default we stay in inference
        if (ei_run_impulse_is_active()) {
            ei_run_impulse();
        }
#endif
    }

}


/**
 * @brief disable interrupt from NDP class
 * 
 */
void ei_syntiant_clear_match(void)
{
    _on_match_enabled = false;
    //NDP.turnOffMicrophone();
    //NDP.noInterrupts();
}

/**
 * @brief enable interrupt from NDP clas
 * 
 */
void ei_syntiant_set_match(void)
{
    _on_match_enabled = true;
    //NDP.turnOnMicrophone();
    //NDP.interrupts();
}

/**
 * @brief Callback when an Error is triggered
 * @note it never exit!
 */
static void error_event(void)
{
    nicla::leds.begin();
    while (1) {
        nicla::leds.setColor(red);
        ThisThread::sleep_for(250);
        nicla::leds.setColor(off);
        ThisThread::sleep_for(250);
    }
    nicla::leds.end();
}

/**
 * @brief Callback when a Match is triggered
 * 
 * @param label The Match label
 */


static void match_event(char* label)
{
    //--------------------------------------------custom code ----------------------------------
    if (_on_match_enabled == true){
        if (strlen(label) > 0) {
            got_match = true;
            nicla::leds.setColor(green);
            ei_printf("\nMatch: %s\n", label);
            ei_printf("\nTEST BEFORE IF STATEMENT\n");
            if (strcmp(label, "NN0:go") == 0) {
                ei_printf("\nTEST AFTER IF STATEMENT\n");
                ei_printf("\nTRYING TO ESTABLISH CONNECTION NOW\n");
                delay(3000);
                Wire.beginTransmission(deviceAddress);
                Wire.write("voice", 5); 
                delay(3000);
                ei_printf("\nSent: voice");
                nicla::leds.setColor(blue);
                byte error = Wire.endTransmission();
                if (error) {
                    ei_printf("\nMessage: 'voice' failed to send. ");
                    nicla::leds.setColor(red);
                }
                else {
                    ei_printf("\nMessage: 'voice' received. ");
                }
            }
            if (strcmp(label, "NN0:stop") == 0) {
                delay(3000);
                Wire.beginTransmission(deviceAddress);
                Wire.write("muted", 5);
                delay(3000);
                ei_printf("\nSent: muted"); 
                nicla::leds.setColor(green);
                byte error = Wire.endTransmission();
                if (error) {
                    ei_printf("\nMessage: 'voice' failed to send. ");
                    nicla::leds.setColor(red);
                }
                else {
                    ei_printf("\nMessage: 'muted' received. ");
                }   
            }
            nicla::leds.setColor(red);
        }
    }
}
//----------------------------------------end of custom code--------------------------------------------
/**
 * @brief 
 * 
 */
static void irq_event(void)
{    
    if (_on_match_enabled == true) {
        got_event = true;
    }
}

Has this been disussed before?
Any tips otherwise?

I am attaching Nicla Vision code too just in case:

#SETUP_START

from pyb import I2C
from machine import LED
import sensor
import time
import image
import tf
import os
import gc
import uos

# Variables - FPS clock
clock = time.clock()

# Variables - I2C communication
i2c = I2C(1, I2C.SLAVE, addr=0x42)

# Variables - Led
led_blue = LED("LED_BLUE")
led_red = LED("LED_RED")
led_green = LED("LED_GREEN")

#led_blue.on()
#led_red.on()
#led_green.on()

def led_off():
    led_blue.off()
    led_red.off()
    led_green.off()

# Variables - Emotion recognition
net = None
labels = None

# Variables - Loop switch
loop_switch = False

# Sensor settings for face detection
sensor.reset()
sensor.set_contrast(3)
sensor.set_gainceiling(16)
sensor.set_framesize(sensor.HQVGA)
sensor.set_pixformat(sensor.GRAYSCALE)

# Load function for I2C communication
def readData():
    global loop_switch
    buf = bytearray(5)  # Adjust buffer size for the string length
    try:
        i2c.recv(buf, timeout=1)
        message = buf.decode('utf-8')  # Convert bytes to string
        print('Message received:', message, '\n')
        if message == 'voice':
            loop_switch = True
            print('loop_switch is', loop_switch)
        if message == 'muted':
            loop_switch = False
            print('loop_switch is', loop_switch)
    except:
        pass


# Load Haar Cascade for face detection
face_cascade = image.HaarCascade("frontalface", stages=25)

# Load emotion recognition model
try:
    net = tf.load("trained.tflite", load_to_fb=uos.stat('trained.tflite')[6] > (gc.mem_free() - (64*1024)))
    labels = [line.rstrip('\n') for line in open("labels.txt")]
except Exception as e:
    print(e)
    raise Exception('Failed to load model or labels.')


#SETUP_END

#Main loop
while True:
    led_off()
    led_green.on()
    led_blue.on()
    readData()
    if loop_switch == True:
        while loop_switch:
            readData()
            clock.tick()
            # Capture snapshot for face detection
            img = sensor.snapshot()

            # Find faces
            faces = img.find_features(face_cascade, threshold=0.9, scale_factor=1.25)
            face_detected = False  # Flag to check if any face is detected

            for face in faces:
                face_detected = True  # Set flag to true as face is detected
                img.draw_rectangle(face)

                # Turn on white LED as soon as a face is detected
                led_off()
                led_blue.on()
                led_red.on()
                led_green.on()

                # Crop the image to the face region
                face_img = img.copy(roi=face)
                # Run emotion recognition on the cropped face image
                emotion_detected = False
                for obj in net.classify(face_img, min_scale=1.0, scale_mul=0.8, x_overlap=0.5, y_overlap=0.5):
                    predictions_list = list(zip(labels, obj.output()))
                    for label, confidence in predictions_list:
                        print("%s = %f" % (label, confidence))
                        if label == 'sad' and confidence > 0.4:
                            led_off()
                            led_blue.on()
                            time.sleep(1.2)
                            emotion_detected = True
                            break
                        elif label == 'happy' and confidence > 0.8:
                            led_off()
                            led_green.on()
                            time.sleep(1.2)
                            emotion_detected = True
                            break
                    if emotion_detected:
                        break
                break  # Exit the loop after processing the first face

            if not face_detected:
                led_off()
                led_red.on()  # Turn on red LED if no face is detected

            print(clock.fps(), "fps")

            if loop_switch == 0:
                led_off()
                led_green.on()
                led_blue.on()
                i2c = I2C(1, I2C.SLAVE, addr=0x42)
                readData()

Eoin · February 12, 2024, 12:28pm

Hi @Cevec

First two things to make sure is that both Nicla Voice and Nicla Vision share a common ground connection. Also for I2C you may need a to use a pull-up resistors on the I2C lines if they’re not internally provided.

Then start with a smaller setup to test the comms is working:

Best

Eoin

Cevec · February 12, 2024, 11:00pm

I appreciate your response Eoin and I will look into it,
but as I have written earlier, the comms are working correctly when i tested them on a smaller scale with less complicated code. The problem is not the if the communication works, but the integration into the code you provided in the tutorial here: On your Syntiant TinyML Board - Edge Impulse Documentation

The main part that I think is the problem is putting the Wire.begin() in the ei_setup() I cant seem to find the proper spot to put the Wire.begin() to initiate the connection. I did some tests to find the place for this function but it breaks the code so often and compiling the code with building the firmware process is so tedious that I decided to look for help here.

This is how my setup looks for now, usually i plug both boards to my usb ports:

Cevec · February 12, 2024, 11:04pm

Digression:
The full code was supposed to be here but the hyperlink to the github page is not working.

This part is with the missing link: