Inference on Saeed Xiao BLE Sense with BLE

AiPTL · March 2, 2023, 1:04pm

Hello,

I’ve trained a ML model which works perfectly on vibration continuous mode. I want to integrate bluetooth so I can send the inference results to the MIT app inventor.

I’ve tried everything, following other examples (which samples every 2 second) etc… to include bluetooth, but I can’t get it to work properly.

I’ve attached the clean code of the inferencing, does anyone know how to allow your device to connect to BLE whilst running in this continuous mode?

/* Edge Impulse ingestion SDK
 * Copyright (c) 2022 EdgeImpulse Inc.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 *
 */

/* Includes ---------------------------------------------------------------- */
#include <AnomolyDetect_inferencing.h>
#include <LSM6DS3.h> //Click here to get the library: http://librarymanager/All#Arduino_LSM9DS1
#include <ArduinoBLE.h> 

/* Constant defines -------------------------------------------------------- */
#define CONVERT_G_TO_MS2    9.80665f
#define MAX_ACCEPTED_RANGE  2.0f        // starting 03/2022, models are generated setting range to +-2, but this example use Arudino library which set range to +-4g. If you are using an older model, ignore this value and use 4.0f instead

/*
 ** NOTE: If you run into TFLite arena allocation issue.
 **
 ** This may be due to may dynamic memory fragmentation.
 ** Try defining "-DEI_CLASSIFIER_ALLOCATION_STATIC" in boards.local.txt (create
 ** if it doesn't exist) and copy this file to
 ** `<ARDUINO_CORE_INSTALL_PATH>/arduino/hardware/<mbed_core>/<core_version>/`.
 **
 ** See
 ** (https://support.arduino.cc/hc/en-us/articles/360012076960-Where-are-the-installed-cores-located-)
 ** to find where Arduino installs cores on your machine.
 **
 ** If the problem persists then there's not enough memory for this model and application.
 */

/* Private variables ------------------------------------------------------- */
static bool debug_nn = false; // Set this to true to see e.g. features generated from the raw signal
static uint32_t run_inference_every_ms = 200;
static rtos::Thread inference_thread(osPriorityLow);
static float buffer[EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE] = { 0 };
static float inference_buffer[EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE];

LSM6DS3 myIMU(I2C_MODE, 0x6A);

// Service UUID
BLEService bleService("19B10000-E8F2-537E-4F6C-D104768A1214"); 

// Characteristic UUID 
BLEStringCharacteristic predictionChar("a5e93786-602d-4769-932f-fc62699df97a", BLERead | BLEWrite | BLENotify, 100 );
BLEFloatCharacteristic offcount("be574fe6-9bdc-4f88-abde-4393ec367590", BLERead | BLEWrite | BLENotify);
BLEFloatCharacteristic speed3count("3daa9df0-5ca4-4289-ab23-7c7be9205af6", BLERead | BLEWrite | BLENotify);
BLEFloatCharacteristic speed7count("fae59b4b-ef27-437b-8d2c-35ca21218dc8", BLERead | BLEWrite | BLENotify);
BLEFloatCharacteristic uncertaincount("609a46e0-17b8-4974-86f7-e9f33569a08f", BLERead | BLEWrite | BLENotify);
BLEFloatCharacteristic anomolycount("de4fd609-4981-4c25-8c60-c6cb0640f2d5", BLERead | BLEWrite | BLENotify);



/* Forward declaration */
void run_inference_background();

/**
* @brief      Arduino setup function
*/
void setup()
{
    // put your setup code here, to run once:
    Serial.begin(115200);
    delay(200);


    // Accelerometer Setup 
    myIMU.settings.accelRange = 4;      //Max G force readable.  Can be: 2, 4, 8, 16
    myIMU.settings.accelSampleRate = 1666;  //Hz.  Can be: 13, 26, 52, 104, 208, 416, 833, 1666, 3332, 6664, 13330
    myIMU.settings.accelBandWidth = 400;  //Hz.  Can be: 50, 100, 200, 400;

    // BLE Setup 
    //Set local name and service UUID
    BLE.setDeviceName("XIAOB");
    BLE.setLocalName("XIAO");
    BLE.setAdvertisedService(bleService);
    //add characteristic to the service 
    bleService.addCharacteristic(predictionChar);
    bleService.addCharacteristic(offcount);
    bleService.addCharacteristic(speed3count);
    bleService.addCharacteristic(speed7count);
    bleService.addCharacteristic(uncertaincount);
    bleService.addCharacteristic(anomolycount);
    //add service
    BLE.addService(bleService);
    //set initial values for the characteristics
    predictionChar.writeValue("None");
    offcount.writeValue(0);
    speed3count.writeValue(0);
    speed7count.writeValue(0);
    uncertaincount.writeValue(0);
    anomolycount.writeValue(0);    
    //start advertising
    BLE.advertise();


    // comment out the below line to cancel the wait for USB connection (needed for native USB)
    while (!Serial);

    Serial.println("Edge Impulse Inferencing Demo");

    if (!myIMU.begin()) {
        ei_printf("Failed to initialize IMU!\r\n");
    }
    else {
        ei_printf("IMU initialized\r\n");
    }

    if (EI_CLASSIFIER_RAW_SAMPLES_PER_FRAME != 3) {
        ei_printf("ERR: EI_CLASSIFIER_RAW_SAMPLES_PER_FRAME should be equal to 3 (the 3 sensor axes)\n");
        return;
    }

    inference_thread.start(mbed::callback(&run_inference_background));
}

/**
 * @brief Return the sign of the number
 * 
 * @param number 
 * @return int 1 if positive (or 0) -1 if negative
 */
float ei_get_sign(float number) {
    return (number >= 0.0) ? 1.0 : -1.0;
}

/**
 * @brief      Run inferencing in the background.
 */
void run_inference_background()
{


  // wait until we have a full buffer
  delay((EI_CLASSIFIER_INTERVAL_MS * EI_CLASSIFIER_RAW_SAMPLE_COUNT) + 100);

  // This is a structure that smoothens the output result
  // With the default settings 70% of readings should be the same before classifying.
  ei_classifier_smooth_t smooth;
  ei_classifier_smooth_init(&smooth, 10 /* no. of readings */, 7 /* min. readings the same */, 0.8 /* min. confidence */, 0.3 /* max anomaly */);

  while (1) {
      // copy the buffer
      memcpy(inference_buffer, buffer, EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE * sizeof(float));

      // Turn the raw buffer in a signal which we can the classify
      signal_t signal;
      int err = numpy::signal_from_buffer(inference_buffer, EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE, &signal);
      if (err != 0) {
          ei_printf("Failed to create signal from buffer (%d)\n", err);
          return;
      }

      // Run the classifier
      ei_impulse_result_t result = { 0 };

      err = run_classifier(&signal, &result, debug_nn);
      if (err != EI_IMPULSE_OK) {
          ei_printf("ERR: Failed to run classifier (%d)\n", err);
          return;
      }

      // print the predictions
      ei_printf("Predictions ");
      ei_printf("(DSP: %d ms., Classification: %d ms., Anomaly: %d ms.)",
          result.timing.dsp, result.timing.classification, result.timing.anomaly);
      ei_printf(": ");

      // ei_classifier_smooth_update yields the predicted label
      const char *prediction = ei_classifier_smooth_update(&smooth, &result);
      ei_printf("%s ", prediction);
      // print the cumulative results
      ei_printf(" [ ");
      for (size_t ix = 0; ix < smooth.count_size; ix++) {
          ei_printf("%u", smooth.count[ix]);
          if (ix != smooth.count_size + 1) {
              ei_printf(", ");
          }
          else {
            ei_printf(" ");
          }
      }
      ei_printf("]\n");

      delay(run_inference_every_ms);
  }

  ei_classifier_smooth_free(&smooth);
  
}

/**
* @brief      Get data and run inferencing
*
* @param[in]  debug  Get debug info if true
*/
void loop()
{
  //BLEDevice central = BLE.central();
  
  while (1) {
      // Determine the next tick (and then sleep later)
      uint64_t next_tick = micros() + (EI_CLASSIFIER_INTERVAL_MS * 1000);

      // roll the buffer -3 points so we can overwrite the last one
      numpy::roll(buffer, EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE, -3);

      // read to the end of the buffer
      buffer[EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE - 3]=myIMU.readFloatAccelX();
      buffer[EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE - 2]=myIMU.readFloatAccelY();
      buffer[EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE - 1]=myIMU.readFloatAccelZ();

      for (int i = 0; i < 3; i++) {
          if (fabs(buffer[EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE - 3 + i]) > MAX_ACCEPTED_RANGE) {
              buffer[EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE - 3 + i] = ei_get_sign(buffer[EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE - 3 + i]) * MAX_ACCEPTED_RANGE;
          }
      }

      //buffer[EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE - 3] *= CONVERT_G_TO_MS2;
      //buffer[EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE - 2] *= CONVERT_G_TO_MS2;
      //buffer[EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE - 1] *= CONVERT_G_TO_MS2;

      // and wait for next tick
      uint64_t time_to_wait = next_tick - micros();
      delay((int)floor((float)time_to_wait / 1000.0f));
      delayMicroseconds(time_to_wait % 1000);
  }
  
}

Second Problem:

This is the code for the non- contineous inferacing. I’m able to connect the xiao via bluetooth, however, the inferacing results do not print out. I’m not updating any characteristics, I just want to see if bluetooth and printing results can work together. This code (for debugging purposes) runs until Serial.println(“3”) in void_loop and does not reach to Serialprintln(“4”).

/* Edge Impulse ingestion SDK
 * Copyright (c) 2022 EdgeImpulse Inc.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 *
 */

/* Includes ---------------------------------------------------------------- */
#include <AnomolyDetect_inferencing.h>
#include <LSM6DS3.h> //Click here to get the library: http://librarymanager/All#Arduino_LSM9DS1
#include <ArduinoBLE.h> 

/* Constant defines -------------------------------------------------------- */
#define CONVERT_G_TO_MS2    9.80665f
#define MAX_ACCEPTED_RANGE  2.0f        // starting 03/2022, models are generated setting range to +-2, but this example use Arudino library which set range to +-4g. If you are using an older model, ignore this value and use 4.0f instead

/*
 ** NOTE: If you run into TFLite arena allocation issue.
 **
 ** This may be due to may dynamic memory fragmentation.
 ** Try defining "-DEI_CLASSIFIER_ALLOCATION_STATIC" in boards.local.txt (create
 ** if it doesn't exist) and copy this file to
 ** `<ARDUINO_CORE_INSTALL_PATH>/arduino/hardware/<mbed_core>/<core_version>/`.
 **
 ** See
 ** (https://support.arduino.cc/hc/en-us/articles/360012076960-Where-are-the-installed-cores-located-)
 ** to find where Arduino installs cores on your machine.
 **
 ** If the problem persists then there's not enough memory for this model and application.
 */

/* Private variables ------------------------------------------------------- */
static bool debug_nn = false; // Set this to true to see e.g. features generated from the raw signal
LSM6DS3 myIMU(I2C_MODE, 0x6A);

// Service UUID
BLEService bleService("19B10000-E8F2-537E-4F6C-D104768A1214"); 

// Characteristic UUID 
BLEStringCharacteristic predictionChar("a5e93786-602d-4769-932f-fc62699df97a", BLERead | BLEWrite | BLENotify, 100 );
BLEFloatCharacteristic offcount("be574fe6-9bdc-4f88-abde-4393ec367590", BLERead | BLEWrite | BLENotify);
BLEFloatCharacteristic speed3count("3daa9df0-5ca4-4289-ab23-7c7be9205af6", BLERead | BLEWrite | BLENotify);
BLEFloatCharacteristic speed7count("fae59b4b-ef27-437b-8d2c-35ca21218dc8", BLERead | BLEWrite | BLENotify);
BLEFloatCharacteristic uncertaincount("609a46e0-17b8-4974-86f7-e9f33569a08f", BLERead | BLEWrite | BLENotify);
BLEFloatCharacteristic anomolycount("de4fd609-4981-4c25-8c60-c6cb0640f2d5", BLERead | BLEWrite | BLENotify);

/**
* @brief      Arduino setup function
*/
void setup()
{
    // put your setup code here, to run once:
    Serial.begin(115200);
    delay(2000);

    // Accelerometer Setup 
    myIMU.settings.accelRange = 4;      //Max G force readable.  Can be: 2, 4, 8, 16
    myIMU.settings.accelSampleRate = 1666;  //Hz.  Can be: 13, 26, 52, 104, 208, 416, 833, 1666, 3332, 6664, 13330
    myIMU.settings.accelBandWidth = 400;  //Hz.  Can be: 50, 100, 200, 400;





    // comment out the below line to cancel the wait for USB connection (needed for native USB)
    while (!Serial);
    Serial.println("Edge Impulse Inferencing Demo");
    
    if (!BLE.begin()) {   // initialize BLE
      Serial.println("starting BLE failed!");
      while (1);
    }
        // BLE Setup 
    //Set local name and service UUID
    BLE.setDeviceName("XIAOB");
    BLE.setLocalName("XIAO");
    BLE.setAdvertisedService(bleService);
    //add characteristic to the service 
    bleService.addCharacteristic(predictionChar);
    bleService.addCharacteristic(offcount);
    bleService.addCharacteristic(speed3count);
    bleService.addCharacteristic(speed7count);
    bleService.addCharacteristic(uncertaincount);
    bleService.addCharacteristic(anomolycount);
    //add service
    BLE.addService(bleService);
    //set initial values for the characteristics
    predictionChar.writeValue("None");
    offcount.writeValue(0);
    speed3count.writeValue(0);
    speed7count.writeValue(0);
    uncertaincount.writeValue(0);
    anomolycount.writeValue(0);    
    //start advertising
    BLE.advertise();

    if (!myIMU.begin()) {
        ei_printf("Failed to initialize IMU!\r\n");
    }
    else {
        ei_printf("IMU initialized\r\n");
    }

    if (EI_CLASSIFIER_RAW_SAMPLES_PER_FRAME != 3) {
        ei_printf("ERR: EI_CLASSIFIER_RAW_SAMPLES_PER_FRAME should be equal to 3 (the 3 sensor axes)\n");
        return;
    }
}

/**
 * @brief Return the sign of the number
 * 
 * @param number 
 * @return int 1 if positive (or 0) -1 if negative
 */
float ei_get_sign(float number) {
    return (number >= 0.0) ? 1.0 : -1.0;
}

/**
* @brief      Get data and run inferencing
*
* @param[in]  debug  Get debug info if true
*/
void loop()
{
    

    BLEDevice central = BLE.central();

    if (central.discoverAttributes()) {
      Serial.print("Connected to central: ");
      Serial.println(central.address());

      while (central.connected()) {
        ei_printf("\nStarting inferencing in 1 seconds...\n");

        delay(1000);

        ei_printf("Sampling...\n");

        // Allocate a buffer here for the values we'll read from the IMU
        float buffer[EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE] = { 0 };

        for (size_t ix = 0; ix < EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE; ix += 3) {
            // Determine the next tick (and then sleep later)
            uint64_t next_tick = micros() + (EI_CLASSIFIER_INTERVAL_MS * 1000);
            Serial.println("1");
            buffer[ix + 0] = myIMU.readFloatAccelX();
            buffer[ix + 1] = myIMU.readFloatAccelY();
            buffer[ix + 2] = myIMU.readFloatAccelZ();
            Serial.println("2");
            //IMU.readAcceleration(buffer[ix], buffer[ix + 1], buffer[ix + 2]);

            for (int i = 0; i < 3; i++) {
                if (fabs(buffer[ix + i]) > MAX_ACCEPTED_RANGE) {
                    buffer[ix + i] = ei_get_sign(buffer[ix + i]) * MAX_ACCEPTED_RANGE;
                }
            }
            Serial.println("3");
            //buffer[ix + 0] *= CONVERT_G_TO_MS2;
            //buffer[ix + 1] *= CONVERT_G_TO_MS2;
            //buffer[ix + 2] *= CONVERT_G_TO_MS2;

            delayMicroseconds(next_tick - micros());
            Serial.println("4");
        }

        Serial.println("5");
        // Turn the raw buffer in a signal which we can the classify
        signal_t signal;
        int err = numpy::signal_from_buffer(buffer, EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE, &signal);
        if (err != 0) {
            ei_printf("Failed to create signal from buffer (%d)\n", err);
            return;
        }

        // Run the classifier
        ei_impulse_result_t result = { 0 };
        Serial.println("5");
        err = run_classifier(&signal, &result, debug_nn);
        if (err != EI_IMPULSE_OK) {
            ei_printf("ERR: Failed to run classifier (%d)\n", err);
            return;
        }

        // print the predictions
        ei_printf("Predictions ");
        ei_printf("(DSP: %d ms., Classification: %d ms., Anomaly: %d ms.)",
            result.timing.dsp, result.timing.classification, result.timing.anomaly);
        ei_printf(": \n");
        for (size_t ix = 0; ix < EI_CLASSIFIER_LABEL_COUNT; ix++) {
            ei_printf("    %s: %.5f\n", result.classification[ix].label, result.classification[ix].value);
        }
        Serial.println("6");
    #if EI_CLASSIFIER_HAS_ANOMALY == 1
        ei_printf("    anomaly score: %.3f\n", result.anomaly);
    #endif
      }
    }
}

Thanks.

Joeri · March 4, 2023, 10:58am

I have no experience with Xiao BLE Sense. Still, maybe you can look at: Workshop using SparkFun MicroMod nRF52840. In the Arduino examples, there is an example: inference-over-bluetooth.ino

AiPTL · March 8, 2023, 3:15pm

Hello,

I’ve attempted to adapt that part of the code using the link you provided and still the code gets stuck at the same stage once it’s uploaded. I think this may be a hardware limitation of the saeed xiao such that it cannot run the edge impulse code and operate BLE at the same time. Good learning point for those looking to do the same thing as me.

Joeri · March 8, 2023, 7:30pm

The Micromod nRF52840 and Xiao BLE have the same processor. If it works on the Micromod nRF52840, I would expect it to work on the Xiao.

I notice in your code the following:

Do you connect the device to a USB to display the results on the terminal and simultaneously make a connection over BLE?

AiPTL · March 15, 2023, 8:21am

Apologies for the late reply.

Yes I connected my device to the computer via USB whilst it attempts to connect via bluetooth (which it can connect but then the output in the serial monitor gets stuck on a specific message if you know what i mean)

Joeri · March 17, 2023, 9:16am

@AiPTL

I haven’t much experience with using the BLE function. But it should be possible; check Connecting Nano 33 BLE Devices over Bluetooth. The Xiao has some MCU as the Arduino Nano.

I suggest trying to get it working without connecting to the BLE, only the terminal. If this works, try running the BLE without printing to the terminal, but use, for example, the nRF Connect. If I remember it is shown in Workshop using SparkFun MicroMod nRF52840.

FYI: How to Send Data between PC and Arduino using Bluetooth LE