Maximum Sampling rate for cli-data-forwarder

Hi
I want to understand the maximum sampling rate that is supported by cli-data-forwarder.

Sample Sensor Reading = -0.12,-6.20,-7.90\n
Hex Value = 2D302E31322C2D362E32302C2D372E39300D0A

Number of bytes per transmission = 20 Bytes Approx

Number of baud / Byte(8N1) = 10

Number of bauds = 20 * 10 = 200

Transmission rate = 115200

Max Sampling rate = 115200/200 = 576 Hz

  1. Is time delay between \r\n used for the calculation of sampling time
  2. What will be the maximum safe sampling frequency recommended for using with the cli-data-forwarder.
  3. Should timers be used to add delay between each sensor data write to UART
  4. Is the Calculation above correct

Hi @paulphilip, the actual rate depends a bit on the underlying serial connection. E.g. do you use flow control, do you use buffers between your code and the UART or use an unbuffered serial, how much time does your platform spend formatting your string, etc. But globally your numbers are OK. My suggestion would be to sample at a set interval (e.g. 200Hz) and busy-loop (or use a timer) until the next tick. Example on how we do this on the ST IoT Discovery Kit: https://github.com/edgeimpulse/firmware-st-b-l475e-iot01a/blob/master/source/ingestion-sdk-platform/mbed/ei_sampler.h#L147 . This mitigates things where you might send more bytes (and thus spend more time) and thus have imprecise intervals (e.g. 100.00,100.00,100.00 is more bytes than 0.00,0.00,0.00).

The calculation in the forwarder is done like this: after initial connection the data forwarder samples data for 500 ms, then looks at the number of lines that came in in that time and it will keep that as the sample rate. If the sample rate differs more than 10Hz from the last time you used the forwarder it’ll update the internal sample rate, otherwise it’ll stay the same (even if there are small differences). We’ve done it that way because a consistent sample rate is important in your dataset (even if it’s a few Hz off).

Thank you so much for your comprehensive answer in such a short time. :smile:

1 Like

@janjongboom Is there a limit on the baud rates supported by edge-impulse-data-forwarder?

1 Like

@Hardik, yes, around 400Hz I’d say. That’s the maximum speed you can push through at baud rate 115,200. For anything else I’d recommend to write the data into the data acquisition format and push through the ingestion service straight away (which is what we do on fully supported dev boards).

1 Like

@janjongboom I tried to create a CBOR file by using the ingestion-sdk from : https://github.com/edgeimpulse/ingestion-sdk-c

I hit following built errors

inc/sensor_aq.h:271:131: error: expected ‘;’, ‘,’ or ‘)’ before ‘=’ token
 int sensor_aq_init(sensor_aq_ctx *ctx, sensor_aq_payload_info *payload_info, EI_SENSOR_AQ_STREAM *stream, bool allow_empty_stream = false) {

I think that’s because you cannot set default values for parameters in C.

To fix this, I defined allow_empty_stream outside of function arguments:

int sensor_aq_init(sensor_aq_ctx *ctx, sensor_aq_payload_info *payload_info, EI_SENSOR_AQ_STREAM *stream) {
    bool allow_empty_stream = false;

After that, I added my hmac-key from the project to sensor_aq_init_mbedtls_hs256_ context function in test/main.c

I tried to upload generated binary file to my project using:
edge-impulse-uploader .\ingestion_data.cbor

Edge Impulse uploader v1.6.7
Endpoints:
    API:       https://studio.edgeimpulse.com/v1
    Ingestion: https://ingestion.edgeimpulse.com

Upload configuration:
    Label:     Not set, will be infered from file name
    Category:  training
    Project ID:  2776

[1/1] Failed to upload .\ingestion_data.cbor Invalid message body, was specified to be CBOR but could not decode message (Failed to parse)

I also tried to decode the file on cbor.me and output as text and upload it. I face the same error.

I am not sure what i am doing wrong.

I am copy pasting my file here:

{
"protected": {
	"ver": "v1", 
	"alg": "HS256"}, 
	"signature": "my-generated-signature", 
	"payload": {
	"device_name": "ac:87:a3:0a:2d:1b", 
	"device_type": "DISCO-L475VG-IOT01A", 
	"interval_ms": 10.0, 
	"sensors": [
		{"name": "accX", "units": "m/s2"}, 
		{"name": "accY", "units": "m/s2"}, 
		{"name": "accZ", "units": "m/s2"}], 
		"values": [
			[-9.8100004196167, 0.029999999329447746, 1.2100000381469727], 
			[-9.829999923706055, 0.03999999910593033, 1.2799999713897705], 
			[-9.119999885559082, 0.029999999329447746, 1.2300000190734863], 
			[-9.140000343322754, 0.009999999776482582, 1.25]
		]
	}
}

Hi @Hardik, sorry the example broke. I’ve pushed a new version to ingestion-sdk-c that compiles. If this still yields invalid data for you could you let me know?

@janjongboom Thanks for the quick fix. But my error with uploading cbor file still remains.

@janjongboom I did one more test. I copied the sample cbor file from : https://docs.edgeimpulse.com/reference#data-acquisition-format and replaced the signature with the one I generated . I still see the same error.

My guess would be that something is wrong with the way I am generating the signature.

Could you paste the raw cbor bytes here? Note that the signature needs to be 64 characters long (32 bytes encoded in hex as 2 characters each)

@janjongboom

a3 69 70 72 6f 74 65 63 74 65 64 a2 63 76 65 72 62 76 31 63 61 6c 67 65 48 53 32 35 36 69 73 69 67 6e 61 74 75 72 65 78 40 62 32 34 34 36 39 61 34 34 34 33 65 65 38 62 31 31 33 61 34 39 31 36 63 37 65 62 62 62 32 62 65 33 63 66 61 39 31 34 64 34 30 34 32 30 31 31 37 62 33 62 36 62 63 64 63 32 36 62 62 33 30 61 62 67 70 61 79 6c 6f 61 64 a5 6b 64 65 76 69 63 65 5f 6e 61 6d 65 71 61 63 3a 38 37 3a 61 33 3a 30 61 3a 32 64 3a 31 62 6b 64 65 76 69 63 65 5f 74 79 70 65 73 44 49 53 43 4f 2d 4c 34 37 35 56 47 2d 49 4f 54 30 31 41 6b 69 6e 74 65 72 76 61 6c 5f 6d 73 f9 49 00 67 73 65 6e 73 6f 72 73 83 a2 64 6e 61 6d 65 64 61 63 63 58 65 75 6e 69 74 73 64 6d 2f 73 32 a2 64 6e 61 6d 65 64 61 63 63 59 65 75 6e 69 74 73 64 6d 2f 73 32 a2 64 6e 61 6d 65 64 61 63 63 5a 65 75 6e 69 74 73 64 6d 2f 73 32 66 76 61 6c 75 65 73 9f 83 fa c1 1c f5 c3 fa 3c f5 c2 8f fa 3f 9a e1 48 83 fa c1 1d 47 ae fa 3d 23 d7 0a fa 3f a3 d7 0a 83 fa c1 11 eb 85 fa 3c f5 c2 8f fa 3f 9d 70 a4 83 fa c1 12 3d 71 fa 3c 23 d7 0a f9 3d 00 ff

@Hardik that CBOR structure is valid according to cbor.me, and also uploads for me:

@janjongboom Thanks for checking this. It still does not upload for me. Beginning to wonder what I am missing here. Some project setting?

Hmm… this goes through our ingestion API and there are no project-specific settings in that… This is how I turn your array into a CBOR file:

test.js

const fs = require('fs');

var data = 'a3 69 70 72 6f 74 65 63 74 65 64 a2 63 76 65 72 62 76 31 63 61 6c 67 65 48 53 32 35 36 69 73 69 67 6e 61 74 75 72 65 78 40 62 32 34 34 36 39 61 34 34 34 33 65 65 38 62 31 31 33 61 34 39 31 36 63 37 65 62 62 62 32 62 65 33 63 66 61 39 31 34 64 34 30 34 32 30 31 31 37 62 33 62 36 62 63 64 63 32 36 62 62 33 30 61 62 67 70 61 79 6c 6f 61 64 a5 6b 64 65 76 69 63 65 5f 6e 61 6d 65 71 61 63 3a 38 37 3a 61 33 3a 30 61 3a 32 64 3a 31 62 6b 64 65 76 69 63 65 5f 74 79 70 65 73 44 49 53 43 4f 2d 4c 34 37 35 56 47 2d 49 4f 54 30 31 41 6b 69 6e 74 65 72 76 61 6c 5f 6d 73 f9 49 00 67 73 65 6e 73 6f 72 73 83 a2 64 6e 61 6d 65 64 61 63 63 58 65 75 6e 69 74 73 64 6d 2f 73 32 a2 64 6e 61 6d 65 64 61 63 63 59 65 75 6e 69 74 73 64 6d 2f 73 32 a2 64 6e 61 6d 65 64 61 63 63 5a 65 75 6e 69 74 73 64 6d 2f 73 32 66 76 61 6c 75 65 73 9f 83 fa c1 1c f5 c3 fa 3c f5 c2 8f fa 3f 9a e1 48 83 fa c1 1d 47 ae fa 3d 23 d7 0a fa 3f a3 d7 0a 83 fa c1 11 eb 85 fa 3c f5 c2 8f fa 3f 9d 70 a4 83 fa c1 12 3d 71 fa 3c 23 d7 0a f9 3d 00 ff'

var b = Buffer.from(data.split(' ').map(n => parseInt(n, 16)))

fs.writeFileSync('test.cbor', b)

Then:

$ node test.js

Then:

janjongboom:~/repos/tmp $ edge-impulse-uploader test.cbor
Edge Impulse uploader v1.7.0
Endpoints:
    API:       https://studio.edgeimpulse.com/v1
    Ingestion: https://ingestion.edgeimpulse.com

Upload configuration:
    Label:     Not set, will be infered from file name
    Category:  training
    Project ID:  2657

[1/1] Uploading test.cbor OK (237 ms)

Done. Files uploaded successful: 1. Files that failed to upload: 0.

:confused:

@janjongboom Thank you so much!

I was simply copy pasting the binary data into a file and sending it. I also used cbor.me to decode the binary data and copy into a file and send