Universal header+binary payload data generator

I am an embedded programmer and I do not know JS or python. I would like to prepare the universal data generator for the ingestion service.

I do have data.bin in a separate file and I need a JS script to join in according to the Binary payloads (e.g. for audio) requirements in https://docs.edgeimpulse.com/reference#data-acquisition-format

I struggle to join the header and binary data file.

let header = {
protected: {
ver: “v1”,
alg: “HS256”,
iat: Math.floor(Date.now() / 1000) // epoch time, seconds since 1970
signature: emptySignature,
payload: {
device_name: “00:00:00:00:00:01”,
device_type: “Board name”,
interval_ms: 0.001,
sensors: [
{ name: “raw”, units: “-” },
values: [
[ “Ref-Binary-i16” ],
[ 0xFF ]
console.log('Header: ', Buffer.from(JSON.stringify(header)));

// read binary file
let binary = fs.readFileSync(Path.join(__dirname, ‘data.bin’));
console.log('Binary: ', binary.slice(0));

// concat header and binary file
let data = Buffer.concat([Buffer.from(JSON.stringify(header)), binary.slice(0)]);
console.log('Data: ', data);

// now calculate the HMAC and fill in the signature
let hmac = crypto.createHmac(‘sha256’, HMAC_KEY);
let signature = hmac.digest().toString(‘hex’);

// update the signature in the message and re-encode
header.signature = signature;
data = Buffer.concat([Buffer.from(JSON.stringify(header)), binary.slice(0)]);

// now upload the buffer to Edge Impulse
request.post(‘https://ingestion.edgeimpulse.com/api/training/data’, {
headers: {
‘x-api-key’: API_KEY,
‘x-file-name’: ‘raw_data’,
‘Content-Type’: ‘application/json’
body: data,
encoding: ‘binary’
}, function (err, response, body) {
if (err) return console.error(‘Request failed’, err);

console.log(‘Uploaded file to Edge Impulse’, response.statusCode, body);

Can you please help me?

@hwidvorakinfo If you’re constructing JSON you probably don’t want to do anything with the binary payloads. They’re made for embedded systems and wrap around CBOR. So for audio your payload would just be:

let header = {
    protected: {
        ver: "v1",
        alg: "HS256",
        iat: Math.floor(Date.now() / 1000) // epoch time, seconds since 1970
    signature: emptySignature,
    payload: {
        device_name: "00:00:00:00:00:01",
        device_type: "Board name",
        interval_ms: 0.0625,
        sensors: [{
            name: "audio",
            units: "wav"
        values: [
            0x01, 0x02, 0x03

Where values is int16_t values (so not raw bytes) (PCM data you pull from your audio driver).

The Linux SDK has examples in C++, Python, Node.js and Go that you can run easily, I guess the C++ one is closest to your heart. https://docs.edgeimpulse.com/docs/edge-impulse-for-linux

@janjongboom thank you for the reaction

Please, what does it mean point 3 - Write the CBOR header?

Does it mean I have to encode to JSON the text form header and then with point 4 to add raw data?

  1. Limit your data to a single sensor.
  2. Set the value of the values field to: [ "Ref-Binary-i16" ] (so a string within an array, where i16 indicates int16, alternatively you can use u16 for uint16 or f32 for float32).
  3. Write the CBOR header. Note that this needs to be a valid CBOR structure, and that it requires termination with 0xFF. The easiest to achieve this is by writing the values field as an indefinite length array (which needs to be terminated by 0xFF.
  4. Write your payload (little endian).
  • If you need to align your writes the easiest is to pad the Ref-Binary-i16 string with spaces on the right.
  1. Update the signature with the signature for the full file, including header and payload.
  2. Upload the data to the ingestion service with the Content-Type: application/octet-stream header.

Can we take one step back, what are you trying to achieve and one which platform? You mention JavaScript so I assume you’re not running this on a Cortex-M-type device?

I have a custom board based on Cortex-M7 and this device is being used as a datalogger.

The data are being transfered via UART into my computer (macOS).

The data are about to be uploaded via Ingestion service into Edge Impulse Studio.

So I think the best for me is to use Python to combine the CBORheader and convert data values from a file into text format numbers. I believe this will help me https://github.com/edgeimpulse/linux-sdk-python/blob/master/examples/custom/collect.py

Is it possible to make it this way?

@hwidvorakinfo Yes, just use the collect.py script in that case. No need to do anything custom, just stick the values in there and upload :ok_hand: