Json to Cbor then to Base64 to upload data with Serial Daemon

Help
#1

I am working with the Serial Daemon, trying to port a Wisblock Dev board to be operable through the Web Interface, almost everything working but having problems with the buffer encodings, I have my data formated as a Json, according to my research, I need to encode this to Cbor and then to base64, this is the problem.

This is my data:
{
“protected”: {
“ver”: “v1”,
“alg”: “HS256”
},
“signature”: “efa7140df9b04e9467b8d3248…3324c8437ea90c366ab301f0de0ed6”,
“payload”: {
“device_name”: “EA:0C:FF:14:2F:1A”,
“device_type”: “SEEED_WIO_TERMINAL”,
“interval_ms”: 16,
“sensors”: [
{
“name”: “Illumination”,
“units”: “na”
}
],
“values”: [
5,
6,
5,
6,
7,
7,
6,
6,
6,
6,
5,
5,
6,
6,
]
}
}

This is a functional base64 of a similar data sample of what I want to get:

o2lwcm90ZWN0ZWSiY3ZlcmJ2MWNhbGdlSFMyNTZpc2lnbmF0dXJleEBlZmE3MTQwZGY5YjA0ZTk0NjdiOGQzMjQ4NjljYjUxN2UzMzMyNGM4NDM3ZWE5MGMzNjZhYjMwMWYwZGUwZWQ2Z3BheWxvYWSla2RldmljZV9uYW1lcUVBOjBDOkZGOjE0OjJGOjFBa2RldmljZV90eXBlclNFRUVEX1dJT19URVJNSU5BTGtpbnRlcnZhbF9tc/lMAGdzZW5zb3JzgaJkbmFtZWxJbGx1bWluYXRpb25ldW5pdHNibmFmdmFsdWVzn/lFAPlGAPlFAPlGAPlHAPlHAPlGAPlGAPlGAPlGAPlFAPlFAPlGAPlGAPlFAPlGAPlHAPlGAPlGAPlHAPlFAPlGAPlHAPlFAPlFAPlFAPlGAPlGAPlFAPlGAPlHAPlGAPlGAPlGAPlGAPlHAPlFAPlGAPlFAPlGAPlGAPlGAPlGAPlGAPlGAPlGAPlFAPlGAPlGAPlGAPlGAPlGAPlGAPlGAPlGAPlGAPlFAPlGAPlGAPlGAPlFAPlGAPlHAPlGAPlGAPlHAPlGAPlHAPlFAPlFAPlFAPlGAPlGAPlGAPlGAPlHAPlGAPlGAPlGAPlGAPlGAPlFAPlFAPlGAPlGAPlGAPlFAPlIAPlHAPlHAPlGAPlFAPlGAPlFAPlHAPlHAPlGAPlGAPlGAPlHAPlFAPlGAPlGAPlGAPlFAPlFAPlFAPlGAPlGAPlHAPlHAPlIAPlGAPlGAPlGAPlHAPlGAPlFAPlGAPlGAPlFAPlGAPlHAPlGAPlGAPlGAP8=

I’ve tried different things but I cant make my Json to look like the base64 example even by encoding it with some Arduino libraries out there or online encoders.

Please help, I think that this “base64” string is a combination of Cbor and base64.

ERROR when sampling audio from ESP32 - Timeout or Undeterminated nesting
#2

Hi mcmchris,

To confirm, you’re basically trying to implement the serial interface from our daemon to your wisblock board?

In the firmware we are indeed taking a CBOR formatted sample -> base64 encoding -> and sending over serial to the daemon:
CBOR example code
base64 encoding example sequence
base64 implementation on arduino

The daemon then immediately decodes back to CBOR and uploads that:

So, if you want to interface with our serial daemon exactly, then you will need to match this encoding on the firmware side. I’m not sure at this point where the difference in your implementation lies, but a good starting point would be verifying that your CBOR is valid, like we do in our ingestion SDK example:
https://docs.edgeimpulse.com/reference/c-sdk-usage-guide#usage

And you should be able to upload the CBOR bytes directly via our API:
https://docs.edgeimpulse.com/reference/ingestion-api#example-usage

Then if it looks good, comparing the output of your existing implementation to the base64_encode implementation from the Arduino firmware (linked above).

The other thing worth mentioning is that if any devices/gateways in your network are wifi connected, and you get the json/cbor file to that device, you can upload the CBOR directly via the API, skipping the daemon, and the requirement for a host system entirely.

1 Like
#3

Hi, thanks for your response,
I know how to make it work by WiFi, I am used to the Ingestion service and Http data upload with the ESP32 and boards alike.

The thing is that I am trying to use a board without WiFi (WisBlock).

I tried this code with my ESP32 (https://docs.edgeimpulse.com/reference/c-sdk-usage-guide?_ga=2.44189990.326953493.1640188637-884147057.1636064879#usage) that returns this (CBOR):

a3 69 70 72 6f 74 65 63 74 65 64 a2 63 76 65 72 62 76 31 63 61 6c 67 65 48 53 32 35 36 69 73 69 67 6e 61 74 75 72 65 78 40 62 61 61 61 30 37 63 30 33 62 36 32 61 39 34 32 66 66 64 31 34 35 36 34 32 39 37 32 32 33 66 35 61 37 30 34 39 38 30 30 31 32 33 61 39 65 38 65 33 38 39 63 35 38 33 34 65 36 65 35 35 30 36 63 67 70 61 79 6c 6f 61 64 a5 6b 64 65 76 69 63 65 5f 6e 61 6d 65 77 39 35 3a 31 34 3a 45 45 3a 33 33 3a 45 30 3a 34 37 3a 35 34 3a 32 39 6b 64 65 76 69 63 65 5f 74 79 70 65 70 57 69 73 42 6c 6f 63 6b 5f 52 41 4b 34 36 33 31 6b 69 6e 74 65 72 76 61 6c 5f 6d 73 f9 49 00 67 73 65 6e 73 6f 72 73 83 a2 64 6e 61 6d 65 64 61 63 63 58 65 75 6e 69 74 73 64 6d 2f 73 32 a2 64 6e 61 6d 65 64 61 63 63 59 65 75 6e 69 74 73 64 6d 2f 73 32 a2 64 6e 61 6d 65 64 61 63 63 5a 65 75 6e 69 74 73 64 6d 2f 73 32 66 76 61 6c 75 65 73 9f 83 fa c1 1c f5 c3 fa 3c f5 c2 8f fa 3f 9a e1 48 83 fa c1 1d 47 ae fa 3d 23 d7 0a fa 3f a3 d7 0a 83 fa c1 11 eb 85 fa 3c f5 c2 8f fa 3f 9d 70 a4 83 fa c1 12 3d 71 fa 3c 23 d7 0a f9 3d 00 ff

Also uploaded to my Edge Impulse account and everything fine:

image
image878×642 20.7 KB

I decoded this Cbor and got this on cbor.me :
{“protected”: {“ver”: “v1”, “alg”: “HS256”}, “signature”: “baaa07c03b62a942ffd14564297223f5a7049800123a9e8e389c5834e6e5506c”, “payload”: {“device_name”: “95:14:EE:33:E0:47:54:29”, “device_type”: “WisBlock_RAK4631”, “interval_ms”: 10.0, “sensors”: [{“name”: “accX”, “units”: “m/s2”}, {“name”: “accY”, “units”: “m/s2”}, {“name”: “accZ”, “units”: “m/s2”}], “values”: [[-9.8100004196167, 0.029999999329447746, 1.2100000381469727], [-9.829999923706055, 0.03999999910593033, 1.2799999713897705], [-9.119999885559082, 0.029999999329447746, 1.2300000190734863], [-9.140000343322754, 0.009999999776482582, 1.25]]}}

Everything of this is right, the thing is when I try to take the Cbor (from above), encode it on base64 (by an Arduino Lib or using base64encode web) and try to upload it with the Daemon.

Note: encoding the Cbor to base64 returns me this:
YTM2OTcwNzI2Zjc0NjU2Mzc0NjU2NGEyNjM3NjY1NzI2Mjc2MzE2MzYxNmM2NzY1NDg1MzMyMzUzNjY5NzM2OTY3NmU2MTc0NzU3MjY1Nzg0MDYyNjE2MTYxMzAzNzYzMzAzMzYyMzYzMjYxMzkzNDMyNjY2NjY0MzEzNDM1MzYzNDMyMzkzNzMyMzIzMzY2MzU2MTM3MzAzNDM5MzgzMDMwMzEzMjMzNjEzOTY1Mzg2NTMzMzgzOTYzMzUzODMzMzQ2NTM2NjUzNTM1MzAzNjYzNjc3MDYxNzk2YzZmNjE2NGE1NmI2NDY1NzY2OTYzNjU1ZjZlNjE2ZDY1NzczOTM1M2EzMTM0M2E0NTQ1M2EzMzMzM2E0NTMwM2EzNDM3M2EzNTM0M2EzMjM5NmI2NDY1NzY2OTYzNjU1Zjc0Nzk3MDY1NzA1NzY5NzM0MjZjNmY2MzZiNWY1MjQxNGIzNDM2MzMzMTZiNjk2ZTc0NjU3Mjc2NjE2YzVmNmQ3M2Y5NDkwMDY3NzM2NTZlNzM2ZjcyNzM4M2EyNjQ2ZTYxNmQ2NTY0NjE2MzYzNTg2NTc1NmU2OTc0NzM2NDZkMmY3MzMyYTI2NDZlNjE2ZDY1NjQ2MTYzNjM1OTY1NzU2ZTY5NzQ3MzY0NmQyZjczMzJhMjY0NmU2MTZkNjU2NDYxNjM2MzVhNjU3NTZlNjk3NDczNjQ2ZDJmNzMzMjY2NzY2MTZjNzU2NTczOWY4M2ZhYzExY2Y1YzNmYTNjZjVjMjhmZmEzZjlhZTE0ODgzZmFjMTFkNDdhZWZhM2QyM2Q3MGFmYTNmYTNkNzBhODNmYWMxMTFlYjg1ZmEzY2Y1YzI4ZmZhM2Y5ZDcwYTQ4M2ZhYzExMjNkNzFmYTNjMjNkNzBhZjkzZDAwZmY=

But when I try to upload this through the daemon, this is what it says:
statusCode: 400,
error: ‘Invalid message body, was specified to be CBOR but could not decode message (Failed to parse)’,

#4

Now the Edge Impulse debug says:
‘Missing protected header’

#5

Perfect thanks for testing. So compared to that base64 arduino library/website - how does our implementation compare?:

#6

Hi,
I am using this method (code) to base64 encode, but Edge Impulse says ‘Missing protected header’.

Please clarify me something, my data must be initially a json, then encoded to cbor and finally to base64 and then upload? this is what I am doing.

By researching on Wio Terminal Open source Firmware I noticed that they kind of create a header or something but Edge Impulse documentation is not clear in this point.

PD: this is the string I am trying to upload to Edge with Daemon:

NzkwMWY4N2IyMjcwNzI2Zjc0NjU2Mzc0NjU2NDIyM2E3YjIyNzY2NTcyMjIzYTIyNzYzMTIyMmMyMjYxNmM2NzIyM2EyMjQ4NTMzMjM1MzYyMjJjMjI2OTYxNzQyMjNhMzEzNjMzMzEzMjMyMzMzNDM5MzA3ZDJjMjI3MzY5Njc2ZTYxNzQ3NTcyNjUyMjNhMjIzOTM0MzIzNjYyMzI2MzM4MzUzNzYyMzY2NTMxNjY2NDMxMzczMzM1MzU2NDMwNjIzMzY0MzQ2MjY1MzgzMTMwMzUzODMzNjEzNTM0MzEzOTM4Mzg2MTM1Mzk2MjMwMzQzNjMxNjQ2MTMyMzY2MzY0NjQzNzM0NjMzMDM2NjQ2NTIyMmMyMjcwNjE3OTZjNmY2MTY0MjIzYTdiMjI2NDY1NzY2OTYzNjU1ZjZlNjE2ZDY1MjIzYTIyMzkzNTNhMzEzNDNhNDU0NTNhMzMzMzNhNDUzMDNhMzQzNzNhMzUzNDNhMzIzOTIyMmMyMjY0NjU3NjY5NjM2NTVmNzQ3OTcwNjUyMjNhMjAyMjU3Njk3MzQyNmM2ZjYzNmI1ZjUyNDE0YjM0MzYzMzMxMjIyYzIyNjk2ZTc0NjU3Mjc2NjE2YzVmNmQ3MzIyM2EzMjMwMmMyMjczNjU2ZTczNmY3MjczMjIzYTIwNWI3YjIwMjI2ZTYxNmQ2NTIyM2EyMjQxNmQ2MjY5NjU2ZTc0MjA0YzY5Njc2ODc0MjIyYzIyNzU2ZTY5NzQ3MzIyM2EyMjRlMmY0MTIyMjA3ZDVkMmMyMjc2NjE2Yzc1NjU3MzIyM2E1YjM3MzAzODJjMzczMTM2MmMzNzMwMzIyYzM3MzIzNDJjMzYzOTMxMmMzNzMxMzIyYzM2MzgzNjJjMzczMDMwMmMzNzMxMzMyYzM2MzkzMjJjMzczMDM2MmMzNzMwMzUyYzM2MzkzNDJjMzczMDMyMmMzNjM5MzAyYzM2MzgzNDJjMzYzOTMxMmMzNzMwMzYyYzM3MzAzOTJjMzczMTM5MmMzNzMwMzQyYzM3MzIzMzJjMzYzOTM4MmMzNjM5MzEyYzM2MzkzNzJjMzczMTMyMmMzNzMwMzYyYzM3MzAzNzJjMzczMjMzMmMzNzMyMzEyYzM2MzgzMzJjMzczMTMzMmMzNzMyMzQyYzM2MzgzOTJjMzYzODM2MmMzNzMwMzAyYzM2MzgzNDJjMzczMTMxMmMzNzMwMzgyYzM2MzkzOTJjMzczMDMzMmMzNzMyMzMyYzM3MzEzNTJjMzYzOTM5MmMzNzMxMzQyYzM3MzAzNDJjMzYzOTM0MmMzNzMwMzEyYzM3MzEzMTJjMzczMjMzNWQ3ZDdkAA==

If you base64 decode it and then cbor decode it, you will have the json with my data.

Please help!

#7

Hey mcmchris,

I see thanks for clarifying. So right the daemon expects base64 encoded cbor, which it immediately decodes.

The protected header is defined in your original json, and from your example base64 looks to be correct:
https://docs.edgeimpulse.com/reference/data-acquisition-format#fields

Can you check what a modified daemon is actually decoding off of the serial connection when reading the buffer? Add a print or log here and run from source:

The only header I’m seeing added in the wio source is when the samples are being written to cbor using our ingestion sdk, but definitely point me to the code you were looking at to clarify.

#8

Hi, I did what you suggest and these are the results:

I am sending this to Edge Impulse through the Daemon:

YTM2OTcwNzI2Zjc0NjU2Mzc0NjU2NGEyNjM3NjY1NzI2Mjc2MzE2MzYxNmM2NzY1NDg1MzMyMzUzNjY5NzM2OTY3NmU2MTc0NzU3MjY1Nzg0MDYyNjE2MTYxMzAzNzYzMzAzMzYyMzYzMjYxMzkzNDMyNjY2NjY0MzEzNDM1MzYzNDMyMzkzNzMyMzIzMzY2MzU2MTM3MzAzNDM5MzgzMDMwMzEzMjMzNjEzOTY1Mzg2NTMzMzgzOTYzMzUzODMzMzQ2NTM2NjUzNTM1MzAzNjYzNjc3MDYxNzk2YzZmNjE2NGE1NmI2NDY1NzY2OTYzNjU1ZjZlNjE2ZDY1NzczOTM1M2EzMTM0M2E0NTQ1M2EzMzMzM2E0NTMwM2EzNDM3M2EzNTM0M2EzMjM5NmI2NDY1NzY2OTYzNjU1Zjc0Nzk3MDY1NzA1NzY5NzM0MjZjNmY2MzZiNWY1MjQxNGIzNDM2MzMzMTZiNjk2ZTc0NjU3Mjc2NjE2YzVmNmQ3M2Y5NDkwMDY3NzM2NTZlNzM2ZjcyNzM4M2EyNjQ2ZTYxNmQ2NTY0NjE2MzYzNTg2NTc1NmU2OTc0NzM2NDZkMmY3MzMyYTI2NDZlNjE2ZDY1NjQ2MTYzNjM1OTY1NzU2ZTY5NzQ3MzY0NmQyZjczMzJhMjY0NmU2MTZkNjU2NDYxNjM2MzVhNjU3NTZlNjk3NDczNjQ2ZDJmNzMzMjY2NzY2MTZjNzU2NTczOWY4M2ZhYzExY2Y1YzNmYTNjZjVjMjhmZmEzZjlhZTE0ODgzZmFjMTFkNDdhZWZhM2QyM2Q3MGFmYTNmYTNkNzBhODNmYWMxMTFlYjg1ZmEzY2Y1YzI4ZmZhM2Y5ZDcwYTQ4M2ZhYzExMjNkNzFmYTNjMjNkNzBhZjkzZDAwZmY=

and the Daemon (with the modifications to print the decoded message) prints this:

body: <Buffer 61 33 36 39 37 30 37 32 36 66 37 34 36 35 36 33 37 34 36 35 36 34 61 32 36 33 37 36 36 35 37 32 36 32 37 36 33 31 36 33 36 31 36 63 36 37 36 35 34 38 … 630 more bytes>

But If I decode the message with an Online decoder the result doesn’t match:

image
image735×622 57.8 KB

The result of this online converter turns in my original Json when decoded on Cbor.me

#9

@meme What you seem to do is base64 encode a string representation of the CBOR. But you should encode the raw bytes. E.g. if you look at what the daemon returns:

61 33 36 39 37 30 37 32 36 66 37 34 36 35 36 33 37 34 36 35 36 34 61 32 36 33 37 36 36 35 37 32 36 32 37 36 33 31 36 33 36 31 36 63 36 37 36 35 34 38

This maps to the string:

a36970726f746563746564a26376657262763163616c676548

But it should map to the bytes 0xa3, 0x69, etc.

A base64 decoder application should not return human-readable strings for the base64 message you pass in.

E.g. this message:

{
    "protected": {
        "ver": "v1",
        "alg": "HS256"
    },
    "signature": "efa7140df9b04e9467b8d3248… 3324 c8437ea90c366ab301f0de0ed6",
    "payload": {
        "device_name": "EA: 0 C: FF: 14: 2 F: 1 A",
        "device_type": "SEEED_WIO_TERMINAL",
        "interval_ms": 16,
        "sensors": [{
            "name": "Illumination",
            "units": "na"
        }],
        "values": [
            5, 6, 5, 6, 7, 7, 6, 6, 6, 6, 5, 5, 6, 6
        ]
    }
}

Maps to the CBOR bytes:

0xa3, 0x69, 0x70, 0x72, 0x6f, 0x74, 0x65, 0x63, 0x74, 0x65, 0x64, 0xa2, 0x63, 0x76, 0x65, 0x72, 0x62, 0x76, 0x31, 0x63, 0x61, 0x6c, 0x67, 0x65, 0x48, 0x53, 0x32, 0x35, 0x36, 0x69, 0x73, 0x69, 0x67, 0x6e, 0x61, 0x74, 0x75, 0x72, 0x65, 0x78, 0x3c, 0x65, 0x66, 0x61, 0x37, 0x31, 0x34, 0x30, 0x64, 0x66, 0x39, 0x62, 0x30, 0x34, 0x65, 0x39, 0x34, 0x36, 0x37, 0x62, 0x38, 0x64, 0x33, 0x32, 0x34, 0x38, 0xe2, 0x80, 0xa6, 0x20, 0x33, 0x33, 0x32, 0x34, 0x20, 0x63, 0x38, 0x34, 0x33, 0x37, 0x65, 0x61, 0x39, 0x30, 0x63, 0x33, 0x36, 0x36, 0x61, 0x62, 0x33, 0x30, 0x31, 0x66, 0x30, 0x64, 0x65, 0x30, 0x65, 0x64, 0x36, 0x67, 0x70, 0x61, 0x79, 0x6c, 0x6f, 0x61, 0x64, 0xa5, 0x6b, 0x64, 0x65, 0x76, 0x69, 0x63, 0x65, 0x5f, 0x6e, 0x61, 0x6d, 0x65, 0x78, 0x19, 0x45, 0x41, 0x3a, 0x20, 0x30, 0x20, 0x43, 0x3a, 0x20, 0x46, 0x46, 0x3a, 0x20, 0x31, 0x34, 0x3a, 0x20, 0x32, 0x20, 0x46, 0x3a, 0x20, 0x31, 0x20, 0x41, 0x6b, 0x64, 0x65, 0x76, 0x69, 0x63, 0x65, 0x5f, 0x74, 0x79, 0x70, 0x65, 0x72, 0x53, 0x45, 0x45, 0x45, 0x44, 0x5f, 0x57, 0x49, 0x4f, 0x5f, 0x54, 0x45, 0x52, 0x4d, 0x49, 0x4e, 0x41, 0x4c, 0x6b, 0x69, 0x6e, 0x74, 0x65, 0x72, 0x76, 0x61, 0x6c, 0x5f, 0x6d, 0x73, 0x10, 0x67, 0x73, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x73, 0x81, 0xa2, 0x64, 0x6e, 0x61, 0x6d, 0x65, 0x6c, 0x49, 0x6c, 0x6c, 0x75, 0x6d, 0x69, 0x6e, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x65, 0x75, 0x6e, 0x69, 0x74, 0x73, 0x62, 0x6e, 0x61, 0x66, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x73, 0x8e, 0x5, 0x6, 0x5, 0x6, 0x7, 0x7, 0x6, 0x6, 0x6, 0x6, 0x5, 0x5, 0x6, 0x6

Which base64 encoded would be:

 o2lwcm90ZWN0ZWSiY3ZlcmJ2MWNhbGdlSFMyNTZpc2lnbmF0dXJleDxlZmE3MTQwZGY5YjA0ZTk0NjdiOGQzMjQ44oCmIDMzMjQgYzg0MzdlYTkwYzM2NmFiMzAxZjBkZTBlZDZncGF5bG9hZKVrZGV2aWNlX25hbWV4GUVBOiAwIEM6IEZGOiAxNDogMiBGOiAxIEFrZGV2aWNlX3R5cGVyU0VFRURfV0lPX1RFUk1JTkFMa2ludGVydmFsX21zEGdzZW5zb3JzgaJkbmFtZWxJbGx1bWluYXRpb25ldW5pdHNibmFmdmFsdWVzjgUGBQYHBwYGBgYFBQYG
1 Like