Edge impulse data forwarder error

paulphilip · May 22, 2020, 9:02am

Hi I am having issues when running the edge impulse data forwarder

ER] Connecting to COM3
[SER] Serial is connected
[WS ] Connecting to wss://remote-mgmt.edgeimpulse.com
[WS ] Connected to wss://remote-mgmt.edgeimpulse.com
? To which project do you want to add this device? (Use arrow keys)
? To which project do you want to add this device? paulphilip-project-1
? 3 sensor axes detected. What do you want to call them? Separate the names with   a? 3 sensor axes detected. What do you want to call them? Separate the names with
? What name do you want to give this device? 06:6C:FF:36:39:31:59:4E:43:03:37:06
[WS ] Authenticated
[WS ] Incoming sampling request {
  path: '/api/training/data',
  label: 'I',
  length: 3000,
  interval: 0,
  hmacKey: 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
  sensor: undefined
}
[SER] Waiting 2 seconds...
[SER] Reading data from device...
[SER] Invalid data collected. All values should be numeric, and every line should have 3 values. [
  [ 73, -135, -956 ], [ 73, -135, -955 ], [ 73, -134, -955 ],
  [ 73, -135, -955 ], [ 73, -135, -955 ], [ 72, -135, -956 ],
  [ 73, -135, -956 ], [ 73, -135, -955 ], [ 73, -135, -955 ],
  [ 73, -135, -956 ], [ 73, -135, -955 ], [ 73, -135, -955 ],
  [ 73, -135, -955 ], [ 73, -135, -956 ], [ 73, -135, -955 ],
  [ 73, -135, -956 ], [ 73, -135, -956 ], [ 73, -135, -956 ],
  [ 73, -135, -956 ], [ 73, -134, -956 ], [ 73, -137, -953 ],
  [ 73, -135, -956 ], [ 72, -135, -955 ], [ 73, -135, -956 ],
  [ 72, -134, -957 ], [ 73, -135, -956 ], [ 72, -134, -956 ],
  [ 73, -135, -955 ], [ 73, -137, -953 ], [ 73, -135, -956 ],
  [ 72, -135, -955 ], [ 73, -135, -956 ], [ 72, -134, -957 ],
  [ 73, -135, -956 ], [ 73, -135, -955 ], [ 73, -135, -955 ],
  [ 73, -137, -953 ], [ 73, -135, -955 ], [ 73, -134, -956 ],
  [ 73, -135, -956 ], [ 72, -133, -958 ], [ 0 ]
]
[SER] Failed to sample data Error: Invalid data collected from device, see data forwarder logs
    at WebSocket.<anonymous> (C:\Users\paul\AppData\Roaming\npm\node_modules\edge-impulse-cli\build\serial-daemon\cli\data-forwarder.js:242:31)

I tried with edge-impulse-data-forwarder --clean but still the issue persists. From where can I get the logs and how would you advise to debug this issue.

janjongboom · May 22, 2020, 9:19am

@paulphilip There’s a [ 0 ] at the end of the Invalid data collected message. Where does that come from? Are you only printing sensor readings or also other things over the serial?

paulphilip · May 22, 2020, 9:22am

Yes. I am printing only sensor values. I am attaching a short snippet of the serial output when the device is reset.

    87,   -110,   -959
    88,   -111,   -958
    87,   -111,   -960
    89,   -111,   -958
    89,   -111,   -959
    88,   -112,   -958
    89,   -113,   -956
    88,   -112,   -958
    88,   -112,   -958
    88,   -111,   -958
    89,   -110,   -960
    87,   -112,   -957
    89,   -111,   -957
    88,   -111,   -958
    88,   -114,   -955
    90,   -112,   -957
    87,   -111,   -958
    88,   -112,   -957
    86,   -110,   -960
    88,   -111,   -958
    90,   -112,   -956
    87,   -112,   -958
    90,   -113,   -955
    87,   -112,   -957
    88,   -111,   -959
    88,   -111,   -958
    88,   -110,   -960
    88,   -112,   -957
    89,   -112,   -956
    88,   -112,   -957
    89,   -113,   -955
    89,   -112,   -957
    86,   -111,   -960
    88,   -112,   -957
    88,   -111,   -957
    89,   -111,   -958
    89,   -112,   -956
    88,   -112,   -958
    88,   -112,   -957
    88,   -112,   -958
    87,   -110,   -960
    89,   -111,   -957
    88,   -112,   -958
    88,   -111,   -958
    88,   -113,   -956
    89,   -111,   -957
    89,   -112,   -957
    89,   -112,   -957
    86,   -111,   -959
    88,   -112,   -957
    88,   -112,   -957
    88,   -111,   -958
    89,   -111,   -957
    87,   -111,   -958
    89,   -111,   -957
    88,   -111,   -958
    87,   -112,   -958
    90,   -111,   -957
    87,   -112,   -957
    89,   -111,   -957
    87,   -111,   -957
    88,   -112,   -957
    90,   -111,   -957
    87,   -113,   -957
    89,   -111,   -958
    88,   -112,   -957
    89,   -111,   -958
    88,   -111,   -958
    88,   -111,   -957
    88,   -111,   -958
    88,   -111,   -957
    89,   -114,   -955
    89,   -112,   -958
    88,   -112,   -957
    87,   -112,   -957
    88,   -110,   -959
    88,   -112,   -957
    88,   -111,   -958
    90,   -111,   -958
    87,   -114,   -955
    89,   -111,   -958
    87,   -112,   -958
    88,   -112,   -957
    89,   -110,   -959
    87,   -112,   -957
    90,   -113,   -956
    87,   -111,   -957
    89,   -113,   -955
    90,   -111,   -957
    87,   -111,   -959
    88,   -112,   -958
    86,   -110,   -960
    88,   -111,   -957
    89,   -113,   -955
    89,   -111,   -957
    88,   -113,   -955
    88,   -112,   -958
    87,   -111,   -959
    89,   -112,   -958
    88,   -112,   -958
    88,   -111,   -958
    90,   -112,   -956
    88,   -112,   -957
    88,   -113,   -956
    88,   -111,   -958
    87,   -110,   -960
    88,   -112,   -957
    89,   -112,   -957
    87,   -112,   -957
    89,   -112,   -956
    88,   -112,   -957
    89,   -111,   -957
    88,   -112,   -958
    87,   -111,   -959
    88,   -112,   -957
    89,   -112,   -957

janjongboom · May 22, 2020, 9:44am

OK, let me check this on a Windows machine!

paulphilip · May 22, 2020, 9:57am

Thanks. Let me know if you want any inputs from my side.

janjongboom · May 22, 2020, 9:58am

@paulphilip This runs fine for me with an Arduino Nano 33 BLE Sense with a simple accelerometer sketch based on the LSM9DS1 on both Windows 10 and macOS, so not sure where the problem is exactly. However, I’ve pushed out v1.6.1 of the Edge Impulse CLI (update via npm update edge-impulse-cli -g) which will look for incomplete readings at the end of the stream and discard them. Hopefully that resolves your issue!

paulphilip · May 22, 2020, 10:14am

@janjongboom
Thanks a lot for solving this error. I am attaching edge impulse data forwarder for your confirmation

[SER] Waiting 2 seconds...
[SER] Reading data from device...
[SER] Reading data from device OK (41 samples at 16Hz)
[SER] Uploading sample to https://ingestion.edgeimpulse.com/api/training/data...
[SER] Sampling finished

janjongboom · May 22, 2020, 10:27am

Beautiful, thanks @paulphilip!

paulphilip · June 11, 2020, 4:12am

Hi,
Yesterday I updated the edge impulse cli and I am facing errors now.

edge-impulse-data-forwarder
Edge Impulse data forwarder v1.6.5
This is an experimental feature, please let us know if you run into any issues at
    https://forum.edgeimpulse.com

Endpoints:
    Websocket: wss://remote-mgmt.edgeimpulse.com
    API:       https://studio.edgeimpulse.com/v1
    Ingestion: https://ingestion.edgeimpulse.com

[SER] Connecting to COM3
[SER] Serial is connected
[WS ] Connecting to wss://remote-mgmt.edgeimpulse.com
[WS ] Connected to wss://remote-mgmt.edgeimpulse.com
? What name do you want to give this device? 06:6C:FF:36:39:31:59:4E:43:03:37:06
[WS ] Authenticated
[WS ] Incoming sampling request {
  path: '/api/testing/data',
  label: 'testing',
  length: 5000,
  interval: 71.42857142857143,
  hmacKey: '69f2fc343ec5c811147c0c320da338a5',
  sensor: 'Sensor with 3 axes (acc_x, acc_y, acc_z)'
}
[SER] Waiting 2 seconds...
[SER] Reading data from device...
[SER] Reading data from device OK (67 samples at 14Hz)
[SER] Uploading sample to https://ingestion.edgeimpulse.com/api/testing/data...
[SER] Sampling finished

But I am getting the following error.

I downgraded to v1.6.4 but still I am facing the same issue.
Kindly advise.

paulphilip · June 11, 2020, 4:59am

@janjongboom I think I found the culprit. I had uploaded my phone data also for training and this data doesn’t have axes label. After I removed the data sampled from the phone, it started to work properly.

Ashshak · October 10, 2020, 11:57am

Is there any problem occurred? if I print other things over the Serial