Import Your Data as CSV

Blog
#22

Hey Louis,
i pulled my data so this is a short excerpt, col 1 is the sensor (6 repeating then time and reading (no specific unit). looking at the python example i guess i just read 6 rows and write them out to the api but need to use a numeric label starting 1 and not use the datetime field at all.

10598 24/12/2020 16:43 0.092692241
10599 24/12/2020 16:43 0.187688172
10600 24/12/2020 16:43 0.155966774
10601 24/12/2020 16:43 0.105609968
10597 24/12/2020 16:48 0.031937473
10598 24/12/2020 16:48 0.095218122
10599 24/12/2020 16:48 0.136323869
10600 24/12/2020 16:48 0.165941507
10601 24/12/2020 16:48 0.102731444
10597 24/12/2020 16:53 0.012241724
10598 24/12/2020 16:53 0.062836766
10599 24/12/2020 16:53 0.103517555
10600 24/12/2020 16:53 0.139830455
10601 24/12/2020 16:53 0.104309998
10597 24/12/2020 16:58 0.008909958
10598 24/12/2020 16:58 0.089947239
10599 24/12/2020 16:58 0.174376264
10600 24/12/2020 16:58 0.162320033
10601 24/12/2020 16:58 0.096579425
10597 24/12/2020 17:03 0.001874622
10598 24/12/2020 17:03 0.070911519
10599 24/12/2020 17:03 0.177711084
10600 24/12/2020 17:03 0.18924123
10601 24/12/2020 17:03 0.096544094
10597 24/12/2020 17:08 0.010716454
10598 24/12/2020 17:08 0.078860447
10599 24/12/2020 17:08 0.109745763
10600 24/12/2020 17:08 0.164896011
10601 24/12/2020 17:08 0.096436128
#23

@bettston,

I’ll have a deeper look tomorrow but in the meatime I guess you can start with this piece of code I just adapted:

import pandas as pd
import numpy as np

import json
import time, hmac, hashlib
import requests

# Create a dataframe from csv
df = pd.read_csv('example-csv.csv', delimiter=',', header=None)
print(df)

## do something here to create a list here that is group by the timestamp for example
## custom_list = [{sensorValue1, sensorValue2, sensorValue3, ...}, {sensorValue1, sensorValue2, sensorValue3, ...}]

## create sensor list
sensors = [{ "name": "10597", "units": "count"}, { "name": "10598", "units": "count"}, { "name": "10599", "units": "count"}, { "name": "10600", "units": "count"}, { "name": "10601", "units": "count"}]
print(sensors)


## Ingest training data

HMAC_KEY = "fed53116f20684c067774ebf9e7bcbdc"
API_KEY = "ei_c1b27..."

# empty signature (all zeros). HS256 gives 32 byte signature, and we encode in hex, so we need 64 characters here
emptySignature = ''.join(['0'] * 64)

data = {
    "protected": {
        "ver": "v1",
        "alg": "HS256",
        "iat": time.time() # epoch time, seconds since 1970
    },
    "signature": emptySignature,
    "payload": {
        "device_name": "ac:87:a3:0a:2d:1b",
        "device_type": "CUSTOM_SENSOR",
        "interval_ms": 1000,
        "sensors": sensors,
        "values": custom_list
    }
}

# encode in JSON
encoded = json.dumps(data)
# print(encoded)

# sign message
signature = hmac.new(bytes(HMAC_KEY, 'utf-8'), msg = encoded.encode('utf-8'), digestmod = hashlib.sha256).hexdigest()

# set the signature again in the message, and encode again
data['signature'] = signature
encoded = json.dumps(data)
# print(encoded)

# and upload the file
res = requests.post(url='https://ingestion.edgeimpulse.com/api/training/data',
                    data=encoded,
                    headers={
                        'Content-Type': 'application/json',
                        'x-file-name': 'your_label',
                        'x-api-key': API_KEY
                    })
if (res.status_code == 200):
    print('Uploaded file to Edge Impulse', res.status_code, res.content)
else:
    print('Failed to upload file to Edge Impulse', res.status_code, res.content)

Regards

1 Like
#24

Hey @bettston,

Did you manage to ingest your CSV file or you still need some help?

Do not hesitate to share your piece of code, other users could find it useful :slight_smile:

Regards,

1 Like
#25

Hi,
@louis not yet, unfortunately the main job got in the way. going to have a go tomorrow now that i understand the principle.

Tony

1 Like
#26

Hello @janjongboom

I got a problem uploading cvs file from CLI.

I do not understand…

the same files upload correctly form web interface.

Screenshot_1
Screenshot_11009×406 70.2 KB

is it possible use one option in CLI for debugging ?
It is very frustrating to have no indication of the error.

This conthain data in csv format used for training:
Data for traning

and this is the comands used for upload:
edge-impulse-uploader --category traning --label in “.\Training\edge\in*.csv”

I tried to modify the node.js function “make-image.js” that is in charge for convert the csv to .json

end upload a single .cs file , it seems that the internal conversion to json works well :

Thanks in advance

Mirko Ugolini

#27

@mirko.ugolini Use

training

You have a typo (traning) :slight_smile:

#28

@janjongboom

I’m so sorry, thans a lot , now it works

:slight_smile:

#29

@janjongboom could you please help me?
I would like to know, imagine that I record acc activy during 3 minutes, each day, during one month. I have to submit one file per day or I can do it in only one file?

Thanks!

#30

Hi @ErickGQ,

You can do it in one file as long as the timestamp increases by the same amount (https://docs.edgeimpulse.com/reference/importing-csv-data).
Also make sure that your csv contains data for a unique label (ie: one activity type).

Aurelien