Import Your Data as CSV

Blog
#1

Upon popular request we've added support for importing CSV data into Edge Impulse through both the upload functionality in the Studio (in your project go to Data acquisition, and select the ‘upload’ icon), and in the Edge Impulse CLI. This makes it trivial to bring existing sensor data into Edge Impulse, without the need to convert your data into the Edge Impulse Data Acquisition format yourself.

This is a companion discussion topic for the original entry at https://www.edgeimpulse.com/blog/import-your-data-as-csv
6 Likes
#2

Awesome!

Tell me, is there a blog post or a video that shows the CSV support being put to use? Having trouble wrapping my head around it.

#3

Here is one of the original discussions before CSV was supported. perhaps it will help.

CSV should be a huge help with data uploading. I also will have to try it out to understand it. The first part that confused me originally was the idea that each label entry was it’s own CSV file, or old JSON file. Once I got over that most of it made sense. Say if I am writing a “B” with an accelerometer. I would have many x,y,z entries for every time I tried that one label.

#4

Thanks, that helps.

Would I have a separate CSV file for each instance of a given classification?

For example, in the case of an object recognition model, I upload many images of one label, many images of another label, etc.

So if I wanted to import a CSV of accelerometer data, would I want to break up the raw CSV into smaller samples of every occurrence of a certain movement?

So:

-- label-1
| -- label-1_01.csv
| -- label-1_02.csv
| -- label-1_03.csv
| -- ...

-- label-2
| -- label-2_01.csv
| -- label-2_02.csv
| -- label-2_03.csv
| -- ...
#5

Hi @avielbr that would work. If you want to detect events (e.g. someone hitting a device) you can also just upload a long sample, then click the three dots => Split sample and it will autodetect the events. See https://www.edgeimpulse.com/blog/crop-split-data/

3 Likes
#6

Hi

CSV upload is most welcome :slight_smile:, but it does not like a timestamp in YYYYMMDDHHMMSS format :frowning: ?

Should I convert to Epoch Time?

James

#7

@Kingqwon it’s milliseconds since the start of the file, so if you have a reading every 10 ms. (100Hz):

timestamp
0
10
20

etc.

#8

Hi,
I get the following error when uploading my data: “Cannot read property ‘trim’ of undefined”

This is how my file looks like
pic

Could you please help me?

#9

Hi @budiju4 Can you email me your CSV file at jan@edgeimpulse.com ?

#10

I’ve just sent it, thank you.

#11

Looked at it, and you need to remove trailing empty lines, then it’ll go through. I’ll push a hotfix to the CLI (v1.13.6) and the Studio today! Thanks for the report.

#12

I’ve made a small C# program that can convert CSV files from Arduino serial monitor to Edge Impulse CSV and JSON format, if anyone is interested :wink:

Don’t know where to put it though :slight_smile:

#13

@kelle cool! I’d just throw it on GitHub. Note that if you have a serial connection already the https://docs.edgeimpulse.com/docs/cli-data-forwarder can do this for you too!

1 Like
#14

This is great but maybe i am missing something i want to use this along with the new regression block. Now i have 6 months of data 4 sensors reading every 5 minutes in a sql db that i can easily export as a single csv but from reading the details it looks like a need to create a csv for each set of readings (43000 files) is that correct or am i missing something here. i want to be able to predict several days ahead or at least try

Any suggestions?

#15

Hi @bettston, correct, but if you have so much data it’s probably easier through the API. E.g. example for Python: https://github.com/edgeimpulse/linux-sdk-python/blob/master/examples/custom/collect.py

(The Linux SDK - https://docs.edgeimpulse.com/docs/edge-impulse-for-linux - also has examples in Node.js, Go and C++).

#16

Hi, thankyou, i see so basically take my csv and use python to chop it up and send json files. i assume all the libraries you are using work in windows (anaconda) even though its says linux sdk?

This is going to be interesting

#17

@bettston, yes.

It is just a basic python script that will call our API.

I usually use Google Colab to write jupyter notebooks and use pandas + numpy to parse my csv. But it will work the same way with Anaconda.

Regards,

Louis

#19

Thanks, i am still in the learning (or the what the heck does all this do) stage so really helpful

Tony

#20

If you data is not confidential and relatively simple, feel free to share it here, I can guide you on the script creation, I have done plenty of those :smiley:

Regards,

Louis

#21

very kind, but i think to start i should do it myself its good practice but if i get stuck i will take you up.

Tony

1 Like