Sending CSV data to Edge Impulse using command line

#1

Hi team,

I encountered an issue while sending CSV data from command line to Edge Impulse.
Here is the script and the issue report below;

image
image1446×873 107 KB

The issue report below;

image

Please can any member of the team assist me in resolving this issue.

Thank you

#2

  • You need to post to the ingestion site:

    • https://ingestion.edgeimpulse.com/api/training/files

  • See the Python ingestion example in the man.

1 Like
#3

Thank you very much for your response.

#4

Thank you so much for your support.
Please after replacing the url with the ingestion site, i got this error report,

image
image835×59 5.44 KB

I guess this should be about the format. If that is the case, how can I know the appropriate file format since I am converting streaming data to csv data file.

Thank you

#5

Hi @Netwave_paul1

Please post code as text in future so we can verify easier, and a sample of the CSV output.

The first line should be header information. Each sample should be on a newline.

I suspect you have got somehting incorrect on the dataframe.

Build the data first with headers, and use that to build the dataframe also ensure that the timestamps are in miliseconds:

Something like this:

data = {
    'Timestamps': time_stamps,
    'EEG Data': eeg_data
}

df = pd.Dataframe(data)

Your CSV should look something like this:

Timestamps,EEG Data
2023-09-12 14:00:00.000,1.23
2023-09-12 14:00:00.001,1.24
2023-09-12 14:00:00.002,1.25
2023-09-12 14:00:00.003,1.26
2023-09-12 14:00:00.004,1.27

See the importing CSV guide for more, and the associated git repo:

Hope this helps, please let us know how you get on and share any output / code as text.

Best

Eoin

#6

Thank you very much for your prompt response.
I will send you the text script if i encounter any further issue.

Many thanks @Eoin

#7

Here is the script below,

import pyxdf
import pandas as pd
import requests

Edge Impulse project details

API_KEY = ‘XXXXX’
PROJECT_ID = ‘278572’
EI_API_ENDPOINT = f’https://ingestion.edgeimpulse.com/api/training/files’

Path to your XDF file

XDF_FILE_PATH = ‘C:/Users/apple/bci-essentials-python/sub-P36_mi_ses-MI_task-T1_run-001_eeg.xdf’

Load the XDF file

streams, header = pyxdf.load_xdf(XDF_FILE_PATH)

Extract data from the first data stream (you can customize this based on your XDF structure)

selected_stream = streams[0]

Extract data and timestamps from the selected stream

eeg_data = selected_stream[‘time_series’]
timestamps = selected_stream[‘time_stamps’]

Convert timestamps to milliseconds

timestamps_ms = [int(ts * 1000) for ts in timestamps]

Create a dictionary with headers and data

data = {
‘Timestamps’: timestamps_ms,
‘EEG Data’: eeg_data
}

Create a DataFrame from the data dictionary

df = pd.DataFrame(data)

Save the DataFrame to a CSV file

csv_file_path = ‘C:/Users/apple/bci-essentials-python.csv’
df.to_csv(csv_file_path, index=False)

Send the CSV data to Edge Impulse

with open(csv_file_path, ‘rb’) as file:
files = {‘file’: (csv_file_path, file, ‘application/octet-stream’)}
headers = {‘x-api-key’: API_KEY}
response = requests.post(EI_API_ENDPOINT, files=files, headers=headers)

Check the response from Edge Impulse

if response.status_code == 200:
print(‘XDF data converted to CSV and uploaded to Edge Impulse successfully.’)
else:
print(f’Error uploading CSV data to Edge Impulse: {response.status_code} - {response.text}')

#8

Hi @Netwave_paul1

Did you have further issues? Its not clear from that.

Remember that the timestamp needs to be in mili seconds. Please make sure the CSV output is as expected: Importing CSV data

Best

Eoin

#9

I’m curious where this project landed…

On a related note, I went through the EEG demos / tutorials, which were leveraging eye blinks and some SMR, but not more advanced stakes like sleep v wake, drowsy versus alert. Before I spend too much time on this, I am curious what experts think about a project that has the following components:

-five channels of ear-eeg
-label data captured with eyes open and eyes closed (with alpha response)

Goal
Use EI to build a model to turn on the lights and turn off the lights simply through alpha power modulation.

This demo is not hard to do with classic machine learning, so what are people’s thought using EI?

Then, I would deploy to one of the supported boards.

1 Like
#10

Hello @gobigorbust ,

Welcome to the Edge Impulse community! It’s great to hear about your interest in using Edge Impulse for your EEG-based product. We have a number of products currently using us for EEG and other health/activity tracking sensor data.

You mentioned 5 channels of EEG are you using an NPU? We have a number of supported boards you can try to explore this with, or if you have specific hardware that is not supported we can discuss with the tech team about how to support your hardware.

We also have some features that are upcoming that may fit your use case, and specific DSP blocks for human health that you may be interested in exploring. See our HRV Block announcement for one such feature, that can also be used with EEG and ECG inputs through Sensor Fusion, read more on that in this blog:

With access to our enterprise tier we should be able to estimate and gain additional space on the target hardware to run the logic required to run the control and coms that you wish to add.

Would you be interested in setting up a meeting with our solutions team to explore this further? Please share the project ID and we will have someone take a quick look at the volume of data and hardware you are testing on.

Please let us know a convenient time for you, and we’ll arrange a session to discuss your project in more detail with a representative in your timezone.

Looking forward to hearing from you and excited about the potential of learning more about this innovative application!

Best,

Eoin