Sending CSV data to Edge Impulse using command line

#1

Hi team,

I encountered an issue while sending CSV data from command line to Edge Impulse.
Here is the script and the issue report below;

image
image1446×873 107 KB

The issue report below;

image

Please can any member of the team assist me in resolving this issue.

Thank you

#2

  • You need to post to the ingestion site:

    • https://ingestion.edgeimpulse.com/api/training/files

  • See the Python ingestion example in the man.

1 Like
#3

Thank you very much for your response.

#4

Thank you so much for your support.
Please after replacing the url with the ingestion site, i got this error report,

image
image835×59 5.44 KB

I guess this should be about the format. If that is the case, how can I know the appropriate file format since I am converting streaming data to csv data file.

Thank you

#5

Hi @Netwave_paul1

Please post code as text in future so we can verify easier, and a sample of the CSV output.

The first line should be header information. Each sample should be on a newline.

I suspect you have got somehting incorrect on the dataframe.

Build the data first with headers, and use that to build the dataframe also ensure that the timestamps are in miliseconds:

Something like this:

data = {
    'Timestamps': time_stamps,
    'EEG Data': eeg_data
}

df = pd.Dataframe(data)

Your CSV should look something like this:

Timestamps,EEG Data
2023-09-12 14:00:00.000,1.23
2023-09-12 14:00:00.001,1.24
2023-09-12 14:00:00.002,1.25
2023-09-12 14:00:00.003,1.26
2023-09-12 14:00:00.004,1.27

See the importing CSV guide for more, and the associated git repo:

Hope this helps, please let us know how you get on and share any output / code as text.

Best

Eoin

#6

Thank you very much for your prompt response.
I will send you the text script if i encounter any further issue.

Many thanks @Eoin

#7

Here is the script below,

import pyxdf
import pandas as pd
import requests

Edge Impulse project details

API_KEY = ‘XXXXX’
PROJECT_ID = ‘278572’
EI_API_ENDPOINT = f’https://ingestion.edgeimpulse.com/api/training/files’

Path to your XDF file

XDF_FILE_PATH = ‘C:/Users/apple/bci-essentials-python/sub-P36_mi_ses-MI_task-T1_run-001_eeg.xdf’

Load the XDF file

streams, header = pyxdf.load_xdf(XDF_FILE_PATH)

Extract data from the first data stream (you can customize this based on your XDF structure)

selected_stream = streams[0]

Extract data and timestamps from the selected stream

eeg_data = selected_stream[‘time_series’]
timestamps = selected_stream[‘time_stamps’]

Convert timestamps to milliseconds

timestamps_ms = [int(ts * 1000) for ts in timestamps]

Create a dictionary with headers and data

data = {
‘Timestamps’: timestamps_ms,
‘EEG Data’: eeg_data
}

Create a DataFrame from the data dictionary

df = pd.DataFrame(data)

Save the DataFrame to a CSV file

csv_file_path = ‘C:/Users/apple/bci-essentials-python.csv’
df.to_csv(csv_file_path, index=False)

Send the CSV data to Edge Impulse

with open(csv_file_path, ‘rb’) as file:
files = {‘file’: (csv_file_path, file, ‘application/octet-stream’)}
headers = {‘x-api-key’: API_KEY}
response = requests.post(EI_API_ENDPOINT, files=files, headers=headers)

Check the response from Edge Impulse

if response.status_code == 200:
print(‘XDF data converted to CSV and uploaded to Edge Impulse successfully.’)
else:
print(f’Error uploading CSV data to Edge Impulse: {response.status_code} - {response.text}')

#8

Hi @Netwave_paul1

Did you have further issues? Its not clear from that.

Remember that the timestamp needs to be in mili seconds. Please make sure the CSV output is as expected: Importing CSV data

Best

Eoin