Hello!
I have a large dataset where each record is saved as an individual .npy file (for data generator purposes). Each file contains a list of integers and they are variable length.
Is it possible to upload these .npy -files to Edge Impulse, and build a classifier based on those?
Thanks!
Hi @ympyra,
It is not currently possible to upload your .npy files directly into the studio, however there are many Python scripts out there that you can utilize to convert your list of integers into a .csv file for example, here’s a quick search of one: https://www.geeksforgeeks.org/convert-a-numpy-array-into-a-csv-file/
If you want to send an example .npy file you are trying to upload, happy to assist with the scripting further!
– Jenny
Also, check out our guide on uploading .csv data here: https://docs.edgeimpulse.com/reference/importing-csv-data
Hello @jenny,
Thanks for the prompt reply and the suggestion!
I am aware that this is possible but saving as .csv can introduce some magnitudes of overhead.
For example, I have a list with 1014 integer values ranging from 0-255, where the size of the .npy-file is 1142 bytes and .csv-file 3428 bytes.
Let’s also say that I have a list with 58456 integer values ranging from 0-255. Then the size of the .npy-file is 58584 bytes and .csv-file 197540 bytes.
So when dealing with large datasets, e.g. millions of samples, these overheads can have a significant impact when considering memory usage.
I can manage with the .csv-format for now. But perhaps support for .npy-files could be a beneficial feature 
All the best,
eppane
I see your point, I will submit this as a feature suggestion to our engineering team!
– Jenny