Uploading Training Data Constraints

Question/Issue:
Hello, I am new to Edge Impulse, I apologize for any obvious knowledge gaps. My inquiry is to learn whether there is a way to automate data upload and training. I have hundreds of thousands of images I wish to upload for training data, whenever I try to train the Generate the Features and Train the Model, I face issues with folder size and training duration errors. Breaking the data into smaller data folders works fine, but makes loading the sheer number of folders impractical.

My understanding is that the model can be trained incrementally. Is there an API that I can use to send a folder to update the model, then once complete receive an acknowledgement from EdgeImpulse, and send the next folder. Ensuring I send all files is imperative for this project (not an option to throw some images out).

Hi @crlane2

You should start by looking at the Uploader | Edge Impulse Documentation

But you may also need some object storage like AWS S3 if your dataset is for enterprise usage I’ve put together some docs on that and tutorials but start with the uploader via cli and see how you get on.

Also please share any more detail here we can help, is this for a masters / phd or a enterprise project? Then we can help advise what direction to take, and the most cost effective one, no need to pay for enterprise grade system if you are just pursuing knowledge building :smiley:

Best

Eoin