Failed to upload large amount of .wav files


I am trying to upload files to my project on edge impulse, however when I try calling the uploader, it fails to upload the files. This is what is returned:
[10/74] Failed to upload ./f_clicks/3_11_f_clicks.wav write EPIPE

also this:
[12/74] Failed to upload ./f_clicks/2_21_f_clicks.wav

502 Bad Gateway

502 Bad Gateway

I wonder if this issue is caused because I am trying to upload 74 files at once because the system does work for smaller amount of file uploads.

@cjohnston21 We have had some issue on production today leading to 502 errors. We’re investigating now!

edit: actually I doubt that this was related (the errors in production were 504 timeouts). Could you send me the two files that fail via ?

Actually, I think the problem is the amount of data I’m trying to upload. The folder containing the .wav files is 8.22 GB. Is there a size limit for the uploader?

this is the command I use: edge-impulse-uploader --label ‘f_clicks’ ./f_clicks/*.wav

would it work if I split up the calls for each .wav file instead of all of them at once, given the large size of the folder?

It shouldn’t be, we parallelise this with max. 20 (I think) uploads - but it could be possible we trip over very large files as we expand them in memory, but let me know if that works. I think there’s a parallel option in the uploader to limit concurrency (not behind a computer right now). Would be interested to see if setting that to 1 solves your issue.

Ok. How would I change the uploader configuration to test this. Or would it be better if I sent the folder to you.

So I tried doing it file by file and it worked, but it took a very long time (>10 min) for the configuration step to complete before it said it was uploading

@cjohnston21 how big are the individual wave files? I’ll try and replicate this.

So the .wav files were 50-250 MB each

@cjohnston21, thanks! Will add it to our bug tracker.