Failed to upload large amount of .wav files

#1

Hi!

I am trying to upload files to my project on edge impulse, however when I try calling the uploader, it fails to upload the files. This is what is returned:
[10/74] Failed to upload ./f_clicks/3_11_f_clicks.wav write EPIPE

also this:
[12/74] Failed to upload ./f_clicks/2_21_f_clicks.wav

502 Bad Gateway

502 Bad Gateway

I wonder if this issue is caused because I am trying to upload 74 files at once because the system does work for smaller amount of file uploads.

#2

@cjohnston21 We have had some issue on production today leading to 502 errors. We’re investigating now!

edit: actually I doubt that this was related (the errors in production were 504 timeouts). Could you send me the two files that fail via jan@edgeimpulse.com ?

#3

Actually, I think the problem is the amount of data I’m trying to upload. The folder containing the .wav files is 8.22 GB. Is there a size limit for the uploader?

#4

this is the command I use: edge-impulse-uploader --label ‘f_clicks’ ./f_clicks/*.wav

would it work if I split up the calls for each .wav file instead of all of them at once, given the large size of the folder?

#5

It shouldn’t be, we parallelise this with max. 20 (I think) uploads - but it could be possible we trip over very large files as we expand them in memory, but let me know if that works. I think there’s a parallel option in the uploader to limit concurrency (not behind a computer right now). Would be interested to see if setting that to 1 solves your issue.

#6

Ok. How would I change the uploader configuration to test this. Or would it be better if I sent the folder to you.

#7

So I tried doing it file by file and it worked, but it took a very long time (>10 min) for the configuration step to complete before it said it was uploading

#8

@cjohnston21 how big are the individual wave files? I’ll try and replicate this.

#9

So the .wav files were 50-250 MB each

#10

@cjohnston21, thanks! Will add it to our bug tracker.