Feature generation crashes when number of windows > 2000

#1

I am assuming there is a limit of 2000 windows max that has been set in the feature generator but if that is the case indeed, it would be nice to have a better error message than the current one:

[1500/2066] Creating features...
[1600/2066] Creating features...
[1700/2066] Creating features...
[1800/2066] Creating features...
[1900/2066] Creating features...
[2000/2066] Creating features...
Traceback (most recent call last):
File "/home/create_features.py", line 50, in <module>
    sample_file[sample_file_total] = f['features']
File "/usr/local/lib/python3.7/site-packages/numpy/core/memmap.py", line 331, in __getitem__
    res = super(memmap, self).__getitem__(index)
IndexError: index 2000 is out of bounds for axis 0 with size 2000

ERR: {"active":1,"failed":1,"startTime":"2020-02-04T06:17:40.000Z"}

Job failed (see above)
#2

Hi @kartben2, it’s not actually - we have projects with over 400K windows(!). But we slice 2,000 of these windows into a separate file for the feature explorer, and we are writing out of bounds of this array. Quickfix would be to change the windowing parameters a bit so you have more windows (I think there’s a race-condition where we miscalculate the step between the windows for the feature explorer), and we’ll deploy a fix as part of today’s release.

#3

@kartben2 This fix has now been deployed to production, please let me know if the problem persists.

#4

Hi,

I get different error messages during audio feature generation.

1.

Feature generation output

Creating job…
Failed to create job:
Cannot start job. There already is a job with key “dsp-10” running. Job failed (see above)

2.
The feature generation also crashes at around 4.000 + Windows - which is similar to the error message under this topic.

Any idea what might be the issue?

Regards

#5

Hi

I tried it again and is working now. I assume some sort of hick up on the Edge Impulse server.

Regards

#6

Hi @rn230021, thanks for reporting! @mathijs could you see if there’s anything in the logs that stands out?

#7

Hello @rn230021,

I have investigated the issue you reported.

From what I can see the jobs in question were terminated due to exceeding the job execution time limit of 10 minutes (for power users we can increase the job execution time limit, if your application requires a longer time limit, please let us know!).

Unfortunately due to an infrastructure issue a notification to inform you about the time limit was not shown. I have identified and fixed this issue.