Generate Features OOM

Help
#1

I’m trying to generate the features for my keyword detection, and I am running into the following situation:

Creating job… OK (ID: 1396159)
Scheduling job in cluster…
Job started
Creating windows from 15296 files…
[2/5] Pre-caching files…
[3/5] Pre-caching files…
[5/5] Pre-caching files…
Pre-caching files OK

[ 1/15296] Creating windows from files…
[ 142/15296] Creating windows from files…
[ 143/15296] Creating windows from files…
[ 153/15296] Creating windows from files…
[ 163/15296] Creating windows from files…
[ 176/15296] Creating windows from files…
[ 194/15296] Creating windows from files…
[ 202/15296] Creating windows from files…
[ 216/15296] Creating windows from files…
[ 228/15296] Creating windows from files…
[ 238/15296] Creating windows from files…
/home/create_features.sh: line 3: 8 Killed node “/app/node/windowing/build/window-time-series.js” “/home/input.json” “/home/output.json”
Application exited with code 137 (OOMKilled)
Job failed (see above)

Feature explorer

No features generated yet.

I know that there is a lot of data, but will cutting down on it solve this problem, is the data I have too high resolution etc. If you could give me some direction for how to solve this issue it would be greatly appreciated.

Thanks!

© 2021 EdgeImpulse Inc. All rights reserve

#2

@sunrobbie can you share your project ID? (go to Dashboard page, bottom right)

#3

@yodaimpulse https://studio.edgeimpulse.com/public/47834/latest this should be the link, thank you!

#4

Hi @sunrobbie I am able to reproduce the error but not yet able to figure out what is causing it. Will investigate with our team and let you know asap.

#5

@sunrobbie I’ve upped the memory allocation for the project now, while we investigate what’s causing this on your dataset.

#6

@sunrobbie - we’ve upped the memory limits for all jobs to be less stringent (they can go over memory limits without being killed immediately) and this should resolve all OOMKilled issues. We’re monitoring actively to see if any others happen and can tweak the limits if that’s the case.