Error after training with FOMO

Report bugs
#1

Question/Issue:
When I trained FOMO model, after the training, the project failed to save the results.

Project ID:
595003

Context/Use case:
I have 100,000 images labeled.

Summary:
The error is as following:

/app/run-python-with-venv.sh: line 17:   773 Killed                  /app/$VENV_NAME/.venv/bin/python3 -u $ARGS
Application exited with code 137 (OOMKilled)
2025-01-09T16:17:01.049Z logger=server level=error Failed job execution
Error: Job 27874741 finished
    at /home/node/studio/build/server/server/start-daemon.js:264:48
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2025-01-09T16:17:01.049Z logger=server level=error Failed job execution
Error: Job 27874741 finished
    at /home/node/studio/build/server/server/start-daemon.js:264:48
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Application exited with code 1
Job failed (see above)

Steps to Reproduce:

  1. Create the images features 320x320
  2. Train the FOMO model

Expected Results:
Model should be saved

Actual Results:
Error while saving

Reproducibility:

  • [ X] Always
  • [ ] Sometimes
  • [ ] Rarely

Environment:

  • Platform: ESP32-EYE
  • Build Environment Details: Arduino 2.0.4
  • OS Version: Windows 10

Logs/Attachments:

/app/run-python-with-venv.sh: line 17:   773 Killed                  /app/$VENV_NAME/.venv/bin/python3 -u $ARGS
Application exited with code 137 (OOMKilled)
2025-01-09T16:17:01.049Z logger=server level=error Failed job execution
Error: Job 27874741 finished
    at /home/node/studio/build/server/server/start-daemon.js:264:48
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2025-01-09T16:17:01.049Z logger=server level=error Failed job execution
Error: Job 27874741 finished
    at /home/node/studio/build/server/server/start-daemon.js:264:48
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Application exited with code 1
Job failed (see above)