Deleting/moving raw-data samples in one class causes reproducible, deterministic retrain instability affecting OTHER classes (project 1067977)

Help
,
#1

Question/Issue:
[Describe the question or issue in detail]
Project ID: 1067977 (Classifier learn block id=3, Keras/NN classifier, MFE features, 4-class audio: background/chainsaw/elephant_call/gunshot).

Summary: after deleting and moving a small number of chainsaw-labeled raw-data samples between the training/testing categories (24 deletions, 22 category moves, all via the Studio API, all confirmed by ID to be chainsaw-only), retraining the Keras classifier – with no other change to any block config – went from a healthy, converging run to a reproducibly unstable one, and this instability affects gunshot, a class none of our changes touched.

Before (retrain job earlier the same day, before the chainsaw raw-data changes): training loss/accuracy converged normally, val_accuracy settled in the 0.68-0.75 range after early epochs, best checkpoint val_loss=0.718 (val_accuracy=0.744). Resulting classify-all: overall accuracy 72%, gunshot F1 0.928, chainsaw F1 0.596.

After (three separate retrain attempts post-changes – two via POST /api/{projectId}/jobs/retrain, one via the Studio UI’s “Retrain model” button): all three produced bit-for-bit identical loss/accuracy at every one of 100 epochs. val_accuracy repeatedly collapses to 0.17-0.31 throughout the entire run (not just early epochs), best checkpoint val_loss=1.028 (val_accuracy=0.707) – worse than the “before” best, and visibly less stable throughout. Resulting classify-all: gunshot F1 collapses to 0.343, chainsaw F1 to 0.000.

What we’ve ruled out (each independently verified before posting):

  1. Data corruption from the resync – audited every raw-data sample currently labeled gunshot/background/elephant_call: counts match a pre-change snapshot exactly (e.g. gunshot testing count identical, 335 before and after), spot-checked samples are correctly labeled/enabled/retrievable, and cross-referenced the exact 41 distinct sample IDs touched by our delete/move calls against every current gunshot/background/elephant_call sample ID – zero overlap.
  2. Stale/cached auto class weights – autoClassWeights: true is enabled, but weights come from ei_tensorflow.training.get_class_weights(Y_train), computed fresh at retrain time.
  3. Learn-block config drift – learningRate (0.005), trainingCycles (100), and trainTestSplit (0.2) are identical between before/after runs.
  4. API vs. UI trigger difference – ruled out directly, UI-triggered run matched the API runs bit-for-bit.
  5. Random chance – ruled out by the bit-for-bit reproducibility itself.

Our core question: is the internal training/validation split (trainTestSplit: 0.2) computed once, globally, across the whole “training” category regardless of class, such that deleting/moving samples in ONE class shifts the internal validation fold for OTHER classes too?

Job IDs: before (healthy) = 51777054; after (unstable, all 3 identical) = 51779988, 51780370, 51781246.
Project ID:
[Provide the project ID]

Context/Use case:
[Provide context or use case where the issue is encountered]

Steps Taken:

  1. [Step 1]
  2. [Step 2]
  3. [Step 3]

Expected Outcome:
[Describe what you expected to happen]

Actual Outcome:
[Describe what actually happened]

Reproducibility:

  • [ ] Always
  • [ ] Sometimes
  • [ ] Rarely

Environment:

  • Platform: [e.g., Raspberry Pi, nRF9160 DK, etc.]
  • Build Environment Details: [e.g., Arduino IDE 1.8.19 ESP32 Core for Arduino 2.0.4]
  • OS Version: [e.g., Ubuntu 20.04, Windows 10]
  • Edge Impulse Version (Firmware): [e.g., 1.2.3]
  • To find out Edge Impulse Version:
  • if you have pre-compiled firmware: run edge-impulse-run-impulse --raw and type AT+INFO. Look for Edge Impulse version in the output.
  • if you have a library deployment: inside the unarchived deployment, open model-parameters/model_metadata.h and look for EI_STUDIO_VERSION_MAJOR, EI_STUDIO_VERSION_MINOR, EI_STUDIO_VERSION_PATCH
  • Edge Impulse CLI Version: [e.g., 1.5.0]
  • Project Version: [e.g., 1.0.0]
  • Custom Blocks / Impulse Configuration: [Describe custom blocks used or impulse configuration]
    Logs/Attachments:
    [Include any logs or screenshots that may help in diagnosing the issue]

Additional Information:
[Any other information that might be relevant]