Deleting/moving raw-data samples in one class causes reproducible, deterministic retrain instability affecting OTHER classes (project 1067977)

Help
,
#1

Question/Issue:
[Describe the question or issue in detail]
Project ID: 1067977 (Classifier learn block id=3, Keras/NN classifier, MFE features, 4-class audio: background/chainsaw/elephant_call/gunshot).

Summary: after deleting and moving a small number of chainsaw-labeled raw-data samples between the training/testing categories (24 deletions, 22 category moves, all via the Studio API, all confirmed by ID to be chainsaw-only), retraining the Keras classifier – with no other change to any block config – went from a healthy, converging run to a reproducibly unstable one, and this instability affects gunshot, a class none of our changes touched.

Before (retrain job earlier the same day, before the chainsaw raw-data changes): training loss/accuracy converged normally, val_accuracy settled in the 0.68-0.75 range after early epochs, best checkpoint val_loss=0.718 (val_accuracy=0.744). Resulting classify-all: overall accuracy 72%, gunshot F1 0.928, chainsaw F1 0.596.

After (three separate retrain attempts post-changes – two via POST /api/{projectId}/jobs/retrain, one via the Studio UI’s “Retrain model” button): all three produced bit-for-bit identical loss/accuracy at every one of 100 epochs. val_accuracy repeatedly collapses to 0.17-0.31 throughout the entire run (not just early epochs), best checkpoint val_loss=1.028 (val_accuracy=0.707) – worse than the “before” best, and visibly less stable throughout. Resulting classify-all: gunshot F1 collapses to 0.343, chainsaw F1 to 0.000.

What we’ve ruled out (each independently verified before posting):

  1. Data corruption from the resync – audited every raw-data sample currently labeled gunshot/background/elephant_call: counts match a pre-change snapshot exactly (e.g. gunshot testing count identical, 335 before and after), spot-checked samples are correctly labeled/enabled/retrievable, and cross-referenced the exact 41 distinct sample IDs touched by our delete/move calls against every current gunshot/background/elephant_call sample ID – zero overlap.
  2. Stale/cached auto class weights – autoClassWeights: true is enabled, but weights come from ei_tensorflow.training.get_class_weights(Y_train), computed fresh at retrain time.
  3. Learn-block config drift – learningRate (0.005), trainingCycles (100), and trainTestSplit (0.2) are identical between before/after runs.
  4. API vs. UI trigger difference – ruled out directly, UI-triggered run matched the API runs bit-for-bit.
  5. Random chance – ruled out by the bit-for-bit reproducibility itself.

Our core question: is the internal training/validation split (trainTestSplit: 0.2) computed once, globally, across the whole “training” category regardless of class, such that deleting/moving samples in ONE class shifts the internal validation fold for OTHER classes too?

Job IDs: before (healthy) = 51777054; after (unstable, all 3 identical) = 51779988, 51780370, 51781246.
Project ID:
[Provide the project ID]

Context/Use case:
[Provide context or use case where the issue is encountered]

Steps Taken:

  1. [Step 1]
  2. [Step 2]
  3. [Step 3]

Expected Outcome:
[Describe what you expected to happen]

Actual Outcome:
[Describe what actually happened]

Reproducibility:

  • [ ] Always
  • [ ] Sometimes
  • [ ] Rarely

Environment:

  • Platform: [e.g., Raspberry Pi, nRF9160 DK, etc.]
  • Build Environment Details: [e.g., Arduino IDE 1.8.19 ESP32 Core for Arduino 2.0.4]
  • OS Version: [e.g., Ubuntu 20.04, Windows 10]
  • Edge Impulse Version (Firmware): [e.g., 1.2.3]
  • To find out Edge Impulse Version:
  • if you have pre-compiled firmware: run edge-impulse-run-impulse --raw and type AT+INFO. Look for Edge Impulse version in the output.
  • if you have a library deployment: inside the unarchived deployment, open model-parameters/model_metadata.h and look for EI_STUDIO_VERSION_MAJOR, EI_STUDIO_VERSION_MINOR, EI_STUDIO_VERSION_PATCH
  • Edge Impulse CLI Version: [e.g., 1.5.0]
  • Project Version: [e.g., 1.0.0]
  • Custom Blocks / Impulse Configuration: [Describe custom blocks used or impulse configuration]
    Logs/Attachments:
    [Include any logs or screenshots that may help in diagnosing the issue]

Additional Information:
[Any other information that might be relevant]

#2

Hey @Haniiiyee, apologies for the poor experience. We’re looking at this for you right now, we’ll let you know as soon as we have an answer.

1 Like
#3

Hey @Haniiiyee, we spent some time looking at your project and the history and evolution of its dataset, here’s what we think is happening.

  • Your initial dataset had 2443 training samples, you then moved 22 chainsaw ones away from the training set, taking its size down to 2421.

  • We verified that indeed all samples that were moved were chainsaw ones, everything else stayed the same as you said.

  • When you go to train your modal, our internal train/val split performs a single global shuffle of all your training samples (across all classes) and takes a percentage (which in your case you left as the default 20%) for the validation fold.

  • The shuffle uses a fixed random seed that’s deterministic, which is why your last three retrain attempts were identical.

  • However, an important point of how the sklearn train_test_split function works is that it’s computed over range(N) (where N is the total training count), when N changes (like in your case going from 2443 to 2421) the entire permutation changes, potentially significantly. You can simulate this directly with something like the following

    from sklearn.model_selection import train_test_split

N_before = 2443
N_after  = 2421

_, val_before = train_test_split(range(N_before), test_size=0.2, random_state=3)
_, val_after  = train_test_split(range(N_after),  test_size=0.2, random_state=3)

shifted = set(val_before).symmetric_difference(set(val_after))
print(f"Indices that changed fold: {len(shifted)}")

  • In your case, 468 out of the ~489 validation slots changed, so almost the entire validation set composition.

  • As an additional nuance, the split operates at the window level, not the sample one. A single raw audio sample produces multiple overlapping windows (due to your window size and stride settings) and those windows are shuffled independently, which means that some windows from the same recording might end up in train while other end up in validation. This is worth keeping in mind as a relevant factor when looking at the validation accuracy of your model.

Now, our suggestion in this case would be to experiment with explicit validation, which gives you full control and visibility over which samples end up in each fold. You can enable it under Data Acquisition → Dataset → three dots menu → Advanced Settings; once enabled, your validation set remains fixed regardless of what changes you make in the other categories. This also has an impact on the potential window-leakage issue I described in the last bullet point, since in this case all windows from a given sample stay on the same side of the fold.

Hope this helps!

2 Likes