Strange behavior in model testing

Hi there. I am experiencing problems identifying simple drops in a cloned project that works essentially well without them added !

With drops recordings added and model retrained untouched, i get a perfect confusion matrix and classes repartition, as this can be seen in the cloned project :

But, when running a live classification on a live 5 seconds recording carried-out from my desktop with absolutely not water dropping around, my recording is identified as being a drop 100% of the sample.

Could one have a look on this project and help me understand what i am missing here?

Many thanks for your help

Hi @Cramredan ,

Thanks for sharing your project, it made this much easier to look into!

Your drop samples are about 60 seconds long each, so a good chunk of each one is quiet time between drops. Every window from those samples gets the drop label during training, so the model has learned to link a quiet room with drops. The tutorial’s noise samples are mostly busy household sounds, so the model hasn’t seen silence anywhere else yet.

A couple of things might help:

  1. Trim the drop samples to just the drop sounds, using Split sample or Crop sample in Data acquisition: Data acquisition - Edge Impulse Documentation
  2. Add some quiet-room recordings from your Portenta to the noise class (or to a new background class), so the model learns what silence sounds like.
  3. If you can, record a few drop samples with the Portenta too. Right now the drop clips are 48 kHz from a different mic, so this keeps every class consistent.
  4. Test with recordings from a different session to get a more realistic idea of how the model will do in practice.

After retraining, the Feature explorer is a great way to check that silence and drops are separating nicely.

Hope this helps, let us know how it goes!

Thanks Jenny for having quickly examined my project to help sorting this out.

I fully agree and understand your suggestions. Trimming the drop samples will be my first move … but, how to automatically isolate in separate files many many drops? Hundreds of them ! I tried the split feature, but cannot get any result out of the interface.The control over the samples is tedious and unpractical.

Maybe could you suggest a process that would help me automate the process, eg. in an external sound editing application maybe?

Kind regards. Marc

Hi Marc,

Here’s a few options for you:

Option 1: Label the drops in Studio with AI (no splitting needed)

Studio has a built-in Audio labeling with AudioSet block. It uses the open source Audio Spectrogram Transformer (AST) model, and AudioSet includes a Drip class. The block’s code is open source too.

First, tag your drop samples with metadata so the AI labeling action can find them:

  1. Go to Data acquisition > Dataset. Click the filter icon above the sample list, select Labels, tick only drop, and apply the filter.
  2. Click the multi-select icon above the list, then tick the box in the header to select every sample in the filtered list. Choose Add metadata, enter label as the key and drop as the value, and save.
  3. Do the same on the Test tab.

Note: AI labeling replaces each sample’s single drop label with drop and noise regions, but the label: drop metadata stays. You can still find your original drop recordings later with the metadata filter in Dataset. If you record more drops, tag them the same way and run the action again.

Then create the AI labeling action:

  1. Go to Data acquisition > AI labeling and click + Add new label action.
  2. For Step 1, choose Audio labeling with AudioSet.
  3. Set the parameters:
    • Labels of interest: replace speech and music with these two lines:
      Drip (drop)
      
      The name in parentheses renames AudioSet’s label to your own.
    • Other label: noise
    • Window size (ms) and Window stride (ms): change the default of 2000 to something shorter, such as 500, so each drop gets its own short region.
    • Minimum confidence threshold: start at the default of 0.2.
  4. Set Run AI labeling on to Data, filtered by metadata, and pick the key label with the value drop. The Total results count should match the number of drop samples, and your other classes are left alone.
  5. Click Label preview data and listen to a few results. If it misses drops, lower the threshold. If it labels silence as drop, raise it.
  6. (Optional) Under Set metadata after running AI labeling, enter labeling-source: audioset. This marks which samples were AI-labeled, so you can filter them later or compare experiments with and without them.
  7. Click Label all data.

Tip: Labels of interest must use exact AudioSet names, one per line with no blank lines. Type them by hand rather than pasting. If the job fails, the bottom of the job log shows Invalid label: "..." with the line it couldn’t match.

Your 60-second files stay whole. Each one becomes a multi-label sample, with the drips labeled drop and the quiet gaps labeled noise. Then go to Create impulse and set multi-label handling to Use label X if anywhere present in the window with drop selected. Any window containing a drop is labeled drop, and the silent windows teach the model what a quiet room sounds like.

Option 2: Run Split sample on every file with a script

Split sample uses two API endpoints, Find segments and Segment sample, so a script can run it on all your files at once. Get your API key from Dashboard > Keys:

import requests

API_KEY = "ei_..."        # Dashboard > Keys
PROJECT_ID = 12345
LABEL = "drop"
CATEGORY = "training"     # run again with "testing"
SEGMENT_LEN = 1000        # ms, match your impulse window size

BASE = f"https://studio.edgeimpulse.com/v1/api/{PROJECT_ID}"
HEADERS = {"x-api-key": API_KEY, "Content-Type": "application/json"}

samples, offset = [], 0
while True:
    r = requests.get(f"{BASE}/raw-data", headers=HEADERS, params={
        "category": CATEGORY, "labels": f'["{LABEL}"]',
        "limit": 100, "offset": offset}).json()
    samples += r["samples"]
    if len(r["samples"]) < 100:
        break
    offset += 100

for s in samples:
    seg = requests.post(f"{BASE}/raw-data/{s['id']}/find-segments", headers=HEADERS,
                        json={"shiftSegments": True, "segmentLengthMs": SEGMENT_LEN}).json()
    if not seg.get("success") or not seg["segments"]:
        print(f"Sample {s['id']}: no segments found")
        continue
    res = requests.post(f"{BASE}/raw-data/{s['id']}/segment", headers=HEADERS,
                        json={"segments": seg["segments"]}).json()
    print(f"Sample {s['id']}: {len(seg['segments'])} segments",
          "OK" if res.get("success") else res.get("error"))

Splitting marks each original sample as deleted. You can get an original back with Crop sample on one of its segments, but try the script on a clone of your project first.

Option 3: Split the files before uploading (Audacity)

If you’d rather do it offline, the free Audacity editor can do it:

  1. Open a recording and go to Analyze > Label Sounds. Set the threshold and minimum silence so each drop gets its own label.
  2. Go to File > Export Audio and choose Multiple files, split by labels. Each drop is saved as its own WAV file.
  3. While you’re there, resample to 16 kHz (Tracks > Resample) so the drops match your Portenta recordings.
  4. Upload the files with the drop label.

Then check the results in Studio

  • Add 5 to 10 minutes of quiet-room recordings from your Portenta to noise, unless Option 1 already did this for you.
  • In Data acquisition, use the Length filter to find clips that are too short or too long. Disable any clips that don’t contain a drop instead of deleting them, so you can bring them back later.
  • After you generate features, check the Feature explorer. Silence and drops should form separate clusters. Click any drop point that lands in the silence cluster to listen to it.
  • Test with Live classification on a fresh Portenta recording of a quiet room. It should come back as noise now.

Let us know which option works best for you!

2 Likes

Jenny, you’re an angel to my rescue… many thanks and gratitude.

I followed your Option1) by the line (Label the drops in Studio with AI no splitting needed) and ended up with the following Job Log upon clicking on “Label Preview Data”. My problem now is that “drop” label do not show in the rankings, while “music”, “vehicle”, “car”, “sound effect” do ! Result is “noise” each line.

I’m I right assessing “drop” should normally appear in the log, whatever the % probability returned?

In your instructions, you stated :

Labels of interest: replace speech and music with these two lines:

Drip (drop)

I only see one line : “Drip (drop)” and changing Drip to drip doesn’t help.

Any idea?

Creating job… OK (ID: 54867867)

:heavy_check_mark: Job scheduled at 13 Oct 2026 15:28:36
:heavy_check_mark: Job started at 13 Oct 2026 15:28:38
/usr/local/lib/python3.10/site-packages/pydub/utils.py:170: RuntimeWarning: Couldn’t find ffmpeg or avconv - defaulting to ffmpeg, but may not work
warn(“Couldn’t find ffmpeg or avconv - defaulting to ffmpeg, but may not work”, RuntimeWarning)
Labeling data using Audio Spectrogram Transformers

Detecting audio:
Audioset labels: drip (drop)
Other label: noise
Min. confidence: 0.2
Window size: 500ms.
Window stride: 500ms.
IDs: 3159235169, 3159235174, 3159235176, 3159235178, 3159235181 and 1 others
Project ID: 1129839
[1/6] Labeling drops_16-09-2026 16-00_normalized-20dB (ID 3159235169), length=60475ms:
[0 - 500ms.] top results: music (0.154), vehicle (0.076), car (0.061): result=noise
[500 - 1000ms.] top results: vehicle (0.103), music (0.083), car (0.078): result=noise
[1000 - 1500ms.] top results: vehicle (0.126), car (0.084), sound effect (0.067): result=noise
[1500 - 2000ms.] top results: vehicle (0.099), music (0.074), sound effect (0.069): result=noise
[2000 - 2500ms.] top results: vehicle (0.117), car (0.075), sound effect (0.071): result=noise
[2500 - 3000ms.] top results: vehicle (0.099), sound effect (0.088), music (0.064): result=noise
[3000 - 3500ms.] top results: music (0.118), vehicle (0.102), car (0.073): result=noise
[3500 - 4000ms.] top results: music (0.115), vehicle (0.098), car (0.072): result=noise
[4000 - 4500ms.] top results: vehicle (0.098), sound effect (0.085), car (0.066): result=noise
[4500 - 5000ms.] top results: music (0.107), vehicle (0.094), car (0.079): result=noise
[5000 - 5500ms.] top results: vehicle (0.104), car (0.085), music (0.081): result=noise
[5500 - 6000ms.] top results: vehicle (0.109), sound effect (0.074), car (0.065): result=noise
[6000 - 6500ms.] top results: vehicle (0.103), car (0.079), music (0.076): result=noise
[6500 - 7000ms.] top results: vehicle (0.084), car (0.083), scrape (0.064): result=noise
[7000 - 7500ms.] top results: vehicle (0.178), car (0.131), music (0.064): result=noise
[7500 - 8000ms.] top results: sound effect (0.089), vehicle (0.072), scrape (0.059): result=noise
[8000 - 8500ms.] top results: music (0.106), vehicle (0.093), car (0.077): result=noise
[8500 - 9000ms.] top results: vehicle (0.092), sound effect (0.082), car (0.057): result=noise
[9000 - 9500ms.] top results: music (0.112), vehicle (0.074), sound effect (0.068): result=noise

1 Like

Hi again @Cramredan !

Your label setup works – the log line Audioset labels: drip (drop) means the block found and accepted the label. Matching ignores upper and lower case, so Drip and drip work the same.

Why drop never shows up: the Hugging Face API sends the block only the top 5 of AudioSet’s 527 classes for each window, and the log prints the top 3. A window is labeled drop only if Drip is in that top 5 and scores above your Minimum confidence threshold. In your log, even the top scores are only around 0.1, spread across “vehicle”, “music”, and “car”. That means the model isn’t hearing anything it recognizes, and drip isn’t getting into the top 5.

The main cause is probably the 500 ms window. The model works on about 10 seconds of audio and pads anything shorter with zeros. A 500 ms window holding one small drip doesn’t give it much to work with. Here’s what I’d try, on one or two samples with Label preview data first:

  1. Use a longer window with a shorter stride. Try a 2000 ms window and a 500 ms stride. The stride can’t be longer than the window.
  2. Add related AudioSet classes and map them all to drop, so any of them in the top 5 counts. Type these one per line by hand, with no blank lines:
Drip (drop)
Raindrop (drop)
Splash, splatter (drop)
Water (drop)

Water is broad, so check the preview to make sure it isn’t labeling other sounds as drop.
3. Lower the Minimum confidence threshold to around 0.1. This only helps once a drip-related class is reaching the top 5, so do steps 1 and 2 first. Then listen to the previewed regions to check they really are drips.

If drip still doesn’t show up, this model may just not recognize your drops yet without more data added. You can also try the following options:

Option 2 (script): the script runs Split sample on every drop file for you. It uses these Studio API endpoints:

  • List samples gets all the samples with the drop label.
  • Find segments finds the “non-noise events” in each sample. That should pick out each drop against a quiet background.
  • Segment sample splits each sample at those segments.

You can find your API key under Dashboard > Keys (API keys docs). Splitting marks each original sample as deleted. You can get it back with Crop sample on one of its segments, but try the script on a clone of your project first.

Option 3 (Audacity): Analyze > Label Sounds marks anything above a level threshold as a sound. Raise the Threshold level until the silence between drops stops being labeled. If drops come less than a second apart, lower Minimum silence duration and Minimum label interval from their 1-second defaults so each drop gets its own label. Then use File > Export Audio > Multiple files, split by labels, to save one WAV per drop. The Label Sounds manual page covers all the settings.

Either way, keep adding those quiet-room Portenta recordings to noise. That’s what stops the model from calling silence a drop.

Let us know how it goes!

1 Like