[Resolved] Issues with processing blocks / feature explorer / data explorer

#1

Update #2: This issue has been fixed, but we were facing DB stability issues (see [Stable] 503s on Studio / API) on our primary DB - this seems to also be stable now.

Update: The issue should be resolved. You’ll need to re-generate features to fix your existing projects. Either fully retrain your model (via “Retrain model” in the navbar); or go to your processing block and click ‘Generate features’ again.

Hi,

We’re seeing a bunch of issues on production right now for projects that involve data uploaded since July 29 (older projects are fine). E.g.:

  • Data explorer / feature explorer shows “Sample #-2147186607 was deleted.” for samples.
  • When re-generating features (e.g. after you added more data) the resulting dataset ends up empty, or ends up being very small (feature explorer is empty, and validation set contains no / little data).

The root cause is because we are storing the sample IDs in an int32 data field (inside some NPY files); but we crossed 2^32 no. of data samples today (thanks all!) - causing an overflow here.

If you’re re-generating features and this causes you to end up with 0 data (or very little data) in your dataset; a workaround is:

  • Go to the processing block (e.g. Spectral Analysis or Image)
  • Change any parameter
  • Click Save
  • Change the parameter back to what it was before
  • Click Save again
  • Generate features, this will generate features from scratch

We’re estimating a fix to be deployed somewhere before July 30, 2PM GMT.

edit July 30, 7:48AM GMT: AI Labeling is functional again.
edit July 30, 1PM GMT: A release is going to production now.
edit July 30, 1:11PM GMT: Release is on production, issue should be resolved.

3 Likes
Failed to upload
500 Internal Server Error when uploading samples"
Something wrong
I can't upload my WAV files
Failed to upload
#2

Update: we assumed this also affected model testing; but this does not seem to be the case. If you do see weird errors there please let us know.

#3

Excuse me,in this case,it may have no feature explorer like this,then I can’t delete some data to make the model better.When I generate features,it says success,but it seems nothing appears on the right.

111
1111542×811 114 KB

#4

@wei_solitude Workaround is to delete the processing block from “Create impulse” and re-creating it… Subsequent features will work again.

We’re rolling out a fix today.

1 Like
#5

thanks for your reply,but I try as you say,it can’t work either,the result is the same,it seems not to be a good solution

#6

@wei_solitude The underlying issue has been resolved; if you regenerate features again you’ll see your whole dataset again.

#7

Now it all works well as before.
Thank you and your team a lot.

1 Like