[Resolved] Issues with processing blocks / feature explorer / data explorer

Update #2: This issue has been fixed, but we were facing DB stability issues (see [Stable] 503s on Studio / API) on our primary DB - this seems to also be stable now.

Update: The issue should be resolved. You’ll need to re-generate features to fix your existing projects. Either fully retrain your model (via “Retrain model” in the navbar); or go to your processing block and click ‘Generate features’ again.

Hi,

We’re seeing a bunch of issues on production right now for projects that involve data uploaded since July 29 (older projects are fine). E.g.:

  • Data explorer / feature explorer shows “Sample #-2147186607 was deleted.” for samples.
  • When re-generating features (e.g. after you added more data) the resulting dataset ends up empty, or ends up being very small (feature explorer is empty, and validation set contains no / little data).

The root cause is because we are storing the sample IDs in an int32 data field (inside some NPY files); but we crossed 2^32 no. of data samples today (thanks all!) - causing an overflow here.

If you’re re-generating features and this causes you to end up with 0 data (or very little data) in your dataset; a workaround is:

  • Go to the processing block (e.g. Spectral Analysis or Image)
  • Change any parameter
  • Click Save
  • Change the parameter back to what it was before
  • Click Save again
  • Generate features, this will generate features from scratch

We’re estimating a fix to be deployed somewhere before July 30, 2PM GMT.

edit July 30, 7:48AM GMT: AI Labeling is functional again.
edit July 30, 1PM GMT: A release is going to production now.
edit July 30, 1:11PM GMT: Release is on production, issue should be resolved.

3 Likes

Update: we assumed this also affected model testing; but this does not seem to be the case. If you do see weird errors there please let us know.

Excuse me,in this case,it may have no feature explorer like this,then I can’t delete some data to make the model better.When I generate features,it says success,but it seems nothing appears on the right.

@wei_solitude Workaround is to delete the processing block from “Create impulse” and re-creating it… Subsequent features will work again.

We’re rolling out a fix today.

1 Like

thanks for your reply,but I try as you say,it can’t work either,the result is the same,it seems not to be a good solution

@wei_solitude The underlying issue has been resolved; if you regenerate features again you’ll see your whole dataset again.

1 Like

Now it all works well as before.
Thank you and your team a lot.

1 Like