Studio unavailable (update: available again!)

Hi,

Due to issues with the network storage the Studio is currently unavailable (ingestion is available). We’re working on a solution at the moment.

Update (15:57 CET): We’ve rebuilt the network storage layer and are now restoring a backup of earlier today. This does not affect any raw data, labels, or DSP config; just intermediate state of some projects. The restore is currently 30% done.

Update (17:04 CET): We’re half way through the restore.

Update (18:18 CET): We’re at ~62%, so slowing down a bit, but still progressing.

Update (19:23 CET): Restore still running. We think it’s mostly done, but that it’s verifying the storage now.

Update (20:08 CET): Restoring the last 14GB of data (out of >3TB, you create a lot of data :wink: ), but these are small files so they take a bit longer.

Update (20:26 CET): Backup seems to be complete, we’ll verify the integrity of the restore and route the Studio traffic to the new storage cluster.

Update (20:56 CET): Backup was verified (it seems), we’re bringing the Studio back up.

Update (21:22 CET): The studio is back up, we’re switching DNS now, you should be able to access the Studio shortly. Thanks for the patience. There are a few (<100) users where some intermediate data might be lost, this can be solved by retraining your model. If you’re affected you’ll get an email from me in a few minutes. There has been no data loss.

6 Likes

Thanks for the updates.

1 Like