Jobs failing with ENOSPC

#1

Hi,

We’re having issues with our network storage being full (and not being cleaned up) for the past day and a half, and this has been leading to ENOSPC errors on jobs. We’re working on a solution.

#2

We’ve fixed the immediate issue, but not the root cause yet; we’ll keep monitoring this :slight_smile:

#3

Unfortunately I still see some events in the logs here, most users shouldn’t be affected right now (it affects 2/10 storage nodes). Will check with infra in the morning.