Some quantum lab users can't start their notebooks

Incident Report for IBM Quantum Experience

Resolved

We believe we have new boot scripts that resolve this for all active and future users.

Posted Sep 22, 2020 - 18:54 UTC

Update

Followup fixes have been implemented, we are monitoring the results.

Posted Sep 22, 2020 - 18:37 UTC

Update

Working on a follow up fix for setting up jupyter environment correctly on boot

Posted Sep 22, 2020 - 14:37 UTC

Monitoring

There is now a script running every 60 seconds to catch pods that start in a corrupted state, fix them, and restart them.

Posted Sep 22, 2020 - 12:10 UTC

Identified

There was an underlying issue with object store permissions, these seem to have been fixed with COS plugin upgrade. There is a different issue with bad symlinks having been created in user images during the outage, which is preventing otherwise healthy environments from working in IQX. Looking at fixes for that.

Posted Sep 22, 2020 - 11:59 UTC

Investigating

Investigating permissions errors with storage

Posted Sep 22, 2020 - 10:43 UTC