Past Incidents

Our tinkering worked! We found what we believe was the root cause of the performance issues. The fleet of edge routers has been upgrade to a new release of OpenZiti & are now back in full operational mode.

We are implementing some adjustments to the edge router configuration in our deployed fleet over the next hour. There will be some additional rolling restarts as part of this effort. If the system starts to become unresponsive again, we will be revert again.

Reverting to the previous version has restored the system to the previous state. To keep the system functional, we have placed a rolling restart of our edge router fleet every hour. Long-running share connections will impacted by these rolling restarts.

Again the upgrade is causing issues with new services. We are capturing logs & reverting to the previous version.

Deployment of zrok 0.4.23 was successful

The OpenZiti team has made some improvements on the link management that we believe will increase the stability of the system. We will be proceeding with deploying ziti 0.32.1 today at 1:00pm EST

Reverting to the previous version has restored the system to the previous state. To keep the system functional, we have placed a rolling restart of our edge router fleet every hour. Long-running share connections will impacted by these rolling restarts.

We are actively working on a fix for this. As soon as we've put out the fires, we plan on providing more information about what went wrong, how we fixed it, and what we're doing to improve our reliability and scalability in the long-term.

Upgrade caused issues with new services, so we reverted.

Upgrade was successful. Metrics are not flowing as expected, we are looking into it now.