Resolved: Service Disruption EU7

Follow
On August 17 a Service Disruption occurred on the EU7 instance.
 
Start Time: 2:03 am CEST, Aug 17
End Time: 5:15 am CEST, Aug 17
Duration: 3 hours, 12 minutes
 
INCIDENT HISTORY
 
ROOT CAUSE
The problem was caused by an unexpected process error while performing a routine maintenance on network tier and manual intervention was required to restore normal service.
 
UPDATE
5:20 am CEST, Aug 17
Upon further investigation, the Technology team has determined that the likely cause of this issue is related to a code change made on August 16 to the host firewall that supports the app servers. The team is now rolling back that change in an effort to resolve the issue. The site switch decision has been put on hold in the interim.
 
INCIDENT RESOLVED
5:15 am CEST, Aug 17
 
PERFORMANCE DEGRADATION ENDED
5:15 am CEST, Aug 17
Duration: 1 minute
 
PERFORMANCE DEGRADATION BEGAN
5:14 am CEST, Aug 17
During a performance degradation, end users are able to access the service, however, some functionality within the service may not be available or running at optimal performance.
 
SERVICE DISRUPTION ENDED
5:14 am CEST, Aug 17
Duration: 2 hours
 
UPDATE
4:55 am CEST, Aug 17
The Salesforce Technology team continues to investigate the service disruption impacting the EU7 instance. While the team is still evaluating the possibility of performing a site switch in order to resolve the issue, they are also exploring a possible network connectivity issue with the application servers that could be the source of the disruption.
 
UPDATE
4:38 am CEST, Aug 17
The Salesforce Technology team continues to investigate the service disruption impacting the EU7 instance. It is currently exploring the possibility of performing a site switch in order to resolve the issue.
 
UPDATE
4:08 am CEST, Aug 17
Upon investigation, The Technology team determined that the cause of the issue was related to high page loading times. In order to resolve the issue, the Technology team decided to restart EU7’s application tier, but that action proved unsuccessful in resolving the issue. The team then decided to roll back a release patch that was implemented on August 16, 2018, but the rollback did not resolve the issue. The team is now evaluating whether or not to perform a site switch to resolve the issue.
 
INVESTIGATING
3:58 am CEST, Aug 17
We are investigating a potential cause of the issue.
 
INVESTIGATING
3:26 am CEST, Aug 17
We are currently investigating the cause of the issue.
 
SERVICE DISRUPTION BEGAN
3:14 am CEST, Aug 17
During a service disruption, end users are unable to access the service.
 
PERFORMANCE DEGRADATION ENDED
3:14 am CEST, Aug 17
Duration: 6 minutes
 
PERFORMANCE DEGRADATION BEGAN
3:08 am CEST, Aug 17
During a performance degradation, end users are able to access the service, however, some functionality within the service may not be available or running at optimal performance.
 
SERVICE DISRUPTION ENDED
3:08 am CEST, Aug 17
Duration: 1 hour, 5 minutes
 
INVESTIGATING
2:45 am CEST, Aug 17
We are currently investigating the cause of the issue.
 
SERVICE DISRUPTION BEGAN
2:03 am CEST, Aug 17
During a service disruption, end users are unable to access the service.
Have more questions? Submit a request

Comments

Powered by Zendesk