We sincerely apologize for the increased latency which occurred yesterday, August 20th, from 6:48AM PDT to 8:20AM PDT. We know you depend on us to do your work in your ministry, and we take that responsibility very seriously. A full summary of the day, including steps we plan on taking moving forward, are as follows.

ISSUE SUMMARY

From 6:48AM PDT to 8:20AM PDT, requests to all applications were met with slow response times and users had an especially difficult time using Check-ins. The root cause was an increase in load on our database server which in turn slowed all requests.

TIMELINE

6:48AM PDT We began receiving alerts of increased latency across several applications.

7:11AM PDT We identified a change made to Check-Ins earlier this week which caused the increased load on the database.

7:35AM PDT The change was reverted from Check-Ins and we began monitoring for recovery.

8:20AM PDT Load began decreasing and response times improved.

ROOT CAUSE

A slow SQL query was deployed earlier in the week that triggered each time Check-Ins searches the list of People. The performance issue did not become apparent until increased traffic on Sunday morning.

RESOLUTION AND RECOVERY

At 8:20AM PDT load decreased and the issue was resolved.

CORRECTIVE AND PREVENTATIVE MEASURES

After a complete review and analysis of our performance yesterday, we will take the following steps to prevent this from happening again.

We are reviewing critical queries in Check-Ins and optimizing them to ensure they can perform well during high traffic periods. We will evaluate our current infrastructure to ensure we are using the most appropriate hardware for our volume of requests. Planning Center is committed to continually improving our technology and operational processes to prevent outages.

We appreciate your patience and again deeply apologize for the impact to you and your church.
Thank you for your business and continued support.