Note: Usually our incident postmortem report comes out within days of the incident. However, since this issue was the result of an error at Stripe, we wanted to wait for their report before posting ours here. With that report in hand, we can now include their internal changes that will prevent this kind of issue going forward.
On April 13, a mistake made by our payment processor, Stripe, resulted in a critical API key being deleted. This caused a temporary disruption across several of our products, including Giving and Registrations. During this outage, payments made on Church Center (web or mobile app) showed an error for one-off donations and payments. Recurring donations (which are programmatically triggered) also hit the same error, but they were quickly and successfully reprocessed once the new key was in place.
While the core issue was resolved quickly in the early morning hours, the automated email that followed the initial disconnection caused unnecessary confusion. It implied that churches had disconnected their own Stripe accounts, which wasn’t the case, and its urgent tone only added to the misunderstanding. To make matters worse, many customers had no way of knowing the issue had already been fixed unless they were familiar with our status page and knew to check there, which left some unsure whether their donation system was still down.
One of the biggest takeaways from this incident was the importance of fast, plainspoken communication. Usually, churches want to know the details about what error messages were shown, exactly what parts of the system were impacted, and most of all what happened with the recurring donations. We wanted to speak to all of that accurately and confidently, but that cost us time. As soon as the system was back online we should have sent a quick communication just to say “Ignore that alert email, everything is fine. More details will follow” and then sent a followup email later.
Over at Stripe, on April 13th, a front line triage agent was responding to a fraud incident and deleted the wrong keys. As a result of this incident, they’ve removed the ability for the triage agents to delete a platform’s API key directly. To manage a platform’s API key (like Planning Center) there’s now a different internal escalation path that requires signoff at multiple levels within Stripe.
We’re taking several steps to strengthen both our technical systems and our communication process, including:
We know your church relies on Planning Center to operate smoothly, especially for things like donations and event registrations. We’re committed to learning from this and making our systems—and our communication—stronger moving forward.
Thank you for your continued partnership and grace.