Elevated errors related to processing payments and donations.

Incident Report for Planning Center

Postmortem

Note: Usually our incident postmortem report comes out within days of the incident. However, since this issue was the result of an error at Stripe, we wanted to wait for their report before posting ours here. With that report in hand, we can now include their internal changes that will prevent this kind of issue going forward.

Incident root cause: Accidental disconnection from Stripe

On April 13, a mistake made by our payment processor, Stripe, resulted in a critical API key being deleted. This caused a temporary disruption across several of our products, including Giving and Registrations. During this outage, payments made on Church Center (web or mobile app) showed an error for one-off donations and payments. Recurring donations (which are programmatically triggered) also hit the same error, but they were quickly and successfully reprocessed once the new key was in place.

Customer impact: Disconnection emails caused confusion 

While the core issue was resolved quickly in the early morning hours, the automated email that followed the initial disconnection caused unnecessary confusion. It implied that churches had disconnected their own Stripe accounts, which wasn’t the case, and its urgent tone only added to the misunderstanding. To make matters worse, many customers had no way of knowing the issue had already been fixed unless they were familiar with our status page and knew to check there, which left some unsure whether their donation system was still down.

What we learned: Faster communication is better

One of the biggest takeaways from this incident was the importance of fast, plainspoken communication. Usually, churches want to know the details about what error messages were shown, exactly what parts of the system were impacted, and most of all what happened with the recurring donations. We wanted to speak to all of that accurately and confidently, but that cost us time. As soon as the system was back online we should have sent a quick communication just to say “Ignore that alert email, everything is fine. More details will follow” and then sent a followup email later.

What Stripe learned: Stronger protection measures for important keys 

Over at Stripe, on April 13th, a front line triage agent was responding to a fraud incident and deleted the wrong keys. As a result of this incident, they’ve removed the ability for the triage agents to delete a platform’s API key directly. To manage a platform’s API key (like Planning Center) there’s now a different internal escalation path that requires signoff at multiple levels within Stripe. 

What we’re changing going forward

We’re taking several steps to strengthen both our technical systems and our communication process, including:

  • Updating our on-call protocols to ensure that the right people on our side are collaborating more quickly
  • Biasing for speed of communication instead of just accuracy—starting with brief, reassuring updates, followed by detailed explanations.
  • Making the status page more visible throughout the helpdesk, the products themselves, and in email communications.

We know your church relies on Planning Center to operate smoothly, especially for things like donations and event registrations. We’re committed to learning from this and making our systems—and our communication—stronger moving forward. 

Thank you for your continued partnership and grace.

Posted May 15, 2025 - 16:11 PDT

Resolved

This incident has been resolved.
Posted Apr 13, 2025 - 06:24 PDT

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Apr 13, 2025 - 05:49 PDT

Identified

Donations and payments are being processed successfully now in Giving and Registrations. We are currently working on an issue related to connecting new Stripe accounts in Accounts.
Posted Apr 13, 2025 - 05:31 PDT

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Apr 13, 2025 - 04:49 PDT

Identified

We have identified the issue and are working on a fix.
Posted Apr 13, 2025 - 04:18 PDT

Investigating

We are investigating errors when trying to submit donations or add new payment methods in Giving and Registrations
Posted Apr 13, 2025 - 04:12 PDT
This incident affected: Accounts, Church Center, Giving, and Registrations.