Post-Mortem: Service Disruption Caused by Excessive Outbound API Requests to Multitracks

Date of Incident: November 9, 2025
Duration: ~1 hour
Impacted: MultiTracks integration; indirect impact to Services and Check-Ins

Overview

On November 9, Planning Center experienced a disruption in our MultiTracks integration caused by an internal change to our caching logic. This change unintentionally triggered a large spike in outbound API requests to MultiTracks, which led to their service throttling our traffic. The resulting failures also caused secondary errors in Planning Center services that rely on responses from that API.

This issue was rooted in Planning Center’s system behavior, and we take full responsibility for the impact.

What Happened

A caching pattern change dramatically increased the number of API requests we sent to MultiTracks. Their service began throttling our traffic, and our retry behavior amplified the load on both systems. This led to elevated errors in the MultiTracks integration and related Planning Center processes.

The impact to customers was somewhat minimized because Music Stand apps use local caching, but some experienced errors, slow responses, or missing data.

Resolution

We quickly identified the regression and deployed a revert of the caching change. Once reverted, our outbound traffic returned to normal and MultiTracks’ throttling stopped, allowing dependent systems to recover.

Root Cause

• A caching change that generated excessive outbound API requests
• Retry and timeout behaviors that worsened cascading failures

Preventive Actions

1. Review and tune third-party timeouts to avoid cascading failures
2. Improve graceful degradation strategies for external dependencies
3. Enhance pre-deployment checks for changes affecting outbound request volume

Current Status

The MultiTracks integration is stable. We are implementing the above improvements to prevent similar issues and ensure our systems remain resilient even when dependencies become unstable.