Resolved -
We've resolved the incident and restored all data that was inadvertently modified.
This bug impacted profile information accessed in People either by an administrator or programmatically via a third-party integration between Friday, October 17, at 12:34 PM PT, and Sunday, October 19, at 9:49 AM PT. This affected various profile fields, including Permissions, Membership Type, Birthday, Anniversary, Campus, and Photos on a person’s profile. This did not include Custom Fields or Contact Information.
What this means:
While we restored as much data as possible, any manual changes made between Friday afternoon and Sunday morning were unfortunately lost.
There is one important exception: if an administrator intentionally removed someone's overall account permissions during this period, those removals were preserved. No one has permissions higher than what they should have.
What you can do:
Organization Administrators can visit https://accounts.planningcenteronline.com/affected_profiles to see profiles in your account that may have been affected by this issue. We recommend reviewing these profiles to ensure everything is as it should be.
Due to this bug, you may also notice some inconsistencies in the activity logs of these profiles.
Additional details:
* Approximately 3% of churches were impacted by this incident, affecting roughly 0.01% of all profiles
* The median number of affected profiles per church is 1, though a very small number of churches will see significantly more profiles affected, in some cases hundreds
* This issue was unrelated to but complicated by the recent AWS outage
* We have confirmed that no unauthorized access to data occurred during this incident
Our teams are conducting a thorough post-incident analysis to identify improvements to prevent similar issues and enhance our response processes.
We sincerely apologize for the disruption and appreciate your patience as we worked to resolve this issue. An email with similar information to this update has been sent to Organization Administrators of affected accounts.
Oct 22, 13:44 PDT
Update -
The large majority of restore work has been completed. The team continues to clean up some final pieces. We will post another update in the morning.
Oct 21, 17:57 PDT
Update -
We've completed the Groups restoration and are continuing to work towards fully restoring People collaborator permissions for things like workfows, lists, etc.
Oct 21, 15:19 PDT
Update -
We've completed our data restore work in Home, Services, Chat, Calendar, Registrations, Giving, Publishing, and Check-Ins. We're getting closer to restoring the secondary permission for People and Groups.
Oct 21, 11:55 PDT
Update -
Our team is continuing to make progress restoring and validating affected data and permissions. If you are still experiencing issues with logging in, in some cases clearing your browser's cache has shown to be effective.
Oct 21, 11:51 PDT
Update -
We've notified affected organization administrators with details about the issue, impact, and our estimated completion timeframe. The team is continuing to make progress restoring permissions and data across our products.
Oct 20, 21:54 PDT
Update -
Recovery work has resumed and is progressing well. We've restored core profile data and are validating that restoration, while also working to restore associated data across all applications.
Oct 20, 18:20 PDT
Update -
Recovery efforts have been paused due to the major outage in our upstream provider.
We will continue work in the morning. If any profiles in your account were affected you will receive an email with specifics later this week.
Oct 20, 00:51 PDT
Update -
We're still actively working on this. Thank you for your patience.
Oct 19, 18:35 PDT
Update -
We've identified the users who were impacted. We're now evaluating what changes we can revert and will update as soon as we know more.
Oct 19, 12:43 PDT
Update -
We're continuing to investigate to determine our options. We'll share an update as soon as we have more information.
Oct 19, 11:07 PDT
Monitoring -
We've implemented a fix to prevent further profile changes. We are now working to understand our options for affected profiles and will update as soon as we have clarity.
Oct 19, 09:58 PDT
Update -
Our team is still actively investigating reports of unintended profile updates and permission updates. We’ll share updates as soon as we know more.
Oct 19, 09:12 PDT
Investigating -
We are currently investigating reports of unintended profile updates and permission changes occurring during page refreshes.
Oct 19, 08:15 PDT
