Update: Mar 19, 2015, 10:15am (MDT): Our engineers believe the issue was caused primarily by a saturated state table on our primary loadbalancers that was compounded by the indeterminate uplink state of our redundant circuits. The problems combined to effectively black-hole new inbound connections. We'll continue evaluating the situation for potential improvements we can make to prevent this cascading effect in the future!
On Wednesday, March 18 around 12:32pm (MDT), our engineers became aware of connectivity issues with the Church Community Builder application and posted to the status twitter account (@ccbtech) that we were investigating the issue.
We believe the connectivity issues were the result of switch interface failures on one of the circuit links at our primary datacenter. Though we have redundant uplinks in the event an interface completely fails, this particular failure was not automatically detected and did not trigger a failover event. In order to restore service, we worked with datacenter technicians to bounce our uplink interfaces and we verified restored connectivity at approximately 12:55pm MDT. You can monitor the system uptime by going to status.churchcommunitybuilder.com.
Our engineers are working to fully debrief the connectivity issue and improve the performance of our networking equipment so this doesn't happen again. We know the impact that an untimely outage has on your church and we sincerely apologize for any inconvenience this has caused.
Thanks for allowing us to serve you!
The Church Community Builder Team
Be sure to follow @ccbtech on twitter to get quick updates on system status. You can also visit http://status.churchcommunitybuilder.com/ for an instant view of our uptime. While we try our hardest to prevent any outages, we know they can happen. We recommend ensuring your church has a Check-In Back-Up Plan in the event of another untimely outage.