Sync service issues
Reported on
The sync service is currently down. The issue is being looked into. Sorry for the disruption.
Update 1: The service is up, but not 100% stable yet. We are continuing to work on it.
Update 2: Unfortunately the sync service is still not in a stable condition. Our apologies for the unexpected service disruption.
Update 3: We are running some database operations in order to help with the sync issues. This process is unfortunately expected to take several hours.
Update 4: Work is continuing on trying to stabilize the sync service. Our apologies for the unreliable service.
Update 5: Currently two-thirds of sync accounts are being served as normal, but the other third are not. The root cause of this disruption is a database performance issue, where the database is no longer able to serve requests at a sufficient rate to keep up with traffic. We have added another database shard to alleviate the load and are rebalancing the data to the new shard.
Update 6: We are sorry for the slow updates, we have had our hands full trying to get our database backend for sync into shape again. It's been a slow process and we are still working on adding more resources to the current setup. We have had plans to add even more resources to our backend, but the growth in synced data stored by our users has been exceeding our plans. We are adding more resources to our current setup, but the process to balance that out before we try to start our sync services up again is taking some time. The data transfer to the new database resources is still in progress and we need to perform further tests before bringing the overloaded parts of our db back into production. We are really sorry for the downtime as we put pride in keeping everything running smoothly at all times.