Over the past 24 hours we’ve had a series of unscheduled downtime windows for both FormAssembly.com and FormAssembly Enterprise Cloud customers. We’re still working to resolve the issues, but we wanted to take a moment to tell you the causes and our plans for the future.
At 07:07 EST, our hosting provider while performing routine maintenance that should not have impacted operations caused an overload in one of their power supplies, causing a fraction of FormAssembly servers to go offline. We worked with the provider to get our servers back up and running, which was accomplished for the majority within 60 minutes.
At 16:10 EST, while attempting to repair the damage caused by the first incident, our hosting provider caused the same fraction of FormAssembly servers to offline for a second time. We’ve currently brought all servers back online.
To be clear, no data has been lost, but we’re currently working to ensure the integrity and safety of your data.
For the moment we’ve configured FormAssembly to operate in a ‘safe mode’ that will process data more slowly, but will be less like to experience further problems. We’ll remain in this mode for the next few hours until we satisfied with the state of affairs with our provider.
We realize you trust us with your data, and this sort of experience is unacceptable. We’re currently finalizing plans to distribute our resources more widely to prevent this type of single point of failure from ever occurring again.
Again we, and I, deeply apologize,
Drew + The FormAssembly Team