Incident Report – May 11th 2010 – 30mn downtime

Today, at 4:15pm CST (21:15 GMT) what appears to be a hardware failure rendered our service inaccessible for about 30mn. We have switched our traffic to a secondary datacenter and operations were back to normal at 4:46pm CST. No data was lost, but form processing was down during that time.
We’re very sorry for the trouble caused by this interruption of service. We are still investigating the issue and will update this report when more information is available.
Thank you for your patience and understanding.
Our uptime report is available here:
You can follow us on twitter for the most current status reports.
UPDATE 5/12/2010: Although form processing was restored at 4:46CST on 5/11/2010, some users remained impacted by lingering issues.
1. Salesforce users who rely on a login IP range instead of a security token to authorize FormAssembly need to add our backup IP address to the list.
For more information visit our Salesforce documentation.
If you have responses that didn’t get into Salesforce because our backup server wasn’t authorized, you can resend the data from the connector log, or open a support request if you want us to take care of it. No data is lost, but your Salesforce org. will not be updated until you authorize our backup server.
2. A configuration error on our backup caused PayPal-integrated forms that uses the “encrypted cart” option to fail. The problem started at 4:46pm CST yesterday and was resolved at 12:20 CST today. Responses submitted during that time were received properly, but customers could not proceed with the PayPal payment part. In FormAssembly, these responses are visible and shown as complete, but there’s no payment receipt attached. We’re contacting affected users directly to help them recover their payments.

Don’t just collect data — leverage it.