Incident Report: Notification Deliverability April 16-May 26, 2017
The following is the incident report for intermittent email errors with FormAssembly that have been occurring over the past several weeks. We know that you rely on our email notification system to ensure a quick and smooth process in your organization and we are very sorry for the additional work and time these issues have caused many of you.
From April 16th to today (May 26) you may have seen intermittent failures in the delivery of FormAssembly transactional emails, including email notifications of submitted responses. The emails were sent, but blocked and not delivered to the recipients’ inbox.
Some users have been able to bypass our abuse prevention systems and use our platform to send unsolicited emails. This affected our reputation as an email sender with several spam-blocking services and resulted in legitimate emails being intermittently rejected.
Resolution and recovery
May 26 Update
Following our initial resolution, we continued to get reports of email notification issues, and are still working to eradicate the issue permanently.
We are releasing an update next week to restrict access to send email notifications and autoresponders for new trial customers, this should alleviate the spam issue at its core.
Initial Resolution: April 24
As of Monday, April 24, all the rejected emails were cleared, FormAssembly email servers were completely off of block lists and emails were delivering normally.
After receiving alerts from spam-blocking services and identifying the problem, FormAssembly’s incident response team started working towards an immediate resolution. We worked with spam blockers and email vendors to remove our email servers from the block lists. We have always been very vigilant about email abuse and have systems in place to keep spammers out of our application, however we take this incident as an opportunity to augment the measures in place.
Corrective and Preventative Measures
During the end of the initial phase of these email notification issues we conducted an internal review and analysis of the outage. We took and are still taking the following actions to address the underlying causes of the issue and to help prevent recurrence:
- Add additional restrictions and verifications before users can send emails through our service.
- Expand our email infrastructure to mitigate the risk and impact of blacklisting.
It also became clear that we needed a better mechanism to quickly notify our customers of incidents related to their forms and data processing. We’re going to address this as well.
FormAssembly is committed to continually and quickly improving our technology and operational processes to prevent outages. We appreciate your patience and again apologize for the impact to you, your users, and your organization. We thank you for your business and continued support.
FormAssembly’s Incident Response Team