Increase Mautic Sending Speed with Amazon SES API
Amazon SES, a transactional email service (i.e. third-party email gateway), has two methods that web applications such as Mautic can use to connect with it: SMTP and API. SMTP (Simple Message Transfer Protocol) is the same protocol used by email clients like Thunderbird or Outlook to send outgoing messages through your email account. One of SMTP’s biggest limitations it that many applications can only send 1 email per SMTP connection, leading to a slow sending speed. Each time the sending server needs to reconnect to a SMTP server, there is a lot of overhead in renegotiating the connection.
For this reason, most transactional email services recommend web applications use their HTTP API to queue messages instead of traditional SMTP. Mautic supports the HTTP API for a variety of other email services including Elastic Email, SendGrid, and SparkPost, but support for API sending with Amazon SES is notably missing. According to GitHub comments from Mautic core developer, @escopecz, adopting the Amazon SES API should result in the ability to send emails in Mautic much faster at the rate limit allowed by Amazon SES (14 emails/sec = 840 emails/min or over 50K emails/hour for most new accounts) compared to a rate of 1 email/sec with SMTP. Theoretically, the HTTP-based Amazon SES API can process thousands of emails per request. Amazon SES users can always request a sending or rate limit increase by filing a support ticket from their AWS dashboard. Based on the history of the Mautic Github repo, Amazon SES API support has been widely requested since as early as 2016.
Many high-volume email senders prefer Amazon SES because they boast some of the lowest costs in the industry (US$0.10/1,000 messages) with no monthly minimums. Bounce tracking is also a pay-as-you-use service through Amazon SNS ($0.06/100K notification deliveries over HTTP) making it more cost effective than subscribing to Elastic Email’s Email API Pro plan, which applies a $1/day fee ($30/month) in addition to costs for any emails sent. With the sunsetting of SparkPost and SendGrid’s legacy free plans which allowed up to 40K emails/month (with no daily limits), Amazon SES remains the only major email gateway with bounce tracking via HTTP callback, with without a minimum monthly fee.
Amazon is also one of the largest senders of transactional email in the world, meaning that they have a proactive “abuse” department that monitors (and suspends) senders with high bounce and complaint rates. To maintain a good sender reputation, Amazon SES users are expected to have a bounce rate less than 5% and complaint rate less than 0.1%. By using Amazon SES, you can be confident that your Mautic email is using a shared IP pool with a reasonably good reputation, although your deliverability ultimately depends on the quality of your domain reputation as well. Domain reputation can be earned by sending enough emails over time with a low bounce & complaint rate. Domain reputation is portable between email providers, but before the advent of open source marketing automation systems such as Mautic, IP reputation was more difficult to maintain if you switched to a different email sending platform (ESP) – for example, from Constant Contact to MailChimp. With Amazon SES and Mautic together, the IP reputation that you build up in Amazon SES can be shared among all of your web applications, including other email marketing tools in your organization. Think of your Amazon SES sender score like a “credit score” that is portable from bank-to-bank – When you earn a good sender reputation in Amazon SES, you’re rewarded by higher deliverability rates to webmail providers such as Gmail and Outlook, as well as corporate email inboxes using commercial spam filters such as SpamAssassin or Barracuda. If you send a high volume of email regularly, you can even provision a dedicated IP address from Amazon SES, although that is only recommended for senders with a predictable sending pattern. Believe it or not, spam filters can treat a low volume of email from an IP as unfavorably as ramping up an email send too quickly (as both can be an indication of spamminess).
Going back to the issue of Mautic missing support for the Amazon SES API. As of Mautic 2.15.3, it can send only using the slower SMTP method “out of the box.” Although the Mautic community has developed a pull request with support for Amazon SES API, it has yet to be merged into the stable version of Mautic. Fortunately, there is a way to backport the Amazon SES API feature from the development branch (2.16.0-dev to 2.15.3) but it requires some technical knowledge of the command line / SSH and Composer, a PHP package manager. If you are not comfortable doing this yourself, we provide professional support with modifying your Mautic installation.
As of this writing (December 2019), it remains yet to be seen whether support for the Amazon SES API will be merged into the next release of Mautic, 2.16.0. Compared to previous attempts to develop Amazon SES API support for Mautic, it looks like this pull request is passing all the integration tests, and developer @bondas83 has marked this as a must-have feature for the next release.
Early Access to Amazon SES API Support in Mautic
So, if you want to have this “bleeding edge” feature on your current 2.15.3 instance of Mautic, you can follow the instructions below to install it. Please understand that this is pre-release code, and the final, released version may still change. If you patch your Mautic installation using the below code, don’t upgrade to the next Mautic version until it’s confirmed that Amazon SES API support was officially merged.
The instructions below assume that your instance of Mautic was installed by downloading the .zip file for a tagged (stable) release – not installed using Git, which does not include Composer out of the box. You first need to add Composer to your Mautic install directory, to ensure all of the dependencies are present.
Install Composer and Add Amazon SES API Support to Mautic 2.15.3
For your own protection, make a backup (e.g. cloud snapshot, or download files via FTP and mysqldump the Mautic database) of your Mautic instance before you proceed.
The below commands are assuming that the web server user is apache (on CentOS). If using Apache on Ubuntu, the default web server user is www-data, and if using Nginx (on any platform), it is nginx.
1 – Connect to your Mautic instance by SSH and change the PHP memory limit (memory_limit) in php.ini to at least 512M (1024M recommended). Restart the php-fpm service with sudo systemctl restart php-fpm if using PHP-FPM.
Also, add a swap file (e.g. 2G) to your system, if it does not already have one.
sudo dd if=/dev/zero of=/swapfile bs=1M count=2048
sudo mkswap /swapfile
sudo chmod 600 /swapfile
sudo swapon /swapfile
Add the following line to /etc/fstab.
/swapfile swap swap defaults 0 0
2 – Change to the directory, for example /var/www/html/, where your Mautic installation is stored. Get the composer.json and composer.lock files from the master (not staging) branch of the Mautic Github repo.
cd /var/www/html/
wget https://raw.githubusercontent.com/mautic/mautic/master/composer.json
wget https://raw.githubusercontent.com/mautic/mautic/master/composer.lock
3 – Install Composer locally in the Mautic directory.
php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');"
php -r "if (hash_file('sha384', 'composer-setup.php') === 'baf1608c33254d00611ac1705c1d9958c817a1a33bce370c0595974b342601bd80b92a3f46067da89e3b06bff421f182') { echo 'Installer verified'; } else { echo 'Installer corrupt'; unlink('composer-setup.php'); } echo PHP_EOL;"
php composer-setup.php
php -r "unlink('composer-setup.php');"
4 – Create the Composer cache directories.
mkdir -p /usr/share/httpd/.cache/composer/repo/https---repo.packagist.org/
mkdir -p /usr/share/httpd/.cache/composer/files/
mkdir -p /usr/share/httpd/.cache/composer/vcs/
sudo chown -R apache:apache /usr/share/httpd/.cache/
5 – Apply the patch for Amazon SES API support using this diffs file, modded by the Autoize team.
You cannot directly apply the pull request on Github as it makes references to PHP files only present in the development (not stable) version of Mautic.
sudo curl -L https://gist.githubusercontent.com/autoize/ee2fba90941dd39979d7990351e3d46e/raw/3691cc983eb1cfe8167aad96084d2b1514de2e4a/6977.diff | sudo -u apache git apply -v
6 – Run the Composer install and update, then clear Mautic cache.
If you run into any out of memory errors in Composer, go back to step 1 and ensure you have both increased the PHP memory limit and configured a swap file.
If this step doesn’t run correctly (to install all of the dependencies using Composer), you will encounter Mautic 500 errors such as the below:
mautic.CRITICAL: Uncaught PHP Exception Symfony\Component\Debug\Exception\ClassNotFoundException: "Attempted to load class "SingleProcessStorage" from namespace "bandwidthThrottle\tokenBucket\storage". Did you forget a "use" statement for another namespace?" at /var/www/html/app/bundles/EmailBundle/Swiftmailer/Transport/AmazonApiTransport.php line 319
sudo -u apache php composer.phar install
sudo -u apache php composer.phar update
sudo -u apache php app/console cache:clear
7 – Obtain a programmatic API key from the IAM service in Amazon AWS with the AmazonSESFullAccess permission.
If using Amazon SES in the us-east-1 (N. Virginia) region, visit https://console.aws.amazon.com/iam/home?region=us-east-1#/users$new?step=details for the IAM dashboard.
Select a username for this user (can be anything you like) and for Access type, check Programmatic access. Click Next: Permissions in the bottom right corner.
Select Attach existing policies directly, search for the AmazonSESFullAccess policy, and select the checkbox beside it in the list. Click Next: Tags (you can skip this step) and click Next: Review. Finally, click Create User and note down the access and secret keys in the final step.
8 – Login to your Mautic dashboard and go to Configuration > Email Settings.
From the Service to send email through dropdown, you will now see a new option that did not previously exist, Amazon SES – API. After you select that option, select the correct Amazon SES Region where your sending domain is verified (e.g. US East – N. Virginia).
For Username paste in the access key obtained in the previous step, and Password, the secret key. Finally, ensure emails are set to Send immediately and not to Queue.
Click Save & Close to apply the settings. Clicking Test Connection may reveal an error message like “Missing required client configuration options: region: (string) A “region” configuration value is required for the “email” service (e.g., “us-west-2”).” It is not necessary to test the connection, in order to save the email settings. This message can be safely ignored.
After the Email Settings are applied and saved, try sending a Test Email to yourself by clicking the Send test email button. This email will be sent to the email address set in your Mautic user profile. If you did everything above correctly, the email will be sent over the Amazon SES API transport and you’re now ready to send email campaigns up to 14X faster using the power of Mautic open source marketing automation.
If you encounter any issues, or would like professional assistance with setup, our Mautic consultants offer paid support with getting Amazon SES API email gateway set up on your Mautic instance.
