+++
categories = ["software"]
tags = ["automation"]
date = 2024-06-18T02:47:45Z
description = ""
draft = false
slug = "email-alerts"
title = "📧 Email Alerts with Postfix & Amazon SES"
author = "nicholas"
+++
I have many jobs on a server that run on a schedule using `cron` jobs. I would like to be notified when these jobs succeed, and especially when they fail. I will send and receive these alerts through email.Since it is impractical to run one's own email server in 2024, I am using AWS SES (Amazon Web Services - Simple Email Service) to send the emails. I do not write anything about this here.
## Prerequisites
There are a few parts to configure to get the machine working.
- Email server
- `postfix`
- `mailutils`
## Postfix
This may prompt configuration GUI (`dpkg-reconfigure postfix`) in the terminal. Select "No configuration".
### Create auth file
An auth file is needed to store credentials before hashing.
```bash
sudo apt install postfix
sudo nano /etc/postfix/aws_ses_sasl
```
`SMTP_USERNAME` (SES Access Key for IAM User) and corresponding SMTP_PASSWORD, e.g.:
```
[email-smtp.us-east-2.amazonaws.com]:587 VMHPJVEZMOQJLJMPCHX:WF62TVHpGhqObyn2RvCb92sVxTP5OiihL3IvRmivHRk4j
```
Now, I use the postmap tool to generate a hashed database file that will eventually be used by Postfix to authenticate to the remote SMTP email server. It will output a .db file of the same name as the input file – in this case, `aws_ses_sasl.db`.
```shell
sudo postmap hash:/etc/postfix/aws_ses_sasl
```
### Create postfix configuration file
The auth file created above is referenced in the Postfix configuration file `main.cf` that is created below.
```shell
sudo nano /etc/postfix/main.cf
```
The last line above references a hashed database file. This file allows me to map my local email addresses to other addresses.
```sh
relayhost = [email-smtp.us-east-2.amazonaws.com]:587
smtp_sasl_auth_enable = yes
smtp_sasl_security_options = noanonymous
smtp_sasl_password_maps = hash:/etc/postfix/aws_ses_sasl
smtp_use_tls = yes
smtp_tls_security_level = encrypt
smtp_tls_note_starttls_offer = yes
smtp_generic_maps = hash:/etc/postfix/generic
```
### Alias Configuration
Server-generated email are by default sent to email address `{user}@{hostname}.localdomain`. I need to change this by configuring an alias for users `root` and `nicholas` in `/etc/aliases` so that all emails will be forwarded to my personal email instead of just ending up unread in my syslog.
```shell
sudo nano /etc/aliases
```
```shell
root: nicholas@email.com
nicholas: nicholas@email.com
```
### Create `generic` file
This will configure the sender address. Emails sent from the system will appear to be sent from the emails named here.
```shell
sudo nano /etc/postfix/generic
```
#### `generic`
```
root@server.localdomain example@example.com
nicholas@server.localdomain example@example.com
```
#### Create hash database file
```shell
sudo postmap hash:/etc/postfix/generic
```
### Restart Postfix
```shell
sudo service postfix restart
```
### Test Email
I want to test whether I have configured the email system correctly. I will need `mailutils`.
```shell
sudo apt install mailutils
```
I can set some variables in the shell to reference in a command:
```shell
email_body="test email body"
email_subject="test email subject"
email_to="test@example.com"
email_from="alerts@uuard.com"
```
Now I attempt to send the email.
```shell
echo "${email_body}" | mail -s "${email_subject}" -r "${email_from}" "${email_to}"
```
I check the mail log to verify the email was sent.
```shell
sudo cat /var/log/mail.log | grep to=
2024-06-17T19:30:17.705051-05:00 nas postfix/smtp[3797056]: D9D331661A14: to=, relay=email-smtp.us-east-2.amazonaws.com[3.22.8.243]:587, delay=0.82, delays=0.01/0/0.56/0.25, dsn=2.0.0, status=sent (250 Ok 010f019028c06bff-df62265f-49b2-4920-b73a-8a7e4b362f2c-000000)
```
Notice the `250 Ok`. This means the email was sent successfully. This is confirmed when I check my email inbox, which shows the email has been delivered:
{{< image
src="images/email-success.png"
caption="Test Email" >}}
## Shell Scripts
As I mentioned before, the purpose of configuring the system to send mail is to alert me to the outcome of the execution of scheduled jobs. To do this, I will create shell scripts for each job, within which will contain logic to send custom emails containing information that might be interesting to me.
I have a file `test.txt` that I output using `cat` command. Both the command and the output will be captured by the script, and the script will use the `mailutils` package `mail` command to send this over email. Here is what happens when I run the command manually:
### Formatting emails
The email I sent is not very pretty, and the lack of formatting will result in a nigh unreadable jumble of text if a job produces an error. I want to be able to quickly discern what and why a job failed, so I will format my emails.
The idea is to separate the command that resulted in exix code indicating error, from the error itself. To test this, I will use `cat` as the command, and the contents of a text file as the output (since `cat` will output the contents of the text file).
```shell
cat test.txt
testing
testing...
testing.......
testing..........
it works.
```
If I configure my script correctly, this exact command and output should show up in the email in separate columns. First, I configure the `test.sh` script.
```shell
cmd="cat test.txt"
cmd_output=$($cmd)
email_subject="cron job - show file contents"
email_to="test@email.com"
email_from="alerts@uuard.com"
# check if the cat command succeeded
if [ $? -ne 0 ]; then
email_subject="error!"
exit 1
fi
#create variable email_body
read -r -d '' email_body << EOM
| command |
${cmd}
|
| output |
${cmd_output} |
EOM
echo -e "${email_body}" | mail --content-type=text/html -s "${email_subject}" -r "${email_from}" "${email_to}"
```
{{< image
src="images/email-formatted.png"
caption="Email Formatted" >}}
## Improvements
### Templating
It occurs to me that it will be tedious and costly to maintain these configs if I want to change things. I decided to lift out the HTML template into a separate file so that I need only edit a single file. Now I have a few files to manage, but fewer lines of duplication between all of the configurations.
#### Template Script
```bash {filename="template-html-email.sh"}
template=$(cat <<'EOF'
| command |
${cmd}
|
| output |
${cmd_output} |
EOF
)
```
#### Send Error Email Script
```bash {filename="send-error-email.sh"}
#!/bin/bash
source dry-run-config.sh
cmd="$1"
cmd_output="$2"
export cmd
export cmd_output
email_body=$(echo "$template" | envsubst)
#echo "Command: $cmd"
#echo "Command Output: $cmd_output"
#echo "Email Body: $email_body"
printf "%s" "$email_body" | mail --content-type=text/html -s "${email_subject}" -r "${email_from}" "${email_to}"
```
#### Rclone Dry Run Config
```bash {filename="dry-run-config.sh"}
#!/bin/bash
send_error_email_script="send-error-email.sh"
template_html_email_script="template-html-email.sh"
email_subject="cron alert"
email_to="nicolaspatrickward@gmail.com"
email_from="alert@uuard.com"
source "$template_html_email_script"
```
#### Dry Run
```bash
#!/bin/bash
# dry run config -- shared variables
source dry-run-config.sh
# dry-run command parameters
src_dir="/source"
dest_remote="dry-run" #this is a remote rclone dir
dest_dir="/dest"
# save both the command as executed, and the command output. Pass along to email script for formatting
cmd="docker exec -t rclone rclone copy -v $src_dir $dest_remote:$dest_dir --dry-run"
cmd_output=$($cmd)
if [ $? -ne 0 ]; then
$send_error_email_script "$cmd" "$cmd_output"
fi
```
Done.