8.5 KiB
+++ categories = ["software"] tags = ["automation"] date = 2024-06-18T02:47:45Z description = "Configuring Postfix to relay cron-job success and failure alerts through Amazon SES, with reusable shell email formatting." draft = false slug = "email-alerts" title = "📧 Email Alerts with Postfix & Amazon SES" author = "nicholas" +++
I have many jobs on a server that run on a schedule using cron jobs. I would like to be notified when these jobs succeed, and especially when they fail. I will send and receive these alerts through email.Since it is impractical to run one's own email server in 2024, I am using AWS SES (Amazon Web Services - Simple Email Service) to send the emails. I do not write anything about this here.
Prerequisites
There are a few parts to configure to get the machine working.
- Email server
postfixmailutils
Postfix
This may prompt configuration GUI (dpkg-reconfigure postfix) in the terminal. Select "No configuration".
Create auth file
An auth file is needed to store credentials before hashing.
sudo apt install postfix
sudo nano /etc/postfix/aws_ses_sasl
SMTP_USERNAME (SES Access Key for IAM User) and corresponding SMTP_PASSWORD, e.g.:
[email-smtp.us-east-2.amazonaws.com]:587 VMHPJVEZMOQJLJMPCHX:WF62TVHpGhqObyn2RvCb92sVxTP5OiihL3IvRmivHRk4j
Now, I use the postmap tool to generate a hashed database file that will eventually be used by Postfix to authenticate to the remote SMTP email server. It will output a .db file of the same name as the input file – in this case, aws_ses_sasl.db.
sudo postmap hash:/etc/postfix/aws_ses_sasl
Create postfix configuration file
The auth file created above is referenced in the Postfix configuration file main.cf that is created below.
sudo nano /etc/postfix/main.cf
The last line above references a hashed database file. This file allows me to map my local email addresses to other addresses.
relayhost = [email-smtp.us-east-2.amazonaws.com]:587
smtp_sasl_auth_enable = yes
smtp_sasl_security_options = noanonymous
smtp_sasl_password_maps = hash:/etc/postfix/aws_ses_sasl
smtp_use_tls = yes
smtp_tls_security_level = encrypt
smtp_tls_note_starttls_offer = yes
smtp_generic_maps = hash:/etc/postfix/generic
Alias Configuration
Server-generated email are by default sent to email address {user}@{hostname}.localdomain. I need to change this by configuring an alias for users root and nicholas in /etc/aliases so that all emails will be forwarded to my personal email instead of just ending up unread in my syslog.
sudo nano /etc/aliases
root: nicholas@email.com
nicholas: nicholas@email.com
Create generic file
This will configure the sender address. Emails sent from the system will appear to be sent from the emails named here.
sudo nano /etc/postfix/generic
generic
root@server.localdomain example@example.com
nicholas@server.localdomain example@example.com
Create hash database file
sudo postmap hash:/etc/postfix/generic
Restart Postfix
sudo service postfix restart
Test Email
I want to test whether I have configured the email system correctly. I will need mailutils.
sudo apt install mailutils
I can set some variables in the shell to reference in a command:
email_body="test email body"
email_subject="test email subject"
email_to="test@example.com"
email_from="alerts@uuard.com"
Now I attempt to send the email.
echo "${email_body}" | mail -s "${email_subject}" -r "${email_from}" "${email_to}"
I check the mail log to verify the email was sent.
sudo cat /var/log/mail.log | grep to=
2024-06-17T19:30:17.705051-05:00 nas postfix/smtp[3797056]: D9D331661A14: to=, relay=email-smtp.us-east-2.amazonaws.com[3.22.8.243]:587, delay=0.82, delays=0.01/0/0.56/0.25, dsn=2.0.0, status=sent (250 Ok 010f019028c06bff-df62265f-49b2-4920-b73a-8a7e4b362f2c-000000)
Notice the 250 Ok. This means the email was sent successfully. This is confirmed when I check my email inbox, which shows the email has been delivered:
{{< image src="images/email-success.png" caption="Test Email" >}}
Shell Scripts
As I mentioned before, the purpose of configuring the system to send mail is to alert me to the outcome of the execution of scheduled jobs. To do this, I will create shell scripts for each job, within which will contain logic to send custom emails containing information that might be interesting to me.
I have a file test.txt that I output using cat command. Both the command and the output will be captured by the script, and the script will use the mailutils package mail command to send this over email. Here is what happens when I run the command manually:
Formatting emails
The email I sent is not very pretty, and the lack of formatting will result in a nigh unreadable jumble of text if a job produces an error. I want to be able to quickly discern what and why a job failed, so I will format my emails.
The idea is to separate the command that resulted in exix code indicating error, from the error itself. To test this, I will use cat as the command, and the contents of a text file as the output (since cat will output the contents of the text file).
cat test.txt
testing
testing...
testing.......
testing..........
it works.
If I configure my script correctly, this exact command and output should show up in the email in separate columns. First, I configure the test.sh script.
cmd="cat test.txt"
cmd_output=$($cmd)
email_subject="cron job - show file contents"
email_to="test@email.com"
email_from="alerts@uuard.com"
# check if the cat command succeeded
if [ $? -ne 0 ]; then
email_subject="error!"
exit 1
fi
#create variable email_body
read -r -d '' email_body << EOM
<html>
<head>
<style>
td, th {
font-family: helvetica;
color: #FBF7EF;
text-align:justify;
padding: 1em;
vertical-align:center;
}
table {
background: #2B303B;
}
</style>
</head>
<body>
<table>
<tr>
<th>command</th>
<td><pre>${cmd}</pre></th>
</tr>
<tr>
<th>output</th>
<td><pre>${cmd_output}</pre></td>
</table>
</body>
</html>
EOM
echo -e "${email_body}" | mail --content-type=text/html -s "${email_subject}" -r "${email_from}" "${email_to}"
{{< image src="images/email-formatted.png" caption="Email Formatted" >}}
Improvements
Templating
It occurs to me that it will be tedious and costly to maintain these configs if I want to change things. I decided to lift out the HTML template into a separate file so that I need only edit a single file. Now I have a few files to manage, but fewer lines of duplication between all of the configurations.
Template Script
template=$(cat <<'EOF'
<html lang="en">
<head>
<style>
td, th {
font-family: helvetica;
color: #FBF7EF;
text-align:justify;
padding: 1em;
vertical-align:center;
}
table {
background: #2B303B;
}
</style>
</head>
<body>
<table>
<tr>
<th>command</th>
<td><pre>${cmd}</pre></th>
</tr>
<tr>
<th>output</th>
<td><pre>${cmd_output}</pre></td>
</table>
</body>
</html>
EOF
)
Send Error Email Script
#!/bin/bash
source dry-run-config.sh
cmd="$1"
cmd_output="$2"
export cmd
export cmd_output
email_body=$(echo "$template" | envsubst)
#echo "Command: $cmd"
#echo "Command Output: $cmd_output"
#echo "Email Body: $email_body"
printf "%s" "$email_body" | mail --content-type=text/html -s "${email_subject}" -r "${email_from}" "${email_to}"
Rclone Dry Run Config
#!/bin/bash
send_error_email_script="send-error-email.sh"
template_html_email_script="template-html-email.sh"
email_subject="cron alert"
email_to="nicolaspatrickward@gmail.com"
email_from="alert@uuard.com"
source "$template_html_email_script"
Dry Run
#!/bin/bash
# dry run config -- shared variables
source dry-run-config.sh
# dry-run command parameters
src_dir="/source"
dest_remote="dry-run" #this is a remote rclone dir
dest_dir="/dest"
# save both the command as executed, and the command output. Pass along to email script for formatting
cmd="docker exec -t rclone rclone copy -v $src_dir $dest_remote:$dest_dir --dry-run"
cmd_output=$($cmd)
if [ $? -ne 0 ]; then
$send_error_email_script "$cmd" "$cmd_output"
fi
Done.