307 lines
8.4 KiB
Markdown
307 lines
8.4 KiB
Markdown
+++
|
||
categories = ["software"]
|
||
tags = ["automation"]
|
||
date = 2024-06-18T02:47:45Z
|
||
description = ""
|
||
draft = false
|
||
slug = "email-alerts"
|
||
title = "📧 Email Alerts with Postfix & Amazon SES"
|
||
author = "nicholas"
|
||
+++
|
||
|
||
I have many jobs on a server that run on a schedule using `cron` jobs. I would like to be notified when these jobs succeed, and especially when they fail. I will send and receive these alerts through email.Since it is impractical to run one's own email server in 2024, I am using AWS SES (Amazon Web Services - Simple Email Service) to send the emails. I do not write anything about this here.
|
||
|
||
## Prerequisites
|
||
There are a few parts to configure to get the machine working.
|
||
- Email server
|
||
- `postfix`
|
||
- `mailutils`
|
||
|
||
## Postfix
|
||
This may prompt configuration GUI (`dpkg-reconfigure postfix`) in the terminal. Select "No configuration".
|
||
|
||
### Create auth file
|
||
An auth file is needed to store credentials before hashing.
|
||
```bash
|
||
sudo apt install postfix
|
||
sudo nano /etc/postfix/aws_ses_sasl
|
||
```
|
||
|
||
`SMTP_USERNAME` (SES Access Key for IAM User) and corresponding SMTP_PASSWORD, e.g.:
|
||
|
||
```
|
||
[email-smtp.us-east-2.amazonaws.com]:587 VMHPJVEZMOQJLJMPCHX:WF62TVHpGhqObyn2RvCb92sVxTP5OiihL3IvRmivHRk4j
|
||
```
|
||
|
||
Now, I use the postmap tool to generate a hashed database file that will eventually be used by Postfix to authenticate to the remote SMTP email server. It will output a .db file of the same name as the input file – in this case, `aws_ses_sasl.db`.
|
||
|
||
```shell
|
||
sudo postmap hash:/etc/postfix/aws_ses_sasl
|
||
```
|
||
|
||
### Create postfix configuration file
|
||
The auth file created above is referenced in the Postfix configuration file `main.cf` that is created below.
|
||
|
||
```shell
|
||
sudo nano /etc/postfix/main.cf
|
||
```
|
||
|
||
The last line above references a hashed database file. This file allows me to map my local email addresses to other addresses.
|
||
|
||
```sh
|
||
relayhost = [email-smtp.us-east-2.amazonaws.com]:587
|
||
smtp_sasl_auth_enable = yes
|
||
smtp_sasl_security_options = noanonymous
|
||
smtp_sasl_password_maps = hash:/etc/postfix/aws_ses_sasl
|
||
smtp_use_tls = yes
|
||
smtp_tls_security_level = encrypt
|
||
smtp_tls_note_starttls_offer = yes
|
||
smtp_generic_maps = hash:/etc/postfix/generic
|
||
```
|
||
|
||
### Alias Configuration
|
||
Server-generated email are by default sent to email address `{user}@{hostname}.localdomain`. I need to change this by configuring an alias for users `root` and `nicholas` in `/etc/aliases` so that all emails will be forwarded to my personal email instead of just ending up unread in my syslog.
|
||
|
||
```shell
|
||
sudo nano /etc/aliases
|
||
```
|
||
|
||
```shell
|
||
root: nicholas@email.com
|
||
nicholas: nicholas@email.com
|
||
```
|
||
|
||
### Create `generic` file
|
||
This will configure the sender address. Emails sent from the system will appear to be sent from the emails named here.
|
||
|
||
```shell
|
||
sudo nano /etc/postfix/generic
|
||
```
|
||
|
||
#### `generic`
|
||
|
||
```
|
||
root@server.localdomain example@example.com
|
||
nicholas@server.localdomain example@example.com
|
||
```
|
||
|
||
#### Create hash database file
|
||
|
||
```shell
|
||
sudo postmap hash:/etc/postfix/generic
|
||
```
|
||
|
||
### Restart Postfix
|
||
|
||
```shell
|
||
sudo service postfix restart
|
||
```
|
||
|
||
### Test Email
|
||
I want to test whether I have configured the email system correctly. I will need `mailutils`.
|
||
|
||
```shell
|
||
sudo apt install mailutils
|
||
```
|
||
|
||
I can set some variables in the shell to reference in a command:
|
||
```shell
|
||
email_body="test email body"
|
||
email_subject="test email subject"
|
||
email_to="test@example.com"
|
||
email_from="alerts@uuard.com"
|
||
```
|
||
|
||
Now I attempt to send the email.
|
||
|
||
```shell
|
||
echo "${email_body}" | mail -s "${email_subject}" -r "${email_from}" "${email_to}"
|
||
```
|
||
|
||
I check the mail log to verify the email was sent.
|
||
|
||
```shell
|
||
sudo cat /var/log/mail.log | grep to=
|
||
2024-06-17T19:30:17.705051-05:00 nas postfix/smtp[3797056]: D9D331661A14: to=, relay=email-smtp.us-east-2.amazonaws.com[3.22.8.243]:587, delay=0.82, delays=0.01/0/0.56/0.25, dsn=2.0.0, status=sent (250 Ok 010f019028c06bff-df62265f-49b2-4920-b73a-8a7e4b362f2c-000000)
|
||
```
|
||
|
||
Notice the `250 Ok`. This means the email was sent successfully. This is confirmed when I check my email inbox, which shows the email has been delivered:
|
||
|
||
{{< image
|
||
src="images/email-success.png"
|
||
caption="Test Email" >}}
|
||
|
||
## Shell Scripts
|
||
As I mentioned before, the purpose of configuring the system to send mail is to alert me to the outcome of the execution of scheduled jobs. To do this, I will create shell scripts for each job, within which will contain logic to send custom emails containing information that might be interesting to me.
|
||
|
||
I have a file `test.txt` that I output using `cat` command. Both the command and the output will be captured by the script, and the script will use the `mailutils` package `mail` command to send this over email. Here is what happens when I run the command manually:
|
||
|
||
### Formatting emails
|
||
The email I sent is not very pretty, and the lack of formatting will result in a nigh unreadable jumble of text if a job produces an error. I want to be able to quickly discern what and why a job failed, so I will format my emails.
|
||
|
||
The idea is to separate the command that resulted in exix code indicating error, from the error itself. To test this, I will use `cat` as the command, and the contents of a text file as the output (since `cat` will output the contents of the text file).
|
||
|
||
```shell
|
||
cat test.txt
|
||
testing
|
||
testing...
|
||
testing.......
|
||
testing..........
|
||
it works.
|
||
```
|
||
|
||
If I configure my script correctly, this exact command and output should show up in the email in separate columns. First, I configure the `test.sh` script.
|
||
|
||
```shell
|
||
cmd="cat test.txt"
|
||
cmd_output=$($cmd)
|
||
|
||
email_subject="cron job - show file contents"
|
||
email_to="test@email.com"
|
||
email_from="alerts@uuard.com"
|
||
|
||
# check if the cat command succeeded
|
||
if [ $? -ne 0 ]; then
|
||
email_subject="error!"
|
||
exit 1
|
||
fi
|
||
|
||
#create variable email_body
|
||
read -r -d '' email_body << EOM
|
||
<html>
|
||
<head>
|
||
<style>
|
||
td, th {
|
||
font-family: helvetica;
|
||
color: #FBF7EF;
|
||
text-align:justify;
|
||
padding: 1em;
|
||
vertical-align:center;
|
||
}
|
||
table {
|
||
background: #2B303B;
|
||
}
|
||
</style>
|
||
</head>
|
||
<body>
|
||
<table>
|
||
<tr>
|
||
<th>command</th>
|
||
<td><pre>${cmd}</pre></th>
|
||
</tr>
|
||
<tr>
|
||
<th>output</th>
|
||
<td><pre>${cmd_output}</pre></td>
|
||
</table>
|
||
</body>
|
||
</html>
|
||
EOM
|
||
|
||
echo -e "${email_body}" | mail --content-type=text/html -s "${email_subject}" -r "${email_from}" "${email_to}"
|
||
|
||
```
|
||
|
||
{{< image
|
||
src="images/email-formatted.png"
|
||
caption="Email Formatted" >}}
|
||
|
||
|
||
## Improvements
|
||
### Templating
|
||
It occurs to me that it will be tedious and costly to maintain these configs if I want to change things. I decided to lift out the HTML template into a separate file so that I need only edit a single file. Now I have a few files to manage, but fewer lines of duplication between all of the configurations.
|
||
|
||
#### Template Script
|
||
|
||
```bash {filename="template-html-email.sh"}
|
||
template=$(cat <<'EOF'
|
||
<html lang="en">
|
||
<head>
|
||
<style>
|
||
td, th {
|
||
font-family: helvetica;
|
||
color: #FBF7EF;
|
||
text-align:justify;
|
||
padding: 1em;
|
||
vertical-align:center;
|
||
}
|
||
table {
|
||
background: #2B303B;
|
||
}
|
||
</style>
|
||
</head>
|
||
<body>
|
||
<table>
|
||
<tr>
|
||
<th>command</th>
|
||
<td><pre>${cmd}</pre></th>
|
||
</tr>
|
||
<tr>
|
||
<th>output</th>
|
||
<td><pre>${cmd_output}</pre></td>
|
||
</table>
|
||
</body>
|
||
</html>
|
||
EOF
|
||
)
|
||
```
|
||
|
||
#### Send Error Email Script
|
||
|
||
```bash {filename="send-error-email.sh"}
|
||
#!/bin/bash
|
||
source dry-run-config.sh
|
||
|
||
cmd="$1"
|
||
cmd_output="$2"
|
||
export cmd
|
||
export cmd_output
|
||
|
||
email_body=$(echo "$template" | envsubst)
|
||
|
||
#echo "Command: $cmd"
|
||
#echo "Command Output: $cmd_output"
|
||
#echo "Email Body: $email_body"
|
||
|
||
printf "%s" "$email_body" | mail --content-type=text/html -s "${email_subject}" -r "${email_from}" "${email_to}"
|
||
|
||
```
|
||
#### Rclone Dry Run Config
|
||
|
||
```bash {filename="dry-run-config.sh"}
|
||
#!/bin/bash
|
||
|
||
send_error_email_script="send-error-email.sh"
|
||
template_html_email_script="template-html-email.sh"
|
||
email_subject="cron alert"
|
||
email_to="nicolaspatrickward@gmail.com"
|
||
email_from="alert@uuard.com"
|
||
|
||
source "$template_html_email_script"
|
||
|
||
```
|
||
|
||
#### Dry Run
|
||
|
||
```bash
|
||
#!/bin/bash
|
||
|
||
# dry run config -- shared variables
|
||
source dry-run-config.sh
|
||
|
||
# dry-run command parameters
|
||
src_dir="/source"
|
||
dest_remote="dry-run" #this is a remote rclone dir
|
||
dest_dir="/dest"
|
||
|
||
# save both the command as executed, and the command output. Pass along to email script for formatting
|
||
cmd="docker exec -t rclone rclone copy -v $src_dir $dest_remote:$dest_dir --dry-run"
|
||
cmd_output=$($cmd)
|
||
|
||
if [ $? -ne 0 ]; then
|
||
$send_error_email_script "$cmd" "$cmd_output"
|
||
fi
|
||
|
||
```
|
||
|
||
Done.
|