Clean history, git LFS

2025-07-05 15:36:34 +00:00
commit 70423ce723
141 changed files with 8484 additions and 0 deletions
@@ -0,0 +1,306 @@
+++
+categories = ["software"]
+tags = ["automation"]
+date = 2024-06-18T02:47:45Z
+description = ""
+draft = false
+slug = "email-alerts"
+title = "📧 Email Alerts with Postfix & Amazon SES"
+author = "nicholas"
+++
+
+I have many jobs on a server that run on a schedule using `cron` jobs. I would like to be notified when these jobs succeed, and especially when they fail. I will send and receive these alerts through email.Since it is impractical to run one's own email server in 2024, I am using AWS SES (Amazon Web Services - Simple Email Service) to send the emails. I do not write anything about this here.
+
+## Prerequisites
+There are a few parts to configure to get the machine working.
+- Email server
+- `postfix`
+- `mailutils`
+
+## Postfix
+This may prompt configuration GUI (`dpkg-reconfigure postfix`) in the terminal. Select "No configuration".
+
+### Create auth file
+An auth file is needed to store credentials before hashing.
+```bash
+sudo apt install postfix
+sudo nano /etc/postfix/aws_ses_sasl
+```
+
+`SMTP_USERNAME` (SES Access Key for IAM User) and corresponding SMTP_PASSWORD, e.g.:
+
+```
+[email-smtp.us-east-2.amazonaws.com]:587 VMHPJVEZMOQJLJMPCHX:WF62TVHpGhqObyn2RvCb92sVxTP5OiihL3IvRmivHRk4j
+```
+
+Now, I use the postmap tool to generate a hashed database file that will eventually be used by Postfix to authenticate to the remote SMTP email server. It will output a .db file of the same name as the input file – in this case, `aws_ses_sasl.db`.
+
+```shell
+sudo postmap hash:/etc/postfix/aws_ses_sasl
+```
+
+### Create postfix configuration file
+The auth file created above is referenced in the Postfix configuration file `main.cf` that is created below.
+
+```shell
+sudo nano /etc/postfix/main.cf
+```
+
+The last line above references a hashed database file. This file allows me to map my local email addresses to other addresses.
+
+```sh
+relayhost = [email-smtp.us-east-2.amazonaws.com]:587
+smtp_sasl_auth_enable = yes
+smtp_sasl_security_options = noanonymous
+smtp_sasl_password_maps = hash:/etc/postfix/aws_ses_sasl
+smtp_use_tls = yes
+smtp_tls_security_level = encrypt
+smtp_tls_note_starttls_offer = yes
+smtp_generic_maps = hash:/etc/postfix/generic
+```
+
+### Alias Configuration
+Server-generated email are by default sent to email address `{user}@{hostname}.localdomain`. I need to change this by configuring an alias for users `root` and `nicholas` in `/etc/aliases` so that all emails will be forwarded to my personal email instead of just ending up unread in my syslog.
+
+```shell
+sudo nano /etc/aliases
+```
+
+```shell
+root: nicholas@email.com
+nicholas: nicholas@email.com
+```
+
+### Create `generic` file
+This will configure the sender address. Emails sent from the system will appear to be sent from the emails named here.
+
+```shell
+sudo nano /etc/postfix/generic
+```
+
+#### `generic`
+
+```
+root@server.localdomain example@example.com
+nicholas@server.localdomain example@example.com
+```
+
+#### Create hash database file
+
+```shell
+sudo postmap hash:/etc/postfix/generic
+```
+
+### Restart Postfix
+
+```shell
+sudo service postfix restart
+```
+
+### Test Email
+I want to test whether I have configured the email system correctly. I will need `mailutils`.
+
+```shell
+sudo apt install mailutils
+```
+
+I can set some variables in the shell to reference in a command:
+```shell
+email_body="test email body"
+email_subject="test email subject"
+email_to="test@example.com"
+email_from="alerts@uuard.com"
+```
+
+Now I attempt to send the email.
+
+```shell
+echo "${email_body}" | mail -s "${email_subject}" -r "${email_from}" "${email_to}"
+```
+
+I check the mail log to verify the email was sent.
+
+```shell
+sudo cat /var/log/mail.log | grep to=
+2024-06-17T19:30:17.705051-05:00 nas postfix/smtp[3797056]: D9D331661A14: to=, relay=email-smtp.us-east-2.amazonaws.com[3.22.8.243]:587, delay=0.82, delays=0.01/0/0.56/0.25, dsn=2.0.0, status=sent (250 Ok 010f019028c06bff-df62265f-49b2-4920-b73a-8a7e4b362f2c-000000)
+```
+
+Notice the `250 Ok`. This means the email was sent successfully. This is confirmed when I check my email inbox, which shows the email has been delivered:
+
+{{< image 
+    src="images/email-success.png" 
+    caption="Test Email" >}}
+
+## Shell Scripts
+As I mentioned before, the purpose of configuring the system to send mail is to alert me to the outcome of the execution of scheduled jobs. To do this, I will create shell scripts for each job, within which will contain logic to send custom emails containing information that might be interesting to me.
+
+I have a file `test.txt` that I output using `cat` command. Both the command and the output will be captured by the script, and the script will use the `mailutils` package `mail` command to send this over email. Here is what happens when I run the command manually:
+
+### Formatting emails
+The email I sent is not very pretty, and the lack of formatting will result in a nigh unreadable jumble of text if a job produces an error. I want to be able to quickly discern what and why a job failed, so I will format my emails.
+
+The idea is to separate the command that resulted in exix code indicating error, from the error itself. To test this, I will use `cat` as the command, and the contents of a text file as the output (since `cat` will output the contents of the text file).
+
+```shell
+cat test.txt
+testing
+testing...
+testing.......
+testing..........
+it works.
+```
+
+If I configure my script correctly, this exact command and output should show up in the email in separate columns. First, I configure the `test.sh` script.
+
+```shell
+cmd="cat test.txt"
+cmd_output=$($cmd)
+
+email_subject="cron job - show file contents"
+email_to="test@email.com"
+email_from="alerts@uuard.com"
+
+# check if the cat command succeeded
+if [ $? -ne 0 ]; then
+    email_subject="error!"
+exit 1
+fi
+
+#create variable email_body
+read -r -d '' email_body << EOM
+<html> 
+  <head>
+  <style>
+    td, th {
+      font-family: helvetica;
+      color: #FBF7EF;
+      text-align:justify;
+      padding: 1em;
+      vertical-align:center;
+    }
+    table {
+      background: #2B303B;
+    }
+  </style>
+  </head>
+  <body>
+    <table>
+    <tr>
+      <th>command</th>
+      <td><pre>${cmd}</pre></th>
+    </tr>
+    <tr>
+      <th>output</th>
+      <td><pre>${cmd_output}</pre></td>
+    </table>
+  </body>
+</html>
+EOM
+
+echo -e "${email_body}" | mail --content-type=text/html -s "${email_subject}" -r "${email_from}" "${email_to}"
+
+```
+
+{{< image 
+    src="images/email-formatted.png" 
+    caption="Email Formatted" >}}
+
+
+## Improvements 
+### Templating
+It occurs to me that it will be tedious and costly to maintain these configs if I want to change things. I decided to lift out the HTML template into a separate file so that I need only edit a single file. Now I have a few files to manage, but fewer lines of duplication between all of the configurations. 
+
+#### Template Script
+
+```bash {filename="template-html-email.sh"}
+template=$(cat <<'EOF'
+<html lang="en">
+  <head>
+  <style>
+    td, th {
+      font-family: helvetica;
+      color: #FBF7EF;
+      text-align:justify;
+      padding: 1em;
+      vertical-align:center;
+    }
+    table {
+      background: #2B303B;
+    }
+  </style>
+  </head>
+  <body>
+    <table>
+    <tr>
+      <th>command</th>
+      <td><pre>${cmd}</pre></th>
+    </tr>
+    <tr>
+      <th>output</th>
+      <td><pre>${cmd_output}</pre></td>
+    </table>
+  </body>
+</html>
+EOF
+)
+```
+
+#### Send Error Email Script
+
+```bash {filename="send-error-email.sh"}
+#!/bin/bash
+source dry-run-config.sh
+
+cmd="$1"
+cmd_output="$2"
+export cmd
+export cmd_output
+
+email_body=$(echo "$template" | envsubst)
+
+#echo "Command: $cmd"
+#echo "Command Output: $cmd_output"
+#echo "Email Body: $email_body"
+
+printf "%s" "$email_body" | mail --content-type=text/html -s "${email_subject}" -r "${email_from}" "${email_to}"
+
+```
+#### Rclone Dry Run Config
+
+```bash {filename="dry-run-config.sh"}
+#!/bin/bash
+
+send_error_email_script="send-error-email.sh"
+template_html_email_script="template-html-email.sh"
+email_subject="cron alert"
+email_to="nicolaspatrickward@gmail.com"
+email_from="alert@uuard.com"
+
+source "$template_html_email_script"
+
+```
+
+#### Dry Run
+
+```bash
+#!/bin/bash
+
+# dry run config -- shared variables
+source dry-run-config.sh
+
+# dry-run command parameters
+src_dir="/source"
+dest_remote="dry-run" #this is a remote rclone dir
+dest_dir="/dest"
+
+# save both the command as executed, and the command output. Pass along to email script for formatting
+cmd="docker exec -t rclone rclone copy -v $src_dir $dest_remote:$dest_dir --dry-run"
+cmd_output=$($cmd)
+
+if [ $? -ne 0 ]; then
+     $send_error_email_script "$cmd" "$cmd_output"
+fi
+
+```
+
+Done.