Compare commits
35 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 0d49d5fa72 | |||
| 691c7bd1ec | |||
| f10cc230af | |||
| 1ef0b96be4 | |||
| 0978154f43 | |||
| 9fdd3408a0 | |||
| 1eb46d98ef | |||
| 93faa527fe | |||
| b1ae9efe36 | |||
| 4422fba707 | |||
| 649a55eecd | |||
| 1936646b20 | |||
| 528497094b | |||
| fedd2d4f34 | |||
| 119f8b1152 | |||
| 5b0a087a78 | |||
| 29a34ce4e5 | |||
| 0540b1083c | |||
| 0ecaa2ce11 | |||
| 25c26e4f79 | |||
| b2dfb76a7e | |||
| 6312d0f563 | |||
| d959b5ec86 | |||
| 4b6db1e23d | |||
| e6a81e6d8d | |||
| faa8d9cc57 | |||
| a93f44600c | |||
| ae75afc69a | |||
| 95baab8e06 | |||
| 947740232c | |||
| 081a0fad4b | |||
| 1e1265ef1b | |||
| 9862689c0c | |||
| bed6c2a398 | |||
| 27c2be664a |
232
DMS/README.md
Normal file
232
DMS/README.md
Normal file
@@ -0,0 +1,232 @@
|
||||
# Spam-Scoring im Mailfluss — Stand und offene Punkte
|
||||
|
||||
Dieses Dokument hält fest, warum die rspamd-Konfiguration hier vom Standard
|
||||
abweicht und was noch offen ist. Hintergrund, der sich weder aus der Config
|
||||
noch aus der Git-History erschließt, gehört hierher.
|
||||
|
||||
## Mailfluss
|
||||
|
||||
```
|
||||
Externe Mail → SES → S3/SQS → email-worker (Container) → SMTP :25 → DMS
|
||||
```
|
||||
|
||||
Daraus folgt die zentrale Eigenheit: **rspamd sieht als Client-IP immer den
|
||||
Worker-Container**, nie den echten Absender-Server. Jede verbindungsbezogene
|
||||
Prüfung misst damit unsere eigene Infrastruktur. Ohne Gegenmaßnahme fallen
|
||||
systematisch Strafpunkte an, die nichts mit der Mail zu tun haben:
|
||||
|
||||
| Symbol | Gewicht |
|
||||
|---|---|
|
||||
| `HFILTER_HOSTNAME_UNKNOWN` | 6.0 (kein PTR für 172.x — exakt die `add_header`-Schwelle) |
|
||||
| `R_SPF_FAIL` / `R_SPF_SOFTFAIL` | 4.5 / 2.5 |
|
||||
| `R_DKIM_NA` / `R_DKIM_PERMFAIL` | 1.0 / 4.5 |
|
||||
|
||||
SPF/DKIM/DMARC gehen dadurch nicht verloren: SES prüft sie bereits mit der
|
||||
echten Absender-IP, das Ergebnis liegt im SQS-Payload
|
||||
(`receipt.spamVerdict`, `spfVerdict`, `dkimVerdict`, `dmarcVerdict`).
|
||||
|
||||
## Aktueller Stand
|
||||
|
||||
Schwellwerte in `docker-data/dms/config/rspamd/override.d/actions.conf`:
|
||||
`add_header = 6`, `reject = 500` (praktisch nie), `greylist = 500` (aus).
|
||||
Mit `MOVE_SPAM_TO_JUNK=1` bedeutet ein Score ab 6 also: Mail landet im Junk.
|
||||
|
||||
Die Settings-Regeln stehen in
|
||||
`docker-data/dms/config/rspamd/append.d/settings.conf`, ausgewertet nach
|
||||
Priorität, es greift nur der erste Treffer:
|
||||
|
||||
| Prio | Regel | Gilt für | Wirkung |
|
||||
|---|---|---|---|
|
||||
| 20 | `ses_worker_forwarded` | `X-SES-Worker-Processed: forwarded\|ooo-reply` | Auth-Gruppen aus **plus** `SPOOF_REPLYTO`, `FREEMAIL_REPLYTO*` |
|
||||
| 10 | `ses_worker_injected` | Worker-Header vorhanden (also `delivered`) | nur Auth-Gruppen aus |
|
||||
| 5 | `local_reinjection` | Docker-Netz ohne Worker-Header (Sieve-Redirects, Roundcube) | ersetzt die alte `DOCKER_WL`-Whitelist |
|
||||
|
||||
Alle drei sind zusätzlich an `172.16.0.0/12` / `127.0.0.0/8` gebunden. Ohne
|
||||
diese IP-Bedingung könnte ein externer Absender den Header selbst setzen und
|
||||
die Prüfungen abschalten.
|
||||
|
||||
**Was bewusst entfernt wurde:** die frühere `DOCKER_WL`-Regel (-50 Punkte für
|
||||
jede Mail mit eigener Absenderdomain) samt erzwungenem „no action" aus
|
||||
`force_actions.conf`. Sie war ein Spoofing-Loch — mit `SPOOF_PROTECTION=0`
|
||||
genügte ein `MAIL FROM: <x@bayarea-cc.com>` für garantierte Zustellung — und
|
||||
hat echten Spam an weitergeleitete Adressen durchgereicht.
|
||||
|
||||
---
|
||||
|
||||
## Offener Punkt B: Weiterleitungen ohne Neukomposition
|
||||
|
||||
**Status:** offen, bewusst zurückgestellt. Punkt A (die Settings-Regel
|
||||
`ses_worker_forwarded`) ist am 2026-08-15 umgesetzt worden und behebt das
|
||||
Symptom. B ist die strukturelle Lösung.
|
||||
|
||||
### Das Problem
|
||||
|
||||
`buildForwardMessage()` in
|
||||
`email-worker-nodejs/src/email/rules-processor.ts:212` baut für jede
|
||||
Weiterleitung eine **neue** Mail:
|
||||
|
||||
```ts
|
||||
from: recipient, // support@bayarea-cc.com — die getroffene lokale Adresse
|
||||
to: forwardTo, // ebrunks@bayarea-cc.com — das Weiterleitungsziel
|
||||
subject: `FWD: ${originalSubject}`,
|
||||
replyTo: originalFrom, // andreas.knuth@gmail.com — der Originalabsender
|
||||
```
|
||||
|
||||
Bei einer Weiterleitung an einen Kollegen in derselben Domain ergibt das
|
||||
formal die Signatur eines CEO-Fraud-Angriffs: Absender und Empfänger intern,
|
||||
die Antwort geht nach außen an eine Freemail-Adresse.
|
||||
|
||||
Belegt an Queue-ID `F228E320085` (2026-08-15 07:53:38):
|
||||
|
||||
```
|
||||
6.00 SPOOF_REPLYTO bayarea-cc.com;gmail.com
|
||||
2.00 FREEMAIL_REPLYTO_NEQ_FROM
|
||||
-0.10 MIME_GOOD
|
||||
0.05 MANY_INVISIBLE_PARTS
|
||||
----
|
||||
7.95 → add header → Junk
|
||||
```
|
||||
|
||||
Dieselbe Mail als Original an `support@` (`EC702320085`, eine Sekunde später):
|
||||
**-1.05, no action**. Der Unterschied kommt allein aus der Bauform der
|
||||
Weiterleitung, nicht aus dem Inhalt.
|
||||
|
||||
### Der Vorschlag
|
||||
|
||||
Statt eine neue Mail zu komponieren, die Originalmail **unverändert
|
||||
weiterreichen** (Remailing): `From:` bleibt der Originalabsender,
|
||||
Envelope-From wird eine lokale Bounce-Adresse, kein `Reply-To` mehr,
|
||||
kein `FWD:`-Präfix.
|
||||
|
||||
Vorteile über die Spam-Bewertung hinaus:
|
||||
|
||||
- Threading in den Clients bleibt intakt (`Message-ID`, `References`)
|
||||
- Originalbetreff und Originalheader bleiben erhalten
|
||||
- Anhänge müssen nicht neu verpackt werden
|
||||
- Das Muster, das `SPOOF_REPLYTO` auslöst, entsteht gar nicht erst — die
|
||||
Ausnahme in `ses_worker_forwarded` könnte danach wieder enger gefasst
|
||||
oder ganz entfernt werden
|
||||
|
||||
### Warum es nicht einfach nur „besser" ist
|
||||
|
||||
Das ist ein echter Eingriff mit Folgen, die vor der Umsetzung geklärt werden
|
||||
müssen:
|
||||
|
||||
1. **Bounces.** Envelope-From muss auf eine Adresse zeigen, die wir selbst
|
||||
auswerten — sonst gehen Zustellfehler an den Originalabsender, der mit
|
||||
unserer Weiterleitung nichts zu tun hat. Klassische Lösung ist SRS;
|
||||
`ENABLE_SRS=0` ist derzeit gesetzt.
|
||||
2. **DMARC bei externen Zielen — das ist der harte Punkt.**
|
||||
`handleForwards()` unterscheidet: interne Ziele gehen über
|
||||
`sendInternalEmail()` auf Port 25, externe über `ses.sendRawEmail()`.
|
||||
Beide setzen den Envelope-From auf `recipient`, also auf unsere eigene
|
||||
Domain. Heute ist das für externe Weiterleitungen sogar ein Vorteil: SES
|
||||
signiert mit unserem DKIM, SPF passt, und weil `From:` ebenfalls unsere
|
||||
Domain ist, ist DMARC sauber ausgerichtet.
|
||||
|
||||
Genau das würde Remailing zerstören. Mit `From:` = Originalabsender prüft
|
||||
Gmail DMARC gegen dessen Domain — und wir erfüllen weder deren SPF (die
|
||||
Mail kommt von SES) noch deren DKIM (wir haben den Body umgebaut).
|
||||
Das ist das klassische Forwarding-Problem. SRS löst den SPF-Teil,
|
||||
`ENABLE_SRS=0` ist derzeit gesetzt; für DKIM hilft nur, die Mail wirklich
|
||||
**byte-identisch** durchzureichen, inklusive Originalheader.
|
||||
|
||||
Praktische Konsequenz: B ließe sich zunächst **nur für interne Ziele**
|
||||
umsetzen und für externe beim jetzigen Verhalten bleiben. Das löst den
|
||||
`SPOOF_REPLYTO`-Fall — der tritt ohnehin nur intern auf, weil er
|
||||
`From`-Domain == `To`-Domain voraussetzt — ohne die externe
|
||||
Zustellbarkeit anzufassen.
|
||||
3. **Absenderdarstellung.** Im Posteingang steht dann der Originalabsender,
|
||||
nicht mehr `support@`. Ob das gewünscht ist, ist eine fachliche
|
||||
Entscheidung — Mailinglisten lösen es über
|
||||
`From: "Name via support" <support@bayarea-cc.com>`, was allerdings
|
||||
`SPOOF_REPLYTO` teilweise zurückholt.
|
||||
|
||||
### Abnahmekriterium
|
||||
|
||||
Eine Weiterleitung einer externen Mail an eine interne Adresse zeigt in
|
||||
`rspamd_report.sh --detail` einen Score unter 6 **ohne** die Ausnahme für
|
||||
`SPOOF_REPLYTO`, und ein Reply des Empfängers erreicht den Originalabsender.
|
||||
|
||||
---
|
||||
|
||||
## Weitere offene Punkte
|
||||
|
||||
### `fuzzy_check` wieder einschalten
|
||||
|
||||
Abgeschaltet in `docker-data/dms/config/rspamd/local.d/fuzzy_check.conf`, weil
|
||||
UDP 11335 ausgehend nicht durchkommt (vermutlich Egress-Filter des Providers).
|
||||
Jede Mail lief in denselben Timeout und kostete konstant ~4,3 s Scanzeit ohne
|
||||
jeden Gegenwert. Nach dem Abschalten liegen die Scans bei 70–580 ms.
|
||||
|
||||
Fuzzy erkennt Massen-Spam über Hashes — genau die Klasse, die weder
|
||||
Auth-Prüfungen noch ein frisch angelernter Bayes zuverlässig erwischt. Das
|
||||
lohnt sich mit dem Provider zu klären. Test:
|
||||
|
||||
```
|
||||
docker exec mailserver rspamadm fuzzy_ping -s rspamd.com
|
||||
```
|
||||
|
||||
### Bayes anlernen
|
||||
|
||||
`BAYES_*` feuert erst ab 200 gelernten Nachrichten **je Klasse**, aktuell also
|
||||
gar nicht. `docker-data/dms/config/train_bayes.sh` lernt einmalig aus den
|
||||
vorhandenen Postfächern (Junk → Spam, INBOX → Ham), Default ist Trockenlauf.
|
||||
|
||||
Wichtig: **nur kuratierte Postfächer** über `--only` einbeziehen. Ohne Filter
|
||||
läuft das Skript über alle Postfächer aller Domains unter `/var/mail` — bei
|
||||
Weiterleitungsadressen ist die INBOX ein ungefilterter Durchlauf und als
|
||||
Ham-Quelle unbrauchbar.
|
||||
|
||||
Reihenfolge beachten: erst falsch einsortierte Mail abstellen, dann lernen.
|
||||
Solange legitime Weiterleitungen im Junk landen, lernt `RSPAMD_LEARN=1` sie
|
||||
dort automatisch als Spam an und vergiftet den Klassifikator.
|
||||
|
||||
### SES-Verdicts als Header stempeln
|
||||
|
||||
Der Worker könnte `receipt.spfVerdict` / `dkimVerdict` / `dmarcVerdict` aus
|
||||
dem SQS-Payload als Header setzen. Dann ließen sich die echten,
|
||||
mit der richtigen Absender-IP ermittelten Auth-Ergebnisse per multimap wieder
|
||||
bepunkten, statt sie ersatzlos abzuschalten.
|
||||
|
||||
---
|
||||
|
||||
## Betrieb
|
||||
|
||||
### Auswertung
|
||||
|
||||
```
|
||||
docker exec mailserver bash /tmp/docker-mailserver/rspamd_report.sh -n 20
|
||||
docker exec mailserver bash /tmp/docker-mailserver/rspamd_report.sh --detail <QID>
|
||||
docker exec mailserver bash /tmp/docker-mailserver/rspamd_report.sh --slow 1000
|
||||
```
|
||||
|
||||
Das Feld `Regel:` in der Detailansicht zeigt, welche Settings-Regel gegriffen
|
||||
hat — der schnellste Weg zu prüfen, ob eine Änderung überhaupt aktiv ist.
|
||||
Voraussetzung ist `level = "info"` in `rspamd/local.d/logging.inc`; der
|
||||
DMS-Default `silent` protokolliert nichts.
|
||||
|
||||
### Änderungen an `append.d/` ausrollen
|
||||
|
||||
`user-patches.sh` hängt die Dateien aus `rspamd/append.d/` an die
|
||||
`local.d`-Pendants an und überspringt das, wenn der Marker `DMS-CUSTOM-APPEND`
|
||||
dort schon steht. `/etc/rspamd` liegt im Container-Dateisystem und überlebt
|
||||
`docker restart` — **ein Neustart übernimmt Änderungen also nicht.** Nötig ist:
|
||||
|
||||
```
|
||||
docker compose up -d --force-recreate mailserver
|
||||
```
|
||||
|
||||
Das Anhängen statt Überschreiben ist zwingend: DMS legt in `settings.conf`
|
||||
selbst eine Regel für authentifizierte User ab (`DMS::SED_TAG::1`), die ein
|
||||
`cp` löschen würde — womit `RSPAMD_CHECK_AUTHENTICATED=0` wirkungslos wäre.
|
||||
|
||||
### DNS
|
||||
|
||||
rspamds Blocklisten brauchen einen eigenen Recursor. Über Dockers eingebautes
|
||||
DNS (127.0.0.11) stufte Spamhaus die Anfragen als von einem offenen Resolver
|
||||
kommend ein und antwortete nur mit `DBL_BLOCKED_OPENRESOLVER`. Gelöst über den
|
||||
Service `dns-recursor` (PowerDNS) im eigenen Netz `dms_dns` mit fester IP —
|
||||
fest deshalb, weil Dockers `dns:`-Option nur IPs akzeptiert.
|
||||
|
||||
`ENABLE_UNBOUND` existiert in DMS nicht und war ein No-Op.
|
||||
58
DMS/batch_imapsync.sh
Normal file
58
DMS/batch_imapsync.sh
Normal file
@@ -0,0 +1,58 @@
|
||||
#!/bin/bash
|
||||
# batch_imapsync.sh - Führt IMAP-Sync für alle User im Hintergrund aus
|
||||
# Format der CSV: email@domain.com,SecretPassword123
|
||||
|
||||
HOST1=$1
|
||||
HOST2=$2
|
||||
CSV_FILE=$3
|
||||
|
||||
if [ -z "$HOST1" ] || [ -z "$HOST2" ] || [ -z "$CSV_FILE" ]; then
|
||||
echo "Usage: $0 <source-host> <target-host> <users.csv>"
|
||||
echo "Beispiel: $0 secure.emailsrvr.com 147.93.132.244 stxmaterials.csv"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# ======================================================================
|
||||
# Die eigentliche Sync-Funktion (wird in den Hintergrund geschickt)
|
||||
# ======================================================================
|
||||
run_sync_jobs() {
|
||||
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
|
||||
LOG_DIR="sync_logs_$TIMESTAMP"
|
||||
mkdir -p "$LOG_DIR"
|
||||
|
||||
echo "Beginne Sync-Lauf am $(date)" > "batch_master_${TIMESTAMP}.log"
|
||||
|
||||
while IFS=, read -r email password; do
|
||||
email=$(echo "$email" | tr -d '\r' | xargs)
|
||||
password=$(echo "$password" | tr -d '\r' | xargs)
|
||||
|
||||
[ -z "$email" ] && continue
|
||||
|
||||
LOGFILE="$LOG_DIR/imapsync_${email}.log"
|
||||
echo "[$(date)] Syncing $email -> $LOGFILE" >> "batch_master_${TIMESTAMP}.log"
|
||||
|
||||
# Führe Docker imapsync für den aktuellen User aus
|
||||
docker run --rm gilleslamiral/imapsync imapsync \
|
||||
--host1 "$HOST1" --user1 "$email" --password1 "$password" --ssl1 \
|
||||
--host2 "$HOST2" --user2 "$email" --password2 "$password" --ssl2 \
|
||||
--automap > "$LOGFILE" 2>&1 < /dev/null
|
||||
|
||||
done < "$CSV_FILE"
|
||||
|
||||
echo "Alle Sync-Jobs beendet am $(date)" >> "batch_master_${TIMESTAMP}.log"
|
||||
}
|
||||
|
||||
# ======================================================================
|
||||
# Skript-Start: Entkopplung vom Terminal
|
||||
# ======================================================================
|
||||
echo "🚀 Starte Batch-IMAP-Sync im Hintergrund..."
|
||||
|
||||
# Rufe die Funktion auf, leite alle restlichen Ausgaben ins Nichts und schicke sie in den Hintergrund (&)
|
||||
run_sync_jobs </dev/null >/dev/null 2>&1 &
|
||||
|
||||
echo "✅ Der Job läuft jetzt autark im Hintergrund (sequenziell)."
|
||||
echo "Du kannst das SSH-Terminal jetzt bedenkenlos schließen!"
|
||||
echo "Überwache den Gesamtfortschritt mit:"
|
||||
echo " tail -f batch_master_*.log"
|
||||
echo "Oder die Details eines einzelnen Postfachs mit:"
|
||||
echo " tail -f sync_logs_*/imapsync_<email>.log"
|
||||
@@ -9,10 +9,10 @@ services:
|
||||
|
||||
# Node-spezifischer Hostname - A-Record zeigt auf DIESEN Server.
|
||||
# email-srvr.com selbst zeigt auf einen anderen Server und wird hier NICHT verwendet.
|
||||
hostname: node1.email-srvr.com
|
||||
hostname: ${NODE_HOSTNAME}
|
||||
|
||||
ports:
|
||||
- "25:25"
|
||||
- "127.0.0.1:25:25"
|
||||
- "587:587"
|
||||
- "465:465"
|
||||
- "143:143"
|
||||
@@ -26,7 +26,13 @@ services:
|
||||
- ./docker-data/dms/mail-state/:/var/mail-state/
|
||||
- ./docker-data/dms/mail-logs/:/var/log/mail/
|
||||
- ./docker-data/dms/config/:/tmp/docker-mailserver/
|
||||
- ./docker-data/dms/config/dovecot/conf.d/95-sieve-redirect.conf:/etc/dovecot/conf.d/95-sieve-redirect.conf:ro
|
||||
# ENTFERNT: Mount von dovecot/conf.d/95-sieve-redirect.conf
|
||||
# Der Host-Pfad existiert nicht, Docker hat daraufhin ein VERZEICHNIS
|
||||
# unter /etc/dovecot/conf.d/95-sieve-redirect.conf angelegt (nachweisbar
|
||||
# per `docker exec mailserver cat ...` -> "Is a directory"). Der Mount war
|
||||
# also wirkungslos. Falls dort spaeter wirklich etwas gebraucht wird
|
||||
# (z.B. sieve_redirect_envelope_from), erst die Datei auf dem Host
|
||||
# anlegen, dann den Mount wieder aufnehmen.
|
||||
- /etc/localtime:/etc/localtime:ro
|
||||
- ./sync_dynamodb_to_sieve.py:/scripts/sync.py:ro
|
||||
- ./sieve-cron:/etc/cron.d/sieve-sync:ro
|
||||
@@ -61,11 +67,16 @@ services:
|
||||
# Kundendomain-SNI wird über postfix-main.cf + dovecot-sni.cf gesteuert.
|
||||
# -------------------------------------------------------
|
||||
- SSL_TYPE=manual
|
||||
- SSL_CERT_PATH=/etc/mail/certs/node1.email-srvr.com/node1.email-srvr.com.crt
|
||||
- SSL_KEY_PATH=/etc/mail/certs/node1.email-srvr.com/node1.email-srvr.com.key
|
||||
- SSL_CERT_PATH=/etc/mail/certs/${NODE_HOSTNAME}/${NODE_HOSTNAME}.crt
|
||||
- SSL_KEY_PATH=/etc/mail/certs/${NODE_HOSTNAME}/${NODE_HOSTNAME}.key
|
||||
|
||||
# SPAM / Rspamd
|
||||
- ENABLE_OPENDKIM=1
|
||||
# OpenDKIM deaktiviert: es liegen keine Keys unter config/opendkim/, der
|
||||
# Milter hat also nie etwas signiert. Ausgehende Mail wird von Amazon SES
|
||||
# signiert (Easy DKIM), eingehende DKIM-Pruefung macht rspamd selbst.
|
||||
# DMS warnt beim Start ausdruecklich davor, OpenDKIM parallel zu rspamd
|
||||
# zu betreiben.
|
||||
- ENABLE_OPENDKIM=0
|
||||
- ENABLE_OPENDMARC=0
|
||||
- ENABLE_POLICYD_SPF=0
|
||||
- ENABLE_RSPAMD=1
|
||||
@@ -80,7 +91,10 @@ services:
|
||||
|
||||
# Sicherheit
|
||||
- ENABLE_FAIL2BAN=1
|
||||
- ENABLE_UNBOUND=1
|
||||
# ENABLE_UNBOUND entfernt: die Variable existiert in DMS nicht (mehr) und
|
||||
# war wirkungslos - rspamd hat ueber Dockers eingebautes DNS (127.0.0.11)
|
||||
# aufgeloest und bekam von Spamhaus nur DBL_BLOCKED_OPENRESOLVER zurueck.
|
||||
# Ersetzt durch den dns-recursor-Service weiter unten.
|
||||
|
||||
# Sonstige
|
||||
- ENABLE_MANAGESIEVE=0
|
||||
@@ -95,7 +109,10 @@ services:
|
||||
|
||||
# Amazon SES Relay
|
||||
- RELAY_HOST=email-smtp.us-east-2.amazonaws.com
|
||||
- RELAY_PORT=587
|
||||
# Node-spezifisch: manche Provider blockieren ausgehend 587.
|
||||
# Wert pro Server in der .env setzen (z.B. RELAY_PORT=2587),
|
||||
# ohne Eintrag bleibt es bei 587.
|
||||
- RELAY_PORT=${RELAY_PORT:-587}
|
||||
- RELAY_USER=${SES_SMTP_USER}
|
||||
- RELAY_PASSWORD=${SES_SMTP_PASSWORD}
|
||||
|
||||
@@ -107,7 +124,7 @@ services:
|
||||
# Postfix
|
||||
# POSTFIX_OVERRIDE_HOSTNAME: Was Postfix im EHLO/HELO Banner sendet.
|
||||
# node1.email-srvr.com passt zum TLS-Cert und ist der echte Hostname.
|
||||
- POSTFIX_OVERRIDE_HOSTNAME=node1.email-srvr.com
|
||||
- POSTFIX_OVERRIDE_HOSTNAME=${NODE_HOSTNAME}
|
||||
- POSTFIX_MYNETWORKS=172.16.0.0/12 172.17.0.0/12 172.18.0.0/12 [::1]/128 [fe80::]/64
|
||||
- POSTFIX_MAILBOX_SIZE_LIMIT=0
|
||||
- POSTFIX_MESSAGE_SIZE_LIMIT=0
|
||||
@@ -116,11 +133,47 @@ services:
|
||||
- NET_ADMIN
|
||||
- SYS_PTRACE
|
||||
restart: unless-stopped
|
||||
|
||||
# -------------------------------------------------------
|
||||
# DNS: rspamd fragt DNS-Blocklisten (Spamhaus DBL/ZEN, SURBL) ab. Diese
|
||||
# Dienste beantworten keine Anfragen, die ueber oeffentliche Resolver
|
||||
# kommen - man bekommt DBL_BLOCKED_OPENRESOLVER statt eines Ergebnisses.
|
||||
# Deshalb ein eigener rekursiver Resolver, siehe Service dns-recursor.
|
||||
#
|
||||
# ACHTUNG: Damit ersetzt der Recursor Dockers eingebautes DNS (127.0.0.11)
|
||||
# fuer DIESEN Container. Der mailserver kann dann keine Container-Namen
|
||||
# mehr aufloesen (roundcube-db, email-worker-ts, ...). Das ist hier
|
||||
# unkritisch, weil er von sich aus nur nach aussen verbindet (SES-Relay).
|
||||
# Faellt der Recursor aus, hat der mailserver kein DNS mehr - deshalb
|
||||
# restart: always beim Recursor.
|
||||
# -------------------------------------------------------
|
||||
dns:
|
||||
- 10.99.0.53
|
||||
depends_on:
|
||||
- dns-recursor
|
||||
|
||||
networks:
|
||||
mail_network:
|
||||
aliases:
|
||||
- mailserver
|
||||
- node1.email-srvr.com
|
||||
- ${NODE_HOSTNAME}
|
||||
dns_network:
|
||||
|
||||
# ---------------------------------------------------------
|
||||
# Rekursiver DNS-Resolver ausschliesslich fuer den mailserver.
|
||||
# Er fragt die Root-Server selbst ab, statt an einen Upstream-Resolver
|
||||
# weiterzuleiten - nur so akzeptiert Spamhaus die Anfragen.
|
||||
# Braucht keine Konfiguration: das Image laeuft per Default als Recursor
|
||||
# und erlaubt Anfragen aus den RFC1918-Netzen (10.99.0.0/24 liegt darin).
|
||||
# ---------------------------------------------------------
|
||||
dns-recursor:
|
||||
image: powerdns/pdns-recursor-53:5.3.10
|
||||
container_name: dns-recursor
|
||||
restart: always
|
||||
stop_grace_period: 0s
|
||||
networks:
|
||||
dns_network:
|
||||
ipv4_address: 10.99.0.53
|
||||
|
||||
roundcube:
|
||||
image: roundcube/roundcubemail:latest
|
||||
@@ -136,10 +189,10 @@ services:
|
||||
- ROUNDCUBEMAIL_DB_USER=roundcube
|
||||
- ROUNDCUBEMAIL_DB_PASSWORD=${ROUNDCUBE_DB_PASSWORD}
|
||||
# Roundcube verbindet intern über den Docker-Alias
|
||||
- ROUNDCUBEMAIL_DEFAULT_HOST=ssl://node1.email-srvr.com
|
||||
- ROUNDCUBEMAIL_DEFAULT_HOST=ssl://${NODE_HOSTNAME}
|
||||
- ROUNDCUBEMAIL_DEFAULT_PORT=993
|
||||
# Interner Traffic ohne TLS
|
||||
- ROUNDCUBEMAIL_SMTP_SERVER=ssl://node1.email-srvr.com
|
||||
- ROUNDCUBEMAIL_SMTP_SERVER=ssl://${NODE_HOSTNAME}
|
||||
- ROUNDCUBEMAIL_SMTP_PORT=465
|
||||
|
||||
# WICHTIG: Variablen LEER lassen, damit Roundcube keine Authentifizierung versucht!
|
||||
@@ -176,3 +229,16 @@ services:
|
||||
networks:
|
||||
mail_network:
|
||||
external: true
|
||||
|
||||
# Eigenes Netz nur fuer mailserver <-> dns-recursor.
|
||||
# Bewusst NICHT im externen mail_network: dort vergibt Docker die Adressen
|
||||
# dynamisch, eine feste IP koennte mit einem anderen Container kollidieren.
|
||||
# Das Subnetz muss fest sein, weil Dockers `dns:`-Option nur IPs akzeptiert.
|
||||
# Falls 10.99.0.0/24 bei euch schon belegt ist (VPN, AWS-VPC), hier und bei
|
||||
# `dns:` sowie `ipv4_address` oben gemeinsam auf ein freies Netz aendern.
|
||||
dns_network:
|
||||
name: dms_dns
|
||||
ipam:
|
||||
driver: default
|
||||
config:
|
||||
- subnet: 10.99.0.0/24
|
||||
|
||||
135
DMS/docker-data/dms/config/rspamd/append.d/settings.conf
Normal file
135
DMS/docker-data/dms/config/rspamd/append.d/settings.conf
Normal file
@@ -0,0 +1,135 @@
|
||||
# DMS-CUSTOM-APPEND
|
||||
# ===========================================================================
|
||||
# WICHTIG: Diese Datei wird von user-patches.sh an /etc/rspamd/local.d/settings.conf
|
||||
# ANGEHAENGT und nicht darueber kopiert. DMS legt in dieser Datei selbst eine
|
||||
# Regel fuer authentifizierte User ab (DMS::SED_TAG::1), die erhalten bleiben muss.
|
||||
# ===========================================================================
|
||||
#
|
||||
# Hintergrund
|
||||
# -----------
|
||||
# Externe Mail erreicht DMS nie direkt: SES -> S3/SQS -> email-worker -> SMTP.
|
||||
# Rspamd sieht als Client-IP daher immer den Worker-Container (172.x) statt des
|
||||
# echten Absender-Servers. Alle verbindungsbezogenen Pruefungen messen damit
|
||||
# unsere eigene Infrastruktur und liefern systematisch Strafpunkte:
|
||||
#
|
||||
# HFILTER_HOSTNAME_UNKNOWN 6.0 (kein PTR fuer 172.x -> exakt die add_header-Schwelle)
|
||||
# R_SPF_FAIL / R_SPF_SOFTFAIL 4.5 / 2.5
|
||||
# R_DKIM_NA / R_DKIM_PERMFAIL 1.0 / 4.5
|
||||
#
|
||||
# Diese Gruppen werden fuer Einlieferungen aus dem Docker-Netz deaktiviert.
|
||||
# Inhaltsbezogene Pruefungen (Bayes, Fuzzy, URL-/Surbl-Blocklisten, MIME,
|
||||
# Attachment-Checks) bleiben vollstaendig aktiv - echter Spam wird also weiter
|
||||
# erkannt, nur eben anhand des Inhalts statt anhand einer Transportstrecke,
|
||||
# die rspamd hier gar nicht beurteilen kann.
|
||||
#
|
||||
# SPF/DKIM/DMARC gehen dadurch nicht verloren: SES prueft sie bereits mit der
|
||||
# echten Absender-IP. Das Ergebnis liegt im SQS-Payload (receipt.spamVerdict,
|
||||
# spfVerdict, dkimVerdict, dmarcVerdict). Sobald der Worker diese Verdicts als
|
||||
# Header stempelt, koennen sie hier per multimap wieder bepunktet werden.
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1) Vom Worker SELBST komponierte Mail: Weiterleitungen und OOO-Antworten
|
||||
#
|
||||
# Zusaetzlich zu den Auth-Symbolen aus Regel 2 fallen hier inhaltliche Symbole
|
||||
# an, die auf die Bauform der Weiterleitung reagieren. buildForwardMessage()
|
||||
# in email-worker-nodejs/src/email/rules-processor.ts setzt:
|
||||
#
|
||||
# From: die getroffene lokale Adresse (support@bayarea-cc.com)
|
||||
# To: das Weiterleitungsziel (ebrunks@bayarea-cc.com)
|
||||
# Reply-To: den Originalabsender (andreas.knuth@gmail.com)
|
||||
#
|
||||
# Damit sieht jede Weiterleitung an einen Kollegen in derselben Domain formal
|
||||
# aus wie CEO-Fraud: Absender und Empfaenger intern, Antwort geht nach aussen
|
||||
# an eine Freemail-Adresse. Rspamd bepunktet das - zu Recht, nur ist die
|
||||
# Praemisse hier falsch, weil wir die Mail selbst gebaut haben:
|
||||
#
|
||||
# SPOOF_REPLYTO 6.0 From-Domain == To-Domain != Reply-To-Domain
|
||||
# FREEMAIL_REPLYTO_NEQ_FROM 2.0 Reply-To ist Freemail und != From
|
||||
#
|
||||
# Belegt an F228E320085 (2026-08-15 07:53:38): Score 7.95, also ueber
|
||||
# add_header = 6 - die Weiterleitung landete im Junk, waehrend dieselbe Mail
|
||||
# als Original an support@ mit -1.05 sauber durchlief (EC702320085).
|
||||
#
|
||||
# WARUM EINE EIGENE REGEL statt symbols_disabled in Regel 2:
|
||||
# Regel 2 gilt auch fuer "delivered", also fuer echte eingehende Fremdmail.
|
||||
# Dort ist SPOOF_REPLYTO genau das Signal, das BEC-Angriffe erkennt - das
|
||||
# darf nicht verloren gehen. Diese Regel prueft deshalb den WERT des Headers
|
||||
# und greift nur fuer Mail aus unserer eigenen Komposition.
|
||||
#
|
||||
# Die Auth-Gruppen muessen wiederholt werden: rspamd wertet nach Prioritaet
|
||||
# aus und wendet nur die erste passende Regel an, Regel 2 greift hier also
|
||||
# nicht mehr.
|
||||
#
|
||||
# Falls der Regexp-Vergleich nicht zieht, faellt die Mail auf Regel 2 zurueck
|
||||
# und behaelt ihre 7.95 - der Fehler ist damit sichtbar, nicht still. Pruefen
|
||||
# laesst sich das am Feld "Regel:" in:
|
||||
# docker exec mailserver bash /tmp/docker-mailserver/rspamd_report.sh --detail <QID>
|
||||
# Erwartet wird ses_worker_forwarded.
|
||||
# ---------------------------------------------------------------------------
|
||||
ses_worker_forwarded {
|
||||
priority = 20;
|
||||
ip = ["172.16.0.0/12", "127.0.0.0/8"];
|
||||
header = {
|
||||
"X-SES-Worker-Processed" = "^(forwarded|ooo-reply)$";
|
||||
}
|
||||
apply {
|
||||
groups_disabled = ["spf", "dkim", "dmarc", "hfilter"];
|
||||
symbols_disabled = ["RDNS_NONE", "SPOOF_REPLYTO",
|
||||
"FREEMAIL_REPLYTO_NEQ_FROM", "FREEMAIL_REPLYTO"];
|
||||
}
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2) Uebrige Einlieferungen des email-workers (externe Mail, "delivered")
|
||||
#
|
||||
# Der Worker setzt X-SES-Worker-Processed auf JEDER Mail, die er einliefert:
|
||||
# delivered = Originalmail an das lokale Postfach -> diese Regel
|
||||
# forwarded = neu komponierte Weiterleitung -> Regel 1
|
||||
# ooo-reply = Abwesenheitsantwort -> Regel 1
|
||||
#
|
||||
# Hier werden ausschliesslich die Transportpruefungen abgeschaltet. Alle
|
||||
# inhaltlichen Symbole bleiben aktiv, insbesondere SPOOF_REPLYTO.
|
||||
#
|
||||
# Die IP-Bedingung ist die Absicherung: ohne sie koennte ein externer Absender
|
||||
# den Header selbst setzen und die Auth-Pruefungen abschalten. Beide
|
||||
# Bedingungen werden von rspamd implizit mit UND verknuepft.
|
||||
# ---------------------------------------------------------------------------
|
||||
ses_worker_injected {
|
||||
priority = 10;
|
||||
ip = ["172.16.0.0/12", "127.0.0.0/8"];
|
||||
header = {
|
||||
"X-SES-Worker-Processed" = true;
|
||||
}
|
||||
apply {
|
||||
groups_disabled = ["spf", "dkim", "dmarc", "hfilter"];
|
||||
symbols_disabled = ["RDNS_NONE"];
|
||||
}
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3) Lokale Re-Injection ohne Worker-Header
|
||||
#
|
||||
# Betrifft:
|
||||
# - Sieve-Redirects fuer INTERNE Mail (accounting@ -> support@ -> Forwards).
|
||||
# Diese Mail laeuft nie ueber SES/Worker, hat also keinen Worker-Header.
|
||||
# - Roundcube, das ohne SASL-Auth aus dem Docker-Netz einliefert
|
||||
# (ROUNDCUBEMAIL_SMTP_USER ist bewusst leer).
|
||||
#
|
||||
# Beides wurde bisher von der DOCKER_WL-Whitelist (-50) aufgefangen. Diese
|
||||
# Regel ersetzt sie - deutlich enger gefasst, weil sie an das Docker-Netz
|
||||
# gebunden ist statt an eine faelschbare Absenderdomain.
|
||||
#
|
||||
# Niedrigste Prioritaet: rspamd wertet Regeln nach Prioritaet aus und nimmt
|
||||
# den ersten Treffer, Worker-Mail landet also immer in Regel 1 oder 2.
|
||||
#
|
||||
# ACHTUNG: Diese Regel setzt voraus, dass Port 25 NICHT oeffentlich erreichbar
|
||||
# ist (docker-compose.yml bindet ihn auf 127.0.0.1). Wird das je geaendert,
|
||||
# muss diese Regel weg - sonst umgeht jeder externe Absender die Pruefung.
|
||||
# ---------------------------------------------------------------------------
|
||||
local_reinjection {
|
||||
priority = 5;
|
||||
ip = ["172.16.0.0/12", "127.0.0.0/8"];
|
||||
apply {
|
||||
groups_enabled = ["dkim"];
|
||||
}
|
||||
}
|
||||
42
DMS/docker-data/dms/config/rspamd/local.d/fuzzy_check.conf
Normal file
42
DMS/docker-data/dms/config/rspamd/local.d/fuzzy_check.conf
Normal file
@@ -0,0 +1,42 @@
|
||||
# documentation: https://rspamd.com/doc/modules/fuzzy_check.html
|
||||
#
|
||||
# Das Modul ist abgeschaltet, weil die oeffentliche Fuzzy-Storage von
|
||||
# rspamd.com von diesem Server aus nicht erreichbar ist. Im Log lief JEDE
|
||||
# eingehende Mail in denselben Timeout:
|
||||
#
|
||||
# fuzzy_check_timer_callback: got IO timeout with server
|
||||
# fuzzy1.rspamd.com:11335(80.241.57.6:11335), after 1/1 retransmits
|
||||
#
|
||||
# Die Server werden sauber aufgeloest, es scheitert also nicht an DNS,
|
||||
# sondern an der UDP-Verbindung auf Port 11335 (vermutlich Egress-Filter
|
||||
# des Providers).
|
||||
#
|
||||
# Kosten des Defaults: timeout = 2s bei retransmits = 1, also zwei Versuche
|
||||
# a 2 Sekunden = ~4 Sekunden Wartezeit pro Mail - genau die konstanten
|
||||
# 4,3 Sekunden Scanzeit, die im Log zu sehen waren. Ohne jeden Gegenwert,
|
||||
# da nie eine Antwort kommt.
|
||||
#
|
||||
# Zusaetzlich ein Betriebsrisiko: DMS setzt soft_reject_on_timeout = true.
|
||||
# Solange nur dieses eine Modul haengt, passiert nichts; laeuft der Task als
|
||||
# Ganzes in den Timeout, wird Mail temporaer abgelehnt statt zugestellt.
|
||||
#
|
||||
# ---------------------------------------------------------------------------
|
||||
# WIEDER EINSCHALTEN, sobald UDP 11335 ausgehend erreichbar ist:
|
||||
# Pruefen laesst sich das direkt mit rspamds eigenem Werkzeug:
|
||||
#
|
||||
# docker exec mailserver rspamadm fuzzy_ping -s rspamd.com
|
||||
#
|
||||
# Antwortet die Storage, diese Datei loeschen (dann gilt wieder der Default)
|
||||
# oder die Zeile unten auskommentieren und stattdessen die Timeouts knapp
|
||||
# halten, damit ein erneuter Ausfall nicht wieder 4 Sekunden kostet:
|
||||
#
|
||||
# timeout = 1s;
|
||||
# retransmits = 0;
|
||||
#
|
||||
# Fuzzy ist inhaltlich wertvoll - es erkennt Massen-Spam ueber Hashes, also
|
||||
# genau die Klasse, die weder von Auth-Pruefungen noch von einem frisch
|
||||
# angelernten Bayes zuverlaessig erwischt wird. Es lohnt sich, das mit dem
|
||||
# Provider zu klaeren, statt es dauerhaft abgeschaltet zu lassen.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
enabled = false;
|
||||
19
DMS/docker-data/dms/config/rspamd/local.d/logging.inc
Normal file
19
DMS/docker-data/dms/config/rspamd/local.d/logging.inc
Normal file
@@ -0,0 +1,19 @@
|
||||
# documentation: https://rspamd.com/doc/configuration/logging.html
|
||||
#
|
||||
# Uebersteuert den DMS-Default (level = "silent"), bei dem rspamd ueberhaupt
|
||||
# nichts protokolliert und Spam-Scores nicht nachvollziehbar sind.
|
||||
#
|
||||
# type = "console" bleibt bewusst stehen: Supervisor leitet stdout/stderr von
|
||||
# rspamd bereits nach /var/log/mail/rspamd.log um (siehe dms-services.conf).
|
||||
# Ein eigenes type = "file" mit demselben Pfad haette zwei Schreiber auf
|
||||
# derselben Datei zur Folge.
|
||||
#
|
||||
# Mit level = "info" schreibt rspamd pro Mail eine Zeile mit Aktion, Score und
|
||||
# allen ausgeloesten Symbolen samt Einzelgewicht:
|
||||
#
|
||||
# docker exec mailserver sh -c 'grep rspamd_task_write_log /var/log/mail/rspamd.log | tail -20'
|
||||
|
||||
type = "console";
|
||||
level = "info";
|
||||
color = false;
|
||||
systemd = false;
|
||||
@@ -1,8 +1,18 @@
|
||||
DOCKER_WL {
|
||||
type = "from";
|
||||
filter = "email:domain";
|
||||
map = "/etc/rspamd/override.d/docker_whitelist.map";
|
||||
symbol = "DOCKER_WHITELIST";
|
||||
description = "Whitelist fuer eigene Domains";
|
||||
score = -50.0;
|
||||
}
|
||||
# documentation: https://rspamd.com/doc/modules/multimap.html
|
||||
#
|
||||
# Die fruehere Regel DOCKER_WL (-50 Punkte fuer jede Mail, deren Envelope-From
|
||||
# auf eine unserer eigenen Domains zeigt) ist bewusst entfernt worden:
|
||||
#
|
||||
# 1. Sie war ein Spoofing-Loch. Mit SPOOF_PROTECTION=0 genuegte einem
|
||||
# beliebigen externen Absender ein "MAIL FROM: <x@bayarea-cc.com>", um
|
||||
# -50 Punkte plus das erzwungene "no action" aus force_actions.conf zu
|
||||
# bekommen - also garantierte Zustellung in die Inbox.
|
||||
# 2. Sie hat echten Spam an weitergeleitete Adressen durchgereicht, weil der
|
||||
# email-worker seine Weiterleitungen mit Envelope-From = Empfaengeradresse
|
||||
# (also eigene Domain) einliefert.
|
||||
# 3. Sie hat nur ein Symptom kompensiert: die falschen Auth-/Hostname-Scores,
|
||||
# die entstehen, weil alle Mail aus dem Worker-Container kommt. Das wird
|
||||
# jetzt an der Ursache geloest - siehe rspamd/append.d/settings.conf.
|
||||
#
|
||||
# Die Datei bleibt (leer) bestehen, damit user-patches.sh sie beim Start ueber
|
||||
# eine evtl. noch im Container liegende alte Version kopiert.
|
||||
|
||||
@@ -1,6 +1,10 @@
|
||||
rules {
|
||||
DOCKER_WHITELIST_FORCE {
|
||||
action = "no action";
|
||||
expression = "DOCKER_WHITELIST";
|
||||
}
|
||||
}
|
||||
# documentation: https://rspamd.com/doc/modules/force_actions.html
|
||||
#
|
||||
# Die Regel DOCKER_WHITELIST_FORCE ("no action" sobald DOCKER_WHITELIST greift)
|
||||
# ist entfernt. Sie hat die Spam-Entscheidung fuer jede Mail mit eigener
|
||||
# Absenderdomain hart ueberschrieben - inklusive echtem Spam an weitergeleitete
|
||||
# Adressen. Das dazugehoerige Symbol existiert nicht mehr, siehe
|
||||
# rspamd/local.d/multimap.conf.
|
||||
#
|
||||
# Keine rules{}-Sektion = force_actions ist wirkungslos. Datei bleibt bestehen,
|
||||
# damit DMS sie beim Start ueber eine alte Version im Container kopiert.
|
||||
|
||||
179
DMS/docker-data/dms/config/rspamd_report.sh
Executable file
179
DMS/docker-data/dms/config/rspamd_report.sh
Executable file
@@ -0,0 +1,179 @@
|
||||
#!/bin/bash
|
||||
# ===========================================================================
|
||||
# rspamd_report.sh - lesbare Auswertung von /var/log/mail/rspamd.log
|
||||
#
|
||||
# Die Logzeilen von rspamd enthalten alles, sind aber unlesbar. Dieses Skript
|
||||
# macht daraus zwei Ansichten:
|
||||
#
|
||||
# Uebersicht: eine Zeile je Mail mit Score, Aktion, Scanzeit, Von, An
|
||||
# Detail: die Symbolaufschluesselung einer Mail, nach Gewicht sortiert,
|
||||
# mit Summe - also warum genau diese Mail diesen Score hat
|
||||
#
|
||||
# AUSFUEHRUNG - immer IM Container:
|
||||
#
|
||||
# docker exec mailserver bash /tmp/docker-mailserver/rspamd_report.sh
|
||||
# docker exec mailserver bash /tmp/docker-mailserver/rspamd_report.sh -n 40
|
||||
# docker exec mailserver bash /tmp/docker-mailserver/rspamd_report.sh --detail
|
||||
# docker exec mailserver bash /tmp/docker-mailserver/rspamd_report.sh --detail C0D8A320066
|
||||
#
|
||||
# Optionen:
|
||||
# -n N Anzahl Mails in der Uebersicht (Default 15)
|
||||
# --detail [ID] Symbolaufschluesselung. ID ist ein beliebiges Fragment aus
|
||||
# Queue-ID oder Message-ID. Ohne ID die letzte Mail.
|
||||
# --slow MS Nur Mails, deren Scan laenger als MS Millisekunden dauerte
|
||||
# --log PFAD Anderes Logfile (Default /var/log/mail/rspamd.log)
|
||||
#
|
||||
# Hinweis: Das Log enthaelt nur Mails, die seit dem Umstellen auf
|
||||
# level = "info" (rspamd/local.d/logging.inc) verarbeitet wurden.
|
||||
# ===========================================================================
|
||||
|
||||
set -uo pipefail
|
||||
|
||||
LOGFILE="/var/log/mail/rspamd.log"
|
||||
COUNT=15
|
||||
DETAIL=0
|
||||
DETAIL_ID=""
|
||||
SLOW_MS=0
|
||||
|
||||
usage() { sed -n '2,30p' "$0" | sed 's/^# \{0,1\}//' ; }
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
-n) COUNT="${2:-15}"; shift 2 ;;
|
||||
--detail) DETAIL=1
|
||||
if [[ ${2:-} != "" && ${2:-} != -* ]]; then DETAIL_ID="$2"; shift 2; else shift; fi ;;
|
||||
--slow) SLOW_MS="${2:-0}"; shift 2 ;;
|
||||
--log) LOGFILE="${2:-}"; shift 2 ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
*) echo "Unbekannte Option: $1" >&2; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [[ ! -r ${LOGFILE} ]]; then
|
||||
echo "FEHLER: ${LOGFILE} nicht lesbar. Laeuft das Skript wirklich IM Container?" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
TASKS=$(grep 'rspamd_task_write_log' "${LOGFILE}" 2>/dev/null)
|
||||
if [[ -z ${TASKS} ]]; then
|
||||
echo "Keine ausgewerteten Mails im Log gefunden."
|
||||
echo "Steht in rspamd/local.d/logging.inc wirklich level = \"info\"?"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Detailansicht einer einzelnen Mail
|
||||
# ---------------------------------------------------------------------------
|
||||
if (( DETAIL )); then
|
||||
if [[ -n ${DETAIL_ID} ]]; then
|
||||
LINE=$(grep -F "${DETAIL_ID}" <<<"${TASKS}" | tail -1)
|
||||
[[ -z ${LINE} ]] && { echo "Keine Mail mit '${DETAIL_ID}' im Log gefunden." >&2; exit 1; }
|
||||
else
|
||||
LINE=$(tail -1 <<<"${TASKS}")
|
||||
fi
|
||||
|
||||
awk '
|
||||
{
|
||||
# Kopfdaten
|
||||
ts = substr($0, 1, 19)
|
||||
match($0, /qid: <[^>]*>/); qid = extract()
|
||||
match($0, /from: <[^>]*>/); from = extract()
|
||||
match($0, /rcpts: <[^>]*>/); rcpt = extract()
|
||||
match($0, /settings_id: [^,]*/); sid = substr($0, RSTART+13, RLENGTH-13)
|
||||
match($0, /time: [0-9.]+ms/); tm = substr($0, RSTART+6, RLENGTH-8)
|
||||
match($0, /dns req: [0-9]+/); dns = substr($0, RSTART+9, RLENGTH-9)
|
||||
|
||||
# Aktion und Score: (default: F (no action): [4.74/500.00]
|
||||
if (match($0, /\([a-z ]+\): \[-?[0-9.]+\//)) {
|
||||
chunk = substr($0, RSTART, RLENGTH)
|
||||
action = substr(chunk, 2, index(chunk, ")") - 2)
|
||||
score = substr(chunk, index(chunk, "[") + 1, length(chunk) - index(chunk, "[") - 1)
|
||||
}
|
||||
|
||||
printf "Zeit: %s\n", ts
|
||||
printf "Queue-ID: %s\n", qid
|
||||
printf "Von: %s\n", from
|
||||
printf "An: %s\n", rcpt
|
||||
printf "Regel: %s\n", sid
|
||||
printf "Aktion: %s Score: %s Scanzeit: %s ms DNS: %s\n\n", action, score, tm, dns
|
||||
|
||||
# Symbolblock zwischen "] [" und "]), len:"
|
||||
i = index($0, "] [")
|
||||
if (i == 0) { print " (keine Symbole im Log)"; exit }
|
||||
rest = substr($0, i + 3)
|
||||
j = index(rest, "]), len:")
|
||||
if (j == 0) j = length(rest)
|
||||
syms = substr(rest, 1, j - 1)
|
||||
|
||||
# auf Kommas der obersten Ebene splitten (Klammern koennen Kommas enthalten)
|
||||
depth = 0; cur = ""; n = 0
|
||||
for (k = 1; k <= length(syms); k++) {
|
||||
c = substr(syms, k, 1)
|
||||
if (c == "{") depth++
|
||||
else if (c == "}") depth--
|
||||
if (c == "," && depth == 0) { if (cur != "") entries[++n] = cur; cur = ""; continue }
|
||||
cur = cur c
|
||||
}
|
||||
if (cur != "") entries[++n] = cur
|
||||
|
||||
printf " %8s %-38s %s\n", "GEWICHT", "SYMBOL", "DETAIL"
|
||||
printf " %8s %-38s %s\n", "-------", "--------------------------------------", "------"
|
||||
|
||||
total = 0
|
||||
# zwei Durchlaeufe: erst die mit Gewicht, dann die neutralen
|
||||
for (pass = 1; pass <= 2; pass++) {
|
||||
for (e = 1; e <= n; e++) {
|
||||
entry = entries[e]
|
||||
p1 = index(entry, "(")
|
||||
if (p1 == 0) continue
|
||||
name = substr(entry, 1, p1 - 1)
|
||||
p2 = index(entry, ")")
|
||||
w = substr(entry, p1 + 1, p2 - p1 - 1)
|
||||
payload = ""
|
||||
b1 = index(entry, "{")
|
||||
if (b1 > 0) payload = substr(entry, b1 + 1, length(entry) - b1 - 1)
|
||||
gsub(/;$/, "", payload)
|
||||
|
||||
if (pass == 1 && w + 0 == 0) continue
|
||||
if (pass == 2 && w + 0 != 0) continue
|
||||
if (pass == 1) total += w
|
||||
|
||||
printf " %8.2f %-38s %s\n", w, name, substr(payload, 1, 60)
|
||||
}
|
||||
if (pass == 1) printf " %8s %s\n", "-------", ""
|
||||
if (pass == 1) printf " %8.2f %s\n\n neutrale Symbole (Gewicht 0):\n", total, "SUMME"
|
||||
}
|
||||
}
|
||||
function extract( s) { s = substr($0, RSTART, RLENGTH); return substr(s, index(s, "<") + 1, length(s) - index(s, "<") - 1) }
|
||||
' <<<"${LINE}"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Uebersicht
|
||||
# ---------------------------------------------------------------------------
|
||||
printf '%-19s %7s %-12s %7s %-30s %-30s\n' "ZEIT" "SCORE" "AKTION" "MS" "VON" "AN"
|
||||
printf -- '--------------------------------------------------------------------------------------------------------------------\n'
|
||||
|
||||
awk -v slow="${SLOW_MS}" '
|
||||
{
|
||||
ts = substr($0, 1, 19)
|
||||
match($0, /from: <[^>]*>/); from = extract()
|
||||
match($0, /rcpts: <[^>]*>/); rcpt = extract()
|
||||
match($0, /time: [0-9.]+ms/); tm = substr($0, RSTART+6, RLENGTH-8) + 0
|
||||
|
||||
if (match($0, /\([a-z ]+\): \[-?[0-9.]+\//)) {
|
||||
chunk = substr($0, RSTART, RLENGTH)
|
||||
action = substr(chunk, 2, index(chunk, ")") - 2)
|
||||
score = substr(chunk, index(chunk, "[") + 1, length(chunk) - index(chunk, "[") - 1) + 0
|
||||
} else { action = "?"; score = 0 }
|
||||
|
||||
if (tm < slow) next
|
||||
printf "%-19s %7.2f %-12s %7.0f %-30.30s %-30.30s\n", ts, score, action, tm, from, rcpt
|
||||
}
|
||||
function extract( s) { s = substr($0, RSTART, RLENGTH); return substr(s, index(s, "<") + 1, length(s) - index(s, "<") - 1) }
|
||||
' <<<"${TASKS}" | tail -n "${COUNT}"
|
||||
|
||||
echo
|
||||
echo "Detail zu einer Mail: $0 --detail <Queue-ID>"
|
||||
echo "Nur langsame Scans: $0 --slow 1000"
|
||||
222
DMS/docker-data/dms/config/train_bayes.sh
Executable file
222
DMS/docker-data/dms/config/train_bayes.sh
Executable file
@@ -0,0 +1,222 @@
|
||||
#!/bin/bash
|
||||
# ===========================================================================
|
||||
# train_bayes.sh - Bayes-Klassifikator von rspamd aus den vorhandenen
|
||||
# Postfaechern anlernen.
|
||||
#
|
||||
# Hintergrund: rspamd traegt mit BAYES_* erst dann zum Score bei, wenn genug
|
||||
# gelernt wurde (Default: 200 Spam UND 200 Ham). Bis dahin liefert der
|
||||
# Klassifikator gar nichts - genau das war im Log zu sehen, dort tauchte kein
|
||||
# einziges BAYES_-Symbol auf. Statt monatelang auf manuelles Verschieben in
|
||||
# den Junk-Ordner zu warten (das macht RSPAMD_LEARN=1 bereits automatisch),
|
||||
# lernt dieses Skript einmalig aus dem, was ohnehin auf der Platte liegt.
|
||||
#
|
||||
# AUSFUEHRUNG - immer IM Container:
|
||||
#
|
||||
# docker exec mailserver bash /tmp/docker-mailserver/train_bayes.sh
|
||||
# -> Trockenlauf: zaehlt nur, lernt nichts. Das ist der Default.
|
||||
#
|
||||
# docker exec mailserver bash /tmp/docker-mailserver/train_bayes.sh --apply
|
||||
# -> lernt tatsaechlich
|
||||
#
|
||||
# Optionen:
|
||||
# --apply Wirklich lernen (ohne das passiert nichts)
|
||||
# --max N Max. Nachrichten je Postfach und Klasse (Default 400)
|
||||
# --include-sent Gesendete Mails zusaetzlich als Ham lernen
|
||||
# --only MUSTER Nur Postfaecher, deren Adresse eines der Muster
|
||||
# enthaelt. Mehrfach angebbar oder kommasepariert:
|
||||
# --only support@,andreas.knuth@
|
||||
# WICHTIG: Ohne diese Option laeuft das Skript ueber
|
||||
# ALLE Postfaecher aller Domains unter /var/mail.
|
||||
# --host HOST:PORT rspamd-Controller (Default 127.0.0.1:11334)
|
||||
#
|
||||
# Passwort: Falls der Controller eines verlangt (bei euch in
|
||||
# rspamd/override.d/worker-controller.inc gesetzt), vorweg setzen:
|
||||
# docker exec -e RSPAMC_PASSWORD='...' mailserver bash /tmp/.../train_bayes.sh
|
||||
#
|
||||
# WICHTIG zur Datenqualitaet:
|
||||
# - Ham kommt aus der INBOX. Liegt dort unerkannter Spam, wird er als Ham
|
||||
# gelernt und verschlechtert die Erkennung. Vorher grob durchsehen lohnt.
|
||||
# - Spam kommt aus Junk. Liegt dort faelschlich einsortierte legitime Mail,
|
||||
# gilt dasselbe umgekehrt.
|
||||
# - Ein sehr schiefes Verhaeltnis (z.B. 8000 Ham zu 50 Spam) macht den
|
||||
# Klassifikator einseitig. Der Trockenlauf zeigt das Verhaeltnis an -
|
||||
# danach ggf. mit --max nachsteuern.
|
||||
# ===========================================================================
|
||||
|
||||
set -uo pipefail
|
||||
|
||||
MAILBASE="${MAILBASE:-/var/mail}"
|
||||
RSPAMC_HOST="127.0.0.1:11334"
|
||||
MAX_PER_CLASS=400
|
||||
MAX_BYTES=5242880 # 5 MB - groessere Mails bringen fuers Lernen nichts
|
||||
BATCH=50 # Dateien pro rspamc-Aufruf
|
||||
APPLY=0
|
||||
INCLUDE_SENT=0
|
||||
ONLY_PATTERNS=()
|
||||
|
||||
usage() { sed -n '2,40p' "$0" | sed 's/^# \{0,1\}//' ; }
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--apply) APPLY=1; shift ;;
|
||||
--max) MAX_PER_CLASS="${2:-}"; shift 2 ;;
|
||||
--include-sent) INCLUDE_SENT=1; shift ;;
|
||||
--only) IFS=',' read -ra _pat <<<"${2:-}"
|
||||
ONLY_PATTERNS+=("${_pat[@]}"); shift 2 ;;
|
||||
--host) RSPAMC_HOST="${2:-}"; shift 2 ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
*) echo "Unbekannte Option: $1" >&2; echo "Hilfe: $0 --help" >&2; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [[ ! -d ${MAILBASE} ]]; then
|
||||
echo "FEHLER: ${MAILBASE} existiert nicht. Laeuft das Skript wirklich IM Container?" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
RSPAMC=(rspamc -h "${RSPAMC_HOST}")
|
||||
[[ -n ${RSPAMC_PASSWORD:-} ]] && RSPAMC+=(-P "${RSPAMC_PASSWORD}")
|
||||
|
||||
if ! command -v rspamc >/dev/null 2>&1; then
|
||||
echo "FEHLER: rspamc nicht gefunden." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! "${RSPAMC[@]}" stat >/dev/null 2>&1; then
|
||||
echo "FEHLER: Controller unter ${RSPAMC_HOST} nicht erreichbar oder Passwort noetig." >&2
|
||||
echo " Versuche es mit: docker exec -e RSPAMC_PASSWORD='...' mailserver bash $0 ..." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Dateien einsammeln: bis LIMIT regulaere Dateien unterhalb der uebergebenen
|
||||
# Maildir-Verzeichnisse, groessenbegrenzt. Neueste zuerst waere schoener, ist
|
||||
# aber bei zehntausenden Dateien zu teuer - Maildir-Namen beginnen mit dem
|
||||
# Timestamp, daher liefert die Sortierung eine brauchbare Naeherung.
|
||||
# ---------------------------------------------------------------------------
|
||||
gather() {
|
||||
local limit="$1"; shift
|
||||
local dir
|
||||
for dir in "$@"; do
|
||||
[[ -d ${dir} ]] || continue
|
||||
find "${dir}" -maxdepth 1 -type f -size -"${MAX_BYTES}"c -print 2>/dev/null
|
||||
done | sort -r | head -n "${limit}"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Lernen in Batches. Gibt "gelernt fehlgeschlagen" zurueck.
|
||||
# ---------------------------------------------------------------------------
|
||||
learn() {
|
||||
local cmd="$1"; shift
|
||||
local -a files=("$@")
|
||||
local total=${#files[@]}
|
||||
local ok=0 fail=0 idx=0 out
|
||||
|
||||
while (( idx < total )); do
|
||||
out=$("${RSPAMC[@]}" "${cmd}" "${files[@]:idx:BATCH}" 2>&1)
|
||||
ok=$(( ok + $(grep -c 'success = true' <<<"${out}") ))
|
||||
fail=$(( fail + $(grep -ci 'error' <<<"${out}") ))
|
||||
idx=$(( idx + BATCH ))
|
||||
done
|
||||
|
||||
echo "${ok} ${fail}"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Hauptlauf
|
||||
# ---------------------------------------------------------------------------
|
||||
if (( APPLY )); then
|
||||
echo "=== Bayes-Training (SCHREIBEND) ==="
|
||||
else
|
||||
echo "=== Bayes-Training (Trockenlauf - es wird nichts gelernt) ==="
|
||||
fi
|
||||
echo "Basis: ${MAILBASE} | max ${MAX_PER_CLASS} je Postfach und Klasse"
|
||||
if (( ${#ONLY_PATTERNS[@]} > 0 )); then
|
||||
echo "Filter: nur Postfaecher mit '${ONLY_PATTERNS[*]}'"
|
||||
else
|
||||
echo "Filter: KEINER - es werden ALLE Postfaecher unter ${MAILBASE} erfasst"
|
||||
fi
|
||||
printf '\n%-42s %8s %8s\n' "POSTFACH" "SPAM" "HAM"
|
||||
printf -- '---------------------------------------------------------------\n'
|
||||
|
||||
total_spam=0
|
||||
total_ham=0
|
||||
learned_spam=0
|
||||
learned_ham=0
|
||||
failed=0
|
||||
|
||||
for domain_dir in "${MAILBASE}"/*/; do
|
||||
[[ -d ${domain_dir} ]] || continue
|
||||
domain=$(basename "${domain_dir}")
|
||||
|
||||
for user_dir in "${domain_dir}"*/; do
|
||||
user_dir="${user_dir%/}"
|
||||
[[ -d ${user_dir}/cur ]] || continue # keine Maildir -> ueberspringen
|
||||
|
||||
mailbox="$(basename "${user_dir}")@${domain}"
|
||||
|
||||
if (( ${#ONLY_PATTERNS[@]} > 0 )); then
|
||||
matched=0
|
||||
for pat in "${ONLY_PATTERNS[@]}"; do
|
||||
[[ ${mailbox} == *"${pat}"* ]] && { matched=1; break; }
|
||||
done
|
||||
(( matched )) || continue
|
||||
fi
|
||||
|
||||
mapfile -t spam_files < <(gather "${MAX_PER_CLASS}" \
|
||||
"${user_dir}/.Junk/cur" "${user_dir}/.Junk/new" \
|
||||
"${user_dir}/.Spam/cur" "${user_dir}/.Spam/new")
|
||||
|
||||
ham_dirs=("${user_dir}/cur" "${user_dir}/new")
|
||||
if (( INCLUDE_SENT )); then
|
||||
ham_dirs+=("${user_dir}/.Sent/cur" "${user_dir}/.Sent Messages/cur")
|
||||
fi
|
||||
mapfile -t ham_files < <(gather "${MAX_PER_CLASS}" "${ham_dirs[@]}")
|
||||
|
||||
n_spam=${#spam_files[@]}
|
||||
n_ham=${#ham_files[@]}
|
||||
(( n_spam == 0 && n_ham == 0 )) && continue
|
||||
|
||||
printf '%-42s %8d %8d\n' "${mailbox}" "${n_spam}" "${n_ham}"
|
||||
total_spam=$(( total_spam + n_spam ))
|
||||
total_ham=$(( total_ham + n_ham ))
|
||||
|
||||
if (( APPLY )); then
|
||||
if (( n_spam > 0 )); then
|
||||
read -r ok fail <<<"$(learn learn_spam "${spam_files[@]}")"
|
||||
learned_spam=$(( learned_spam + ok )); failed=$(( failed + fail ))
|
||||
fi
|
||||
if (( n_ham > 0 )); then
|
||||
read -r ok fail <<<"$(learn learn_ham "${ham_files[@]}")"
|
||||
learned_ham=$(( learned_ham + ok )); failed=$(( failed + fail ))
|
||||
fi
|
||||
fi
|
||||
done
|
||||
done
|
||||
|
||||
printf -- '---------------------------------------------------------------\n'
|
||||
printf '%-42s %8d %8d\n\n' "SUMME" "${total_spam}" "${total_ham}"
|
||||
|
||||
if (( total_spam > 0 )); then
|
||||
ratio=$(( total_ham / (total_spam > 0 ? total_spam : 1) ))
|
||||
if (( ratio >= 5 )); then
|
||||
echo "WARNUNG: Ham/Spam-Verhaeltnis etwa ${ratio}:1 - sehr einseitig."
|
||||
echo " Besser mit kleinerem --max fahren, z.B. --max $(( total_spam / 4 + 10 ))."
|
||||
echo
|
||||
fi
|
||||
fi
|
||||
|
||||
if (( ! APPLY )); then
|
||||
echo "Trockenlauf beendet. Zum echten Lernen dasselbe Kommando mit --apply."
|
||||
echo "rspamd braucht je Klasse mindestens 200 Nachrichten, bevor BAYES_* feuert."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "Gelernt: ${learned_spam} Spam, ${learned_ham} Ham, ${failed} Fehler/Duplikate."
|
||||
echo
|
||||
echo "=== rspamc stat ==="
|
||||
"${RSPAMC[@]}" stat 2>/dev/null | grep -iE 'learn|classifier|statfile' || true
|
||||
echo
|
||||
echo "Kontrolle an der naechsten eingehenden Mail:"
|
||||
echo " docker exec mailserver sh -c 'grep rspamd_task_write_log /var/log/mail/rspamd.log | tail -3'"
|
||||
echo "In der Symbolliste sollte jetzt BAYES_HAM oder BAYES_SPAM auftauchen."
|
||||
@@ -1,47 +1,61 @@
|
||||
#!/bin/bash
|
||||
# user-patches.sh laeuft bei jedem Start von DMS automatisch
|
||||
# user-patches.sh laeuft bei jedem Start von DMS automatisch,
|
||||
# nach dem Setup und bevor die Daemons gestartet werden.
|
||||
|
||||
ACCOUNTS_FILE="/tmp/docker-mailserver/postfix-accounts.cf"
|
||||
WHITELIST_FILE="/etc/rspamd/override.d/docker_whitelist.map"
|
||||
SRC_BASE="/tmp/docker-mailserver/rspamd"
|
||||
LOCAL_D="/etc/rspamd/local.d"
|
||||
|
||||
# --- Rspamd Whitelist generieren ---
|
||||
STATIC_DOMAINS=(
|
||||
"bayarea-cc.com"
|
||||
"ruehrgedoens.de"
|
||||
"annavillesda.org"
|
||||
"bizmatch.net"
|
||||
"biz-match.com"
|
||||
"qrmaster.net"
|
||||
"nqsltd.com"
|
||||
"iitwelders.com"
|
||||
)
|
||||
|
||||
echo "Patching: Generiere Rspamd Whitelist aus Accounts + statischen Domains..."
|
||||
|
||||
{
|
||||
for domain in "${STATIC_DOMAINS[@]}"; do
|
||||
echo "$domain"
|
||||
done
|
||||
if [ -f "$ACCOUNTS_FILE" ]; then
|
||||
awk -F'|' '{print $1}' "$ACCOUNTS_FILE" | cut -d'@' -f2
|
||||
fi
|
||||
} | sort | uniq > "$WHITELIST_FILE"
|
||||
|
||||
chmod 644 "$WHITELIST_FILE"
|
||||
chown _rspamd:_rspamd "$WHITELIST_FILE" 2>/dev/null || true
|
||||
echo "Whitelist erstellt:"
|
||||
cat "$WHITELIST_FILE"
|
||||
|
||||
# --- local.d configs manuell kopieren (DMS kopiert local.d nicht automatisch) ---
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1) local.d configs kopieren
|
||||
# DMS kopiert nur rspamd/override.d automatisch, local.d nicht.
|
||||
# ---------------------------------------------------------------------------
|
||||
echo "Patching: Kopiere custom rspamd local.d configs..."
|
||||
SRC="/tmp/docker-mailserver/rspamd/local.d"
|
||||
DST="/etc/rspamd/local.d"
|
||||
if [ -d "$SRC" ]; then
|
||||
for f in "$SRC"/*; do
|
||||
if [ -d "${SRC_BASE}/local.d" ]; then
|
||||
for f in "${SRC_BASE}/local.d"/*; do
|
||||
[ -f "$f" ] || continue
|
||||
cp "$f" "$DST/$(basename "$f")"
|
||||
chown root:root "$DST/$(basename "$f")"
|
||||
chmod 644 "$DST/$(basename "$f")"
|
||||
echo " Kopiert: $(basename "$f") -> $DST/"
|
||||
base=$(basename "$f")
|
||||
cp "$f" "${LOCAL_D}/${base}"
|
||||
chown root:root "${LOCAL_D}/${base}"
|
||||
chmod 644 "${LOCAL_D}/${base}"
|
||||
echo " Kopiert: ${base} -> ${LOCAL_D}/"
|
||||
done
|
||||
fi
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2) append.d: an bestehende local.d-Dateien ANHAENGEN statt sie zu ersetzen
|
||||
#
|
||||
# Zwingend fuer settings.conf: DMS legt dort selbst die Regel fuer
|
||||
# authentifizierte User ab (DMS::SED_TAG::1). Ein cp darueber wuerde sie
|
||||
# loeschen und RSPAMD_CHECK_AUTHENTICATED=0 wirkungslos machen.
|
||||
#
|
||||
# Der Marker DMS-CUSTOM-APPEND macht das Anhaengen idempotent - /etc/rspamd
|
||||
# liegt im Container-Dateisystem und ueberlebt einen `docker restart`.
|
||||
# ---------------------------------------------------------------------------
|
||||
echo "Patching: Haenge custom rspamd Snippets an local.d configs an..."
|
||||
if [ -d "${SRC_BASE}/append.d" ]; then
|
||||
for f in "${SRC_BASE}/append.d"/*; do
|
||||
[ -f "$f" ] || continue
|
||||
base=$(basename "$f")
|
||||
target="${LOCAL_D}/${base}"
|
||||
|
||||
if grep -q 'DMS-CUSTOM-APPEND' "$target" 2>/dev/null; then
|
||||
echo " Bereits vorhanden, uebersprungen: ${base}"
|
||||
continue
|
||||
fi
|
||||
|
||||
printf '\n' >>"$target"
|
||||
cat "$f" >>"$target"
|
||||
chown root:root "$target"
|
||||
chmod 644 "$target"
|
||||
echo " Angehaengt: ${base} -> ${target}"
|
||||
done
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# HINWEIS: Die frueher hier erzeugte Rspamd-Whitelist
|
||||
# (/etc/rspamd/override.d/docker_whitelist.map aus postfix-accounts.cf +
|
||||
# statischer Domainliste) ist entfernt. Die zugehoerige multimap-Regel
|
||||
# DOCKER_WL (-50) und das erzwungene "no action" aus force_actions.conf gibt
|
||||
# es nicht mehr - sie waren ein Spoofing-Loch und haben echten Spam an
|
||||
# weitergeleitete Adressen durchgereicht. Ersatz: rspamd/append.d/settings.conf.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -32,13 +32,27 @@ class email_config extends rcube_plugin
|
||||
function email_config_form()
|
||||
{
|
||||
$rcmail = rcube::get_instance();
|
||||
$email = $rcmail->user->get_username();
|
||||
$email = $rcmail->user->get_username();
|
||||
|
||||
$secret_key = 'SHARED_SECRET_KEY_987654321';
|
||||
$config_url = 'https://config.email-bayarea.com';
|
||||
$expires = time() + 3600;
|
||||
$data = $email . '|' . $expires;
|
||||
|
||||
// mailadmin.<domain der adresse>, z.B. support@bayarea-cc.com
|
||||
// -> https://mailadmin.bayarea-cc.com
|
||||
$at = strrpos($email, '@');
|
||||
if ($at === false) {
|
||||
// Roundcube liefert ausnahmsweise keinen vollen Namen: hier kann kein
|
||||
// sinnvoller Link gebaut werden.
|
||||
return '';
|
||||
}
|
||||
$domain = strtolower(substr($email, $at + 1));
|
||||
$config_url = 'https://mailadmin.' . $domain;
|
||||
|
||||
$expires = time() + 3600;
|
||||
$data = $email . '|' . $expires;
|
||||
$signature = hash_hmac('sha256', $data, $secret_key);
|
||||
$url = $config_url . '/?email=' . urlencode($email) . '&expires=' . $expires . '&signature=' . $signature;
|
||||
$url = $config_url . '/?email=' . urlencode($email)
|
||||
. '&expires=' . $expires
|
||||
. '&signature=' . $signature;
|
||||
|
||||
$out = '
|
||||
<div class="box" style="max-width: 600px; margin: 40px auto; padding: 30px; background: #fff; border-radius: 8px; box-shadow: 0 2px 8px rgba(0,0,0,0.1);">
|
||||
|
||||
@@ -27,7 +27,12 @@ CERTS_BASE_PATH=${CERTS_BASE_PATH:-"/etc/mail/certs"}
|
||||
|
||||
# Node-Hostname: Fallback-Cert für DMS (kein Wildcard, direktes Cert)
|
||||
# Muss mit dem 'hostname' in docker-compose.yml übereinstimmen.
|
||||
NODE_HOSTNAME=${NODE_HOSTNAME:-"node1.email-srvr.com"}
|
||||
if [ -z "$NODE_HOSTNAME" ]; then
|
||||
echo "❌ NODE_HOSTNAME ist nicht gesetzt!"
|
||||
echo "Beispiel:"
|
||||
echo " DMS_CONTAINER=mailserver NODE_HOSTNAME=node2.email-srvr.com ./setup-dms-tls.sh"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "============================================================"
|
||||
echo " 🔐 DMS TLS SNI Setup (Multi-Domain)"
|
||||
|
||||
@@ -34,7 +34,7 @@ aws s3api put-public-access-block \
|
||||
--public-access-block-configuration "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"
|
||||
|
||||
# Lebenszyklus-Konfiguration hinzufügen
|
||||
echo "Lebenszyklus-Konfiguration hinzufügen (E-Mails werden nach 90 Tagen gelöscht)..."
|
||||
echo "Lebenszyklus-Konfiguration hinzufügen (E-Mails werden nach 7 Tagen gelöscht)..."
|
||||
aws s3api put-bucket-lifecycle-configuration \
|
||||
--bucket ${S3_BUCKET_NAME} \
|
||||
--lifecycle-configuration '{
|
||||
@@ -43,7 +43,7 @@ aws s3api put-bucket-lifecycle-configuration \
|
||||
"ID": "DeleteOldEmails",
|
||||
"Status": "Enabled",
|
||||
"Expiration": {
|
||||
"Days": 14
|
||||
"Days": 7
|
||||
},
|
||||
"Filter": {
|
||||
"Prefix": ""
|
||||
@@ -76,6 +76,14 @@ aws s3api put-bucket-policy \
|
||||
]
|
||||
}'
|
||||
|
||||
# ------------------------
|
||||
# Cost Allocation Tags setzen
|
||||
# ------------------------
|
||||
echo "Setze Cost Allocation Tag (BucketName)..."
|
||||
aws s3api put-bucket-tagging \
|
||||
--bucket ${S3_BUCKET_NAME} \
|
||||
--tagging "TagSet=[{Key=BucketName,Value=${S3_BUCKET_NAME}}]"
|
||||
|
||||
echo "S3 Bucket $S3_BUCKET_NAME wurde erfolgreich erstellt und konfiguriert."
|
||||
echo "Bucket-ARN: arn:aws:s3:::$S3_BUCKET_NAME"
|
||||
|
||||
|
||||
@@ -7,9 +7,14 @@
|
||||
# Standard: mail.${DOMAIN_NAME}
|
||||
# Override: export MAIL_FROM_SUBDOMAIN="mailfrom" (nur der Prefix, ohne Domain)
|
||||
#
|
||||
# Lambda Deployment:
|
||||
# Standard: Überspringt den Deploy (SKIP_LAMBDA_DEPLOY=true)
|
||||
# Override: export SKIP_LAMBDA_DEPLOY="false" (um Lambda neu auszurollen)
|
||||
#
|
||||
# Beispiel:
|
||||
# export DOMAIN_NAME="buddelectric.net"
|
||||
# export MAIL_FROM_SUBDOMAIN="mailfrom" # → mailfrom.buddelectric.net
|
||||
# export SKIP_LAMBDA_DEPLOY="false" # → Lambda wird aktualisiert
|
||||
# ./awsses.sh
|
||||
|
||||
set -e
|
||||
@@ -18,17 +23,20 @@ set -e
|
||||
if ! command -v jq &> /dev/null; then echo "Fehler: 'jq' fehlt."; exit 1; fi
|
||||
if [ -z "$DOMAIN_NAME" ]; then echo "Fehler: DOMAIN_NAME ist nicht gesetzt."; exit 1; fi
|
||||
|
||||
# Prüfen ob Python Code da ist
|
||||
PYTHON_FILE="ses_sns_shim_global.py"
|
||||
if [ ! -f "$PYTHON_FILE" ]; then
|
||||
echo "Fehler: $PYTHON_FILE nicht gefunden!"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# --- VARIABLEN ---
|
||||
AWS_REGION=${AWS_REGION:-"us-east-2"}
|
||||
EMAIL_PREFIX=${EMAIL_PREFIX:-""}
|
||||
CONFIGURATION_SET_NAME="relay-outbound"
|
||||
SKIP_LAMBDA_DEPLOY=${SKIP_LAMBDA_DEPLOY:-"true"}
|
||||
|
||||
# Prüfen ob Python Code da ist (nur wenn auch deployt werden soll)
|
||||
PYTHON_FILE="ses_sns_shim_global.py"
|
||||
if [ "$SKIP_LAMBDA_DEPLOY" != "true" ]; then
|
||||
if [ ! -f "$PYTHON_FILE" ]; then
|
||||
echo "Fehler: $PYTHON_FILE nicht gefunden!"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
# MAIL FROM Subdomain (konfigurierbar)
|
||||
MAIL_FROM_SUBDOMAIN=${MAIL_FROM_SUBDOMAIN:-"mail"}
|
||||
@@ -49,6 +57,7 @@ LAMBDA_ROLE_NAME="SesShimGlobalRole"
|
||||
echo "=========================================================="
|
||||
echo " SES Setup (S3 -> Global Lambda Shim -> SQS) für $DOMAIN_NAME"
|
||||
echo " MAIL FROM: $MAIL_FROM_DOMAIN"
|
||||
echo " SKIP_LAMBDA: $SKIP_LAMBDA_DEPLOY"
|
||||
echo "=========================================================="
|
||||
|
||||
# ---------------------------------------------------------
|
||||
@@ -113,41 +122,43 @@ echo " -> Permissions aktualisiert."
|
||||
sleep 5
|
||||
|
||||
# ---------------------------------------------------------
|
||||
# 4. Lambda Funktion erstellen/updaten (Global!)
|
||||
# 4 & 5. Lambda Funktion + SES Permissions (optional)
|
||||
# ---------------------------------------------------------
|
||||
echo "[4/6] Global Lambda Shim deployen..."
|
||||
# Zip erstellen
|
||||
cp "$PYTHON_FILE" lambda_function.py
|
||||
zip -q lambda.zip lambda_function.py
|
||||
# Keine Env-Vars nötig, da dynamisch
|
||||
ROLE_ARN=$(aws iam get-role --role-name "$LAMBDA_ROLE_NAME" --query 'Role.Arn' --output text)
|
||||
if ! aws lambda get-function --function-name "$LAMBDA_NAME" --region "$AWS_REGION" >/dev/null 2>&1; then
|
||||
echo " -> Erstelle neue Lambda-Funktion..."
|
||||
aws lambda create-function --function-name "$LAMBDA_NAME" \
|
||||
--runtime python3.11 --handler lambda_function.lambda_handler \
|
||||
--role "$ROLE_ARN" --zip-file fileb://lambda.zip \
|
||||
--region "$AWS_REGION" >/dev/null
|
||||
if [ "$SKIP_LAMBDA_DEPLOY" = "true" ]; then
|
||||
echo "[4/6] Global Lambda Shim deployen... (ÜBERSPRUNGEN)"
|
||||
echo "[5/6] SES Permission für Lambda... (ÜBERSPRUNGEN)"
|
||||
else
|
||||
echo " -> Aktualisiere existierende Lambda-Funktion..."
|
||||
aws lambda update-function-code --function-name "$LAMBDA_NAME" --zip-file fileb://lambda.zip --region "$AWS_REGION" >/dev/null
|
||||
|
||||
# Warte kurz
|
||||
sleep 2
|
||||
|
||||
aws lambda update-function-configuration --function-name "$LAMBDA_NAME" --region "$AWS_REGION" >/dev/null
|
||||
fi
|
||||
# Aufräumen
|
||||
rm lambda.zip lambda_function.py
|
||||
echo "[4/6] Global Lambda Shim deployen..."
|
||||
# Zip erstellen
|
||||
cp "$PYTHON_FILE" lambda_function.py
|
||||
zip -q lambda.zip lambda_function.py
|
||||
# Keine Env-Vars nötig, da dynamisch
|
||||
ROLE_ARN=$(aws iam get-role --role-name "$LAMBDA_ROLE_NAME" --query 'Role.Arn' --output text)
|
||||
if ! aws lambda get-function --function-name "$LAMBDA_NAME" --region "$AWS_REGION" >/dev/null 2>&1; then
|
||||
echo " -> Erstelle neue Lambda-Funktion..."
|
||||
aws lambda create-function --function-name "$LAMBDA_NAME" \
|
||||
--runtime python3.11 --handler lambda_function.lambda_handler \
|
||||
--role "$ROLE_ARN" --zip-file fileb://lambda.zip \
|
||||
--region "$AWS_REGION" >/dev/null
|
||||
else
|
||||
echo " -> Aktualisiere existierende Lambda-Funktion..."
|
||||
aws lambda update-function-code --function-name "$LAMBDA_NAME" --zip-file fileb://lambda.zip --region "$AWS_REGION" >/dev/null
|
||||
|
||||
# Warte kurz
|
||||
sleep 2
|
||||
|
||||
aws lambda update-function-configuration --function-name "$LAMBDA_NAME" --region "$AWS_REGION" >/dev/null
|
||||
fi
|
||||
# Aufräumen
|
||||
rm lambda.zip lambda_function.py
|
||||
|
||||
# ---------------------------------------------------------
|
||||
# 5. Permission: SES darf Lambda aufrufen (Global, einmalig)
|
||||
# ---------------------------------------------------------
|
||||
echo "[5/6] SES Permission für Lambda..."
|
||||
aws lambda add-permission --function-name "$LAMBDA_NAME" \
|
||||
--statement-id "AllowSESInvoke-Global" \
|
||||
--action "lambda:InvokeFunction" \
|
||||
--principal "ses.amazonaws.com" \
|
||||
--region "$AWS_REGION" 2>/dev/null || true
|
||||
echo "[5/6] SES Permission für Lambda..."
|
||||
aws lambda add-permission --function-name "$LAMBDA_NAME" \
|
||||
--statement-id "AllowSESInvoke-Global" \
|
||||
--action "lambda:InvokeFunction" \
|
||||
--principal "ses.amazonaws.com" \
|
||||
--region "$AWS_REGION" 2>/dev/null || true
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------
|
||||
# 6. SES Rule (S3 + Global Lambda)
|
||||
|
||||
90
basic_setup/check_logins.py
Executable file
90
basic_setup/check_logins.py
Executable file
@@ -0,0 +1,90 @@
|
||||
#!/usr/bin/env python3
|
||||
import csv
|
||||
import imaplib
|
||||
import sys
|
||||
import time
|
||||
|
||||
# Konfiguration
|
||||
IMAP_SERVER = "secure.emailsrvr.com"
|
||||
IMAP_PORT = 993
|
||||
DELAY_SECONDS = 1 # Kurze Pause, um Rate-Limiting oder Fail2Ban zu vermeiden
|
||||
|
||||
def check_imap_login(email, password):
|
||||
try:
|
||||
# Verbindung zum IMAP-Server via SSL herstellen
|
||||
mail = imaplib.IMAP4_SSL(IMAP_SERVER, IMAP_PORT)
|
||||
mail.login(email, password)
|
||||
mail.logout()
|
||||
return True
|
||||
except imaplib.IMAP4.error:
|
||||
# Login fehlgeschlagen (falsches Passwort/User)
|
||||
return False
|
||||
except Exception as e:
|
||||
print(f" [!] Netzwerk- oder Serverfehler bei {email}: {e}")
|
||||
return False
|
||||
|
||||
def main(csv_filepath):
|
||||
erfolgreich = []
|
||||
fehlgeschlagen = []
|
||||
uebersprungen = []
|
||||
|
||||
try:
|
||||
with open(csv_filepath, mode='r', encoding='utf-8') as f:
|
||||
# Nutze csv.reader für sauberes Parsing der Kommas
|
||||
reader = csv.reader(f)
|
||||
for row_num, row in enumerate(reader, start=1):
|
||||
if not row:
|
||||
continue
|
||||
|
||||
email = row[0].strip()
|
||||
# Prüfen, ob ein zweites Feld (Passwort) existiert und nicht leer ist
|
||||
password = row[1].strip() if len(row) > 1 else ""
|
||||
|
||||
if not password:
|
||||
print(f"[{row_num}] Überspringe {email} (Kein Passwort)")
|
||||
uebersprungen.append(email)
|
||||
continue
|
||||
|
||||
print(f"[{row_num}] Prüfe {email}... ", end="", flush=True)
|
||||
|
||||
if check_imap_login(email, password):
|
||||
print("OK")
|
||||
erfolgreich.append(email)
|
||||
else:
|
||||
print("FEHLGESCHLAGEN")
|
||||
fehlgeschlagen.append(email)
|
||||
|
||||
# Kurze Pause einlegen, um den Mailserver nicht zu fluten
|
||||
time.sleep(DELAY_SECONDS)
|
||||
|
||||
except FileNotFoundError:
|
||||
print(f"\nFehler: Die Datei '{csv_filepath}' wurde nicht gefunden.")
|
||||
sys.exit(1)
|
||||
except Exception as e:
|
||||
print(f"\nEin unerwarteter Fehler ist aufgetreten: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
# Ausgabe der Zusammenfassung
|
||||
print("\n" + "="*40)
|
||||
print("ZUSAMMENFASSUNG DER PRÜFUNG")
|
||||
print("="*40)
|
||||
|
||||
print(f"\nErfolgreich ({len(erfolgreich)}):")
|
||||
for e in erfolgreich:
|
||||
print(f" - {e}")
|
||||
|
||||
print(f"\nFehlgeschlagen ({len(fehlgeschlagen)}):")
|
||||
for e in fehlgeschlagen:
|
||||
print(f" - {e}")
|
||||
|
||||
print(f"\nÜbersprungen (kein Passwort) ({len(uebersprungen)}):")
|
||||
for e in uebersprungen:
|
||||
print(f" - {e}")
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 2:
|
||||
print("Verwendung: ./check_logins.py <pfad_zur_datei.csv>")
|
||||
sys.exit(1)
|
||||
|
||||
csv_file = sys.argv[1]
|
||||
main(csv_file)
|
||||
@@ -5,15 +5,22 @@
|
||||
# Setzt mail/imap/smtp/pop Subdomains für domain-spezifischen Mailserver-Zugang
|
||||
#
|
||||
# MIGRATIONS-FLAGS:
|
||||
# SKIP_CLIENT_DNS=true → Abschnitt 8 (imap/smtp/pop/webmail) + 10 (SRV) überspringen
|
||||
# Nutzen: Client-Subdomains bleiben beim alten Provider
|
||||
# SKIP_DMARC=true → Abschnitt 7 (DMARC) überspringen
|
||||
# Nutzen: Bestehenden DMARC-Record nicht anfassen
|
||||
# SKIP_CLIENT_DNS=true → Abschnitt 8 (mail/imap/smtp/pop/webmail) überspringen
|
||||
# Nutzen: Client-Subdomains bleiben komplett beim alten Provider
|
||||
# SKIP_PROTOCOL_DNS=true → nur imap/smtp/pop überspringen, aber mail + webmail setzen
|
||||
# Nutzen: Webmail/Autodiscover vorbereiten, Mailclients bleiben beim alten Provider
|
||||
# SKIP_DMARC=true → Abschnitt 7 (DMARC) überspringen
|
||||
# Nutzen: Bestehenden DMARC-Record nicht anfassen
|
||||
#
|
||||
# Typischer Migrations-Ablauf:
|
||||
# Phase 0 (Vorbereitung): SKIP_CLIENT_DNS=true SKIP_DMARC=true → nur SES + SPF
|
||||
# Phase 1 (MX Cutover): MX umstellen (manuell)
|
||||
# Phase 2 (Client Switch): ohne SKIP Flags → alle Records setzen
|
||||
# Phase 0a (Vorbereitung, Client-Records bleiben alt):
|
||||
# SKIP_CLIENT_DNS=true SKIP_DMARC=true → nur SES + SPF/DKIM/MailFrom
|
||||
# Phase 0b (Webmail vorbereiten, imap/smtp/pop bleiben alt):
|
||||
# SKIP_CLIENT_DNS=false SKIP_PROTOCOL_DNS=true SKIP_DMARC=false
|
||||
# Phase 1 (MX Cutover):
|
||||
# MX manuell umstellen
|
||||
# Phase 2 (Client Switch):
|
||||
# SKIP_CLIENT_DNS=false SKIP_PROTOCOL_DNS=false → alle Client-Records setzen
|
||||
|
||||
set -e
|
||||
|
||||
@@ -23,6 +30,7 @@ DRY_RUN=${DRY_RUN:-"false"}
|
||||
|
||||
# Migrations-Flags (NEU)
|
||||
SKIP_CLIENT_DNS=${SKIP_CLIENT_DNS:-"false"}
|
||||
SKIP_PROTOCOL_DNS=${SKIP_PROTOCOL_DNS:-"false"}
|
||||
SKIP_DMARC=${SKIP_DMARC:-"false"}
|
||||
|
||||
# IP des Mailservers - PFLICHT wenn keine CNAME-Kette gewünscht
|
||||
@@ -50,7 +58,8 @@ echo " 🌍 Region: $AWS_REGION"
|
||||
echo " 📬 Mail-Server Target: $TARGET_MAIL_SERVER"
|
||||
[ -n "$MAIL_SERVER_IP" ] && echo " 🖥️ Server IP: $MAIL_SERVER_IP"
|
||||
[ "$DRY_RUN" = "true" ] && echo " ⚠️ DRY RUN MODE - Keine Änderungen!"
|
||||
[ "$SKIP_CLIENT_DNS" = "true" ] && echo " ⏭️ SKIP: Client-Subdomains (imap/smtp/pop/webmail/SRV)"
|
||||
[ "$SKIP_CLIENT_DNS" = "true" ] && echo " ⏭️ SKIP: alle Client-Subdomains (mail/imap/smtp/pop/webmail)"
|
||||
[ "$SKIP_PROTOCOL_DNS" = "true" ] && echo " ⏭️ SKIP: Protokoll-Subdomains imap/smtp/pop bleiben unverändert"
|
||||
[ "$SKIP_DMARC" = "true" ] && echo " ⏭️ SKIP: DMARC Record"
|
||||
echo "============================================================"
|
||||
|
||||
@@ -255,14 +264,23 @@ else
|
||||
fi
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# SCHRITT 6: Root Domain MX (nur Info, wird nicht geändert)
|
||||
# SCHRITT 6: Root Domain MX
|
||||
# ------------------------------------------------------------------
|
||||
echo ""
|
||||
echo "--- 6. Root Domain MX (nur Info, wird nicht geändert) ---"
|
||||
CURRENT_MX=$(curl -s -X GET "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records?type=MX&name=$DOMAIN_NAME" \
|
||||
echo "--- 6. Root Domain MX ---"
|
||||
# Prüfen, ob bereits MX-Records für die Root-Domain existieren
|
||||
MX_COUNT=$(curl -s -X GET "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records?type=MX&name=$DOMAIN_NAME" \
|
||||
-H "Authorization: Bearer $CF_API_TOKEN" -H "Content-Type: application/json" \
|
||||
| jq -r '.result[0].content // "keiner"')
|
||||
echo " ℹ️ MX vorhanden: $CURRENT_MX (wird nicht geändert)"
|
||||
| jq -r '.result | length')
|
||||
|
||||
if [ "$MX_COUNT" -eq 0 ]; then
|
||||
echo " ℹ️ Kein MX-Record vorhanden. Setze initialen SES Inbound MX..."
|
||||
ensure_record "MX" "$DOMAIN_NAME" "inbound-smtp.${AWS_REGION}.amazonaws.com" false 10
|
||||
else
|
||||
echo " ⛔ MX-Record(s) für @ bereits vorhanden ($MX_COUNT Eintrag/Einträge)."
|
||||
echo " → Wird zum Schutz der alten Postfächer AUF KEINEN FALL automatisch geändert!"
|
||||
echo " → Muss nach der Migration manuell umgestellt werden."
|
||||
fi
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# SCHRITT 7: DMARC
|
||||
@@ -283,11 +301,12 @@ echo ""
|
||||
echo "--- 8. Mailclient Subdomains (A + CNAME) ---"
|
||||
if [ "$SKIP_CLIENT_DNS" = "true" ]; then
|
||||
echo " ⏭️ Übersprungen (SKIP_CLIENT_DNS=true)"
|
||||
echo " ℹ️ imap/smtp/pop/webmail bleiben beim alten Provider."
|
||||
echo " ℹ️ mail/imap/smtp/pop/webmail bleiben beim alten Provider."
|
||||
echo " ℹ️ Setze SKIP_CLIENT_DNS=false nach MX-Cutover + Client-Umstellung."
|
||||
else
|
||||
if [ -n "$MAIL_SERVER_IP" ]; then
|
||||
# A-Record für mail.<domain> direkt auf Server-IP
|
||||
# Wichtig: mail muss DNS-only bleiben; Cloudflare Proxy funktioniert nicht für SMTP/IMAP/POP.
|
||||
ensure_record "A" "mail.$DOMAIN_NAME" "$MAIL_SERVER_IP" false
|
||||
else
|
||||
# CNAME auf externen Ziel-Host (nur wenn verschieden)
|
||||
@@ -296,11 +315,19 @@ else
|
||||
fi
|
||||
fi
|
||||
|
||||
# imap, smtp, pop, webmail → CNAME auf mail.<domain>
|
||||
ensure_record "CNAME" "imap.$DOMAIN_NAME" "mail.$DOMAIN_NAME" false
|
||||
ensure_record "CNAME" "smtp.$DOMAIN_NAME" "mail.$DOMAIN_NAME" false
|
||||
ensure_record "CNAME" "pop.$DOMAIN_NAME" "mail.$DOMAIN_NAME" false
|
||||
# Webmail kann bereits vorbereitet werden, auch wenn imap/smtp/pop noch beim alten Provider bleiben.
|
||||
ensure_record "CNAME" "webmail.$DOMAIN_NAME" "mail.$DOMAIN_NAME" false
|
||||
|
||||
if [ "$SKIP_PROTOCOL_DNS" = "true" ]; then
|
||||
echo " ⏭️ Überspringe imap/smtp/pop (SKIP_PROTOCOL_DNS=true)"
|
||||
echo " ℹ️ imap/smtp/pop bleiben unverändert beim alten Provider."
|
||||
else
|
||||
# imap/smtp/pop → CNAME auf mail.<domain>
|
||||
# Wichtig: diese Records müssen DNS-only bleiben; Cloudflare Proxy funktioniert nicht für Mail-Protokolle.
|
||||
ensure_record "CNAME" "imap.$DOMAIN_NAME" "mail.$DOMAIN_NAME" false
|
||||
ensure_record "CNAME" "smtp.$DOMAIN_NAME" "mail.$DOMAIN_NAME" false
|
||||
ensure_record "CNAME" "pop.$DOMAIN_NAME" "mail.$DOMAIN_NAME" false
|
||||
fi
|
||||
fi
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
@@ -318,7 +345,12 @@ if [ "$SKIP_CLIENT_DNS" = "true" ]; then
|
||||
echo ""
|
||||
echo " ⚠️ Client-Subdomains wurden NICHT geändert."
|
||||
echo " Nach MX-Cutover + Worker-Validierung erneut ausführen mit:"
|
||||
echo " SKIP_CLIENT_DNS=false SKIP_DMARC=false ./cloudflareMigrationDns.sh"
|
||||
echo " SKIP_CLIENT_DNS=false SKIP_PROTOCOL_DNS=false SKIP_DMARC=false ./cloudflareMigrationDns.sh"
|
||||
elif [ "$SKIP_PROTOCOL_DNS" = "true" ]; then
|
||||
echo ""
|
||||
echo " ⚠️ imap/smtp/pop wurden NICHT geändert."
|
||||
echo " Für den finalen Client-Cutover erneut ausführen mit:"
|
||||
echo " SKIP_CLIENT_DNS=false SKIP_PROTOCOL_DNS=false SKIP_DMARC=false ./cloudflareMigrationDns.sh"
|
||||
fi
|
||||
echo ""
|
||||
echo " Mailclient-Konfiguration für Kunden:"
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
#!/bin/bash
|
||||
# create-queue.sh (v2 — mit SNS Fan-Out + Standby Queue)
|
||||
# Usage: DOMAIN=andreasknuth.de ./create-queue.sh
|
||||
# Usage: DOMAIN_NAME=andreasknuth.de ./create-queue.sh
|
||||
#
|
||||
# Erstellt pro Domain:
|
||||
# - Primary Queue + DLQ (wie bisher, für Contabo)
|
||||
@@ -12,13 +12,13 @@ set -e
|
||||
|
||||
AWS_REGION="us-east-2"
|
||||
|
||||
if [ -z "$DOMAIN" ]; then
|
||||
echo "Error: DOMAIN environment variable not set"
|
||||
echo "Usage: DOMAIN=andreasknuth.de $0"
|
||||
if [ -z "$DOMAIN_NAME" ]; then
|
||||
echo "Error: DOMAIN_NAME environment variable not set"
|
||||
echo "Usage: DOMAIN_NAME=andreasknuth.de $0"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
DOMAIN_SLUG="${DOMAIN//./-}"
|
||||
DOMAIN_SLUG="${DOMAIN_NAME//./-}"
|
||||
QUEUE_NAME="${DOMAIN_SLUG}-queue"
|
||||
DLQ_NAME="${QUEUE_NAME}-dlq"
|
||||
STANDBY_QUEUE_NAME="${DOMAIN_SLUG}-standby-queue"
|
||||
@@ -31,9 +31,9 @@ echo "========================================"
|
||||
echo "Creating SQS + SNS for Email Delivery"
|
||||
echo "========================================"
|
||||
echo ""
|
||||
echo "📧 Domain: $DOMAIN"
|
||||
echo " Region: $AWS_REGION"
|
||||
echo " Account: $ACCOUNT_ID"
|
||||
echo "📧 Domain: $DOMAIN_NAME"
|
||||
echo " Region: $AWS_REGION"
|
||||
echo " Account: $ACCOUNT_ID"
|
||||
echo ""
|
||||
|
||||
# ============================================================
|
||||
@@ -184,7 +184,7 @@ echo ""
|
||||
# Zusammenfassung
|
||||
# ============================================================
|
||||
echo "========================================"
|
||||
echo "✅ Setup complete for $DOMAIN"
|
||||
echo "✅ Setup complete for $DOMAIN_NAME"
|
||||
echo "========================================"
|
||||
echo ""
|
||||
echo "Primary (Contabo):"
|
||||
|
||||
32
basic_setup/legacy/s3-retention.sh
Executable file
32
basic_setup/legacy/s3-retention.sh
Executable file
@@ -0,0 +1,32 @@
|
||||
#!/bin/bash
|
||||
|
||||
echo "Aktualisiere Lifecycle-Regeln (7 Tage) für alle E-Mail-Buckets..."
|
||||
|
||||
for BUCKET in $(aws s3api list-buckets --query 'Buckets[].Name' --output text); do
|
||||
|
||||
# Prüfen, ob der Name auf '-emails' endet
|
||||
if [[ "$BUCKET" == *-emails ]]; then
|
||||
echo "⚙️ Setze 7-Tage-Regel für: $BUCKET"
|
||||
|
||||
aws s3api put-bucket-lifecycle-configuration \
|
||||
--bucket "$BUCKET" \
|
||||
--lifecycle-configuration '{
|
||||
"Rules": [
|
||||
{
|
||||
"ID": "DeleteOldEmails",
|
||||
"Status": "Enabled",
|
||||
"Expiration": {
|
||||
"Days": 7
|
||||
},
|
||||
"Filter": {
|
||||
"Prefix": ""
|
||||
}
|
||||
}
|
||||
]
|
||||
}'
|
||||
else
|
||||
echo "⏭️ Überspringe (kein E-Mail-Bucket): $BUCKET"
|
||||
fi
|
||||
done
|
||||
|
||||
echo "Fertig! Alle E-Mail-Buckets löschen jetzt Objekte nach 7 Tagen."
|
||||
31
basic_setup/legacy/sync-s3-tags.sh
Executable file
31
basic_setup/legacy/sync-s3-tags.sh
Executable file
@@ -0,0 +1,31 @@
|
||||
#!/bin/bash
|
||||
# sync-s3-tags.sh - Synchronisiert Bucket Tags welche fuer die Abrechung verwendet werden
|
||||
echo "Passe Bucket-Tags an..."
|
||||
|
||||
for BUCKET in $(aws s3api list-buckets --query 'Buckets[].Name' --output text); do
|
||||
# Aktuellen Tag abfragen (Fehler unterdrücken)
|
||||
TAG=$(aws s3api get-bucket-tagging --bucket "$BUCKET" --query 'TagSet[?Key==`BucketName`].Value' --output text 2>/dev/null)
|
||||
|
||||
# Prüfen, ob der Name auf '-emails' endet
|
||||
if [[ "$BUCKET" == *-emails ]]; then
|
||||
# Soll getaggt sein
|
||||
if [ -z "$TAG" ] || [ "$TAG" == "None" ]; then
|
||||
echo "➕ Setze fehlendes Tag für: $BUCKET"
|
||||
aws s3api put-bucket-tagging \
|
||||
--bucket "$BUCKET" \
|
||||
--tagging "TagSet=[{Key=BucketName,Value=$BUCKET}]"
|
||||
else
|
||||
echo "✅ OK (bereits getaggt): $BUCKET"
|
||||
fi
|
||||
else
|
||||
# Soll NICHT getaggt sein
|
||||
if [ -n "$TAG" ] && [ "$TAG" != "None" ]; then
|
||||
echo "🗑️ Entferne Tag von: $BUCKET"
|
||||
aws s3api delete-bucket-tagging --bucket "$BUCKET"
|
||||
else
|
||||
echo "✅ OK (ohne Tag): $BUCKET"
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
echo "Fertig! Alle '-emails' Buckets sind getaggt, bei allen anderen wurden die Tags entfernt."
|
||||
158
basic_setup/mailadminDns.sh
Executable file
158
basic_setup/mailadminDns.sh
Executable file
@@ -0,0 +1,158 @@
|
||||
#!/bin/bash
|
||||
# mailadminDns.sh
|
||||
# ------------------------------------------------------------------
|
||||
# Setzt AUSSCHLIESSLICH den DNS-Record fuer den mailadmin-Zugang.
|
||||
# Faesst NICHTS anderes an (kein SES, SPF, DKIM, MX, keine anderen
|
||||
# Subdomains). Kann gefahrlos auf bereits migrierten Domains laufen.
|
||||
#
|
||||
# Verwendet dieselbe ensure_record-Logik wie cloudflareMigrationDns.sh,
|
||||
# damit das Verhalten (Create/Update/Skip-bei-identisch) identisch ist.
|
||||
#
|
||||
# ------------------------------------------------------------------
|
||||
# PFLICHT-VARIABLEN:
|
||||
# DOMAIN_NAME Kundendomain, z.B. innungsapp.com
|
||||
# CF_API_TOKEN Cloudflare API Token
|
||||
#
|
||||
# OPTIONALE VARIABLEN:
|
||||
# MAILADMIN_HOST Subdomain-Label, Default: "mailadmin"
|
||||
# -> ergibt mailadmin.<DOMAIN_NAME>
|
||||
# MAILADMIN_TARGET CNAME-Ziel, Default: "mail.<DOMAIN_NAME>"
|
||||
# Beispiele:
|
||||
# mail.innungsapp.com (pro-Domain Modell)
|
||||
# mailadmin.bayarea-cc.com (zentrales Modell)
|
||||
# DRY_RUN "true" -> zeigt nur an, aendert nichts
|
||||
#
|
||||
# ------------------------------------------------------------------
|
||||
# BEISPIELE:
|
||||
#
|
||||
# # Standard: mailadmin.innungsapp.com -> mail.innungsapp.com
|
||||
# DOMAIN_NAME=innungsapp.com CF_API_TOKEN=xxx ./mailadminDns.sh
|
||||
#
|
||||
# # Zentrales Modell: mailadmin.innungsapp.com -> mailadmin.bayarea-cc.com
|
||||
# DOMAIN_NAME=innungsapp.com \
|
||||
# MAILADMIN_TARGET=mailadmin.bayarea-cc.com \
|
||||
# CF_API_TOKEN=xxx ./mailadminDns.sh
|
||||
#
|
||||
# # Erst testen ohne zu aendern
|
||||
# DOMAIN_NAME=innungsapp.com CF_API_TOKEN=xxx DRY_RUN=true ./mailadminDns.sh
|
||||
#
|
||||
# ------------------------------------------------------------------
|
||||
set -e
|
||||
|
||||
# --- KONFIGURATION ---
|
||||
DRY_RUN=${DRY_RUN:-"false"}
|
||||
MAILADMIN_HOST=${MAILADMIN_HOST:-"mailadmin"}
|
||||
MAILADMIN_TARGET=${MAILADMIN_TARGET:-"mail.${DOMAIN_NAME}"}
|
||||
|
||||
# --- CHECKS ---
|
||||
if [ -z "$DOMAIN_NAME" ]; then echo "❌ Fehler: DOMAIN_NAME fehlt."; exit 1; fi
|
||||
if [ -z "$CF_API_TOKEN" ]; then echo "❌ Fehler: CF_API_TOKEN fehlt."; exit 1; fi
|
||||
if ! command -v jq &> /dev/null; then echo "❌ Fehler: 'jq' fehlt."; exit 1; fi
|
||||
if ! command -v curl &> /dev/null; then echo "❌ Fehler: 'curl' fehlt."; exit 1; fi
|
||||
|
||||
RECORD_NAME="${MAILADMIN_HOST}.${DOMAIN_NAME}"
|
||||
|
||||
echo "============================================================"
|
||||
echo " 🔧 mailadmin DNS Setup"
|
||||
echo " 🌐 Domain: $DOMAIN_NAME"
|
||||
echo " 📍 Record: CNAME $RECORD_NAME → $MAILADMIN_TARGET"
|
||||
[ "$DRY_RUN" = "true" ] && echo " ⚠️ DRY RUN MODE - Keine Änderungen!"
|
||||
echo "============================================================"
|
||||
|
||||
# Schutz: CNAME auf sich selbst macht keinen Sinn
|
||||
if [ "$RECORD_NAME" == "$MAILADMIN_TARGET" ]; then
|
||||
echo "❌ Fehler: CNAME-Ziel ist identisch mit dem Record-Namen."
|
||||
echo " $RECORD_NAME kann nicht auf sich selbst zeigen."
|
||||
echo " Setze MAILADMIN_TARGET auf einen anderen Host."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# 1. ZONE ID HOLEN
|
||||
echo "🔍 Suche Cloudflare Zone ID..."
|
||||
ZONE_ID=$(curl -s -X GET "https://api.cloudflare.com/client/v4/zones?name=$DOMAIN_NAME" \
|
||||
-H "Authorization: Bearer $CF_API_TOKEN" -H "Content-Type: application/json" | jq -r '.result[0].id')
|
||||
|
||||
if [ "$ZONE_ID" == "null" ] || [ -z "$ZONE_ID" ]; then
|
||||
echo "❌ Zone nicht gefunden für $DOMAIN_NAME."
|
||||
exit 1
|
||||
fi
|
||||
echo " ✅ Zone ID: $ZONE_ID"
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# FUNKTION: ensure_record
|
||||
# Identisch zur Logik in cloudflareMigrationDns.sh, reduziert auf
|
||||
# die Typen die wir hier brauchen (CNAME).
|
||||
# ------------------------------------------------------------------
|
||||
ensure_record() {
|
||||
local type=$1
|
||||
local name=$2
|
||||
local content=$3
|
||||
local proxied=${4:-false}
|
||||
|
||||
echo " ⚙️ Prüfe $type $name..."
|
||||
|
||||
local search_res=$(curl -s -X GET "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records?type=$type&name=$name" \
|
||||
-H "Authorization: Bearer $CF_API_TOKEN" -H "Content-Type: application/json")
|
||||
|
||||
local rec_id=$(echo "$search_res" | jq -r '.result[0].id')
|
||||
local rec_content=$(echo "$search_res" | jq -r '.result[0].content')
|
||||
|
||||
[ -z "$rec_id" ] && rec_id="null"
|
||||
[ -z "$rec_content" ] && rec_content="null"
|
||||
|
||||
local json_data=$(jq -n --arg t "$type" --arg n "$name" --arg c "$content" --argjson p "$proxied" \
|
||||
'{type: $t, name: $n, content: $c, ttl: 3600, proxied: $p}')
|
||||
|
||||
if [ "$rec_id" == "null" ]; then
|
||||
if [ "$DRY_RUN" = "true" ]; then
|
||||
echo " [DRY] Würde ERSTELLEN: $type $name → $content"
|
||||
else
|
||||
local res=$(curl -s -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records" \
|
||||
-H "Authorization: Bearer $CF_API_TOKEN" -H "Content-Type: application/json" --data "$json_data")
|
||||
if [ "$(echo $res | jq -r .success)" == "true" ]; then
|
||||
echo " ✅ Erstellt: $type $name → $content"
|
||||
else
|
||||
echo " ❌ Fehler beim Erstellen: $(echo $res | jq -r '.errors[0].message')"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
else
|
||||
if [ "$rec_content" == "$content" ]; then
|
||||
echo " 🆗 Identisch ($rec_content). Überspringe."
|
||||
else
|
||||
if [ "$DRY_RUN" = "true" ]; then
|
||||
echo " [DRY] Würde UPDATEN: '$rec_content' → '$content'"
|
||||
else
|
||||
local res=$(curl -s -X PUT "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records/$rec_id" \
|
||||
-H "Authorization: Bearer $CF_API_TOKEN" -H "Content-Type: application/json" --data "$json_data")
|
||||
if [ "$(echo $res | jq -r .success)" == "true" ]; then
|
||||
echo " ✅ Aktualisiert: '$rec_content' → '$content'"
|
||||
else
|
||||
echo " ❌ Fehler beim Updaten: $(echo $res | jq -r '.errors[0].message')"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
}
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Der eigentliche Record
|
||||
# proxied=false ist hier wichtig: der mailadmin laeuft hinter Caddy
|
||||
# mit eigenem TLS-Zertifikat. Cloudflare-Proxy davor wuerde die
|
||||
# Zertifikatskette stoeren bzw. doppeltes TLS-Terminieren.
|
||||
# ------------------------------------------------------------------
|
||||
echo ""
|
||||
echo "--- mailadmin CNAME ---"
|
||||
ensure_record "CNAME" "$RECORD_NAME" "$MAILADMIN_TARGET" false
|
||||
|
||||
echo ""
|
||||
echo "============================================================"
|
||||
echo "✅ Fertig."
|
||||
echo ""
|
||||
echo " mailadmin erreichbar unter: https://$RECORD_NAME"
|
||||
echo ""
|
||||
echo " ⚠️ Wichtig: Caddy muss ein Zertifikat für $RECORD_NAME"
|
||||
echo " ausstellen. Stelle sicher, dass der Hostname in der"
|
||||
echo " Caddy-Konfiguration enthalten ist."
|
||||
echo "============================================================"
|
||||
@@ -8,24 +8,38 @@ from botocore.exceptions import ClientError
|
||||
import time
|
||||
import random
|
||||
|
||||
# Logging konfigurieren
|
||||
logger = logging.getLogger()
|
||||
logger.setLevel(logging.INFO)
|
||||
|
||||
sqs = boto3.client('sqs')
|
||||
sns = boto3.client('sns')
|
||||
sts_account_id = None
|
||||
|
||||
# Retry-Konfiguration
|
||||
MAX_RETRIES = 3
|
||||
BASE_BACKOFF = 1 # Sekunden
|
||||
BASE_BACKOFF = 1
|
||||
|
||||
def exponential_backoff(attempt):
|
||||
"""Exponential Backoff mit Jitter"""
|
||||
return BASE_BACKOFF * (2 ** attempt) + random.uniform(0, 1)
|
||||
|
||||
def get_account_id():
|
||||
global sts_account_id
|
||||
if sts_account_id is None:
|
||||
sts_account_id = boto3.client('sts').get_caller_identity()['Account']
|
||||
return sts_account_id
|
||||
|
||||
def get_topic_arn(domain):
|
||||
"""
|
||||
Generiert Topic-ARN aus Domain.
|
||||
Konvention: domain.tld -> domain-tld-topic
|
||||
"""
|
||||
topic_name = domain.replace('.', '-') + '-topic'
|
||||
region = os.environ.get('AWS_REGION', 'us-east-2')
|
||||
account_id = get_account_id()
|
||||
return f"arn:aws:sns:{region}:{account_id}:{topic_name}"
|
||||
|
||||
def get_queue_url(domain):
|
||||
"""
|
||||
Generiert Queue-Namen aus Domain und holt URL.
|
||||
Konvention: domain.tld -> domain-tld-queue
|
||||
Fallback: Direkter SQS-Send für Domains ohne SNS-Topic.
|
||||
"""
|
||||
queue_name = domain.replace('.', '-') + '-queue'
|
||||
try:
|
||||
@@ -38,54 +52,91 @@ def get_queue_url(domain):
|
||||
else:
|
||||
raise
|
||||
|
||||
def publish_to_sns(topic_arn, message_body, msg_id):
|
||||
attempt = 0
|
||||
while attempt < MAX_RETRIES:
|
||||
try:
|
||||
sns.publish(
|
||||
TopicArn=topic_arn,
|
||||
Message=message_body
|
||||
)
|
||||
logger.info(f"✅ Published {msg_id} to SNS: {topic_arn}")
|
||||
return True
|
||||
except ClientError as e:
|
||||
error_code = e.response['Error']['Code']
|
||||
# Fallback auf SQS bei Topic-nicht-gefunden ODER fehlender Berechtigung
|
||||
if error_code in ('NotFound', 'NotFoundException', 'AuthorizationError'):
|
||||
logger.info(f"ℹ️ SNS unavailable for {topic_arn} ({error_code}) — falling back to SQS")
|
||||
return False
|
||||
attempt += 1
|
||||
logger.warning(f"Retry {attempt}/{MAX_RETRIES} SNS: {error_code}")
|
||||
if attempt == MAX_RETRIES:
|
||||
raise
|
||||
time.sleep(exponential_backoff(attempt))
|
||||
return False
|
||||
|
||||
def send_to_sqs(queue_url, message_body, msg_id):
|
||||
"""Fallback: Direkter SQS-Send (wie bisher)."""
|
||||
attempt = 0
|
||||
while attempt < MAX_RETRIES:
|
||||
try:
|
||||
sqs.send_message(
|
||||
QueueUrl=queue_url,
|
||||
MessageBody=message_body
|
||||
)
|
||||
logger.info(f"✅ Sent {msg_id} to SQS: {queue_url}")
|
||||
return
|
||||
except ClientError as e:
|
||||
attempt += 1
|
||||
error_code = e.response['Error']['Code']
|
||||
logger.warning(f"Retry {attempt}/{MAX_RETRIES} SQS: {error_code}")
|
||||
if attempt == MAX_RETRIES:
|
||||
raise
|
||||
time.sleep(exponential_backoff(attempt))
|
||||
|
||||
def lambda_handler(event, context):
|
||||
"""
|
||||
Nimmt SES Event entgegen, extrahiert Domain dynamisch,
|
||||
verpackt Metadaten als 'Fake SNS' und sendet an die domain-spezifische SQS.
|
||||
Mit integrierter Retry-Logik für SQS-Send.
|
||||
Nimmt SES Event entgegen, extrahiert Domain dynamisch.
|
||||
Strategie: SNS Publish (Fan-Out an Primary + Standby Queue).
|
||||
Fallback: Direkter SQS-Send falls kein SNS-Topic existiert.
|
||||
"""
|
||||
try:
|
||||
records = event.get('Records', [])
|
||||
logger.info(f"Received event with {len(records)} records.")
|
||||
|
||||
|
||||
for record in records:
|
||||
ses_data = record.get('ses', {})
|
||||
if not ses_data:
|
||||
logger.warning(f"Invalid SES event: Missing 'ses' in record: {record}")
|
||||
logger.warning(f"Invalid SES event: Missing 'ses' in record")
|
||||
continue
|
||||
|
||||
|
||||
mail = ses_data.get('mail', {})
|
||||
receipt = ses_data.get('receipt', {})
|
||||
|
||||
# Domain extrahieren (aus erstem Recipient)
|
||||
|
||||
recipients = receipt.get('recipients', []) or mail.get('destination', [])
|
||||
if not recipients:
|
||||
logger.warning("No recipients in event - skipping")
|
||||
continue
|
||||
|
||||
|
||||
first_recipient = recipients[0]
|
||||
domain = first_recipient.split('@')[-1].lower()
|
||||
if not domain:
|
||||
logger.error("Could not extract domain from recipient")
|
||||
continue
|
||||
|
||||
# Wichtige Metadaten loggen
|
||||
|
||||
msg_id = mail.get('messageId', 'unknown')
|
||||
source = mail.get('source', 'unknown')
|
||||
logger.info(f"Processing Message-ID: {msg_id} for domain: {domain}")
|
||||
logger.info(f" From: {source}")
|
||||
logger.info(f" To: {recipients}")
|
||||
|
||||
# SES JSON als String serialisieren
|
||||
logger.info(f" From: {source}")
|
||||
logger.info(f" To: {recipients}")
|
||||
|
||||
ses_json_string = json.dumps(ses_data)
|
||||
|
||||
# Payload Größe loggen und checken (Safeguard)
|
||||
|
||||
payload_size = len(ses_json_string.encode('utf-8'))
|
||||
logger.info(f" Metadata Payload Size: {payload_size} bytes")
|
||||
if payload_size > 200000: # Arbitrary Limit < SQS 256KB
|
||||
raise ValueError("Payload too large for SQS")
|
||||
|
||||
# Fake SNS Payload
|
||||
logger.info(f" Metadata Payload Size: {payload_size} bytes")
|
||||
if payload_size > 200000:
|
||||
raise ValueError("Payload too large")
|
||||
|
||||
fake_sns_payload = {
|
||||
"Type": "Notification",
|
||||
"MessageId": str(uuid.uuid4()),
|
||||
@@ -94,30 +145,20 @@ def lambda_handler(event, context):
|
||||
"Message": ses_json_string,
|
||||
"Timestamp": datetime.utcnow().isoformat() + "Z"
|
||||
}
|
||||
|
||||
# Queue URL dynamisch holen
|
||||
queue_url = get_queue_url(domain)
|
||||
|
||||
# SQS Send mit Retries
|
||||
attempt = 0
|
||||
while attempt < MAX_RETRIES:
|
||||
try:
|
||||
sqs.send_message(
|
||||
QueueUrl=queue_url,
|
||||
MessageBody=json.dumps(fake_sns_payload)
|
||||
)
|
||||
logger.info(f"✅ Successfully forwarded {msg_id} to SQS: {queue_url}")
|
||||
break
|
||||
except ClientError as e:
|
||||
attempt += 1
|
||||
error_code = e.response['Error']['Code']
|
||||
logger.warning(f"Retry {attempt}/{MAX_RETRIES} for SQS send: {error_code} - {str(e)}")
|
||||
if attempt == MAX_RETRIES:
|
||||
raise
|
||||
time.sleep(exponential_backoff(attempt))
|
||||
|
||||
|
||||
message_body = json.dumps(fake_sns_payload)
|
||||
|
||||
# Strategie: SNS zuerst, SQS als Fallback
|
||||
topic_arn = get_topic_arn(domain)
|
||||
sns_success = publish_to_sns(topic_arn, message_body, msg_id)
|
||||
|
||||
if not sns_success:
|
||||
# Kein SNS-Topic für diese Domain → direkt in SQS (wie bisher)
|
||||
queue_url = get_queue_url(domain)
|
||||
send_to_sqs(queue_url, message_body, msg_id)
|
||||
|
||||
return {'status': 'ok'}
|
||||
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"❌ Critical Error in Lambda Shim: {str(e)}", exc_info=True)
|
||||
raise e
|
||||
@@ -7,7 +7,16 @@ RUN xcaddy build ${CADDY_VERSION} \
|
||||
--with github.com/caddy-dns/cloudflare \
|
||||
--with github.com/caddyserver/replace-response
|
||||
|
||||
# Autodiscover Handler in Go bauen (Go ist im Builder-Image bereits verfügbar)
|
||||
COPY autodiscover-handler.go /src/autodiscover-handler.go
|
||||
WORKDIR /src
|
||||
RUN CGO_ENABLED=0 go build -ldflags="-s -w" -o /usr/bin/autodiscover-handler autodiscover-handler.go
|
||||
|
||||
FROM caddy:${CADDY_VERSION}
|
||||
COPY --from=builder /usr/bin/caddy /usr/bin/caddy
|
||||
RUN mkdir -p /var/log/caddy
|
||||
COPY --from=builder /usr/bin/autodiscover-handler /usr/local/bin/autodiscover-handler
|
||||
COPY start.sh /usr/local/bin/start.sh
|
||||
RUN chmod +x /usr/local/bin/start.sh /usr/local/bin/autodiscover-handler \
|
||||
&& mkdir -p /var/log/caddy
|
||||
|
||||
CMD ["/usr/local/bin/start.sh"]
|
||||
109
caddy/autodiscover-handler.go
Normal file
109
caddy/autodiscover-handler.go
Normal file
@@ -0,0 +1,109 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"regexp"
|
||||
"strings"
|
||||
)
|
||||
|
||||
const port = "8280"
|
||||
|
||||
var emailRegex = regexp.MustCompile(`(?i)<EMailAddress>([^<]+)</EMailAddress>`)
|
||||
|
||||
func main() {
|
||||
http.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
|
||||
w.WriteHeader(http.StatusOK)
|
||||
fmt.Fprint(w, "OK")
|
||||
})
|
||||
|
||||
http.HandleFunc("/autodiscover/autodiscover.xml", handleAutodiscover)
|
||||
// Outlook sendet manchmal mit Großbuchstaben
|
||||
http.HandleFunc("/Autodiscover/Autodiscover.xml", handleAutodiscover)
|
||||
http.HandleFunc("/AutoDiscover/AutoDiscover.xml", handleAutodiscover)
|
||||
|
||||
log.Printf("[autodiscover] Listening on port %s", port)
|
||||
if err := http.ListenAndServe(":"+port, nil); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func handleAutodiscover(w http.ResponseWriter, r *http.Request) {
|
||||
var email string
|
||||
|
||||
if r.Method == http.MethodPost {
|
||||
body, err := io.ReadAll(r.Body)
|
||||
if err == nil {
|
||||
if match := emailRegex.FindStringSubmatch(string(body)); len(match) > 1 {
|
||||
email = strings.TrimSpace(match[1])
|
||||
}
|
||||
}
|
||||
r.Body.Close()
|
||||
}
|
||||
|
||||
var domain string
|
||||
if email != "" {
|
||||
parts := strings.SplitN(email, "@", 2)
|
||||
if len(parts) == 2 {
|
||||
domain = parts[1]
|
||||
}
|
||||
}
|
||||
if domain == "" {
|
||||
domain = extractDomainFromHost(r.Host)
|
||||
}
|
||||
|
||||
log.Printf("[autodiscover] %s from %s - email=%q domain=%s", r.Method, r.RemoteAddr, email, domain)
|
||||
|
||||
w.Header().Set("Content-Type", "application/xml")
|
||||
w.WriteHeader(http.StatusOK)
|
||||
fmt.Fprint(w, buildResponse(domain, email))
|
||||
}
|
||||
|
||||
func extractDomainFromHost(host string) string {
|
||||
// Strip port
|
||||
if idx := strings.Index(host, ":"); idx >= 0 {
|
||||
host = host[:idx]
|
||||
}
|
||||
parts := strings.Split(host, ".")
|
||||
if len(parts) >= 3 && strings.EqualFold(parts[0], "autodiscover") {
|
||||
return strings.Join(parts[1:], ".")
|
||||
}
|
||||
if len(parts) >= 2 {
|
||||
return strings.Join(parts[len(parts)-2:], ".")
|
||||
}
|
||||
return host
|
||||
}
|
||||
|
||||
func buildResponse(domain, loginName string) string {
|
||||
return fmt.Sprintf(`<?xml version="1.0" encoding="utf-8"?>
|
||||
<Autodiscover xmlns="http://schemas.microsoft.com/exchange/autodiscover/responseschema/2006">
|
||||
<Response xmlns="http://schemas.microsoft.com/exchange/autodiscover/outlook/responseschema/2006a">
|
||||
<Account>
|
||||
<AccountType>email</AccountType>
|
||||
<Action>settings</Action>
|
||||
<Protocol>
|
||||
<Type>IMAP</Type>
|
||||
<Server>imap.%s</Server>
|
||||
<Port>993</Port>
|
||||
<DomainRequired>off</DomainRequired>
|
||||
<LoginName>%s</LoginName>
|
||||
<SPA>off</SPA>
|
||||
<SSL>on</SSL>
|
||||
<AuthRequired>on</AuthRequired>
|
||||
</Protocol>
|
||||
<Protocol>
|
||||
<Type>SMTP</Type>
|
||||
<Server>smtp.%s</Server>
|
||||
<Port>465</Port>
|
||||
<DomainRequired>off</DomainRequired>
|
||||
<LoginName>%s</LoginName>
|
||||
<SPA>off</SPA>
|
||||
<SSL>on</SSL>
|
||||
<AuthRequired>on</AuthRequired>
|
||||
</Protocol>
|
||||
</Account>
|
||||
</Response>
|
||||
</Autodiscover>`, domain, loginName, domain, loginName)
|
||||
}
|
||||
@@ -13,6 +13,7 @@ services:
|
||||
- 'host.docker.internal:host-gateway'
|
||||
networks:
|
||||
- mail_network
|
||||
- mailadmin_network
|
||||
volumes:
|
||||
- $PWD/Caddyfile:/etc/caddy/Caddyfile
|
||||
- $PWD/mail_certs:/etc/caddy/mail_certs
|
||||
@@ -29,3 +30,5 @@ services:
|
||||
networks:
|
||||
mail_network:
|
||||
external: true
|
||||
mailadmin_network:
|
||||
external: true
|
||||
|
||||
@@ -35,7 +35,7 @@
|
||||
<body>
|
||||
|
||||
<div class="card">
|
||||
<img src="/logo.png" alt="Logo" class="logo">
|
||||
<img src="/email-setup/logo.png" alt="Logo" class="logo">
|
||||
|
||||
<div id="input-section">
|
||||
<h1>Email Setup</h1>
|
||||
|
||||
8
caddy/start.sh
Normal file
8
caddy/start.sh
Normal file
@@ -0,0 +1,8 @@
|
||||
#!/bin/sh
|
||||
set -e
|
||||
|
||||
# Autodiscover handler im Hintergrund starten
|
||||
/usr/local/bin/autodiscover-handler &
|
||||
|
||||
# Caddy im Vordergrund
|
||||
exec caddy run --config /etc/caddy/Caddyfile --adapter caddyfile
|
||||
@@ -78,36 +78,7 @@ OUTPUT="${OUTPUT}(email_settings) {\n"
|
||||
# --- 1. Outlook Classic Autodiscover (POST + GET XML) ---
|
||||
OUTPUT="${OUTPUT} # Outlook Autodiscover (XML) - POST und GET\n"
|
||||
OUTPUT="${OUTPUT} route /autodiscover/autodiscover.xml {\n"
|
||||
OUTPUT="${OUTPUT} header Content-Type \"application/xml\"\n"
|
||||
OUTPUT="${OUTPUT} respond \`<?xml version=\"1.0\" encoding=\"utf-8\"?>\n"
|
||||
OUTPUT="${OUTPUT}<Autodiscover xmlns=\"http://schemas.microsoft.com/exchange/autodiscover/responseschema/2006\">\n"
|
||||
OUTPUT="${OUTPUT} <Response xmlns=\"http://schemas.microsoft.com/exchange/autodiscover/outlook/responseschema/2006a\">\n"
|
||||
OUTPUT="${OUTPUT} <Account>\n"
|
||||
OUTPUT="${OUTPUT} <AccountType>email</AccountType>\n"
|
||||
OUTPUT="${OUTPUT} <Action>settings</Action>\n"
|
||||
OUTPUT="${OUTPUT} <Protocol>\n"
|
||||
OUTPUT="${OUTPUT} <Type>IMAP</Type>\n"
|
||||
OUTPUT="${OUTPUT} <Server>imap.{labels.1}.{labels.0}</Server>\n"
|
||||
OUTPUT="${OUTPUT} <Port>993</Port>\n"
|
||||
OUTPUT="${OUTPUT} <DomainRequired>off</DomainRequired>\n"
|
||||
OUTPUT="${OUTPUT} <LoginName></LoginName>\n"
|
||||
OUTPUT="${OUTPUT} <SPA>off</SPA>\n"
|
||||
OUTPUT="${OUTPUT} <SSL>on</SSL>\n"
|
||||
OUTPUT="${OUTPUT} <AuthRequired>on</AuthRequired>\n"
|
||||
OUTPUT="${OUTPUT} </Protocol>\n"
|
||||
OUTPUT="${OUTPUT} <Protocol>\n"
|
||||
OUTPUT="${OUTPUT} <Type>SMTP</Type>\n"
|
||||
OUTPUT="${OUTPUT} <Server>smtp.{labels.1}.{labels.0}</Server>\n"
|
||||
OUTPUT="${OUTPUT} <Port>465</Port>\n"
|
||||
OUTPUT="${OUTPUT} <DomainRequired>off</DomainRequired>\n"
|
||||
OUTPUT="${OUTPUT} <LoginName></LoginName>\n"
|
||||
OUTPUT="${OUTPUT} <SPA>off</SPA>\n"
|
||||
OUTPUT="${OUTPUT} <SSL>on</SSL>\n"
|
||||
OUTPUT="${OUTPUT} <AuthRequired>on</AuthRequired>\n"
|
||||
OUTPUT="${OUTPUT} </Protocol>\n"
|
||||
OUTPUT="${OUTPUT} </Account>\n"
|
||||
OUTPUT="${OUTPUT} </Response>\n"
|
||||
OUTPUT="${OUTPUT}</Autodiscover>\` 200\n"
|
||||
OUTPUT="${OUTPUT} reverse_proxy localhost:8280\n"
|
||||
OUTPUT="${OUTPUT} }\n"
|
||||
OUTPUT="${OUTPUT}\n"
|
||||
|
||||
@@ -135,14 +106,14 @@ OUTPUT="${OUTPUT} <hostname>imap.{labels.1}.{labels.0}</hostname>\n"
|
||||
OUTPUT="${OUTPUT} <port>993</port>\n"
|
||||
OUTPUT="${OUTPUT} <socketType>SSL</socketType>\n"
|
||||
OUTPUT="${OUTPUT} <authentication>password-cleartext</authentication>\n"
|
||||
OUTPUT="${OUTPUT} <username>%%EMAILADDRESS%%</username>\n"
|
||||
OUTPUT="${OUTPUT} <username>%EMAILADDRESS%</username>\n"
|
||||
OUTPUT="${OUTPUT} </incomingServer>\n"
|
||||
OUTPUT="${OUTPUT} <outgoingServer type=\"smtp\">\n"
|
||||
OUTPUT="${OUTPUT} <hostname>smtp.{labels.1}.{labels.0}</hostname>\n"
|
||||
OUTPUT="${OUTPUT} <port>465</port>\n"
|
||||
OUTPUT="${OUTPUT} <socketType>SSL</socketType>\n"
|
||||
OUTPUT="${OUTPUT} <authentication>password-cleartext</authentication>\n"
|
||||
OUTPUT="${OUTPUT} <username>%%EMAILADDRESS%%</username>\n"
|
||||
OUTPUT="${OUTPUT} <username>%EMAILADDRESS%</username>\n"
|
||||
OUTPUT="${OUTPUT} </outgoingServer>\n"
|
||||
OUTPUT="${OUTPUT} </emailProvider>\n"
|
||||
OUTPUT="${OUTPUT}</clientConfig>\` 200\n"
|
||||
@@ -289,6 +260,13 @@ for domain in $DOMAINS; do
|
||||
OUTPUT="${OUTPUT} format console\n"
|
||||
OUTPUT="${OUTPUT} }\n"
|
||||
OUTPUT="${OUTPUT}}\n\n"
|
||||
|
||||
# Mailadmin Block
|
||||
OUTPUT="${OUTPUT}# MailAdmin UI\n"
|
||||
OUTPUT="${OUTPUT}mailadmin.${domain} {\n"
|
||||
OUTPUT="${OUTPUT} encode gzip\n"
|
||||
OUTPUT="${OUTPUT} reverse_proxy mailadmin:3000\n"
|
||||
OUTPUT="${OUTPUT}}\n\n"
|
||||
|
||||
# Autodiscover / Autoconfig Block
|
||||
OUTPUT="${OUTPUT}# Autodiscover/Autoconfig für $domain\n"
|
||||
|
||||
@@ -10,12 +10,14 @@ services:
|
||||
ports:
|
||||
- "9000:8000" # Prometheus metrics (Host:Container)
|
||||
- "9090:8080" # Health check (Host:Container)
|
||||
# Connect to DMS on the host or Docker network
|
||||
extra_hosts:
|
||||
- "host.docker.internal:host-gateway"
|
||||
environment:
|
||||
- SMTP_HOST=host.docker.internal
|
||||
- SMTP_HOST=mailserver
|
||||
- SMTP_PORT=25
|
||||
networks:
|
||||
- mail_network
|
||||
|
||||
volumes:
|
||||
worker-logs:
|
||||
networks:
|
||||
mail_network:
|
||||
external: true
|
||||
@@ -48,6 +48,9 @@ export const config = {
|
||||
// Monitoring
|
||||
metricsPort: parseInt(process.env.METRICS_PORT ?? '8000', 10),
|
||||
healthPort: parseInt(process.env.HEALTH_PORT ?? '8080', 10),
|
||||
|
||||
queueSuffix: process.env.QUEUE_SUFFIX ?? '-queue',
|
||||
standbyMode: (process.env.STANDBY_MODE ?? 'false').toLowerCase() === 'true',
|
||||
} as const;
|
||||
|
||||
export type Config = typeof config;
|
||||
@@ -106,7 +109,7 @@ export function isInternalAddress(email: string): boolean {
|
||||
|
||||
/** Convert domain to SQS queue name: bizmatch.net → bizmatch-net-queue */
|
||||
export function domainToQueueName(domain: string): string {
|
||||
return domain.replace(/\./g, '-') + '-queue';
|
||||
return domain.replace(/\./g, '-') + config.queueSuffix;
|
||||
}
|
||||
|
||||
/** Convert domain to S3 bucket name: bizmatch.net → bizmatch-net-emails */
|
||||
|
||||
@@ -36,6 +36,9 @@ export class RulesProcessor {
|
||||
workerName: string,
|
||||
metricsCallback?: MetricsCallback,
|
||||
): Promise<boolean> {
|
||||
if (config.standbyMode) {
|
||||
return false;
|
||||
}
|
||||
const rule = await this.dynamodb.getEmailRules(recipient.toLowerCase());
|
||||
if (!rule) return false;
|
||||
|
||||
|
||||
@@ -26,7 +26,7 @@ import { BlocklistChecker } from '../email/blocklist.js';
|
||||
import { BounceHandler } from '../email/bounce-handler.js';
|
||||
import { parseEmail, isProcessedByWorker } from '../email/parser.js';
|
||||
import { RulesProcessor } from '../email/rules-processor.js';
|
||||
|
||||
import { config } from '../config.js';
|
||||
// ---------------------------------------------------------------------------
|
||||
// Processor
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -258,34 +258,40 @@ export class MessageProcessor {
|
||||
|
||||
if (totalHandled === recipients.length) {
|
||||
if (blockedRecipients.length === recipients.length) {
|
||||
// All blocked
|
||||
try {
|
||||
await this.s3.markAsBlocked(
|
||||
domain,
|
||||
messageId,
|
||||
blockedRecipients,
|
||||
fromAddrFinal,
|
||||
workerName,
|
||||
);
|
||||
await this.s3.deleteBlockedEmail(domain, messageId, workerName);
|
||||
} catch (err: any) {
|
||||
log(`⚠ Failed to handle blocked email: ${err.message ?? err}`, 'ERROR', workerName);
|
||||
return false;
|
||||
// All blocked — im Standby kein S3 anfassen
|
||||
if (!config.standbyMode) {
|
||||
try {
|
||||
await this.s3.markAsBlocked(
|
||||
domain,
|
||||
messageId,
|
||||
blockedRecipients,
|
||||
fromAddrFinal,
|
||||
workerName,
|
||||
);
|
||||
await this.s3.deleteBlockedEmail(domain, messageId, workerName);
|
||||
} catch (err: any) {
|
||||
log(`⚠ Failed to handle blocked email: ${err.message ?? err}`, 'ERROR', workerName);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
} else if (successful.length > 0) {
|
||||
await this.s3.markAsProcessed(
|
||||
domain,
|
||||
messageId,
|
||||
workerName,
|
||||
failedPermanent.length > 0 ? failedPermanent : undefined,
|
||||
);
|
||||
if (!config.standbyMode) {
|
||||
await this.s3.markAsProcessed(
|
||||
domain,
|
||||
messageId,
|
||||
workerName,
|
||||
failedPermanent.length > 0 ? failedPermanent : undefined,
|
||||
);
|
||||
}
|
||||
} else if (failedPermanent.length > 0) {
|
||||
await this.s3.markAsAllInvalid(
|
||||
domain,
|
||||
messageId,
|
||||
failedPermanent,
|
||||
workerName,
|
||||
);
|
||||
if (!config.standbyMode) {
|
||||
await this.s3.markAsAllInvalid(
|
||||
domain,
|
||||
messageId,
|
||||
failedPermanent,
|
||||
workerName,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// Summary
|
||||
|
||||
317
lambda/relay-outbound.py
Normal file
317
lambda/relay-outbound.py
Normal file
@@ -0,0 +1,317 @@
|
||||
"""
|
||||
relay-outbound Lambda
|
||||
|
||||
Handles SES Configuration Set events forwarded via EventBridge.
|
||||
Writes to two DynamoDB tables:
|
||||
|
||||
- ses-outbound-messages : EXISTING table for bounce forensics
|
||||
(kept unchanged so existing tooling keeps working)
|
||||
- ses-events : NEW table for billing-style aggregation
|
||||
of sends/bounces/complaints
|
||||
|
||||
We list mailadmin domains to filter out events from non-mailadmin
|
||||
addresses (e.g. SES console test sends).
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import boto3
|
||||
|
||||
dynamodb = boto3.resource('dynamodb')
|
||||
ses_client = boto3.client('ses')
|
||||
|
||||
# Existing bounce-detail table — preserved as-is
|
||||
bounce_table = dynamodb.Table('ses-outbound-messages')
|
||||
|
||||
# New unified events table for billing
|
||||
events_table = dynamodb.Table('ses-events')
|
||||
|
||||
# ------------------------------------------------------------
|
||||
# Domain filter
|
||||
# ------------------------------------------------------------
|
||||
# We only want to track events for mailboxes managed by mailadmin.
|
||||
# Easiest source-of-truth: SES verified domain identities. Any address
|
||||
# whose domain is verified in SES is "ours". Cached for 5 minutes so
|
||||
# we don't hammer SES on every event.
|
||||
# ------------------------------------------------------------
|
||||
|
||||
_domain_cache = {'expires_at': 0, 'domains': set()}
|
||||
_DOMAIN_CACHE_TTL_SEC = 300
|
||||
|
||||
|
||||
def get_managed_domains():
|
||||
"""Return the set of SES verified domain identities (lowercased)."""
|
||||
now = time.time()
|
||||
if _domain_cache['expires_at'] > now and _domain_cache['domains']:
|
||||
return _domain_cache['domains']
|
||||
|
||||
try:
|
||||
# ListIdentities is paginated; for ~50 domains one page is enough,
|
||||
# but handle pagination defensively.
|
||||
domains = set()
|
||||
next_token = None
|
||||
while True:
|
||||
kwargs = {'IdentityType': 'Domain', 'MaxItems': 100}
|
||||
if next_token:
|
||||
kwargs['NextToken'] = next_token
|
||||
resp = ses_client.list_identities(**kwargs)
|
||||
for d in resp.get('Identities', []):
|
||||
domains.add(d.lower())
|
||||
next_token = resp.get('NextToken')
|
||||
if not next_token:
|
||||
break
|
||||
|
||||
_domain_cache['domains'] = domains
|
||||
_domain_cache['expires_at'] = now + _DOMAIN_CACHE_TTL_SEC
|
||||
return domains
|
||||
except Exception as exc:
|
||||
print(f"WARN: could not list SES identities, falling back to cached set: {exc}")
|
||||
return _domain_cache['domains']
|
||||
|
||||
|
||||
def domain_of(email):
|
||||
if not email or '@' not in email:
|
||||
return None
|
||||
return email.split('@', 1)[1].strip().lower()
|
||||
|
||||
|
||||
def is_managed_address(email):
|
||||
d = domain_of(email)
|
||||
if not d:
|
||||
return False
|
||||
return d in get_managed_domains()
|
||||
|
||||
|
||||
# ------------------------------------------------------------
|
||||
# DynamoDB helpers
|
||||
# ------------------------------------------------------------
|
||||
|
||||
def _ttl_two_years_from_now():
|
||||
return int((datetime.now(timezone.utc) + timedelta(days=730)).timestamp())
|
||||
|
||||
|
||||
def _month_bucket(ts_iso):
|
||||
"""
|
||||
Convert a SES event timestamp (ISO 8601) to 'YYYY-MM'.
|
||||
Falls back to current UTC month if parsing fails.
|
||||
"""
|
||||
try:
|
||||
if ts_iso.endswith('Z'):
|
||||
ts_iso = ts_iso[:-1] + '+00:00'
|
||||
dt = datetime.fromisoformat(ts_iso)
|
||||
except Exception:
|
||||
dt = datetime.now(timezone.utc)
|
||||
return dt.strftime('%Y-%m')
|
||||
|
||||
|
||||
def write_event_row(*, event_type, source, recipients, message_id, timestamp_iso,
|
||||
size_bytes=None, subject=None, bounce_type=None, complaint_type=None,
|
||||
smtp_response=None, reporting_mta=None):
|
||||
"""
|
||||
Write a row to the unified ses-events table.
|
||||
|
||||
Skips silently if the source address isn't from a managed domain
|
||||
(per user choice: ignore non-mailadmin events).
|
||||
"""
|
||||
if not is_managed_address(source):
|
||||
print(f"Skipping event for non-managed sender: {source}")
|
||||
return
|
||||
|
||||
domain = domain_of(source)
|
||||
ym = _month_bucket(timestamp_iso)
|
||||
|
||||
pk = f"{domain}#{source}#{ym}"
|
||||
sk = f"{timestamp_iso}#{message_id}"
|
||||
|
||||
item = {
|
||||
'pk': pk,
|
||||
'sk': sk,
|
||||
'event_type': event_type,
|
||||
'from': source,
|
||||
'domain': domain,
|
||||
'recipients': recipients or [],
|
||||
'message_id': message_id or '',
|
||||
'timestamp': timestamp_iso,
|
||||
'ttl_unix': _ttl_two_years_from_now(),
|
||||
}
|
||||
if size_bytes is not None:
|
||||
item['size_bytes'] = int(size_bytes)
|
||||
if subject:
|
||||
item['subject'] = subject[:200]
|
||||
if bounce_type:
|
||||
item['bounce_type'] = bounce_type
|
||||
if complaint_type:
|
||||
item['complaint_type'] = complaint_type
|
||||
if smtp_response:
|
||||
item['smtp_response'] = smtp_response[:500]
|
||||
if reporting_mta:
|
||||
item['reporting_mta'] = reporting_mta
|
||||
|
||||
events_table.put_item(Item=item)
|
||||
print(f"ses-events <- {event_type} {pk} {sk}")
|
||||
|
||||
|
||||
# ------------------------------------------------------------
|
||||
# Existing bounce handler (unchanged behavior, plus the new write)
|
||||
# ------------------------------------------------------------
|
||||
|
||||
def handle_bounce(detail):
|
||||
bounce = detail.get('bounce', {}) or {}
|
||||
mail = detail.get('mail', {}) or {}
|
||||
|
||||
feedback_id = bounce.get('feedbackId')
|
||||
bounce_type = bounce.get('bounceType', 'Unknown')
|
||||
bounce_subtype = bounce.get('bounceSubType', 'Unknown')
|
||||
bounced_recipients = [r.get('emailAddress') for r in bounce.get('bouncedRecipients', []) if r.get('emailAddress')]
|
||||
original_source = mail.get('source')
|
||||
original_message_id = mail.get('messageId')
|
||||
timestamp = bounce.get('timestamp') or mail.get('timestamp') or datetime.now(timezone.utc).isoformat()
|
||||
|
||||
print(f"Bounce: id={feedback_id} from={original_source} type={bounce_type}/{bounce_subtype}")
|
||||
|
||||
# 1) EXISTING: store full bounce forensics in ses-outbound-messages
|
||||
bounce_table.put_item(Item={
|
||||
'MessageId': feedback_id,
|
||||
'original_message_id': original_message_id,
|
||||
'original_source': original_source,
|
||||
'bounceType': bounce_type,
|
||||
'bounceSubType': bounce_subtype,
|
||||
'bouncedRecipients': bounced_recipients,
|
||||
'timestamp': timestamp,
|
||||
'event_type': 'bounce',
|
||||
})
|
||||
|
||||
# 2) NEW: also record into ses-events for billing aggregation
|
||||
write_event_row(
|
||||
event_type='bounce',
|
||||
source=original_source,
|
||||
recipients=bounced_recipients,
|
||||
message_id=original_message_id or feedback_id,
|
||||
timestamp_iso=timestamp,
|
||||
bounce_type=f"{bounce_type}/{bounce_subtype}",
|
||||
)
|
||||
|
||||
|
||||
# ------------------------------------------------------------
|
||||
# New: send handler
|
||||
# ------------------------------------------------------------
|
||||
|
||||
def handle_send(detail):
|
||||
mail = detail.get('mail', {}) or {}
|
||||
source = mail.get('source')
|
||||
message_id = mail.get('messageId')
|
||||
timestamp = mail.get('timestamp') or datetime.now(timezone.utc).isoformat()
|
||||
|
||||
# SES events do not include the message size directly. The closest
|
||||
# info available is mail.headersTruncated and mail.commonHeaders, but
|
||||
# neither contains the byte size. We approximate from the sum of all
|
||||
# known header values + 0 (we'll log 0 if unknown). If size_bytes is
|
||||
# critical, the alternative is to request the 'sendingPoolMetadata'
|
||||
# via SES API, but for billing-grade tracking the SES Send Quota
|
||||
# itself is per-message anyway.
|
||||
common = mail.get('commonHeaders', {}) or {}
|
||||
subject = common.get('subject')
|
||||
|
||||
# Recipients: prefer 'to' from commonHeaders, fall back to destination
|
||||
to_field = common.get('to') or mail.get('destination') or []
|
||||
if isinstance(to_field, str):
|
||||
to_field = [to_field]
|
||||
|
||||
# Try to get size from SES if exposed (newer event format includes it
|
||||
# under 'mail.headersTruncated' — not always present)
|
||||
size_bytes = mail.get('messageSize') or mail.get('size') or 0
|
||||
|
||||
print(f"Send: id={message_id} from={source} to={to_field} size={size_bytes}")
|
||||
|
||||
write_event_row(
|
||||
event_type='send',
|
||||
source=source,
|
||||
recipients=to_field,
|
||||
message_id=message_id,
|
||||
timestamp_iso=timestamp,
|
||||
size_bytes=size_bytes,
|
||||
subject=subject,
|
||||
)
|
||||
|
||||
|
||||
# ------------------------------------------------------------
|
||||
# New: complaint handler
|
||||
# ------------------------------------------------------------
|
||||
|
||||
def handle_complaint(detail):
|
||||
complaint = detail.get('complaint', {}) or {}
|
||||
mail = detail.get('mail', {}) or {}
|
||||
|
||||
source = mail.get('source')
|
||||
message_id = mail.get('messageId')
|
||||
timestamp = complaint.get('timestamp') or mail.get('timestamp') or datetime.now(timezone.utc).isoformat()
|
||||
complaint_type = complaint.get('complaintFeedbackType') or 'unknown'
|
||||
complained_recipients = [r.get('emailAddress') for r in complaint.get('complainedRecipients', []) if r.get('emailAddress')]
|
||||
|
||||
print(f"Complaint: id={message_id} from={source} type={complaint_type}")
|
||||
|
||||
write_event_row(
|
||||
event_type='complaint',
|
||||
source=source,
|
||||
recipients=complained_recipients,
|
||||
message_id=message_id,
|
||||
timestamp_iso=timestamp,
|
||||
complaint_type=complaint_type,
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------
|
||||
# New: delivery handler
|
||||
# ------------------------------------------------------------
|
||||
def handle_delivery(detail):
|
||||
delivery = detail.get('delivery', {}) or {}
|
||||
mail = detail.get('mail', {}) or {}
|
||||
|
||||
source = mail.get('source')
|
||||
message_id = mail.get('messageId')
|
||||
timestamp = delivery.get('timestamp') or mail.get('timestamp') or datetime.now(timezone.utc).isoformat()
|
||||
recipients = delivery.get('recipients') or []
|
||||
smtp_response = delivery.get('smtpResponse') # wörtliche M365-Antwort
|
||||
reporting_mta = delivery.get('reportingMTA')
|
||||
processing_ms = delivery.get('processingTimeMillis')
|
||||
|
||||
print(f"Delivery: id={message_id} to={recipients} smtp={smtp_response!r} mta={reporting_mta}")
|
||||
|
||||
write_event_row(
|
||||
event_type='delivery',
|
||||
source=source,
|
||||
recipients=recipients,
|
||||
message_id=message_id,
|
||||
timestamp_iso=timestamp,
|
||||
smtp_response=smtp_response,
|
||||
reporting_mta=reporting_mta,
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------
|
||||
# Entry point
|
||||
# ------------------------------------------------------------
|
||||
|
||||
def lambda_handler(event, context):
|
||||
print(f"Received event: {json.dumps(event)[:500]}")
|
||||
|
||||
event_type = event.get('detail-type')
|
||||
detail = event.get('detail', {}) or {}
|
||||
|
||||
try:
|
||||
if event_type == 'Email Bounced':
|
||||
handle_bounce(detail)
|
||||
elif event_type == 'Email Sent':
|
||||
handle_send(detail)
|
||||
elif event_type == 'Email Complaint Received':
|
||||
handle_complaint(detail)
|
||||
elif event_type == 'Email Delivered':
|
||||
handle_delivery(detail)
|
||||
else:
|
||||
print(f"Ignoring event type: {event_type}")
|
||||
except Exception as exc:
|
||||
# Don't fail the Lambda — losing one event is preferable to
|
||||
# endless EventBridge retries that pile up DLQs.
|
||||
print(f"ERROR processing {event_type}: {exc}")
|
||||
|
||||
return {'statusCode': 200}
|
||||
Reference in New Issue
Block a user