Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/actions/notify-deploy-authors/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
.env
99 changes: 99 additions & 0 deletions .github/actions/notify-deploy-authors/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
# Notify Deploy Authors Action

DMs everyone whose code is in a deploy, telling them their change is live and pointing them at the dashboard to watch.

## Why it runs after the deploy

There is nothing to monitor until the rollout has finished, so this belongs in the deploy path and nowhere
earlier. The step is gated on the deploy succeeding, which is the point at which "your change is live" is
actually true β€” anything running before that can only promise a change is on its way, which is not a useful cue
to go and watch a dashboard.

Keeping it a separate job also means it only fires for things that are actually deployed, and that it can never
hold up or affect the deploy itself.

## Usage

```yaml
- name: Notify deploy authors
uses: monta-app/github-workflows/.github/actions/notify-deploy-authors@main
with:
previous-ref: ${{ needs.deploy.outputs.previous-image-tag }}
current-ref: ${{ needs.deploy.outputs.image-tag }}
service-name: "Monta PHP Monolith"
stage: production
dashboard-url: "https://montaapp.grafana.net/d/ja52q4d/server-error-dashboard?from=now-30m&to=now&timezone=browser&var-container=$__all"
dashboard-label: "Server Error Dashboard"
slack-token: ${{ secrets.SLACK_APP_TOKEN }}
github-token: ${{ secrets.GITHUB_TOKEN }}
identity-api-url: "https://project-tracker.vpn.internal.monta.app"
```

Gate the step on a successful deploy (`if: needs.deploy.result == 'success'`) β€” there is no point asking anyone
to watch a rollout that failed.

## Inputs

| Input | Required | Default | Description |
|-------|----------|---------|-------------|
| `previous-ref` | Yes | - | Commit SHA or tag the deploy starts after |
| `current-ref` | Yes | - | Commit SHA or tag the deploy ends on |
| `service-name` | Yes | - | Human-readable service name |
| `dashboard-url` | Yes | - | Dashboard the authors should watch |
| `slack-token` | Yes | - | Needs `users:read.email`, `im:write`, `chat:write` |
| `github-token` | Yes | - | Token that can read the repository |
| `dashboard-label` | No | `Error Dashboard` | Link text for the dashboard |
| `stage` | No | `''` | Only affects wording ("… is now deployed to Production.") |
| `ref-url` | No | `''` | Links the deployed ref, when the caller has a URL for it |
| `identity-api-url` | No | `''` | Identity resolver base URL β€” see below |
| `monitoring-window` | No | `the next 30 minutes` | How long to ask them to watch |
| `max-authors` | No | `15` | Refuse to send if the range has more authors than this |
| `dry-run` | No | `false` | Print the payloads instead of sending them |
| `strict` | No | `false` | Exit non-zero on failure instead of warning |

## Reaching the author in Slack

This is the hard part. A GitHub login does not get you to a Slack account, and the obvious routes mostly fail:

| Signal | Reality (measured on `monta-app/server`) |
|---|---|
| Public GitHub profile email | Set on 4 of 17 sampled org members |
| …and wrong when it is set | `Casperhr` is `cr@monta.app` on GitHub but `cr@monta.com` in Slack β€” the lookup fails |
| Commit author email | 43% `users.noreply.github.com`, 24% `@monta.com`, 18% `@monta.app`, 13% personal |

So set `identity-api-url` to project-tracker's resolver, which holds each person's work email β€” the address
Slack actually knows. Measured on one real deploy range, with the resolver 2 of 2 authors were reachable;
without it, 0 of 2 (one author's GitHub login is literally `838`).

Resolution order per person: the resolver's Slack id β†’ the resolver's work email β†’ the `Co-authored-by:` trailer
email β†’ the public GitHub profile email. Each address is looked up via `users.lookupByEmail`.

The resolver is **VPN-only**, so a GitHub-hosted runner cannot reach it. Join the runner to the tailnet with
`tailscale/github-action` (the `TAILSCALE_AUTHKEY` pattern used elsewhere in this org) or use a self-hosted
runner. Without it the action still runs, just reaching fewer people, and it names everyone it could not reach.

## Behaviour worth knowing

- **It never fails your deploy.** Every runtime problem is a `::warning::` and a zero exit. Use `strict: true`
while testing the wiring.
- **It refuses to mass-DM.** More than `max-authors` people in the range is treated as wrong refs, not as a
busy deploy: nothing is sent and the count is logged. A 3-day range on the monolith resolves to 34 authors.
- **Bots are skipped** β€” both `…[bot]` logins and anyone the resolver flags as a bot or a leaver.
- **One DM per person**, even when they appear both as a commit author and in a `Co-authored-by:` trailer.

## Testing

`./test-local.sh` runs the script against a real ref range with `dry-run` on, so it prints the exact Slack
payloads without messaging anyone. `GITHUB_TOKEN` is picked up from `gh auth token` if unset. Export the rest
for your own repo, service and dashboard - none of it is specific to this action, so there's no example file:

```shell
export GITHUB_REPOSITORY=owner/repo
export PREVIOUS_REF=... CURRENT_REF=... # any two refs with commits between them
export SERVICE_NAME="My Service" STAGE=production
export DASHBOARD_URL=... DASHBOARD_LABEL=...
export IDENTITY_API_URL=https://project-tracker.vpn.internal.monta.app
./test-local.sh
```

Or drop the same variables in a local `.env` (gitignored) and `test-local.sh` will source it for you.
86 changes: 86 additions & 0 deletions .github/actions/notify-deploy-authors/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
name: 'Notify Deploy Authors'
description: 'DM the authors of everything in a deploy, asking them to watch a dashboard now their change is live'
author: 'Monta'

inputs:
previous-ref:
description: 'Commit SHA or tag the deploy starts after (e.g. the previous image tag)'
required: true
current-ref:
description: 'Commit SHA or tag the deploy ends on (e.g. the image tag just deployed)'
required: true
service-name:
description: 'Human-readable service name, e.g. "Monta PHP Monolith"'
required: true
dashboard-url:
description: 'Dashboard the authors should watch'
required: true
slack-token:
description: 'Slack bot token. Needs users:read.email, im:write and chat:write'
required: true
github-token:
description: 'Token that can read the repository'
required: true
dashboard-label:
description: 'Link text for the dashboard'
required: false
default: 'Error Dashboard'
stage:
description: 'Deployment stage, e.g. production. Only used in the message wording'
required: false
default: ''
ref-url:
description: 'URL to link the deployed ref to. Rendered as plain text when omitted'
required: false
default: ''
identity-api-url:
description: >-
Base URL of the identity resolver (project-tracker) used to map GitHub logins to Slack
accounts. Strongly recommended: without it only authors whose public GitHub email matches
their Slack account are reachable, which is a minority. Needs tailnet access from the runner.
required: false
default: ''
monitoring-window:
description: 'How long to ask authors to watch for'
required: false
default: 'the next 30 minutes'
max-authors:
description: >-
Refuse to send anything if the ref range contains more authors than this. A range far larger
than a normal deploy almost always means the refs are wrong, and mass-DMing is worse than
sending nothing.
required: false
default: '15'
dry-run:
description: 'Print what would be sent instead of sending it'
required: false
default: 'false'
strict:
description: >-
Exit non-zero when the notification cannot be sent. Off by default, because a monitoring
nudge should never fail a deploy. Useful while testing the wiring.
required: false
default: 'false'

runs:
using: 'composite'
steps:
- name: DM the authors in this deploy
shell: bash
env:
PREVIOUS_REF: ${{ inputs.previous-ref }}
CURRENT_REF: ${{ inputs.current-ref }}
SERVICE_NAME: ${{ inputs.service-name }}
DASHBOARD_URL: ${{ inputs.dashboard-url }}
DASHBOARD_LABEL: ${{ inputs.dashboard-label }}
SLACK_TOKEN: ${{ inputs.slack-token }}
GITHUB_TOKEN: ${{ inputs.github-token }}
STAGE: ${{ inputs.stage }}
REF_URL: ${{ inputs.ref-url }}
IDENTITY_API_URL: ${{ inputs.identity-api-url }}
MONITORING_WINDOW: ${{ inputs.monitoring-window }}
MAX_AUTHORS: ${{ inputs.max-authors }}
DRY_RUN: ${{ inputs.dry-run }}
STRICT: ${{ inputs.strict }}
run: |
"${{ github.action_path }}/notify-deploy-authors.sh"
204 changes: 204 additions & 0 deletions .github/actions/notify-deploy-authors/notify-deploy-authors.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,204 @@
#!/usr/bin/env bash
#
# DM the authors of everything in a deploy, telling them their change is live and pointing them at
# the dashboard to watch. Run this as a step after the rollout has finished.
#
# Never fails the caller: a monitoring nudge is not worth breaking a deploy over, so every runtime
# problem is a ::warning:: and a zero exit. Set STRICT=true to exit non-zero instead, which is what
# you want while testing the wiring.
#
# Needs: bash, curl, jq. No associative arrays, so this also runs on the bash 3.2 that ships with
# macOS - handy for testing by hand before wiring it up.
#
set -uo pipefail

STRICT="${STRICT:-false}"
DRY_RUN="${DRY_RUN:-false}"
STAGE="${STAGE:-}"
IDENTITY_API_URL="${IDENTITY_API_URL:-}"
DASHBOARD_LABEL="${DASHBOARD_LABEL:-Error Dashboard}"
MONITORING_WINDOW="${MONITORING_WINDOW:-the next 30 minutes}"
MAX_AUTHORS="${MAX_AUTHORS:-15}"

# Anything we can't do is a warning, not a failed deploy. Note there is deliberately no `set -e`
# or ERR trap: an ERR trap fires on every non-zero command, including the many ordinary
# "grep matched nothing" cases below, which would abort the run halfway through.
give_up() {
echo "::warning::notify-deploy-authors: $1"
[ "$STRICT" = "true" ] && exit 1
exit 0
}

for required in GITHUB_REPOSITORY GITHUB_TOKEN CURRENT_REF DASHBOARD_URL SERVICE_NAME; do
eval "value=\${$required:-}"
[ -n "$value" ] || give_up "missing required input: $required"
done
if [ "$DRY_RUN" != "true" ] && [ -z "${SLACK_TOKEN:-}" ]; then
give_up "missing required input: SLACK_TOKEN"
fi

# A first deploy has nothing to compare against, which is not an error.
[ -n "${PREVIOUS_REF:-}" ] || give_up "no previous ref given - nothing to compare, skipping"
[ "$PREVIOUS_REF" != "$CURRENT_REF" ] || give_up "previous and current ref are identical - skipping"

US=$(printf '\037')

gh_api() { curl -sSf -H "Authorization: Bearer $GITHUB_TOKEN" -H "Accept: application/vnd.github+json" "$@"; }

slack_api() {
method="$1"; shift
curl -sS -H "Authorization: Bearer $SLACK_TOKEN" -H "Content-type: application/json; charset=utf-8" \
"https://slack.com/api/$method" "$@"
}

# --- who is in this deploy --------------------------------------------------------------------
# The compare API gives every commit between the two refs, which is all we need: commit authors are
# logins, and Co-authored-by trailers give us the people GitHub doesn't attribute.
compare_json="$(gh_api "https://api.github.com/repos/$GITHUB_REPOSITORY/compare/$PREVIOUS_REF...$CURRENT_REF?per_page=250")" ||
give_up "could not compare $PREVIOUS_REF...$CURRENT_REF (do both refs exist?)"

# Bots never need to monitor anything. `[bot]` catches the GitHub App suffix; the identity
# resolver's isBot flag catches the rest further down.
logins="$(jq -r '[.commits[].author.login | select(. != null)] | unique | .[]' <<<"$compare_json" |
grep -v '\[bot\]$' || true)"

coauthor_emails="$(jq -r '.commits[].commit.message' <<<"$compare_json" |
grep -iEo 'co-authored-by:[[:space:]]*[^<]*<[^>]+>' |
sed -E 's/.*<([^>]+)>.*/\1/' |
grep -viE 'users\.noreply\.github\.com|noreply@' | sort -u || true)"

if [ -z "$logins" ] && [ -z "$coauthor_emails" ]; then
echo "No human authors between $PREVIOUS_REF and $CURRENT_REF - nothing to do."
exit 0
fi

author_count=$(printf '%s\n%s\n' "$logins" "$coauthor_emails" | grep -cve '^$' || true)
echo "Authors in this deploy ($author_count): $(tr '\n' ' ' <<<"$logins") $(tr '\n' ' ' <<<"$coauthor_emails")"

# A range far bigger than a normal deploy usually means the refs are wrong. Better to say so than
# to DM half the company.
if [ "$author_count" -gt "$MAX_AUTHORS" ]; then
give_up "$author_count authors exceeds MAX_AUTHORS=$MAX_AUTHORS - refusing to DM. Check PREVIOUS_REF/CURRENT_REF."
fi

# --- resolve them to Slack --------------------------------------------------------------------
# A GitHub login alone does not reach anyone in Slack: public profile emails are set on a small
# minority of the org, and commit emails are often noreply or personal addresses. The identity
# resolver knows each person's work email, which is the address Slack has.
#
# Records are: key, email, slackUserId, displayName - delimited by a unit separator rather than a
# tab, because tab counts as IFS whitespace and `read` would collapse runs of empty fields,
# shifting everything after a blank one into the wrong slot.
people=""
resolved=""

if [ -n "$IDENTITY_API_URL" ]; then
query=""
[ -n "$logins" ] && query="github=$(paste -sd, - <<<"$logins")"
[ -n "$coauthor_emails" ] && query="${query:+$query&}email=$(paste -sd, - <<<"$coauthor_emails")"

# Optional and best-effort: off the tailnet this fails and we fall back to GitHub data.
if identity_json="$(curl -sSf --max-time 15 "$IDENTITY_API_URL/api/identity/resolve?$query" 2>/dev/null)"; then
# personId as the key means one person reached once, even when they turn up both as a commit
# author and as a co-author email. The resolver flags bots and leavers rather than filtering
# them; we want neither.
people="$(jq -r '
[.github // {}, .email // {}] | add | to_entries[]
| select(.value != null)
| select(.value.isBot != true and .value.isActive != false)
| [(.value.personId // .key), (.value.email // ""), (.value.slackUserId // ""), (.value.displayName // .key)]
| join("\u001f")' <<<"$identity_json" | sort -u)"

resolved="$(jq -r '[.github // {}, .email // {}] | add | to_entries[]
| select(.value != null) | .key' <<<"$identity_json" | tr '[:upper:]' '[:lower:]' | sort -u)"
else
echo "::warning::Identity resolver unreachable at $IDENTITY_API_URL - falling back to GitHub emails, which reach far fewer people"
fi
fi

is_resolved() { [ -n "$resolved" ] && grep -qxF "$(tr '[:upper:]' '[:lower:]' <<<"$1")" <<<"$resolved"; }

# Fall back to the public GitHub profile email for anyone the resolver didn't cover.
while read -r login; do
[ -z "$login" ] && continue
is_resolved "$login" && continue
profile_email="$(gh_api "https://api.github.com/users/$login" | jq -r '.email // empty' || true)"
people="$people
$(printf '%s%s%s%s%s%s' "$login" "$US" "$profile_email" "$US" "$US" "$login")"
done <<<"$logins"

# A co-author trailer email the resolver didn't know is still worth a direct Slack lookup.
if [ -n "$coauthor_emails" ]; then
while read -r email; do
[ -z "$email" ] && continue
is_resolved "$email" && continue
people="$people
$(printf '%s%s%s%s%s%s' "$email" "$US" "$email" "$US" "$US" "$email")"
done <<<"$coauthor_emails"
fi

# Drop blanks, then drop repeats of an address we have already queued under another key.
people="$(grep -v '^[[:space:]]*$' <<<"$people" | sort -u |
awk -F"$US" '{ if ($2 == "" || !seen[$2]++) print }')"

# --- the message ------------------------------------------------------------------------------
stage_text=""
[ -n "$STAGE" ] && stage_text=" to $(tr '[:lower:]' '[:upper:]' <<<"${STAGE:0:1}")${STAGE:1}"

ref_link="$CURRENT_REF"
[ -n "${REF_URL:-}" ] && ref_link="<$REF_URL|$CURRENT_REF>"

headline=":rocket: *Your changes are live* - $SERVICE_NAME $ref_link is now deployed${stage_text}."
ask="Please keep an eye out for errors over $MONITORING_WINDOW:
β€’ <$DASHBOARD_URL|$DASHBOARD_LABEL>"
fallback="Your changes are live in $SERVICE_NAME $CURRENT_REF - please monitor for errors"

sent=0
unreachable=""
while IFS="$US" read -r key email slack_id name; do
[ -z "${key:-}" ] && continue

if [ -z "$slack_id" ]; then
if [ -z "$email" ]; then
unreachable="$unreachable $name"
continue
fi
if [ "$DRY_RUN" = "true" ]; then
slack_id="(would look up $email)"
else
lookup="$(slack_api "users.lookupByEmail?email=$email")"
slack_id="$(jq -r '.user.id // empty' <<<"$lookup")"
if [ -z "$slack_id" ]; then
echo "::warning::No Slack account for $name <$email>: $(jq -r '.error // "unknown"' <<<"$lookup")"
unreachable="$unreachable $name"
continue
fi
fi
fi

payload="$(jq -nc --arg ch "$slack_id" --arg text "$fallback" --arg h "$headline" --arg a "$ask" '{
channel: $ch, text: $text,
blocks: [
{type:"section", text:{type:"mrkdwn", text:$h}},
{type:"section", text:{type:"mrkdwn", text:$a}}
]}')"

if [ "$DRY_RUN" = "true" ]; then
echo "--- would DM $name [$slack_id] ---"
jq . <<<"$payload"
else
# Posting to a user id opens (or reuses) the DM with them.
result="$(slack_api chat.postMessage -d "$payload")"
if [ "$(jq -r '.ok' <<<"$result")" = "true" ]; then
echo "DM sent to $name"
sent=$((sent + 1))
else
echo "::warning::DM to $name failed: $(jq -r '.error' <<<"$result")"
fi
fi
done <<<"$people"

echo "Done. DMs sent: $sent. Unreachable:${unreachable:- none}"
if [ -n "$unreachable" ]; then
echo "::warning::Could not reach in Slack:$unreachable. Set identity-api-url (and give the runner tailnet access) so authors resolve without a public GitHub email."
fi
Loading
Loading