141 lines
8.1 KiB
Markdown
141 lines
8.1 KiB
Markdown
# Operations: messenger workers and cron
|
||
|
||
Reference for how queued work actually gets executed in production. Describes the setup as
|
||
it is — not a plan.
|
||
|
||
The governing fact, from which most of the rest follows:
|
||
|
||
> **Nothing in this repository starts a worker.** Both workers are crontab entries on the
|
||
> host, outside version control. `deploy.php` only ever *stops* workers, and even that is
|
||
> largely ineffective (see below). If you change how a queue is consumed, change it here
|
||
> too, or the next person has to reverse-engineer it from the config.
|
||
|
||
---
|
||
|
||
## Transports
|
||
|
||
Defined in `config/packages/messenger.yaml`, all backed by the same Doctrine table
|
||
(`MESSENGER_TRANSPORT_DSN=doctrine://default?auto_setup=0`):
|
||
|
||
| Transport | `queue_name` | Carries | Consumed by |
|
||
|-----------|--------------|---------|-------------|
|
||
| `async` | `default` | all ordinary mail, chat and SMS notifications | the async cron worker |
|
||
| `mailing` | `mailing` | bulk teamer mailings only | the mailing cron worker |
|
||
| `failed` | `failed` | anything that exhausted its retries | nobody — inspected by hand |
|
||
| `sync` | — | dev and test only | the request itself |
|
||
|
||
The two live workers are isolated **by the `queue_name` column**, not by how they are
|
||
started: the Doctrine transport's `get()` filters on it, so neither worker can see the
|
||
other's messages. Ordinary mail therefore never queues behind a mailing of several hundred
|
||
recipients.
|
||
|
||
Overlapping runs of the same worker are harmless. `Connection::get()` selects
|
||
`FOR UPDATE SKIP LOCKED` inside a transaction and stamps `delivered_at` before committing,
|
||
so two consumers can never be handed the same row. No `flock` is needed; an overlap only
|
||
costs a second process.
|
||
|
||
## The cron entries
|
||
|
||
> **The async line below is a placeholder.** Read the real one off the host
|
||
> (`crontab -l` as `p704161`) and transcribe it verbatim, including any differences in
|
||
> flags, php path or logging — those differences are information.
|
||
|
||
```cron
|
||
# ordinary mail, notifications
|
||
*/15 * * * * cd /home/www/p704161/html/myep-team/current && /usr/local/bin/php bin/console messenger:consume async --time-limit=... --memory-limit=... --quiet >> /home/www/p704161/html/myep-team/shared/var/log/cron-messenger-async.log 2>&1
|
||
|
||
# bulk teamer mailings
|
||
*/15 * * * * cd /home/www/p704161/html/myep-team/current && /usr/local/bin/php bin/console messenger:consume mailing --time-limit=300 --memory-limit=128M --quiet >> /home/www/p704161/html/myep-team/shared/var/log/cron-messenger-mailing.log 2>&1
|
||
```
|
||
|
||
Why each part is the way it is:
|
||
|
||
- **`/usr/local/bin/php`, not bare `php`.** `deploy.php` sets `bin/php` to this path
|
||
explicitly for both hosts, because the host's default CLI is not the right one. Cron's
|
||
`PATH` is typically only `/usr/bin:/bin`. `vendor/` is rsynced from a developer machine
|
||
and there is no `deploy:vendors`, so the CLI must match the PHP the dependencies were
|
||
installed against (8.4). Deployments run `composer check-platform-reqs` locally before
|
||
rsync; because `config.platform.php` is pinned in `composer.json`, this only verifies
|
||
that the locked dependencies resolve against the PHP and extension set we target — it
|
||
does not probe the staging or production runtime. Keeping both FPM and
|
||
`/usr/local/bin/php` on staging and production at PHP 8.4 is a manual precondition of
|
||
every deploy.
|
||
- **`--time-limit=300`** on the mailing worker. An 800-recipient mailing needs roughly
|
||
200–360 seconds of worker time — the fan-out renders 800 Twig bodies and inserts 800
|
||
rows, then the mails go out at about 3–5 per second over a reused SMTP connection. So a
|
||
mailing usually drains in one window and occasionally spills into the next; worst case
|
||
end to end is about half an hour. Raise it toward 840 for lower latency at the cost of an
|
||
effectively always-on process, lower it for a lighter footprint. Anything under 900
|
||
guarantees two runs cannot overlap.
|
||
- **Stopping mid-mailing is safe.** The time limit is only checked *between* messages
|
||
(`StopWorkerOnTimeLimitListener` listens on `WorkerRunningEvent`), so it never truncates
|
||
a send or a half-finished fan-out. Unhandled rows keep `delivered_at = NULL` and the next
|
||
run picks them up.
|
||
- **`--memory-limit` is a recycle trigger, not a safety net.** It too is only checked
|
||
between messages, so it cannot protect the fan-out, which is the memory peak. That is
|
||
down to the CLI `memory_limit` ini — keep it at 256M or more.
|
||
- **`--quiet` plus a redirect.** `messenger:consume` prints a banner on every run and
|
||
unredirected cron output is mailed to the crontab user, which would be 96 mails a day.
|
||
Nothing is lost: errors still reach `var/log/framework.prod.log` through the prod monolog
|
||
handler. Redirect into the **shared** `var/log`, which is in `shared_dirs` and survives
|
||
deploys — never anywhere under `var/cache`, which is per-release by design.
|
||
- **No `APP_ENV=prod` on the line.** `bin/console` loads Dotenv relative to the project
|
||
dir, and `APP_ENV` lives in the shared `.env.local`. A real environment variable takes
|
||
precedence over that file, so putting it in the crontab would create a second source of
|
||
truth and would need a different line for staging, which runs on the same host under the
|
||
same user.
|
||
|
||
## Deploys do not restart workers
|
||
|
||
`deploy.php` ends with `deploy:stop-workers`, which runs `messenger:stop-workers`. That
|
||
writes a restart timestamp into a cache pool under `%kernel.cache_dir%`, i.e. into the
|
||
**new release**. A worker that is already running was started from the previous release and
|
||
stays pinned to that release's `var/cache` for its whole life, so it never sees the flag.
|
||
|
||
**`--time-limit` is therefore what actually bounds a worker's life and what makes new code
|
||
take effect** — up to five minutes after a deploy for the mailing worker. This is not worth
|
||
"fixing" by moving `var/cache` into `shared_dirs`; that would break cache warming and
|
||
opcache invalidation far worse than the problem it solves.
|
||
|
||
Related: `keep_releases: 3`. A worker must not outlive three deploys, or it will be running
|
||
out of a directory that has been deleted. At a five-minute lifetime this cannot happen.
|
||
|
||
## When something goes wrong
|
||
|
||
```bash
|
||
bin/console messenger:stats # queue depths, including failed
|
||
bin/console messenger:failed:show # what failed and why
|
||
bin/console messenger:failed:show <id> -vv # the full exception
|
||
bin/console messenger:failed:retry # requeue, interactively
|
||
```
|
||
|
||
Nothing watches the `failed` transport, so a failed mailing stays silent until somebody
|
||
looks. Check it after any large mailing.
|
||
|
||
The scheduler's `email_on_failure` does not help here, and covers less than it appears to:
|
||
a task only counts as failed when its command exits non-zero, and `App\Command\CronCommand`
|
||
always returns `Command::SUCCESS` — even where it renders `$io->error()` for a failed log
|
||
flush. So none of the five services it orchestrates can currently raise an alert.
|
||
|
||
Three failure modes worth knowing because they are quiet rather than loud:
|
||
|
||
- **`.env.local` missing from the deploy path.** `APP_ENV` falls back to `dev` and
|
||
`MAILING_MAILER_DSN` to `null://null`. The worker then consumes every message and
|
||
delivers nothing, reporting success and recording no failure. The quietest failure in the
|
||
whole system.
|
||
- **No worker consuming `mailing`.** Mailings queue up indefinitely while the admin is told
|
||
they are on their way. `messenger:stats` shows it immediately.
|
||
- **Wrong PHP CLI.** A version mismatch against the rsynced `vendor/` fails at parse time,
|
||
inside a cron job nobody reads. This is why the log redirect exists.
|
||
|
||
## Scheduled tasks are a different mechanism
|
||
|
||
`config/packages/zenstruck_schedule.yaml` holds the daily jobs (`app:cron`,
|
||
`app:bpn-import`, `app:teamer-status`, …), driven by a separate `schedule:run` cron entry.
|
||
|
||
**Do not put `messenger:consume` in there.** The bundle runs tasks in-process and
|
||
sequentially, so a consume task holds the entire schedule for its duration — every later
|
||
task waits, and the next `schedule:run` starts concurrently because no task declares
|
||
`withoutOverlapping`. This was tried and reverted (`dc796849`, "remove messenger from
|
||
scheduler config to be run separately"). Workers belong in their own crontab lines.
|