myep-team/docs/operations.md

Operations: messenger workers and cron

Reference for how queued work actually gets executed in production. Describes the setup as it is — not a plan.

The governing fact, from which most of the rest follows:

Nothing in this repository starts a worker. Both workers are crontab entries on the host, outside version control. deploy.php only ever stops workers, and even that is largely ineffective (see below). If you change how a queue is consumed, change it here too, or the next person has to reverse-engineer it from the config.

---

Transports

Defined in config/packages/messenger.yaml, all backed by the same Doctrine table (MESSENGER_TRANSPORT_DSN=doctrine://default?auto_setup=0):

Transport	queue_name	Carries	Consumed by
async	default	all ordinary mail, chat and SMS notifications	the async cron worker
mailing	mailing	bulk teamer mailings only	the mailing cron worker
failed	failed	anything that exhausted its retries	nobody — inspected by hand
sync	—	dev and test only	the request itself

The two live workers are isolated by the queue_name column, not by how they are started: the Doctrine transport's get() filters on it, so neither worker can see the other's messages. Ordinary mail therefore never queues behind a mailing of several hundred recipients.

Overlapping runs of the same worker are harmless. Connection::get() selects FOR UPDATE SKIP LOCKED inside a transaction and stamps delivered_at before committing, so two consumers can never be handed the same row. No flock is needed; an overlap only costs a second process.

The cron entries

The async line below is a placeholder. Read the real one off the host (crontab -l as p704161) and transcribe it verbatim, including any differences in flags, php path or logging — those differences are information.

# ordinary mail, notifications
*/15 * * * * cd /home/www/p704161/html/myep-team/current && /usr/local/bin/php bin/console messenger:consume async --time-limit=... --memory-limit=... --quiet >> /home/www/p704161/html/myep-team/shared/var/log/cron-messenger-async.log 2>&1

# bulk teamer mailings
*/15 * * * * cd /home/www/p704161/html/myep-team/current && /usr/local/bin/php bin/console messenger:consume mailing --time-limit=300 --memory-limit=128M --quiet >> /home/www/p704161/html/myep-team/shared/var/log/cron-messenger-mailing.log 2>&1

Why each part is the way it is:

• /usr/local/bin/php, not bare php. deploy.php sets bin/php to this path explicitly for both hosts, because the host's default CLI is not the right one. Cron's PATH is typically only /usr/bin:/bin. vendor/ is rsynced from a developer machine and there is no deploy:vendors, so the CLI must match the PHP the dependencies were installed against (8.3).
• --time-limit=300 on the mailing worker. An 800-recipient mailing needs roughly 200–360 seconds of worker time — the fan-out renders 800 Twig bodies and inserts 800 rows, then the mails go out at about 3–5 per second over a reused SMTP connection. So a mailing usually drains in one window and occasionally spills into the next; worst case end to end is about half an hour. Raise it toward 840 for lower latency at the cost of an effectively always-on process, lower it for a lighter footprint. Anything under 900 guarantees two runs cannot overlap.
• Stopping mid-mailing is safe. The time limit is only checked between messages (StopWorkerOnTimeLimitListener listens on WorkerRunningEvent), so it never truncates a send or a half-finished fan-out. Unhandled rows keep delivered_at = NULL and the next run picks them up.
• --memory-limit is a recycle trigger, not a safety net. It too is only checked between messages, so it cannot protect the fan-out, which is the memory peak. That is down to the CLI memory_limit ini — keep it at 256M or more.
• --quiet plus a redirect. messenger:consume prints a banner on every run and unredirected cron output is mailed to the crontab user, which would be 96 mails a day. Nothing is lost: errors still reach var/log/framework.prod.log through the prod monolog handler. Redirect into the shared var/log, which is in shared_dirs and survives deploys — never anywhere under var/cache, which is per-release by design.
• No APP_ENV=prod on the line. bin/console loads Dotenv relative to the project dir, and APP_ENV lives in the shared .env.local. A real environment variable takes precedence over that file, so putting it in the crontab would create a second source of truth and would need a different line for staging, which runs on the same host under the same user.

Deploys do not restart workers

deploy.php ends with deploy:stop-workers, which runs messenger:stop-workers. That writes a restart timestamp into a cache pool under %kernel.cache_dir%, i.e. into the new release. A worker that is already running was started from the previous release and stays pinned to that release's var/cache for its whole life, so it never sees the flag.

--time-limit is therefore what actually bounds a worker's life and what makes new code take effect — up to five minutes after a deploy for the mailing worker. This is not worth "fixing" by moving var/cache into shared_dirs; that would break cache warming and opcache invalidation far worse than the problem it solves.

Related: keep_releases: 3. A worker must not outlive three deploys, or it will be running out of a directory that has been deleted. At a five-minute lifetime this cannot happen.

When something goes wrong

bin/console messenger:stats                      # queue depths, including failed
bin/console messenger:failed:show                # what failed and why
bin/console messenger:failed:show <id> -vv       # the full exception
bin/console messenger:failed:retry               # requeue, interactively

Nothing watches the failed transport, so a failed mailing stays silent until somebody looks. Check it after any large mailing.

The scheduler's email_on_failure does not help here, and covers less than it appears to: a task only counts as failed when its command exits non-zero, and App\Command\CronCommand always returns Command::SUCCESS — even where it renders $io->error() for a failed log flush. So none of the five services it orchestrates can currently raise an alert.

Three failure modes worth knowing because they are quiet rather than loud:

• .env.local missing from the deploy path. APP_ENV falls back to dev and MAILING_MAILER_DSN to null://null. The worker then consumes every message and delivers nothing, reporting success and recording no failure. The quietest failure in the whole system.
• No worker consuming mailing. Mailings queue up indefinitely while the admin is told they are on their way. messenger:stats shows it immediately.
• Wrong PHP CLI. A version mismatch against the rsynced vendor/ fails at parse time, inside a cron job nobody reads. This is why the log redirect exists.

Scheduled tasks are a different mechanism

config/packages/zenstruck_schedule.yaml holds the daily jobs (app:cron, app:bpn-import, app:teamer-status, …), driven by a separate schedule:run cron entry.

Do not put messenger:consume in there. The bundle runs tasks in-process and sequentially, so a consume task holds the entire schedule for its duration — every later task waits, and the next schedule:run starts concurrently because no task declares withoutOverlapping. This was tried and reverted (dc796849, "remove messenger from scheduler config to be run separately"). Workers belong in their own crontab lines.