myep-team/docs/statistics.md

Statistics: event collection and evaluation

Reference for how statistics are collected, what is stored, and how to read or extend it. Describes the behaviour as implemented — not a plan. Evaluation screens for these metrics do not exist yet; this is what they will be built on.

The governing rule, from which the rest follows:

Columns are for what you GROUP BY. The payload is for what you read back.

A value that will end up in a GROUP BY needs its own indexed column. Everything else — old and new values, reasons, scopes, flags — lives in the JSON payload. That split is what lets a new metric ship without a migration, and it is the only rule that has to hold as the table grows.

Two corollaries that surprise people:

• Rows are never updated or deleted. A statistics event is a record of something that happened, not a view of current state. If the answer can be recomputed from live data at any time, it does not belong here — see What does not belong below.
• Nothing here is a foreign key. assignment_id and friends are plain integers, so statistics outlive the rows they describe. Applications are hard deleted; their history is not.

---

Why the table exists

Three of the four metrics could not be answered from live data:

• Assignment::$jobProfile holds only the current value. Every edit destroyed the previous one permanently.
• Applications are hard deleted (no soft delete), so any count over them silently shrank over time.
• Call-offs persisted, but not when they happened — updatedAt is overwritten by any later edit — nor at what scope.

The log table is not an alternative. It is a monolog dump (App\Logging\DatabaseHandler) with free-form context/extra, excluded from migrations by schema_filter: ~^(?!log)~ in config/packages/doctrine.yaml. It is an audit trail for humans to read, not a schema to aggregate over.

---

The table

statistics_event, created by migrations/Version20260826000000.php, mapped by src/Entity/StatisticsEvent.php.

Column	Type	Purpose
id	int, PK
name	varchar(64)	the metric, enumType: StatisticsEventName
occurred_at	datetime	when it happened
actor_id	int, null	User::getId() of whoever acted
actor_label	varchar(180), null	their getUserIdentifier()
actor_role	varchar(32), null	enumType: StatisticsActorRole
assignment_id	int, null	dimension
disposition_id	int, null	dimension
application_id	int, null	dimension
teamer_id	int, null	dimension
destination_id	int, null	dimension
job_profile_id	int, null	dimension
hotel_code	varchar(8), null	normalized code, frozen at write time
payload	json	everything metric-specific

Indexed on (name, occurred_at), (name, hotel_code), (assignment_id), (teamer_id).

Two traps in the columns

hotel_code is normalized here and raw in feedback. This table stores ALB; feedback.hotel_code stores SERALB and normalizes at query time. Same column name, opposite conventions — comparing or joining them directly returns nothing, silently. The SER-stripping rule lives in Destination::getHotelCodeNormalized() (src/Entity/Destination.php), and is reimplemented in DQL in DispositionRepository and in raw SQL in FeedbackRepository. Reuse the entity method in PHP; do not add a fourth copy.

Dimensions are frozen at write time, deliberately. A later correction to a destination's hotel code does not rewrite past periods. This is the same pattern Feedback::fromAssignment() (src/Entity/Feedback.php) already uses.

---

The four metrics

Defined in src/Enum/StatisticsEventName.php. Adding a case here plus one collector that records it is the entire cost of a new metric.

Name	Recorded when	Payload
assignment.job_profile_changed	the job profile column changes on any write path	from, to (each {id, name} or null)
assignment.called_off	an assignment's status becomes called_off	dispositions_affected, previous_status
disposition.called_off	a disposition's status becomes called_off	called_off_by, scope, reason, previous_status
application.created	an application is persisted, by any path	—

Which dimensions each one fills:

Metric	assignment	destination	job_profile	hotel_code	teamer	disposition	application
assignment.job_profile_changed	✓	✓	✓	✓	—	—	—
assignment.called_off	✓	✓	✓	✓	—	—	—
disposition.called_off	✓	✓	✓	✓	✓	✓	—
application.created	✓	✓	✓	✓	✓	—	✓

Call-offs: three different "who" and "how far"

This is the part most likely to be misread when building a chart.

• actor_role — who clicked, from the security token. Always admin for a call-off; only administrative users can perform one.
• payload.called_off_by — whose decision it was, teamer or office, as declared by an admin on the call-off form (src/Form/DispositionCallOffType.php). There is no teamer-initiated call-off route; a teamer's decision is recorded on their behalf.
• payload.scope — how far it reached, assignment or disposition (src/Enum/CallOffScope.php). The office can cancel one placement just as a whole trip can fall away, so called_off_by does not imply this.

Cancelling a trip cascades onto every disposition on it in one flush, so one office decision produces one assignment.called_off row plus N disposition.called_off rows. Counting the disposition rows therefore counts affected placements, not decisions. To count decisions:

-- decisions
SELECT COUNT(*) FROM statistics_event WHERE name = 'assignment.called_off';
SELECT COUNT(*) FROM statistics_event
 WHERE name = 'disposition.called_off'
   AND JSON_UNQUOTE(JSON_EXTRACT(payload,'$.scope')) = 'disposition';

-- affected placements
SELECT COUNT(*) FROM statistics_event WHERE name = 'disposition.called_off';

assignment.called_off carries dispositions_affected so a cancelled trip's cost reads off a single row. It is frequently 0 — a trip cancelled before anyone was staffed is still a decision worth counting.

---

How events are recorded

One mechanism, writing through App\Service\Statistics\StatisticsRecorder.

Doctrine changesets — src/EventListener/StatisticsChangeSetListener.php, on onFlush and postFlush. Used for state changes. onFlush is the only place the previous value still exists, and it catches every write path — edit form, duplication, imports, console — rather than the one route someone remembered to hook. Recording happens in postFlush, after commit, so a rolled-back flush leaves no phantom statistics: the batch belongs to one flush, is cleared at the start of the next, and goes away with the service reset that a worker does between messages.

The listener itself knows nothing about any metric. It owns only the timing, and delegates to collectors — implementations of App\Service\Statistics\Collector\StatisticsCollectorInterface, autoconfigured by interface so a new one needs no wiring and cannot be silently forgotten.

A collector receives the whole flush (StatisticsFlush), not one entity at a time. That is deliberate: CallOffCollector has to see which assignments are becoming called_off before it can tell a cascaded disposition from a teamer dropping out, and it counts the cascade to fill dispositions_affected. Neither is expressible from a single entity's changeset, and that distinction exists only inside the flush.

Group collectors by concern, not by metric. Both call-off metrics live in one collector because they can only be told apart together; a job profile change shares nothing with them and lives in its own. Order between collectors is irrelevant and nothing may depend on it.

Collectors see insertions as well as updates, so a metric about something being created is a collector too. ApplicationCreatedCollector reads insertionsOf(Application::class) rather than subscribing to ApplicationCreatedEvent, which is what an earlier version did. Reading the flush counts every path that persists an application; subscribing counted only the paths that remember to dispatch, and would have silently missed an admin applying on someone's behalf, an import, or a fixture.

A newly inserted entity has no id during onFlush. That is fine — Doctrine assigns it during the commit, and recording happens afterwards in postFlush.

There is deliberately only one mechanism. If a future metric is not a persisted state change at all — an email sent, a login, a document downloaded — call StatisticsRecorder directly from wherever that happens. Do not reintroduce a second listener layer for it.

Writes go through raw DBAL, not the ORM, so recording never touches the UnitOfWork — a listener running inside a flush would otherwise trigger a nested flush. Failures are caught and reported to the audit logger: statistics are a side effect and must never break the business action that produced them.

---

Reading the data

src/Repository/StatisticsEventRepository.php.

public function countGroupedBy(
    StatisticsEventName $name,
    string $dimension,                        // whitelisted
    ?\DateTimeImmutable $dateFrom = null,
    ?\DateTimeImmutable $dateTo = null,
    StatisticsDateBasis $dateBasis = StatisticsDateBasis::SEASON,
): array

getCountGroupedByQuery() returns the same query unexecuted, which is how its semantics are pinned in tests without a database.

A period means the season, not when the row was written

StatisticsDateBasis::SEASON (the default) filters on the assignment's dates, falling back to the destination's — the translation of Assignment::getEffectivePeriod() into DQL, which lives in src/Repository/Filter/SeasonPeriodFilter.php and is shared with DispositionRepository. This matches every other statistics screen and StatisticsFilterType. Change what a season means there, once, not per screen.

StatisticsDateBasis::OCCURRENCE filters occurred_at instead. Reach for it deliberately, for genuine time series such as call-offs per month.

The two routinely disagree, because an application for the coming winter arrives months before the season it belongs to. Picking the wrong one does not fail — it quietly answers a different question from the screen next to it.

SEASON reaches the assignment through an arbitrary join and therefore excludes events with no assignment_id. All four metrics set it.

Crossing with live data

The plain-integer dimension columns cost nothing in queryability — Doctrine's arbitrary join works:

SELECT destination.hotel AS hotel, COUNT(event.id) AS applications
  FROM App\Entity\StatisticsEvent event
  JOIN App\Entity\Assignment assignment WITH assignment.id = event.assignmentId
  JOIN assignment.destination destination
 WHERE event.name = :name
 GROUP BY hotel

Group by hotel_code, though, not by the hotel name: they are not one-to-one. One normalized code can span several destinations with different names.

---

Backfilling

bin/console app:statistics:backfill seeds what current data still contains. Idempotent — it skips subjects already recorded — and supports --dry-run. It is never run automatically: nothing in CronCommand, the scheduler, or any deploy script invokes it.

It writes only to statistics_event, so it is safe to run against production at any time.

Metric	Backfillable	occurred_at from
application.created	yes	createdAt — exact
disposition.called_off	yes	updatedAt — approximate
assignment.called_off	yes	updatedAt — approximate
assignment.job_profile_changed	no	history does not exist

Every seeded row carries "backfilled": true; the approximate ones also carry "approximate_date": true. So the choice is per chart, never table-wide:

WHERE JSON_EXTRACT(payload,'$.backfilled') IS NULL   -- organically recorded only

Three limits on seeded rows, all of which matter when charting:

• Dates from updatedAt are a proxy, since any later edit overwrites it. Exclude those rows from time series; keep them for totals.
• actor_role is system. The command runs on the CLI with no security token, so the original actor is not recoverable.
• Frozen dimensions are frozen at backfill time, not at the time of the event, and scope is left unset because it cannot be established retroactively.

---

Extending it

The cost depends entirely on whether the new metric needs a new dimension or just a new name. Almost all need only a name.

No migration — a new kind of event. Add a case to StatisticsEventName, then one StatisticsCollectorInterface implementation that records it. Metric-specific detail goes in payload. StatisticsChangeSetListener is never edited. This covers most requests — application rejections, contract upload delays, feedback submitted, any status transition, anything created or deleted.

Reuse an existing collector when the new metric is part of a concern already covered — especially when it needs to see the same entities in the same flush. Write a new one when it is genuinely independent.

One migration — a new dimension to slice by. Only when you need to group by something no column carries (a season, a country, a fee band). Add the column and an index, fill it in StatisticsRecorder, and accept NULL on older rows. Resist when the dimension is already reachable through a join.

Migration plus backfill — history for something not yet recorded. Extend StatisticsBackfillCommand if current data still holds the answer. If it has been overwritten, as with job profiles, the metric simply starts the day it ships.

---

What does not belong here

Anything recomputable from live data that cannot be lost. Feedback statistics (src/Controller/Administrative/Statistics/) are the worked example: Feedback already freezes its own dimensions at creation, is never deleted, and its aggregations average a per-question ratings array. It is already an append-only record; copying it into statistics_event would duplicate every row and lose the structure the charts need. This table counts occurrences — it has no shape for averaging structured arrays.

Current state. "How many applications are open right now" is a question for the application table. This table answers "how many were ever received", which is a different and equally valid question — just not the same one.

An audit trail for humans. That is the log table and /admin/log.

---

Where things live

Concern	Path
Entity	src/Entity/StatisticsEvent.php
Repository	src/Repository/StatisticsEventRepository.php
Enums	src/Enum/Statistics*.php
Write path	src/Service/Statistics/StatisticsRecorder.php
Flush orchestration	src/EventListener/StatisticsChangeSetListener.php
Collectors (one per concern)	src/Service/Statistics/Collector/
Flush context passed to collectors	src/Service/Statistics/StatisticsFlush.php
What a period means, shared with the disposition statistics	src/Repository/Filter/SeasonPeriodFilter.php
Backfill	src/Command/StatisticsBackfillCommand.php
Migration	migrations/Version20260826000000.php
Tests	tests/EventListener/, tests/Repository/, tests/Service/Statistics/

Existing statistics screens, for the filter/handler pattern a future evaluation UI should reuse: src/Controller/Administrative/Statistics/, src/Service/Common/StatisticsFilterHandler.php, src/Model/StatisticsFilterDto.php, src/Form/StatisticsFilterType.php.