myep-team/docs/statistics.md

# Statistics: event collection and evaluation

Reference for how statistics are collected, what is stored, and how to read or extend it.
Describes the behaviour as implemented — not a plan. Evaluation screens for these metrics do
not exist yet; this is what they will be built on.

The governing rule, from which the rest follows:

> **Columns are for what you `GROUP BY`. The payload is for what you read back.**
>
> A value that will end up in a `GROUP BY` needs its own indexed column. Everything else —
> old and new values, reasons, scopes, flags — lives in the JSON `payload`. That split is
> what lets a new metric ship without a migration, and it is the only rule that has to hold
> as the table grows.

Two corollaries that surprise people:

- **Rows are never updated or deleted.** A statistics event is a record of something that
  happened, not a view of current state. If the answer can be recomputed from live data at
  any time, it does not belong here — see *What does not belong* below.
- **Nothing here is a foreign key.** `assignment_id` and friends are plain integers, so
  statistics outlive the rows they describe. Applications are hard deleted; their history is
  not.

---

## Why the table exists

Three of the four metrics could not be answered from live data:

- `Assignment::$jobProfile` holds only the current value. Every edit destroyed the previous
  one permanently.
- Applications are hard deleted (no soft delete), so any count over them silently shrank
  over time.
- Call-offs persisted, but not *when* they happened — `updatedAt` is overwritten by any
  later edit — nor at what scope.

The `log` table is not an alternative. It is a monolog dump (`App\Logging\DatabaseHandler`)
with free-form `context`/`extra`, excluded from migrations by
`schema_filter: ~^(?!log)~` in `config/packages/doctrine.yaml`. It is an audit trail for
humans to read, not a schema to aggregate over.

---

## The table

`statistics_event`, created by `migrations/Version20260826000000.php`, mapped by
`src/Entity/StatisticsEvent.php`.

| Column | Type | Purpose |
|--------|------|---------|
| `id` | int, PK | |
| `name` | varchar(64) | the metric, `enumType: StatisticsEventName` |
| `occurred_at` | datetime | when it happened |
| `actor_id` | int, null | `User::getId()` of whoever acted |
| `actor_label` | varchar(180), null | their `getUserIdentifier()` |
| `actor_role` | varchar(32), null | `enumType: StatisticsActorRole` |
| `assignment_id` | int, null | dimension |
| `disposition_id` | int, null | dimension |
| `application_id` | int, null | dimension |
| `teamer_id` | int, null | dimension |
| `destination_id` | int, null | dimension |
| `job_profile_id` | int, null | dimension |
| `hotel_code` | varchar(8), null | **normalized** code, frozen at write time |
| `payload` | json | everything metric-specific |

Indexed on `(name, occurred_at)`, `(name, hotel_code)`, `(assignment_id)`, `(teamer_id)`.

### Two traps in the columns

**`hotel_code` is normalized here and raw in `feedback`.** This table stores `ALB`;
`feedback.hotel_code` stores `SERALB` and normalizes at query time. Same column name,
opposite conventions — comparing or joining them directly returns nothing, silently. The
`SER`-stripping rule lives in `Destination::getHotelCodeNormalized()`
(`src/Entity/Destination.php`), and is reimplemented in DQL in `DispositionRepository` and
in raw SQL in `FeedbackRepository`. Reuse the entity method in PHP; do not add a fourth copy.

**Dimensions are frozen at write time, deliberately.** A later correction to a destination's
hotel code does not rewrite past periods. This is the same pattern `Feedback::fromAssignment()`
(`src/Entity/Feedback.php`) already uses.

---

## The four metrics

Defined in `src/Enum/StatisticsEventName.php`. Adding a case here plus one collector that
records it is the entire cost of a new metric.

| Name | Recorded when | Payload |
|------|---------------|---------|
| `assignment.job_profile_changed` | the job profile column changes on any write path | `from`, `to` (each `{id, name}` or null) |
| `assignment.called_off` | an assignment's status becomes `called_off` | `dispositions_affected`, `previous_status` |
| `disposition.called_off` | a disposition's status becomes `called_off` | `called_off_by`, `scope`, `reason`, `previous_status` |
| `application.created` | an application is persisted, by any path | — |

Which dimensions each one fills:

| Metric | assignment | destination | job_profile | hotel_code | teamer | disposition | application |
|--------|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| `assignment.job_profile_changed` | ✓ | ✓ | ✓ | ✓ | — | — | — |
| `assignment.called_off` | ✓ | ✓ | ✓ | ✓ | — | — | — |
| `disposition.called_off` | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | — |
| `application.created` | ✓ | ✓ | ✓ | ✓ | ✓ | — | ✓ |

### Call-offs: three different "who" and "how far"

This is the part most likely to be misread when building a chart.

- **`actor_role`** — who *clicked*, from the security token. Always `admin` for a call-off;
  only administrative users can perform one.
- **`payload.called_off_by`** — whose *decision* it was, `teamer` or `office`, as declared by
  an admin on the call-off form (`src/Form/DispositionCallOffType.php`). There is no
  teamer-initiated call-off route; a teamer's decision is recorded on their behalf.
- **`payload.scope`** — how far it *reached*, `assignment` or `disposition`
  (`src/Enum/CallOffScope.php`). The office can cancel one placement just as a
  whole trip can fall away, so `called_off_by` does not imply this.

Cancelling a trip cascades onto every disposition on it in one flush, so **one office
decision produces one `assignment.called_off` row plus N `disposition.called_off` rows**.
Counting the disposition rows therefore counts *affected placements*, not decisions. To count
decisions:

```sql
-- decisions
SELECT COUNT(*) FROM statistics_event WHERE name = 'assignment.called_off';
SELECT COUNT(*) FROM statistics_event
 WHERE name = 'disposition.called_off'
   AND JSON_UNQUOTE(JSON_EXTRACT(payload,'$.scope')) = 'disposition';

-- affected placements
SELECT COUNT(*) FROM statistics_event WHERE name = 'disposition.called_off';
```

`assignment.called_off` carries `dispositions_affected` so a cancelled trip's cost reads off
a single row. It is frequently `0` — a trip cancelled before anyone was staffed is still a
decision worth counting.

---

## How events are recorded

One mechanism, writing through `App\Service\Statistics\StatisticsRecorder`.

**Doctrine changesets** — `src/EventListener/StatisticsChangeSetListener.php`, on `onFlush`
and `postFlush`. Used for state changes. `onFlush` is the only place the *previous* value
still exists, and it catches every write path — edit form, duplication, imports, console —
rather than the one route someone remembered to hook. Recording happens in `postFlush`, after
commit, so a rolled-back flush leaves no phantom statistics: the batch belongs to one flush,
is cleared at the start of the next, and goes away with the service reset that a worker does
between messages.

The listener itself knows nothing about any metric. It owns only the timing, and delegates
to **collectors** — implementations of
`App\Service\Statistics\Collector\StatisticsCollectorInterface`, autoconfigured by interface
so a new one needs no wiring and cannot be silently forgotten.

A collector receives the whole flush (`StatisticsFlush`), not one entity at a time. That is
deliberate: `CallOffCollector` has to see which *assignments* are becoming `called_off`
before it can tell a cascaded disposition from a teamer dropping out, and it counts the
cascade to fill `dispositions_affected`. Neither is expressible from a single entity's
changeset, and that distinction exists only inside the flush.

**Group collectors by concern, not by metric.** Both call-off metrics live in one collector
because they can only be told apart together; a job profile change shares nothing with them
and lives in its own. Order between collectors is irrelevant and nothing may depend on it.

Collectors see **insertions** as well as updates, so a metric about something being created
is a collector too. `ApplicationCreatedCollector` reads `insertionsOf(Application::class)`
rather than subscribing to `ApplicationCreatedEvent`, which is what an earlier version did.
Reading the flush counts every path that persists an application; subscribing counted only
the paths that remember to dispatch, and would have silently missed an admin applying on
someone's behalf, an import, or a fixture.

A newly inserted entity has no id during `onFlush`. That is fine — Doctrine assigns it during
the commit, and recording happens afterwards in `postFlush`.

> **There is deliberately only one mechanism.** If a future metric is not a persisted state
> change at all — an email sent, a login, a document downloaded — call `StatisticsRecorder`
> directly from wherever that happens. Do not reintroduce a second listener layer for it.

**Writes go through raw DBAL**, not the ORM, so recording never touches the UnitOfWork — a
listener running inside a flush would otherwise trigger a nested flush. Failures are caught
and reported to the audit logger: statistics are a side effect and must never break the
business action that produced them.

---

## Reading the data

`src/Repository/StatisticsEventRepository.php`.

```php
public function countGroupedBy(
    StatisticsEventName $name,
    string $dimension,                        // whitelisted
    ?\DateTimeImmutable $dateFrom = null,
    ?\DateTimeImmutable $dateTo = null,
    StatisticsDateBasis $dateBasis = StatisticsDateBasis::SEASON,
): array
```

`getCountGroupedByQuery()` returns the same query unexecuted, which is how its semantics are
pinned in tests without a database.

### A period means the season, not when the row was written

`StatisticsDateBasis::SEASON` (the default) filters on the assignment's dates, falling back to
the destination's — the translation of `Assignment::getEffectivePeriod()` into DQL, which
lives in `src/Repository/Filter/SeasonPeriodFilter.php` and is shared with
`DispositionRepository`. This matches every other statistics screen and
`StatisticsFilterType`. Change what a season means there, once, not per screen.

`StatisticsDateBasis::OCCURRENCE` filters `occurred_at` instead. Reach for it deliberately,
for genuine time series such as call-offs per month.

The two routinely disagree, because an application for the coming winter arrives months
before the season it belongs to. Picking the wrong one does not fail — it quietly answers a
different question from the screen next to it.

`SEASON` reaches the assignment through an arbitrary join and therefore excludes events with
no `assignment_id`. All four metrics set it.

### Crossing with live data

The plain-integer dimension columns cost nothing in queryability — Doctrine's arbitrary join
works:

```
SELECT destination.hotel AS hotel, COUNT(event.id) AS applications
  FROM App\Entity\StatisticsEvent event
  JOIN App\Entity\Assignment assignment WITH assignment.id = event.assignmentId
  JOIN assignment.destination destination
 WHERE event.name = :name
 GROUP BY hotel
```

Group by `hotel_code`, though, not by the hotel *name*: they are not one-to-one. One
normalized code can span several destinations with different names.

---

## Backfilling

`bin/console app:statistics:backfill` seeds what current data still contains. Idempotent —
it skips subjects already recorded — and supports `--dry-run`. It is never run automatically:
nothing in `CronCommand`, the scheduler, or any deploy script invokes it.

It writes only to `statistics_event`, so it is safe to run against production at any time.

| Metric | Backfillable | `occurred_at` from |
|--------|--------------|--------------------|
| `application.created` | yes | `createdAt` — exact |
| `disposition.called_off` | yes | `updatedAt` — **approximate** |
| `assignment.called_off` | yes | `updatedAt` — **approximate** |
| `assignment.job_profile_changed` | **no** | history does not exist |

Every seeded row carries `"backfilled": true`; the approximate ones also carry
`"approximate_date": true`. So the choice is per chart, never table-wide:

```sql
WHERE JSON_EXTRACT(payload,'$.backfilled') IS NULL   -- organically recorded only
```

Three limits on seeded rows, all of which matter when charting:

- **Dates from `updatedAt` are a proxy**, since any later edit overwrites it. Exclude those
  rows from time series; keep them for totals.
- **`actor_role` is `system`.** The command runs on the CLI with no security token, so the
  original actor is not recoverable.
- **Frozen dimensions are frozen at *backfill* time**, not at the time of the event, and
  `scope` is left unset because it cannot be established retroactively.

---

## Extending it

The cost depends entirely on whether the new metric needs a new *dimension* or just a new
*name*. Almost all need only a name.

**No migration — a new kind of event.** Add a case to `StatisticsEventName`, then one
`StatisticsCollectorInterface` implementation that records it. Metric-specific detail goes in
`payload`. `StatisticsChangeSetListener` is never edited. This covers most requests —
application rejections, contract upload delays, feedback submitted, any status transition,
anything created or deleted.

Reuse an existing collector when the new metric is part of a concern already covered —
especially when it needs to see the same entities in the same flush. Write a new one when it
is genuinely independent.

**One migration — a new dimension to slice by.** Only when you need to group by something no
column carries (a season, a country, a fee band). Add the column and an index, fill it in
`StatisticsRecorder`, and accept `NULL` on older rows. Resist when the dimension is already
reachable through a join.

**Migration plus backfill — history for something not yet recorded.** Extend
`StatisticsBackfillCommand` if current data still holds the answer. If it has been
overwritten, as with job profiles, the metric simply starts the day it ships.

---

## What does not belong here

**Anything recomputable from live data that cannot be lost.** Feedback statistics
(`src/Controller/Administrative/Statistics/`) are the worked example: `Feedback` already
freezes its own dimensions at creation, is never deleted, and its aggregations average a
per-question `ratings` array. It is already an append-only record; copying it into
`statistics_event` would duplicate every row and lose the structure the charts need. This
table counts occurrences — it has no shape for averaging structured arrays.

**Current state.** "How many applications are open right now" is a question for the
`application` table. This table answers "how many were ever received", which is a different
and equally valid question — just not the same one.

**An audit trail for humans.** That is the `log` table and `/admin/log`.

---

## Where things live

| Concern | Path |
|---------|------|
| Entity | `src/Entity/StatisticsEvent.php` |
| Repository | `src/Repository/StatisticsEventRepository.php` |
| Enums | `src/Enum/Statistics*.php` |
| Write path | `src/Service/Statistics/StatisticsRecorder.php` |
| Flush orchestration | `src/EventListener/StatisticsChangeSetListener.php` |
| Collectors (one per concern) | `src/Service/Statistics/Collector/` |
| Flush context passed to collectors | `src/Service/Statistics/StatisticsFlush.php` |
| What a period means, shared with the disposition statistics | `src/Repository/Filter/SeasonPeriodFilter.php` |
| Backfill | `src/Command/StatisticsBackfillCommand.php` |
| Migration | `migrations/Version20260826000000.php` |
| Tests | `tests/EventListener/`, `tests/Repository/`, `tests/Service/Statistics/` |

Existing statistics screens, for the filter/handler pattern a future evaluation UI should
reuse: `src/Controller/Administrative/Statistics/`,
`src/Service/Common/StatisticsFilterHandler.php`, `src/Model/StatisticsFilterDto.php`,
`src/Form/StatisticsFilterType.php`.