receipity/receipt-parsing-spec.md

# Receipt Parsing Logic Specification

**Implementation guide (how to add a shop):** [`docs/receipt-parsing.md`](docs/receipt-parsing.md)
**Keyword config:** [`receipt-parsing-keywords.json`](receipt-parsing-keywords.json)

Purpose: define rule-based (non-AI, regex/heuristic) logic for turning raw
OCR text lines from a supermarket receipt into structured data: items,
quantities, unit prices, line totals, discounts, and receipt totals.

This spec is based on analysis of real receipts from multiple Dutch stores
(Jumbo, Lidl, Kruidvat, World Toko), which use at least four different line
layouts for the same underlying data. The parser must handle all of them
without assuming a single fixed format.

---

## 1. Input assumptions

- Input is raw OCR text, one line per detected text line, in top-to-bottom
  order as it appeared on the receipt.
- Prices use European format: comma as decimal separator (e.g. `2,89`),
  period as thousands separator if present (rare on receipts this size).
- OCR may introduce noise: misread characters, extra/missing spaces,
  occasional dropped or merged lines. The parser should be tolerant of
  minor spacing irregularities but does not need to do spell-correction.

## 2. Preprocessing (run before classification)

1. Trim leading/trailing whitespace on every line; collapse multiple
   internal spaces to one.
2. Normalize decimal separators: treat `,` as the decimal point in any
   number matching `\d+[,.]\d{2}`. Convert to a standard `.` internally
   for numeric storage, but remember the original format was comma-based
   (useful for locale-aware re-parsing if needed).
3. Drop fully empty lines.
4. Do NOT drop short lines outright (e.g. a lone `BROCCOLI` or a lone
   `2 X 0,95`) — both are meaningful in different receipt layouts.

## 3. Core line classification

Classify every line into one of these categories, in this priority order
(check top-down, first match wins):

### 3.1 STORE_HEADER
- The first 1–5 lines of the receipt, before the first recognizable
  item/price line appears.
- Often contains a store name in large/stylized text (e.g. "JUMBO",
  "Lidl", "Kruidvat"), an address, phone number, or website.
- Heuristic: lines before the first line matching an ITEM or
  QUANTITY_MODIFIER pattern (see below), or before a line matching
  `OMSCHRIJVING` / `Artikel` / similar column-header keywords.
- Extract the store name from this block using a known store-name list
  (maintain and grow this list — seen so far: JUMBO, LIDL, KRUIDVAT,
  ALBERT HEIJN, ALDI, and independent/ethnic grocers whose name appears
  as free text, e.g. "World Toko" — for these, take the first non-address
  line as a best-effort store name).

### 3.2 COLUMN_HEADER (ignore, not an item)
- Lines that are just table headers, not data.
- Match keywords: `OMSCHRIJVING`, `BEDRAG IN`, `Artikel`, `Prijs`,
  `Aantal Artikel`, or lines that are purely `=`, `-`, or `#` separator
  characters.

### 3.3 ITEM (product line)
- A line containing a product name AND a price at or near the end of
  the line.
- Price pattern: `\d+[,.]\d{2}` anchored near line end (allow trailing
  single-letter suffixes like ` B` or ` C` — see section 6, BTW category
  codes — these are not part of the price).
- Everything before the matched price is the product name (trim
  trailing spaces).
- This is the "complete" case: name + price on the same line, quantity
  of 1 implied unless a QUANTITY_MODIFIER line follows (section 3.4).

### 3.4 QUANTITY_MODIFIER (sub-line, belongs to previous or next ITEM)
- A line matching the pattern: `<int> X <price>` optionally followed by
  a unit word (e.g. `PER STUK`), e.g.:
  - `2 X 2,79`
  - `2 X 3,99 PER STUK`
- This line does NOT introduce a new product — it modifies the item on
  an adjacent line.
- Two observed variants — the parser must detect which applies:
  - **Variant A (Jumbo-style):** product name appears on its own line
    with NO price. The next line is `N X unit_price`. The line total
    (`N × unit_price`) appears as its own value either on the same
    `N X price` line's right-hand column, or as a separate line
    immediately after. Associate this quantity/total back to the
    product name line above it.
  - **Variant B (Kruidvat-style):** product name and `N X unit_price`
    appear on the SAME line, description-style (e.g.
    `ZEEPTABLET NEUTRAL 2 X 3,99 PER STUK`), with the computed total
    price in a separate right-aligned amount column/line.
- In both variants: `quantity = N`, `unit_price = price`,
  `line_total = N * unit_price` (compute this yourself, then cross-check
  against any total value found nearby — if they don't match within
  0.01, flag the item for manual review rather than guessing).

### 3.5 DISCOUNT
- A line representing a price reduction, not a product.
- Detection rules (match ANY of):
  - Line starts with or contains `ACTIE` (Jumbo pattern), e.g.
    `ACTIE AARDBEIEN` with an associated negative amount.
  - Line contains `KORTING` (Kruidvat/general Dutch for "discount"),
    e.g. `KORTING NEUTRAL ZE.HP`.
  - Line contains `In prijs verlaagd` (Lidl pattern — "price reduced").
  - The associated amount is negative (prefixed with `-`), e.g. `-2,00`,
    `-1,55`, `-0,21`, `-0,50`.
- A discount line should be linked to the item immediately preceding it
  when possible (the discount usually applies to the last product
  listed above it). Store it as a `discount` field on that item rather
  than as a separate negative "product."
- If a discount line cannot be confidently linked to a specific item
  (e.g. it's a receipt-wide discount near the total, like
  `Totaal korting: -2,21`), store it as a receipt-level
  `total_discount` field instead.

### 3.6 SUBTOTAL / TOTAL
- Lines matching keywords: `Totaal`, `SUBTOTAAL`, `Total`, `TOTAAL`.
- There may be MULTIPLE lines containing "Totaal" on one receipt
  (e.g. a running subtotal, then a final Totaal, then
  "Totaal korting" near the bottom, then "Totaal incl. BTW"). Rules:
  - The FIRST large bold "Totaal" line following the last item is the
    **receipt grand total** — use this as the authoritative total to
    validate your parsed items against.
  - Any "Totaal korting" line is the **sum of all discounts** — use
    this as a cross-check: sum of all `discount` fields you extracted
    should equal this value (within 0.01).
  - "Totaal incl. BTW" / "Totaal excl. BTW" relate to tax breakdown,
    not needed for item-level data, but can be stored as metadata.

### 3.7 PAYMENT (ignore for item extraction, optionally store as metadata)
- Lines matching: `Betaald`, `BETALING`, `VISA`, `Mastercard`, `PIN`,
  `Contactloze betaling`, `AKKOORD`, `Kaartnr`, `Transactie`,
  `Terminal`, `Merchant`, `Auth. code`, `Kaart`.
- These lines and everything below/around them (transaction IDs, auth
  codes, terminal numbers) should be excluded from item parsing
  entirely. Treat "first payment-related line after the last Totaal
  line" as the end of the item-relevant section of the receipt.

### 3.8 TAX / BTW BREAKDOWN (ignore for item extraction)
- Lines matching: `BTW`, `Bedr.Excl`, `Bedr.Incl`, a table with columns
  like `B 9%` / `C 21%`, `BTW-Code`.
- These describe tax categories, not individual products. Some product
  lines end in a single letter (`B` or `C`) marking their tax category
  — strip this letter before/after price extraction, don't treat it as
  part of the price or name.

### 3.9 NOISE / FOOTER (ignore entirely)
- Loyalty program mentions, website URLs, "Bewaar uw kassabon",
  return policy text, barcodes represented as text/numbers, store
  hours, social media handles, "Bedankt en graag tot ziens",
  promotional taglines. Match a broad keyword/pattern list and treat
  anything after the payment block as low-priority/ignorable by default.

## 4. Item construction algorithm (pseudocode)

```
items = []
pending_item = None   # name found, price/qty not yet resolved

for line in classified_lines:
    if line.type == ITEM:
        if pending_item:
            items.append(pending_item)   # close out previous item first
        pending_item = { name: line.name, unit_price: line.price,
                          quantity: 1, line_total: line.price,
                          discount: 0 }

    elif line.type == QUANTITY_MODIFIER:
        if pending_item and pending_item.unit_price is None:
            # Variant A: name-only line was pending, this line completes it
            pending_item.quantity = line.qty
            pending_item.unit_price = line.price
            pending_item.line_total = line.qty * line.price
        elif pending_item:
            # Variant B: name+price already set, this line refines/
            # confirms it (overwrite quantity/unit price, recompute total)
            pending_item.quantity = line.qty
            pending_item.unit_price = line.price
            pending_item.line_total = line.qty * line.price
        else:
            # Orphan quantity line with no preceding item — flag for
            # manual review rather than silently dropping
            flag_for_review(line)

    elif line.type == DISCOUNT:
        if pending_item:
            pending_item.discount += line.amount   # amount is negative
            pending_item.line_total += line.amount
        else:
            receipt.total_discount += line.amount

    elif line.type in (SUBTOTAL, TOTAL):
        if pending_item:
            items.append(pending_item)
            pending_item = None
        if is_grand_total(line):
            receipt.total = line.amount
            break   # stop item parsing here

# catch any trailing pending item if receipt ended oddly
if pending_item:
    items.append(pending_item)
```

## 5. Validation pass (do this after parsing, before showing results to user)

1. Sum `line_total` across all parsed items.
2. Add `receipt.total_discount` if it wasn't already folded into
   individual item discounts.
3. Compare the result to `receipt.total` (the grand total line).
4. If they match within 0.01 → high confidence, no flags needed.
5. If they don't match → mark the WHOLE receipt as "needs review" and
   surface it clearly in the review/edit screen (don't just flag
   individual lines — a mismatch means something in the whole parse is
   probably off, e.g. a missed line or a misclassified discount).

## 6. Known store-specific quirks to encode as configurable rules

Keep these as a lookup table / config, not hardcoded logic, so new
stores can be added without rewriting the parser:

| Store | Quirk |
|---|---|
| Jumbo | Multi-unit items: name on own line, then `N X price` line, then total. Discounts prefixed `ACTIE <name>` with negative amount. Tax category letter (`B`/`C`) sometimes appended to far-right column, not the price. |
| Lidl | Discounts appear as `In prijs verlaagd -0,50` style lines. Tax breakdown at the bottom uses `Bedr.Excl` / `BTW` / `Bedr.Incl` labels with `B 9%` / `C 21%` category rows — do not parse these rows as products. |
| Kruidvat | Multi-unit items combine name + `N X price PER STUK` on one line. `KORTING <description>` lines carry the discount, may appear one or two lines after the item, not always immediately after. |
| World Toko | Simple `Artikel` / `Prijs` two-column layout, mostly one item per line with price directly alongside; some multi-unit items still use a following `N X price` line (Variant A). |

Add new rows to this table as new receipt formats are encountered — this
is meant to grow with real-world use rather than be exhaustive on day one.

## 7. Output schema (what the parser should produce per receipt)

```json
{
  "store": "Jumbo",
  "date": "2026-08-30",
  "items": [
    {
      "name": "Jumbo Bio Volle Melk",
      "quantity": 1,
      "unit_price": 2.09,
      "discount": 0.00,
      "line_total": 2.09
    },
    {
      "name": "Biologisch Ei 6 st",
      "quantity": 2,
      "unit_price": 2.79,
      "discount": 0.00,
      "line_total": 5.58
    },
    {
      "name": "Aardbeien",
      "quantity": 2,
      "unit_price": 3.99,
      "discount": -2.00,
      "line_total": 5.98
    }
  ],
  "total_discount": -2.21,
  "receipt_total": 82.99,
  "validation_passed": true
}
```

## 8. Explicit non-goals for this parser

- No attempt to categorize items (groceries vs household, etc.) —
  out of scope.
- No attempt to normalize/deduplicate product names across receipts at
  parse time (e.g. matching "Jumbo Bio Volle Melk" to a previous
  purchase) — that's a separate matching step done after storage, not
  part of this parsing spec.
- No language detection/translation — assume Dutch keyword lists for
  now (`ACTIE`, `KORTING`, `Totaal`, `Betaald`, etc.), but structure the
  keyword lists as swappable config so other languages can be added.