sharedinbox/DB-SYNC.md

# DB Sync Status

This document covers the mail-to-database sync layer only, not the UI.

## Implemented features

### JMAP

- JMAP accounts can be stored in the database.
- JMAP endpoint discovery is implemented.
- JMAP connection testing is implemented.
- `sync_state` table stores server-side state tokens per (account, resource type).
- `pending_changes` table provides a protocol-agnostic outbound mutation queue.
- `JmapClient` fetches the JMAP Session object, extracts `apiUrl` and `accountId`,
  and provides a `call()` helper for API requests.
- `syncMailboxes` for JMAP: first run uses `Mailbox/get`; subsequent runs use
  `Mailbox/changes` with the stored state token.
- `syncEmails` for JMAP: first run uses `Email/query` + `Email/get`; subsequent runs
  use `Email/changes`. Chains both calls in a single API request via `#ids` back-reference.
- `Email/query` pagination: cursor-based loop with `position` offset and `calculateTotal`
  handles mailboxes larger than 500 emails.
- JMAP background sync worker (`_JmapAccountSync`): session → flush outbound queue →
  syncMailboxes → syncEmails per mailbox → 30 s poll → repeat. Exponential backoff
  5–300 s on failure.
- Local mutations (flag, move, delete) on JMAP accounts are written to `pending_changes`
  with an optimistic local update. `flushPendingChanges` drains the queue via `Email/set`
  at the start of each sync cycle.
- Email bodies are fetched on demand via `Email/get` with `bodyValues` and cached in
  `email_bodies` so subsequent opens are instant.
- `syncEmails` fetches `bodyValues` during the sync pass so bodies are cached without
  a separate on-demand fetch.
- `flushPendingChanges` passes `ifInState` to every `Email/set`; a `stateMismatch`
  response clears the local checkpoint and triggers a full re-sync before retrying.
- **JMAP send**: outgoing mail uses `EmailSubmission/set` when the server advertises
  the submission capability; falls back to SMTP otherwise.
- **JMAP push**: `_JmapAccountSync._wait()` subscribes to the server's SSE
  `eventSourceUrl` via `watchJmapPush`; falls back to 30 s polling when push
  is unavailable or the server does not advertise the URL.
- `notUpdated`/`notDestroyed` per-item errors from `Email/set` are treated as
  permanent failures and discarded immediately (no retry).

### IMAP

- Background sync starts automatically for IMAP accounts.
- Mailbox lists and mailbox counters are synced into the local database.
- Email headers, flags, and attachment metadata are pulled from IMAP into the local database.
- Email bodies are fetched on demand and cached locally.
- All mailboxes (not just INBOX) are synced each cycle.
- Incremental sync: `(lastUid, uidValidity)` checkpoint stored in `sync_state`; only
  new UIDs are fetched on subsequent runs; UID-validity change triggers a full re-scan.
- Deletion reconciliation: server UID set is compared against local rows; any email
  absent from the server is removed from the local DB.
- Local mutations (flag, move, delete) are written to `pending_changes` with an
  optimistic local update; `flushPendingChanges` drains the queue over a single
  IMAP connection at the start of each sync cycle.
- Sent messages are appended to the Sent folder after SMTP delivery.
- IMAP move remap: after a `MOVE` is flushed, the local row id is rewritten in
  place using the RFC 4315 `COPYUID` response code (UIDPLUS); if the server
  doesn't support UIDPLUS, the new UID is looked up via `UID SEARCH HEADER
  Message-ID …` in the destination mailbox. Cached bodies (`email_bodies`),
  threads, queued pending changes, and undo entries follow the new id.
  Deletion reconciliation skips rows whose `move`/`snooze`/`unsnooze` is still
  in `pending_changes` so the optimistic local move isn't wiped mid-flight.
- Sync retries use exponential backoff after failures.

### Cross-protocol

- `sync_log` table records each sync cycle's account, result (ok / error), error
  message, start time, and finish time. Used for debugging and "last synced" UI.
- **Conflict-resolution policy: server-wins.** The next sync cycle always
  overwrites local state with server values. Outbound mutations in
  `pending_changes` are retried up to 5 times before being evicted, preventing
  unbounded queue growth. Permanent per-item JMAP errors (`notFound`,
  `forbidden`) are discarded immediately; transient errors (network, 500)
  are retried up to the limit.
- Integration test (`test/integration/concurrent_sync_test.dart`) concurrently
  syncs an IMAP account (alice) and a JMAP account (bob) against a real Stalwart
  server and verifies the Drift DB cache is consistent (no duplicates, correct
  counts, no pending changes).

---

## Next steps

All planned sync-layer features are implemented. Possible future work:

- **IMAP CONDSTORE / QRESYNC**: use `MODSEQ` for faster incremental sync on
  servers that support RFC 7162.
- **JMAP blob expiry**: detect and re-fetch body blobs that the server has
  purged (currently the cache is assumed permanent).
- **Offline compose queue**: surface `pending_changes` failures in the UI so
  the user can retry or discard stuck outbound mutations.