Thomas GüttlerandClaude Sonnet 4.6
650c7a70f5
Move JMAP send, push, and conflict-resolution items from Next steps into Implemented features. Replace the next-steps section with optional future work (CONDSTORE, blob expiry, UI for stuck mutations). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
86 lines
4.5 KiB
Markdown
86 lines
4.5 KiB
Markdown
# DB Sync Status
|
||
|
||
This document covers the mail-to-database sync layer only, not the UI.
|
||
|
||
## Implemented features
|
||
|
||
### JMAP
|
||
|
||
- JMAP accounts can be stored in the database.
|
||
- JMAP endpoint discovery is implemented.
|
||
- JMAP connection testing is implemented.
|
||
- `sync_state` table stores server-side state tokens per (account, resource type).
|
||
- `pending_changes` table provides a protocol-agnostic outbound mutation queue.
|
||
- `JmapClient` fetches the JMAP Session object, extracts `apiUrl` and `accountId`,
|
||
and provides a `call()` helper for API requests.
|
||
- `syncMailboxes` for JMAP: first run uses `Mailbox/get`; subsequent runs use
|
||
`Mailbox/changes` with the stored state token.
|
||
- `syncEmails` for JMAP: first run uses `Email/query` + `Email/get`; subsequent runs
|
||
use `Email/changes`. Chains both calls in a single API request via `#ids` back-reference.
|
||
- `Email/query` pagination: cursor-based loop with `position` offset and `calculateTotal`
|
||
handles mailboxes larger than 500 emails.
|
||
- JMAP background sync worker (`_JmapAccountSync`): session → flush outbound queue →
|
||
syncMailboxes → syncEmails per mailbox → 30 s poll → repeat. Exponential backoff
|
||
5–300 s on failure.
|
||
- Local mutations (flag, move, delete) on JMAP accounts are written to `pending_changes`
|
||
with an optimistic local update. `flushPendingChanges` drains the queue via `Email/set`
|
||
at the start of each sync cycle.
|
||
- Email bodies are fetched on demand via `Email/get` with `bodyValues` and cached in
|
||
`email_bodies` so subsequent opens are instant.
|
||
- `syncEmails` fetches `bodyValues` during the sync pass so bodies are cached without
|
||
a separate on-demand fetch.
|
||
- `flushPendingChanges` passes `ifInState` to every `Email/set`; a `stateMismatch`
|
||
response clears the local checkpoint and triggers a full re-sync before retrying.
|
||
- **JMAP send**: outgoing mail uses `EmailSubmission/set` when the server advertises
|
||
the submission capability; falls back to SMTP otherwise.
|
||
- **JMAP push**: `_JmapAccountSync._wait()` subscribes to the server's SSE
|
||
`eventSourceUrl` via `watchJmapPush`; falls back to 30 s polling when push
|
||
is unavailable or the server does not advertise the URL.
|
||
- `notUpdated`/`notDestroyed` per-item errors from `Email/set` are treated as
|
||
permanent failures and discarded immediately (no retry).
|
||
|
||
### IMAP
|
||
|
||
- Background sync starts automatically for IMAP accounts.
|
||
- Mailbox lists and mailbox counters are synced into the local database.
|
||
- Email headers, flags, and attachment metadata are pulled from IMAP into the local database.
|
||
- Email bodies are fetched on demand and cached locally.
|
||
- All mailboxes (not just INBOX) are synced each cycle.
|
||
- Incremental sync: `(lastUid, uidValidity)` checkpoint stored in `sync_state`; only
|
||
new UIDs are fetched on subsequent runs; UID-validity change triggers a full re-scan.
|
||
- Deletion reconciliation: server UID set is compared against local rows; any email
|
||
absent from the server is removed from the local DB.
|
||
- Local mutations (flag, move, delete) are written to `pending_changes` with an
|
||
optimistic local update; `flushPendingChanges` drains the queue over a single
|
||
IMAP connection at the start of each sync cycle.
|
||
- Sent messages are appended to the Sent folder after SMTP delivery.
|
||
- Sync retries use exponential backoff after failures.
|
||
|
||
### Cross-protocol
|
||
|
||
- `sync_log` table records each sync cycle's account, result (ok / error), error
|
||
message, start time, and finish time. Used for debugging and "last synced" UI.
|
||
- **Conflict-resolution policy: server-wins.** The next sync cycle always
|
||
overwrites local state with server values. Outbound mutations in
|
||
`pending_changes` are retried up to 5 times before being evicted, preventing
|
||
unbounded queue growth. Permanent per-item JMAP errors (`notFound`,
|
||
`forbidden`) are discarded immediately; transient errors (network, 500)
|
||
are retried up to the limit.
|
||
- Integration test (`test/integration/concurrent_sync_test.dart`) concurrently
|
||
syncs an IMAP account (alice) and a JMAP account (bob) against a real Stalwart
|
||
server and verifies the Drift DB cache is consistent (no duplicates, correct
|
||
counts, no pending changes).
|
||
|
||
---
|
||
|
||
## Next steps
|
||
|
||
All planned sync-layer features are implemented. Possible future work:
|
||
|
||
- **IMAP CONDSTORE / QRESYNC**: use `MODSEQ` for faster incremental sync on
|
||
servers that support RFC 7162.
|
||
- **JMAP blob expiry**: detect and re-fetch body blobs that the server has
|
||
purged (currently the cache is assumed permanent).
|
||
- **Offline compose queue**: surface `pending_changes` failures in the UI so
|
||
the user can retry or discard stuck outbound mutations.
|