feat: JMAP body caching during sync and ifInState conflict detection
- Include bodyValues/textBody/htmlBody/attachments in every Email/get call during syncEmails; _upsertJmapEmails writes to email_bodies so first open is instant even for freshly synced messages - Extract _parseJmapBody helper shared by sync path and on-demand fetch - Add JmapStateMismatchException; _applyPendingChangeJmap passes ifInState and returns newState; on stateMismatch the local checkpoint is cleared so the next cycle does a full re-sync before retrying the mutation - Update DB-SYNC.md to reflect what has been implemented Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Thomas Güttler
co-authored by
Claude Sonnet 4.6
Claude Sonnet 4.6
parent
db548a7d8b
commit
7e34ca45de
+44
-109
@@ -9,6 +9,24 @@ This document covers the mail-to-database sync layer only, not the UI.
|
||||
- JMAP accounts can be stored in the database.
|
||||
- JMAP endpoint discovery is implemented.
|
||||
- JMAP connection testing is implemented.
|
||||
- `sync_state` table stores server-side state tokens per (account, resource type).
|
||||
- `pending_changes` table provides a protocol-agnostic outbound mutation queue.
|
||||
- `JmapClient` fetches the JMAP Session object, extracts `apiUrl` and `accountId`,
|
||||
and provides a `call()` helper for API requests.
|
||||
- `syncMailboxes` for JMAP: first run uses `Mailbox/get`; subsequent runs use
|
||||
`Mailbox/changes` with the stored state token.
|
||||
- `syncEmails` for JMAP: first run uses `Email/query` + `Email/get`; subsequent runs
|
||||
use `Email/changes`. Chains both calls in a single API request via `#ids` back-reference.
|
||||
- `Email/query` pagination: cursor-based loop with `position` offset and `calculateTotal`
|
||||
handles mailboxes larger than 500 emails.
|
||||
- JMAP background sync worker (`_JmapAccountSync`): session → flush outbound queue →
|
||||
syncMailboxes → syncEmails per mailbox → 30 s poll → repeat. Exponential backoff
|
||||
5–300 s on failure.
|
||||
- Local mutations (flag, move, delete) on JMAP accounts are written to `pending_changes`
|
||||
with an optimistic local update. `flushPendingChanges` drains the queue via `Email/set`
|
||||
at the start of each sync cycle.
|
||||
- Email bodies are fetched on demand via `Email/get` with `bodyValues` and cached in
|
||||
`email_bodies` so subsequent opens are instant.
|
||||
|
||||
### IMAP
|
||||
|
||||
@@ -16,123 +34,40 @@ This document covers the mail-to-database sync layer only, not the UI.
|
||||
- Mailbox lists and mailbox counters are synced into the local database.
|
||||
- Email headers, flags, and attachment metadata are pulled from IMAP into the local database.
|
||||
- Email bodies are fetched on demand and cached locally.
|
||||
- User-triggered changes are sent to the server immediately: seen, flagged, move, delete, and send.
|
||||
- All mailboxes (not just INBOX) are synced each cycle.
|
||||
- Incremental sync: `(lastUid, uidValidity)` checkpoint stored in `sync_state`; only
|
||||
new UIDs are fetched on subsequent runs; UID-validity change triggers a full re-scan.
|
||||
- Deletion reconciliation: server UID set is compared against local rows; any email
|
||||
absent from the server is removed from the local DB.
|
||||
- Local mutations (flag, move, delete) are written to `pending_changes` with an
|
||||
optimistic local update; `flushPendingChanges` drains the queue over a single
|
||||
IMAP connection at the start of each sync cycle.
|
||||
- Sent messages are appended to the Sent folder after SMTP delivery.
|
||||
- Sync retries use exponential backoff after failures.
|
||||
|
||||
---
|
||||
### Cross-protocol
|
||||
|
||||
## Plan
|
||||
|
||||
Goal: make bidirectional DB↔JMAP sync easy and correct. JMAP is the preferred protocol
|
||||
long-term because its state-based change tracking is cleaner than IMAP's UID/MODSEQ model.
|
||||
All DB foundations are protocol-agnostic so IMAP can use the same tables later.
|
||||
|
||||
### Step 1 — `sync_state` table `[x]`
|
||||
|
||||
A single table that stores the server-side state token per (account, resource type).
|
||||
For JMAP this is the opaque `state` string returned by `Mailbox/get` and `Email/get`.
|
||||
For IMAP it will hold a JSON checkpoint (last UID, MODSEQ) per mailbox.
|
||||
|
||||
Schema:
|
||||
|
||||
```sql
|
||||
sync_state (
|
||||
account_id TEXT NOT NULL,
|
||||
resource_type TEXT NOT NULL, -- e.g. "Mailbox", "Email", "INBOX"
|
||||
state TEXT NOT NULL, -- JMAP state string or IMAP checkpoint JSON
|
||||
synced_at DATETIME NOT NULL,
|
||||
PRIMARY KEY (account_id, resource_type)
|
||||
)
|
||||
```
|
||||
|
||||
### Step 2 — `pending_changes` table `[x]`
|
||||
|
||||
Protocol-agnostic outbound queue. Any local mutation (flag, move, delete) is written
|
||||
here first. A sync worker drains the queue and sends to server. Enables offline-first.
|
||||
|
||||
Schema:
|
||||
|
||||
```sql
|
||||
pending_changes (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
account_id TEXT NOT NULL,
|
||||
resource_type TEXT NOT NULL, -- "Email"
|
||||
resource_id TEXT NOT NULL, -- local email id
|
||||
change_type TEXT NOT NULL, -- "flag_seen" | "flag_flagged" | "move" | "delete"
|
||||
payload TEXT NOT NULL, -- JSON, e.g. {"seen": true} or {"dest": "Archive"}
|
||||
created_at DATETIME NOT NULL,
|
||||
attempts INTEGER NOT NULL DEFAULT 0,
|
||||
last_error TEXT
|
||||
)
|
||||
```
|
||||
|
||||
### Step 3 — JMAP session client `[x]`
|
||||
|
||||
Implement `JmapSession`: parse the JMAP Session object from `GET {jmapUrl}`,
|
||||
extract `apiUrl`, primary `accountId`, and capabilities. Store nothing extra in the
|
||||
DB (re-fetch session on start). Provide a `call(methodCalls)` helper that POSTs to
|
||||
`apiUrl` and decodes responses.
|
||||
|
||||
### Step 4 — JMAP Mailbox sync `[x]`
|
||||
|
||||
Implement `syncMailboxes(accountId)` for JMAP:
|
||||
|
||||
- First run: `Mailbox/get` → upsert all mailboxes, persist state in `sync_state`.
|
||||
- Subsequent runs: `Mailbox/changes` using stored state → apply additions, updates,
|
||||
removals, then update state.
|
||||
|
||||
Reuse the existing `Mailboxes` table. No new DB columns needed.
|
||||
|
||||
### Step 5 — JMAP Email sync `[x]`
|
||||
|
||||
Implement `syncEmails(accountId, mailboxId)` for JMAP:
|
||||
|
||||
- First run: `Email/query` (sorted by receivedAt desc, limit 500) + `Email/get` for
|
||||
the returned ids → upsert into `Emails`, persist state.
|
||||
- Subsequent runs: `Email/changes` using stored state → fetch new/changed via
|
||||
`Email/get`, delete removed rows, update state.
|
||||
|
||||
No new DB columns needed beyond `sync_state`.
|
||||
|
||||
### Step 6 — JMAP background sync worker `[x]`
|
||||
|
||||
Add JMAP handling to `AccountSyncManager`:
|
||||
|
||||
- When a JMAP account appears, start a `_JmapAccountSync` loop.
|
||||
- Loop: session → syncMailboxes → syncEmails for each mailbox → wait (poll or
|
||||
EventSource if server supports it) → repeat.
|
||||
- Reuse the existing exponential backoff pattern from `_AccountSync`.
|
||||
|
||||
### Step 7 — JMAP outbound changes `[x]`
|
||||
|
||||
Wire local mutations (flag, move, delete) for JMAP accounts into `pending_changes`
|
||||
instead of direct server calls. Add a queue-draining step at the start of each sync
|
||||
loop that issues `Email/set` for queued changes and removes them on success.
|
||||
- `sync_log` table records each sync cycle's account, result (ok / error), error
|
||||
message, start time, and finish time. Used for debugging and "last synced" UI.
|
||||
|
||||
---
|
||||
|
||||
## Missing features (to be addressed after the plan above)
|
||||
## Next steps
|
||||
|
||||
### JMAP hardening
|
||||
|
||||
### JMAP missing features
|
||||
- **Body caching during sync**: `syncEmails` currently syncs headers only. Include
|
||||
`bodyValues` + `htmlBody`/`textBody` in the `Email/get` properties list so bodies
|
||||
are written to `email_bodies` during the sync pass, not just on first open.
|
||||
- **JMAP send**: implement outgoing mail via `EmailSubmission/set` in addition to the
|
||||
current SMTP path.
|
||||
- **Push instead of polling**: upgrade `_JmapAccountSync._wait()` to use an
|
||||
`EventSource` connection to the JMAP push URL when the server advertises push
|
||||
capability. Fall back to 30 s polling when push is unavailable.
|
||||
- **Conflict handling**: pass `ifInState` to `Email/set` in `flushPendingChanges` so
|
||||
the server can reject a stale mutation; retry the affected change after re-syncing.
|
||||
|
||||
- Everything in the plan above (Steps 3–7).
|
||||
- No conflict handling (deferred; JMAP's `ifInState` provides the hook for it later).
|
||||
- No sync log in database (deferred).
|
||||
### Shared / cross-protocol
|
||||
|
||||
### IMAP missing features
|
||||
|
||||
- Background sync refreshes only INBOX; other folders need the same treatment.
|
||||
- No incremental sync checkpoints (will use `sync_state` once Step 1 is done).
|
||||
- No durable outbound queue (will use `pending_changes` once Step 2 is done).
|
||||
- No full reconciliation for remote deletions.
|
||||
- No explicit conflict-resolution strategy.
|
||||
- No sync log or audit trail.
|
||||
|
||||
## Current summary
|
||||
|
||||
- IMAP: partially implemented and already usable, but not full bidirectional sync.
|
||||
- JMAP: account setup exists, but actual sync is still missing.
|
||||
- Plan above targets JMAP first; IMAP improvements follow naturally once the shared
|
||||
DB foundations (Steps 1–2) are in place.
|
||||
- **Explicit conflict-resolution strategy**: decide and document the policy (last-write-
|
||||
wins vs. server-wins) and implement it consistently across both protocols.
|
||||
|
||||
Reference in New Issue
Block a user