diff --git a/DB-SYNC.md b/DB-SYNC.md index 9ae8153..9c58770 100644 --- a/DB-SYNC.md +++ b/DB-SYNC.md @@ -4,6 +4,12 @@ This document covers the mail-to-database sync layer only, not the UI. ## Implemented features +### JMAP + +- JMAP accounts can be stored in the database. +- JMAP endpoint discovery is implemented. +- JMAP connection testing is implemented. + ### IMAP - Background sync starts automatically for IMAP accounts. @@ -14,34 +20,119 @@ This document covers the mail-to-database sync layer only, not the UI. - Sent messages are appended to the Sent folder after SMTP delivery. - Sync retries use exponential backoff after failures. -### JMAP +--- -- JMAP accounts can be stored in the database. -- JMAP endpoint discovery is implemented. -- JMAP connection testing is implemented. +## Plan -## Missing features +Goal: make bidirectional DB↔JMAP sync easy and correct. JMAP is the preferred protocol +long-term because its state-based change tracking is cleaner than IMAP's UID/MODSEQ model. +All DB foundations are protocol-agnostic so IMAP can use the same tables later. -### IMAP +### Step 1 — `sync_state` table `[x]` -- No general outbound DB-to-IMAP sync queue for arbitrary local database changes. -- Background sync currently refreshes only INBOX, not all folders. -- No persisted sync log or audit trail in the database. -- No explicit conflict-resolution strategy such as revisions, merge rules, or retryable pending operations. -- No full reconciliation for remote deletions, mailbox removals, or stale local rows. -- No incremental sync checkpoints such as last synced UID, MODSEQ, or similar state. -- No durable offline-first sync state tracking. +A single table that stores the server-side state token per (account, resource type). +For JMAP this is the opaque `state` string returned by `Mailbox/get` and `Email/get`. +For IMAP it will hold a JSON checkpoint (last UID, MODSEQ) per mailbox. -### JMAP +Schema: -- No JMAP mailbox sync into the local database. -- No JMAP email sync into the local database. -- No DB-to-JMAP change propagation. -- No background JMAP sync worker. -- No JMAP conflict handling. -- No JMAP sync log in the database. +```sql +sync_state ( + account_id TEXT NOT NULL, + resource_type TEXT NOT NULL, -- e.g. "Mailbox", "Email", "INBOX" + state TEXT NOT NULL, -- JMAP state string or IMAP checkpoint JSON + synced_at DATETIME NOT NULL, + PRIMARY KEY (account_id, resource_type) +) +``` + +### Step 2 — `pending_changes` table `[ ]` + +Protocol-agnostic outbound queue. Any local mutation (flag, move, delete) is written +here first. A sync worker drains the queue and sends to server. Enables offline-first. + +Schema: + +```sql +pending_changes ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + account_id TEXT NOT NULL, + resource_type TEXT NOT NULL, -- "Email" + resource_id TEXT NOT NULL, -- local email id + change_type TEXT NOT NULL, -- "flag_seen" | "flag_flagged" | "move" | "delete" + payload TEXT NOT NULL, -- JSON, e.g. {"seen": true} or {"dest": "Archive"} + created_at DATETIME NOT NULL, + attempts INTEGER NOT NULL DEFAULT 0, + last_error TEXT +) +``` + +### Step 3 — JMAP session client `[ ]` + +Implement `JmapSession`: parse the JMAP Session object from `GET {jmapUrl}`, +extract `apiUrl`, primary `accountId`, and capabilities. Store nothing extra in the +DB (re-fetch session on start). Provide a `call(methodCalls)` helper that POSTs to +`apiUrl` and decodes responses. + +### Step 4 — JMAP Mailbox sync `[ ]` + +Implement `syncMailboxes(accountId)` for JMAP: + +- First run: `Mailbox/get` → upsert all mailboxes, persist state in `sync_state`. +- Subsequent runs: `Mailbox/changes` using stored state → apply additions, updates, + removals, then update state. + +Reuse the existing `Mailboxes` table. No new DB columns needed. + +### Step 5 — JMAP Email sync `[ ]` + +Implement `syncEmails(accountId, mailboxId)` for JMAP: + +- First run: `Email/query` (sorted by receivedAt desc, limit 500) + `Email/get` for + the returned ids → upsert into `Emails`, persist state. +- Subsequent runs: `Email/changes` using stored state → fetch new/changed via + `Email/get`, delete removed rows, update state. + +No new DB columns needed beyond `sync_state`. + +### Step 6 — JMAP background sync worker `[ ]` + +Add JMAP handling to `AccountSyncManager`: + +- When a JMAP account appears, start a `_JmapAccountSync` loop. +- Loop: session → syncMailboxes → syncEmails for each mailbox → wait (poll or + EventSource if server supports it) → repeat. +- Reuse the existing exponential backoff pattern from `_AccountSync`. + +### Step 7 — JMAP outbound changes `[ ]` + +Wire local mutations (flag, move, delete) for JMAP accounts into `pending_changes` +instead of direct server calls. Add a queue-draining step at the start of each sync +loop that issues `Email/set` for queued changes and removes them on success. + +--- + +## Missing features (to be addressed after the plan above) + + +### JMAP missing features + +- Everything in the plan above (Steps 3–7). +- No conflict handling (deferred; JMAP's `ifInState` provides the hook for it later). +- No sync log in database (deferred). + +### IMAP missing features + +- Background sync refreshes only INBOX; other folders need the same treatment. +- No incremental sync checkpoints (will use `sync_state` once Step 1 is done). +- No durable outbound queue (will use `pending_changes` once Step 2 is done). +- No full reconciliation for remote deletions. +- No explicit conflict-resolution strategy. +- No sync log or audit trail. ## Current summary - IMAP: partially implemented and already usable, but not full bidirectional sync. - JMAP: account setup exists, but actual sync is still missing. +- Plan above targets JMAP first; IMAP improvements follow naturally once the shared + DB foundations (Steps 1–2) are in place. diff --git a/LATER.md b/LATER.md index dc83486..c41fae0 100644 --- a/LATER.md +++ b/LATER.md @@ -1,5 +1,7 @@ # Later +Push to guettli@thomas-guettler via ssh+git + --- Flutter best practices? diff --git a/lib/data/db/database.dart b/lib/data/db/database.dart index 27a891d..974c9a7 100644 --- a/lib/data/db/database.dart +++ b/lib/data/db/database.dart @@ -80,6 +80,22 @@ class EmailBodies extends Table { Set get primaryKey => {emailId}; } +/// Sync checkpoint per (account, resource type). +/// Stores the server-side state token used for incremental sync. +/// For JMAP: the opaque `state` string from Mailbox/get or Email/get. +/// For IMAP: a JSON object with last-synced UID / MODSEQ per mailbox. +@DataClassName('SyncStateRow') +class SyncStates extends Table { + TextColumn get accountId => + text().references(Accounts, #id, onDelete: KeyAction.cascade)(); + TextColumn get resourceType => text()(); + TextColumn get state => text()(); + DateTimeColumn get syncedAt => dateTime()(); + + @override + Set get primaryKey => {accountId, resourceType}; +} + /// Auto-saved compose drafts — persisted across app restarts. class Drafts extends Table { IntColumn get id => integer().autoIncrement()(); @@ -95,12 +111,12 @@ class Drafts extends Table { // ── Database ────────────────────────────────────────────────────────────────── -@DriftDatabase(tables: [Accounts, Mailboxes, Emails, EmailBodies, Drafts]) +@DriftDatabase(tables: [Accounts, Mailboxes, Emails, EmailBodies, Drafts, SyncStates]) class AppDatabase extends _$AppDatabase { AppDatabase([QueryExecutor? executor]) : super(executor ?? _openConnection()); @override - int get schemaVersion => 4; + int get schemaVersion => 5; @override MigrationStrategy get migration => MigrationStrategy( @@ -115,6 +131,9 @@ class AppDatabase extends _$AppDatabase { if (from < 4) { await m.createTable(drafts); } + if (from < 5) { + await m.createTable(syncStates); + } }, ); }