feat: add sync_state table (Step 1 — DB foundation for incremental sync)
Protocol-agnostic checkpoint table stores one state token per (account_id, resource_type). JMAP uses the opaque state string from Mailbox/get and Email/get; IMAP will use a JSON checkpoint per mailbox. DB schema bumped to v5. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Thomas Güttler
co-authored by
Claude Sonnet 4.6
Claude Sonnet 4.6
parent
4cefc8aac3
commit
475ba34d28
+111
-20
@@ -4,6 +4,12 @@ This document covers the mail-to-database sync layer only, not the UI.
|
||||
|
||||
## Implemented features
|
||||
|
||||
### JMAP
|
||||
|
||||
- JMAP accounts can be stored in the database.
|
||||
- JMAP endpoint discovery is implemented.
|
||||
- JMAP connection testing is implemented.
|
||||
|
||||
### IMAP
|
||||
|
||||
- Background sync starts automatically for IMAP accounts.
|
||||
@@ -14,34 +20,119 @@ This document covers the mail-to-database sync layer only, not the UI.
|
||||
- Sent messages are appended to the Sent folder after SMTP delivery.
|
||||
- Sync retries use exponential backoff after failures.
|
||||
|
||||
### JMAP
|
||||
---
|
||||
|
||||
- JMAP accounts can be stored in the database.
|
||||
- JMAP endpoint discovery is implemented.
|
||||
- JMAP connection testing is implemented.
|
||||
## Plan
|
||||
|
||||
## Missing features
|
||||
Goal: make bidirectional DB↔JMAP sync easy and correct. JMAP is the preferred protocol
|
||||
long-term because its state-based change tracking is cleaner than IMAP's UID/MODSEQ model.
|
||||
All DB foundations are protocol-agnostic so IMAP can use the same tables later.
|
||||
|
||||
### IMAP
|
||||
### Step 1 — `sync_state` table `[x]`
|
||||
|
||||
- No general outbound DB-to-IMAP sync queue for arbitrary local database changes.
|
||||
- Background sync currently refreshes only INBOX, not all folders.
|
||||
- No persisted sync log or audit trail in the database.
|
||||
- No explicit conflict-resolution strategy such as revisions, merge rules, or retryable pending operations.
|
||||
- No full reconciliation for remote deletions, mailbox removals, or stale local rows.
|
||||
- No incremental sync checkpoints such as last synced UID, MODSEQ, or similar state.
|
||||
- No durable offline-first sync state tracking.
|
||||
A single table that stores the server-side state token per (account, resource type).
|
||||
For JMAP this is the opaque `state` string returned by `Mailbox/get` and `Email/get`.
|
||||
For IMAP it will hold a JSON checkpoint (last UID, MODSEQ) per mailbox.
|
||||
|
||||
### JMAP
|
||||
Schema:
|
||||
|
||||
- No JMAP mailbox sync into the local database.
|
||||
- No JMAP email sync into the local database.
|
||||
- No DB-to-JMAP change propagation.
|
||||
- No background JMAP sync worker.
|
||||
- No JMAP conflict handling.
|
||||
- No JMAP sync log in the database.
|
||||
```sql
|
||||
sync_state (
|
||||
account_id TEXT NOT NULL,
|
||||
resource_type TEXT NOT NULL, -- e.g. "Mailbox", "Email", "INBOX"
|
||||
state TEXT NOT NULL, -- JMAP state string or IMAP checkpoint JSON
|
||||
synced_at DATETIME NOT NULL,
|
||||
PRIMARY KEY (account_id, resource_type)
|
||||
)
|
||||
```
|
||||
|
||||
### Step 2 — `pending_changes` table `[ ]`
|
||||
|
||||
Protocol-agnostic outbound queue. Any local mutation (flag, move, delete) is written
|
||||
here first. A sync worker drains the queue and sends to server. Enables offline-first.
|
||||
|
||||
Schema:
|
||||
|
||||
```sql
|
||||
pending_changes (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
account_id TEXT NOT NULL,
|
||||
resource_type TEXT NOT NULL, -- "Email"
|
||||
resource_id TEXT NOT NULL, -- local email id
|
||||
change_type TEXT NOT NULL, -- "flag_seen" | "flag_flagged" | "move" | "delete"
|
||||
payload TEXT NOT NULL, -- JSON, e.g. {"seen": true} or {"dest": "Archive"}
|
||||
created_at DATETIME NOT NULL,
|
||||
attempts INTEGER NOT NULL DEFAULT 0,
|
||||
last_error TEXT
|
||||
)
|
||||
```
|
||||
|
||||
### Step 3 — JMAP session client `[ ]`
|
||||
|
||||
Implement `JmapSession`: parse the JMAP Session object from `GET {jmapUrl}`,
|
||||
extract `apiUrl`, primary `accountId`, and capabilities. Store nothing extra in the
|
||||
DB (re-fetch session on start). Provide a `call(methodCalls)` helper that POSTs to
|
||||
`apiUrl` and decodes responses.
|
||||
|
||||
### Step 4 — JMAP Mailbox sync `[ ]`
|
||||
|
||||
Implement `syncMailboxes(accountId)` for JMAP:
|
||||
|
||||
- First run: `Mailbox/get` → upsert all mailboxes, persist state in `sync_state`.
|
||||
- Subsequent runs: `Mailbox/changes` using stored state → apply additions, updates,
|
||||
removals, then update state.
|
||||
|
||||
Reuse the existing `Mailboxes` table. No new DB columns needed.
|
||||
|
||||
### Step 5 — JMAP Email sync `[ ]`
|
||||
|
||||
Implement `syncEmails(accountId, mailboxId)` for JMAP:
|
||||
|
||||
- First run: `Email/query` (sorted by receivedAt desc, limit 500) + `Email/get` for
|
||||
the returned ids → upsert into `Emails`, persist state.
|
||||
- Subsequent runs: `Email/changes` using stored state → fetch new/changed via
|
||||
`Email/get`, delete removed rows, update state.
|
||||
|
||||
No new DB columns needed beyond `sync_state`.
|
||||
|
||||
### Step 6 — JMAP background sync worker `[ ]`
|
||||
|
||||
Add JMAP handling to `AccountSyncManager`:
|
||||
|
||||
- When a JMAP account appears, start a `_JmapAccountSync` loop.
|
||||
- Loop: session → syncMailboxes → syncEmails for each mailbox → wait (poll or
|
||||
EventSource if server supports it) → repeat.
|
||||
- Reuse the existing exponential backoff pattern from `_AccountSync`.
|
||||
|
||||
### Step 7 — JMAP outbound changes `[ ]`
|
||||
|
||||
Wire local mutations (flag, move, delete) for JMAP accounts into `pending_changes`
|
||||
instead of direct server calls. Add a queue-draining step at the start of each sync
|
||||
loop that issues `Email/set` for queued changes and removes them on success.
|
||||
|
||||
---
|
||||
|
||||
## Missing features (to be addressed after the plan above)
|
||||
|
||||
|
||||
### JMAP missing features
|
||||
|
||||
- Everything in the plan above (Steps 3–7).
|
||||
- No conflict handling (deferred; JMAP's `ifInState` provides the hook for it later).
|
||||
- No sync log in database (deferred).
|
||||
|
||||
### IMAP missing features
|
||||
|
||||
- Background sync refreshes only INBOX; other folders need the same treatment.
|
||||
- No incremental sync checkpoints (will use `sync_state` once Step 1 is done).
|
||||
- No durable outbound queue (will use `pending_changes` once Step 2 is done).
|
||||
- No full reconciliation for remote deletions.
|
||||
- No explicit conflict-resolution strategy.
|
||||
- No sync log or audit trail.
|
||||
|
||||
## Current summary
|
||||
|
||||
- IMAP: partially implemented and already usable, but not full bidirectional sync.
|
||||
- JMAP: account setup exists, but actual sync is still missing.
|
||||
- Plan above targets JMAP first; IMAP improvements follow naturally once the shared
|
||||
DB foundations (Steps 1–2) are in place.
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
# Later
|
||||
|
||||
Push to guettli@thomas-guettler via ssh+git
|
||||
|
||||
---
|
||||
|
||||
Flutter best practices?
|
||||
|
||||
@@ -80,6 +80,22 @@ class EmailBodies extends Table {
|
||||
Set<Column> get primaryKey => {emailId};
|
||||
}
|
||||
|
||||
/// Sync checkpoint per (account, resource type).
|
||||
/// Stores the server-side state token used for incremental sync.
|
||||
/// For JMAP: the opaque `state` string from Mailbox/get or Email/get.
|
||||
/// For IMAP: a JSON object with last-synced UID / MODSEQ per mailbox.
|
||||
@DataClassName('SyncStateRow')
|
||||
class SyncStates extends Table {
|
||||
TextColumn get accountId =>
|
||||
text().references(Accounts, #id, onDelete: KeyAction.cascade)();
|
||||
TextColumn get resourceType => text()();
|
||||
TextColumn get state => text()();
|
||||
DateTimeColumn get syncedAt => dateTime()();
|
||||
|
||||
@override
|
||||
Set<Column> get primaryKey => {accountId, resourceType};
|
||||
}
|
||||
|
||||
/// Auto-saved compose drafts — persisted across app restarts.
|
||||
class Drafts extends Table {
|
||||
IntColumn get id => integer().autoIncrement()();
|
||||
@@ -95,12 +111,12 @@ class Drafts extends Table {
|
||||
|
||||
// ── Database ──────────────────────────────────────────────────────────────────
|
||||
|
||||
@DriftDatabase(tables: [Accounts, Mailboxes, Emails, EmailBodies, Drafts])
|
||||
@DriftDatabase(tables: [Accounts, Mailboxes, Emails, EmailBodies, Drafts, SyncStates])
|
||||
class AppDatabase extends _$AppDatabase {
|
||||
AppDatabase([QueryExecutor? executor]) : super(executor ?? _openConnection());
|
||||
|
||||
@override
|
||||
int get schemaVersion => 4;
|
||||
int get schemaVersion => 5;
|
||||
|
||||
@override
|
||||
MigrationStrategy get migration => MigrationStrategy(
|
||||
@@ -115,6 +131,9 @@ class AppDatabase extends _$AppDatabase {
|
||||
if (from < 4) {
|
||||
await m.createTable(drafts);
|
||||
}
|
||||
if (from < 5) {
|
||||
await m.createTable(syncStates);
|
||||
}
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user