docs: SYNC.md — full email action lifecycle (D3) (#54)
This commit was merged in pull request #54.
This commit is contained in:
Bot of Thomas Güttler
@@ -0,0 +1,206 @@
|
||||
# Email Sync Architecture
|
||||
|
||||
This document describes the full lifecycle of an email action — from the moment the user taps
|
||||
a button to server confirmation — covering the IMAP IDLE loop, JMAP push/poll, the pending-change
|
||||
queue, exponential backoff, and the undo/cancel mechanism.
|
||||
|
||||
For the database schema and protocol-level implementation details see [DB-SYNC.md](DB-SYNC.md).
|
||||
|
||||
---
|
||||
|
||||
## 1. Components
|
||||
|
||||
| Component | File | Role |
|
||||
|-----------|------|------|
|
||||
| `AccountSyncManager` | `lib/core/sync/account_sync_manager.dart` | Owns one `_SyncLoop` per account; starts, stops, and wakes sync loops |
|
||||
| `_AccountSync` | same file | IMAP sync loop (IDLE + incremental fetch) |
|
||||
| `_JmapAccountSync` | same file | JMAP sync loop (SSE push + poll fallback) |
|
||||
| `EmailRepositoryImpl` | `lib/data/repositories/email_repository_impl.dart` | All DB reads/writes and network calls |
|
||||
| `pending_changes` table | `lib/data/db/database.dart` | Protocol-agnostic outbound mutation queue |
|
||||
| `UndoService` | `lib/core/services/undo_service.dart` | Persisted undo history; cancel-or-reverse logic |
|
||||
|
||||
---
|
||||
|
||||
## 2. Lifecycle of an email mutation (e.g. "Mark as read")
|
||||
|
||||
```
|
||||
User taps "Mark as read"
|
||||
│
|
||||
▼
|
||||
EmailRepository.setFlag(id, seen: true)
|
||||
│
|
||||
├─ 1. Write optimistic update to local DB
|
||||
│ emails.is_seen = true
|
||||
│
|
||||
└─ 2. Insert row into pending_changes
|
||||
{ type: 'flag_seen', email_id: id, payload: {seen: true} }
|
||||
(IMAP: includes uid + mailboxPath for the STORE command)
|
||||
(JMAP: includes just the flag map for Email/set)
|
||||
|
||||
[UI immediately reflects the change via Drift's reactive streams]
|
||||
|
||||
│
|
||||
▼ (next sync cycle, triggered by IMAP IDLE / JMAP push / wakeUp)
|
||||
_SyncLoop._flush() / flushPendingChanges()
|
||||
│
|
||||
├─ IMAP: open connection → STORE uid +FLAGS (\Seen) → close
|
||||
│
|
||||
└─ JMAP: Email/set { update: { id: { keywords: { "$seen": true } } } }
|
||||
If stateMismatch → clear checkpoint → full re-sync
|
||||
|
||||
│
|
||||
▼
|
||||
pending_changes row deleted on success
|
||||
(on permanent error: retry count incremented; evicted after 5 failures)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. IMAP sync loop
|
||||
|
||||
The IMAP loop runs one coroutine per account (`_AccountSync`):
|
||||
|
||||
```
|
||||
start()
|
||||
│
|
||||
▼
|
||||
[forever loop]
|
||||
├─ flushPendingChanges() ← drain outbound queue first
|
||||
├─ syncMailboxes() ← detect new/removed mailboxes
|
||||
├─ for each mailbox:
|
||||
│ syncEmails() ← incremental: fetch only UIDs > lastUid
|
||||
│ deletion reconciliation: remove rows
|
||||
│ whose UID is absent from the server
|
||||
└─ _idle() ← IMAP IDLE for up to 25 min (RFC 2177)
|
||||
│ Wakes on: server EXISTS/EXPUNGE/FLAGS
|
||||
│ or syncNow() signal from UI
|
||||
└─ repeat
|
||||
```
|
||||
|
||||
**Incremental sync checkpoint** — `sync_state` table stores `(accountId, mailbox, lastUid, uidValidity)`.
|
||||
On each run, only UIDs greater than `lastUid` are fetched. If `uidValidity` changes the full
|
||||
folder is re-scanned and the checkpoint is reset.
|
||||
|
||||
**IDLE cap** — IDLE sessions are limited to 25 minutes per the RFC. The loop also wakes
|
||||
immediately if `syncNow()` is called (e.g. user pulls-to-refresh).
|
||||
|
||||
---
|
||||
|
||||
## 4. JMAP sync loop
|
||||
|
||||
The JMAP loop (`_JmapAccountSync`) follows a similar structure but uses HTTP:
|
||||
|
||||
```
|
||||
start()
|
||||
│
|
||||
▼
|
||||
[forever loop]
|
||||
├─ flushPendingChanges() ← Email/set for queued mutations
|
||||
├─ syncMailboxes() ← Mailbox/get or Mailbox/changes
|
||||
├─ for each mailbox:
|
||||
│ syncEmails() ← Email/query + Email/get (first run)
|
||||
│ Email/changes (subsequent runs, state token)
|
||||
└─ _wait()
|
||||
├─ If server advertises eventSourceUrl: subscribe to SSE push
|
||||
│ wake on "Email" change event
|
||||
└─ Otherwise: sleep 30 s (poll fallback)
|
||||
```
|
||||
|
||||
**State tokens** — each `Mailbox/changes` / `Email/changes` call uses the server-provided
|
||||
`state` token stored in `sync_state`. A `stateMismatch` error clears the token and triggers
|
||||
a full re-fetch.
|
||||
|
||||
**JMAP send** — outgoing mail uses `EmailSubmission/set` when the server advertises the
|
||||
`urn:ietf:params:jmap:submission` capability; falls back to SMTP otherwise.
|
||||
|
||||
---
|
||||
|
||||
## 5. Exponential backoff
|
||||
|
||||
Both loops share the same backoff policy:
|
||||
|
||||
| Outcome | Backoff |
|
||||
|---------|---------|
|
||||
| Sync succeeded | Reset to 5 s |
|
||||
| Network / server error | Double previous backoff, capped at 900 s (15 min) |
|
||||
|
||||
The backoff counter (`_backoffSeconds`) is per-account and per-process; it resets to 5 s
|
||||
on the next successful cycle.
|
||||
|
||||
The last error message is written to `sync_log` and surfaced in the UI via
|
||||
`syncLastErrorProvider` (the red `MaterialBanner` in the email list).
|
||||
|
||||
---
|
||||
|
||||
## 6. Pending-change queue
|
||||
|
||||
`pending_changes` is a protocol-agnostic table that stores every outbound mutation before it
|
||||
reaches the server:
|
||||
|
||||
| Column | Description |
|
||||
|--------|-------------|
|
||||
| `id` | Auto-increment primary key |
|
||||
| `email_id` | The email being mutated |
|
||||
| `type` | `flag_seen`, `flag_flagged`, `move`, `delete`, `snooze` |
|
||||
| `payload` | JSON-encoded protocol-specific arguments |
|
||||
| `retry_count` | Incremented on each failed flush attempt |
|
||||
| `created_at` | For ordering and debug |
|
||||
|
||||
**Optimistic UI** — every mutation writes the local change first, then inserts into
|
||||
`pending_changes`. The Drift reactive stream delivers the update to the UI before
|
||||
the network round-trip completes.
|
||||
|
||||
**Conflict resolution** — the server always wins. On the next sync cycle the server's
|
||||
state overwrites local rows. Outbound mutations are retried up to 5 times; after that
|
||||
they are evicted and a `FailedMutation` record is created. Permanent per-item JMAP
|
||||
errors (`notFound`, `forbidden`) skip the retry counter and evict immediately.
|
||||
|
||||
---
|
||||
|
||||
## 7. Undo and cancel
|
||||
|
||||
When the user triggers an undoable action the UI calls:
|
||||
```
|
||||
ref.read(undoServiceProvider.notifier).pushAction(UndoAction(...))
|
||||
```
|
||||
|
||||
`UndoService` persists the action to the `undo_actions` table (max 10 entries, FIFO).
|
||||
A `SnackBar` with an **Undo** button appears for a few seconds.
|
||||
|
||||
When the user taps Undo, `UndoService.undo()` executes this sequence for each affected email:
|
||||
|
||||
```
|
||||
1. cancelPendingChange(id, originalType)
|
||||
└─ Deletes the pending_changes row if it has not been flushed yet.
|
||||
Returns true if cancelled, false if the server already processed it.
|
||||
|
||||
2. If the email row was hard-deleted (DELETE action):
|
||||
restoreEmails([original])
|
||||
└─ Re-inserts the row with its pre-deletion state,
|
||||
placed in the correct mailbox (source if cancelled, dest otherwise).
|
||||
|
||||
3. moveEmail(id, sourceMailboxPath)
|
||||
└─ Optimistic local move back to the original folder.
|
||||
If step 1 returned false (already sent to server), this enqueues
|
||||
a reverse-move in pending_changes so the server move is undone too.
|
||||
|
||||
4. If step 1 returned true (cancelled before flush):
|
||||
cancelPendingChange(id, 'move')
|
||||
└─ The reverse-move from step 3 is redundant; remove it.
|
||||
```
|
||||
|
||||
The net result is: if the mutation was still in the queue it is silently cancelled with no
|
||||
server round-trip; if it had already been flushed, a compensating move is queued.
|
||||
|
||||
---
|
||||
|
||||
## 8. Key invariants
|
||||
|
||||
- **Order**: pending changes are flushed before syncing. This prevents the server from
|
||||
overwriting an optimistic local state that the server hasn't seen yet.
|
||||
- **Idempotency**: `flushPendingChanges` is safe to call multiple times. Each row is
|
||||
deleted only after the server acknowledges the change.
|
||||
- **No silent data loss**: permanent server errors surface as `FailedMutation` records
|
||||
visible in the UI (Settings → Failed mutations).
|
||||
- **UI layer isolation**: `lib/ui/` never imports `lib/data/`; all interaction goes
|
||||
through `core/` interfaces. The `check-layers` Taskfile task enforces this.
|
||||
Reference in New Issue
Block a user