sharedinbox/SYNC.md

Email Sync Architecture

This document describes the full lifecycle of an email action — from the moment the user taps a button to server confirmation — covering the IMAP IDLE loop, JMAP push/poll, the pending-change queue, exponential backoff, and the undo/cancel mechanism.

For the database schema and protocol-level implementation details see DB-SYNC.md.

---

1. Components

Component	File	Role
AccountSyncManager	lib/core/sync/account_sync_manager.dart	Owns one _SyncLoop per account; starts, stops, and wakes sync loops
_AccountSync	same file	IMAP sync loop (IDLE + incremental fetch)
_JmapAccountSync	same file	JMAP sync loop (SSE push + poll fallback)
EmailRepositoryImpl	lib/data/repositories/email_repository_impl.dart	All DB reads/writes and network calls
pending_changes table	lib/data/db/database.dart	Protocol-agnostic outbound mutation queue
UndoService	lib/core/services/undo_service.dart	Persisted undo history; cancel-or-reverse logic

---

2. Lifecycle of an email mutation (e.g. "Mark as read")

User taps "Mark as read"
        │
        ▼
EmailRepository.setFlag(id, seen: true)
        │
        ├─ 1. Write optimistic update to local DB
        │      emails.is_seen = true
        │
        └─ 2. Insert row into pending_changes
               { type: 'flag_seen', email_id: id, payload: {seen: true} }
               (IMAP: includes uid + mailboxPath for the STORE command)
               (JMAP: includes just the flag map for Email/set)

[UI immediately reflects the change via Drift's reactive streams]

        │
        ▼ (next sync cycle, triggered by IMAP IDLE / JMAP push / wakeUp)
_SyncLoop._flush() / flushPendingChanges()
        │
        ├─ IMAP: open connection → STORE uid +FLAGS (\Seen) → close
        │
        └─ JMAP: Email/set { update: { id: { keywords: { "$seen": true } } } }
               If stateMismatch → clear checkpoint → full re-sync

        │
        ▼
pending_changes row deleted on success
(on permanent error: retry count incremented; evicted after 5 failures)

---

3. IMAP sync loop

The IMAP loop runs one coroutine per account (_AccountSync):

start()
  │
  ▼
[forever loop]
  ├─ flushPendingChanges()   ← drain outbound queue first
  ├─ syncMailboxes()         ← detect new/removed mailboxes
  ├─ for each mailbox:
  │    syncEmails()          ← incremental: fetch only UIDs > lastUid
  │                             deletion reconciliation: remove rows
  │                             whose UID is absent from the server
  └─ _idle()                 ← IMAP IDLE for up to 25 min (RFC 2177)
       │                        Wakes on: server EXISTS/EXPUNGE/FLAGS
       │                        or syncNow() signal from UI
       └─ repeat

Incremental sync checkpoint — sync_state table stores (accountId, mailbox, lastUid, uidValidity). On each run, only UIDs greater than lastUid are fetched. If uidValidity changes the full folder is re-scanned and the checkpoint is reset.

IDLE cap — IDLE sessions are limited to 25 minutes per the RFC. The loop also wakes immediately if syncNow() is called (e.g. user pulls-to-refresh).

---

4. JMAP sync loop

The JMAP loop (_JmapAccountSync) follows a similar structure but uses HTTP:

start()
  │
  ▼
[forever loop]
  ├─ flushPendingChanges()   ← Email/set for queued mutations
  ├─ syncMailboxes()         ← Mailbox/get or Mailbox/changes
  ├─ for each mailbox:
  │    syncEmails()          ← Email/query + Email/get (first run)
  │                             Email/changes (subsequent runs, state token)
  └─ _wait()
       ├─ If server advertises eventSourceUrl: subscribe to SSE push
       │    wake on "Email" change event
       └─ Otherwise: sleep 30 s (poll fallback)

State tokens — each Mailbox/changes / Email/changes call uses the server-provided state token stored in sync_state. A stateMismatch error clears the token and triggers a full re-fetch.

JMAP send — outgoing mail uses EmailSubmission/set when the server advertises the urn:ietf:params:jmap:submission capability; falls back to SMTP otherwise.

---

5. Exponential backoff

Both loops share the same backoff policy:

Outcome	Backoff
Sync succeeded	Reset to 5 s
Network / server error	Double previous backoff, capped at 900 s (15 min)

The backoff counter (_backoffSeconds) is per-account and per-process; it resets to 5 s on the next successful cycle.

The last error message is written to sync_log and surfaced in the UI via syncLastErrorProvider (the red MaterialBanner in the email list).

---

6. Pending-change queue

pending_changes is a protocol-agnostic table that stores every outbound mutation before it reaches the server:

Column	Description
id	Auto-increment primary key
email_id	The email being mutated
type	flag_seen, flag_flagged, move, delete, snooze
payload	JSON-encoded protocol-specific arguments
retry_count	Incremented on each failed flush attempt
created_at	For ordering and debug

Optimistic UI — every mutation writes the local change first, then inserts into pending_changes. The Drift reactive stream delivers the update to the UI before the network round-trip completes.

Conflict resolution — the server always wins. On the next sync cycle the server's state overwrites local rows. Outbound mutations are retried up to 5 times; after that they are evicted and a FailedMutation record is created. Permanent per-item JMAP errors (notFound, forbidden) skip the retry counter and evict immediately.

---

7. Undo and cancel

When the user triggers an undoable action the UI calls:

ref.read(undoServiceProvider.notifier).pushAction(UndoAction(...))

UndoService persists the action to the undo_actions table (max 10 entries, FIFO). A SnackBar with an Undo button appears for a few seconds.

When the user taps Undo, UndoService.undo() executes this sequence for each affected email:

1. cancelPendingChange(id, originalType)
      └─ Deletes the pending_changes row if it has not been flushed yet.
         Returns true if cancelled, false if the server already processed it.

2. If the email row was hard-deleted (DELETE action):
      restoreEmails([original])
         └─ Re-inserts the row with its pre-deletion state,
            placed in the correct mailbox (source if cancelled, dest otherwise).

3. moveEmail(id, sourceMailboxPath)
      └─ Optimistic local move back to the original folder.
         If step 1 returned false (already sent to server), this enqueues
         a reverse-move in pending_changes so the server move is undone too.

4. If step 1 returned true (cancelled before flush):
      cancelPendingChange(id, 'move')
         └─ The reverse-move from step 3 is redundant; remove it.

The net result is: if the mutation was still in the queue it is silently cancelled with no server round-trip; if it had already been flushed, a compensating move is queued.

---

8. Key invariants

• Order: pending changes are flushed before syncing. This prevents the server from overwriting an optimistic local state that the server hasn't seen yet.
• Idempotency: flushPendingChanges is safe to call multiple times. Each row is deleted only after the server acknowledges the change.
• No silent data loss: permanent server errors surface as FailedMutation records visible in the UI (Settings → Failed mutations).
• UI layer isolation: lib/ui/ never imports lib/data/; all interaction goes through core/ interfaces. The check-layers Taskfile task enforces this.