sharedinbox/SYNC.md

Email Sync Architecture

This document describes the full lifecycle of an email action — from the moment the user taps a button to server confirmation — covering the IMAP IDLE loop, JMAP push/poll, the pending-change queue, exponential backoff, and the undo/cancel mechanism.

For the database schema and protocol-level implementation details see DB-SYNC.md.

---

1. Components

Component	File	Role
AccountSyncManager	lib/core/sync/account_sync_manager.dart	Owns one _SyncLoop per account; starts, stops, and wakes sync loops
_AccountSync	same file	IMAP sync loop (IDLE + incremental fetch)
_JmapAccountSync	same file	JMAP sync loop (SSE push + poll fallback)
EmailRepositoryImpl	lib/data/repositories/email_repository_impl.dart	All DB reads/writes and network calls
pending_changes table	lib/data/db/database.dart	Protocol-agnostic outbound mutation queue
UndoService	lib/core/services/undo_service.dart	Persisted undo history; cancel-or-reverse logic

---

2. Lifecycle of an email mutation (e.g. "Mark as read")

User taps "Mark as read"
        │
        ▼
EmailRepository.setFlag(id, seen: true)
        │
        ├─ 1. Write optimistic update to local DB
        │      emails.is_seen = true
        │
        └─ 2. Insert row into pending_changes
               { type: 'flag_seen', email_id: id, payload: {seen: true} }
               (IMAP: includes uid + mailboxPath for the STORE command)
               (JMAP: includes just the flag map for Email/set)

[UI immediately reflects the change via Drift's reactive streams]

        │
        ▼ (next sync cycle, triggered by IMAP IDLE / JMAP push / wakeUp)
_SyncLoop._flush() / flushPendingChanges()
        │
        ├─ IMAP: open connection → STORE uid +FLAGS (\Seen) → close
        │
        └─ JMAP: Email/set { update: { id: { keywords: { "$seen": true } } } }
               If stateMismatch → clear checkpoint → full re-sync

        │
        ▼
pending_changes row deleted on success
(on permanent error: retry count incremented; evicted after 5 failures)

---

3. IMAP sync loop

The IMAP loop runs one coroutine per account (_AccountSync):

start()
  │
  ▼
[forever loop]
  ├─ flushPendingChanges()   ← drain outbound queue first
  ├─ syncMailboxes()         ← detect new/removed mailboxes
  ├─ for each mailbox:
  │    syncEmails()          ← incremental: fetch only UIDs > lastUid
  │                             deletion reconciliation: remove rows
  │                             whose UID is absent from the server
  └─ _idle()                 ← IMAP IDLE for up to 25 min (RFC 2177)
       │                        Wakes on: server EXISTS/EXPUNGE/FLAGS
       │                        or syncNow() signal from UI
       └─ repeat

Incremental sync checkpoint — sync_state table stores (accountId, mailbox, lastUid, uidValidity). On each run, only UIDs greater than lastUid are fetched. If uidValidity changes the full folder is re-scanned and the checkpoint is reset.

IMAP move remap — IMAP UIDs are mailbox-scoped, so a moved message gets a new UID in its destination folder. When a move/snooze/unsnooze change is flushed, the local row id (accountId:mailboxPath:uid) is rewritten in place to point at the new UID. The new UID is taken from the RFC 4315 COPYUID response code returned by MOVE; if the server does not advertise UIDPLUS, a UID SEARCH HEADER Message-ID … in the destination mailbox is used as a fallback. email_bodies, threads, pending_changes, and undo_actions rows that reference the old id are updated atomically so cached bodies and pending undo operations keep tracking the same physical message. Deletion reconciliation also skips rows whose move is still queued, so the optimistic local move never gets wiped mid-flight.

IDLE cap — IDLE sessions are limited to 25 minutes per the RFC. The loop also wakes immediately if syncNow() is called (e.g. user pulls-to-refresh).

---

4. JMAP sync loop

The JMAP loop (_JmapAccountSync) follows a similar structure but uses HTTP:

start()
  │
  ▼
[forever loop]
  ├─ flushPendingChanges()   ← Email/set for queued mutations
  ├─ syncMailboxes()         ← Mailbox/get or Mailbox/changes
  ├─ for each mailbox:
  │    syncEmails()          ← Email/query + Email/get (first run)
  │                             Email/changes (subsequent runs, state token)
  └─ _wait()
       ├─ If server advertises eventSourceUrl: subscribe to SSE push
       │    wake on "Email" change event
       └─ Otherwise: sleep 30 s (poll fallback)

State tokens — each Mailbox/changes / Email/changes call uses the server-provided state token stored in sync_state. A stateMismatch error clears the token and triggers a full re-fetch.

JMAP send — outgoing mail uses EmailSubmission/set when the server advertises the urn:ietf:params:jmap:submission capability; falls back to SMTP otherwise.

---

5. Exponential backoff

Both loops share the same backoff policy:

Outcome	Backoff
Sync succeeded	Reset to 5 s
Network / server error	Double previous backoff, capped at 900 s (15 min)

The backoff counter (_backoffSeconds) is per-account and per-process; it resets to 5 s on the next successful cycle.

The last error message is written to sync_log and surfaced in the UI via syncLastErrorProvider (the red MaterialBanner in the email list).

---

6. Pending-change queue

pending_changes is a protocol-agnostic table that stores every outbound mutation before it reaches the server:

Column	Description
id	Auto-increment primary key
email_id	The email being mutated
type	flag_seen, flag_flagged, move, delete, snooze
payload	JSON-encoded protocol-specific arguments
retry_count	Incremented on each failed flush attempt
created_at	For ordering and debug

Optimistic UI — every mutation writes the local change first, then inserts into pending_changes. The Drift reactive stream delivers the update to the UI before the network round-trip completes.

Conflict resolution — the server always wins. On the next sync cycle the server's state overwrites local rows. Outbound mutations are retried up to 5 times; after that they are evicted and a FailedMutation record is created. Permanent per-item JMAP errors (notFound, forbidden) skip the retry counter and evict immediately.

---

7. Undo and cancel

When the user triggers an undoable action the UI calls:

ref.read(undoServiceProvider.notifier).pushAction(UndoAction(...))

UndoService persists the action to the undo_actions table (max 10 entries, FIFO). A SnackBar with an Undo button appears for a few seconds.

When the user taps Undo, UndoService.undo() executes this sequence for each affected email:

1. cancelPendingChange(id, originalType)
      └─ Deletes the pending_changes row if it has not been flushed yet.
         Returns true if cancelled, false if the server already processed it.

2. If the email row was hard-deleted (DELETE action):
      restoreEmails([original])
         └─ Re-inserts the row with its pre-deletion state,
            placed in the correct mailbox (source if cancelled, dest otherwise).

3. moveEmail(id, sourceMailboxPath)
      └─ Optimistic local move back to the original folder.
         If step 1 returned false (already sent to server), this enqueues
         a reverse-move in pending_changes so the server move is undone too.

4. If step 1 returned true (cancelled before flush):
      cancelPendingChange(id, 'move')
         └─ The reverse-move from step 3 is redundant; remove it.

The net result is: if the mutation was still in the queue it is silently cancelled with no server round-trip; if it had already been flushed, a compensating move is queued.

---

8. Key invariants

• Order: pending changes are flushed before syncing. This prevents the server from overwriting an optimistic local state that the server hasn't seen yet.
• Idempotency: flushPendingChanges is safe to call multiple times. Each row is deleted only after the server acknowledges the change.
• No silent data loss: permanent server errors surface as FailedMutation records visible in the UI (Settings → Failed mutations).
• UI layer isolation: lib/ui/ never imports lib/data/; all interaction goes through core/ interfaces. The check-layers Taskfile task enforces this.