Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Thomas SharedInboxandClaude Sonnet 4.6
|
a6ad4183f6 |
@@ -0,0 +1,206 @@
|
||||
# Email Sync Architecture
|
||||
|
||||
This document describes the full lifecycle of an email action — from the moment the user taps
|
||||
a button to server confirmation — covering the IMAP IDLE loop, JMAP push/poll, the pending-change
|
||||
queue, exponential backoff, and the undo/cancel mechanism.
|
||||
|
||||
For the database schema and protocol-level implementation details see [DB-SYNC.md](DB-SYNC.md).
|
||||
|
||||
---
|
||||
|
||||
## 1. Components
|
||||
|
||||
| Component | File | Role |
|
||||
|-----------|------|------|
|
||||
| `AccountSyncManager` | `lib/core/sync/account_sync_manager.dart` | Owns one `_SyncLoop` per account; starts, stops, and wakes sync loops |
|
||||
| `_AccountSync` | same file | IMAP sync loop (IDLE + incremental fetch) |
|
||||
| `_JmapAccountSync` | same file | JMAP sync loop (SSE push + poll fallback) |
|
||||
| `EmailRepositoryImpl` | `lib/data/repositories/email_repository_impl.dart` | All DB reads/writes and network calls |
|
||||
| `pending_changes` table | `lib/data/db/database.dart` | Protocol-agnostic outbound mutation queue |
|
||||
| `UndoService` | `lib/core/services/undo_service.dart` | Persisted undo history; cancel-or-reverse logic |
|
||||
|
||||
---
|
||||
|
||||
## 2. Lifecycle of an email mutation (e.g. "Mark as read")
|
||||
|
||||
```
|
||||
User taps "Mark as read"
|
||||
│
|
||||
▼
|
||||
EmailRepository.setFlag(id, seen: true)
|
||||
│
|
||||
├─ 1. Write optimistic update to local DB
|
||||
│ emails.is_seen = true
|
||||
│
|
||||
└─ 2. Insert row into pending_changes
|
||||
{ type: 'flag_seen', email_id: id, payload: {seen: true} }
|
||||
(IMAP: includes uid + mailboxPath for the STORE command)
|
||||
(JMAP: includes just the flag map for Email/set)
|
||||
|
||||
[UI immediately reflects the change via Drift's reactive streams]
|
||||
|
||||
│
|
||||
▼ (next sync cycle, triggered by IMAP IDLE / JMAP push / wakeUp)
|
||||
_SyncLoop._flush() / flushPendingChanges()
|
||||
│
|
||||
├─ IMAP: open connection → STORE uid +FLAGS (\Seen) → close
|
||||
│
|
||||
└─ JMAP: Email/set { update: { id: { keywords: { "$seen": true } } } }
|
||||
If stateMismatch → clear checkpoint → full re-sync
|
||||
|
||||
│
|
||||
▼
|
||||
pending_changes row deleted on success
|
||||
(on permanent error: retry count incremented; evicted after 5 failures)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. IMAP sync loop
|
||||
|
||||
The IMAP loop runs one coroutine per account (`_AccountSync`):
|
||||
|
||||
```
|
||||
start()
|
||||
│
|
||||
▼
|
||||
[forever loop]
|
||||
├─ flushPendingChanges() ← drain outbound queue first
|
||||
├─ syncMailboxes() ← detect new/removed mailboxes
|
||||
├─ for each mailbox:
|
||||
│ syncEmails() ← incremental: fetch only UIDs > lastUid
|
||||
│ deletion reconciliation: remove rows
|
||||
│ whose UID is absent from the server
|
||||
└─ _idle() ← IMAP IDLE for up to 25 min (RFC 2177)
|
||||
│ Wakes on: server EXISTS/EXPUNGE/FLAGS
|
||||
│ or syncNow() signal from UI
|
||||
└─ repeat
|
||||
```
|
||||
|
||||
**Incremental sync checkpoint** — `sync_state` table stores `(accountId, mailbox, lastUid, uidValidity)`.
|
||||
On each run, only UIDs greater than `lastUid` are fetched. If `uidValidity` changes the full
|
||||
folder is re-scanned and the checkpoint is reset.
|
||||
|
||||
**IDLE cap** — IDLE sessions are limited to 25 minutes per the RFC. The loop also wakes
|
||||
immediately if `syncNow()` is called (e.g. user pulls-to-refresh).
|
||||
|
||||
---
|
||||
|
||||
## 4. JMAP sync loop
|
||||
|
||||
The JMAP loop (`_JmapAccountSync`) follows a similar structure but uses HTTP:
|
||||
|
||||
```
|
||||
start()
|
||||
│
|
||||
▼
|
||||
[forever loop]
|
||||
├─ flushPendingChanges() ← Email/set for queued mutations
|
||||
├─ syncMailboxes() ← Mailbox/get or Mailbox/changes
|
||||
├─ for each mailbox:
|
||||
│ syncEmails() ← Email/query + Email/get (first run)
|
||||
│ Email/changes (subsequent runs, state token)
|
||||
└─ _wait()
|
||||
├─ If server advertises eventSourceUrl: subscribe to SSE push
|
||||
│ wake on "Email" change event
|
||||
└─ Otherwise: sleep 30 s (poll fallback)
|
||||
```
|
||||
|
||||
**State tokens** — each `Mailbox/changes` / `Email/changes` call uses the server-provided
|
||||
`state` token stored in `sync_state`. A `stateMismatch` error clears the token and triggers
|
||||
a full re-fetch.
|
||||
|
||||
**JMAP send** — outgoing mail uses `EmailSubmission/set` when the server advertises the
|
||||
`urn:ietf:params:jmap:submission` capability; falls back to SMTP otherwise.
|
||||
|
||||
---
|
||||
|
||||
## 5. Exponential backoff
|
||||
|
||||
Both loops share the same backoff policy:
|
||||
|
||||
| Outcome | Backoff |
|
||||
|---------|---------|
|
||||
| Sync succeeded | Reset to 5 s |
|
||||
| Network / server error | Double previous backoff, capped at 900 s (15 min) |
|
||||
|
||||
The backoff counter (`_backoffSeconds`) is per-account and per-process; it resets to 5 s
|
||||
on the next successful cycle.
|
||||
|
||||
The last error message is written to `sync_log` and surfaced in the UI via
|
||||
`syncLastErrorProvider` (the red `MaterialBanner` in the email list).
|
||||
|
||||
---
|
||||
|
||||
## 6. Pending-change queue
|
||||
|
||||
`pending_changes` is a protocol-agnostic table that stores every outbound mutation before it
|
||||
reaches the server:
|
||||
|
||||
| Column | Description |
|
||||
|--------|-------------|
|
||||
| `id` | Auto-increment primary key |
|
||||
| `email_id` | The email being mutated |
|
||||
| `type` | `flag_seen`, `flag_flagged`, `move`, `delete`, `snooze` |
|
||||
| `payload` | JSON-encoded protocol-specific arguments |
|
||||
| `retry_count` | Incremented on each failed flush attempt |
|
||||
| `created_at` | For ordering and debug |
|
||||
|
||||
**Optimistic UI** — every mutation writes the local change first, then inserts into
|
||||
`pending_changes`. The Drift reactive stream delivers the update to the UI before
|
||||
the network round-trip completes.
|
||||
|
||||
**Conflict resolution** — the server always wins. On the next sync cycle the server's
|
||||
state overwrites local rows. Outbound mutations are retried up to 5 times; after that
|
||||
they are evicted and a `FailedMutation` record is created. Permanent per-item JMAP
|
||||
errors (`notFound`, `forbidden`) skip the retry counter and evict immediately.
|
||||
|
||||
---
|
||||
|
||||
## 7. Undo and cancel
|
||||
|
||||
When the user triggers an undoable action the UI calls:
|
||||
```
|
||||
ref.read(undoServiceProvider.notifier).pushAction(UndoAction(...))
|
||||
```
|
||||
|
||||
`UndoService` persists the action to the `undo_actions` table (max 10 entries, FIFO).
|
||||
A `SnackBar` with an **Undo** button appears for a few seconds.
|
||||
|
||||
When the user taps Undo, `UndoService.undo()` executes this sequence for each affected email:
|
||||
|
||||
```
|
||||
1. cancelPendingChange(id, originalType)
|
||||
└─ Deletes the pending_changes row if it has not been flushed yet.
|
||||
Returns true if cancelled, false if the server already processed it.
|
||||
|
||||
2. If the email row was hard-deleted (DELETE action):
|
||||
restoreEmails([original])
|
||||
└─ Re-inserts the row with its pre-deletion state,
|
||||
placed in the correct mailbox (source if cancelled, dest otherwise).
|
||||
|
||||
3. moveEmail(id, sourceMailboxPath)
|
||||
└─ Optimistic local move back to the original folder.
|
||||
If step 1 returned false (already sent to server), this enqueues
|
||||
a reverse-move in pending_changes so the server move is undone too.
|
||||
|
||||
4. If step 1 returned true (cancelled before flush):
|
||||
cancelPendingChange(id, 'move')
|
||||
└─ The reverse-move from step 3 is redundant; remove it.
|
||||
```
|
||||
|
||||
The net result is: if the mutation was still in the queue it is silently cancelled with no
|
||||
server round-trip; if it had already been flushed, a compensating move is queued.
|
||||
|
||||
---
|
||||
|
||||
## 8. Key invariants
|
||||
|
||||
- **Order**: pending changes are flushed before syncing. This prevents the server from
|
||||
overwriting an optimistic local state that the server hasn't seen yet.
|
||||
- **Idempotency**: `flushPendingChanges` is safe to call multiple times. Each row is
|
||||
deleted only after the server acknowledges the change.
|
||||
- **No silent data loss**: permanent server errors surface as `FailedMutation` records
|
||||
visible in the UI (Settings → Failed mutations).
|
||||
- **UI layer isolation**: `lib/ui/` never imports `lib/data/`; all interaction goes
|
||||
through `core/` interfaces. The `check-layers` Taskfile task enforces this.
|
||||
@@ -1,158 +0,0 @@
|
||||
import 'package:flutter/material.dart';
|
||||
import 'package:flutter_riverpod/flutter_riverpod.dart';
|
||||
import 'package:flutter_test/flutter_test.dart';
|
||||
|
||||
import 'package:sharedinbox/core/models/email.dart';
|
||||
import 'package:sharedinbox/di.dart';
|
||||
|
||||
import 'helpers.dart';
|
||||
|
||||
// Fixed-date emails so golden files don't change day to day.
|
||||
final _kDate = DateTime(2024, 6);
|
||||
|
||||
Email _email({
|
||||
String id = 'acc-1:1',
|
||||
String subject = 'Hello world',
|
||||
bool isSeen = true,
|
||||
bool isFlagged = false,
|
||||
}) =>
|
||||
Email(
|
||||
id: id,
|
||||
accountId: 'acc-1',
|
||||
mailboxPath: 'INBOX',
|
||||
uid: int.parse(id.split(':').last),
|
||||
subject: subject,
|
||||
receivedAt: _kDate,
|
||||
sentAt: _kDate,
|
||||
from: const [EmailAddress(name: 'Bob', email: 'bob@example.com')],
|
||||
to: const [EmailAddress(email: 'alice@example.com')],
|
||||
cc: const [],
|
||||
isSeen: isSeen,
|
||||
isFlagged: isFlagged,
|
||||
hasAttachment: false,
|
||||
);
|
||||
|
||||
List<Override> _overrides({
|
||||
List<Email> emails = const [],
|
||||
List<Email> searchResults = const [],
|
||||
String? syncError,
|
||||
}) =>
|
||||
[
|
||||
accountRepositoryProvider.overrideWithValue(
|
||||
FakeAccountRepository([kTestAccount]),
|
||||
),
|
||||
mailboxRepositoryProvider.overrideWithValue(
|
||||
FakeMailboxRepository([kTestMailbox]),
|
||||
),
|
||||
emailRepositoryProvider.overrideWithValue(
|
||||
FakeEmailRepository(emails: emails, searchResults: searchResults),
|
||||
),
|
||||
draftRepositoryProvider.overrideWithValue(FakeDraftRepository()),
|
||||
searchHistoryRepositoryProvider.overrideWithValue(
|
||||
FakeSearchHistoryRepository(),
|
||||
),
|
||||
syncLastErrorProvider.overrideWith(
|
||||
(ref, _) => Stream.value(syncError),
|
||||
),
|
||||
];
|
||||
|
||||
void main() {
|
||||
group('EmailListScreen goldens', () {
|
||||
testWidgets('golden: empty state', (tester) async {
|
||||
await tester.pumpWidget(
|
||||
buildApp(
|
||||
initialLocation: '/accounts/acc-1/mailboxes/INBOX/emails',
|
||||
overrides: _overrides(),
|
||||
),
|
||||
);
|
||||
await tester.pumpAndSettle();
|
||||
|
||||
await expectLater(
|
||||
find.byType(MaterialApp),
|
||||
matchesGoldenFile('goldens/email_list_empty.png'),
|
||||
);
|
||||
});
|
||||
|
||||
testWidgets('golden: list with emails', (tester) async {
|
||||
await tester.pumpWidget(
|
||||
buildApp(
|
||||
initialLocation: '/accounts/acc-1/mailboxes/INBOX/emails',
|
||||
overrides: _overrides(
|
||||
emails: [
|
||||
_email(subject: 'Team standup notes', isSeen: false),
|
||||
_email(id: 'acc-1:2', subject: 'Q3 review', isFlagged: true),
|
||||
_email(id: 'acc-1:3', subject: 'Welcome to the project'),
|
||||
],
|
||||
),
|
||||
),
|
||||
);
|
||||
await tester.pumpAndSettle();
|
||||
|
||||
await expectLater(
|
||||
find.byType(MaterialApp),
|
||||
matchesGoldenFile('goldens/email_list_with_emails.png'),
|
||||
);
|
||||
});
|
||||
|
||||
testWidgets('golden: selection mode', (tester) async {
|
||||
await tester.pumpWidget(
|
||||
buildApp(
|
||||
initialLocation: '/accounts/acc-1/mailboxes/INBOX/emails',
|
||||
overrides: _overrides(
|
||||
emails: [
|
||||
_email(subject: 'Team standup notes', isSeen: false),
|
||||
_email(id: 'acc-1:2', subject: 'Q3 review'),
|
||||
],
|
||||
),
|
||||
),
|
||||
);
|
||||
await tester.pumpAndSettle();
|
||||
|
||||
await tester.longPress(find.text('Team standup notes'));
|
||||
await tester.pumpAndSettle();
|
||||
|
||||
await expectLater(
|
||||
find.byType(MaterialApp),
|
||||
matchesGoldenFile('goldens/email_list_selection.png'),
|
||||
);
|
||||
});
|
||||
|
||||
testWidgets('golden: search with results', (tester) async {
|
||||
await tester.pumpWidget(
|
||||
buildApp(
|
||||
initialLocation: '/accounts/acc-1/mailboxes/INBOX/emails',
|
||||
overrides: _overrides(
|
||||
searchResults: [
|
||||
_email(id: 'acc-1:5', subject: 'Project proposal'),
|
||||
],
|
||||
),
|
||||
),
|
||||
);
|
||||
await tester.pumpAndSettle();
|
||||
|
||||
await tester.enterText(find.byType(SearchBar), 'project');
|
||||
await tester.testTextInput.receiveAction(TextInputAction.search);
|
||||
await tester.pumpAndSettle();
|
||||
|
||||
await expectLater(
|
||||
find.byType(MaterialApp),
|
||||
matchesGoldenFile('goldens/email_list_search_results.png'),
|
||||
);
|
||||
});
|
||||
|
||||
testWidgets('golden: error banner', (tester) async {
|
||||
await tester.pumpWidget(
|
||||
buildApp(
|
||||
initialLocation: '/accounts/acc-1/mailboxes/INBOX/emails',
|
||||
overrides: _overrides(syncError: 'Connection refused'),
|
||||
),
|
||||
);
|
||||
await tester.pumpAndSettle();
|
||||
|
||||
await expectLater(
|
||||
find.byType(MaterialApp),
|
||||
matchesGoldenFile('goldens/email_list_error_banner.png'),
|
||||
);
|
||||
});
|
||||
});
|
||||
}
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 32 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 33 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 32 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 33 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 33 KiB |
Reference in New Issue
Block a user