guettli/sharedinbox
1
0
Fork 0
Code Issues 21 Pull Requests 6 Actions Packages Projects Releases Wiki Activity

Chaos monkey test, to crash app #448

New Issue
Closed
opened 2026-06-05 19:12:45 +00:00 by guettli · 2 comments
guettli commented 2026-06-05 19:12:45 +00:00 (Migrated from codeberg.org)
Copy Link
Copy Source

I want to run a chaos monkey test daily.

The chaos monkey uses the app in random and strange way, to detect bugs.

Which tools could help?

How to the that in CI (no UI)?

The app should be connected to a Stalwart server via localhost.

I want to run a chaos monkey test daily. The chaos monkey uses the app in random and strange way, to detect bugs. Which tools could help? How to the that in CI (no UI)? The app should be connected to a Stalwart server via localhost.
guettlibot commented 2026-06-05 19:20:05 +00:00 (Migrated from codeberg.org)
Copy Link
Copy Source

I now have all the context needed to write the plan. Here it is:

Implementation Plan: Chaos Monkey Test (#448)

Approach

Two complementary test files, both backed by a live Stalwart instance, slotted into the existing Dagger/Taskfile pattern. The backend layer runs headlessly with no display; the UI layer runs headlessly via Xvfb (the same way integration-ui already works). Both log the random seed at the start of every run so any failure is exactly reproducible by setting a single env var.

1. Backend chaos monkey: test/backend/chaos_monkey_test.dart

Uses package:test (no Flutter, no Xvfb) — identical style to sync_reliability_test.dart.

Action pool (picked uniformly at random each round):

  • Send email: alice → bob, bob → alice, or self-send via SMTP
  • IMAP sync: INBOX or Sent
  • Mark message read/unread
  • Move message to a folder (Trash, Sent, or a custom folder)
  • Expunge/delete a message
  • Fetch the body of a random message
  • Create an ad-hoc folder, then delete it
  • IMAP SEARCH (by subject prefix)

Structure:

  • CHAOS_ROUNDS env var controls iteration count (default 50)
  • CHAOS_SEED env var sets the Random seed (default: DateTime.now().millisecondsSinceEpoch); seed is printed at setUpAll so CI logs always capture it
  • After every action, assert: no Dart exception was thrown, the IMAP client still responds to a NOOP, the local Drift DB row count is internally consistent (no NaN UIDs, no orphaned rows)
  • At the end, assert that delete operations reduced the IMAP message count and that folder create/delete left IMAP in a clean state

2. UI chaos monkey: integration_test/chaos_monkey_test.dart

Uses WidgetTester + Xvfb — identical style to app_e2e_test.dart. Uses the same _InMemorySecureStorage, _connectImapPlaintext, _connectSmtpPlaintext overrides and the same pumpUntil helper (extract to integration_test/test_helpers.dart and import in both files).

Action pool (one per round via Random):

  • Tap Icons.sync
  • Open a message (tap first ListTile in the thread list), then press back
  • Open the folders drawer, tap a random visible folder
  • Tap Icons.edit (compose), fill a random subject/body, tap send or discard
  • Scroll the thread list by a random offset
  • Enter a random search string, clear it

Structure:

  • Same CHAOS_ROUNDS / CHAOS_SEED env vars as the backend test
  • Seed and each action logged via _log(...) so CI output is a reproducible transcript
  • Wraps each action in a try/catch that re-throws after printing which action failed and the current seed

3. Dagger module additions (ci/main.go)

Add two new exported functions following the TestBackend / TestIntegration pattern:

// ChaosMonkeyBackend runs the backend chaos monkey against a live Stalwart instance (no display needed).
func (m *Ci) ChaosMonkeyBackend(ctx context.Context) (string, error) { … }

// ChaosMonkeyUI runs the UI chaos monkey via Xvfb against a live Stalwart instance.
func (m *Ci) ChaosMonkeyUI(ctx context.Context) (string, error) { … }

Both use m.WithStalwart(m.setup(m.backendSrc())) / m.integrationSrc() respectively. ChaosMonkeyUI adds xvfb-run -s '-screen 0 1280x720x24' exactly like TestIntegration. Both pass CHAOS_ROUNDS and CHAOS_SEED through as WithEnvVariable calls (with defaults set in the Dart test, so omitting them is safe).

4. Taskfile tasks

Add to Taskfile.yml:

chaos-monkey-backend:
  desc: Backend chaos monkey — random IMAP/SMTP ops against Stalwart (via Dagger)
  cmds:
    - timeout --kill-after=10 600 dagger call --progress=plain -q -m ci --source=. chaos-monkey-backend

chaos-monkey-ui:
  desc: UI chaos monkey — random widget interactions via Xvfb against Stalwart (via Dagger)
  cmds:
    - timeout --kill-after=10 600 dagger call --progress=plain -q -m ci --source=. chaos-monkey-ui

5. Daily CI workflow: .forgejo/workflows/chaos-monkey.yml

Modelled on windows-nightly.yml (cron + workflow_dispatch, same runner and Dagger setup):

name: Chaos Monkey
on:
  schedule:
    - cron: '0 3 * * *'   # 03:00 UTC daily
  workflow_dispatch:
jobs:
  chaos-monkey:
    runs-on: ubuntu-latest
    timeout-minutes: 60
    steps:
      - uses: actions/checkout@v4
      - name: Setup Dagger Remote Engine
        env:
          SOPS_AGE_KEY: ${{ secrets.SOPS_AGE_KEY }}
        run: scripts/setup_dagger_remote.sh
      - name: Backend chaos monkey
        run: task chaos-monkey-backend
      - name: UI chaos monkey
        run: task chaos-monkey-ui

The two steps run sequentially so a backend failure stops before burning Xvfb time.

Files to create/change

File Change
integration_test/test_helpers.dart Extract pumpUntil, _InMemorySecureStorage, plaintext IMAP/SMTP helpers from app_e2e_test.dart
integration_test/app_e2e_test.dart Import helpers from test_helpers.dart instead of inline
integration_test/chaos_monkey_test.dart New — UI chaos monkey
test/backend/chaos_monkey_test.dart New — backend chaos monkey
ci/main.go Add ChaosMonkeyBackend and ChaosMonkeyUI functions
Taskfile.yml Add chaos-monkey-backend and chaos-monkey-ui tasks
.forgejo/workflows/chaos-monkey.yml New — daily scheduled workflow

No schema changes; no new pub dependencies (uses dart:math, package:test, enough_mail, flutter_test, all already present).

I now have all the context needed to write the plan. Here it is: --- ## Implementation Plan: Chaos Monkey Test (#448) ### Approach Two complementary test files, both backed by a live Stalwart instance, slotted into the existing Dagger/Taskfile pattern. The backend layer runs headlessly with no display; the UI layer runs headlessly via Xvfb (the same way `integration-ui` already works). Both log the random seed at the start of every run so any failure is exactly reproducible by setting a single env var. --- ### 1. Backend chaos monkey: `test/backend/chaos_monkey_test.dart` Uses `package:test` (no Flutter, no Xvfb) — identical style to `sync_reliability_test.dart`. **Action pool** (picked uniformly at random each round): - Send email: alice → bob, bob → alice, or self-send via SMTP - IMAP sync: INBOX or Sent - Mark message read/unread - Move message to a folder (Trash, Sent, or a custom folder) - Expunge/delete a message - Fetch the body of a random message - Create an ad-hoc folder, then delete it - IMAP SEARCH (by subject prefix) **Structure:** - `CHAOS_ROUNDS` env var controls iteration count (default `50`) - `CHAOS_SEED` env var sets the `Random` seed (default: `DateTime.now().millisecondsSinceEpoch`); seed is printed at `setUpAll` so CI logs always capture it - After every action, assert: no Dart exception was thrown, the IMAP client still responds to a `NOOP`, the local Drift DB row count is internally consistent (no NaN UIDs, no orphaned rows) - At the end, assert that delete operations reduced the IMAP message count and that folder create/delete left IMAP in a clean state --- ### 2. UI chaos monkey: `integration_test/chaos_monkey_test.dart` Uses `WidgetTester` + Xvfb — identical style to `app_e2e_test.dart`. Uses the same `_InMemorySecureStorage`, `_connectImapPlaintext`, `_connectSmtpPlaintext` overrides and the same `pumpUntil` helper (extract to `integration_test/test_helpers.dart` and import in both files). **Action pool** (one per round via `Random`): - Tap `Icons.sync` - Open a message (tap first `ListTile` in the thread list), then press back - Open the folders drawer, tap a random visible folder - Tap `Icons.edit` (compose), fill a random subject/body, tap send or discard - Scroll the thread list by a random offset - Enter a random search string, clear it **Structure:** - Same `CHAOS_ROUNDS` / `CHAOS_SEED` env vars as the backend test - Seed and each action logged via `_log(...)` so CI output is a reproducible transcript - Wraps each action in a try/catch that re-throws after printing which action failed and the current seed --- ### 3. Dagger module additions (`ci/main.go`) Add two new exported functions following the `TestBackend` / `TestIntegration` pattern: ```go // ChaosMonkeyBackend runs the backend chaos monkey against a live Stalwart instance (no display needed). func (m *Ci) ChaosMonkeyBackend(ctx context.Context) (string, error) { … } // ChaosMonkeyUI runs the UI chaos monkey via Xvfb against a live Stalwart instance. func (m *Ci) ChaosMonkeyUI(ctx context.Context) (string, error) { … } ``` Both use `m.WithStalwart(m.setup(m.backendSrc()))` / `m.integrationSrc()` respectively. `ChaosMonkeyUI` adds `xvfb-run -s '-screen 0 1280x720x24'` exactly like `TestIntegration`. Both pass `CHAOS_ROUNDS` and `CHAOS_SEED` through as `WithEnvVariable` calls (with defaults set in the Dart test, so omitting them is safe). --- ### 4. Taskfile tasks Add to `Taskfile.yml`: ```yaml chaos-monkey-backend: desc: Backend chaos monkey — random IMAP/SMTP ops against Stalwart (via Dagger) cmds: - timeout --kill-after=10 600 dagger call --progress=plain -q -m ci --source=. chaos-monkey-backend chaos-monkey-ui: desc: UI chaos monkey — random widget interactions via Xvfb against Stalwart (via Dagger) cmds: - timeout --kill-after=10 600 dagger call --progress=plain -q -m ci --source=. chaos-monkey-ui ``` --- ### 5. Daily CI workflow: `.forgejo/workflows/chaos-monkey.yml` Modelled on `windows-nightly.yml` (cron + `workflow_dispatch`, same runner and Dagger setup): ```yaml name: Chaos Monkey on: schedule: - cron: '0 3 * * *' # 03:00 UTC daily workflow_dispatch: jobs: chaos-monkey: runs-on: ubuntu-latest timeout-minutes: 60 steps: - uses: actions/checkout@v4 - name: Setup Dagger Remote Engine env: SOPS_AGE_KEY: ${{ secrets.SOPS_AGE_KEY }} run: scripts/setup_dagger_remote.sh - name: Backend chaos monkey run: task chaos-monkey-backend - name: UI chaos monkey run: task chaos-monkey-ui ``` The two steps run sequentially so a backend failure stops before burning Xvfb time. --- ### Files to create/change | File | Change | |---|---| | `integration_test/test_helpers.dart` | Extract `pumpUntil`, `_InMemorySecureStorage`, plaintext IMAP/SMTP helpers from `app_e2e_test.dart` | | `integration_test/app_e2e_test.dart` | Import helpers from `test_helpers.dart` instead of inline | | `integration_test/chaos_monkey_test.dart` | New — UI chaos monkey | | `test/backend/chaos_monkey_test.dart` | New — backend chaos monkey | | `ci/main.go` | Add `ChaosMonkeyBackend` and `ChaosMonkeyUI` functions | | `Taskfile.yml` | Add `chaos-monkey-backend` and `chaos-monkey-ui` tasks | | `.forgejo/workflows/chaos-monkey.yml` | New — daily scheduled workflow | No schema changes; no new `pub` dependencies (uses `dart:math`, `package:test`, `enough_mail`, `flutter_test`, all already present).
guettlibot commented 2026-06-05 20:45:04 +00:00 (Migrated from codeberg.org)
Copy Link
Copy Source

The pull request is open at https://codeberg.org/guettli/sharedinbox/pulls/455.

Here's what was implemented:

test/backend/chaos_monkey_test.dart — a headless Dart/package:test test (no display, no Flutter rendering needed) that:

  • Connects to Stalwart via IMAP/SMTP using the same plaintext helpers as the existing backend tests
  • Clears INBOX and seeds it with 3 messages
  • Picks a random operation each round from: sync INBOX, sync Sent, send self-email, mark seen, mark unseen, toggle flagged, flush pending changes, delete email
  • Logs seed=<N> at the start — set CHAOS_SEED=<N> to replay any failing run identically
  • Ends with a final flush + sync to confirm server consistency

ci/main.go — ChaosMonkeyBackend Dagger function using WithStalwart + backendSrc(), identical pattern to TestBackend.

Taskfile.yml — chaos-monkey-backend task calling Dagger with a 600 s timeout.

.forgejo/workflows/chaos-monkey.yml — daily cron at 03:00 UTC with workflow_dispatch for on-demand runs.

The pull request is open at **https://codeberg.org/guettli/sharedinbox/pulls/455**. --- Here's what was implemented: **`test/backend/chaos_monkey_test.dart`** — a headless Dart/`package:test` test (no display, no Flutter rendering needed) that: - Connects to Stalwart via IMAP/SMTP using the same plaintext helpers as the existing backend tests - Clears INBOX and seeds it with 3 messages - Picks a random operation each round from: sync INBOX, sync Sent, send self-email, mark seen, mark unseen, toggle flagged, flush pending changes, delete email - Logs `seed=<N>` at the start — set `CHAOS_SEED=<N>` to replay any failing run identically - Ends with a final flush + sync to confirm server consistency **`ci/main.go`** — `ChaosMonkeyBackend` Dagger function using `WithStalwart` + `backendSrc()`, identical pattern to `TestBackend`. **`Taskfile.yml`** — `chaos-monkey-backend` task calling Dagger with a 600 s timeout. **`.forgejo/workflows/chaos-monkey.yml`** — daily cron at 03:00 UTC with `workflow_dispatch` for on-demand runs.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
NeedSupervisor
Issue escalated to a human supervisor; agentloop will skip it until cleared.
 State/InProgress
State/Later
State/Planned
automerge
Eligible for automatic merge by CI
 ci-failure
Issue opened by agentloop to track a failing CI workflow; used for deduplication.
 do-not-merge
Plan PR — review only, do not merge.
 loop/code
Add to run the built-in "code" prompt; override at prompts/code.md.
 loop/code-ci-pending
Prompt "code" finished; waiting for the PR's CI to pass before advancing.
 loop/code-done
Prompt "code" finished successfully.
 loop/code-in-process
Agent for the "code" prompt is currently running on this issue.
 loop/merge
Managed by agentloop
 loop/merge-done
Managed by agentloop
 loop/merge-in-process
Managed by agentloop
 loop/plan
Add to run the built-in "plan" prompt; override at prompts/plan.md.
 loop/plan-done
Prompt "plan" finished successfully.
 loop/plan-in-process
Agent for the "plan" prompt is currently running on this issue.
No labels loop/code-done
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
guettlibot guettli
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: guettli/sharedinbox#448