Three unguarded blocking calls caused CI to hang until the 60-min timeout:
- dagger query prune steps had no timeout; || true only catches errors, not hangs
- docker info (added in d905cd6) had no timeout if Docker socket is unresponsive
- until portfile loop in check-dagger spun forever if otel-receiver.py crashed
Fixes: timeout 120 on all dagger query prune calls, timeout 30 on docker info,
and a kill -0 process-alive guard on the portfile until loop with fallback.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Switch from the bespoke 1136-line Python orchestrator to the community
agentloop tool (https://github.com/guettli/agentloop). The new tool
handles the issue → agent → PR pipeline via a label state machine using
loop/plan and loop/code labels, running every 5 minutes via cron.
Removes: scripts/agent_loop.py, scripts/test_agent_loop.py
Removes: .forgejo/workflows/monitor.yml (no heartbeat concept in agentloop)
Updates: AGENTS.md to document the new loop/ label workflow
agentloop config lives in ~/agentloop/loop/sharedinbox/ on the host.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Track a heartbeat timestamp in ~/.sharedinbox-agent-heartbeat at the
start of each _run_loop() invocation so we can tell when it last ran.
- Add `agent_loop.py monitor` subcommand that exits 1 with a WARNING
message if the heartbeat is missing, corrupted, or older than 2 hours.
- Add .forgejo/workflows/monitor.yml scheduled workflow that runs the
monitor check every 2 hours on the self-hosted runner; a CI failure
serves as the warning when the loop is stalled.
- Add 7 unit tests covering all monitor / heartbeat scenarios.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When a catch-up PR merge fails (PR stays open after the merge command), the loop sets the issue to State/Question and comments on it. But on the next cron tick the same PR is still open with passing CI, so it tries again — spamming the issue with identical comments every minute.
Fix: before attempting a catch-up merge, fetch the issue's current labels via `_get_issue_labels()`. If `State/Question` is already set, skip the PR entirely.
Closes#239
Co-authored-by: Thomas SharedInbox <sharedinbox@thomas-guettler.de>
Reviewed-on: https://codeberg.org/guettli/sharedinbox/pulls/242
Issues labelled State/ToPlan are now picked up by a dedicated planning
agent before any implementation happens. The agent posts a plan as an
issue comment, then the loop transitions the label to State/Planned and
leaves a resume command in a follow-up comment. A human reviews the plan
and manually promotes the issue to State/Ready to trigger implementation.
Planning agents run at higher priority than Ready issues.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Forgejo reports deploy.yml (scheduled/dispatch) runs with event=push
and prettyref=main, identical to ci.yml push runs. The event-only
filter was insufficient — adding workflow_id == "ci.yml" prevents
deploy.yml runs from blocking or triggering false CI fix agents.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
_latest_main_ci_run() was using event != pull_request which still
matched deploy.yml schedule runs when their prettyref == "main",
blocking the loop from picking up new issues.
_latest_ci_run_for_branch() had the same issue: the else branch matched
any non-pull_request event including schedule runs.
Both functions now explicitly filter for event == "push" only.
Tests updated: rename _latest_ci_run → _latest_main_ci_run, mock
_open_issue_prs to prevent real API calls in unit tests, and update
_find_pr_for_branch side_effect to reflect the upstream post-merge
PR-still-open verification check.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The previous tests patched google_auth_httplib2 and googleapiclient which
no longer exist in the new implementation. Rewrite to mock AuthorizedSession
and _upload_aab_resumable, covering the same scenarios: happy path, retry
on transient errors, backoff delays, and exhausted attempts.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
httplib2 treats 308 Resume Incomplete responses (used by Google's
resumable upload API) as redirects and raises RedirectMissingLocation
when the response lacks a Location header. Switch to
google.auth.transport.requests.AuthorizedSession + direct HTTP calls
so the upload uses the requests library, which handles 308 correctly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Wrap the resumable bundle upload in a loop of up to _MAX_UPLOAD_ATTEMPTS (3)
attempts. On httplib2.error.RedirectMissingLocation, recreate MediaFileUpload
(resumable uploads cannot reuse the same object) and wait 10 s / 20 s before
retrying. After all attempts are exhausted, raise RuntimeError chained to the
last exception. Add tests covering the retry path, backoff delays, fresh
MediaFileUpload on each attempt, and exhaustion.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Switch deploy_playstore.py from requests/AuthorizedSession to the
googleapiclient.discovery client with google-auth-httplib2, so that
AuthorizedHttp(timeout=300) enforces a hard socket timeout on all
requests and num_retries=3 on every .execute() call enables automatic
retries for transient failures.
Update flake.nix and ci/main.go to install the new dependencies
(google-api-python-client, google-auth-httplib2, httplib2) instead of
the old google-auth + requests pair.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The Forgejo workflow_runs API has no head_branch field. For pull_request
events the branch lives in event_payload["pull_request"]["head"]["ref"];
for push events it is in prettyref. The old code used run.get("head_branch")
which always returned None, causing _latest_ci_run_for_branch to never find
the run and the loop to declare "no CI run after 15 min" and set the issue to
State/Question — even when CI had already passed.
Also fixes a pre-existing test mock that was missing the session_name kwarg.
Adds TestLatestCiRunForBranch covering both event types and the regression.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The Dagger container running generate_build_history.py may not always
reach the deployment server (network constraints on the Dagger engine).
Rather than aborting the entire publish-website pipeline, log the SSH
verbose output (already added in the previous debug commit) and return
an empty file list so Hugo still builds and rsync still deploys the
site — just without updated build-history pages.
This unblocks the cron deploy that has been failing since c259d2da.
Temporary: print verbose SSH output on failure to identify why the
connection fails from inside the dagger container.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
All other ssh/scp calls in the dagger module use explicit -i /root/.ssh/id_ed25519.
This one was missing it, causing exit 255 inside the dagger container.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- agent_loop.py: create log dir with mode 0700 and enforce it on
existing dirs; open log files with mode 0600; chmod state file
to 0600 after every write. Prevents other local processes from
reading agent output (which may contain credential paths) or
tampering with the state file's pid field.
- ci/main.go (TestAndroidFirebase): replace
echo "$FIREBASE_SA_KEY" > /tmp/key.json
with bash process substitution
--key-file=<(echo "$FIREBASE_SA_KEY")
The key is now passed via a file descriptor — it never touches
disk, so it cannot be stranded by a failed gcloud auth call or
snapshotted into the Dagger layer cache.
- ci.yml / deploy.yml: add "Cleanup TLS credentials" step
(if: always()) at the end of every job that calls
setup_dagger_remote.sh. Removes /tmp/dagger-tls,
/tmp/stunnel-dagger.conf, /tmp/stunnel.pid from the self-hosted
runner after each job, so client certs do not accumulate between
job runs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Only pick up issues created by guettli, guettlibot, or guettlibot2
to prevent the loop from acting on external/bot issues.
- Post an explanatory comment on the issue whenever the loop sets
State/Question (agent killed, no CI run, no push detected), so the
reason is visible without digging through cron logs. Closes#158.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- agent_loop.py: agents now create an `issue-N-fix` branch and open a PR;
the loop discovers the PR via `fgj pr list`, tracks its CI run, squash-merges
on green, and falls back to the global-CI path if no PR exists (backward compat).
Adds `_find_pr_for_branch`, `_latest_ci_run_for_branch`, `_merge_pr` helpers.
- .forgejo/workflows/ci.yml: strip to the single fast `check` job only
(removes build-linux, deploy-playstore, publish-website).
- .forgejo/workflows/deploy.yml (new, replaces android-emulator-tests.yml):
scheduled hourly + workflow_dispatch; runs firebase tests, Play Store deploy,
Linux build/deploy, website publish; on completion sets CI/Full-Pass or
CI/Full-Fail label on the repo's DEPLOY_HEALTH_ISSUE tracking issue.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
If the agent exits immediately (e.g. rate-limit), the loop was closing the
pending issue against the *previous* CI run, which was still green.
Fix: record the latest CI run ID when an issue agent starts. If the run ID
hasn't changed when the agent exits, the agent pushed nothing → set
State/Question instead of closing.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Capture gcloud auth stderr separately and fail on unexpected output;
ignore the two known informational lines ("Activated service account
credentials for: [...]" and "Updated property [core/project].") while
keeping a strict "fail if unknown stderr" check for anything else.
- Replace the narrow pattern grep (non-retryable error|infrastructure_failure|
test execution failed) with a broad whole-word case-insensitive grep for
'error', so any infrastructure or Firebase error in the output causes CI
failure.
- Verify that the number of device result rows in the result table matches
the expected device count (1), so a silent test-run failure cannot slip
through.
- Add scripts/test_firebase_check.sh with 18 unit tests for the three new
bash patterns (auth stderr filter, error-word detection, device count).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add -qq to apt-get update/install in Dagger toolchain to suppress
verbose package-list output (hundreds of lines on cold cache)
- Wrap sdkmanager in silent-on-success pattern — only shows output
on failure, like the build_runner and flutter pub get steps
- Set debug = warning in stunnel config to suppress LOG5 (info/notice)
startup lines while keeping LOG4 (warning) and above
- Add org.gradle.welcome=never to android/gradle.properties to
suppress the "Welcome to Gradle N.NN!" banner
- Filter SKIPPED Gradle tasks, Gradle Daemon startup messages, and
gcloud support-page promo lines in run_firebase_test.sh
Errors and warnings are preserved in all cases.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
`tea api` exits 0 even on 401 responses, so `_close_issue` and
`_set_labels` appeared to succeed but did nothing. Issues were never
actually closed, causing them to be picked up again every cron tick.
Switch all write operations (close issue, set labels) and issue-list
reads to `fgj`, which has proper authentication. Keep `tea api` only
for CI run fetches where `fgj` times out (504). Add ~/go/bin to the
cron PATH so fgj is found.
Also add an error check in `_tea_get` for API-level error responses,
and strip State/InProgress when closing an issue.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previously issue agents were instructed to close the issue via prompt text
immediately after pushing. If CI then failed, the issue was already closed.
Now the loop tracks a pending_issue across cron ticks:
- When an agent finishes (issue or ci-fix), the issue number is extracted
from state before it is cleared.
- If CI is still running, a "pending-ci" state preserves the issue number.
- If CI fails, the ci-fix agent is started with the issue number in state
so it survives the fix cycle.
- Once CI passes, _close_issue() is called from Python — never by the agent.
The agent prompt no longer instructs the agent to close the issue.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
