AI agents that learn from their failed runs
Two Markdown files beside your project carry what the last session worked out: a rolling hand-off brief and a rule-book of what works here. Both are read back on the next run — and with the teacher turned on, a run that failed three times writes the next rule itself.
Why does my AI coding agent keep making the same mistake?
Because nothing wrote the lesson down. An agent starts every run with an empty context window, so a correction you gave it yesterday is gone today unless it lives in a file the next run reads — which is why every guide on this tells you to hand-maintain AGENTS.md, CLAUDE.md or .cursorrules. PortBay keeps two such files for you: .portbay/HANDOFF.md, a rolling brief of what the last session did, and .portbay/LEARNINGS.md, an index of one-line rules each with a Why. Both are inlined into the next dispatch prompt, filtered to the card being worked. When a card fails three times it stops retrying and parks in Blocked for you. Turn the teacher on and that third strike also fires one post-mortem call through the agent CLI you already pay for — no API key — which writes the lesson as a rule.
Verified against the product, 2026-09-06.
Three sessions in, the rule-book knows things the first one didn't
The rule-book as a diff: a rule an agent wrote, a rule the opt-in teacher distilled from a failed run and marked [unverified], and a pinned rule surviving the cap that evicts an older one.
How a lesson survives a /clear
- A run ends with a brief, not a shrug
- The next dispatch reads it back
- A lesson gets promoted to a rule
- Three failed runs park the card
- Optionally, the failure becomes a rule
A run ends with a brief, not a shrug
When an agent reports a card Done it writes the hand-off in the same call. The entry is prepended to .portbay/HANDOFF.md under a timestamp and the author's name. If nobody writes one, PortBay derives a skeleton from board state and marks it auto-generated.
portbay_task_complete(status="Done", handoff="what changed; what is next")The next dispatch reads it back
Before an agent starts a card, PortBay inlines the latest brief and the learnings that match the card's touchpoints, labels and title under the header “what works here; trust it”. Entries that don't match are withheld, and the prompt says how many, so the agent knows the book is bigger than its slice.
[Project learnings — what works here; trust it]A lesson gets promoted to a rule
A rule is one line with a Why and an optional How to apply. Adding one that already exists is a no-op, so an agent re-asserting the same lesson never bloats the book. Up to three rules can be pinned; an append that could only fit by evicting a pinned rule is refused rather than silently taken.
portbay_learning_add(rule, why, how)Three failed runs park the card
A run that ends without reaching Done takes a strike, and a card that carries a reasoning effort retries one level harder instead of repeating identically. At the cap — three by default — auto-retry stops and the card lands in Blocked, with a comment in its thread naming how many runs ended and what the last outcome was. Opt-in rescue re-points the card at a different agent or model on the attempt before that.
outcome=reclaimed model=sonnet strike=3/3 → BlockedOptionally, the failure becomes a rule
With teacher escalation on, hitting the cap fires one background print-mode call through an agent CLI you already have installed. It reads the failed trace and distills a single portable lesson into the rule-book, marked [unverified] because a model wrote it from an untrusted transcript. It is off by default and costs nothing until you turn it on.
automation.teacher = trueWhat the memory layer includes
.portbay/HANDOFF.md — newest entry on top, every entry attributed to the agent, the CLI, you, or PortBay.
Set per project in the file's own frontmatter. An over-sized entry sheds its least valuable sections first, so the landmines are the last thing a cap takes.
.portbay/LEARNINGS.md is an index of one-line rules; each body lives uncapped at .portbay/learnings/<id>.md.
6,000-character index, 500 characters per entry, up to 3 pinned entries that eviction cannot touch.
Only the rules matching the card go into the prompt; the rest are counted as withheld rather than silently dropped.
Configurable. A run that ends without Done takes a strike; a card that already carries a reasoning effort retries one level harder.
Opt-in, and not an extra attempt: on the last try before the cap it re-points the card at a different agent or model. With no rescue agent chosen it is a plain same-agent retry.
Opt-in. One print-mode call through an agent CLI you already have — claude -p, codex exec, gemini -p, qwen -p only. It runs on the subscription you already pay for, never an API key. Its rules are marked [unverified].
portbay_handoff_get, portbay_handoff_update, portbay_learning_add, portbay_learning_get. The CLI has portbay handoff show and portbay handoff update.
Verified against the app source: none of this is entitlement-gated. Both files, the strike system and both escalations ship in every tier.
Keeping context by hand
Without a place for it, continuity lives in the context window, and the context window is the thing that ends. The usual substitute is one notes file that grows forever and gets read in full on every run.
PortBay writes the brief when the run ends, caps both files, selects the rules that match the card, and archives whatever the cap prunes.
# The manual version: one notes file you maintain yourself$ cat >> CLAUDE.md <<'EOF'## 2026-09-05- In a worktree, run `npx --no-install vitest run <file>`; pnpm purges the main clone's node_modules through the symlink.EOF # Nothing prunes it. Nothing ranks it. Nothing picks the lines that# apply to the task at hand — every agent reads all of it, every run.$ /clear# The session brief lived in the context window. It is gone. # And the run that failed three times taught nobody anything:$ git log --oneline -1# a commit. Not a reason.
One app, not a stack of tools
Agent task board
A Kanban board both you and your agents work. Cards are Markdown in the repo; moving one to To Do dispatches the agent.
Explore →Tasks as Markdown
Every board card is one Markdown file at .portbay/tasks/<id>.md — edit it with any tool and the board re-renders in about 150 ms.
Explore →Sandboxed runs
Wrap a cloned repo or agent-modified project in a sandbox-exec profile: writes confined, network by policy, every denial logged.
Explore →Asked before downloading
Is this real memory, or just a file the agent reads?
PortBay's agent memory is a file the agent reads, and the page will not dress that up. There is no vector store, no embedding index and no model fine-tuning. What accumulates is written rules and written briefs, in Markdown, that PortBay inlines into the next dispatch prompt. The thing that makes it work is not the storage, it is the discipline around it: the rule-book is deduplicated, capped, ranked, filtered against the card being worked, and archived rather than deleted when the cap prunes it.
What does PortBay write into HANDOFF.md?
One entry per session, prepended under a timestamp and the author's name. Entries carry what changed, what is next, open items and landmines — the things a next session cannot reconstruct from the board or the diff. The whole file is capped at 1,200 characters by default. When an append pushes it over, the oldest entries are pruned from the bottom; when a single entry is over the cap on its own, it sheds sections in rank order and landmines is the last one to go. Everything pruned is moved verbatim into .portbay/HANDOFF.archive.md.
Does PortBay run rescue and teacher escalation out of the box?
No. Both default to off in the app source, and with teacher off that code path costs nothing. The three-strike cap itself is always on: a run that ends without Done takes a strike, and at three the card stops being re-dispatched and lands in Blocked with a comment. Rescue and teacher are what you add when you want the failure to change something rather than just stop.
Can I trust a rule PortBay's teacher agent wrote?
Treat it as a hint to check. The teacher distills a rule from the failed run's trace, which is output the model that just struck out produced, so the trace is fenced as untrusted on the way in and the resulting rule is prefixed [unverified] in the book. That marker survives into the prompt and survives the consolidation pass, and clearing it is a person editing the heading. Rules a human wrote carry no marker, so a reader can always tell which is which.
What does PortBay's agent learning loop cost to run?
Nothing beyond what you already pay your agent vendor. None of this is behind Pro — the hand-off brief, the rule-book, card-scoped selection, the strike system, rescue and teacher escalation read no entitlement in the app source; Pro changes capacity (the project cap, sync between devices, larger task attachments), not whether a project can remember. And the teacher, when you turn it on, shells out to an agent CLI you already have installed in print mode — claude -p, codex exec, gemini -p or qwen -p. It runs on the subscription you are already paying for. PortBay never asks you for an API key and never meters your memory per token.
Give your projects and your agents a real local home.
Download for macOSFree & open source · macOS 11+ on Apple Silicon · Pro from $10/mo