--- framework: Agentic Engineering Maturity — Factory Edition version: 2026.06.0 kind: checklist lens: team title: Team / Project Checklist --- # Team / Project Checklist **Version `2026.06.0`** · Score a team, repository, or organization's standards and infrastructure. **Project / Team:** ________________________________ **Date:** ________________ **Assessed by:** ________________________ Each item carries a **stage** (`S0`–`S4`) and **theme tags**. A team is at a stage within a dimension when the standard holds *in practice* — provable from a repo file, a config, a dashboard, or recent PRs — not just written in a wiki. Your overall stage is the lowest that holds across the dimensions you treat as load-bearing. See the companion guide for the *why*, *what good looks like*, *how to check*, and the *evidence* behind every item. > Stages: **S0** Improvised · **S1** Assisted · **S2** Delegated · **S3** Orchestrated · **S4** Autonomous Agents --- ## 1. Intent & Specification *Is the why externalized before implementation — and kept?* - [ ] A spec / intent doc precedes non-trivial implementation and lives in the repo or tracker, not a chat thread `S2` `intent` `spec` - [ ] A shared spec template captures goals, constraints, non-goals, edge cases, and a checkable definition of done `S2` `spec` `template` - [ ] Decision logs / ADRs are standard and captured at decision time — the *why* is a first-class artifact `S3` `intent` `decision-log` - [ ] `AGENTS.md` is the team's intent ledger (conventions, boundaries, "we don't do X because…"), not an autogenerated file `S3` `intent` `harness` - [ ] The definition of done is explicit and shared beyond "functionally correct" — the quality bar (perf / a11y / security) is named `S3` `spec` `quality` - [ ] The team treats intent debt as real: load-bearing rationale is externalized so agents, which start every session cold, don't guess it `S4` `intent` `compounding` ## 2. Context, Memory & Harness *Is the project set up to give agents the right context — durably?* - [ ] A root-level `AGENTS.md` is the canonical place agents learn the repo's conventions, the lines they must not cross, and how to build and test `S2` `harness` `context` - [ ] The README and docs are AI-consumable — key paths, commands, and architecture an agent can navigate without guessing `S2` `context` - [ ] Architecture and domain knowledge live in machine-readable artifacts an agent can cite — decision records, specs, a glossary, and schema / API definitions `S2` `context` - [ ] The harness is a real, owned artifact — prompts, tools, hooks, and context policies are versioned and improved, not left to each dev `S3` `harness` - [ ] Each recurring kind of work — new feature, fix, refactor, data migration — has a ready-made context bundle or rule set the agent loads `S3` `context` `harness` - [ ] Long-running work has a memory standard — state files, progress logs, hand-off briefs — so agents survive context resets `S3` `harness` `memory` - [ ] Team knowledge (past PRs, incidents, decisions) is indexed so people and agents can retrieve it `S3` `context` `knowledge` ## 3. Loops, Autonomy & Orchestration *Is autonomy deliberate, and are loops engineered rather than improvised?* - [ ] Every workflow is assigned an autonomy tier from suggest-only up to auto-merge, and the riskier the work the tighter the tier `S2` `autonomy` - [ ] The exact conditions that force an agent or loop to halt and hand back to a human are written down `S3` `autonomy` `loop` - [ ] The team uses loop primitives deliberately — scheduled automations, worktree isolation, sub-agents, connectors, a state file — not ad hoc scripts `S4` `loop` - [ ] Maker/checker separation is standard for non-trivial work: generation and evaluation are different agents or stages `S3` `loop` `verification` - [ ] Autonomous and cloud agents run under explicit rules: who may start one, the blast radius it's allowed, and how its results return for review `S3` `autonomy` `governance` - [ ] Parallel-execution patterns (worktrees, best-of-N) are documented and scaled to the team's *review* capacity, not the tool's spawn limit `S3` `orchestration-tax` `loop` - [ ] Overnight / self-improving loops, where used, have guardrails: feature branches, read-only auto-approve, sandboxes, stop conditions, human PR at the end `S4` `loop` `safety` ## 4. Verification & Review *Is review tiered, adversarial, evidence-gated, and human-owned?* - [ ] Review depth is tiered by blast radius, not by author — risk tiers and their required gates are written down `S2` `review` `blast-radius` - [ ] AI-generated code clears the same review bar as human code, with a human owning the merge on load-bearing paths `S2` `review` `accountability` - [ ] The team runs *heterogeneous* AI reviewers on high-risk PRs — two built differently, not two copies of one `S3` `review` `heterogeneous` - [ ] An intake bar is enforced: PRs arrive with stated intent, a readable / small diff, and evidence the tests ran — or they bounce `S3` `review` `intake` - [ ] Review explicitly checks the test diff for assertions weakened or rewritten to match changed behavior `S3` `review` `testing` - [ ] A fast-fail / triage step routes trivial agent PRs to a light path and flags high-maintenance ones before a human invests time `S4` `review` `metrics` - [ ] Agents are instructed to keep PRs small; oversized unreviewable diffs are sent back, not rubber-stamped `S2` `review` `quality` - [ ] The review checklist names AI-specific risks (hallucinated imports, shallow tests, duplication, untrusted input flowing into a prompt) `S2` `review` `quality` ## 5. Tests & Deterministic Gates *Do real tests and immovable CI carry the weight human reading can't?* - [ ] Tests are part of the AI-assisted workflow from the start; coverage expectations apply equally to AI-written code `S2` `testing` - [ ] Red/green TDD is the norm for non-trivial logic — tests written and seen to fail before implementation `S3` `testing` `tdd` - [ ] CI is the immovable wall: lint, types, tests, security and dependency scans are required and cannot be skipped to go green `S2` `gates` `ci` - [ ] The pipeline detects agents quietly lowering the bar — removed tests, skipped lint, dropped coverage thresholds `S3` `gates` `ci` - [ ] Mutation testing or equivalent guards the suite on critical paths where coverage alone misleads `S4` `testing` `quality` - [ ] Verify-after-generate is required by the workflow and enforced by the review checklist `S2` `testing` `quality` ## 6. Safety, Security & Guardrails *Are dangerous actions blocked by construction across the org?* - [ ] Ignore files keep secrets and sensitive paths out of agent context by default `S2` `security` - [ ] A secrets-management policy is explicit *and technically enforced* — not just "don't paste secrets" `S2` `security` `governance` - [ ] Tool / MCP permissions are scoped (never "allow all" for prod-adjacent work); credentials come from env, not hard-coding `S2` `security` `permissions` - [ ] No MCP server or third-party tool is adopted until it clears a provenance-and-risk check — publisher, signature, reachable scope, dependency exposure — and it's version-pinned `S3` `security` `mcp` - [ ] Destructive actions are denied at the system level whatever the agent intends — no force-push, production migration, bulk delete, or secret writes `S3` `security` `guardrails` - [ ] Agent-executed code handling untrusted input runs in sandboxes; indirect prompt-injection defenses are in place `S3` `security` `sandbox` `injection` - [ ] Documented "what AI must NOT do" boundaries exist (never auto-merge to main, never touch billing / prod config) and are enforced `S3` `governance` `guardrails` - [ ] Enterprise / privacy / data-residency constraints are documented, enforced, and AI usage is auditable where compliance requires `S3` `governance` `security` ## 7. Comprehension & Judgment *Does the team still understand what it ships?* - [ ] The team treats comprehension as a real risk — it doesn't assume "reviewed" means "understood" `S3` `comprehension` - [ ] The team calls out and pushes back on known bad habits — shipping without a spec, accepting output unread, recycling prompts blindly, rubber-stamping under deadline `S2` `comprehension` `surrender` - [ ] For load-bearing systems, at least one human keeps a current mental model — ownership of "why it's built this way" is explicit `S3` `comprehension` `intent` - [ ] AI literacy is differentiated by role (driver vs. reviewer expectations), and onboarding teaches the agentic workflow `S3` `knowledge` `people` - [ ] Friction is designed in where it matters — a required spec, a confirm before merge, a checklist before deploy — to interrupt heuristic acceptance `S3` `surrender` `guardrails` - [ ] The team values and makes time for the engineer who can tell a safe refactor from one quietly shifting behavior users depend on `S4` `comprehension` `compounding` ## 8. Reusable Capability & the Ratchet *Does the team compound capability instead of repeating fixes?* - [ ] A versioned, team-maintained library of skills, commands, and rules exists and covers the common task types `S3` `ratchet` `skills` - [ ] The ratchet is institutional — recurring agent mistakes become permanent guards (rules, hooks, reviewer checks), not repeat incidents `S4` `ratchet` - [ ] Skills, rules, and hooks are treated as code — reviewed in PRs, tested, and retired when models make them redundant `S4` `ratchet` `skills` - [ ] Lifecycle hooks automate the repeatable (format, test, secret-scan, notify) with "silent success / verbose failure" discipline `S3` `harness` `hooks` - [ ] Lessons from real agent sessions are fed back into `AGENTS.md`, rules, and skills — the memory loop is closed, not write-once `S4` `ratchet` `compounding` - [ ] Skills are shared across repos / teams (e.g. packaged as plugins) so the org compounds rather than each team rebuilding `S4` `knowledge` `compounding` ## 9. Measurement, Cost & Compounding *Can the team prove the factory is getting better, not just faster?* - [ ] A named owner publishes a regular readout of how well the AI tooling performs — suggestion-acceptance, revert / rework, time-to-merge, cycle time `S3` `metrics` - [ ] DORA-style metrics carry an AI-adoption dimension so quality regressions (churn, defects, zero-review merges) stay visible behind velocity `S3` `metrics` `quality` - [ ] Changes to prompts, rules, and skills run against an eval set, and the pass rate is trended like any other quality signal `S4` `evals` - [ ] Every agent run leaves a replayable trail — the tool calls made, the choices taken, the model version — for debugging and audit `S3` `observability` - [ ] Token / cost is visible per team or project, with budgets, quotas, or alerts and a documented response `S2` `cost` `governance` - [ ] AI-attributable incidents are counted and reviewed; post-incident reviews ask "was this AI-related?" `S3` `metrics` `quality` - [ ] AI authorship / provenance is attributed in commits or PRs, and license / IP policy for generated code is documented `S2` `governance` - [ ] The team can show the factory compounding — fewer repeat mistakes, rising eval pass rates, stable quality as volume grows `S4` `compounding` `metrics` --- *62 items across 9 dimensions. Pair with the companion guide for the "why," "what good looks like," "how to check," and "evidence" behind each.*