A skeleton for putting AI agents into a real test stack
Web and native mobile. It ships the agents, the skills they use, and the wiring that lets them work against whatever language, runner, CI and tracker a project already picked.
The hard part is keeping the agents stack agnostic while still letting them write code that fits the repo. One config file and a profile system solve that. Adding Java plus Selenium means writing four files and changing zero agents.
What it is
Five QA workflows, working against one project:
- Requirement refinement and test design
- Assisted exploratory and manual testing
- Automated test generation
- Test maintenance and refactoring
- Quality auditing of automation suites
It is a scaffold you drop into a project, point at that project's stack, and use. It is not a test suite for one product, and it is not tied to a runner.
Stack agnostic
Agents read qa.config.yml and the active profile before doing anything. Nothing hardcodes Playwright, pytest or a CI provider.
Runs for real
Two demo suites go green against a public site in CI. The examples in the docs execute rather than describe.
Says what it did not do
Every workflow reports what it skipped and why. A design that covers everything and an audit that finds nothing are both wrong.
Fits a repo that already has tests
/qa:profile reads your suite and writes down the house style it already has. The policy hook only flags lines an edit touched, so it never fights code you inherited. How that goes.
How it works
Two layers with one contract between them.
The AI layer, agents/ and skills/, knows QA. It knows nothing about Playwright or pytest.
The profile layer, profiles/<name>/, knows one stack. How to run it, where files go, what good code looks like there. No QA reasoning in it at all.
The contract is qa.config.yml at the project root, plus a profile.yaml in each profile.
# qa.config.yml
stacks:
web:
profile: ts-playwright
root: e2e/
base_url_env: BASE_URL
ci:
provider: github-actions # github-actions | gitlab-ci | jenkins
tracker:
provider: jira # jira | github-issues | none
project_key: SHOP
policies:
test_id_attribute: data-testid
max_test_runtime_s: 60
forbid: [hard_sleeps, absolute_xpath, conditional_assertions]# profiles/ts-playwright/profile.yaml
commands:
install: npm ci && npx playwright install --with-deps
test: npx playwright test
test_one: npx playwright test {file} -g "{title}"
layout:
tests: tests/
test_glob: "**/*.spec.ts"
selector_priority: [getByRole, getByLabel, getByTestId, getByText]When an agent needs to rerun one test it reads commands.test_one. When it needs to write a page object it reads the template and CONVENTIONS.md. That is the whole trick, and it is why the framework is open on language, runner and CI.
One command gives an agent all of it:
$ qa-context web
config: /repo/qa.config.yml
project root: /repo
web: ts-playwright -> /plugin/profiles/ts-playwright
===== /repo/qa.config.yml =====
...
===== /plugin/profiles/ts-playwright/profile.yaml =====
...
conventions for web: /plugin/profiles/ts-playwright/CONVENTIONS.mdInstall
As a Claude Code plugin:
/plugin marketplace add eduramos21/qa-automation-framework
/plugin install qa@eduramos21-qaThen, in a project:
/qa:initThat works out the language, runner and CI from the repo, asks about what it cannot see, writes qa.config.yml, and drops in a pipeline file. It describes what the project already does rather than prescribing what it should, so the config is true on day one.
To try it without installing:
git clone https://github.com/eduramos21/qa-automation-framework
cd qa-automation-framework
claude --plugin-dir .Prerequisites, connecting Jira, and what this looks like on a repo that already has tests are all on the setup page.
Or run the demo suites, which need no setup and no credentials:
cd examples/web-ts
npm ci && npx playwright install chromium && npx playwright test
# 9 passed (2.6s)The five workflows
Each is a skill. The three that read a lot of files hand off to an agent, so the reading stays out of the conversation.
| Command | Does | Produces |
|---|---|---|
/qa:refine |
Turns a ticket into testable acceptance criteria, an ambiguity list and a risk table. Asks the questions that need a human | qa/requirements/<id>.md |
/qa:design |
Risk scored coverage matrix. What to check, at which layer, what to automate, what not to cover | qa/design/<id>.md |
/qa:explore |
Session based exploratory testing in a real browser or on a device, through MCP. Keeps the session sheet | qa/sessions/<date>-<charter>.md |
/qa:generate |
Writes tests in the active profile's style. Runs them, then breaks them on purpose to prove they can fail | test files, plus a report on what it did not write |
/qa:fix |
Triages a red test through five causes before changing anything, then fixes the cause once | edits, plus what it found and did not fix |
/qa:profile |
Reads a suite that already exists and writes down its house style, so generated code matches the repo rather than a generic example | <project>/profiles/<name>/ |
/qa:audit |
Scores a suite against a nine area rubric with evidence, and ranks findings by cost over effort | qa/audit/<date>.md |
Full catalog of agents, skills and what each one refuses to do is in the reference.
A worked example
One ticket, all the way through, against the same public demo site the test suites run on. Every artifact below is committed in examples/artifacts/, and the numbers were checked against the running site rather than assumed.
The ticket, as written
"The checkout overview page should show the item total, the tax and the final total. Tax is 8%." Three acceptance criteria, one of which is "tax is calculated correctly". A developer commented that it should be quick.
1. /qa:refine
Five acceptance criteria, a decision table, a risk table and five open questions. Two of those questions are the reason the story was not ready to build:
- Round half up, half even, or truncate? The ticket says "correctly", which is exactly the ambiguity the story turns on.
- Does the total add the displayed tax or the raw one? These differ by a cent on some carts.
Three developers would pick three answers and all three would satisfy "calculated correctly". One cent per order, nobody reports it, finance finds it at month end.
2. /qa:design
Thirteen conditions, risk scored, each assigned to a layer. Eight are unit tests, three are e2e, one waits on an open question, one is deliberately not automated. Plus a table of what is not being covered and why, and two charters for the parts a script cannot judge.
The design picks its test cart on purpose: $29.99 plus $9.99 gives a raw tax of $3.1984. Truncating gives $3.19, rounding gives $3.20. A cart where both agree would have passed either way and told you nothing.
3. /qa:generate
Three e2e tests written, run, then broken on purpose to confirm they fail for the right reason. The eight unit rows were left unwritten, because this project has no unit layer and promoting them to e2e would make them slow and make them fail without saying which of forty things broke.
It also found that half of one acceptance criterion is untestable: the requirement says the confirmation repeats the total, and the confirmation page shows no amount at all. That gap is written in a comment above the test rather than papered over.
4. /qa:explore
A 55 minute session against the charter the design produced. Seven findings, three of them real defects on the live site:
- The browser back button after confirming an order returns to a live Finish button, and pressing it confirms again with nothing to say it was a repeat.
- Emptying the cart in a second tab does not stop the first tab confirming.
- Navigating straight to the overview URL with an empty cart shows
Item total: $0next toTax: $0.00and lets a $0.00 order through.
All three look like one cause: the cart is trusted from the rendered page instead of re-read at confirm. Saying that in the report turns three fixes into one.
5. /qa:audit
The suite scores 5 on speed, isolation, waiting and assertions, and 2 on layers and coverage. Both twos are the same cause: the highest risk item in the requirement is a unit row, and there is no unit layer, so nothing in the repo would catch a rounding error.
Which is the point. Every one of those gaps was a recorded decision, and the audit's job is noticing that the decisions have not been acted on since.
Built in phases
| Phase | What | State |
|---|---|---|
| 0 | Skeleton, config contract, profile template, validator | done |
| 1 | TypeScript and Python web profiles, runnable demos | done |
| 2 | Requirement refinement and test design | done |
| 3 | Automated test generation | done |
| 4 | Assisted exploratory and manual testing | done |
| 5 | Test maintenance and suite auditing | done |
| 6 | Mobile profile and Appium demo | done |
| 7 | This site | done |
| 8 | Brownfield: changed-only policy hook, /qa:profile, setup docs | done |
Java plus Selenium plus JUnit 5 is deliberately not shipped. It is the obvious next profile, and writing it from profiles/_template is the fastest way to find out whether this design holds up.