feat: add autonomous E2E CI failure fix workflow with skills and Playwright agents#4516
feat: add autonomous E2E CI failure fix workflow with skills and Playwright agents#4516zdrapela wants to merge 11 commits intoredhat-developer:mainfrom
Conversation
|
Skipping CI for Draft Pull Request. |
|
The container image build workflow finished with status: |
zdrapela
force-pushed
the
skill-agent-e2e-fix
branch
from
e8cd40c to
d237b44
Compare
|
The container image build workflow finished with status: |
|
The container image build workflow finished with status: |
|
The container image build workflow finished with status: |
e2e-tests/playwright/seed.spec.ts
Outdated
| @@ -0,0 +1,22 @@ | |||
| import { test, expect } from "@playwright/test"; | |||
There was a problem hiding this comment.
This file sits in playwright/ alongside real tests and no project in projects.json explicitly includes or excludes it. It'll get picked up by any project with a broad testMatch glob. If it's scaffolding for the Playwright Test Generator agent, should it live somewhere outside the test directory, or be excluded in playwright.config.ts?
There was a problem hiding this comment.
Good point, looks like the seed test file need to live there and it's good to have our own seed file in the repo (instead of the generic one). I excluded it.
| env | grep -E "^(BASE_URL|K8S_CLUSTER|CONTAINER_PLATFORM|IS_OPENSHIFT|JOB_NAME|IMAGE_|TAG_NAME|NAME_SPACE|GITHUB_APP|...)" > .env | ||
| ``` | ||
|
|
||
| The `.env` file is already in `.gitignore` — never commit it. |
There was a problem hiding this comment.
This says .env is already in .gitignore, but e2e-tests/.gitignore doesn't have a .env entry. The instructions above dump Vault secrets, K8S tokens, and GitHub app credentials into that file — needs an actual gitignore entry to back up this claim.
There was a problem hiding this comment.
It actually does have it 🤔
Line 31 in 2dff713
| "rules", | ||
| "commands" | ||
| "commands", | ||
| "skills" |
There was a problem hiding this comment.
Is the rulesync CI check (rulesync-check.yaml) validated to work with the skills feature? This is the first time it's being added. Worth confirming the check handles skills sync the same way it handles rules and commands.
There was a problem hiding this comment.
I tested it for both Claude code and OpenCode. I couldn't verify for Cursor, because the Playwright agents aren't supported for it
| ```bash | ||
| cd e2e-tests | ||
| source local-test-setup.sh <showcase|rbac> | ||
| env | grep -E "^(BASE_URL|K8S_CLUSTER|CONTAINER_PLATFORM|IS_OPENSHIFT|JOB_NAME|IMAGE_|TAG_NAME|NAME_SPACE|GITHUB_APP|...)" > .env |
There was a problem hiding this comment.
That trailing |... in the grep regex will match any three characters, not just serve as a placeholder. If someone copy-pastes this, it'll capture unintended variables. Either list the actual env var prefixes exhaustively or use a different approach (like env > .env with a note to review).
There was a problem hiding this comment.
I added the .env generation to local-test-setup.sh That's a more systematic approach
|
|
||
| See https://playwright.dev/docs/test-agents for the full list of supported tools and options. | ||
|
|
||
| This creates configuration files with the Playwright MCP server and agent definitions. The generated files are local tooling — do NOT commit them. |
There was a problem hiding this comment.
This says "do NOT commit them", but the PR commits e2e-tests/.mcp.json, e2e-tests/opencode.json, and all the agent/prompt markdown files in e2e-tests/.claude/agents/ and e2e-tests/.opencode/prompts/ — which are exactly what npx playwright init-agents generates. If these are meant to be checked in as the project's canonical agent configs, this note needs updating. If they shouldn't be committed, they shouldn't be in this PR.
There was a problem hiding this comment.
I added them to gitignore
e2e-tests/opencode.json
Outdated
| } | ||
| }, | ||
| "tools": { | ||
| "playwright*": false |
There was a problem hiding this comment.
This silently disables all Playwright MCP tools outside agent contexts. If a developer tries to use them directly (without going through a subagent), they'll get no feedback about why they're unavailable. Worth a comment in the file or the skill docs explaining this design choice.
There was a problem hiding this comment.
Good point, I rather gitignored the files generated by the Playwright agent init
| ```bash | ||
| # Make the change locally | ||
| # Then commit and push | ||
| git add -A |
There was a problem hiding this comment.
git add -A can accidentally stage the .env file from the earlier step (especially since it's not actually gitignored — see other comment on diagnose-and-fix). Should use git add <specific-files> instead, which is also what the project's own commit conventions recommend.
|
|
||
| 1. **Mark the test as `test.fixme()`** with a descriptive comment: | ||
| ```typescript | ||
| test.fixme('RHIDP-XXXX: Button no longer visible after version upgrade'); |
There was a problem hiding this comment.
Playwright's test.fixme() signature is test.fixme(condition?, description?). This string-only form works (the string is truthy) but it's non-standard and could confuse contributors checking the Playwright docs. The canonical form would be test.fixme(true, 'RHIDP-XXXX: description').
There was a problem hiding this comment.
I rewrote it to use instead this format, which we use the most often
// TODO: https://jira.link
test.fixme();
``
d947b22 to
0a20e05
Compare
|
Addressing review comment #4516 (comment):
Fixed. |
|
Addressing review comment #4516 (comment):
Fixed. Replaced |
zdrapela
force-pushed
the
skill-agent-e2e-fix
branch
from
0a20e05 to
def054a
Compare
Code Review by Qodo
1.
|
Review Summary by QodoAdd autonomous E2E CI failure fix workflow with Playwright agents and multi-platform skill distribution
WalkthroughsDescription• Add comprehensive autonomous E2E CI failure investigation and fix workflow with 7 AI agent skills • Implement /fix-e2e command orchestrating full lifecycle: parse failure, create branch, deploy RHDH, reproduce, diagnose/fix with Playwright agents, verify, and submit PR with Qodo review • Initialize Playwright Test Agents (healer/generator/planner) with MCP server configuration for live browser interaction • Distribute all skills and rules across rulesync, Claude, Cursor, and OpenCode platforms for consistent AI agent access • Add seed test for E2E initialization and .env file generation for Playwright Test Agents environment setup • Enable rulesync skills feature and configure MCP server for Playwright test automation Diagramflowchart LR
A["CI Failure"] -->|parse-ci-failure| B["Extract Metadata"]
B -->|setup-fix-branch| C["Create Fix Branch"]
C -->|deploy-rhdh| D["Deploy RHDH"]
D -->|reproduce-failure| E["Classify Failure"]
E -->|diagnose-and-fix| F["Fix with Playwright Agents"]
F -->|verify-fix| G["Stability & Quality Check"]
G -->|submit-and-review| H["Submit PR & Qodo Review"]
H --> I["Monitor CI"]
J["Skills & Rules"] -->|rulesync| K["Claude/Cursor/OpenCode"]
L["MCP Server"] -->|Playwright Agents| F
File Changes1. e2e-tests/playwright.config.ts
|
|
The container image build workflow finished with status: |
|
The container image build workflow finished with status: |
zdrapela
force-pushed
the
skill-agent-e2e-fix
branch
from
a22f6c2 to
3b547f3
Compare
|
/test ? |
|
@zdrapela: The following commands are available to trigger required jobs: The following commands are available to trigger optional jobs: Use DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
zdrapela
force-pushed
the
skill-agent-e2e-fix
branch
from
3b547f3 to
d70a000
Compare
|
The container image build workflow finished with status: |
|
The container image build workflow finished with status: |
…wright agents Add 6 AI agent skills and /fix-e2e command for autonomous E2E CI failure investigation and fix workflow. Skills: e2e-parse-ci-failure, e2e-deploy-rhdh, e2e-reproduce-failure, e2e-diagnose-and-fix, e2e-verify-fix, e2e-submit-and-review. Key features: - Playwright Test Agents (healer/generator/planner) with MCP server - Auto-generate .env via local-test-setup.sh --env with secure file handling - Sourcebot/Context7 optional with gh/clone fallbacks - OCP vs K8s job differentiation for deploy-only mode - Playwright MCP for parsing HTML test reports - All skills managed via rulesync, synced to OpenCode, Claude Code, and Cursor Assisted-by: OpenCode
…mit-and-review Document required vs optional job sections, shortened job names, and update the mapping table with actual /test commands from the bot. Assisted-by: OpenCode
Assisted-by: OpenCode
Assisted-by: OpenCode
Assisted-by: OpenCode
…Qodo polling - Fix duplicate ### 5 numbering in e2e-verify-fix (Review the Diff is now ### 6) - Add Playwright report URL to fix-e2e.md input list - Add scripted polling loop for Qodo review (15s interval, 5 min timeout) - Update command to match new Qodo polling interval Assisted-by: OpenCode
…skills Assisted-by: OpenCode
…e job Present classified failures, ask user which to fix, group by shared root cause when possible. Assisted-by: OpenCode
… optional Assisted-by: OpenCode
The healer agent is only supported in OpenCode and Claude Code. Add notes to all 3 healer-using skills directing Cursor users to direct execution and manual diagnosis instead. Assisted-by: OpenCode
Assisted-by: OpenCode
zdrapela
force-pushed
the
skill-agent-e2e-fix
branch
from
2863dcb to
138c106
Compare
|
The container image build workflow finished with status: |
|
2 participants
Summary
/fix-e2ecommand that orchestrates the full lifecycle: parse failure, create branch, deploy RHDH, reproduce, diagnose/fix with Playwright agents, verify, and submit PR.envfile vialocal-test-setup.sh --envwith proper quoting for multiline secretsseed.spec.tsfor Playwright Test Generator agent (excluded from all projects exceptany-test)Skills
e2e-parse-ci-failuree2e-setup-fix-branche2e-deploy-rhdhe2e-reproduce-failuree2e-diagnose-and-fixe2e-verify-fixe2e-submit-and-reviewKey changes since initial review
e2e-prefix for claritygh search code/ local clone fallbacks-sdeploy-only flagtest.fixme(), useRHDHBUGSJira project.envvia--envflag with single-quoted values (handles multiline PEM certs)git add -Awith specific file staging.playwright-mcp, generated agent files to.gitignoreseed.spec.tsfrom showcase/showcase-k8s/showcase-operator projects