feat: per-test configuration for triggers and alerts#74
Open
metalwarrior665 wants to merge 23 commits intomasterfrom
Open
feat: per-test configuration for triggers and alerts#74metalwarrior665 wants to merge 23 commits intomasterfrom
metalwarrior665 wants to merge 23 commits intomasterfrom
Conversation
Tests and suites now accept `runWhen` and `alerts` options in their config object, enabling fine-grained control over which GitHub Actions trigger causes each test to execute and which alerting channels fire on failure. - `lib/trigger.ts` — reads `TEST_TRIGGER` env var; exposes `getCurrentTrigger()` and `shouldRunForTrigger(runWhen)` (falls back to `'locally'` when unset) - `lib/types.ts` — adds `TriggerType`, `RunWhenConfig`, `AlertsConfig`; extends `ActorTestOptions` with `runWhen` and `alerts` - `lib/lib.ts` — `describe`, `testActor`, `testStandbyActor` all apply the trigger check (ANDed with the existing platform guard); `runWhen`/`alerts` are stripped before being forwarded to vitest; `alertsRegistry` map is exported for post-run reporting; `getCurrentTrigger` re-exported - `index.ts` — exports new types and helpers https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd
…l merging - `describe` and `testActor`/`testStandbyActor` now accept a config object with `name` as their first/second param; `fn` always follows immediately, keeping the test body visible before any trailing options - Hierarchical merging via a config stack: `default → describe → nested describe → testActor`; `runWhen` is fully overridden by the innermost layer, `alerts` shallow-merges so children can add/override keys - Backward-compatible: old-style `describe(name, fn, options?)` and `testActor(actorId, name, fn, options?)` still work via typeof detection - Remove `'locally'` from `TriggerType`; `TEST_TRIGGER` unset now means "run everything" (no filtering) rather than matching a fallback value - Remove `onCall` from `AlertsConfig` - `runWhen`/`alerts` removed from `ActorTestOptions` (moved to `DescribeConfig` / `TestActorConfig`) - Export `DescribeConfig` and `TestActorConfig` from index https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd
Instead of a separate module-level registry, alerts config is embedded in `task.meta.alerts` at the start of each test body. Vitest's JSON reporter already serializes task.meta into test-output.json (the same mechanism used for runLink/actorName), so report-tests can read it without any extra files or IPC. - Remove `alertsRegistry` — superseded by task.meta - `report-tests` reads `meta.alerts.slack` per failing test: - undefined (no alerts config) → notify, for backward compatibility - `slack: true` → notify - `slack: false` → skip Slack for that specific test https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd
Extract pure functions to make core logic directly testable: - `mergeInheritedConfigs(layers)` exported from lib/lib.ts - `shouldNotifySlack(alerts)` exported from bin/test-report.ts New test files (33 tests): - test/unit/trigger.test.ts — getCurrentTrigger (valid/invalid/missing TEST_TRIGGER) and shouldRunForTrigger (all combinations of trigger presence and runWhen config) - test/unit/config-merging.test.ts — mergeInheritedConfigs: empty stack, runWhen innermost-wins semantics, alerts shallow-merge, multi-layer combined scenarios - test/unit/test-report-alerts.test.ts — shouldNotifySlack: undefined alerts (backward compat), explicit opt-in, explicit opt-out https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd
Previously the Slack filter was baked into the failedAssertions assembly loop, meaning any future channel (email, PagerDuty, etc.) would require duplicating that loop. Now: - `failedAssertions` collects all failures with their `alerts` metadata - `slackAssertions` is derived by filtering `failedAssertions` via `shouldNotifySlack` immediately before the Slack send - Adding a new reporting channel is a single `.filter()` line on the same `failedAssertions` base https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd
…ield runWhen merge, deduplicated actor test helpers
- `DescribeConfig` and `TestActorConfig` now use `triggers: { runWhen, alerts }` and `options: { timeout, concurrent/retry }` sub-objects; `name` stays at top level
- `mergeInheritedConfigs` now shallow-merges `runWhen` field-by-field (same as `alerts`) so children only need to override specific trigger keys rather than replacing the entire object
- Extract `resolveActorTestConfig` helper to eliminate duplication between `testActor` and `testStandbyActor`
- Remove `shouldNotifySlack` function (single-property check); inline as `alerts?.slack !== false` with a backward-compat comment
- Export new types: `TriggerConfig`, `DescribeOptions`, `ActorOptions`
- Update config-merging unit tests for new shallow-merge semantics; delete test-report-alerts unit test
https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd
…InheritedTriggers Aligns internal names with the `triggers` field and `TriggerConfig` type used in the public API. https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd
Previously runWhen[trigger] === true was required to run, meaning any unset
trigger implicitly disabled the test. Now runWhen[trigger] !== false, so all
triggers are enabled by default and users only need to set false to opt out.
e.g. runWhen: { pullRequest: false } disables only PR runs while
hourly and daily continue to run — no need to list every other trigger.
https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd
…eck function Rather than shouldRunForTrigger using !== false (silently enabling new trigger types for all tests), DEFAULT_TRIGGERS is prepended to the merge stack and typed as Required<RunWhenConfig>. Adding a new TriggerType now causes a compile error on DEFAULT_TRIGGERS, forcing an explicit opt-in/opt-out decision — preventing accidental rollout to all existing tests. shouldRunForTrigger reverts to === true; the opt-out default is purely a config concern, not a predicate concern. https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd
…out semantics - getMergedTriggers now accepts an optional extra layer, eliminating the duplicate DEFAULT_TRIGGERS reference in resolveActorTestConfig - describe/testActor/testStandbyActor JSDoc examples updated to show opt-out style (pullRequest: false) instead of misleading opt-in (daily: true / hourly: true) - RunWhenConfig and shouldRunForTrigger docs no longer reference internal implementation details (DEFAULT_TRIGGERS location) - Remove orphaned section divider comment https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd
…abled per directory
DEFAULT_TRIGGERS.hourly is now false — only tests that explicitly set
runWhen: { hourly: true } will run on the hourly trigger.
For backward compatibility without touching test files: set the
BACKWARD_COMPATIBLE_HOURLY_DIR env var (e.g. 'core') in the workflow.
The library uses call-stack inspection to detect which test file is
registering a describe/testActor, and promotes hourly to true for tests
defined under that directory. The Actions workflow no longer needs a
subtest directory input — run all of /platform and let the lib handle
the scoping.
BACKWARD_COMPATIBLE_HOURLY_DIR is exported from the package so callers
can reference the constant name rather than hardcoding the string.
https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd
…ll sites The layers are now explicit at each call site, making the merge order readable without jumping to a helper. https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd
Move DEFAULT_TRIGGERS/DEFAULT_DESCRIBE_OPTIONS/DEFAULT_TEST_ACTOR_OPTIONS to consts.ts and actor-run helpers (createStartRunFn, createStartStandbyFn, createStandbyTask, generateRunLink) to utils.ts. lib.ts now contains only module-level init, the trigger stack, and the exported API (describe, testActor, testStandbyActor). Also inlines BACKWARD_COMPATIBLE_HOURLY_DIR as process.env access so it is no longer exported from index.ts. https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd
No need to pass callerFile as a parameter — both functions live in lib.ts, so all frames are already filtered by THIS_FILE_BASE. Avoids calling getCallerFile when BACKWARD_COMPATIBLE_HOURLY_DIR is not set (fast path). https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd
slack defaults to true (opt-out model, consistent with runWhen). Required<AlertsConfig> gives a compile error if a new alert channel is added without an explicit decision. report-tests keeps its !== false guard since it operates across a process boundary and may be used with older library versions. https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd
The hourly override was returning only { runWhen: {...} }, dropping
alerts from DEFAULT_TRIGGERS. Fixed by spreading DEFAULT_TRIGGERS first
and then overriding only runWhen. Also corrects the describe() JSDoc
example to show alerts: { slack: false } to illustrate opt-out semantics.
https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd
…tion Remove the old "Differences in writing tests" before/after migration guide (everyone has migrated). Add examples for runWhen/alerts triggers config: opting out of triggers, inheriting through describe hierarchy, disabling Slack alerts, hourly tests via BACKWARD_COMPATIBLE_HOURLY_DIR. Update the core workflow to use backward_compatible_hourly_dir input instead of subtest: core. https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd
…le_hourly_dir from workflow
How BACKWARD_COMPATIBLE_HOURLY_DIR is passed to the workflow is TBD.
New tests should opt in to hourly via triggers: { runWhen: { hourly: true } }.
https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd
ruocco-l
reviewed
Apr 8, 2026
Contributor
ruocco-l
left a comment
There was a problem hiding this comment.
Other than the pullRequest to be explicitly opt in, LGTM
Patai5
reviewed
Apr 8, 2026
Contributor
Patai5
left a comment
There was a problem hiding this comment.
I trust the unit tests that were made by the LLM to do their job 😎
Patai5
approved these changes
Apr 8, 2026
Member
Author
|
I have reverted the refactoring that moved some stuff to utils.ts, it was making the diff longer and making merging master harder. Let's do it later. |
Member
Author
|
Corrected after merge conflicts. I will let it sit for a few days, then will do the last review, ping, and merge. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This will need some small changes in source actions but for repos, it is backward compatible
AI SUMMARY BELOW
Summary
This PR introduces a trigger-based test filtering system and alert configuration framework to the actor testing library. Tests and test suites can now be conditionally run based on trigger types (hourly, daily, pull request) and configured to send alerts to specific channels (Slack) when they fail.
Key Changes
New trigger system (
lib/trigger.ts):getCurrentTrigger(): Reads theTEST_TRIGGERenvironment variable to determine the active trigger typeshouldRunForTrigger(): Evaluates whether a test should run based on itsrunWhenconfiguration and the current triggerhourly,daily,pullRequestHierarchical trigger inheritance:
mergeInheritedTriggers()function that merges trigger configurations field-by-field through the describe hierarchydescribeblocks, allowing child tests to inherit and override parent trigger settingsrunWhenandalertsconfigurations are shallow-merged so children only need to override specific keysUpdated
describe()API:DescribeConfig) or legacy string-based argumentsname,triggers, andoptionspropertiesUpdated
testActor()andtestStandbyActor()APIs:TestActorConfig) or legacy string-based argumentsname,triggers, andoptionspropertiesdescribeblocks with test-level overridesNew type definitions (
lib/types.ts):TriggerType: Union of valid trigger typesRunWhenConfig: Controls which triggers enable a testAlertsConfig: Defines alerting channels (currently Slack)TriggerConfig: CombinesrunWhenandalertsDescribeConfigandTestActorConfig: Configuration objects for the new API styleDescribeOptionsandActorOptions: Vitest-level test optionsUpdated test reporting (
bin/test-report.ts):alertsmetadata from test resultsalerts.slack !== falseComprehensive test coverage:
test/unit/triggers-merging.test.ts: Tests the trigger merging logic across various inheritance scenariostest/unit/trigger.test.ts: Tests trigger detection and filtering logicExported public API (
index.ts):getCurrentTrigger()functionNotable Implementation Details
https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd