Skip to content

feat: per-test configuration for triggers and alerts#74

Open
metalwarrior665 wants to merge 23 commits intomasterfrom
claude/configurable-test-triggers-li7wc
Open

feat: per-test configuration for triggers and alerts#74
metalwarrior665 wants to merge 23 commits intomasterfrom
claude/configurable-test-triggers-li7wc

Conversation

@metalwarrior665
Copy link
Copy Markdown
Member

@metalwarrior665 metalwarrior665 commented Apr 5, 2026

This will need some small changes in source actions but for repos, it is backward compatible

AI SUMMARY BELOW

Summary

This PR introduces a trigger-based test filtering system and alert configuration framework to the actor testing library. Tests and test suites can now be conditionally run based on trigger types (hourly, daily, pull request) and configured to send alerts to specific channels (Slack) when they fail.

Key Changes

  • New trigger system (lib/trigger.ts):

    • getCurrentTrigger(): Reads the TEST_TRIGGER environment variable to determine the active trigger type
    • shouldRunForTrigger(): Evaluates whether a test should run based on its runWhen configuration and the current trigger
    • Supports three trigger types: hourly, daily, pullRequest
  • Hierarchical trigger inheritance:

    • Added mergeInheritedTriggers() function that merges trigger configurations field-by-field through the describe hierarchy
    • Implements a trigger stack that tracks nested describe blocks, allowing child tests to inherit and override parent trigger settings
    • Both runWhen and alerts configurations are shallow-merged so children only need to override specific keys
  • Updated describe() API:

    • Now accepts either a config object (DescribeConfig) or legacy string-based arguments
    • Config object supports name, triggers, and options properties
    • Maintains backward compatibility with existing string-based syntax
    • Pushes/pops triggers onto the stack during suite collection
  • Updated testActor() and testStandbyActor() APIs:

    • Now accept either a config object (TestActorConfig) or legacy string-based arguments
    • Config object supports name, triggers, and options properties
    • Merges inherited triggers from enclosing describe blocks with test-level overrides
    • Embeds alerts configuration in task metadata for downstream processing by the report tool
  • New type definitions (lib/types.ts):

    • TriggerType: Union of valid trigger types
    • RunWhenConfig: Controls which triggers enable a test
    • AlertsConfig: Defines alerting channels (currently Slack)
    • TriggerConfig: Combines runWhen and alerts
    • DescribeConfig and TestActorConfig: Configuration objects for the new API style
    • DescribeOptions and ActorOptions: Vitest-level test options
  • Updated test reporting (bin/test-report.ts):

    • Reads alerts metadata from test results
    • Filters failed assertions by alert configuration (defaults to true for backward compatibility)
    • Only sends Slack notifications for assertions with alerts.slack !== false
  • Comprehensive test coverage:

    • test/unit/triggers-merging.test.ts: Tests the trigger merging logic across various inheritance scenarios
    • test/unit/trigger.test.ts: Tests trigger detection and filtering logic
  • Exported public API (index.ts):

    • Exports getCurrentTrigger() function
    • Exports new type definitions for trigger and alert configuration

Notable Implementation Details

  • The trigger stack is maintained during vitest's synchronous suite collection phase, ensuring consistent state
  • Trigger merging uses shallow object spread to allow field-by-field overrides without losing unspecified keys
  • The system is backward-compatible: tests without trigger configuration run unconditionally, and the legacy string-based API is fully supported
  • Alert configuration is embedded in task metadata so the JSON reporter can serialize it for downstream processing

https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd

claude added 9 commits April 4, 2026 15:23
Tests and suites now accept `runWhen` and `alerts` options in their config
object, enabling fine-grained control over which GitHub Actions trigger causes
each test to execute and which alerting channels fire on failure.

- `lib/trigger.ts` — reads `TEST_TRIGGER` env var; exposes `getCurrentTrigger()`
  and `shouldRunForTrigger(runWhen)` (falls back to `'locally'` when unset)
- `lib/types.ts` — adds `TriggerType`, `RunWhenConfig`, `AlertsConfig`; extends
  `ActorTestOptions` with `runWhen` and `alerts`
- `lib/lib.ts` — `describe`, `testActor`, `testStandbyActor` all apply the
  trigger check (ANDed with the existing platform guard); `runWhen`/`alerts` are
  stripped before being forwarded to vitest; `alertsRegistry` map is exported for
  post-run reporting; `getCurrentTrigger` re-exported
- `index.ts` — exports new types and helpers

https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd
…l merging

- `describe` and `testActor`/`testStandbyActor` now accept a config object
  with `name` as their first/second param; `fn` always follows immediately,
  keeping the test body visible before any trailing options
- Hierarchical merging via a config stack: `default → describe → nested
  describe → testActor`; `runWhen` is fully overridden by the innermost
  layer, `alerts` shallow-merges so children can add/override keys
- Backward-compatible: old-style `describe(name, fn, options?)` and
  `testActor(actorId, name, fn, options?)` still work via typeof detection
- Remove `'locally'` from `TriggerType`; `TEST_TRIGGER` unset now means
  "run everything" (no filtering) rather than matching a fallback value
- Remove `onCall` from `AlertsConfig`
- `runWhen`/`alerts` removed from `ActorTestOptions` (moved to
  `DescribeConfig` / `TestActorConfig`)
- Export `DescribeConfig` and `TestActorConfig` from index

https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd
Instead of a separate module-level registry, alerts config is embedded in
`task.meta.alerts` at the start of each test body. Vitest's JSON reporter
already serializes task.meta into test-output.json (the same mechanism used
for runLink/actorName), so report-tests can read it without any extra files
or IPC.

- Remove `alertsRegistry` — superseded by task.meta
- `report-tests` reads `meta.alerts.slack` per failing test:
  - undefined (no alerts config) → notify, for backward compatibility
  - `slack: true` → notify
  - `slack: false` → skip Slack for that specific test

https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd
Extract pure functions to make core logic directly testable:
- `mergeInheritedConfigs(layers)` exported from lib/lib.ts
- `shouldNotifySlack(alerts)` exported from bin/test-report.ts

New test files (33 tests):
- test/unit/trigger.test.ts — getCurrentTrigger (valid/invalid/missing
  TEST_TRIGGER) and shouldRunForTrigger (all combinations of trigger
  presence and runWhen config)
- test/unit/config-merging.test.ts — mergeInheritedConfigs: empty stack,
  runWhen innermost-wins semantics, alerts shallow-merge, multi-layer
  combined scenarios
- test/unit/test-report-alerts.test.ts — shouldNotifySlack: undefined
  alerts (backward compat), explicit opt-in, explicit opt-out

https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd
Previously the Slack filter was baked into the failedAssertions assembly
loop, meaning any future channel (email, PagerDuty, etc.) would require
duplicating that loop.

Now:
- `failedAssertions` collects all failures with their `alerts` metadata
- `slackAssertions` is derived by filtering `failedAssertions` via
  `shouldNotifySlack` immediately before the Slack send
- Adding a new reporting channel is a single `.filter()` line on the
  same `failedAssertions` base

https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd
…ield runWhen merge, deduplicated actor test helpers

- `DescribeConfig` and `TestActorConfig` now use `triggers: { runWhen, alerts }` and `options: { timeout, concurrent/retry }` sub-objects; `name` stays at top level
- `mergeInheritedConfigs` now shallow-merges `runWhen` field-by-field (same as `alerts`) so children only need to override specific trigger keys rather than replacing the entire object
- Extract `resolveActorTestConfig` helper to eliminate duplication between `testActor` and `testStandbyActor`
- Remove `shouldNotifySlack` function (single-property check); inline as `alerts?.slack !== false` with a backward-compat comment
- Export new types: `TriggerConfig`, `DescribeOptions`, `ActorOptions`
- Update config-merging unit tests for new shallow-merge semantics; delete test-report-alerts unit test

https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd
…InheritedTriggers

Aligns internal names with the `triggers` field and `TriggerConfig` type used in the public API.

https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd
Previously runWhen[trigger] === true was required to run, meaning any unset
trigger implicitly disabled the test. Now runWhen[trigger] !== false, so all
triggers are enabled by default and users only need to set false to opt out.

e.g. runWhen: { pullRequest: false } disables only PR runs while
hourly and daily continue to run — no need to list every other trigger.

https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd
…eck function

Rather than shouldRunForTrigger using !== false (silently enabling new trigger
types for all tests), DEFAULT_TRIGGERS is prepended to the merge stack and typed
as Required<RunWhenConfig>. Adding a new TriggerType now causes a compile error
on DEFAULT_TRIGGERS, forcing an explicit opt-in/opt-out decision — preventing
accidental rollout to all existing tests.

shouldRunForTrigger reverts to === true; the opt-out default is purely a config
concern, not a predicate concern.

https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd
@metalwarrior665 metalwarrior665 changed the title Add trigger-based test filtering and alert configuration feat: per-test configuration for triggers and alerts Apr 5, 2026
claude added 9 commits April 5, 2026 12:45
…out semantics

- getMergedTriggers now accepts an optional extra layer, eliminating the duplicate
  DEFAULT_TRIGGERS reference in resolveActorTestConfig
- describe/testActor/testStandbyActor JSDoc examples updated to show opt-out style
  (pullRequest: false) instead of misleading opt-in (daily: true / hourly: true)
- RunWhenConfig and shouldRunForTrigger docs no longer reference internal
  implementation details (DEFAULT_TRIGGERS location)
- Remove orphaned section divider comment

https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd
…abled per directory

DEFAULT_TRIGGERS.hourly is now false — only tests that explicitly set
runWhen: { hourly: true } will run on the hourly trigger.

For backward compatibility without touching test files: set the
BACKWARD_COMPATIBLE_HOURLY_DIR env var (e.g. 'core') in the workflow.
The library uses call-stack inspection to detect which test file is
registering a describe/testActor, and promotes hourly to true for tests
defined under that directory. The Actions workflow no longer needs a
subtest directory input — run all of /platform and let the lib handle
the scoping.

BACKWARD_COMPATIBLE_HOURLY_DIR is exported from the package so callers
can reference the constant name rather than hardcoding the string.

https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd
…ll sites

The layers are now explicit at each call site, making the merge order
readable without jumping to a helper.

https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd
Move DEFAULT_TRIGGERS/DEFAULT_DESCRIBE_OPTIONS/DEFAULT_TEST_ACTOR_OPTIONS to
consts.ts and actor-run helpers (createStartRunFn, createStartStandbyFn,
createStandbyTask, generateRunLink) to utils.ts. lib.ts now contains only
module-level init, the trigger stack, and the exported API (describe,
testActor, testStandbyActor). Also inlines BACKWARD_COMPATIBLE_HOURLY_DIR
as process.env access so it is no longer exported from index.ts.

https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd
No need to pass callerFile as a parameter — both functions live in lib.ts,
so all frames are already filtered by THIS_FILE_BASE. Avoids calling
getCallerFile when BACKWARD_COMPATIBLE_HOURLY_DIR is not set (fast path).

https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd
slack defaults to true (opt-out model, consistent with runWhen).
Required<AlertsConfig> gives a compile error if a new alert channel is
added without an explicit decision. report-tests keeps its !== false
guard since it operates across a process boundary and may be used with
older library versions.

https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd
The hourly override was returning only { runWhen: {...} }, dropping
alerts from DEFAULT_TRIGGERS. Fixed by spreading DEFAULT_TRIGGERS first
and then overriding only runWhen. Also corrects the describe() JSDoc
example to show alerts: { slack: false } to illustrate opt-out semantics.

https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd
…tion

Remove the old "Differences in writing tests" before/after migration guide
(everyone has migrated). Add examples for runWhen/alerts triggers config:
opting out of triggers, inheriting through describe hierarchy, disabling
Slack alerts, hourly tests via BACKWARD_COMPATIBLE_HOURLY_DIR. Update
the core workflow to use backward_compatible_hourly_dir input instead of
subtest: core.

https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd
…le_hourly_dir from workflow

How BACKWARD_COMPATIBLE_HOURLY_DIR is passed to the workflow is TBD.
New tests should opt in to hourly via triggers: { runWhen: { hourly: true } }.

https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd
Copy link
Copy Markdown
Contributor

@ruocco-l ruocco-l left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other than the pullRequest to be explicitly opt in, LGTM

Copy link
Copy Markdown
Contributor

@Patai5 Patai5 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I trust the unit tests that were made by the LLM to do their job 😎

@metalwarrior665
Copy link
Copy Markdown
Member Author

I have reverted the refactoring that moved some stuff to utils.ts, it was making the diff longer and making merging master harder. Let's do it later.

@metalwarrior665
Copy link
Copy Markdown
Member Author

Corrected after merge conflicts. I will let it sit for a few days, then will do the last review, ping, and merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants