0.14.0: listener can remain deleted after AutoscalingRunnerSet patch when minRunners > 0, leaving jobs queued

[tonypottera24]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.14.0

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes.

To Reproduce

1. Deploy the official gha-runner-scale-set-controller / gha-runner-scale-set charts at 0.14.0.
2. Configure an org-level scale set with minRunners > 0 (in our case minRunners: 2, maxRunners: 4).
3. Let the pool keep warm idle runners.
4. Patch the AutoscalingRunnerSet spec (in our case we changed minRunners / maxRunners a few times while tuning the pool).
5. The controller deletes the out-of-date listener.
6. If warm runners are still present, the listener is not recreated and new GitHub jobs targeting that scale set remain queued.

Describe the bug

We are seeing a repeated listener deadlock on 0.14.0 with a warm ARC build pool.

After patching the build AutoscalingRunnerSet, the controller deletes the listener and then gets stuck in this state:

• AutoscalingListener does not exist.
• Creating a new AutoscalingListener is waiting for the running and pending runners to finish

The important detail is that the remaining runners are not active jobs anymore; they are idle warm runners from minRunners > 0.

When this happens:

• the scale set has no listener pod
• at least one warm EphemeralRunner is still present / Running
• new workflow jobs targeting the scale set stay queued indefinitely

Manual workaround: deleting the idle EphemeralRunner immediately allows ARC to recreate the listener, and the queued workflow starts running.

This looks very similar to #4200, but we are reproducing it on 0.14.0, which should already include #4289.

Describe the expected behavior

Patching an AutoscalingRunnerSet with minRunners > 0 should not leave the scale set without a listener while idle warm runners still exist.

ARC should either:

• recreate the listener while warm runners still exist, or
• correctly distinguish idle warm runners from runners that should block listener recreation

New jobs should not remain queued indefinitely after a scale set patch.

Additional Context

Relevant AutoscalingRunnerSet config:

spec:
  runnerScaleSetName: <build-scale-set>
  githubConfigUrl: https://github.com/<redacted-org>
  minRunners: 2
  maxRunners: 4
  listenerTemplate:
    spec:
      nodeSelector:
        kubernetes.io/os: linux
        kubernetes.io/arch: amd64
  template:
    spec:
      nodeSelector:
        kubernetes.io/os: linux
        kubernetes.io/arch: amd64
      containers:
        - name: runner
          image: ghcr.io/actions/actions-runner@<redacted-digest>
          resources:
            requests:
              cpu: "4"
              memory: 8Gi
            limits:
              cpu: "16"
              memory: 32Gi
        - name: dind
          image: docker:dind@<redacted-digest>
          resources:
            requests:
              cpu: "4"
              memory: 8Gi
            limits:
              cpu: "16"
              memory: 32Gi
          securityContext:
            privileged: true

Observed on 2026-04-04.

One concrete affected workflow run was a build job in a private repository that stayed queued until we manually deleted the idle EphemeralRunner.

Controller Logs

Most relevant excerpt:

2026-04-04T15:28:00Z INFO AutoscalingRunnerSet RunnerScaleSetListener is out of date. Deleting it so that it is recreated {"autoscalingrunnerset":{"name":"<build-scale-set>","namespace":"github-actions-runners"},"name":"<build-scale-set>-listener"}
2026-04-04T15:28:00Z INFO AutoscalingRunnerSet Deleted RunnerScaleSetListener since existing one is out of date {"autoscalingrunnerset":{"name":"<build-scale-set>","namespace":"github-actions-runners"}}
2026-04-04T15:28:02Z INFO AutoscalingRunnerSet AutoscalingListener does not exist. {"autoscalingrunnerset":{"name":"<build-scale-set>","namespace":"github-actions-runners"}}
2026-04-04T15:28:02Z INFO AutoscalingRunnerSet Creating a new AutoscalingListener is waiting for the running and pending runners to finish. Waiting for the running and pending runners to finish: {"autoscalingrunnerset":{"name":"<build-scale-set>","namespace":"github-actions-runners"},"running":2,"pending":0}
2026-04-04T15:31:01Z INFO AutoscalingRunnerSet AutoscalingListener does not exist. {"autoscalingrunnerset":{"name":"<build-scale-set>","namespace":"github-actions-runners"}}
2026-04-04T15:31:01Z INFO AutoscalingRunnerSet Creating a new AutoscalingListener is waiting for the running and pending runners to finish. Waiting for the running and pending runners to finish: {"autoscalingrunnerset":{"name":"<build-scale-set>","namespace":"github-actions-runners"},"running":1,"pending":0}

If additional controller or runner logs would be useful, I can provide a further-redacted subset privately in follow-up.

Runner Pod Logs

The runner pods themselves were not crash-looping. In our case the issue was that idle warm runners remained present while the listener was gone, which blocked new job assignment until we manually deleted the idle EphemeralRunner.

[github-actions]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

[nikola-jokic]

Hey @tonypottera24,

What update strategy are you using when you installed the controller?

[tonypottera24]

We are using updateStrategy: eventual on the controller side.

This is set via the official gha-runner-scale-set-controller chart values (flags.updateStrategy: eventual) on 0.14.0.

If it helps, I can also provide the exact controller values block (redacted) or try reproducing the same behavior with a different update strategy.

[tonypottera24]

We reproduced the same behavior again today, and it is not limited to our build scale set.

We now see the same deadlock on a separate CI scale set with the same minRunners > 0 pattern:

• AutoscalingListener for the scale set does not exist
• the corresponding EphemeralRunnerSet still reports warm runners as running
• GitHub still shows those runners as online
• new jobs targeting the scale set remain queued

One important refinement to the workaround from my original report:

• deleting a single idle EphemeralRunner is not always sufficient when minRunners > 0
• ARC can immediately create replacement warm runners, and listener recreation stays blocked
• the reliable manual recovery for us was:
  1. temporarily scale the corresponding EphemeralRunnerSet.spec.replicas to 0
  2. wait for the listener to be recreated
  3. then allow the warm pool to come back

So at least from our side, this still looks like a listener recreation deadlock tied to warm runners, not something specific to one scale set type or to Docker-in-Docker.

[nikola-jokic]

Hey @tonypottera24,

This is exactly described behavior for that upgrade type:

actions-runner-controller/charts/gha-runner-scale-set-controller/values.yaml

Line 123 in 79e7b17

## - "eventual": The controller will remove the listener and ephemeral runner set

[tonypottera24]

Thanks, that part makes sense.

We understand that with updateStrategy: eventual, ARC is expected to delay recreating the listener / runner set until pending and running runners are drained.

What seems off on our side is that this does not just delay the rollout, it can get stuck in a state that requires manual intervention:

• the listener stays absent
• warm idle runners remain present / online
• new jobs targeting the scale set stay queued
• the system does not appear to recover on its own in a reasonable time

In our recent reproductions, the reliable recovery was to temporarily scale the corresponding EphemeralRunnerSet.spec.replicas to 0, so that the listener could be recreated.

So the main question from our side is:
is it expected with eventual that warm idle runners can indefinitely block listener recreation and new job assignment after a patch, or would you consider that part unexpected?

[nikola-jokic]

Thanks! now I understand the issue. Can you please send the controller log at the time that occurred as well? In the meantime, I will try to reproduce the issue. Thank you for explaining it!

[tonypottera24]

The controller log excerpt currently in the issue body is the most relevant redacted snippet we captured from the time of occurrence.

It is the sequence where ARC deletes the out-of-date listener, then repeatedly reports:

• AutoscalingListener does not exist
• Creating a new AutoscalingListener is waiting for the running and pending runners to finish

with warm runners still counted as running.

If that is not enough context, I can try to provide either:

• a longer redacted time window around that event, or
• a shorter redacted recovery timeline showing what happened until we manually recovered by scaling the corresponding EphemeralRunnerSet.spec.replicas to 0.

If there is a specific surrounding log range or additional controller field you want, let me know and I can narrow it down.

[tonypottera24]

We reproduced this again and confirmed the same consistent conditions across occurrences:

• minRunners > 0
• listener recreation is triggered
• warm runners are still counted as present
• new jobs stay queued while the listener remains absent

We also confirmed the following manual recovery sequence works reliably for us:

1. temporarily stop the affected AutoscalingRunnerSet from creating warm runners
2. remove the stale EphemeralRunnerSet
3. wait for ARC to recreate the AutoscalingListener
4. restore the original warm-pool settings