feat: cache versioned kubelet kubectl package binaries by awesomenix · Pull Request #8287 · Azure/AgentBaker

awesomenix · 2026-04-12T08:56:44Z

What this does

This change avoids unnecessary kubelet/kubectl package installation work during CSE when the corresponding binaries are already
available on the VHD.

Today, Ubuntu and Mariner/Azure Linux cache the kubelet/kubectl PMC packages on the VHD, but CSE still installs them via the package
manager/runtime package flow. That has a few downsides:

it re-runs package installation logic during provisioning
on Ubuntu, dpkg -i triggers package postinst behavior for kubelet.service
we pay extra provisioning latency even though AKS already owns kubelet service configuration and startup

This PR changes the flow so VHD build materializes versioned kubelet/kubectl binaries from the cached package artifacts into:

/opt/bin/kubelet-<k8sVersion>
/opt/bin/kubectl-<k8sVersion>

Then, during CSE, if the requested version is already present in /opt/bin, we reuse the existing cache-first path and do the final
rename into place instead of reinstalling from the package.

Changes

VHD build

extend vhdbuilder/packer/install-dependencies.sh so cached kubelet/kubectl package artifacts also produce versioned binaries
under /opt/bin
support both:
- Ubuntu: extract /usr/bin/<tool> from cached .deb
- Mariner/Azure Linux: extract /usr/bin/<tool> from cached .rpm

CSE

add a shared helper to detect whether versioned kubelet/kubectl binaries already exist in /opt/bin
update Ubuntu and Mariner package-based install paths to:
- use the cached versioned binaries when available
- fall back to existing package install behavior when the cache is not present
- continue respecting SHOULD_ENFORCE_KUBE_PMC_INSTALL=true

Tests

add/extend coverage for the cache-first kubelet/kubectl flow
extend Linux VHD content validation to verify package-backed kubelet/kubectl versions also materialize versioned binaries in
/opt/bin

Why

This keeps kubelet/kubectl aligned with the existing kubernetes-binaries flow:

cache versioned binaries on the image
do a cheap final move during provisioning

Expected benefits:

lower CSE latency
avoid unnecessary package-manager work during provisioning
avoid Ubuntu kubelet package postinst side effects when the VHD already has the requested binary

Notes

this does not remove the existing package fallback path
if the versioned binary is missing, CSE still falls back to the current package installation behavior
SHOULD_ENFORCE_KUBE_PMC_INSTALL=true still forces the package path for validation / test scenarios

Copilot

Pull request overview

This PR optimizes Linux node provisioning by reusing kubelet/kubectl binaries already cached on the VHD (materialized as versioned files under /opt/bin) instead of reinstalling via the package manager during CSE.

Changes:

VHD build: extract kubelet/kubectl binaries from cached .deb/.rpm artifacts into /opt/bin/<tool>-<k8sVersion>.
CSE: add shared helpers to detect/move cached versioned kube binaries; update Ubuntu and Mariner package-based kubelet/kubectl install paths to prefer cache unless SHOULD_ENFORCE_KUBE_PMC_INSTALL=true.
Tests: extend VHD content validation and add/extend ShellSpec coverage for the cache-first behavior.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
`vhdbuilder/packer/install-dependencies.sh`	Adds VHD-build extraction of kubelet/kubectl binaries from cached package artifacts into versioned `/opt/bin` paths.
`vhdbuilder/packer/test/linux-vhd-content-test.sh`	Extends VHD content tests to validate versioned package-backed kubelet/kubectl binaries exist and match expected versions.
`parts/linux/cloud-init/artifacts/cse_install.sh`	Adds shared helpers to detect and move cached versioned kubelet/kubectl binaries; reuses them in the URL-based install flow.
`parts/linux/cloud-init/artifacts/ubuntu/cse_install_ubuntu.sh`	Updates Ubuntu package-based kubelet/kubectl install path to prefer cached versioned binaries.
`parts/linux/cloud-init/artifacts/mariner/cse_install_mariner.sh`	Updates Mariner/AzureLinux package-based kubelet/kubectl install path to prefer cached versioned binaries.
`spec/parts/linux/cloud-init/artifacts/cse_install_ubuntu_spec.sh`	Adds ShellSpec coverage for Ubuntu cache-first kubelet/kubectl install behavior.
`spec/parts/linux/cloud-init/artifacts/cse_install_mariner_spec.sh`	Adds ShellSpec coverage for Mariner cache-first kubelet/kubectl install behavior.

spec/parts/linux/cloud-init/artifacts/cse_install_ubuntu_spec.sh

spec/parts/linux/cloud-init/artifacts/cse_install_mariner_spec.sh

parts/linux/cloud-init/artifacts/ubuntu/cse_install_ubuntu.sh

parts/linux/cloud-init/artifacts/mariner/cse_install_mariner.sh

vhdbuilder/packer/install-dependencies.sh

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

vhdbuilder/packer/test/linux-vhd-content-test.sh

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Copilot · 2026-04-12T18:57:28Z

parts/linux/cloud-init/artifacts/azlosguard/cse_install_osguard.sh

+    echo "Unpacking usr/bin/${packageName} from ${downloadDir}/${packageName}-${desiredVersion}*"
+    mkdir -p "$(dirname "${targetPath}")"
    # This assumes that the binary will either be in /usr/bin or /usr/local/bin, but not both.
-    rpm2cpio "${rpmFile}" | cpio -i --to-stdout "./usr/bin/${rpmBinaryName}" "./usr/local/bin/${rpmBinaryName}" | install -m0755 /dev/stdin "${targetBinDir}/${targetBinaryName}"
-	rm -rf ${downloadDir}
+    rpm2cpio "${rpmFile}" | cpio -i --to-stdout "./usr/bin/${packageName}" "./usr/local/bin/${packageName}" | install -m0755 /dev/stdin "${targetPath}"
+	rm -rf "${downloadDir}"


This rpm2cpio | cpio -i --to-stdout call requests both ./usr/bin/<name> and ./usr/local/bin/<name> at once. If the RPM contains only one of these, cpio can fail (and if both exist it may concatenate both files to stdout), leading to an empty/corrupted binary being installed. Please extract exactly one resolved path (e.g., inspect the RPM file list first, or try /usr/bin/... and fall back to /usr/local/bin/... only if needed).

Copilot · 2026-04-12T18:57:28Z

parts/linux/cloud-init/artifacts/ubuntu/cse_install_ubuntu.sh

+
+    local sourceBinary="${extractDir}/usr/bin/${packageName}"
+    if [ ! -f "${sourceBinary}" ]; then
+        echo "Failed to locate usr/bin/${packageName} in ${debFile}"


Minor: the error message says usr/bin/${packageName} but the intended path is /usr/bin/${packageName} (leading slash missing). Fixing this makes failures clearer when a .deb doesn’t contain the expected binary path.

Suggested change

echo "Failed to locate usr/bin/${packageName} in ${debFile}"

echo "Failed to locate /usr/bin/${packageName} in ${debFile}"

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings April 12, 2026 08:56

awesomenix requested review from AbelHu, Devinwong, SriHarsha001, calvin197, cameronmeissner, djsly, ganeshkumarashok, junjiezhang1997, lilypan26, mxj220, pdamianov-dev, phealy, r2k1, sulixu, surajssd, timmy-wright and zachary-bailey as code owners April 12, 2026 08:56

awesomenix temporarily deployed to test April 12, 2026 08:56 — with GitHub Actions Inactive

Copilot started reviewing on behalf of awesomenix April 12, 2026 08:57 View session

Copilot AI reviewed Apr 12, 2026

View reviewed changes

awesomenix force-pushed the nishp/fastkubeletfrompkg branch from 98894d6 to 9614c74 Compare April 12, 2026 15:27

awesomenix temporarily deployed to test April 12, 2026 15:27 — with GitHub Actions Inactive

awesomenix changed the title ~~Cache versioned kubelet kubectl package binaries~~ feat: cache versioned kubelet kubectl package binaries Apr 12, 2026

Copilot AI review requested due to automatic review settings April 12, 2026 15:34

awesomenix force-pushed the nishp/fastkubeletfrompkg branch from 9614c74 to edbcee6 Compare April 12, 2026 15:34

awesomenix temporarily deployed to test April 12, 2026 15:34 — with GitHub Actions Inactive

Copilot started reviewing on behalf of awesomenix April 12, 2026 15:35 View session

Copilot AI reviewed Apr 12, 2026

View reviewed changes

vhdbuilder/packer/test/linux-vhd-content-test.sh Outdated Show resolved Hide resolved

vhdbuilder/packer/test/linux-vhd-content-test.sh Outdated Show resolved Hide resolved

awesomenix force-pushed the nishp/fastkubeletfrompkg branch from edbcee6 to b1dd3e8 Compare April 12, 2026 17:15

awesomenix temporarily deployed to test April 12, 2026 17:15 — with GitHub Actions Inactive

Copilot AI review requested due to automatic review settings April 12, 2026 18:52

awesomenix force-pushed the nishp/fastkubeletfrompkg branch from b1dd3e8 to 1936c2b Compare April 12, 2026 18:52

awesomenix temporarily deployed to test April 12, 2026 18:52 — with GitHub Actions Inactive

Copilot started reviewing on behalf of awesomenix April 12, 2026 18:53 View session

Copilot AI reviewed Apr 12, 2026

View reviewed changes

Cache versioned kubelet kubectl package binaries

b905626

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

awesomenix force-pushed the nishp/fastkubeletfrompkg branch from 1936c2b to b905626 Compare April 13, 2026 01:53

awesomenix temporarily deployed to test April 13, 2026 01:53 — with GitHub Actions Inactive

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: cache versioned kubelet kubectl package binaries#8287

feat: cache versioned kubelet kubectl package binaries#8287
awesomenix wants to merge 1 commit intomainfrom
nishp/fastkubeletfrompkg

awesomenix commented Apr 12, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 12, 2026

Uh oh!

Copilot AI Apr 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	echo "Failed to locate usr/bin/${packageName} in ${debFile}"
	echo "Failed to locate /usr/bin/${packageName} in ${debFile}"

Conversation

awesomenix commented Apr 12, 2026

What this does

Changes

VHD build

CSE

Tests

Why

Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Apr 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 12, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants