From 9e6848cbf6f7bbdf28c906c1587bf7899b67729e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Henrik=20H=C3=BCttemann?= Date: Thu, 20 Jun 2024 14:19:23 +0200 Subject: [PATCH 1/6] Add package version script --- README.md | 46 +++++++++++++++++++++++++++++++++++++++++++-- get_pkg_versions.sh | 15 +++++++++++++++ 2 files changed, 59 insertions(+), 2 deletions(-) create mode 100755 get_pkg_versions.sh diff --git a/README.md b/README.md index 4e0c04a..e193f46 100644 --- a/README.md +++ b/README.md @@ -22,10 +22,52 @@ steps: - pre-commit run --all-files ``` -If renovate is set up, it'll add and update the pinned digest/hash of the image. +If renovate is set up for your repo, it'll add and update the pinned digest/hash of the image. ## Development +Generally you should have `Docker` or something alike installed. + If you need to copy files into the container, don't forget to add exclusions to the general _exclude all_ in `.dockerignore`. -To update the base image (like `3.12.4-alpine3.20` to a newer Alpine version), manual work is still required. In the `Dockerfile`, update the Alpine version for the image, the renovate comments (`# renovate: datasource=repology depName=alpine_3_20/gcc versioning=loose`), and the package versions for that OS version from the repo (Like on the [Alpine Package Page for gcc](https://pkgs.alpinelinux.org/packages?name=gcc&branch=v3.20&repo=&arch=x86_64)). +To **update the base image** (like `3.12.4-alpine3.20` to a newer Alpine version), manual work is still required, but supported by a little script. **Renovate might not create a PR for newer image tags.** + +1. In the `Dockerfile`, update the Alpine version for the image and the renovate comments (`# renovate: datasource=repology depName=alpine_3_20/gcc versioning=loose`). + + ```diff + - FROM python:3-alpine3.19@sha256:00c0ffeeacab... + + FROM python:3-alpine3.20 # You can omit the sha256 digest, the script prints it out + # ... + + - # renovate: datasource=repology depName=alpine_3_19/build-base versioning=loose + + # renovate: datasource=repology depName=alpine_3_20/build-base versioning=loose + ENV BUILD_BASE_VERSION="0.8.15" + # ... + ``` + +1. Now run `./get_pkg_versions.sh`. It pulls the alpine image from the Dockerfile, prints it's digest and the latest packages it could find via `apk` inside that container and prints out the names and versions. + + Example output of `./get_pkg_versions.sh` for a new image, which is not yet pulled: + + ```plain + Unable to find image 'python:3.12.3-alpine3.18' locally + 3.12.3-alpine3.18: Pulling from library/python + 619be1103602: Pull complete + [...] + 0eb61f1af52e: Pull complete + Digest: sha256:24680ddf8422899b24756d62b31eb5de782fbb42e9c2bb1c70f1f55fcf891721 + Status: Downloaded newer image for python:3.12.3-alpine3.18 + [Script output starts here] + Checking 5/5 latest package versions on python:3.12.3-alpine3.18 + Image digest found: sha256:24680ddf8422899b24756d62b31eb5de782fbb42e9c2bb1c70f1f55fcf891721 + --- + build-base-0.5-r3 + gcc-12.2.1_git20220924-r10 + git-2.40.1-r0 + openssh-keygen-9.3_p2-r1 + ruby-3.2.4-r0 + ``` + +1. Copy the package versions and update the respective `ENV` with it manually in the `Dockerfile`. You also might add the digest to the base image. + +1. Test building the image and you can commit it. diff --git a/get_pkg_versions.sh b/get_pkg_versions.sh new file mode 100755 index 0000000..87b772f --- /dev/null +++ b/get_pkg_versions.sh @@ -0,0 +1,15 @@ +#!/bin/bash +set -euo pipefail +IFS=$'\n\t' + +IMAGE=$(grep -oP 'FROM \K.*alpine[^ ]+' Dockerfile) +PACKAGES=$(grep -oP '#.+depName=alpine.+/\K[^ ]+' Dockerfile) +# shellcheck disable=SC2086 +PACKAGES_NO_BR=$(echo ${PACKAGES} | tr -d '\n') +PACKAGES_VERSIONS=$(docker run --rm -t --entrypoint /bin/sh "$IMAGE" -c "apk --update --no-cache list $PACKAGES_NO_BR | cut -d ' ' -f 1 | grep -v '^fetch$'") +DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' "$IMAGE" | cut -d '@' -f2) + +echo "Checking $(echo "$PACKAGES" | wc -l)/$(echo "$PACKAGES_VERSIONS" | wc -l) latest package versions on $IMAGE" +echo "Image digest found: $DIGEST" +echo "---" +echo "$PACKAGES_VERSIONS" From 62d418d40c06e6c86d8ef1e1ea8065229cd5556e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Henrik=20H=C3=BCttemann?= Date: Thu, 20 Jun 2024 14:28:08 +0200 Subject: [PATCH 2/6] Fix Woodpecker config to run on all file changes Fix linter issues --- .woodpecker.yaml | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/.woodpecker.yaml b/.woodpecker.yaml index 463d1a5..ffb8897 100644 --- a/.woodpecker.yaml +++ b/.woodpecker.yaml @@ -1,10 +1,8 @@ -when: - path: '*Dockerfile*' - steps: build-main: when: - branch: main + - event: push + branch: main image: woodpeckerci/plugin-docker-buildx:4.0.0@sha256:9d24b71c37d7a958d79252e608c4d1a04b02f2e74d4e26003b43e0830038bde0 pull: true settings: @@ -18,8 +16,9 @@ steps: build-branch: when: - branch: - exclude: ['main'] + - event: push + branch: + exclude: ['main'] image: woodpeckerci/plugin-docker-buildx:4.0.0@sha256:9d24b71c37d7a958d79252e608c4d1a04b02f2e74d4e26003b43e0830038bde0 pull: true settings: From 7b022827a10c6cf5ac16ac0991e5327abeb9ac08 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Henrik=20H=C3=BCttemann?= Date: Thu, 20 Jun 2024 16:25:44 +0200 Subject: [PATCH 3/6] Remove dependencies to let pre-commit install them --- Dockerfile | 6 ------ 1 file changed, 6 deletions(-) diff --git a/Dockerfile b/Dockerfile index d5f6e51..9991bd2 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,6 +1,4 @@ FROM python:3.12.4-alpine3.20@sha256:a982997504b8ec596f553d78f4de4b961bbdf5254e0177f6e99bb34f4ef16f95 -COPY --from=koalaman/shellcheck:v0.10.0@sha256:2097951f02e735b613f4a34de20c40f937a6c8f18ecb170612c88c34517221fb /bin/shellcheck /usr/bin/ -COPY --from=ghcr.io/gitleaks/gitleaks:v8.18.4@sha256:f44e526acc67786b7476db413edb993ce2d152660d32fb3eb48d9bca06fa83f8 /usr/bin/gitleaks /usr/bin/ # renovate: datasource=repology depName=alpine_3_20/build-base versioning=loose ENV BUILD_BASE_VERSION="0.5-r3" @@ -14,8 +12,6 @@ ENV GIT_VERSION="2.45.2-r0" ENV OPENSSH_KEYGEN_VERSION="9.7_p1-r3" # renovate: datasource=pypi depName=pre-commit versioning=pep440 ENV PRE_COMMIT_VERSION="3.7.1" -# renovate: datasource=rubygems depName=mdl versioning=ruby -ENV MDL_VERSION="0.13.0" RUN mkdir /data /tmp/pre-commit COPY .pre-commit-config.yaml /tmp/pre-commit @@ -23,13 +19,11 @@ COPY .pre-commit-config.yaml /tmp/pre-commit RUN apk add --update --no-cache \ build-base="${BUILD_BASE_VERSION}" \ gcc="${GCC_VERSION}" \ - ruby="${RUBY_VERSION}" \ ruby-dev="${RUBY_VERSION}" \ git="${GIT_VERSION}" \ openssh-keygen="${OPENSSH_KEYGEN_VERSION}" \ && \ pip install --no-cache-dir pre-commit=="${PRE_COMMIT_VERSION}" && \ - gem install --no-document mdl -v "${MDL_VERSION}" && \ git config --global --add safe.directory /data && \ cd /tmp/pre-commit && \ git init --initial-branch main && \ From 64c3f1fa9c79dd21689cb9b0bbe47e8e78ee05af Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Henrik=20H=C3=BCttemann?= Date: Thu, 20 Jun 2024 16:39:03 +0200 Subject: [PATCH 4/6] Add tests for pre-commit container Run pre-commit on the freshly built container against salt and rc2matrix --- .woodpecker.yaml => .woodpecker/.build.yaml | 23 +++++---- .woodpecker/.test.yaml | 56 +++++++++++++++++++++ 2 files changed, 69 insertions(+), 10 deletions(-) rename .woodpecker.yaml => .woodpecker/.build.yaml (72%) create mode 100644 .woodpecker/.test.yaml diff --git a/.woodpecker.yaml b/.woodpecker/.build.yaml similarity index 72% rename from .woodpecker.yaml rename to .woodpecker/.build.yaml index 463d1a5..f59d616 100644 --- a/.woodpecker.yaml +++ b/.woodpecker/.build.yaml @@ -1,10 +1,8 @@ -when: - path: '*Dockerfile*' - steps: - build-main: + build main: when: - branch: main + - event: push + branch: main image: woodpeckerci/plugin-docker-buildx:4.0.0@sha256:9d24b71c37d7a958d79252e608c4d1a04b02f2e74d4e26003b43e0830038bde0 pull: true settings: @@ -14,12 +12,15 @@ steps: password: from_secret: gitea_token repo: git.verdigado.com/${CI_REPO,,} - tag: 'latest' + tags: + - 'latest' + - ${CI_COMMIT_SHA} - build-branch: + build branch: when: - branch: - exclude: ['main'] + - event: push + branch: + exclude: ['main'] image: woodpeckerci/plugin-docker-buildx:4.0.0@sha256:9d24b71c37d7a958d79252e608c4d1a04b02f2e74d4e26003b43e0830038bde0 pull: true settings: @@ -29,4 +30,6 @@ steps: password: from_secret: gitea_token repo: git.verdigado.com/${CI_REPO,,} - tag: ${CI_COMMIT_BRANCH} + tags: + - ${CI_COMMIT_BRANCH} + - ${CI_COMMIT_SHA} diff --git a/.woodpecker/.test.yaml b/.woodpecker/.test.yaml new file mode 100644 index 0000000..69c9a51 --- /dev/null +++ b/.woodpecker/.test.yaml @@ -0,0 +1,56 @@ +skip_clone: true +when: + - event: push +depends_on: + - build +variables: + - &image 'git.verdigado.com/verdigado-images/container-pre-commit:${CI_COMMIT_SHA}' +steps: + await-image: + image: alpine + environment: + IMAGE: *image + commands: + - apk add --update --no-cache img + - 'while !(( img pull $IMAGE 2>&1 | grep -q "Error: failed to unmount" )) ; do echo "Awaiting image $IMAGE..."; sleep 3; done' + - echo 'found.' + + clone salt: + image: woodpeckerci/plugin-git + settings: + remote: https://git.verdigado.com/verdigado-Privileged/Salt.git + path: salt + sha: '' + ref: refs/heads/master + branch: master + + pre-commit salt: + image: *image + depends_on: + - await-image + - clone salt + environment: + - SKIP=no-commit-to-branch # Ignore "don't commit to protected branch" check + commands: + - cd salt + - pre-commit run --all-files + + clone rocketchat2matrix: + image: woodpeckerci/plugin-git + settings: + remote: https://git.verdigado.com/NB-Public/rocketchat2matrix.git + path: rocketchat2matrix + sha: '' + ref: refs/heads/main + branch: master + + pre-commit rocketchat2matrix: + image: *image + depends_on: + - await-image + - clone rocketchat2matrix + environment: + - SKIP=no-commit-to-branch # Ignore "don't commit to protected branch" check + commands: + - cd rocketchat2matrix + - pre-commit run --all-files From e43b05e0fdd30f4cb56b9f74e25fd18d1ba6a6d8 Mon Sep 17 00:00:00 2001 From: Renovate Bot Date: Tue, 25 Jun 2024 07:02:53 +0000 Subject: [PATCH 5/6] Renovate: Pin dependencies --- .woodpecker/.test.yaml | 6 +++--- Dockerfile | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/.woodpecker/.test.yaml b/.woodpecker/.test.yaml index 69c9a51..b63c742 100644 --- a/.woodpecker/.test.yaml +++ b/.woodpecker/.test.yaml @@ -7,7 +7,7 @@ variables: - &image 'git.verdigado.com/verdigado-images/container-pre-commit:${CI_COMMIT_SHA}' steps: await-image: - image: alpine + image: alpine@sha256:b89d9c93e9ed3597455c90a0b88a8bbb5cb7188438f70953fede212a0c4394e0 environment: IMAGE: *image commands: @@ -16,7 +16,7 @@ steps: - echo 'found.' clone salt: - image: woodpeckerci/plugin-git + image: woodpeckerci/plugin-git@sha256:7af90de3a9aa5dc93cc0d5cd2e67e28cb237d4b8e891ccacfd9031f78f4b05a8 settings: remote: https://git.verdigado.com/verdigado-Privileged/Salt.git path: salt @@ -36,7 +36,7 @@ steps: - pre-commit run --all-files clone rocketchat2matrix: - image: woodpeckerci/plugin-git + image: woodpeckerci/plugin-git@sha256:7af90de3a9aa5dc93cc0d5cd2e67e28cb237d4b8e891ccacfd9031f78f4b05a8 settings: remote: https://git.verdigado.com/NB-Public/rocketchat2matrix.git path: rocketchat2matrix diff --git a/Dockerfile b/Dockerfile index 9991bd2..e988a6a 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,11 +1,11 @@ -FROM python:3.12.4-alpine3.20@sha256:a982997504b8ec596f553d78f4de4b961bbdf5254e0177f6e99bb34f4ef16f95 +FROM python:3.12.4-alpine3.20@sha256:dc095966439c68283a01dde5e5bc9819ba24b28037dddd64ea224bf7aafc0c82 # renovate: datasource=repology depName=alpine_3_20/build-base versioning=loose ENV BUILD_BASE_VERSION="0.5-r3" # renovate: datasource=repology depName=alpine_3_20/gcc versioning=loose ENV GCC_VERSION="13.2.1_git20240309-r0" # renovate: datasource=repology depName=alpine_3_20/ruby versioning=loose -ENV RUBY_VERSION="3.3.1-r0" +ENV RUBY_VERSION="3.3.3-r0" # renovate: datasource=repology depName=alpine_3_20/git versioning=loose ENV GIT_VERSION="2.45.2-r0" # renovate: datasource=repology depName=alpine_3_20/openssh-keygen versioning=loose From 9354a361a36944d8a9faeb87310f895d144550b1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Henrik=20H=C3=BCttemann?= Date: Thu, 20 Jun 2024 14:19:23 +0200 Subject: [PATCH 6/6] Add package version script --- README.md | 46 +++++++++++++++++++++++++++++++++++++++++++-- get_pkg_versions.sh | 15 +++++++++++++++ 2 files changed, 59 insertions(+), 2 deletions(-) create mode 100755 get_pkg_versions.sh diff --git a/README.md b/README.md index 4e0c04a..e193f46 100644 --- a/README.md +++ b/README.md @@ -22,10 +22,52 @@ steps: - pre-commit run --all-files ``` -If renovate is set up, it'll add and update the pinned digest/hash of the image. +If renovate is set up for your repo, it'll add and update the pinned digest/hash of the image. ## Development +Generally you should have `Docker` or something alike installed. + If you need to copy files into the container, don't forget to add exclusions to the general _exclude all_ in `.dockerignore`. -To update the base image (like `3.12.4-alpine3.20` to a newer Alpine version), manual work is still required. In the `Dockerfile`, update the Alpine version for the image, the renovate comments (`# renovate: datasource=repology depName=alpine_3_20/gcc versioning=loose`), and the package versions for that OS version from the repo (Like on the [Alpine Package Page for gcc](https://pkgs.alpinelinux.org/packages?name=gcc&branch=v3.20&repo=&arch=x86_64)). +To **update the base image** (like `3.12.4-alpine3.20` to a newer Alpine version), manual work is still required, but supported by a little script. **Renovate might not create a PR for newer image tags.** + +1. In the `Dockerfile`, update the Alpine version for the image and the renovate comments (`# renovate: datasource=repology depName=alpine_3_20/gcc versioning=loose`). + + ```diff + - FROM python:3-alpine3.19@sha256:00c0ffeeacab... + + FROM python:3-alpine3.20 # You can omit the sha256 digest, the script prints it out + # ... + + - # renovate: datasource=repology depName=alpine_3_19/build-base versioning=loose + + # renovate: datasource=repology depName=alpine_3_20/build-base versioning=loose + ENV BUILD_BASE_VERSION="0.8.15" + # ... + ``` + +1. Now run `./get_pkg_versions.sh`. It pulls the alpine image from the Dockerfile, prints it's digest and the latest packages it could find via `apk` inside that container and prints out the names and versions. + + Example output of `./get_pkg_versions.sh` for a new image, which is not yet pulled: + + ```plain + Unable to find image 'python:3.12.3-alpine3.18' locally + 3.12.3-alpine3.18: Pulling from library/python + 619be1103602: Pull complete + [...] + 0eb61f1af52e: Pull complete + Digest: sha256:24680ddf8422899b24756d62b31eb5de782fbb42e9c2bb1c70f1f55fcf891721 + Status: Downloaded newer image for python:3.12.3-alpine3.18 + [Script output starts here] + Checking 5/5 latest package versions on python:3.12.3-alpine3.18 + Image digest found: sha256:24680ddf8422899b24756d62b31eb5de782fbb42e9c2bb1c70f1f55fcf891721 + --- + build-base-0.5-r3 + gcc-12.2.1_git20220924-r10 + git-2.40.1-r0 + openssh-keygen-9.3_p2-r1 + ruby-3.2.4-r0 + ``` + +1. Copy the package versions and update the respective `ENV` with it manually in the `Dockerfile`. You also might add the digest to the base image. + +1. Test building the image and you can commit it. diff --git a/get_pkg_versions.sh b/get_pkg_versions.sh new file mode 100755 index 0000000..87b772f --- /dev/null +++ b/get_pkg_versions.sh @@ -0,0 +1,15 @@ +#!/bin/bash +set -euo pipefail +IFS=$'\n\t' + +IMAGE=$(grep -oP 'FROM \K.*alpine[^ ]+' Dockerfile) +PACKAGES=$(grep -oP '#.+depName=alpine.+/\K[^ ]+' Dockerfile) +# shellcheck disable=SC2086 +PACKAGES_NO_BR=$(echo ${PACKAGES} | tr -d '\n') +PACKAGES_VERSIONS=$(docker run --rm -t --entrypoint /bin/sh "$IMAGE" -c "apk --update --no-cache list $PACKAGES_NO_BR | cut -d ' ' -f 1 | grep -v '^fetch$'") +DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' "$IMAGE" | cut -d '@' -f2) + +echo "Checking $(echo "$PACKAGES" | wc -l)/$(echo "$PACKAGES_VERSIONS" | wc -l) latest package versions on $IMAGE" +echo "Image digest found: $DIGEST" +echo "---" +echo "$PACKAGES_VERSIONS"