Everyone can contribute! Learn DevOps and Cloud Native in our cafe ☕

Highlights

We went from a quick introduction of the Docker Hub Rate Limits into a thoughtful discussion round of whom is affected and how to mitigate further. Every CI/CD pipeline and job being run in a container is affected, and likewise deployments run in containerized environments such as Kubernetes clusters.

Next to caching proxies, we will see a need for building and maintaining your own Docker images. Following the OCI specification, this needs a central repository hosting the Dockerfile definitions, CI jobs for building, tagging and pushing a local container registry, e.g. in GitLab.

There is no clear prediction on whether Docker CLI or podman or $tool will win. As a developer, we probably do not care as they all implement an API specification and “something” does “something” to format, build, test, deploy our code.

We’ve also discussed the changed image names (short image names vs. absolute registry URLs) and the changes required in every CI/CD config file in every project. Niclas shared their best practice on updating CI configuration files with renovate, similar to a bot updating software dependencies and submitting merge requests.

Building and maintaining container images moves more responsibilities to ops, with security deeply tied into this process. Container scanning will become the standard process: A newly tagged image renders older versions potentially out-of-date and needs automated scanning for vulnerabilities and needed updates.

Conclusion was to pick the solution which fits best into the current environment. Invest into a local container registry and own images, with integrated automation and security. If you cannot afford the resources now, invest into the Docker Hub pro tier and buy some time to schedule an infrastructure shift next year.

Recording

Enjoy the session! 🦊

Cafe Notes

• Docker - check your current pull rate limits with curl
• Mitigate Docker Hub Limits with image caching
• GitLab Dependency Proxy moving to Core

Docker Image Creation

• DinD: https://github.com/docker-library/docker/issues/38
• K8s executor in GitLab Runner: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/46612/diffs

Ideas and discussions

GitLab Infra Insights:

• https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11113#note_438640819
• https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11781#note_440178331

Build own Images

As a Developer, I do not care about Docker, Podman, or others. It is just an API. Similar to CI/CD executors.

How to build container images - is it easy/hard?

• OCI Specification
• https://github.com/GoogleContainerTools/jib
• https://docs.gitlab.com/ee/ci/docker/using_docker_build.html

Rebuilding images, it can be painful to migrate from the short Docker Hub URLs to the longer URLs. Create a CI job, build your image and push it to the registries. Find a good way to update the existing .gitlab-ci.yml configuration file with the image:.

• Pull images, tag them, push them to the GitLab registry. This is a PoC: https://gitlab.com/greg/docker

• Use renovate to update the image in the CI config

  • Example configuration

• Automate this in a project :)

• We also need to take care about security for containers, e.g. Container Scanning in GitLab

• You can edit /etc/docker/daemon.json and add the registry-mirrors key shown in the blog post.

Monitoring

• Monitoring Plugin written in Python 3 by GitLab Developer Evangelists - draft blog post MR](https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_requests/66940/diffs)
• Prometheus Node Exporter? Keep the scrape interval high, every pull counts.
• Are there better APIs available or do we just migrate away from Docker Hub?

Public registries

• AWS
• Digital Ocean
• Harbor

Insights

• Agenda
• Guests: Niclas Mietz, Michael Friedrich, Michael Aigner, Marcel Weinberg, Christop Stoettner
• Next ☕ chat #8 - Keptn

---

Date published: November 4, 2020