Deploy VMware Tanzu Packages from a private Container Registry

Robert Guske included in Platforms Tools

2022-02-09 2470 words 12 minutes

Contents

Offline Requirements

During a recent Tanzu engagement, my customer wanted to make use of the VMware Tanzu Packages to extend the core functionalities of their existing Kubernetes clusters. As a requirement, an offline deployment of the Packages was raised, which in other words means, deployments from their own private container registry. The intention behind it is simple.

All of the Tanzu Packages shown in Table I below are provided by VMware. VMware is ensuring that the images for the Packages are stable, secure and compliant. But in order to deploy a Package, we need unristricted internet acceess to *.projects.registry.vmware.com. To bypass this “online” requirement, deploying the Packages from a private container registry like Harbor for example, is a typical use case for most clients.

Table I: List of User-Managed Packages

Function	Package Package	Repository
Certificate management	cert-manager	tanzu-standard
Container networking	multus-cni	tanzu-standard
Container registry	harbor	tanzu-standard
Ingress control	contour	tanzu-standard
Log forwarding	fluent-bit	tanzu-standard
Monitoring	grafana	tanzu-standard
Monitoring	prometheus	tanzu-standard
Service discovery	external-dns	tanzu-standard

Repository URL

Table column Repository, is the name of the package repository (tanzu-standard) which is configured by default in namespace tanzu-package-repo-global and which normally is configured to use URL projects.registry.vmware.com/tkg/packages/standard/repo.

By following the steps provided in this post, you will be able to deploy the kapp-controller as well as all Tanzu Packages from your own private registry onto your eagerly waiting Kubernetes clusters. We start with some prerequisites.

Prerequisites

We have to equip our shell with some additional CLI’s in order to proceed further.

Tanzu Cli

The tanzu CLI ships alongside a compatible version of the kubectl CLI. Download the CLI(s) from VMware’s Customer Connect portal.

Download - Tanzu CLI

If you are new to the topic, check out the official documentation to get even more details.

Documentation - Tanzu CLI

In short for Linux (MacOS and Windows instructions via Docs):

        
# unpack the tar ball
tar -xvf tanzu-cli-bundle-linux-amd64.tar

        
        
        
    
# unpacked tar ball
.
├── cluster
│   ├── plugin.yaml
│   └── v1.4.1
│       └── tanzu-cluster-linux_amd64
├── core
│   ├── plugin.yaml
│   └── v1.4.1
│       └── tanzu-core-linux_amd64
├── imgpkg-linux-amd64-v0.10.0+vmware.1.gz
├── kapp-linux-amd64-v0.37.0+vmware.1.gz
├── kbld-linux-amd64-v0.30.0+vmware.1.gz
├── kubernetes-release
│   ├── plugin.yaml
│   └── v1.4.1
│       └── tanzu-kubernetes-release-linux_amd64
├── login
│   ├── plugin.yaml
│   └── v1.4.1
│       └── tanzu-login-linux_amd64
├── management-cluster
│   ├── plugin.yaml
│   └── v1.4.1
│       └── tanzu-management-cluster-linux_amd64
├── manifest.yaml
├── package
│   ├── plugin.yaml
│   └── v1.4.1
│       └── tanzu-package-linux_amd64
├── pinniped-auth
│   ├── plugin.yaml
│   └── v1.4.1
│       └── tanzu-pinniped-auth-linux_amd64
├── vendir-linux-amd64-v0.21.1+vmware.1.gz
└── ytt-linux-amd64-v0.34.0+vmware.1.gz

        
# install the tanzu cli
sudo install cli/core/v1.4.0/tanzu-core-linux_amd64 /usr/local/bin/tanzu

# version check
tanzu version

# tanzu cli update
tanzu update

# remove existing plugins from any previous CLI installations
tanzu plugin clean

# install all the plugins for the specific TKG release
tanzu plugin install --local cli all

# check plugin installation status
tanzu plugin list

kubectl

The compatible kubectl version can be downloaded via the provided link as well.

Just to have it mentioned - I used the alias k=kubectl actually for all my code examples in this post.

        
# unpack the binary
gzip -d kubectl-linux-v1.21.2+vmware.1.gz

# make it executable
chmod +x kubectl-linux-v1.21.2+vmware.1

# move the executable to your /usr/local/bin
sudo mv kubectl-linux-v1.21.2+vmware.1 /usr/local/bin/kubectl

# version check
kubectl version

imgpkg

The Carvel tool imgpkg, pronounced “image package”, is a tool that allows you to store a set of arbitrary files as an OCI image and is also being used for e.g. deploying Tanzu Kubernetes Grid in Internet-restricted environments and when building your own machine images.

        
# change into the cli folder of the unpacked Tanzu cli
cd cli

# unpack the Gnu zipped archive
gunzip imgpkg-linux-amd64-v0.10.0+vmware.1.gz

# make it executable
chmod ugo+x imgpkg-linux-amd64-v0.10.0+vmware.1

# move the binary into /usr/local/bin
mv ./imgpkg-linux-amd64-v0.10.0+vmware.1 /usr/local/bin/imgpkg

Create a new Project in Harbor

Create a new project/repository in your registry to which we can push (upload) the packages (images) to. I’ve created a new one called tanzu-packages in my Harbor registry.

/img/posts/202202_tanzu_packages_offline/rguske_post_empty_harbor_repo.png — Figure I: New Harbor Project

I’ve written a couple of blog posts covering e.g. the deployment using helm or Tanzu Mission Control Catalog as well as for configuring specific features of the Harbor container registry.

Check out:

📝 Deploying Tanzu Packages using Tanzu Mission Control Catalog

📝 vSphere 7 with Kubernetes supercharged - Helm, Harbor and Tanzu Kubernetes Grid

📝 Using Harbor with the VMware Event Broker Appliance

Download the Harbor Certificate

You can skip this section if you are using a different registry.

Please have the certificate of your registry by hand because we’ll need it a few times.

Download the Harbor certificate:

        
# enter the FQDN of your Harbor instance (e.g. harbor.jarvis.tanzu)
read REGISTRY

# download the certificate
sudo wget -O ~/Downloads/ca.crt https://$REGISTRY/api/v2.0/systeminfo/getcert --no-check-certificate

Add it to your Docker config:

        
# create a folder named like your registry
sudo mkdir -p /etc/docker/certs.d/$REGISTRY

# download the certificate
sudo wget -O /etc/docker/certs.d/$REGISTRY/ca.crt https://$REGISTRY/api/v2.0/systeminfo/getcert --no-check-certificate

# restart docker daemon
systemctl restart docker

# login
docker login harbor.jarvis.tanzu

Username: admin
Password:
Login Succeeded

`pull` and `push` the Kapp-Controller Image

The first component which we make offline (offline == private registry) available is the kapp-controller in VMware’s own maintained version (v0.23.0_vmware.1).

kapp-controller

kapp-controller’s declarative APIs and layered approach enables you to build, deploy, and manage your own applications.

Source: https://carvel.dev/kapp-controller/

        
# pull the kapp-controller image locally first
docker pull projects.registry.vmware.com/tkg/kapp-controller:v0.23.0_vmware.1

# tag the kapp-controller image to be prepared for the image push
docker tag projects.registry.vmware.com/tkg/kapp-controller:v0.23.0_vmware.1 harbor.jarvis.tanzu/tanzu-packages/kapp-controller:v0.23.0_vmware.1

Locally it should look like the following:

        
docker images

REPOSITORY                                           TAG                IMAGE ID       CREATED        SIZE
harbor.jarvis.tanzu/tanzu-packages/kapp-controller   v0.23.0_vmware.1   0076b17e8c71   5 months ago   639MB
projects.registry.vmware.com/tkg/kapp-controller     v0.23.0_vmware.1   0076b17e8c71   5 months ago   639MB

Push it:

        
# push the image to the destination registry
docker push harbor.jarvis.tanzu/tanzu-packages/kapp-controller:v0.23.0_vmware.1

/img/posts/202202_tanzu_packages_offline/rguske_post_kapp_push_harbor.png — Figure II: Pushed kapp-controller image

Install the Kapp-Controller

In order to install the kapp-controller to our Kubernetes cluster, a deployment manifest file has to be created first. Create a new file called e.g. kapp-controller.yaml and add the provided specifications from the docs.

Open the file and jump to line number 1278. By using vim for example, you can simply enable line numbers. Just enter :set numbers and jump to the line by executing :1278. Otherwise, simply search / for image: projects.

Replace the original repository and image with yours.

        
# replace the old image url
[...]

1278         image: harbor.jarvis.tanzu/tanzu-packages/kapp-controller:v0.23.0_vmware.1
1279         name: kapp-controller
1280         ports:

[...]

Install the kapp-controller:

        
        
        
    
k apply -f kapp-controller.yaml

namespace/tkg-system created
namespace/tanzu-package-repo-global created
apiservice.apiregistration.k8s.io/v1alpha1.data.packaging.carvel.dev created
service/packaging-api created
customresourcedefinition.apiextensions.k8s.io/internalpackagemetadatas.internal.packaging.carvel.dev created
customresourcedefinition.apiextensions.k8s.io/internalpackages.internal.packaging.carvel.dev created
customresourcedefinition.apiextensions.k8s.io/apps.kappctrl.k14s.io created
customresourcedefinition.apiextensions.k8s.io/packageinstalls.packaging.carvel.dev created
customresourcedefinition.apiextensions.k8s.io/packagerepositories.packaging.carvel.dev created
configmap/kapp-controller-config created
deployment.apps/kapp-controller created
serviceaccount/kapp-controller-sa created
clusterrole.rbac.authorization.k8s.io/kapp-controller-cluster-role created
clusterrolebinding.rbac.authorization.k8s.io/kapp-controller-cluster-role-binding created
clusterrolebinding.rbac.authorization.k8s.io/pkg-apiserver:system:auth-delegator created
rolebinding.rbac.authorization.k8s.io/pkgserver-auth-reader created

Notice: The Kubernetes namespaces tkg-system as well as tanzu-package-repo-global was created as part of the deployment. Keep them in mind!

Installation successfully ✅

        
# show Deployment, Pod, Configmap and Replicaset
k -n tkg-system get deploy,po,configmap,replicasets.apps

NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kapp-controller   1/1     1            1           22h

NAME                                   READY   STATUS    RESTARTS   AGE
pod/kapp-controller-5fd59df9dd-xmvmj   1/1     Running   0          21h

NAME                               DATA   AGE
configmap/kapp-controller-config   5      22h
configmap/kube-root-ca.crt         1      22h

NAME                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/kapp-controller-5fd59df9dd   1         1         1       22h

Making Packages offline available

Since our goal is to deploy Tanzu Packages from our private registry, we have to make the packages (images) offline available first. Downloading, bundling and uploading the packages can easily be performed with the following two imgpkg commands:

Registry Certificate

The registry certificate is needed to upload the image bundle (--registry-ca-cert-path=ca.crt).

        
# use the imgpkg copy command to download the packages and to pack them into a tarball
imgpkg copy -b projects.registry.vmware.com/tkg/packages/standard/repo:v1.4.0 --to-tar ~/Downloads/packages/packages.tar

# use the imgpkg copy command to `push` the content into your private container registry
imgpkg copy --tar packages.tar --to-repo harbor.jarvis.tanzu/tanzu-packages/tanzu-packages --registry-ca-cert-path=ca.crt --registry-username=admin --registry-password=$PASSWORD

🎥 Watch the thrilling and somewhat boring recording: (⏩ is your friend 😉)

/img/posts/202202_tanzu_packages_offline/rguske_post_packages_bundle_uploaded.png — Figure III: Pushed kapp-controller image

Update the Kapp-Controller Configmap

Before we are able to announce the new repository to the kapp-controller (tanzu package repository add) we have to make the kapp-controller trust our private registry by providing the certificate. In order to make this happen, the kapp-controller configMap needs an update.

OTHERWISE you will quickly see this error message:

        
tanzu package repository add tanzu-packages-offline --url harbor.jarvis.tanzu/tanzu-packages/tanzu-packages:v1.4.0 -n tanzu-package-repo-global

- Adding package repository 'tanzu-packages-offline'

| Getting package repository 'tanzu-packages-offline'
| Validating provided settings for the package repository
| Updating package repository resource
- Waiting for 'PackageRepository' reconciliation for 'tanzu-packages-offline'

Error: resource reconciliation failed: Error: Syncing directory '0': Syncing directory '.' with imgpkgBundle contents: Imgpkg: exit status 1 (stderr: Error: Checking if image is bundle: Collecting images: Working with harbor.jarvis.tanzu/tanzu-packages/tanzu-packages:v1.4.0: Get "https://harbor.jarvis.tanzu/v2/": x509: certificate signed by unknown authority
)
. Reconcile failed: Fetching resources: Error (see .status.usefulErrorMessage for details)
Error: exit status 1

✖  exit status 1

Certificate Error

x509: certificate signed by unknown authority

And by checking the reconciliation status of the repository you will see:

        
tanzu package repository list -A

| Retrieving repositories...
  NAME                    REPOSITORY                                         TAG     STATUS                   DETAILS                  NAMESPACE
  tanzu-packages-offline  harbor.jarvis.tanzu/tanzu-packages/tanzu-packages  v1.4.0  Reconcile failed: Fe...  Error: Syncing direc...  tanzu-package-repo-global

Certificates… 🙄

Let’s get it done:

Step 1: Edit the kapp-controller configMap:

        
# edit configMap
k -n tkg-system edit cm kapp-controller-config

Step 2: Add your certificate data under section caCerts like shown in my example below:

        
        
        
    
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  caCerts: |
    -----BEGIN CERTIFICATE-----
    MIIDWzCCAkOgAwIBAgIRAMODWpLzIWy3JocU4JBtIrIwDQYJKoZIhvcNAQELBQAw
    LTEXMBUGA1UEChMOUHJvamVjdCBIYXJib3IxEjAQBgNVBAMTCUhhcmJvciBDQTAe
    Fw0yMjAxMTExMDEwMDlaFw0zMjAxMDkxMDEwMDlaMC0xFzAVBgNVBAoTDlByb2pl
    Y3QgSGFyYm9yMRIwEAYDVQQDEwlIYXJib3IgQ0EwggEiMA0GCSqGSIb3DQEBAQUA

    [...]
    -----END CERTIFICATE-----    
  dangerousSkipTLSVerify: ""
  httpProxy: ""
  httpsProxy: ""
  noProxy: ""
kind: ConfigMap
metadata:
  annotations:
    kapp.k14s.io/change-group: apps.kappctrl.k14s.io/kapp-controller-config
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"caCerts":"","dangerousSkipTLSVerify":"","httpProxy":"","httpsProxy":"","noProxy":""},"kind":"ConfigMap","metadata":{"annotations":{"kapp.k14s.io/change-group":"apps.kappctrl.k14s.io/kapp-controller-config"},"name":"kapp-controller-config","namespace":"tkg-system"}}      
  creationTimestamp: "2022-02-08T12:31:35Z"
  name: kapp-controller-config
  namespace: tkg-system
  resourceVersion: "1168617"
  uid: 73e9cc55-02d4-4790-a7ee-eaa38b15c894

Save your adjustments :wq.

Step 3: Restart/delete the kapp-controller pod in order to let the changes take effect:

        
# delete the kapp-controller pod
k -n tkg-system delete pod kapp-controller-5fd59df9dd-xmvmj

Step 4: Add the URL, which is pointing to your private package repository and use the namespace tanzu-package-repo-global:

        
# add the offline package repository
tanzu package repository add tanzu-packages-offline --url harbor.jarvis.tanzu/tanzu-packages/tanzu-packages:v1.4.0 -n tanzu-package-repo-global

Different Namespace

If you choose a different namespace, the Packages will only be visible and deployable in that particular namespace.

Source: beyondelastic.com - Tanzu Packages Explained

❗ Don’t stop here ❗ Adding the certificate to the kapp-controller configMap is only half the way if your repository is not configured as Public (without authentication).

Create a Secret to Authenticate with your Registry

To let the kapp-controller authenticate with your private registry, you have to create a Kubernetes secret which in turn has to be referenced in the PackageRepository custom resource (CR).

Step 1: Create the Kubernetes secret:

        
# create a k8s docker-registry secret to authenticate with your registry
kubectl -n tanzu-package-repo-global create secret docker-registry harbor-creds --docker-server='harbor.jarvis.tanzu' --docker-username='admin' --docker-password='your-password' --docker-email='rguske@vmware.com'

        
        
        
    
# validate the creation of the secret
k get secrets -n tanzu-package-repo-global      
NAME                                                    TYPE                                  DATA   AGE
cert-manager-tanzu-package-repo-global-sa-token-6n57n   kubernetes.io/service-account-token   3      3h6m
default-token-wzv5l                                     kubernetes.io/service-account-token   3      24h
harbor-creds 

Step 2:

Adjust the PackageRepository CR accordingly to use the new secret and to ultimately authenticate with your registry:

        
        
        
    
k -n tanzu-package-repo-global edit packagerepositories.packaging.carvel.dev tanzu-packages-offline 

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: packaging.carvel.dev/v1alpha1
kind: PackageRepository
metadata:
  creationTimestamp: "2022-02-09T10:11:02Z"
  finalizers:
  - finalizers.packagerepository.packaging.carvel.dev/delete
  generation: 1
  name: tanzu-packages-offline
  namespace: tanzu-package-repo-global
  resourceVersion: "1464972"
  uid: d42032e9-5534-4503-a065-469c3434a94d
spec:
  fetch:
    imgpkgBundle:
      image: harbor.jarvis.tanzu/tanzu-packages/tanzu-packages:v1.4.0
      secretRef:
        name: harbor-creds
[...]

Step 3:

Validate that the changes has taken effect and that the kapp-controller can successfully reconcile the repository. The status for the package repository should have changed from Reconcile failed: to Reconcile succeeded.

        
# validate the configuration of the repository
tanzu package repository list -n tanzu-package-repo-global

- Retrieving repositories...
  NAME                    REPOSITORY                                         TAG     STATUS               DETAILS
  tanzu-packages-offline  harbor.jarvis.tanzu/tanzu-packages/tanzu-packages  v1.4.0  Reconcile succeeded

Step 4: Also, validate the available (offline) packages:

        
        
        
    
# check package availability
tanzu package available list -n tanzu-package-repo-global

- Retrieving available packages...
  NAME                           DISPLAY-NAME  SHORT-DESCRIPTION                                                                                           LATEST-VERSION
  cert-manager.tanzu.vmware.com  cert-manager  Certificate management                                                                                      1.1.0+vmware.1-tkg.2
  contour.tanzu.vmware.com       Contour       An ingress controller                                                                                       1.17.1+vmware.1-tkg.1
  external-dns.tanzu.vmware.com  external-dns  This package provides DNS synchronization functionality.                                                    0.8.0+vmware.1-tkg.1
  fluent-bit.tanzu.vmware.com    fluent-bit    Fluent Bit is a fast Log Processor and Forwarder                                                            1.7.5+vmware.1-tkg.1
  grafana.tanzu.vmware.com       grafana       Visualization and analytics software                                                                        7.5.7+vmware.1-tkg.1
  harbor.tanzu.vmware.com        Harbor        OCI Registry                                                                                                2.2.3+vmware.1-tkg.1
  multus-cni.tanzu.vmware.com    multus-cni    This package provides the ability for enabling attaching multiple network interfaces to pods in Kubernetes  3.7.1+vmware.1-tkg.1
  prometheus.tanzu.vmware.com    prometheus    A time series database for your metrics                                                                     2.27.0+vmware.1-tkg.1

Deploy the Tanzu Package Cert-Manager

Now that we have our private repository configured, let’s try deploying a first Tanzu Package to our Kubernetes cluster. The cert-manager supports you to e.g. have self-signed certificates generated for further Tanzu Packages installations as well as to be kind of an issuer of your own provided certificates.

        
# list cert-manager package availability
tanzu package available list cert-manager.tanzu.vmware.com -n tanzu-package-repo-global

# check a specific cert-manager version
tanzu package available get cert-manager.tanzu.vmware.com/1.1.0+vmware.1-tkg.2 -n tanzu-package-repo-global

# install cert-manager version
tanzu package install cert-manager -p cert-manager.tanzu.vmware.com -v 1.1.0+vmware.1-tkg.2 -n tanzu-package-repo-global

The deployment process is going to notify you when the installation of the Tanzu Package has finished.

        
[...]
- Waiting for 'PackageInstall' reconciliation for 'cert-manager'
/ 'PackageInstall' resource install status: Reconciling

 Added installed package 'cert-manager'

🎉 🎉 🎉

The reconciliation status of a specific Package can be verified every time using e.g.:

        
tanzu package installed list -n tanzu-package-repo-global

- Retrieving installed packages...
  NAME          PACKAGE-NAME                   PACKAGE-VERSION       STATUS
  cert-manager  cert-manager.tanzu.vmware.com  1.1.0+vmware.1-tkg.2  Reconcile succeeded

OPTIONAL: kapp-controller secret vs. configMap

Instead of making adjustments to the configMap of the kapp-controller, a creation of a Kubernetes secret can be used as an alternative. I validated both ways successfully. A more detailed description can be found on the official Carvel documentation here: Configuring the Controller

Here’s my example I’ve tested:

        
        
        
    
apiVersion: v1
kind: Secret
metadata:
  # Name must be `kapp-controller-config` for kapp controller to pick it up
  name: kapp-controller-config
  # Namespace must match the namespace kapp-controller is deployed to
  namespace: tkg-system
stringData:
  # A cert chain of trusted ca certs. These will be added to the system-wide
  # cert pool of trusted ca's (optional)
  caCerts: |
    -----BEGIN CERTIFICATE-----
    MIIDWzCCAkOgAwIBAgIRAMODWpLzIWy3JocU4JBtIrIwDQYJKoZIhvcNAQELBQAw

    [...]
    -----END CERTIFICATE-----    
  # The url/ip of a proxy for kapp controller to use when making network
  # requests (optional)
  httpProxy: ""
  # The url/ip of a tls capable proxy for kapp controller to use when
  # making network requests (optional)
  httpsProxy: ""
  # A comma delimited list of domain names which kapp controller should
  # bypass the proxy for when making requests (optional)
  noProxy: ""
  # A comma delimited list of domain names for which kapp controller, when
  # fetching images or imgpkgBundles, will skip TLS verification. (optional)
  dangerousSkipTLSVerify: ""

Resources

VMware Docs - Install the Tanzu CLI and Other Tools
VMware Docs - Install and Configure Packages
beyondelastic.com - Tanzu Packages Explained
beyondelastic.com - Configure vSphere with Tanzu behind a Proxy plus TKG Extensions
Stay here 😉 - Deploying Tanzu Packages using Tanzu Mission Control Catalog
Carvel Docs - Configuring the Controller
Carvel Docs - Authenticating to Private Registries
Kubernetes Docs - Pull an Image from a Private Registry