Harbor Container Registry¶
About this task
Harbor is an open-source container registry with a richer and fuller set of capabilities than the built-in StarlingX container registry. StarlingX end users can use Harbor to manage their own application container images. Harbor secures artifacts with policies and role-based access control, ensures images are scanned and free from vulnerabilities, and signs images as trusted. Harbor has been evolved to a complete OCI compliant cloud-native artifact registry.
With Harbor V2.0, you can manage images, manifest lists, Helm charts, CNABs, OPAs and other artifacts adhering to the OCI image specification. It supports operations such as pulling, pushing, deleting, tagging, replicating, and scanning these artifacts. Additionally, you can now sign images and manifest lists.
Harbor also supports the replication of images between registries, and offers advanced replication of images between registries, and offers advanced security features such as user management, access control and activity auditing.
See https://goharbor.io/docs/2.0.0/ for more details on Harbor.
Harbor Installation¶
Prerequisites
CephFS backed PVCs are recommended for Harbor such that when configuring multiple replicas, for AIO-DX or Standard configurations, both Harbor replicas can read and write to the registry.
Create a secret as described below:
Generate certificates and create secret.
A CA cert and server cert creation procedure using cert-manager is specified below:
Create the certificate for Harbor using Cert-Manager and using the local CA, system-local-ca, as the issuer. Note that the certificate should be created in the
harbor-tls SECRET
in the HarborNAMESPACE
.For example:
Create the following Harbor certificate yaml configuration file:
~(keystone_admin)]$ cat <<EOF > harbor-certificate.yaml --- apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: harbor-certificate namespace: harbor spec: secretName: harbor-tls issuerRef: name: system-local-ca kind: ClusterIssuer duration: 2160h # 90 days renewBefore: 360h # 15 days commonName: < oam floating IP Address or FQDN > subject: organizations: - ABC-Company organizationalUnits: - StarlingX-harbor ipAddresses: - < oam floating IP address > dnsNames: - < harbor dns> # e.g. harbor.yourdomian.com - < notary dns > # optional, required only if exposed on ingress e.g. notary.yourdomian.com EOF
Create the Harbor namespace:
~(keystone_admin)]$ kubectl create namespace harbor
Apply the configuration:
~(keystone_admin)]$ kubectl apply -f harbor-certificate.yaml
Verify the configuration:
~(keystone_admin)]$ kubectl get certificate harbor-certificate -n harbor
After successful configuration, the certificate’s Ready status will be True.
Procedure
Locate the Harbor system application tarball in
/usr/local/share/applications/helm
.For example:
/usr/local/share/applications/helm/harbor-<version>.tgz
Upload the Harbor application.
~(keystone_admin)]$ system application-upload /usr/local/share/applications/helm/harbor-<version>.tgz
Configure the Helm Overrides for Harbor.
Expose the Harbor application externally with either nodePort or Ingress.
nodePort
Create Harbor using NodePort to expose the service
Note
The instructions below assume that the NodePorts 30002, 30003, and 30004 are available (i.e., not used by any other applications). If these ports are unavailable, please choose and configure alternative ports that are not in use.
Put the following nodePort overrides in
values.yaml
:expose: type: nodePort # Type should be nodeport tls: enabled: true certSource: secret secret: # Certificate Source is secret secretName: "harbor-tls" # A secret containing tls.crt and tls.key notarySecretName: "harbor-tls" # A secret containing tls.crt and tls.key nodePort: # The name of NodePort service name: harbor ports: http: # The service port Harbor listens on when serving HTTP port: 80 # The node port Harbor listens on when serving HTTP nodePort: 30002 https: # The service port Harbor listens on when serving HTTPS port: 443 # The node port Harbor listens on when serving HTTPS nodePort: 30003 # Only needed when notary.enabled is set to true notary: # The service port Notary listens on port: 4443 # The node port Notary listens on nodePort: 30004 externalURL: https://harbor.yourdomian.com:30003 # URL of harbor listing on 30003 port
Ingress
Create Harbor using Ingress to expose the service.
Note
The instructions below assume that the URL
harbor.yourdomain.com
has been configured in the DNS server owningyourdomain.com
as theOAM FLOATING IP Address
of StarlingX.Put the following Ingress overrides in
values.yaml
:expose: type: ingress. # Type should be ingress tls: enabled: true certSource: secret secret: # Certificate Source is secret secretName: "harbor-tls" # Above created secret name notarySecretName: "harbor-tls" # Above created secret name ingress: hosts: core: harbor.yourdomian.com # Harbor Domain name notary: notary.yourdomian.com # Notary Domain name annotations: kubernetes.io/ingress.class: nginx. # Add ingressclass name. It would be "nginx" if you are using default ingress controller. nginx.org/client-max-body-size: "0". # Add this notation for nginx otherwise nginx will reject the image pull & push externalURL: https://harbor.yourdomian.com # URL of harbor
For AIO-DX and Standard setup, add the following
storageClass
andaccessMode
overrides for PVC used forHarbor-Jobservice
andHarbor-Registry
microservice.Note
Set the registry size according to your requirements considering the number and size of images that you will have in this registry.
Example for nodePort:
persistence: enabled: true resourcePolicy: "keep" persistentVolumeClaim: registry: existingClaim: "" storageClass: "cephfs" subPath: "" accessMode: ReadWriteMany size: 100Gi annotations: {} jobservice: jobLog: existingClaim: "" storageClass: "cephfs" subPath: "" accessMode: ReadWriteMany size: 1Gi annotations: {}
Example for Ingress:
persistence: enabled: true resourcePolicy: "keep" persistentVolumeClaim: registry: existingClaim: "" storageClass: "cephfs" subPath: "" accessMode: ReadWriteMany size: 100Gi annotations: {} jobservice: jobLog: existingClaim: "" storageClass: "cephfs" subPath: "" accessMode: ReadWriteMany size: 1Gi annotations: {}
Update the Helm overrides.
~(keystone_admin)]$ system helm-override-update harbor harbor harbor --values values.yaml
Execute Helm overrides.
~(keystone_admin)]$ system helm-override-update harbor harbor harbor --values values.yaml
Apply/Create the Harbor system application.
~(keystone_admin)]$ system application-apply harbor
Configure LDAP Authentication for Harbor Registry¶
Prerequisites
The URL for accessing the Harbor web interface is the
externalURL
set in the Helm override above in the step Configure Helm Overrides for Harbor.The default admin username is ‘admin’, and the password is ‘Harbor12345’.
To configure Harbor to use StarlingX Local LDAP for authentication, follow the instructions in Configure LDAP/Active Directory Authentication with the following values.
For StarlingX local LDAP:
LDAP URL: ldap://controller
LDAP search DN: cn=ldapadmin,dc=cgcs,dc=local
LDAP Search Password: <Password of ldapadmin>
LDAP Base DN: dc=cgcs,dc=local
LDAP UID: cn
You can find <Password of ldapadmin>
in /etc/ldap/slapd.conf.backup
.
Push an Image to a <project> in Harbor¶
Note
Depending on your docker setup, you may be required to run all of the following commands with ‘sudo’.
Docker Login.
docker login <harbor_address> -u <username>
Where
<harbor-address>
is either:for ‘Ingress’ expose: harbor.yourdomian.com
for ‘NodePort’ expose: https:// <oam-floating-ip>:30003 and
<username>
is your actual username
Tag the image.
docker tag redis:latest <harbor_address>/<project>/redis:latest
Push the image.
docker push <harbor_address>/<project>/redis:latest
Pull an Image from Harbor¶
Use command to pull an image:
docker pull <harbor_address>/<project>/ redis:latest
Where <harbor-address>
is either:
for
'Ingress' expose: harbor.yourdomian.com
for
'NodePort' expose: https:// <oam-floating-ip>:30003
Push a Helm Chart as an OCI Object¶
helm package <Chart_Path>
helm push <chart_package> oci://<harbor_address>/<project>/
Pull a Helm Chart¶
helm pull oci://<chart_package/<project>/harbor
Project Management¶
A project in Harbor contains all the repositories (images) of a particular application; e.g. the StarlingX project in Harbor would contain all the images from StarlingX.
For more details on creating and configuring a project see: Create Projects and Project Configuration.
Use Images from a Remote Registry¶
A project in a local Harbor registry can be used as a pull-through cache for a remote registry. For example, a Harbor registry project on a StarlingX Subcloud can be used as a pull-through cache of a Harbor registry project on its StarlingX system controller. This can be achieved using a proxy project which acts as a proxy to the remote registry, or using a replication of the remote registry.
Add a proxy project
Proxy cache allows you to use Harbor to proxy and cache images from a target public or private registry. It does not allow to push the image by the user. Below are the steps to create a proxy project:
Create a registry endpoint. See Creating Replication Endpoints.
Create a new project using the above created registry endpoint. See Configure Proxy Cache.
Add Replication Rule
Setting up pull based replication replicates the images from the remote registry based on a trigger. Trigger can be Manual, scheduled, or event based as shown below.
![../../_images/new-replication-rule.png](../../_images/new-replication-rule.png)
Create a replication endpoint. See Creating Replication Endpoints.
Create a new replication rule with replication mode as
pull based
as describe in Creating a Replication Rule.
Add Users to the Project¶
You can add individual users to a project, for more details see Assign Users to a Project.
LDAP/AD users can be added to the project.
LDAP/AD groups to projects and assign a role to the group.
You can see the various roles and user permission associated with the roles in User Permissions By Role.
Configure Signed Images Support¶
For more details see: Implementing Content Trust.
Select cosign or notary for content trust. Harbor will then only allow verified images to be pulled from the project.
Enable content trust by setting the following environment variables on the machine on which you run the Docker client.
export DOCKER_CONTENT_TRUST=1 export DOCKER_CONTENT_TRUST_SERVER=https://<harbor_address>:<notary port[30004]>
If you push the image for the first time, you will be asked to enter the root key passphrase.
Configure Trivy Scanner Plugin¶
Trivy is installed and configured as a default scanner.
![../../_images/interrogation-services.png](../../_images/interrogation-services.png)
Configure Size of Registry DB¶
Registry DB size can be configured by setting following in
values.yaml
under:persistence: registry: size: 5Gi jobservice: jobLog: size: 1Gi
Set the the value (Default set to 5Gi).
system helm-override-update harbor harbor harbor --values values.yaml
Apply the change:
system application-apply harbor
Enforcement of Image Security Policies Using Portieris¶
Portieris allows to configure trust policies for an individual namespace or cluster-wide and checks the image against a signed image list on a specified notary server to enforce the configured image policies. An Administrator can enforce StarlingX image security policies using the Portieris admission controller.
It is required to pull from a registry using a docker-registry
secret.
Enforcing trust for anonymous image pulls is not supported.
To use portieris, an administrator needs to follow below steps:
Install portieris as specified in Install Portieris.
Configure image policy to allow images from Harbor registry + notary as specified Portieris ClusterImagePolicy and ImagePolicy Configuration. Below example shows the policy allowing image from Harbor registry.
apiVersion: portieris.cloud.ibm.com/v1 kind: ImagePolicy metadata: name: allow-custom namespace: harbor spec: repositories: - name: "<harbor-address>:30003/*" policy: trust: enabled: true trustServer: "https://<notary address>:30004" # Optional, custom trust server for repository
Create a SECRET with a Harbor username and password, to use as an ImagePullSecret in a POD spec.
kubectl create secret docker-registry \ -n harbor harbor-registry-secret \ --docker-server=<harbor-address>:port \ --docker-username=admin \ --docker-password=Test@123
Note
If the pod creation with the above secret fails, the user should try with new secret with
--docker-server
as<harbor-address>
.Pull a signed image from Harbor registry in a pod using
harbor-secret
created above. Please note that image policy and pod should be created in the same namespace.apiVersion: v1 kind: Pod metadata: name: test-pod-public spec: containers: - command: - sleep - '3600' image: <harbor-address>:30003/public-demo/redis:latest imagePullPolicy: Always name: test-pod tolerations: - key: "node-role.kubernetes.io/master" operator: "Exists" effect: "NoSchedule" imagePullSecrets: - name: harbor-registry-secret
Limitation¶
Harbor application cannot be deployed during bootstrap due to the bootstrap deployment dependencies such as early availability of storage class.