Secure Inter Host Pod-to-Pod Network Traffic Using IPsec

Storyboard: https://storyboard.openstack.org/#!/story/2011127

The objective of this feature is to configurably secure specific inter host pod-to-pod network traffic for kubernetes applications on StarlingX platform. The proposed mechanism is by using IPsec.

Problem description

Currently on a StarlingX system, inter host traffics among application pods are not explicitly protected by the platform. Though applications may use other mechanism such as HTTPS to protect them, it means more works for the applications. Also existing applications that don’t support HTTPS would need to be upgraded.

This feature proposes to configurably protect specific application traffic by using IPsec to encrypt inter host pod to pod traffic on cluster host network. This protection is provided by the platform which can be utilized by any applications. It protects applications from both passive (i.e. snooping traffic) and active attacks (i.e. attempting to compromise).

Use Cases

With the introduction of this feature, the platform provides enhanced security capability that the applications can utilize to protect their inter host pod-to-pod traffic.

  • End users: End users can add, update or delete IPsec policies (i.e, specifying what services, protocols and ports to protect) for their applications.

    End users can do these operations by updating polices stored in kubernetes CRs, applying a policy yaml file, or by programmatically calling kubernetes APIs.

  • Deployers: Deployers can create IPsec polices along with their application deployment, by including IPsec polices specifications in the application’s deployment yaml file. The application will be protected right after they are deployed.

Proposed change

  • Secure inter pod-to-pod traffic on cluster host network On StarlingX, an application’s pods on different nodes communicate with each other over cluster host network. So the IPsec will be enforced on cluster host network.

  • Using IPsec tunnel mode To secure inter host pod-to-pod communication, IPSec Tunnel mode is selected. This is because, the internal IP addresses of the pods are ephemeral, and IP packets from pods will be dispensed with the cluster host IP address of the node when cross host via cluster host network. The pods are like hosts in a private subnet behind a gateway. IPsec running in tunnel mode, with the cluster host IP addresses of nodes as the endpoints of the tunnel fits well for the objective.

  • Leverage the platform IPsec Platform IPsec for mgmt. network is implemented by strongSwan. In its core is the IPsec daemon called charon that negotiates, establishes and maintains IPsec SAs and tunnels based on its configuration files /etc/swanctl/swanctl.conf and these in /etc/swanctl/conf.d.

    Pod-to-pod IPsec will leverage platform IPsec by including pod-to-pod IPsec policies in charon’s configuration files. It will also reuse the platform IPsec security materials (such as certificates, private keys etc) for IKE negotiation, SA establishment, reauthentication, rekeying etc.

  • Policy management in a cloud native model The IPsec policies for pod-to-pod IPsec will be managed in a cloud native model by Kubernetes custom operators, instead of by platform services such as sysinv.

    Operators are software extensions to Kubernetes that make use of custom resources to manage applications and their components.

    For pod-to-pod IPsec, policies are defined in custom CRs. An operator (namely policy manager) running on controllers will watch IPsec policy CRs for addition/updating/deletion, query apiserver for relevant information, and update internal IPsec configmaps containing the IPsec configurations for each of the hosts in the cluster.

    A second operator (namely policy agent) running on all nodes will watch the IPsec configmaps. Changes to the IPsec configmap for a node will trigger the policy agent running on that node to reconcile, updating the strongSwan’s configuration files on the host, and request IPsec daemon to reload the updated configurations to make them effective.

    The operator can be built by using operator builder SDK. It will be packaged as an StarlingX optional platform application.

Alternatives

  • An application can protect its pod-to-pod traffic by TLS (HTTPS at REST for example). But it has to be implemented by the application itself, which makes the application more complicated. The proposed IPsec solution is transparent to the application, leaving the application focus on its business logic.

Data model impact

  • New kubernetes CRD for IPsec policies New CRD (Custom Resource Definition) for pod-to-pod IPsec policies will be introduced. User defined IPsec policies will be stored in CRs complying to the CRD.

    A IPsec policy is defined in the CR by a name, a service in a namespace and its protocol/ports to be protected.

  • New configmaps for the pod-to-pod IPsec configurations Pod-to-pod IPsec configurations for the nodes in the cluster are generated by policy manager operator and stored in configmaps. The policy agent running on each node generates host strongSwan configuration file from the configmap for that node.

  • New strongSwan IPsec configuration file A new strongSwan IPsec configuration file is generated for pod-to-pod IPsec on host in /etc/swanctl/conf.d directory. It is then included in IPsec config by the clause include conf.d/*.conf in /etc/swanctl/swanctl.conf.

REST API impact

None

Security impact

  • This feature provides a machenism that applications can use to secure their inter host pod-to-pod traffic.

  • New rules to allow ESP packets on cluster host network need to be added into kubernetes’ globalnetworkpolicy: controller-cluster-host-if-gnp, worker-cluster-host-if-gnp.

Other end user impact

  • The feature will be implemented as a StarlingX optional app, the app needs to be applied in order for it to work.

  • If no pod-to-pod IPsec policies defined, the feature is equivalent to be disabled.

Performance Impact

  • A preliminary test results confirm that the TX overhead for encryption is taxed to the sender on the application cores and the RX overhead for decryption is taxed to the platform cores.

Other deployer impact

Please see the “Use Cases” section.

Developer impact

None

Upgrade impact

  • App upgrade The feature will be implemented as a StarlingX optional platform app. Upgrade of the app will be supported the same way as other apps.

  • System upgrade The handling of the app for system upgrade will be the same as existing apps.

Implementation

Assignee(s)

Primary assignee:

Other Contributors:

Repos Impacted

  • starlingx/app-ipsec-policy-operator (new)

Work Items

The following work items are expected, with the understanding that the storyboard will be updated as more work items are found to be necessary.

Spikes and Design

  • Basic testing of IPsec tunnel mode to encrypt inter host pod-to-pod traffic.

  • Understanding and experimenting of Kubernetes Operator pattern.

  • Review of the proposed design.

  • Preliminary evaluation of performance impact on cpu utilization.

Development Work Items

  • Introduce pod-to-pod IPsec policy CRD.

  • Introduce IPsec policy manager operator as Deployment.

  • Enhance IPsec policy manager operator to monitor policy CRs to generate or update IPsec configmaps.

  • Enhance IPsec policy manager operator to monitor services’ endpoints change and update IPsec configmaps.

  • Introduce IPsec policy agent operator as Daemonset.

  • Enhance IPsec policy agent to monitor IPsec configmap and generate or update host strongSwan IPsec configuration file.

  • Create FluxCD manifests to wrap the manager and agent into a StarlingX application.

  • Installation via system application.

Customer Documentation

  • Publish the usage guide for what functionality is available and how to make use of it.

  • Sample code showing how to make use of the functionality.

Dependencies

Testing

System configuration

The tests will be conducted in the following system configurations:

  • AIO-SX

  • AIO-DX

  • Standard

Test Scenarios

  • Functional tests of the IPsec policy manager and agent.

  • Security tests to ensure inter host pod-to-pod traffic of the services specified in policies are protected by IPsec.

  • Performance testing to identify and address any performance impacts.

  • Backup and restore tests.

  • Application upgrade tests.

  • System upgrade tests.

Documentation Impact

The end-user documentation must be created, adding a guide to IPsec Policy Operator Application deployment, policy definition and customizations.

References

  1. strongSwan Documentation

  2. Operator Framework

  3. Kubebuilder

History

Revisions

Release Name

Description

stx-11.0

Introduced