Tungsten Fabric solution guide - Kubernetes integration

Posted by dmrn on Tue, 23 Jun 2020 11:09:18 +0200

Author: Tony Liu translator: TF Compilation Group

1 integration of kubernetes and TF

In the integration scheme, there are two connections between Kubernetes and Tungsten Fabric.

Contract Kube manager and Kube API server
Contrail CNI

1.1 contrail-kube-manager

This service connects to the Kube API server to receive updates. It then connects to the Tungsten Fabric configuration API server to create the necessary configuration (VM, VMI / port, IP, etc.) to connect the container to the overlay layer. It also sends updates to the Kube API server.

1.2 Contrail CNI

Kubelet on each node/minion runs with CNI parameters. When you start the container, kubelet calls CNI to set up the network. Contract CNI to vRouter agent REST API:

1) Get the necessary configuration
2) Insert the container network interface into the vRouter

1.3 Gateway

Tungsten Fabric uses gateway to connect overlay and underlay networks to provide external access. We need to use gateway to support Kubernetes' exposed services and ingress capabilities.

You must create a floating IP pool in Tungsten Fabric.

In / etc / contailctl/ kubemanager.conf The FQ name of the FIP pool configured in to provision.

[KUBERNETES_VNC]public_fip_pool = {'domain': 'default-domain', 'project': 'default', 'network': 'public', 'name': 'public-fip-pool'}

In the container condir Kube manager, the floating IP pool is in / etc / container / container- kubernetes.conf For configuration.

[VNC]public_fip_pool = {'domain': 'default-domain', 'project': 'default', 'network': 'public', 'name': 'public-fip-pool'}

When a service is exposed or an ingress is created in Kubernetes, a FIP is assigned from the pool as the external IP.

2 Namespace

When integrated with Tungsten Fabric, the Kubernetes namespace can be mapped to a project/tenant or virtual network.

2.1 single tenant

If / etc / container / container is set- Kubernetes.conf [Kubernetes]. Cluster in_ Project, which is a single tenant, Kubernetes namespace will map to the virtual network in Tungsten Fabric. All non isolated namespaces are mapped to the default virtual network "cluster network". Each isolation namespace is mapped to a separate virtual network - vn.

Here is an example, which shows that in / etc / contailctl/ kubemanager.conf Set [kuternets]. Cluster in_ Project to enable single tenant scenarios.

[KUBERNETES]cluster_project = {'domain': 'default-domain', 'project': 'kubernetes'}

The following was created by the condir Kube manager during initialization.

Flat IPAM <cluster_ Project >: pod IPAM with subnet
IPAM <cluster_project>: service-ipam
Security groups k8s default default default and k8s default SG of virtual network "cluster network"
Virtual network < cluster_ Project >: cluster network with pod IPAM and service IPAM

(see Appendix A.1)

2.1.1 non isolated namespace

Create a non isolated namespace.

apiVersion: v1kind: Namespacemetadata: name: "dev-unisolated"

When Kubernetes creates a non isolated namespace, Tungsten Fabric creates two SGS, k8s default -- SG and k8s default -- SG. A virtual network is not created here. All containers in non isolated NS will be on cluster network.

Start a Pod in a non isolated namespace.

apiVersion: v1kind: Podmetadata:  name: nginx-1spec:  containers:  - name: nginx    image: docker.io/nginx    imagePullPolicy: IfNotPresentkubectl create -f nginx-1.yaml -n <namespace>kubectl get pods -n <namespace>

When you start Pod in a non isolated namespace, Tungsten Fabric (contract Kube manager) does the following.

Pod s in different non isolated namespaces can connect to each other because they are on the same virtual network in Tungsten Fabric.

2.1.2 isolated namespace

Create an isolated namespace.

apiVersion: v1kind: Namespacemetadata: name: "dev-isolated" annotations: {   "opencontrail.org/isolation" : "true" }

When you create an isolated namespace in Kubernetes, Tungsten Fabric creates the following.

Virtual network < cluster_ project>:-vn
Security groups k8s default -- default and k8s default -- SG

When you start Pod in the isolation namespace, Tungsten Fabric does the following.

Pods in different isolation namespaces cannot connect to each other because the ports are on different virtual networks.

2.2 multi tenant

If / etc / container / container is not set- Kubernetes.conf [Kubernetes]. Cluster in_ Project, which is multi tenant, Kubernetes namespace will map to tenant/project in Tungsten Fabric. The Pod in the non isolated namespace starts on the default virtual network 'cluster network'. Each isolation namespace is mapped to a separate virtual network - vn.

During initialization, the contract Kube manager creates the following:

Provide a project / tenant for each existing Kubenetes namespace, such as default, Kube public, and Kube system.
Flat IPAM default domain: default:pod-ipam
IPAM default domain: default:service-ipam
Security groups for each namespace: k8s default -- SG and k8s default -- SG
Default domain name of virtual network: with pod IPAM default:cluster-network And service IPAM

2.2.1 non isolated namespace

Create a non isolated namespace.

apiVersion: v1kind: Namespacemetadata: name: "dev-unisolated"

The contract Kube manager will create the following.

When Pod is started in a non isolated namespace, conventil Kube manager creates the port:

Pod s in different non isolated namespaces can connect to each other because they are on the same virtual network in Tungsten Fabric.

2.2.2 isolated namespace

Create an isolated namespace.

apiVersion: v1kind: Namespacemetadata: name: "dev-isolated" annotations: {   "opencontrail.org/isolation" : "true" }

The contract cube manager will create the following.

Project default domain:
Virtual network default domain:: - VN and default- domain:default : pod IPAM Association.
Security groups default domain:: k8s default -- default and default domain:: k8s default -- SG

When Pod is started in a non isolated namespace, conventil Kube manager creates the port.

In project default domain:
On virtual network default domain:: - VN
From IPAM default- domain:default : pod IPAM get address
With security groups default domain:: k8s default -- default and default domain:: k8s default -- SG

Pods in different isolation namespaces cannot connect to each other because the ports are on different virtual networks.

2.3 custom namespace

Create a custom namespace.

apiVersion: v1kind: Namespacemetadata: name: "dev-customized" annotations: {   "opencontrail.org/network": '{"domain": "default-domain", "project": "demo", "name": "red"}' }

When you start Pod in a custom namespace, the contract Kube manager creates the port.

In project default-domain:default in
Map to custom namespace on virtual network
Get the address from IPAM associated with the virtual network
Security group?

2.4 specify Pod on virtual network

Start Pod on the specified virtual network.

When Pod is started on the specified virtual network, the conutil Kube manager creates the port.

Map to the specified or default namespace in the project
On the specified virtual network
Get address from IPAM associated with a specific virtual network
Security group?

2.5 Kubernetes network strategy

Kubernetes network policy will run as usual, which is implemented by the security group in Tungsten Fabric. This version will be released with 4.0.1.

2.6 POD SNAT

Tungsten Fabric supports this function. You can configure a router (configuration object) in Tungsten Fabric to become the external gateway of the virtual network that starts the container. This is the same as an external gateway that supports OpenStack.

3 Service

Kubernetes service supports ClusterIP, NodePort, LoadBalancer and ExternalName. It also supports the use of ExternalIP to specify IP. Tungsten Fabric supports ClusterIP, LoadBalancer, and ExternalIP.

When a service is created in Kubernetes, a load balancer is created in Tungsten Fabric. The provider of load balancer is "native", and ECMP load balancing is implemented by vRouter. Floating IP is created as VIP.

Create an application with multiple instances.

apiVersion: v1kind: ReplicationControllermetadata:  name: web-qaspec:  replicas: 2  selector:    app: web-qa  template:    metadata:      name: web-qa      labels:        app: web-qa    spec:      containers:      - name: web        image: docker.io/nginx        imagePullPolicy: IfNotPresent

3.1 ClusterIP

Create services in front of these applications. The default service type is ClusterIP.

kind: ServiceapiVersion: v1metadata:  name: web-qaspec:  selector:    app: web-qa  ports:    - protocol: TCP      port: 80      targetPort: 80

When the service is created, the conventil Kube manager performs the following actions.

When the LB is created, the native LB driver does the following.

Set up port mapping in FIP.
Add VMI of all members to FIP.

When the service type is ClusterIP, the service can only be accessed within the cluster. FIP is allocated from the service FIP pool in the cluster network and mapped to all Pod addresses. When accessing the service address in the cluster, vRouter will balance the traffic between the pods.

3.2 Loadbalancer

Create a service of type LoadBalancer.

kind: ServiceapiVersion: v1metadata:  name: web-qaspec:  selector:    app: web-qa  ports:    - protocol: TCP      port: 80      targetPort: 80  type: LoadBalancer

For the service type LoadBalancer, the service is exposed externally. FIPS are allocated from the service FIP pool for access within the cluster, and FIPS are allocated from the public FIP pool to map to all POD addresses. The FIP will be notified to the gateway, which will perform ECMP load balancing between pods.

Appendix A single tenant

A.1 IPAM

<cluster_project>:pod-ipam

{    "fq_name": [        "default-domain",        "kubernetes",        "pod-ipam"    ],    "uuid": "c9641741-c785-456e-845b-a14a253c3572",    "ipam_subnet_method": "flat-subnet",    "parent_type": "project",    "perms2": {        "owner": "None",        "owner_access": 7,        "global_access": 0,        "share": []    },    "ipam_subnets": {        "subnets": [            {                "subnet": {                    "ip_prefix": "10.32.0.0",                    "ip_prefix_len": 12                },                "dns_server_address": "10.47.255.253",                "enable_dhcp": true,                "created": null,                "default_gateway": "10.47.255.254",                "dns_nameservers": [],                "dhcp_option_list": null,                "subnet_uuid": null,                "alloc_unit": 1,                "last_modified": null,                "host_routes": null,                "addr_from_start": null,                "subnet_name": null,                "allocation_pools": []            }        ]    },    "id_perms": {        "enable": true,        "description": null,        "creator": null,        "created": "2017-12-27T18:45:33.957901",        "uuid": {            "uuid_mslong": 14511749470582293870,            "uuid_lslong": 9537393975711511922        },        "user_visible": true,        "last_modified": "2017-12-27T18:45:33.957901",        "permissions": {            "owner": "cloud-admin",            "owner_access": 7,            "other_access": 7,            "group": "cloud-admin-group",            "group_access": 7        }    },    "display_name": "pod-ipam"}

<cluster_project>:service-ipam

{    "fq_name": [        "default-domain",        "kubernetes",        "service-ipam"    ],    "uuid": "526f554a-0bf4-47c6-a8e4-768a3f98cef4",    "parent_type": "project",    "perms2": {        "owner": "None",        "owner_access": 7,        "global_access": 0,        "share": []    },    "id_perms": {        "enable": true,        "description": null,        "creator": null,        "created": "2017-12-27T18:45:34.000690",        "uuid": {            "uuid_mslong": 5940060210041472966,            "uuid_lslong": 12169982429206466292        },        "user_visible": true,        "last_modified": "2017-12-27T18:45:34.000690",        "permissions": {            "owner": "cloud-admin",            "owner_access": 7,            "other_access": 7,            "group": "cloud-admin-group",            "group_access": 7        }    },    "display_name": "service-ipam"}

A.2 security group

k8s-default-dev-share-default

{    "fq_name": [        "default-domain",        "kubernetes",        "k8s-default-dev-share-default"    ],    "uuid": "ad29de07-5ef6-4f55-86bb-52c44827c09d",    "parent_type": "project",    "perms2": {        "owner": "46c31b9b-d21c-4c27-9445-6c94db948b6d",        "owner_access": 7,        "global_access": 0,        "share": []    },    "security_group_id": 8000010,    "id_perms": {        "enable": true,        "description": "Default security group",        "creator": null,        "created": "2018-01-12T09:02:15.110429",        "uuid": {            "uuid_mslong": 12477748365846007637,            "uuid_lslong": 9708444424704868509        },        "user_visible": true,        "last_modified": "2018-01-12T15:45:08.899388",        "permissions": {            "owner": "cloud-admin",            "owner_access": 7,            "other_access": 7,            "group": "cloud-admin-group",            "group_access": 7        }    },    "security_group_entries": {        "policy_rule": [            {                "direction": ">",                "protocol": "any",                "dst_addresses": [                    {                        "security_group": "local",                        "subnet": null,                        "virtual_network": null,                        "subnet_list": [],                        "network_policy": null                    }                ],                "action_list": null,                "created": null,                "rule_uuid": "dc13bb48-e2a7-4c59-a0b8-740ecfcb9a2c",                "dst_ports": [                    {                        "end_port": 65535,                        "start_port": 0                    }                ],                "application": [],                "last_modified": null,                "ethertype": "IPv4",                "src_addresses": [                    {                        "security_group": null,                        "subnet": {                            "ip_prefix": "0.0.0.0",                            "ip_prefix_len": 0                        },                        "virtual_network": null,                        "subnet_list": [],                        "network_policy": null                    }                ],                "rule_sequence": null,                "src_ports": [                    {                        "end_port": 65535,                        "start_port": 0                    }                ]            },            {                "direction": ">",                "protocol": "any",                "dst_addresses": [                    {                        "security_group": "local",                        "subnet": null,                        "virtual_network": null,                        "subnet_list": [],                        "network_policy": null                    }                ],                "action_list": null,                "created": null,                "rule_uuid": "a84e2d98-2b8f-45ba-aa75-88494da73b11",                "dst_ports": [                    {                        "end_port": 65535,                        "start_port": 0                    }                ],                "application": [],                "last_modified": null,                "ethertype": "IPv6",                "src_addresses": [                    {                        "security_group": null,                        "subnet": {                            "ip_prefix": "::",                            "ip_prefix_len": 0                        },                        "virtual_network": null,                        "subnet_list": [],                        "network_policy": null                    }                ],                "rule_sequence": null,                "src_ports": [                    {                        "end_port": 65535,                        "start_port": 0                    }                ]            },            {                "direction": ">",                "protocol": "any",                "dst_addresses": [                    {                        "security_group": null,                        "subnet": {                            "ip_prefix": "0.0.0.0",                            "ip_prefix_len": 0                        },                        "virtual_network": null,                        "subnet_list": [],                        "network_policy": null                    }                ],                "action_list": null,                "created": null,                "rule_uuid": "b7752ec1-6037-4c7f-97a9-291893fbed64",                "dst_ports": [                    {                        "end_port": 65535,                        "start_port": 0                    }                ],                "application": [],                "last_modified": null,                "ethertype": "IPv4",                "src_addresses": [                    {                        "security_group": "local",                        "subnet": null,                        "virtual_network": null,                        "subnet_list": [],                        "network_policy": null                    }                ],                "rule_sequence": null,                "src_ports": [                    {                        "end_port": 65535,                        "start_port": 0                    }                ]            },            {                "direction": ">",                "protocol": "any",                "dst_addresses": [                    {                        "security_group": null,                        "subnet": {                            "ip_prefix": "::",                            "ip_prefix_len": 0                        },                        "virtual_network": null,                        "subnet_list": [],                        "network_policy": null                    }                ],                "action_list": null,                "created": null,                "rule_uuid": "ea5cd2a8-2d47-47c4-a9ab-390de2317246",                "dst_ports": [                    {                        "end_port": 65535,                        "start_port": 0                    }                ],                "application": [],                "last_modified": null,                "ethertype": "IPv6",                "src_addresses": [                    {                        "security_group": "local",                        "subnet": null,                        "virtual_network": null,                        "subnet_list": [],                        "network_policy": null                    }                ],                "rule_sequence": null,                "src_ports": [                    {                        "end_port": 65535,                        "start_port": 0                    }                ]            }        ]    },    "annotations": {        "key_value_pair": [            {                "key": "namespace",                "value": "dev-share"            },            {                "key": "cluster",                "value": "k8s-default"            },            {                "key": "kind",                "value": "Namespace"            },            {                "key": "project",                "value": "kubernetes"            },            {                "key": "name",                "value": "k8s-default-dev-share-default"            },            {                "key": "owner",                "value": "k8s"            }        ]    },    "display_name": "k8s-default-dev-share-default"}

k8s-default-dev-share-sg

{    "fq_name": [        "default-domain",        "kubernetes",        "k8s-default-dev-share-sg"    ],    "uuid": "791f1c7e-a66e-4c47-ba05-409f00ee2c8e",    "parent_type": "project",    "perms2": {        "owner": "46c31b9b-d21c-4c27-9445-6c94db948b6d",        "owner_access": 7,        "global_access": 0,        "share": []    },    "security_group_id": 8000017,    "id_perms": {        "enable": true,        "description": "Namespace security group",        "creator": null,        "created": "2018-01-12T09:02:15.236401",        "uuid": {            "uuid_mslong": 8727725933151013959,            "uuid_lslong": 13404190917597736078        },        "user_visible": true,        "last_modified": "2018-01-12T09:02:15.275407",        "permissions": {            "owner": "cloud-admin",            "owner_access": 7,            "other_access": 7,            "group": "cloud-admin-group",            "group_access": 7        }    },    "display_name": "k8s-default-dev-share-sg",    "annotations": {        "key_value_pair": [            {                "key": "namespace",                "value": "dev-share"            },            {                "key": "cluster",                "value": "k8s-default"            },            {                "key": "kind",                "value": "Namespace"            },            {                "key": "project",                "value": "kubernetes"            },            {                "key": "name",                "value": "k8s-default-dev-share-sg"            },            {                "key": "owner",                "value": "k8s"            }        ]    }}

A.3 virtual network

<cluster_project>:cluster-network

{    "virtual_network_properties": {        "forwarding_mode": "l3",        "allow_transit": null,        "network_id": null,        "mirror_destination": false,        "vxlan_network_identifier": null,        "rpf": null    },    "fq_name": [        "default-domain",        "kubernetes",        "cluster-network"    ],    "uuid": "1b9f7f74-17f0-493a-9108-729f91b43598",    "address_allocation_mode": "user-defined-subnet-only",    "mac_aging_time": 300,    "parent_type": "project",    "perms2": {        "owner": "None",        "owner_access": 7,        "global_access": 0,        "share": []    },    "display_name": "cluster-network",    "pbb_evpn_enable": false,    "mac_learning_enabled": false,    "id_perms": {        "enable": true,        "description": null,        "creator": null,        "created": "2017-12-27T18:45:34.062865",        "uuid": {            "uuid_mslong": 1990449696915605818,            "uuid_lslong": 10450728964983109016        },        "user_visible": true,        "last_modified": "2017-12-29T10:29:20.685414",        "permissions": {            "owner": "cloud-admin",            "owner_access": 7,            "other_access": 7,            "group": "cloud-admin-group",            "group_access": 7        }    },    "flood_unknown_unicast": false,    "layer2_control_word": false,    "port_security_enabled": true,    "network_ipam_refs": [        {            "to": [                "default-domain",                "kubernetes",                "service-ipam"            ],            "href": "http://127.0.0.1:8082/network-ipam/526f554a-0bf4-47c6-a8e4-768a3f98cef4",            "attr": {                "ipam_subnets": [                    {                        "subnet": {                            "ip_prefix": "10.167.0.0",                            "ip_prefix_len": 16                        },                        "dns_server_address": "10.167.255.253",                        "enable_dhcp": true,                        "created": null,                        "default_gateway": "10.167.255.254",                        "dns_nameservers": [],                        "dhcp_option_list": null,                        "subnet_uuid": "10a8de65-9de8-419b-b14c-180bf2ab3dc9",                        "alloc_unit": 1,                        "last_modified": null,                        "host_routes": null,                        "addr_from_start": null,                        "subnet_name": null,                        "allocation_pools": []                    }                ],                "host_routes": null            },            "uuid": "526f554a-0bf4-47c6-a8e4-768a3f98cef4"        },        {            "to": [                "default-domain",                "kubernetes",                "pod-ipam"            ],            "href": "http://127.0.0.1:8082/network-ipam/c9641741-c785-456e-845b-a14a253c3572",            "attr": {                "ipam_subnets": [                    {                        "subnet": null,                        "dns_server_address": null,                        "enable_dhcp": true,                        "created": null,                        "default_gateway": null,                        "dns_nameservers": [],                        "dhcp_option_list": null,                        "subnet_uuid": "d2b090ce-cbcc-4b00-b50a-cc1ed5468b00",                        "alloc_unit": 1,                        "last_modified": null,                        "host_routes": null,                        "addr_from_start": null,                        "subnet_name": null,                        "allocation_pools": []                    }                ],                "host_routes": null            },            "uuid": "c9641741-c785-456e-845b-a14a253c3572"        }    ],    "pbb_etree_enable": false,    "virtual_network_network_id": 5}

<cluster_project>:dev-vn

{    "virtual_network_properties": {        "forwarding_mode": "l3",        "allow_transit": null,        "network_id": null,        "mirror_destination": false,        "vxlan_network_identifier": null,        "rpf": null    },    "fq_name": [        "default-domain",        "kubernetes",        "dev-vn"    ],    "uuid": "ce01826b-e3e6-407f-8798-80612018e89c",    "address_allocation_mode": "flat-subnet-only",    "mac_aging_time": 300,    "parent_type": "project",    "perms2": {        "owner": "None",        "owner_access": 7,        "global_access": 0,        "share": []    },    "display_name": "dev-vn",    "pbb_evpn_enable": false,    "mac_learning_enabled": false,    "id_perms": {        "enable": true,        "description": null,        "creator": null,        "created": "2018-01-09T11:40:06.196335",        "uuid": {            "uuid_mslong": 14844289246686494847,            "uuid_lslong": 9770700546218977436        },        "user_visible": true,        "last_modified": "2018-01-09T12:18:55.796399",        "permissions": {            "owner": "cloud-admin",            "owner_access": 7,            "other_access": 7,            "group": "cloud-admin-group",            "group_access": 7        }    },    "flood_unknown_unicast": false,    "layer2_control_word": false,    "port_security_enabled": true,    "network_ipam_refs": [        {            "to": [                "default-domain",                "kubernetes",                "pod-ipam"            ],            "href": "http://127.0.0.1:8082/network-ipam/c9641741-c785-456e-845b-a14a253c3572",            "attr": {                "ipam_subnets": [                    {                        "subnet": null,                        "dns_server_address": null,                        "enable_dhcp": true,                        "created": null,                        "default_gateway": null,                        "dns_nameservers": [],                        "dhcp_option_list": null,                        "subnet_uuid": "48ed8235-efcd-44a1-998c-659e4f5840f4",                        "alloc_unit": 1,                        "last_modified": null,                        "host_routes": null,                        "addr_from_start": null,                        "subnet_name": null,                        "allocation_pools": []                    }                ],                "host_routes": null            },            "uuid": "c9641741-c785-456e-845b-a14a253c3572"        }    ],    "annotations": {        "key_value_pair": [            {                "key": "cluster",                "value": "k8s-default"            },            {                "key": "kind",                "value": "Namespace"            },            {                "key": "namespace",                "value": "dev"            },            {                "key": "isolated",                "value": "True"            },            {                "key": "project",                "value": "kubernetes"            },            {                "key": "name",                "value": "dev"            },            {                "key": "owner",                "value": "k8s"            }        ]    },    "pbb_etree_enable": false,    "virtual_network_network_id": 11}

A.4 virtual machine interface and instance IP

Non isolated port

{    "fq_name": [        "default-domain",        "kubernetes",        "dev-web-k528t__5a1fc03e-f7ab-11e7-8f66-52540065dced"    ],    "virtual_machine_interface_mac_addresses": {        "mac_address": [            "02:5a:1f:c0:3e:f7"        ]    },    "display_name": "dev-share__dev-web-k528t",    "security_group_refs": [        {            "to": [                "default-domain",                "kubernetes",                "k8s-default-dev-share-default"            ],            "href": "http://127.0.0.1:8082/security-group/ad29de07-5ef6-4f55-86bb-52c44827c09d",            "attr": null,            "uuid": "ad29de07-5ef6-4f55-86bb-52c44827c09d"        },        {            "to": [                "default-domain",                "kubernetes",                "k8s-default-dev-share-sg"            ],            "href": "http://127.0.0.1:8082/security-group/791f1c7e-a66e-4c47-ba05-409f00ee2c8e",            "attr": null,            "uuid": "791f1c7e-a66e-4c47-ba05-409f00ee2c8e"        }    ],    "routing_instance_refs": [        {            "to": [                "default-domain",                "kubernetes",                "cluster-network",                "cluster-network"            ],            "href": "http://127.0.0.1:8082/routing-instance/5ed7608a-28bb-4735-a8d8-2e9132b03d62",            "attr": {                "direction": "both",                "protocol": null,                "ipv6_service_chain_address": null,                "dst_mac": null,                "mpls_label": null,                "vlan_tag": null,                "src_mac": null,                "service_chain_address": null            },            "uuid": "5ed7608a-28bb-4735-a8d8-2e9132b03d62"        }    ],    "virtual_machine_interface_disable_policy": false,    "parent_type": "project",    "perms2": {        "owner": "None",        "owner_access": 7,        "global_access": 0,        "share": []    },    "virtual_network_refs": [        {            "to": [                "default-domain",                "kubernetes",                "cluster-network"            ],            "href": "http://127.0.0.1:8082/virtual-network/1b9f7f74-17f0-493a-9108-729f91b43598",            "attr": null,            "uuid": "1b9f7f74-17f0-493a-9108-729f91b43598"        }    ],    "id_perms": {        "enable": true,        "description": null,        "creator": null,        "created": "2018-01-12T15:14:58.189964",        "uuid": {            "uuid_mslong": 6494120564367233511,            "uuid_lslong": 10333036915785587949        },        "user_visible": true,        "last_modified": "2018-01-12T15:14:58.253769",        "permissions": {            "owner": "cloud-admin",            "owner_access": 7,            "other_access": 7,            "group": "cloud-admin-group",            "group_access": 7        }    },    "virtual_machine_refs": [        {            "to": [                "dev-web-k528t__708154c6-f7ab-11e7-a9df-98f2b3a36be0"            ],            "href": "http://127.0.0.1:8082/virtual-machine/708154c6-f7ab-11e7-a9df-98f2b3a36be0",            "attr": null,            "uuid": "708154c6-f7ab-11e7-a9df-98f2b3a36be0"        }    ],    "vlan_tag_based_bridge_domain": false,    "port_security_enabled": true,    "annotations": {        "key_value_pair": [            {                "key": "cluster",                "value": "k8s-default"            },            {                "key": "kind",                "value": "Pod"            },            {                "key": "namespace",                "value": "dev-share"            },            {                "key": "project",                "value": "kubernetes"            },            {                "key": "name",                "value": "dev-web-k528t"            },            {                "key": "owner",                "value": "k8s"            }        ]    },    "uuid": "5a1fc03e-f7ab-11e7-8f66-52540065dced"}

IP instance

{    "fq_name": [        "dev-web-k528t__5a2f9cde-f7ab-11e7-8f66-52540065dced"    ],    "uuid": "5a2f9cde-f7ab-11e7-8f66-52540065dced",    "service_health_check_ip": false,    "instance_ip_address": "10.47.255.251",    "perms2": {        "owner": "cloud-admin",        "owner_access": 7,        "global_access": 0,        "share": []    },    "annotations": {        "key_value_pair": [            {                "key": "cluster",                "value": "k8s-default"            },            {                "key": "kind",                "value": "Pod"            },            {                "key": "namespace",                "value": "dev-share"            },            {                "key": "project",                "value": "kubernetes"            },            {                "key": "name",                "value": "dev-web-k528t"            },            {                "key": "owner",                "value": "k8s"            }        ]    },    "subnet_uuid": "d2b090ce-cbcc-4b00-b50a-cc1ed5468b00",    "id_perms": {        "enable": true,        "description": null,        "creator": null,        "created": "2018-01-12T15:14:58.323069",        "uuid": {            "uuid_mslong": 6498585268770771431,            "uuid_lslong": 10333036915785587949        },        "user_visible": true,        "last_modified": "2018-01-12T15:14:58.363792",        "permissions": {            "owner": "cloud-admin",            "owner_access": 7,            "other_access": 7,            "group": "cloud-admin-group",            "group_access": 7        }    },    "virtual_machine_interface_refs": [        {            "to": [                "default-domain",                "kubernetes",                "dev-web-k528t__5a1fc03e-f7ab-11e7-8f66-52540065dced"            ],            "href": "http://127.0.0.1:8082/virtual-machine-interface/5a1fc03e-f7ab-11e7-8f66-52540065dced",            "attr": null,            "uuid": "5a1fc03e-f7ab-11e7-8f66-52540065dced"        }    ],    "service_instance_ip": false,    "instance_ip_local_ip": false,    "virtual_network_refs": [        {            "to": [                "default-domain",                "kubernetes",                "cluster-network"            ],            "href": "http://127.0.0.1:8082/virtual-network/1b9f7f74-17f0-493a-9108-729f91b43598",            "attr": null,            "uuid": "1b9f7f74-17f0-493a-9108-729f91b43598"        }    ],    "instance_ip_secondary": false,    "display_name": "dev-share__dev-web-k528t"}

Isolated port

{    "fq_name": [        "default-domain",        "kubernetes",        "dev-client__c64b3b12-f7b5-11e7-8f66-52540065dced"    ],    "virtual_machine_interface_mac_addresses": {        "mac_address": [            "02:c6:4b:3b:12:f7"        ]    },    "display_name": "dev__dev-client",    "security_group_refs": [        {            "to": [                "default-domain",                "kubernetes",                "k8s-default-dev-sg"            ],            "href": "http://127.0.0.1:8082/security-group/579019d5-038e-4901-b6ab-ed146022dd70",            "attr": null,            "uuid": "579019d5-038e-4901-b6ab-ed146022dd70"        },        {            "to": [                "default-domain",                "kubernetes",                "k8s-default-dev-default"            ],            "href": "http://127.0.0.1:8082/security-group/e43caf6e-6b35-40c3-b336-83c155078efe",            "attr": null,            "uuid": "e43caf6e-6b35-40c3-b336-83c155078efe"        }    ],    "routing_instance_refs": [        {            "to": [                "default-domain",                "kubernetes",                "dev-vn",                "dev-vn"            ],            "href": "http://127.0.0.1:8082/routing-instance/45173786-a1b4-4c75-8ef0-590de67d2d05",            "attr": {                "direction": "both",                "protocol": null,                "ipv6_service_chain_address": null,                "dst_mac": null,                "mpls_label": null,                "vlan_tag": null,                "src_mac": null,                "service_chain_address": null            },            "uuid": "45173786-a1b4-4c75-8ef0-590de67d2d05"        }    ],    "virtual_machine_interface_disable_policy": false,    "parent_type": "project",    "perms2": {        "owner": "None",        "owner_access": 7,        "global_access": 0,        "share": []    },    "virtual_network_refs": [        {            "to": [                "default-domain",                "kubernetes",                "dev-vn"            ],            "href": "http://127.0.0.1:8082/virtual-network/ce01826b-e3e6-407f-8798-80612018e89c",            "attr": null,            "uuid": "ce01826b-e3e6-407f-8798-80612018e89c"        }    ],    "id_perms": {        "enable": true,        "description": null,        "creator": null,        "created": "2018-01-12T16:29:34.640295",        "uuid": {            "uuid_mslong": 14288579195414319591,            "uuid_lslong": 10333036915785587949        },        "user_visible": true,        "last_modified": "2018-01-12T16:29:34.708511",        "permissions": {            "owner": "cloud-admin",            "owner_access": 7,            "other_access": 7,            "group": "cloud-admin-group",            "group_access": 7        }    },    "virtual_machine_refs": [        {            "to": [                "dev-client__c64878a1-f7b5-11e7-9dbb-98f2b3a33b90"            ],            "href": "http://127.0.0.1:8082/virtual-machine/c64878a1-f7b5-11e7-9dbb-98f2b3a33b90",            "attr": null,            "uuid": "c64878a1-f7b5-11e7-9dbb-98f2b3a33b90"        }    ],    "vlan_tag_based_bridge_domain": false,    "port_security_enabled": true,    "annotations": {        "key_value_pair": [            {                "key": "cluster",                "value": "k8s-default"            },            {                "key": "kind",                "value": "Pod"            },            {                "key": "namespace",                "value": "dev"            },            {                "key": "project",                "value": "kubernetes"            },            {                "key": "name",                "value": "dev-client"            },            {                "key": "owner",                "value": "k8s"            }        ]    },    "uuid": "c64b3b12-f7b5-11e7-8f66-52540065dced"}{    "fq_name": [        "dev-client__c65c2a12-f7b5-11e7-8f66-52540065dced"    ],    "uuid": "c65c2a12-f7b5-11e7-8f66-52540065dced",    "service_health_check_ip": false,    "instance_ip_address": "10.47.255.250",    "perms2": {        "owner": "cloud-admin",        "owner_access": 7,        "global_access": 0,        "share": []    },    "annotations": {        "key_value_pair": [            {                "key": "cluster",                "value": "k8s-default"            },            {                "key": "kind",                "value": "Pod"            },            {                "key": "namespace",                "value": "dev"            },            {                "key": "project",                "value": "kubernetes"            },            {                "key": "name",                "value": "dev-client"            },            {                "key": "owner",                "value": "k8s"            }        ]    },    "subnet_uuid": "4b421367-165a-4555-80ab-2cff90cb9401",    "id_perms": {        "enable": true,        "description": null,        "creator": null,        "created": "2018-01-12T16:29:34.763793",        "uuid": {            "uuid_mslong": 14293345578320728551,            "uuid_lslong": 10333036915785587949        },        "user_visible": true,        "last_modified": "2018-01-12T16:29:34.810063",        "permissions": {            "owner": "cloud-admin",            "owner_access": 7,            "other_access": 7,            "group": "cloud-admin-group",            "group_access": 7        }    },    "virtual_machine_interface_refs": [        {            "to": [                "default-domain",                "kubernetes",                "dev-client__c64b3b12-f7b5-11e7-8f66-52540065dced"            ],            "href": "http://127.0.0.1:8082/virtual-machine-interface/c64b3b12-f7b5-11e7-8f66-52540065dced",            "attr": null,            "uuid": "c64b3b12-f7b5-11e7-8f66-52540065dced"        }    ],    "service_instance_ip": false,    "instance_ip_local_ip": false,    "virtual_network_refs": [        {            "to": [                "default-domain",                "kubernetes",                "dev-vn"            ],            "href": "http://127.0.0.1:8082/virtual-network/ce01826b-e3e6-407f-8798-80612018e89c",            "attr": null,            "uuid": "ce01826b-e3e6-407f-8798-80612018e89c"        }    ],    "instance_ip_secondary": false,    "display_name": "dev__dev-client"}

Appendix B Service

B.1 LB VMI

{    "fq_name": [        "default-domain",        "kubernetes",        "svc-dev-web__20c27603-2d0f-45f5-9647-defe4adaba9a"    ],    "virtual_machine_interface_mac_addresses": {        "mac_address": [            "02:20:c2:76:03:2d"        ]    },    "display_name": "dev-share__svc-dev-web",    "security_group_refs": [        {            "to": [                "default-domain",                "kubernetes",                "k8s-default-dev-share-sg"            ],            "href": "http://127.0.0.1:8082/security-group/791f1c7e-a66e-4c47-ba05-409f00ee2c8e",            "attr": null,            "uuid": "791f1c7e-a66e-4c47-ba05-409f00ee2c8e"        },        {            "to": [                "default-domain",                "kubernetes",                "k8s-default-dev-share-default"            ],            "href": "http://127.0.0.1:8082/security-group/ad29de07-5ef6-4f55-86bb-52c44827c09d",            "attr": null,            "uuid": "ad29de07-5ef6-4f55-86bb-52c44827c09d"        }    ],    "routing_instance_refs": [        {            "to": [                "default-domain",                "kubernetes",                "cluster-network",                "cluster-network"            ],            "href": "http://127.0.0.1:8082/routing-instance/5ed7608a-28bb-4735-a8d8-2e9132b03d62",            "attr": {                "direction": "both",                "protocol": null,                "ipv6_service_chain_address": null,                "dst_mac": null,                "mpls_label": null,                "vlan_tag": null,                "src_mac": null,                "service_chain_address": null            },            "uuid": "5ed7608a-28bb-4735-a8d8-2e9132b03d62"        }    ],    "virtual_machine_interface_disable_policy": false,    "parent_type": "project",    "perms2": {        "owner": "None",        "owner_access": 7,        "global_access": 0,        "share": []    },    "virtual_network_refs": [        {            "to": [                "default-domain",                "kubernetes",                "cluster-network"            ],            "href": "http://127.0.0.1:8082/virtual-network/1b9f7f74-17f0-493a-9108-729f91b43598",            "attr": null,            "uuid": "1b9f7f74-17f0-493a-9108-729f91b43598"        }    ],    "id_perms": {        "enable": true,        "description": null,        "creator": null,        "created": "2018-01-12T15:21:05.324801",        "uuid": {            "uuid_mslong": 2360578910708516341,            "uuid_lslong": 10828869012794555034        },        "user_visible": true,        "last_modified": "2018-01-12T15:21:05.365345",        "permissions": {            "owner": "cloud-admin",            "owner_access": 7,            "other_access": 7,            "group": "cloud-admin-group",            "group_access": 7        }    },    "vlan_tag_based_bridge_domain": false,    "virtual_machine_interface_device_owner": "K8S:LOADBALANCER",    "port_security_enabled": true,    "uuid": "20c27603-2d0f-45f5-9647-defe4adaba9a"}

B.2 LB IP instance and floating IP