On Openstack network
This version is Stein version and adopts OVS network
Test environment: 1 control node, 2 computing nodes
1, Network structure diagram
Borrow 2 online pictures
Figure 1 👇:
Figure 2 👇:
2, OVS analysis
As shown in Figure 2
vm1 represents an instance
QBR XXX stands for Linux virtual bridge
You can view it through the command #brctl show //compute node
[root@compute02 ~]# brctl show|grep bfda qbrbfda8db6-7b 8000.5e414374148d no qvbbfda8db6-7b tapbfda8db6-7b
It can be seen that there are two virtual network ports on the virtual bridge (qbrbfda8db6-7b), as shown in A and C corresponding to figure 2
Br int: it can be understood that the layer-2 switch of the virtual machine can access different instances in the same subnet. At the same time, it also has the function of VLAN tag, which can distinguish different network segments. You can view that there are corresponding ports on BR int through: #ovs-vsctl show, corresponding to D in Figure 2[root@compute02 ~]# ovs-vsctl show|grep bfda
Port "qvobfda8db6-7b"
Interface "qvobfda8db6-7b"
[root@compute02 ~]# ovs-vsctl show|grep patch-tun
options: {peer=patch-tun}
Port patch-tun
Interface patch-tun
When instances in the same subnet are not on the same computer node, data needs to be forwarded through br tun. Process:
- The packet sent by the instance reaches br int through the Linux bridge and is marked with VLAN ID Tag
- Arrive at br Tun, convert VLAN ID into Tunnel ID, send it from GRE Tunnel and arrive at another compute node
- On the other compute node, the reverse process occurs
On the br Tun node, there will also be a virtual machine interface connected to br int, corresponding to port F in Figure 2
[root@compute02 ~]# ovs-vsctl show|grep patch-in
Port patch-int
Interface patch-int
options: {peer=patch-int}
Br Tun connection between different compute node s, as shown in ports G and H in Figure 2
[root@compute02 ~]# ovs-vsctl show
6febea71-608a-4161-af6b-7bfb24fea765
Manager "ptcp:6640:127.0.0.1"
is_connected: true
Bridge br-tun
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
datapath_type: system
Port "vxlan-0aa80711"
Interface "vxlan-0aa80711"
type: vxlan
options: {df_default="true", egress_pkt_mark="0", in_key=flow, local_ip="10.168.7.132", out_key=flow, remote_ip="10.168.7.17"}
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Port br-tun
Interface br-tun
type: internal
Port "vxlan-0aa80710"
Interface "vxlan-0aa80710"
type: vxlan
options: {df_default="true", egress_pkt_mark="0", in_key=flow, local_ip="10.168.7.132", out_key=flow, remote_ip="10.168.7.16"}
As mentioned above, there are two vxlan XXX ports on the br Tun virtual bridge (only one compute node in Figure 2 and two compute nodes in my test environment). Connect the br tun of the two compute nodes respectively, so that the network node and two compute nodes can be connected through the vxlan XXX port of BR tun.
To sum up, the network on the same compute node and the network between different compute nodes can be connected
3, DHCP service
Each subnet has a corresponding DHCP service provided by the network node, and each DHCP has a corresponding namespace to isolate the disconnected DHCP services
[root@controller ~]# ip netns list qdhcp-96c5a292-af46-4ae8-ae42-cf47ab21dd7a (id: 8) qrouter-db3c68f9-e7ba-4a75-abff-ad78bccb34de (id: 15) qrouter-c8edeee3-d95d-45f2-9290-4c21e7928404 qdhcp-fb12f0d5-40c5-41a5-8ea8-156ea71db5fa (id: 14) qdhcp-0dd98fc2-d30a-433d-bd9a-466970c8b72b (id: 13) qdhcp-8e73b70d-0f88-4803-94ad-870e99e36428 (id: 12) qdhcp-07b7235a-b6a3-46c3-af32-d3bd341038e9 (id: 11) qdhcp-4fba4040-4b9c-4631-8563-9647d1eed5b8 (id: 10) qdhcp-6a840bae-e75d-4e14-9a63-10424c5d5f4d (id: 9) qdhcp-0c17d541-2035-472f-8441-0d3750e137a2 (id: 7) qdhcp-45b132ab-0fba-4ffb-8b91-ad12cf083d15 (id: 4) qdhcp-74024eeb-1506-45a2-a776-fa30ff114071 (id: 6) qdhcp-69663c23-ad87-4ffd-931d-5e3e85eb0bb2 (id: 5) qdhcp-4633daf8-4495-4c4a-ab66-ab16875aa036 (id: 2) qdhcp-f678d5c8-b5e0-4783-97aa-28f0db680172 (id: 3) qdhcp-28c483f7-e520-4fc8-b363-ac8f4a1cb74f (id: 0) qrouter-be384c66-1834-4265-8f8a-9c85c016ee79 (id: 1)
[root@controller ~]# ip netns exec qdhcp-96c5a292-af46-4ae8-ae42-cf47ab21dd7a ifconfig
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
tap46db5863-b8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 172.12.0.2 netmask 255.255.255.0 broadcast 172.12.0.255
inet6 fe80::f816:3eff:fe6b:8949 prefixlen 64 scopeid 0x20<link>
ether fa:16:3e:6b:89:49 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 5 bytes 446 (446.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
It can be seen that in the corresponding ns There is a corresponding virtual network port tap46db5863-b8，Corresponding to figure 2 P Mouth. At the other end br-int Above, corresponding to Fig. 2 O mouth
[root@controller ~]# ovs-vsctl show|grep -C10 46db
type: patch
options: {peer=int-br-ex}
Bridge br-int
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "tap645ced8c-52"
tag: 14
Interface "tap645ced8c-52"
type: internal
Port "tap46db5863-b8"
tag: 19
Interface "tap46db5863-b8"
type: internal
Port "tapefd340f0-9d"
tag: 32
Interface "tapefd340f0-9d"
type: internal
Port "qr-9959547d-8f"
tag: 4095
Interface "qr-9959547d-8f"
type: internal
Port "tapc39b592a-14"
4, Route service
Router can provide Internet access, and each router also corresponds to a namespace
[root@controller ~]# ip netns list qdhcp-b613251e-b959-4e8a-b959-515823a51d30 (id: 17) qdhcp-74bd9951-523f-4732-96e9-5b7eccf01407 (id: 16) qdhcp-96c5a292-af46-4ae8-ae42-cf47ab21dd7a (id: 8) qrouter-db3c68f9-e7ba-4a75-abff-ad78bccb34de (id: 15) qrouter-c8edeee3-d95d-45f2-9290-4c21e7928404 qrouter-be384c66-1834-4265-8f8a-9c85c016ee79 (id: 1)
[root@controller ~]# ip netns exec qrouter-be384c66-1834-4265-8f8a-9c85c016ee79 ifconfig
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 52 bytes 5219 (5.0 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 52 bytes 5219 (5.0 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qg-77688753-4e: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.100.18 netmask 255.255.255.0 broadcast 192.168.100.255
inet6 fe80::f816:3eff:fe14:8bf2 prefixlen 64 scopeid 0x20<link>
ether fa:16:3e:14:8b:f2 txqueuelen 1000 (Ethernet)
RX packets 94919400 bytes 93216606173 (86.8 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 36054483 bytes 4787900088 (4.4 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qr-104bbda7-ca: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 192.168.198.1 netmask 255.255.255.0 broadcast 192.168.198.255
inet6 fe80::f816:3eff:feb9:3ec prefixlen 64 scopeid 0x20<link>
ether fa:16:3e:b9:03:ec txqueuelen 1000 (Ethernet)
RX packets 15972 bytes 466577 (455.6 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 19 bytes 1250 (1.2 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qr-3b8aae49-1a: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 192.168.199.1 netmask 255.255.255.0 broadcast 192.168.199.255
inet6 fe80::f816:3eff:feea:57f9 prefixlen 64 scopeid 0x20<link>
ether fa:16:3e:ea:57:f9 txqueuelen 1000 (Ethernet)
RX packets 16105 bytes 475741 (464.5 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 85 bytes 8071 (7.8 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qr-42ff6e11-9f: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 10.11.11.1 netmask 255.255.255.0 broadcast 10.11.11.255
inet6 fe80::f816:3eff:feb0:d3a3 prefixlen 64 scopeid 0x20<link>
ether fa:16:3e:b0:d3:a3 txqueuelen 1000 (Ethernet)
RX packets 36987969 bytes 4483620005 (4.1 GiB)
RX errors 0 dropped 123 overruns 0 frame 0
TX packets 50672255 bytes 91520023500 (85.2 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Among them, QR xxxxx is the gateway of the subnet and forms a pair with QR XXXX on BR int, corresponding to ports N and M in Figure 2
QG XXXX is the outgoing network card, which is equivalent to the WAN port of the Router and corresponds to port K in Figure 2
5, Floating IP
The nat table is configured in the Router's namespace
[root@controller ~]# ip netns exec qrouter-be384c66-1834-4265-8f8a-9c85c016ee79 iptables -t nat -nvL
Chain PREROUTING (policy ACCEPT 53108 packets, 6285K bytes)
pkts bytes target prot opt in out source destination
3148K 438M neutron-l3-agent-PREROUTING all -- * * 0.0.0.0/0 0.0.0.0/0
Chain INPUT (policy ACCEPT 1728 packets, 231K bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 52 packets, 3878 bytes)
pkts bytes target prot opt in out source destination
156 10876 neutron-l3-agent-OUTPUT all -- * * 0.0.0.0/0 0.0.0.0/0
Chain POSTROUTING (policy ACCEPT 657 packets, 34260 bytes)
pkts bytes target prot opt in out source destination
606K 37M neutron-l3-agent-POSTROUTING all -- * * 0.0.0.0/0 0.0.0.0/0
589K 36M neutron-postrouting-bottom all -- * * 0.0.0.0/0 0.0.0.0/0
Chain neutron-l3-agent-OUTPUT (1 references)
pkts bytes target prot opt in out source destination
0 0 DNAT all -- * * 0.0.0.0/0 192.168.100.41 to:10.11.11.81
0 0 DNAT all -- * * 0.0.0.0/0 192.168.100.63 to:10.11.11.63
0 0 DNAT all -- * * 0.0.0.0/0 192.168.100.16 to:10.11.11.25
0 0 DNAT all -- * * 0.0.0.0/0 192.168.100.237 to:10.11.11.66
0 0 DNAT all -- * * 0.0.0.0/0 192.168.100.131 to:10.11.11.210
0 0 DNAT all -- * * 0.0.0.0/0 192.168.100.239 to:10.11.11.30
0 0 DNAT all -- * * 0.0.0.0/0 192.168.100.33 to:10.11.11.118
0 0 DNAT all -- * * 0.0.0.0/0 192.168.100.43 to:10.11.11.148
Chain neutron-l3-agent-POSTROUTING (1 references)
pkts bytes target prot opt in out source destination
17074 1023K ACCEPT all -- * !qg-77688753-4e 0.0.0.0/0 0.0.0.0/0 ! ctstate DNAT
Chain neutron-l3-agent-PREROUTING (1 references)
pkts bytes target prot opt in out source destination
241 14460 REDIRECT tcp -- qr-+ * 0.0.0.0/0 169.254.169.254 tcp dpt:80 redir ports 9697
5 348 DNAT all -- * * 0.0.0.0/0 192.168.100.41 to:10.11.11.81
259 13468 DNAT all -- * * 0.0.0.0/0 192.168.100.63 to:10.11.11.63
11 758 DNAT all -- * * 0.0.0.0/0 192.168.100.16 to:10.11.11.25
4 1416 DNAT all -- * * 0.0.0.0/0 192.168.100.237 to:10.11.11.66
581 39061 DNAT all -- * * 0.0.0.0/0 192.168.100.131 to:10.11.11.210
2799 151K DNAT all -- * * 0.0.0.0/0 192.168.100.239 to:10.11.11.30
4 260 DNAT all -- * * 0.0.0.0/0 192.168.100.33 to:10.11.11.118
46 8716 DNAT all -- * * 0.0.0.0/0 192.168.100.43 to:10.11.11.148
Chain neutron-l3-agent-float-snat (1 references)
pkts bytes target prot opt in out source destination
631 37931 SNAT all -- * * 10.11.11.81 0.0.0.0/0 to:192.168.100.41
1270 80695 SNAT all -- * * 10.11.11.63 0.0.0.0/0 to:192.168.100.63
2054 136K SNAT all -- * * 10.11.11.25 0.0.0.0/0 to:192.168.100.16
19283 1135K SNAT all -- * * 10.11.11.66 0.0.0.0/0 to:192.168.100.237
154K 9113K SNAT all -- * * 10.11.11.210 0.0.0.0/0 to:192.168.100.131
11594 779K SNAT all -- * * 10.11.11.30 0.0.0.0/0 to:192.168.100.239
54356 3424K SNAT all -- * * 10.11.11.118 0.0.0.0/0 to:192.168.100.33
102K 5455K SNAT all -- * * 10.11.11.148 0.0.0.0/0 to:192.168.100.43
Chain neutron-l3-agent-snat (1 references)
pkts bytes target prot opt in out source destination
589K 36M neutron-l3-agent-float-snat all -- * * 0.0.0.0/0 0.0.0.0/0
163K 11M SNAT all -- * qg-77688753-4e 0.0.0.0/0 0.0.0.0/0 to:192.168.100.18
0 0 SNAT all -- * * 0.0.0.0/0 0.0.0.0/0 mark match ! 0x2/0xffff ctstate DNAT to:192.168.100.18
Chain neutron-postrouting-bottom (1 references)
pkts bytes target prot opt in out source destination
589K 36M neutron-l3-agent-snat all -- * * 0.0.0.0/0 0.0.0.0/0 /* Perform source NAT on outgoing traffic. */
6, Br ex
Br ex bind a physical port for outgoing
[root@controller ~]# ovs-vsctl show
c46120c7-9020-477f-93c6-33dda84ba833
Manager "ptcp:6640:127.0.0.1"
is_connected: true
Manager "tcp:127.0.0.1:6640"
is_connected: true
Bridge br-ex
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port br-ex
Interface br-ex
type: internal
Port "em4"
Interface "em4"
Port phy-br-ex
Interface phy-br-ex
type: patch
options: {peer=int-br-ex}
Where " em4"Is the outgoing network card and binds it to the br-ex，Connect the Internet to em4，And in openstack of Route Configure the corresponding Internet address on the
7, Security group
The instance's security group is implemented through iptables on the corresponding compute node, and the rules in the security group will be reflected in iptables
iptables controls the instance interface, similar to tap XXXX,
For example, the interface of an instance is: (0db0e25b-000f)
You can see the corresponding iptables rules on compute node
[root@compute01 ~]# iptables -nvL neutron-openvswi-o0db0e25b-0
Chain neutron-openvswi-o0db0e25b-0 (2 references)
pkts bytes target prot opt in out source destination
0 0 RETURN udp -- * * 0.0.0.0 255.255.255.255 udp spt:68 dpt:67 /* Allow DHCP client traffic. */
26207 1994K neutron-openvswi-s0db0e25b-0 all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 RETURN udp -- * * 0.0.0.0/0 0.0.0.0/0 udp spt:68 dpt:67 /* Allow DHCP client traffic. */
0 0 DROP udp -- * * 0.0.0.0/0 0.0.0.0/0 udp spt:67 dpt:68 /* Prevent DHCP Spoofing by VM. */
288 29909 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */
1 40 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID /* Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack. */
19835 1470K neutron-openvswi-sg-fallback all -- * * 0.0.0.0/0 0.0.0.0/0 /* Send unmatched traffic to the fallback chain. */
Moreover, the forward chain is rejected by default, so the instance cannot forward traffic without closing the port security and security group. I didn't post all the tables here, because there are too many self built chains in openstack, so I need to find the corresponding chain myself. I only posted the final policy. The last one below
[root@compute01 ~]# iptables -nvL neutron-openvswi-i0db0e25b-0
Chain neutron-openvswi-i0db0e25b-0 (1 references)
pkts bytes target prot opt in out source destination
1019 73957 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */
0 0 RETURN udp -- * * 0.0.0.0/0 172.16.0.249 udp spt:67 dpt:68
2 781 RETURN udp -- * * 0.0.0.0/0 255.255.255.255 udp spt:67 dpt:68
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID /* Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack. */
Moreover, floating IP cannot take effect without turning off port security
If there is something wrong, please correct it