ingress-nginx LoadBalancer 基于Metallb BGP模式(三)

Kubernetes / 2022-12-16

采用flannel组网

部署frr软路由
# 默认配置启动frr容器
docker run  -d --name AlmaLinux-BGP  frrouting/frr

# 提取镜像默认配置到/etc/frr,持久化存储配置做准备
mkdir -p /etc/frr
docker cp AlmaLinux-BGP:/etc/frr/daemons /etc/frr/daemons

# 修改配置文件,启动BGP
sed -i 's#bgpd=no#bgpd=yes#g' /etc/frr/daemons

# 关闭默认配置frr容器
docker stop AlmaLinux-BGP && docker rm AlmaLinux-BGP

# 持久化存储启动frr容器
docker run  -d --restart=always \
        --name AlmaLinux-BGP  \
        -v /etc/frr:/etc/frr:Z \
        --net=host \
        --privileged \
        frrouting/frr
配置BGP

docker exec -it AlmaLinux-BGP vtysh进入frr容器,配置BGP

config
router bgp 65000
bgp router-id 10.20.13.10
no bgp ebgp-requires-policy
neighbor k8s peer-group
neighbor k8s remote-as 65001
neighbor k8s password k8s
neighbor 10.20.13.11 peer-group k8s
neighbor 10.20.13.11 password k8s
neighbor 10.20.13.12 peer-group k8s
neighbor 10.20.13.12 password k8s

只加入node节点即可

修改configmap(没有则创建,详看上一篇)
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    # The peers section tells MetalLB what BGP routers to connect too. There
    # is one entry for each router you want to peer with.
    peers:
    - peer-address: 10.20.13.10
      peer-asn: 65000
      my-asn: 65001
      password: "k8s"
    address-pools:
    - name: default
      protocol: bgp
      addresses:
      - 192.168.0.100-192.168.0.200

修改配置后删除Metallb 删除控制器原来的pod,k8s自动重启新pod

kubectl delete pod -n metallb-system -l app=metallb,component=controller
kubectl get svc -n ingress-nginx
NAME                                 TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller             LoadBalancer   10.109.134.0    10.20.13.90   80:30926/TCP,443:30424/TCP   26h
ingress-nginx-controller-admission   ClusterIP      10.98.231.139   <none>        443/TCP                      26h

验证

cka-1# show ip bgp  peer-group

BGP peer-group k8s, remote AS 65001
  Peer-group type is external
  Configured address-families: IPv4 Unicast;
  Peer-group members:
    10.20.13.11  Established
    10.20.13.12  Established

确保状态为Established

查看BGP状态
cka-1# show ip route bgp
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

B>* 10.20.13.90/32 [20/0] via 10.20.13.11, ens192, weight 1, 00:59:13
B>* 10.20.13.91/32 [20/0] via 10.20.13.11, ens192, weight 1, 00:59:13
  *                       via 10.20.13.12, ens192, weight 1, 00:59:13

查看路由表
cka-1# show ip bgp
BGP table version is 7, local router ID is 10.20.13.10, vrf id 0
Default local pref 100, local AS 65000
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

   Network          Next Hop            Metric LocPrf Weight Path
*> 10.20.13.90/32   10.20.13.11                            0 65001 ?
*= 10.20.13.91/32   10.20.13.12                            0 65001 ?
*>                  10.20.13.11                            0 65001 ?

Displayed  2 routes and 3 total paths
测试结果

curl http://192.168.0.100

Hostname: echoserver-8585bfb456-8brrb