参考文献

环境说明

  • 主机: MBP M1 macOS 14.1.2

  • 拓扑结构: 一个Master节点,一个Node节点

    角色 hostname IP
    Master node-1 192.168.11.173
    Node node-2 192.168.11.174

  • 虚拟机版本: Ubuntu 20.04.5 LTS

系统配置

  • 注: 以下操作需要在所有节点上执行
配置固定IP
配置内核模块

  • 创建/etc/modules-load.d/containerd.conf配置文件，确保在系统启动时自动加载所需的内核模块，以满足容器运行时的要求

    1
    2
    3
    4
    cat << EOF > /etc/modules-load.d/containerd.conf
    overlay
    br_netfilter
    EOF

  • 执行以下命令使配置生效

    1
    2
    modprobe overlay
    modprobe br_netfilter

  • 创建/etc/sysctl.d/99-kubernetes-cri.conf配置文件

    1
    2
    3
    4
    5
    6
    cat << EOF > /etc/sysctl.d/99-kubernetes-cri.conf
    net.bridge.bridge-nf-call-ip6tables = 1
    net.bridge.bridge-nf-call-iptables = 1
    net.ipv4.ip_forward = 1
    user.max_user_namespaces=28633
    EOF

  • 执行以下命令使配置生效

    1
    sysctl -p /etc/sysctl.d/99-kubernetes-cri.conf

    在文件名/etc/sysctl.d/99-kubernetes-cri.conf中，“99” 代表文件的优先级或顺序。sysctl是Linux内核参数的配置工具，它可以通过修改/proc/sys/目录下的文件来设置内核参数。在/etc/sysctl.d/目录中，可以放置一系列的配置文件，以便在系统启动时自动加载这些参数。这些配置文件按照文件名的字母顺序逐个加载。数字前缀用于指定加载的顺序，较小的数字表示较高的优先级。

配置ipvs

  • 开启IPVS(IP Virtual Server)的目的是为了优化集群内部的负载均衡性能

    • 性能更高: IPVS 是在 Linux 内核中实现的四层（L4）负载均衡技术，能够处理大量连接并提供更快的连接建立速度。与基于规则匹配的iptables相比，IPVS在处理大规模流量时效率更高。

  • IPVS和iptables的对比

    特性 IPVS iptables
    性能 高,基于哈希表处理大规模规则 较低,线性处理规则集
    负载均衡算法 提供多种负载均衡算法 仅支持简单的轮询算法
    处理方式 内核态处理,性能更高 基于规则匹配,规则越多,性能越差
    状态保持 支持连接持久化 状态保持支持较弱
    适用场景 大规模集群,复杂流量分发 小规模集群,简单流量分发

  • 创建/etc/modules-load.d/ipvs.conf文件，保证在节点重启后能自动加载所需模块

    1
    2
    3
    4
    5
    6
    cat > /etc/modules-load.d/ipvs.conf <<EOF
    ip_vs
    ip_vs_rr
    ip_vs_wrr
    ip_vs_sh
    EOF

  • 使用lsmod | grep -e ip_vs -e nf_conntrack命令查看是否已经正确加载所需的内核模块

  • 接下来还需要确保各个节点上已经安装了ipset软件包，为了便于查看ipvs的代理规则，最好安装一下管理工具ipvsadm。

    1
    apt-get install -y ipset ipvsadm

  • 如果不满足以上前提条件，则即使kube-proxy的配置开启了ipvs模式，也会退回到iptables模式。

关闭swap

  • 关闭swap

    1
    swapoff -a

  • 修改/etc/fstab文件，注释掉 SWAP 的自动挂载，使用free -m确认swap已经关闭

  • swappiness参数调整，修改/etc/sysctl.d/99-kubernetes-cri.conf添加下面一行

    1
    vm.swappiness = 0
    1
    2
    3
    4
    5
    6
    cat /etc/sysctl.d/99-kubernetes-cri.conf
    net.bridge.bridge-nf-call-ip6tables = 1
    net.bridge.bridge-nf-call-iptables = 1
    net.ipv4.ip_forward = 1
    user.max_user_namespaces = 28633
    vm.swappiness = 0

  • 执行sysctl -p /etc/sysctl.d/99-kubernetes-cri.conf使修改生效。

组件安装

  • 注: 以下操作需要在所有节点上执行

部署容器运行时containerd

  • 以下步骤可以参考: 官方文档

  • 下载containerd的二进制包,下载地址.选择对应的版本containerd-<VERSION>-<OS>-<ARCH>.tar.gz,此处我选择containerd-1.7.14-linux-arm64.tar.gz

    1
    wget https://github.com/containerd/containerd/releases/download/v1.7.14/containerd-1.7.14-linux-arm64.tar.gz
    1
    2
    3
    4
    5
    6
    7
    8
    $ tar Cxzvf /usr/local containerd-1.7.14-linux-arm64.tar.gz
    bin/
    bin/containerd-shim-runc-v2
    bin/containerd-shim
    bin/ctr
    bin/containerd-shim-runc-v1
    bin/containerd
    bin/containerd-stress

  • 生成containerd的配置文件

    1
    2
    mkdir -p /etc/containerd
    containerd config default > /etc/containerd/config.toml

    • 根据文档Container runtimes中的内容，对于使用systemd作为init system的Linux的发行版，使用systemd作为容器的cgroup driver可以确保服务器节点在资源紧张的情况更加稳定，因此这里配置各个节点上containerd的cgroup driver为systemd。

    • 修改前面生成的配置文件/etc/containerd/config.toml

      1
      2
      3
      4
      5
      6
      7
      8
      vim /etc/containerd/config.toml
      # 进入编辑模式 输入/SystemdCgroup定位到具体要修改的位置将SystemdCgroup修改为true

        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
          SystemdCgroup = true
          
      # 按Esc,重新搜索/sandbox_image定位后,修改pause镜像获取地址改为国内地址 
          sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.9"

  • 此外配置containerd的镜像加速(注: 以下加速地址可能失效,使用中需要甄别使用)

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
    93
    94
    95
    96
    97
    98
    99
    100
    101
    102
    103
    104
    105
    106
    107
    108
    109
    mkdir -p /etc/containerd/certs.d/docker.io
    cat > /etc/containerd/certs.d/docker.io/hosts.toml << EOF
    server = "https://docker.io"
    [host."https://dockerproxy.com"]
      capabilities = ["pull", "resolve"]

    [host."https://docker.m.daocloud.io"]
      capabilities = ["pull", "resolve"]

    [host."https://reg-mirror.qiniu.com"]
      capabilities = ["pull", "resolve"]

    [host."https://registry.docker-cn.com"]
      capabilities = ["pull", "resolve"]

    [host."http://hub-mirror.c.163.com"]
      capabilities = ["pull", "resolve"]

    EOF

    # registry.k8s.io镜像加速
    mkdir -p /etc/containerd/certs.d/registry.k8s.io
    tee /etc/containerd/certs.d/registry.k8s.io/hosts.toml << 'EOF'
    server = "https://registry.k8s.io"

    [host."https://k8s.m.daocloud.io"]
      capabilities = ["pull", "resolve", "push"]
    EOF

    # docker.elastic.co镜像加速
    mkdir -p /etc/containerd/certs.d/docker.elastic.co
    tee /etc/containerd/certs.d/docker.elastic.co/hosts.toml << 'EOF'
    server = "https://docker.elastic.co"

    [host."https://elastic.m.daocloud.io"]
      capabilities = ["pull", "resolve", "push"]
    EOF

    # gcr.io镜像加速
    mkdir -p /etc/containerd/certs.d/gcr.io
    tee /etc/containerd/certs.d/gcr.io/hosts.toml << 'EOF'
    server = "https://gcr.io"

    [host."https://gcr.m.daocloud.io"]
      capabilities = ["pull", "resolve", "push"]
    EOF

    # ghcr.io镜像加速
    mkdir -p /etc/containerd/certs.d/ghcr.io
    tee /etc/containerd/certs.d/ghcr.io/hosts.toml << 'EOF'
    server = "https://ghcr.io"

    [host."https://ghcr.m.daocloud.io"]
      capabilities = ["pull", "resolve", "push"]
    EOF

    # k8s.gcr.io镜像加速
    mkdir -p /etc/containerd/certs.d/k8s.gcr.io
    tee /etc/containerd/certs.d/k8s.gcr.io/hosts.toml << 'EOF'
    server = "https://k8s.gcr.io"

    [host."https://k8s-gcr.m.daocloud.io"]
      capabilities = ["pull", "resolve", "push"]
    EOF

    # mcr.m.daocloud.io镜像加速
    mkdir -p /etc/containerd/certs.d/mcr.microsoft.com
    tee /etc/containerd/certs.d/mcr.microsoft.com/hosts.toml << 'EOF'
    server = "https://mcr.microsoft.com"

    [host."https://mcr.m.daocloud.io"]
      capabilities = ["pull", "resolve", "push"]
    EOF

    # nvcr.io镜像加速
    mkdir -p /etc/containerd/certs.d/nvcr.io
    tee /etc/containerd/certs.d/nvcr.io/hosts.toml << 'EOF'
    server = "https://nvcr.io"

    [host."https://nvcr.m.daocloud.io"]
      capabilities = ["pull", "resolve", "push"]
    EOF

    # quay.io镜像加速
    mkdir -p /etc/containerd/certs.d/quay.io
    tee /etc/containerd/certs.d/quay.io/hosts.toml << 'EOF'
    server = "https://quay.io"

    [host."https://quay.m.daocloud.io"]
      capabilities = ["pull", "resolve", "push"]
    EOF

    # registry.jujucharms.com镜像加速
    mkdir -p /etc/containerd/certs.d/registry.jujucharms.com
    tee /etc/containerd/certs.d/registry.jujucharms.com/hosts.toml << 'EOF'
    server = "https://registry.jujucharms.com"

    [host."https://jujucharms.m.daocloud.io"]
      capabilities = ["pull", "resolve", "push"]
    EOF

    # rocks.canonical.com镜像加速
    mkdir -p /etc/containerd/certs.d/rocks.canonical.com
    tee /etc/containerd/certs.d/rocks.canonical.com/hosts.toml << 'EOF'
    server = "https://rocks.canonical.com"

    [host."https://rocks-canonical.m.daocloud.io"]
      capabilities = ["pull", "resolve", "push"]
    EOF

  • 将contianerd配置服务,可下载https://raw.githubusercontent.com/containerd/containerd/main/containerd.service到/usr/local/lib/systemd/system/containerd.service,然后应用配置

    1
    2
    systemctl daemon-reload
    systemctl enable --now containerd
    • containerd.service文件内容如下:
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    # Copyright The containerd Authors.
    #
    # Licensed under the Apache License, Version 2.0 (the "License");
    # you may not use this file except in compliance with the License.
    # You may obtain a copy of the License at
    #
    #     http://www.apache.org/licenses/LICENSE-2.0
    #
    # Unless required by applicable law or agreed to in writing, software
    # distributed under the License is distributed on an "AS IS" BASIS,
    # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    # See the License for the specific language governing permissions and
    # limitations under the License.

    [Unit]
    Description=containerd container runtime
    Documentation=https://containerd.io
    After=network.target local-fs.target

    [Service]
    ExecStartPre=-/sbin/modprobe overlay
    ExecStart=/usr/local/bin/containerd

    Type=notify
    Delegate=yes
    KillMode=process
    Restart=always
    RestartSec=5

    # Having non-zero Limit*s causes performance problems due to accounting overhead
    # in the kernel. We recommend using cgroups to do container-local accounting.
    LimitNPROC=infinity
    LimitCORE=infinity

    # Comment TasksMax if your systemd version does not supports it.
    # Only systemd 226 and above support this version.
    TasksMax=infinity
    OOMScoreAdjust=-999

    [Install]
    WantedBy=multi-user.target

  • 安装验证

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    $ containerd -h
    NAME:
       containerd -
                        __        _                     __
      _________  ____  / /_____ _(_)___  ___  _________/ /
     / ___/ __ \/ __ \/ __/ __ `/ / __ \/ _ \/ ___/ __  /
    / /__/ /_/ / / / / /_/ /_/ / / / / /  __/ /  / /_/ /
    \___/\____/_/ /_/\__/\__,_/_/_/ /_/\___/_/   \__,_/

    high performance container runtime


    USAGE:
       containerd [global options] command [command options] [arguments...]

    VERSION:
       v1.7.14

    DESCRIPTION:

    containerd is a high performance container runtime whose daemon can be started
    by using this command. If none of the *config*, *publish*, *oci-hook*, or *help* commands
    are specified, the default action of the **containerd** command is to start the
    containerd daemon in the foreground.


    A default configuration is used if no TOML configuration is specified or located
    at the default file location. The *containerd config* command can be used to
    generate the default configuration for containerd. The output of that command
    can be used and modified as necessary as a custom configuration.

    COMMANDS:
       config    Information on the containerd config
       publish   Binary to publish events to containerd
       oci-hook  Provides a base for OCI runtime hooks to allow arguments to be injected.
       help, h   Shows a list of commands or help for one command

    GLOBAL OPTIONS:
       --config value, -c value     Path to the configuration file (default: "/etc/containerd/config.toml")
       --log-level value, -l value  Set the logging level [trace, debug, info, warn, error, fatal, panic]
       --address value, -a value    Address for containerd's GRPC server
       --root value                 containerd root directory
       --state value                containerd state directory
       --help, -h                   show help
       --version, -v                print the version

部署runc

  • 下载地址,下载对应版本的二进制文件,此处我选择runc.arm64

    1
    wget https://github.com/opencontainers/runc/releases/download/v1.2.0-rc.2/runc.arm64
    1
    $ install -m 755 runc.amd64 /usr/local/sbin/runc

  • 安装验证

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    $ runc -h
    NAME:
       runc - Open Container Initiative runtime

    runc is a command line client for running applications packaged according to
    the Open Container Initiative (OCI) format and is a compliant implementation of the
    Open Container Initiative specification.

    runc integrates well with existing process supervisors to provide a production
    container runtime environment for applications. It can be used with your
    existing process monitoring tools and the container will be spawned as a
    direct child of the process supervisor.

    Containers are configured using bundles. A bundle for a container is a directory
    that includes a specification file named "config.json" and a root filesystem.
    The root filesystem contains the contents of the container.

    To start a new instance of a container:

        # runc run [ -b bundle ] <container-id>

    Where "<container-id>" is your name for the instance of the container that you
    are starting. The name you provide for the container instance must be unique on
    your host. Providing the bundle directory using "-b" is optional. The default
    value for "bundle" is the current directory.

    USAGE:
       runc [global options] command [command options] [arguments...]

    VERSION:
       1.2.0-rc.2
    commit: v1.2.0-rc.2-0-gf2d2ee5e-dirty
    spec: 1.2.0
    go: go1.22.3
    libseccomp: 2.5.5

    COMMANDS:
       checkpoint  checkpoint a running container
       create      create a container
       delete      delete any resources held by the container often used with detached container
       events      display container events such as OOM notifications, cpu, memory, and IO usage statistics
       exec        execute new process inside the container
       kill        kill sends the specified signal (default: SIGTERM) to the container's init process
       list        lists containers started by runc with the given root
       pause       pause suspends all processes inside the container
       ps          ps displays the processes running inside a container
       restore     restore a container from a previous checkpoint
       resume      resumes all processes that have been previously paused
       run         create and run a container
       spec        create a new specification file
       start       executes the user defined process in a created container
       state       output the state of a container
       update      update container resource constraints
       features    show the enabled features
       help, h     Shows a list of commands or help for one command

    GLOBAL OPTIONS:
       --debug             enable debug logging
       --log value         set the log file to write runc logs to (default is '/dev/stderr')
       --log-format value  set the log format ('text' (default), or 'json') (default: "text")
       --root value        root directory for storage of container state (this should be located in tmpfs) (default: "/run/user/1000/runc")
       --systemd-cgroup    enable systemd cgroup support, expects cgroupsPath to be of form "slice:prefix:name" for e.g. "system.slice:runc:434234"
       --rootless value    ignore cgroup permission errors ('true', 'false', or 'auto') (default: "auto")
       --help, -h          show help
       --version, -v       print the version

部署crictl

  • 官网文档

  • 下载地址,此处选择crictl-v1.31.1-linux-arm64.tar.gz

    1
    2
    3
    4
    5
    6
    VERSION="v1.31.1" # check latest version in /releases page
    OS="linux"
    ARCH="arm64"
    wget https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/crictl-$VERSION-$OS-$ARCH.tar.gz
    sudo tar zxvf crictl-$VERSION-$OS-$ARCH.tar.gz -C /usr/local/bin
    rm -f crictl-$VERSION-$OS-$ARCH.tar.gz

  • 配置containerd为crictl首选runtime-endpoint

    1
    2
    3
    4
    5
    6
    7
    # tee /etc/crictl.yaml << 'EOF'
    runtime-endpoint: unix:///run/containerd/containerd.sock
    image-endpoint: unix:///run/containerd/containerd.sock
    timeout: 2
    debug: true
    pull-image-on-create: false
    EOF

  • 安装验证

    1
    2
    3
    4
    5
    6
    7
    8
    # crictl version
    DEBU[0000] get runtime connection
    DEBU[0000] VersionRequest: &VersionRequest{Version:v1,}
    DEBU[0000] VersionResponse: &VersionResponse{Version:0.1.0,RuntimeName:containerd,RuntimeVersion:v1.7.14,RuntimeApiVersion:v1,}
    Version:  0.1.0
    RuntimeName:  containerd
    RuntimeVersion:  v1.7.14
    RuntimeApiVersion:  v1

使用kubeadm部署Kubernetes

  • 组件说明

    • kubeadm：用于引导集群的命令
    • kubelet：在集群中的所有机器上运行的组件，用于启动 Pod 和容器等操作。
    • kubectl：用于与集群通信的命令行实用程序。

  • 在各个节点上安装

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    # 更新 apt 包索引并安装使用 Kubernetes apt 存储库所需的包
    # apt-get update
    # apt-get install -y apt-transport-https ca-certificates curl gpg

    # 下载 Kubernetes 包存储库的公共签名密钥
    # curl -fsSL https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.28/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
        
    # echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.28/deb/ /" | tee /etc/apt/sources.list.d/kubernetes.list
        
    # 更新 apt 包索引，安装 kubelet、kubeadm 和 kubectl，并固定它们的版本
    # apt-get update
    # apt install kubelet kubeadm kubectl
    # apt-mark hold kubelet kubeadm kubectl

  • 在各节点开机启动kubelet服务

    1
    # systemctl enable kubelet.service

初始化集群

  • kubeadm config print init-defaults官方文档说明

  • 使用kubeadm config print init-defaults --component-configs KubeletConfiguration > kubeadmin-init.yaml可以打印集群初始化默认的使用的配置

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    apiVersion: kubeadm.k8s.io/v1beta3
    bootstrapTokens:
    - groups:
      - system:bootstrappers:kubeadm:default-node-token
      token: abcdef.0123456789abcdef
      ttl: 24h0m0s
      usages:
      - signing
      - authentication
    kind: InitConfiguration
    localAPIEndpoint:
    	# 此处修改为Master节点的IP地址
      advertiseAddress: 192.168.11.173
      bindPort: 6443
    nodeRegistration:
      criSocket: unix:///var/run/containerd/containerd.sock
      imagePullPolicy: IfNotPresent
      name: node
      taints: null
    ---
    apiServer:
      timeoutForControlPlane: 4m0s
    apiVersion: kubeadm.k8s.io/v1beta3
    certificatesDir: /etc/kubernetes/pki
    clusterName: kubernetes
    controllerManager: {}
    dns: {}
    etcd:
      local:
        dataDir: /var/lib/etcd
    #此处修改为国内镜像源
    imageRepository: registry.aliyuncs.com/google_containers
    kind: ClusterConfiguration
    kubernetesVersion: 1.28.0
    networking:
      dnsDomain: cluster.local
      serviceSubnet: 10.96.0.0/12
    scheduler: {}
    ---
    apiVersion: kubelet.config.k8s.io/v1beta1
    authentication:
      anonymous:
        enabled: false
      webhook:
        cacheTTL: 0s
        enabled: true
      x509:
        clientCAFile: /etc/kubernetes/pki/ca.crt
    authorization:
      mode: Webhook
      webhook:
        cacheAuthorizedTTL: 0s
        cacheUnauthorizedTTL: 0s
    # 设置kubelet的cgroupDriver为systemd    
    cgroupDriver: systemd
    clusterDNS:
    - 10.96.0.10
    clusterDomain: cluster.local
    containerRuntimeEndpoint: ""
    cpuManagerReconcilePeriod: 0s
    evictionPressureTransitionPeriod: 0s
    fileCheckFrequency: 0s
    healthzBindAddress: 127.0.0.1
    healthzPort: 10248
    httpCheckFrequency: 0s
    imageMinimumGCAge: 0s
    kind: KubeletConfiguration
    logging:
      flushFrequency: 0
      options:
        json:
          infoBufferSize: "0"
      verbosity: 0
    memorySwap: {}
    nodeStatusReportFrequency: 0s
    nodeStatusUpdateFrequency: 0s
    resolvConf: /run/systemd/resolve/resolv.conf
    rotateCertificates: true
    runtimeRequestTimeout: 0s
    shutdownGracePeriod: 0s
    shutdownGracePeriodCriticalPods: 0s
    staticPodPath: /etc/kubernetes/manifests
    streamingConnectionIdleTimeout: 0s
    syncFrequency: 0s
    volumeStatsAggPeriod: 0s
    failSwapOn: false
    ---
    # 设置kube-proxy代理模式为ipvs
    apiVersion: kubeproxy.config.k8s.io/v1alpha1
    kind: KubeProxyConfiguration
    mode: ipvs

    • 初始化之前可以,先拉取镜像kubeadm config images pull --config kubeadm-init.yaml

      1
      2
      3
      4
      5
      6
      7
      8
      # sudo kubeadm config images pull --config kubeadm-init.yaml
      [config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.28.0
      [config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.28.0
      [config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.28.0
      [config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.28.0
      [config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.9
      [config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.5.9-0
      [config/images] Pulled registry.aliyuncs.com/google_containers/coredns:v1.10.1

  • 在node-1上执行sudo kubeadm init --config kubeadm-init.yaml,执行后出现如下信息,则表示初始化集群成功

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    ...
    Your Kubernetes control-plane has initialized successfully!
    To start using your cluster, you need to run the following as a regular user:

      mkdir -p $HOME/.kube
      sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
      sudo chown $(id -u):$(id -g) $HOME/.kube/config
      
    Alternatively, if you are the root user, you can run:

      export KUBECONFIG=/etc/kubernetes/admin.conf

    You should now deploy a pod network to the cluster.
    Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
      https://kubernetes.io/docs/concepts/cluster-administration/addons/
      
    Then you can join any number of worker nodes by running the following on each as root:

    kubeadm join 192.168.11.173:6443 --token bd1pcw.zico7hebekwsw4ya \
    		--discovery-token-ca-cert-hash sha256:e5cf861d41e24e0d1c13e57f3e3de5bd75f624328ad4c94af828116129487f67

  • 根据提示配置,配置常规用户可以使用kubectl访问集群

    1
    2
    3
    $ mkdir -p $HOME/.kube
    $ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    $ sudo chown $(id -u):$(id -g) $HOME/.kube/config

  • 集群初始化如果遇到问题可以根据journalctl -xeu kubelet信息排查，排查完可以使用kubeadm reset命令进行清理,然后重新初始化集群。

加入集群

  • 切换到node-2节点上(注: node-2此时的环境与node-1相比只有kubeadm init命令及以后的命令未执行),加入集群

    1
    2
    # kubeadm join 192.168.11.173:6443 --token bd1pcw.zico7hebekwsw4ya \
    		--discovery-token-ca-cert-hash sha256:e5cf861d41e24e0d1c13e57f3e3de5bd75f624328ad4c94af828116129487f67

  • 若要在node-2也想使用kubectl访问集群,则需要将Master节点(node-1)上的.kube文件夹拷贝到当前需要使用kubectl的家目录下

    1
    2
    # 两台常规用户的用户名都是holelin
    $ scp -r holelin@192.168.11.173:/home/holelin/.kube /home/holelin/

安装网络插件

  • 此处选择部署Pod Network组件Calico
使用kubectl apply -f calico.yaml安装插件
使用helm安装插件

  • 下载地址

    1
    2
    3
    # wget https://get.helm.sh/helm-v3.15.4-linux-arm64.tar.gz
    # tar -zxvf helm-v3.15.4-linux-arm64.tar.gz
    # mv linux-ard64/helm  /usr/local/bin/

  • 验证

    1
    2
    # helm version
    version.BuildInfo{Version:"v3.15.4", GitCommit:"fa9efb07d9d8debbb4306d72af76a383895aa8c4", GitTreeState:"clean", GoVersion:"go1.22.6"}

  • 下载tigera-operator的helm chart

    1
    # wget https://github.com/projectcalico/calico/releases/download/v3.28.1/release-v3.28.1.tgz

  • 查看这个chart的中可定制的配置

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    # helm show values tigera-operator-v3.28.1.tgz > values.yaml

    # cat values.yaml
    # imagePullSecrets is a special helm field which, when specified, creates a secret
    # containing the pull secret which is used to pull all images deployed by this helm chart and the resulting operator.
    # this field is a map where the key is the desired secret name and the value is the contents of the imagePullSecret.
    #
    # Example: --set-file imagePullSecrets.gcr=./pull-secret.json
    imagePullSecrets: {}

    installation:
      enabled: true
      kubernetesProvider: ""
      # imagePullSecrets are configured on all images deployed by the tigera-operator.
      # secrets specified here must exist in the tigera-operator namespace; they won't be created by the operator or helm.
      # imagePullSecrets are a slice of LocalObjectReferences, which is the same format they appear as on deployments.
      #
      # Example: --set installation.imagePullSecrets[0].name=my-existing-secret
      imagePullSecrets: []

    apiServer:
      enabled: true

    defaultFelixConfiguration:
      enabled: false

    certs:
      node:
        key:
        cert:
        commonName:
      typha:
        key:
        cert:
        commonName:
        caBundle:

    # Resource requests and limits for the tigera/operator pod.
    resources: {}

    # Tolerations for the tigera/operator pod.
    tolerations:
    - effect: NoExecute
      operator: Exists
    - effect: NoSchedule
      operator: Exists

    # NodeSelector for the tigera/operator pod.
    nodeSelector:
      kubernetes.io/os: linux

    # Affinity for the tigera/operator pod.
    affinity: {}

    # PriorityClassName for the tigera/operator pod.
    priorityClassName: ""

    # Custom annotations for the tigera/operator pod.
    podAnnotations: {}

    # Custom labels for the tigera/operator pod.
    podLabels: {}

    # Image and registry configuration for the tigera/operator pod.
    tigeraOperator:
      image: tigera/operator
      version: v1.34.3
      registry: quay.io
    calicoctl:
      image: docker.io/calico/ctl
      tag: v3.28.1

    kubeletVolumePluginPath: /var/lib/kubelet

    # Optionally configure the host and port used to access the Kubernetes API server.
    kubernetesServiceEndpoint:
      host: ""
      port: "6443"

  • 使用helm安装calico

    1
    helm install calico tigera-operator-v3.28.1.tgz -n kube-system  --create-namespace -f values.yaml

  • 等待并确认所有pod处于Running状态

    1
    2
    3
    4
    5
    6
    7
    8
    9
    # kubectl get pod -n kube-system | grep tigera-operator

    tigera-operator-5fb55776df-wxbph   1/1     Running   0             5m10s

    # kubectl get pods -n calico-system
    NAME                                       READY   STATUS    RESTARTS   AGE
    calico-kube-controllers-68884f975d-5d7p9   1/1     Running   0          5m24s
    calico-node-twbdh                          1/1     Running   0          5m24s
    calico-typha-7b4bdd99c5-ssdn2              1/1     Running   0          5m24s

  • 查看一下calico向k8s中添加的api资源

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    # kubectl api-resources | grep calico
    bgpconfigurations                              crd.projectcalico.org/v1               false        BGPConfiguration
    bgpfilters                                     crd.projectcalico.org/v1               false        BGPFilter
    bgppeers                                       crd.projectcalico.org/v1               false        BGPPeer
    blockaffinities                                crd.projectcalico.org/v1               false        BlockAffinity
    caliconodestatuses                             crd.projectcalico.org/v1               false        CalicoNodeStatus
    clusterinformations                            crd.projectcalico.org/v1               false        ClusterInformation
    felixconfigurations                            crd.projectcalico.org/v1               false        FelixConfiguration
    globalnetworkpolicies                          crd.projectcalico.org/v1               false        GlobalNetworkPolicy
    globalnetworksets                              crd.projectcalico.org/v1               false        GlobalNetworkSet
    hostendpoints                                  crd.projectcalico.org/v1               false        HostEndpoint
    ipamblocks                                     crd.projectcalico.org/v1               false        IPAMBlock
    ipamconfigs                                    crd.projectcalico.org/v1               false        IPAMConfig
    ipamhandles                                    crd.projectcalico.org/v1               false        IPAMHandle
    ippools                                        crd.projectcalico.org/v1               false        IPPool
    ipreservations                                 crd.projectcalico.org/v1               false        IPReservation
    kubecontrollersconfigurations                  crd.projectcalico.org/v1               false        KubeControllersConfiguration
    networkpolicies                                crd.projectcalico.org/v1               true         NetworkPolicy
    networksets                                    crd.projectcalico.org/v1               true         NetworkSet