Ini adalah catatan bagaimana aku setup kubernetes cluster pakai kubeadm. Aku setup pakai VM (KVM) 1 master dan 1 node worker. Aku pakai vm ubuntu 18.04. Selama ini aku setup kubernetes cluster di debian 10. Sekarang belajar setup di ubuntu 18.04. Meskipun ubuntu berbasis debian, tapi ada beberapa yang berbeda ketika setup kubernetes cluster, yg paling kentara di Debian 10 adalah iptables-legacy.

Rangkuman singkatnya begini:

install docker dan kubeadm di master
setup kubeadm di master
install docker dan kubeadm di node
join node ke cluster pakai kubeadm
Cluster selesai, sisanya pasang aplikasi sesua kebutuhan

Log install kubernetes di Ubuntu 18.04

A. Setup Master

Node IP addr: 192.168.100.198
Node Hostname: master
OS: Ubuntu 18.04
RAM: 2 GB
CPU: 2 core

A.1 Install docker

Ikuti panduan https://docs.docker.com/engine/install/ubuntu/

apt-get update

apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -

add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"

 apt-get update
 apt-get install docker-ce docker-ce-cli containerd.io

A.2 Ganti driver docker ke systemd, awalnya cgroupfs

# Set up the Docker daemon
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

mkdir -p /etc/systemd/system/docker.service.d

# Restart Docker
systemctl daemon-reload
systemctl restart docker
systemctl enable docker

A.3 Install kubeadm

Ikuti panduan https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

cat <<EOF | tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system

apt-get update && apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF | tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl

A.4 Init cluster pakai kubeadm

Matikan swap

swapoff -a

Matikan swap pas reboot pakai crontab

@reboot swapoff -a

Disini mau pakai flannel untuk networknya, oleh karena itu pakai parameter --pod-network-cidr=10.244.0.0/16

kubeadm init --pod-network-cidr=10.244.0.0/16

root@master:/home/sumar# kubectl get po -o wide -A
NAMESPACE     NAME                             READY   STATUS    RESTARTS   AGE     IP                NODE     NOMINATED NODE   READINESS GATES
kube-system   coredns-66bff467f8-jghgf         0/1     Pending   0          4m22s   <none>            <none>   <none>           <none>
kube-system   coredns-66bff467f8-zkcdd         0/1     Pending   0          4m22s   <none>            <none>   <none>           <none>
kube-system   etcd-master                      1/1     Running   0          4m32s   192.168.100.198   master   <none>           <none>
kube-system   kube-apiserver-master            1/1     Running   0          4m32s   192.168.100.198   master   <none>           <none>
kube-system   kube-controller-manager-master   1/1     Running   0          4m32s   192.168.100.198   master   <none>           <none>
kube-system   kube-proxy-lc5ps                 1/1     Running   0          4m22s   192.168.100.198   master   <none>           <none>
kube-system   kube-scheduler-master            1/1     Running   0          4m32s   192.168.100.198   master   <none>           <none>
root@master:/home/sumar#

pod coredns status pending krn belum install flannel,

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

Setelah install flannel, pod coredns jadi jalan

root@master:/home/sumar# kubectl get po -o wide -A
NAMESPACE     NAME                             READY   STATUS    RESTARTS   AGE     IP                NODE     NOMINATED NODE   READINESS GATES
kube-system   coredns-66bff467f8-jghgf         1/1     Running   0          8m19s   10.244.0.3        master   <none>           <none>
kube-system   coredns-66bff467f8-zkcdd         1/1     Running   0          8m19s   10.244.0.2        master   <none>           <none>
kube-system   etcd-master                      1/1     Running   0          8m29s   192.168.100.198   master   <none>           <none>
kube-system   kube-apiserver-master            1/1     Running   0          8m29s   192.168.100.198   master   <none>           <none>
kube-system   kube-controller-manager-master   1/1     Running   0          8m29s   192.168.100.198   master   <none>           <none>
kube-system   kube-flannel-ds-amd64-rb4wc      1/1     Running   0          46s     192.168.100.198   master   <none>           <none>
kube-system   kube-proxy-lc5ps                 1/1     Running   0          8m19s   192.168.100.198   master   <none>           <none>
kube-system   kube-scheduler-master            1/1     Running   0          8m29s   192.168.100.198   master   <none>           <none>

B. Setup Nodesatu

Node IP addr: 192.168.100.199
Node Hostname: nodesatu
OS: Ubuntu 18.04
RAM: 2GB
CPU: 2 core

B.1 Install docker

Ikuti panduan https://docs.docker.com/engine/install/ubuntu/

apt-get update

apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -

add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"

 apt-get update
 apt-get install docker-ce docker-ce-cli containerd.io

B.2 Ganti driver docker ke systemd, awalnya cgroupfs

# Set up the Docker daemon
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

mkdir -p /etc/systemd/system/docker.service.d

# Restart Docker
systemctl daemon-reload
systemctl restart docker
systemctl enable docker

B.3 Install kubeadm

Ikuti panduan https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

cat <<EOF | tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system

apt-get update && apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF | tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl

B.4 Join nodesatu ke cluster

kubeadm join 192.168.100.198:6443 --token 98vdpp.r7tziuipa5ebwpg8 \
    --discovery-token-ca-cert-hash sha256:d6d44f1588470756cd52f70b5576aa7ddd6b5b9a57559b025dbfa21ecee2c0eb

Langkah A dan B adalah langkah untuk setup cluster. Setelah setup master selesai dan setup nodesatu selesai, kita verifikasi:

root@master:~# kubectl cluster-info
Kubernetes master is running at https://192.168.100.198:6443
KubeDNS is running at https://192.168.100.198:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
root@master:~# kubectl get nodes -o wide
NAME       STATUS   ROLES    AGE   VERSION   INTERNAL-IP       EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
master     Ready    master   74m   v1.18.5   192.168.100.198   <none>        Ubuntu 18.04.4 LTS   4.15.0-108-generic   docker://19.3.12
nodesatu   Ready    <none>   41m   v1.18.5   192.168.100.199   <none>        Ubuntu 18.04.4 LTS   4.15.0-108-generic   docker://19.3.12


root@master:~# kubectl -n kube-system get po -o wide
NAME                             READY   STATUS    RESTARTS   AGE   IP                NODE       NOMINATED NODE   READINESS GATES
coredns-66bff467f8-jghgf         1/1     Running   0          77m   10.244.0.3        master     <none>           <none>
coredns-66bff467f8-zkcdd         1/1     Running   0          77m   10.244.0.2        master     <none>           <none>
etcd-master                      1/1     Running   0          77m   192.168.100.198   master     <none>           <none>
kube-apiserver-master            1/1     Running   0          77m   192.168.100.198   master     <none>           <none>
kube-controller-manager-master   1/1     Running   0          77m   192.168.100.198   master     <none>           <none>
kube-flannel-ds-amd64-rb4wc      1/1     Running   0          69m   192.168.100.198   master     <none>           <none>
kube-flannel-ds-amd64-rmdw7      1/1     Running   1          44m   192.168.100.199   nodesatu   <none>           <none>
kube-proxy-hgkn9                 1/1     Running   0          44m   192.168.100.199   nodesatu   <none>           <none>
kube-proxy-lc5ps                 1/1     Running   0          77m   192.168.100.198   master     <none>           <none>
kube-scheduler-master            1/1     Running   0          77m   192.168.100.198   master     <none>           <none>

Dari data diatas, semua pod running. Aku asumsikan semua beres.

C. Pasca Setup Cluster

Setelah cluster berjalan, aku lanjutkan install aplikasi di atas cluster kubernetes. Kita kembali ke master, dan install-install dari master.

C.1 Install Helm3

Helm v3 ini akan membatu banget untuk install-install aplikasi diatas cluster kubernetes.

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3

chmod 700 get_helm.sh

./get_helm.sh

helm repo add stable https://kubernetes-charts.storage.googleapis.com/

helm repo update

C.2 Install metallb

Ikuti https://metallb.universe.tf/installation/

Krn cluster yg kubuat di vm local, dan gak punya dedicated LB, maka aku butuh metallb untuk assign IP ke service bertipe LoadBalancer

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/namespace.yaml

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/metallb.yaml

kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"

Bikin configmap untuk metallb, silakan merujuk ke https://metallb.universe.tf/configuration/#layer-2-configuration

nano metallb-configmap.yaml
# isinya sbb:
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.100.30-192.168.100.40

Yang perlu diperhatikan adalah 192.168.100.30-192.168.100.40, IP ini aku kasih range dari router, ya krn ini VM di local. Kalau di cloud/datacenter range ip tersebut harus ip public supaya service bisa dijangkau dari luar.

kubectl apply -f metallb-configmap.yaml

Verifikasi hasil install metallb

root@master:~# kubectl -n metallb-system get po -o wide
NAME                          READY   STATUS    RESTARTS   AGE   IP                NODE       NOMINATED NODE   READINESS GATES
controller-57f648cb96-b5llj   1/1     Running   0          33m   10.244.1.2        nodesatu   <none>           <none>
speaker-csgx5                 1/1     Running   0          33m   192.168.100.199   nodesatu   <none>           <none>
speaker-mrv7x                 1/1     Running   0          33m   192.168.100.198   master     <none>           <none>

C.3 Install Nginx Ingress Controller

Ikuti https://kubernetes.github.io/ingress-nginx/deploy/#using-helm

Ingress controller ini membantu untuk expose service ke luar cluster, misalnya expose web service via domain.

Nginx Ingress Controller hanyalah satu dari sekian banyak Ingress Controller. Selain nginx ada Traefik, HAProxy, dll. Aku sendiri selama ini pakai Traefik v1.7.24 di production. Dah kali ini aku akan pakai Nginx

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm install my-release ingress-nginx/ingress-nginx

Perintah diatas akan install nginx di namespace default, kita check:

@master:~# kubectl -n default get po,svc -o wide
NAME                                                       READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
pod/my-release-ingress-nginx-controller-7fcf9d598b-k68gs   1/1     Running   0          34m   10.244.1.4   nodesatu   <none>           <none>

NAME                                                    TYPE           CLUSTER-IP             PORT(S)                      AGE   SELECTOR
service/kubernetes                                      ClusterIP      10.96.0.1        <none>           443/TCP                      85m   <none>
service/my-release-ingress-nginx-controller             LoadBalancer   10.110.208.65    192.168.100.30   80:30509/TCP,443:30954/TCP   34m   app.kubernetes.io/component=controller,app.kubernetes.io/instance=my-release,app.kubernetes.io/name=ingress-nginx
service/my-release-ingress-nginx-controller-admission   ClusterIP      10.104.254.133   <none>           443/TCP                      34m   app.kubernetes.io/component=controller,app.kubernetes.io/instance=my-release,app.kubernetes.io/name=ingress-nginx

Berkat bantuan metallb, EXTERNAL-IP dari service/my-release-ingress-nginx-controller tidak pending. Dalam hal ini dia mendapatkan IP 192.168.100.30. IP tsb kita asumsikan IP public yg mana akan dimasukan ke DNS A Record

Untuk memastikan ingress controllernya berfungsi, aku akan test dari luar cluster yakni dari PC.

➜  ~ curl 192.168.100.30
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.19.0</center>
</body>
</html>

Kita coba deploy aplikasi whoareyou

Isi dari whoareyou.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: whoareyou-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: whoareyou
  template:
    metadata:
      labels:
        app: whoareyou
    spec:
      containers:
      - name: whoareyou-container
        image: containous/whoami
---
apiVersion: v1
kind: Service
metadata:
  name: whoareyou-service
spec:
  ports:
  - name: http
    targetPort: 80
    port: 80
  selector:
    app: whoareyou
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: whoareyou-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: whoareyou.kvm.local
    http:
      paths:
      - path: /
        backend:
          serviceName: whoareyou-service
          servicePort: http

kubectl apply -f whoareyou.yaml

Verifikasi hasil deploy:

root@master:~# kubectl -n default get po,svc,ingress
NAME                                                       READY   STATUS    RESTARTS   AGE
pod/my-release-ingress-nginx-controller-7fcf9d598b-k68gs   1/1     Running   0          45m
pod/whoareyou-deployment-74bff9c468-mlnjd                  1/1     Running   0          110s
pod/whoareyou-deployment-74bff9c468-p5bkp                  1/1     Running   0          110s

NAME                                                    TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)                      AGE
service/kubernetes                                      ClusterIP      10.96.0.1        <none>           443/TCP                      97m
service/my-release-ingress-nginx-controller             LoadBalancer   10.110.208.65    192.168.100.30   80:30509/TCP,443:30954/TCP   45m
service/my-release-ingress-nginx-controller-admission   ClusterIP      10.104.254.133   <none>           443/TCP                      45m
service/whoareyou-service                               ClusterIP      10.108.79.85     <none>           80/TCP                       110s

NAME                                   CLASS    HOSTS                 ADDRESS          PORTS   AGE
ingress.extensions/whoareyou-ingress   <none>   whoareyou.kvm.local   192.168.100.30   80      110s

Kita test hasil deploynya. Anggaplah whoareyou.kvm.local nama domain yg kita set dan 192.168.100.30 adalah public ip. Krn ini di local, cara testnya bisa kita masukin ke /etc/hosts komputer di luar cluster. Atau pakai curl dengan param header host. Disini aku test pakai curl

➜  ~ curl -H "Host: whoareyou.kvm.local" 192.168.100.30
Hostname: whoareyou-deployment-74bff9c468-mlnjd
IP: 127.0.0.1
IP: 10.244.1.7
RemoteAddr: 10.244.1.4:33058
GET / HTTP/1.1
Host: whoareyou.kvm.local
User-Agent: curl/7.70.0
Accept: */*
X-Forwarded-For: 10.244.1.1
X-Forwarded-Host: whoareyou.kvm.local
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Real-Ip: 10.244.1.1
X-Request-Id: bce405ea511051351247e01746104180
X-Scheme: http

Sip, sukses. Yg keluar adalah output dari pod whoareyou.

Bagaimana kalau kita pakai nama domain yg lain?

➜  ~ curl -H "Host: sembarangan.com" 192.168.100.30
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.19.0</center>
</body>
</html>

Outputnya adalah 404, karena ingress rule untuk domain sembarangan.com tidak ada.

Kesimpulannya adalah, kubeadm ini sangat memudahkan kita untuk setup kubernetes cluster. Banyak magic-nya. Aku senang dengan adanya kubeadm

Setup Kubernetes Cluster Ubuntu 18.04 Pakai Kubeadm