Ini adalah catatan bagaimana aku setup kubernetes cluster pakai kubeadm. Aku setup pakai VM (KVM) 1 master dan 1 node worker. Aku pakai vm ubuntu 18.04. Selama ini aku setup kubernetes cluster di debian 10. Sekarang belajar setup di ubuntu 18.04. Meskipun ubuntu berbasis debian, tapi ada beberapa yang berbeda ketika setup kubernetes cluster, yg paling kentara di Debian 10 adalah iptables-legacy.
Rangkuman singkatnya begini:
- install docker dan kubeadm di master
- setup kubeadm di master
- install docker dan kubeadm di node
- join node ke cluster pakai kubeadm
- Cluster selesai, sisanya pasang aplikasi sesua kebutuhan
Log install kubernetes di Ubuntu 18.04
A. Setup Master
- Node IP addr: 192.168.100.198
- Node Hostname: master
- OS: Ubuntu 18.04
- RAM: 2 GB
- CPU: 2 core
A.1 Install docker
Ikuti panduan https://docs.docker.com/engine/install/ubuntu/
apt-get update
apt-get install \
apt-transport-https \
ca-certificates \
curl \
gnupg-agent \
software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"
apt-get update
apt-get install docker-ce docker-ce-cli containerd.io
A.2 Ganti driver docker ke systemd, awalnya cgroupfs
# Set up the Docker daemon
cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF
mkdir -p /etc/systemd/system/docker.service.d
# Restart Docker
systemctl daemon-reload
systemctl restart docker
systemctl enable docker
A.3 Install kubeadm
Ikuti panduan https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
cat <<EOF | tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
apt-get update && apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF | tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl
A.4 Init cluster pakai kubeadm
Matikan swap
swapoff -a
Matikan swap pas reboot pakai crontab
@reboot swapoff -a
Disini mau pakai flannel untuk networknya, oleh karena itu pakai parameter --pod-network-cidr=10.244.0.0/16
kubeadm init --pod-network-cidr=10.244.0.0/16
root@master:/home/sumar# kubectl get po -o wide -A
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system coredns-66bff467f8-jghgf 0/1 Pending 0 4m22s <none> <none> <none> <none>
kube-system coredns-66bff467f8-zkcdd 0/1 Pending 0 4m22s <none> <none> <none> <none>
kube-system etcd-master 1/1 Running 0 4m32s 192.168.100.198 master <none> <none>
kube-system kube-apiserver-master 1/1 Running 0 4m32s 192.168.100.198 master <none> <none>
kube-system kube-controller-manager-master 1/1 Running 0 4m32s 192.168.100.198 master <none> <none>
kube-system kube-proxy-lc5ps 1/1 Running 0 4m22s 192.168.100.198 master <none> <none>
kube-system kube-scheduler-master 1/1 Running 0 4m32s 192.168.100.198 master <none> <none>
root@master:/home/sumar#
pod coredns status pending krn belum install flannel,
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
Setelah install flannel, pod coredns jadi jalan
root@master:/home/sumar# kubectl get po -o wide -A
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system coredns-66bff467f8-jghgf 1/1 Running 0 8m19s 10.244.0.3 master <none> <none>
kube-system coredns-66bff467f8-zkcdd 1/1 Running 0 8m19s 10.244.0.2 master <none> <none>
kube-system etcd-master 1/1 Running 0 8m29s 192.168.100.198 master <none> <none>
kube-system kube-apiserver-master 1/1 Running 0 8m29s 192.168.100.198 master <none> <none>
kube-system kube-controller-manager-master 1/1 Running 0 8m29s 192.168.100.198 master <none> <none>
kube-system kube-flannel-ds-amd64-rb4wc 1/1 Running 0 46s 192.168.100.198 master <none> <none>
kube-system kube-proxy-lc5ps 1/1 Running 0 8m19s 192.168.100.198 master <none> <none>
kube-system kube-scheduler-master 1/1 Running 0 8m29s 192.168.100.198 master <none> <none>
B. Setup Nodesatu
- Node IP addr: 192.168.100.199
- Node Hostname: nodesatu
- OS: Ubuntu 18.04
- RAM: 2GB
- CPU: 2 core
B.1 Install docker
Ikuti panduan https://docs.docker.com/engine/install/ubuntu/
apt-get update
apt-get install \
apt-transport-https \
ca-certificates \
curl \
gnupg-agent \
software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"
apt-get update
apt-get install docker-ce docker-ce-cli containerd.io
B.2 Ganti driver docker ke systemd, awalnya cgroupfs
# Set up the Docker daemon
cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF
mkdir -p /etc/systemd/system/docker.service.d
# Restart Docker
systemctl daemon-reload
systemctl restart docker
systemctl enable docker
B.3 Install kubeadm
Ikuti panduan https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
cat <<EOF | tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
apt-get update && apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF | tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl
B.4 Join nodesatu ke cluster
kubeadm join 192.168.100.198:6443 --token 98vdpp.r7tziuipa5ebwpg8 \
--discovery-token-ca-cert-hash sha256:d6d44f1588470756cd52f70b5576aa7ddd6b5b9a57559b025dbfa21ecee2c0eb
Langkah A dan B adalah langkah untuk setup cluster. Setelah setup master selesai dan setup nodesatu selesai, kita verifikasi:
root@master:~# kubectl cluster-info
Kubernetes master is running at https://192.168.100.198:6443
KubeDNS is running at https://192.168.100.198:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
root@master:~# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master Ready master 74m v1.18.5 192.168.100.198 <none> Ubuntu 18.04.4 LTS 4.15.0-108-generic docker://19.3.12
nodesatu Ready <none> 41m v1.18.5 192.168.100.199 <none> Ubuntu 18.04.4 LTS 4.15.0-108-generic docker://19.3.12
root@master:~# kubectl -n kube-system get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-66bff467f8-jghgf 1/1 Running 0 77m 10.244.0.3 master <none> <none>
coredns-66bff467f8-zkcdd 1/1 Running 0 77m 10.244.0.2 master <none> <none>
etcd-master 1/1 Running 0 77m 192.168.100.198 master <none> <none>
kube-apiserver-master 1/1 Running 0 77m 192.168.100.198 master <none> <none>
kube-controller-manager-master 1/1 Running 0 77m 192.168.100.198 master <none> <none>
kube-flannel-ds-amd64-rb4wc 1/1 Running 0 69m 192.168.100.198 master <none> <none>
kube-flannel-ds-amd64-rmdw7 1/1 Running 1 44m 192.168.100.199 nodesatu <none> <none>
kube-proxy-hgkn9 1/1 Running 0 44m 192.168.100.199 nodesatu <none> <none>
kube-proxy-lc5ps 1/1 Running 0 77m 192.168.100.198 master <none> <none>
kube-scheduler-master 1/1 Running 0 77m 192.168.100.198 master <none> <none>
Dari data diatas, semua pod running. Aku asumsikan semua beres.
C. Pasca Setup Cluster
Setelah cluster berjalan, aku lanjutkan install aplikasi di atas cluster kubernetes. Kita kembali ke master, dan install-install dari master.
C.1 Install Helm3
Helm v3 ini akan membatu banget untuk install-install aplikasi diatas cluster kubernetes.
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh
helm repo add stable https://kubernetes-charts.storage.googleapis.com/
helm repo update
C.2 Install metallb
Ikuti https://metallb.universe.tf/installation/
Krn cluster yg kubuat di vm local, dan gak punya dedicated LB, maka aku butuh metallb untuk assign IP ke service bertipe LoadBalancer
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/metallb.yaml
kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
Bikin configmap untuk metallb, silakan merujuk ke https://metallb.universe.tf/configuration/#layer-2-configuration
nano metallb-configmap.yaml
# isinya sbb:
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: default
protocol: layer2
addresses:
- 192.168.100.30-192.168.100.40
Yang perlu diperhatikan adalah 192.168.100.30-192.168.100.40
, IP ini aku kasih range dari router, ya krn ini VM di local. Kalau di cloud/datacenter range ip tersebut harus ip public supaya service bisa dijangkau dari luar.
kubectl apply -f metallb-configmap.yaml
Verifikasi hasil install metallb
root@master:~# kubectl -n metallb-system get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
controller-57f648cb96-b5llj 1/1 Running 0 33m 10.244.1.2 nodesatu <none> <none>
speaker-csgx5 1/1 Running 0 33m 192.168.100.199 nodesatu <none> <none>
speaker-mrv7x 1/1 Running 0 33m 192.168.100.198 master <none> <none>
C.3 Install Nginx Ingress Controller
Ikuti https://kubernetes.github.io/ingress-nginx/deploy/#using-helm
Ingress controller ini membantu untuk expose service ke luar cluster, misalnya expose web service via domain.
Nginx Ingress Controller hanyalah satu dari sekian banyak Ingress Controller. Selain nginx ada Traefik, HAProxy, dll. Aku sendiri selama ini pakai Traefik v1.7.24 di production. Dah kali ini aku akan pakai Nginx
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm install my-release ingress-nginx/ingress-nginx
Perintah diatas akan install nginx di namespace default, kita check:
@master:~# kubectl -n default get po,svc -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/my-release-ingress-nginx-controller-7fcf9d598b-k68gs 1/1 Running 0 34m 10.244.1.4 nodesatu <none> <none>
NAME TYPE CLUSTER-IP PORT(S) AGE SELECTOR
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 85m <none>
service/my-release-ingress-nginx-controller LoadBalancer 10.110.208.65 192.168.100.30 80:30509/TCP,443:30954/TCP 34m app.kubernetes.io/component=controller,app.kubernetes.io/instance=my-release,app.kubernetes.io/name=ingress-nginx
service/my-release-ingress-nginx-controller-admission ClusterIP 10.104.254.133 <none> 443/TCP 34m app.kubernetes.io/component=controller,app.kubernetes.io/instance=my-release,app.kubernetes.io/name=ingress-nginx
Berkat bantuan metallb, EXTERNAL-IP dari service/my-release-ingress-nginx-controller
tidak pending. Dalam hal ini dia mendapatkan IP 192.168.100.30. IP tsb kita asumsikan IP public yg mana akan dimasukan ke DNS A Record
Untuk memastikan ingress controllernya berfungsi, aku akan test dari luar cluster yakni dari PC.
➜ ~ curl 192.168.100.30
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.19.0</center>
</body>
</html>
Kita coba deploy aplikasi whoareyou
Isi dari whoareyou.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: whoareyou-deployment
spec:
replicas: 2
selector:
matchLabels:
app: whoareyou
template:
metadata:
labels:
app: whoareyou
spec:
containers:
- name: whoareyou-container
image: containous/whoami
---
apiVersion: v1
kind: Service
metadata:
name: whoareyou-service
spec:
ports:
- name: http
targetPort: 80
port: 80
selector:
app: whoareyou
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: whoareyou-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- host: whoareyou.kvm.local
http:
paths:
- path: /
backend:
serviceName: whoareyou-service
servicePort: http
kubectl apply -f whoareyou.yaml
Verifikasi hasil deploy:
root@master:~# kubectl -n default get po,svc,ingress
NAME READY STATUS RESTARTS AGE
pod/my-release-ingress-nginx-controller-7fcf9d598b-k68gs 1/1 Running 0 45m
pod/whoareyou-deployment-74bff9c468-mlnjd 1/1 Running 0 110s
pod/whoareyou-deployment-74bff9c468-p5bkp 1/1 Running 0 110s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 97m
service/my-release-ingress-nginx-controller LoadBalancer 10.110.208.65 192.168.100.30 80:30509/TCP,443:30954/TCP 45m
service/my-release-ingress-nginx-controller-admission ClusterIP 10.104.254.133 <none> 443/TCP 45m
service/whoareyou-service ClusterIP 10.108.79.85 <none> 80/TCP 110s
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress.extensions/whoareyou-ingress <none> whoareyou.kvm.local 192.168.100.30 80 110s
Kita test hasil deploynya. Anggaplah whoareyou.kvm.local
nama domain yg kita set dan 192.168.100.30
adalah public ip. Krn ini di local, cara testnya bisa kita masukin ke /etc/hosts komputer di luar cluster. Atau pakai curl dengan param header host. Disini aku test pakai curl
➜ ~ curl -H "Host: whoareyou.kvm.local" 192.168.100.30
Hostname: whoareyou-deployment-74bff9c468-mlnjd
IP: 127.0.0.1
IP: 10.244.1.7
RemoteAddr: 10.244.1.4:33058
GET / HTTP/1.1
Host: whoareyou.kvm.local
User-Agent: curl/7.70.0
Accept: */*
X-Forwarded-For: 10.244.1.1
X-Forwarded-Host: whoareyou.kvm.local
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Real-Ip: 10.244.1.1
X-Request-Id: bce405ea511051351247e01746104180
X-Scheme: http
Sip, sukses. Yg keluar adalah output dari pod whoareyou.
Bagaimana kalau kita pakai nama domain yg lain?
➜ ~ curl -H "Host: sembarangan.com" 192.168.100.30
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.19.0</center>
</body>
</html>
Outputnya adalah 404, karena ingress rule untuk domain sembarangan.com tidak ada.
Kesimpulannya adalah, kubeadm ini sangat memudahkan kita untuk setup kubernetes cluster. Banyak magic-nya. Aku senang dengan adanya kubeadm