Backup and Restore etcd

etcd is the backend data storage solution for your kubernetes cluster. As such, all your kubernetes objects, applications, and configurations are stored in etcd.

Therefore, you will likely want to be able to backup your cluster's data by backing up etcd.

 

Backup etcd

You can backup etcd data using the etcd command line tool, etcdctl.

Use the etcdctl snapshot save command to backup the data.

$ ETCDCTL_API=3 etcdctl --endpoints $ENDPOINT snapshot save <filename>

Restoring etcd

You can restore etcd data from a backup using the etcdctl snapshot restore command.

You will need to supply some additional parameters, as the restore operation creates a new logical cluster.

$ ETCDCTL_API=3 etcdctl snapshot restore <filename>

LAB

Backups are an important part of any resilient system. Kubernetes is no exception. In this lab, you will have the opportunity to practice your skills by backing up and restoring a Kubernetes cluster state stored in etcd. This will help you get comfortable with the setps involved in backing up Kubernetes data.

1.  Look up the value for the key cluster.name in the etcd cluster:

cloud_user@etcd-1:~$ ETCDCTL_API=3 etcdctl get cluster.name \
> --endpoints=https://10.0.1.101:2379 \
> --cacert=/home/cloud_user/etcd-certs/etcd-ca.pem \
> --cert=/home/cloud_user/etcd-certs/etcd-server.crt \
> --key=/home/cloud_user/etcd-certs/etcd-server.key

cluster.name
beebox
cloud_user@etcd-1:~$

2. Back up etcd using etcdctl and the provided etcd certificates:

cloud_user@etcd-1:~$ ETCDCTL_API=3 etcdctl snapshot save /home/cloud_user/etcd_backup.db \
>   --endpoints=https://10.0.1.101:2379 \
>   --cacert=/home/cloud_user/etcd-certs/etcd-ca.pem \
>   --cert=/home/cloud_user/etcd-certs/etcd-server.crt \
>   --key=/home/cloud_user/etcd-certs/etcd-server.key

{"level":"info","ts":1663201128.3303864,"caller":"snapshot/v3_snapshot.go:119","msg":"created temporary db file","path":"/home/cloud_user/etcd_backup.db.part"}
{"level":"info","ts":"2022-09-15T00:18:48.337Z","caller":"clientv3/maintenance.go:200","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":1663201128.3376715,"caller":"snapshot/v3_snapshot.go:127","msg":"fetching snapshot","endpoint":"https://10.0.1.101:2379"}
{"level":"info","ts":"2022-09-15T00:18:48.340Z","caller":"clientv3/maintenance.go:208","msg":"completed snapshot read; closing"}
{"level":"info","ts":1663201128.3435297,"caller":"snapshot/v3_snapshot.go:142","msg":"fetched snapshot","endpoint":"https://10.0.1.101:2379","size":"20 kB","took":0.013078392}
{"level":"info","ts":1663201128.3437078,"caller":"snapshot/v3_snapshot.go:152","msg":"saved","path":"/home/cloud_user/etcd_backup.db"}
Snapshot saved at /home/cloud_user/etcd_backup.db
cloud_user@etcd-1:~$

백업파일이 생성되었다.

cloud_user@etcd-1:~$ ls
etcd-certs  etcd_backup.db
cloud_user@etcd-1:~$

3. Reset etcd by removing all existing etcd data: (모든 etcd 데이터가 날라감)

cloud_user@etcd-1:~$ sudo systemctl stop etcd
[sudo] password for cloud_user: 
cloud_user@etcd-1:~$ sudo rm -rf /var/lib/etcd
cloud_user@etcd-1:~$

Restore the etcd Data from the Backup

4. Restore the etcd data from the backup (this command spins up a temporary etcd cluster, saving the data from the backup file to a new data directory in the same location where the previous data directory was):

cloud_user@etcd-1:~$ sudo ETCDCTL_API=3 etcdctl snapshot restore /home/cloud_user/etcd_backup.db \
>   --initial-cluster etcd-restore=https://10.0.1.101:2380 \
>   --initial-advertise-peer-urls https://10.0.1.101:2380 \
>   --name etcd-restore \
>   --data-dir /var/lib/etcd

{"level":"info","ts":1663201480.2564209,"caller":"snapshot/v3_snapshot.go:296","msg":"restoring snapshot","path":"/home/cloud_user/etcd_backup.db","wal-dir":"/var/lib/etcd/member/wal","data-dir":"/var/lib/etcd","snap-dir":"/var/lib/etcd/member/snap"}
{"level":"info","ts":1663201480.267,"caller":"membership/cluster.go:392","msg":"added member","cluster-id":"be98953e0e8f788","local-member-id":"0","added-peer-id":"d961d83f6a817e88","added-peer-peer-urls":["https://10.0.1.101:2380"]}
{"level":"info","ts":1663201480.2782035,"caller":"snapshot/v3_snapshot.go:309","msg":"restored snapshot","path":"/home/cloud_user/etcd_backup.db","wal-dir":"/var/lib/etcd/member/wal","data-dir":"/var/lib/etcd","snap-dir":"/var/lib/etcd/member/snap"}
cloud_user@etcd-1:~$

cloud_user@etcd-1:~$ sudo ls -l /var/lib/etcd
total 4
drwx------ 4 root root 4096 Sep 15 00:24 member

cloud_user@etcd-1:~$ sudo chown -R etcd:etcd /var/lib/etcd
cloud_user@etcd-1:~$ sudo ls -l /var/lib/etcd
total 4
drwx------ 4 etcd etcd 4096 Sep 15 00:24 member

cloud_user@etcd-1:~$ sudo systemctl start etcd
cloud_user@etcd-1:~

백업의 성공여부 확인

cloud_user@etcd-1:~$ ETCDCTL_API=3 etcdctl get cluster.name \
>   --endpoints=https://10.0.1.101:2379 \
>   --cacert=/home/cloud_user/etcd-certs/etcd-ca.pem \
>   --cert=/home/cloud_user/etcd-certs/etcd-server.crt \
>   --key=/home/cloud_user/etcd-certs/etcd-server.key
cluster.name
beebox
cloud_user@etcd-1:~$