Kubernetes生产实践之Redis-cluster

方案一 自定义yaml文件安装redis cluster

背景

在Kubernetes中部署Redis集群面临挑战，因为每个Redis实例都依赖于一个配置文件，该文件可以跟踪其他集群实例及其角色。为此，需要结合使用Kubernetes StatefulSets和PersistentVolumes实现。

redis cluster架构示意图：

创建StatefulSet yaml文件

---
apiVersion:v1
kind:ConfigMap
metadata:
name:redis-cluster
data:
update.sh:|
#!/bin/sh
REDIS_NODES="/data/nodes.conf"
sed-i-e"/myself/s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/${POD_IP}/"${REDIS_NODES}
exec"$@"
redis.conf:|+
bind0.0.0.0
cluster-enabledyes
cluster-require-full-coverageno
cluster-node-timeout30000
cluster-config-file/data/nodes.conf
cluster-migration-barrier1
appendonlyyes
protected-modeno
---
apiVersion:apps.kruise.io/v1beta1
#apiVersion:apps/v1
kind:StatefulSet
metadata:
name:redis-cluster
spec:
serviceName:redis-cluster
replicas:6
selector:
matchLabels:
app:redis-cluster
template:
metadata:
labels:
app:redis-cluster
spec:
containers:
-name:redis
image:redis:6.2.1-alpine
ports:
-containerPort:6379
name:client
-containerPort:16379
name:gossip
command:["/conf/update.sh","redis-server","/conf/redis.conf"]
env:
-name:POD_IP
valueFrom:
fieldRef:
fieldPath:status.podIP
volumeMounts:
-name:conf
mountPath:/conf
readOnly:false
-name:data
mountPath:/data
readOnly:false
volumes:
-name:conf
configMap:
name:redis-cluster
defaultMode:0755
volumeClaimTemplates:
-metadata:
name:data
spec:
accessModes:["ReadWriteOnce"]
resources:
requests:
storage:100Gi
storageClassName:rbd

注释：

cluster-migration-barrier那些分配后仍然剩余migrationbarrier个从节点的主节点才会触发节点分配，而不是分配前有migrationbarrier个从节点的主节点就会触发节点分配，默认是1，生产环境建议维持默认值
protected-modeno参数是为了禁止外网访问redis，如果启用了，则只能够通过lookbackip（127.0.0.1）访问Redis，如果外网访问redis，会报出异常。
apiVersion:apps.kruise.io/v1beta1控制器这里使用kruise提供的AdvancedStatefulSet，如果集群没有安装kruise，可以使用apps/v1

安装redis-cluster

[root@qd01-stop-k8s-master001redis]#kubectlapply-finstall-redis.yaml
configmap/redis-clustercreated
statefulset.apps.kruise.io/redis-clustercreated

[root@qd01-stop-k8s-master001redis]#kubectlgetpo-nop
NAMEREADYSTATUSRESTARTSAGE
redis-cluster-01/1Running03m26s
redis-cluster-11/1Running03m14s
redis-cluster-21/1Running02m54s
redis-cluster-31/1Running02m23s
redis-cluster-41/1Running02m14s
redis-cluster-51/1Running0114s

创建redis-cluster service

---
apiVersion:v1
kind:Service
metadata:
name:redis-cluster
namespace:op
spec:
type:ClusterIP
ports:
-port:6379
targetPort:6379
name:client
-port:16379
targetPort:16379
name:gossip
selector:
app:redis-cluster

[root@qd01-stop-k8s-master001redis]#kubectlapply-fredis-svc.yml
service/redis-clustercreated
[root@qd01-stop-k8s-master001redis]#kubectlgetsvc-nop
NAMETYPECLUSTER-IPEXTERNAL-IPPORT(S)AGE
redis-clusterClusterIP10.97.197.224<none>6379/TCP,16379/TCP9s

测试能后连通
[root@qd01-stop-k8s-master001redis]#telnet10.97.197.2246379
Trying10.97.197.224...
Connectedto10.97.197.224.
Escapecharacteris'^]'.

初始化redis-cluster

执行如下命令，获取到pod IP，然后使用redis-cli –cluster创建集群

[root@qd01-stop-k8s-master001redis]#kubectl-nopexec-itredis-cluster-0--redis-cli--clustercreate--cluster-replicas1$(kubectl-nopgetpods-lapp=redis-cluster-ojsonpath='{range.items[*]}{.status.podIP}:6379{end}')
>>>Performinghashslotsallocationon6nodes...
Master[0]->Slots0-5460
Master[1]->Slots5461-10922
Master[2]->Slots10923-16383
Addingreplica100.88.43.67:6379to100.64.147.152:6379
Addingreplica100.113.170.5:6379to100.98.174.217:6379
Addingreplica100.64.147.153:6379to100.80.158.227:6379
M:b47b27a3dbddf3fc1370cbe14ae753f4fce20b04100.64.147.152:6379
slots:[0-5460](5461slots)master
M:09543217c903350e963fc4fdf4acb73f8a1b7f8b100.98.174.217:6379
slots:[5461-10922](5462slots)master
M:5389ace495b68eeac85370d6783648dff68f2fb6100.80.158.227:6379
slots:[10923-16383](5461slots)master
S:b1f39714c006ae55b12b18e6537303d7a00e1704100.64.147.153:6379
replicates5389ace495b68eeac85370d6783648dff68f2fb6
S:0113f4668ec2f3ca2e9470c44bd5faab532b0936100.88.43.67:6379
replicatesb47b27a3dbddf3fc1370cbe14ae753f4fce20b04
S:e1e2f18ae66c79f1943390beabb59613abbad38a100.113.170.5:6379
replicates09543217c903350e963fc4fdf4acb73f8a1b7f8b
CanIsettheaboveconfiguration?(type'yes'toaccept):yes
>>>Nodesconfigurationupdated
>>>Assignadifferentconfigepochtoeachnode
>>>SendingCLUSTERMEETmessagestojointhecluster
Waitingfortheclustertojoin
..
>>>PerformingClusterCheck(usingnode100.64.147.152:6379)
M:b47b27a3dbddf3fc1370cbe14ae753f4fce20b04100.64.147.152:6379
slots:[0-5460](5461slots)master
1additionalreplica(s)
S:0113f4668ec2f3ca2e9470c44bd5faab532b0936100.88.43.67:6379
slots:(0slots)slave
replicatesb47b27a3dbddf3fc1370cbe14ae753f4fce20b04
M:09543217c903350e963fc4fdf4acb73f8a1b7f8b100.98.174.217:6379
slots:[5461-10922](5462slots)master
1additionalreplica(s)
M:5389ace495b68eeac85370d6783648dff68f2fb6100.80.158.227:6379
slots:[10923-16383](5461slots)master
1additionalreplica(s)
S:e1e2f18ae66c79f1943390beabb59613abbad38a100.113.170.5:6379
slots:(0slots)slave
replicates09543217c903350e963fc4fdf4acb73f8a1b7f8b
S:b1f39714c006ae55b12b18e6537303d7a00e1704100.64.147.153:6379
slots:(0slots)slave
replicates5389ace495b68eeac85370d6783648dff68f2fb6
[OK]Allnodesagreeaboutslotsconfiguration.
>>>Checkforopenslots...
>>>Checkslotscoverage...
[OK]All16384slotscovered.

验证集群信息

[root@qd01-stop-k8s-master001redis]#kubectl-nopexec-itredis-cluster-0--redis-cliclusterinfo
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:178
cluster_stats_messages_pong_sent:181
cluster_stats_messages_sent:359
cluster_stats_messages_ping_received:176
cluster_stats_messages_pong_received:178
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:359

[root@qd01-stop-k8s-master001redis]#kubectl-nopexec-itredis-cluster-0--redis-cliclusternodes
0113f4668ec2f3ca2e9470c44bd5faab532b0936100.88.43.67:6379@16379slaveb47b27a3dbddf3fc1370cbe14ae753f4fce20b04016153483111561connected
09543217c903350e963fc4fdf4acb73f8a1b7f8b100.98.174.217:6379@16379master-016153483141622connected5461-10922
b47b27a3dbddf3fc1370cbe14ae753f4fce20b04100.64.147.152:6379@16379myself,master-016153483120001connected0-5460
5389ace495b68eeac85370d6783648dff68f2fb6100.80.158.227:6379@16379master-016153483120003connected10923-16383
e1e2f18ae66c79f1943390beabb59613abbad38a100.113.170.5:6379@16379slave09543217c903350e963fc4fdf4acb73f8a1b7f8b016153483131602connected
b1f39714c006ae55b12b18e6537303d7a00e1704100.64.147.153:6379@16379slave5389ace495b68eeac85370d6783648dff68f2fb6016153483121583connected

从输出可以看到，集群总共6个节点，三主三从

方案二 使用kubeDB安装redis

安装kubeDB

1、安装KubeDB

获取AppsCode License

https://license-issuer.appscode.com/

下载KubeDB charts

https://github.com/appscode/charts/tree/master/stable/kubedb-community

[root@qd01-stop-k8s-master001kubedb-community]#unzipkubedb-community-v0.16.2.tgz
[root@qd01-stop-k8s-master001kubedb-community]#cdkubedb-community
[root@qd01-stop-k8s-master001kubedb-community]#ls-al
total96
drwxr-xr-x4rootroot158Mar1015:26.
drwxr-xr-x3rootroot66Mar1015:24..
-rw-r--r--1rootroot351Feb1609:55Chart.yaml
drwxr-xr-x2rootroot28Mar1015:24ci
-rw-r--r--1rootroot493Feb1609:55doc.yaml
-rw-r--r--1rootroot353Feb1609:55.helmignore
-rw-r--r--1rootroot24422Feb1609:55README.md
drwxr-xr-x2rootroot4096Mar1015:24templates
-rw-r--r--1rootroot47437Feb1609:55values.openapiv3_schema.yaml
-rw-r--r--1rootroot5230Feb1609:55values.yaml

修改values.yaml，把License文件放到kubedb-community目录下

2、使用helm安装

[root@qd01-stop-k8s-master001kubedb-community]#helminstallkubedb-community--namespacekube-system--set-filelicense=./kubedb-community-license.txt-fvalues.yaml.
NAME:kubedb-community
LASTDEPLOYED:WedMar1015:38:592021
NAMESPACE:kube-system
STATUS:deployed
REVISION:1
TESTSUITE:None
NOTES:
ToverifythatKubeDBhasstarted,run:
kubectlgetdeployment--namespacekube-system-l"app.kubernetes.io/name=kubedb-community,app.kubernetes.io/instance=kubedb-community"
Nowinstall/upgradeappscode/kubedb-catalogchart.
Toinstall,run:
helminstallkubedb-catalogappscode/kubedb-catalog--versionv0.16.2--namespacekube-system
Toupgrade,run:
helmupgradekubedb-catalogappscode/kubedb-catalog--versionv0.16.2--namespacekube-system

执行如下命令查看是否安装完成
[root@qd01-stop-k8s-master001kubedb-community]#kubectlgetdeployment--namespacekube-system-l"app.kubernetes.io/name=kubedb-community,app.kubernetes.io/instance=kubedb-community"
NAMEREADYUP-TO-DATEAVAILABLEAGE
kubedb-community1/11138s

等待crds注册成功
[root@qd01-stop-k8s-master001kubedb-community]#kubectlgetcrds-lapp.kubernetes.io/name=kubedb-w
NAMECREATEDAT
elasticsearches.kubedb.com2021-03-10T07:39:42Z
elasticsearchversions.catalog.kubedb.com2021-03-10T07:39:45Z
etcds.kubedb.com2021-03-10T07:39:42Z
etcdversions.catalog.kubedb.com2021-03-10T07:39:45Z
memcacheds.kubedb.com2021-03-10T07:39:43Z
memcachedversions.catalog.kubedb.com2021-03-10T07:39:45Z
mongodbs.kubedb.com2021-03-10T07:39:43Z
mongodbversions.catalog.kubedb.com2021-03-10T07:39:45Z
mysqls.kubedb.com2021-03-10T07:39:43Z
mysqlversions.catalog.kubedb.com2021-03-10T07:39:46Z
perconaxtradbs.kubedb.com2021-03-10T07:39:43Z
perconaxtradbversions.catalog.kubedb.com2021-03-10T07:39:46Z
pgbouncers.kubedb.com2021-03-10T07:39:44Z
pgbouncerversions.catalog.kubedb.com2021-03-10T07:39:46Z
postgreses.kubedb.com2021-03-10T07:39:44Z
postgresversions.catalog.kubedb.com2021-03-10T07:39:46Z
proxysqls.kubedb.com2021-03-10T07:39:44Z
proxysqlversions.catalog.kubedb.com2021-03-10T07:39:46Z
redises.kubedb.com2021-03-10T07:39:45Z
redisversions.catalog.kubedb.com2021-03-10T07:39:46Z

3、安装KubeDB Catalog

同样，先下载

https://github.com/appscode/charts/tree/master/stable/kubedb-catalog

[root@qd01-stop-k8s-master001kubedb-catalog]#tar-zxfkubedb-catalog-v0.16.2.tgz
[root@qd01-stop-k8s-master001kubedb-catalog]#cdkubedb-catalog
[root@qd01-stop-k8s-master001kubedb-catalog]#ls-al
total24
drwxr-xr-x3rootroot148Mar1015:48.
drwxr-xr-x3rootroot28Mar1015:48..
-rw-r--r--1rootroot321Jan2620:08Chart.yaml
-rw-r--r--1rootroot467Jan2620:08doc.yaml
-rw-r--r--1rootroot353Jan2620:08.helmignore
-rw-r--r--1rootroot3195Jan2620:08README.md
drwxr-xr-x12rootroot188Mar1015:48templates
-rw-r--r--1rootroot744Jan2620:08values.openapiv3_schema.yaml
-rw-r--r--1rootroot1070Jan2620:08values.yaml

[root@qd01-stop-k8s-master001kubedb-catalog]#helminstallkubedb-catalog--namespacekube-system-fvalues.yaml.
NAME:kubedb-catalog
LASTDEPLOYED:WedMar1015:50:502021
NAMESPACE:kube-system
STATUS:deployed
REVISION:1
TESTSUITE:None

使用kubedb安装redis

1、先看下官方给的redis生命周期示意图

kubedb安装redis支持如下特性

FeaturesAvailability
Clustering✓
InstantBackup✗
ScheduledBackup✗
PersistentVolume✓
InitializeusingSnapshot✗
InitializeusingScript✗
CustomConfiguration✓
UsingCustomdockerimage✓
BuiltinPrometheusDiscovery✓
UsingPrometheusoperator✓

2、查看支持的版本

[root@qd01-stop-k8s-master001kubedb-catalog]#kubectlgetredisversions
NAMEVERSIONDB_IMAGEDEPRECATEDAGE
4.0.114.0.11kubedb/redis:4.0.1115m
4.0.6-v24.0.6kubedb/redis:4.0.6-v215m
5.0.3-v15.0.3kubedb/redis:5.0.3-v115m
6.0.66.0.6kubedb/redis:6.0.615m

3、编辑yaml安装文件

可以参照

https://github.com/kubedb/docs/blob/v2021.01.26/docs/examples/redis/clustering/demo-1.yaml

这里选择安装6.0.6这个版本，我的集群storageClassName: "rbd"，请根据实际修改

如果想自定义redis.conf,请参考

https://github.com/kubedb/docs/blob/v2021.01.26/docs/examples/redis/custom-config/redis-custom.yaml

apiVersion:kubedb.com/v1alpha2
kind:Redis
metadata:
name:redis-cluster
namespace:op
spec:
version:6.0.6
mode:Cluster
cluster:
master:3
replicas:1
storageType:Durable
storage:
resources:
requests:
storage:1Gi
storageClassName:"rbd"
accessModes:
-ReadWriteOnce

执行安装

[root@qd01-stop-k8s-master001kubedb-community]#kubectlapply-fredis-cluster.yaml
redis.kubedb.com/redis-clustercreated

安装完成，可以如下查看
[root@qd01-stop-k8s-master001kubedb-community]#kubectlgetrd,po-nop
NAMEVERSIONSTATUSAGE
redis.kubedb.com/redis-cluster6.0.6Provisioning6m55s

NAMEREADYSTATUSRESTARTSAGE
pod/redis-cluster-shard0-01/1Running06m54s
pod/redis-cluster-shard0-11/1Running06m18s
pod/redis-cluster-shard1-01/1Running05m38s
pod/redis-cluster-shard1-11/1Running05m1s
pod/redis-cluster-shard2-01/1Running04m30s
pod/redis-cluster-shard2-11/1Running04m8s

[root@qd01-stop-k8s-master001redis]#kubectlgetsvc-nop
NAMETYPECLUSTER-IPEXTERNAL-IPPORT(S)AGE
redis-clusterClusterIP10.97.197.224<none>6379/TCP5h16m
redis-cluster-podsClusterIPNone<none>6379/TCP17m

4、验证集群

[root@qd01-stop-k8s-master001kubedb-community]#kubectlgetpods-nop-ojsonpath='{range.items[*]}{.metadata.name}----------{.status.podIP}:6379{"\t\n"}{end}'|grepredis
redis-cluster-shard0-0----------100.64.147.156:6379
redis-cluster-shard0-1----------100.98.174.218:6379
redis-cluster-shard1-0----------100.126.252.204:6379
redis-cluster-shard1-1----------100.113.170.6:6379
redis-cluster-shard2-0----------100.107.55.69:6379
redis-cluster-shard2-1----------100.78.230.4:6379

[root@qd01-stop-k8s-master001redis]#kubectl-nopexec-itredis-cluster-shard0-0--redis-cliclusterinfo
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:3
cluster_my_epoch:1
cluster_stats_messages_ping_sent:864
cluster_stats_messages_pong_sent:882
cluster_stats_messages_sent:1746
cluster_stats_messages_ping_received:879
cluster_stats_messages_pong_received:864
cluster_stats_messages_meet_received:3
cluster_stats_messages_received:1746

[root@qd01-stop-k8s-master001redis]#kubectl-nopexec-itredis-cluster-shard0-0--redis-cliclusternodes
1895cb4b9c31b848666c61000e502f55a29a8255100.64.147.155:6379@16379master-016153651620082connected5461-10922
30bdbf2ca37001774498a9b935afbc1cd2ce389c100.126.252.203:6379@16379slave2c06092fafa99e0158e39e6237a04fed25be3550016153651630001connected
9b2cfbd5c1b417121d410141b6da9512ad29ce3c100.78.230.3:6379@16379slavee83446c368839c5fdccf5f70e3b1004eb67cb651016153651635123connected
2c06092fafa99e0158e39e6237a04fed25be3550100.82.197.130:6379@16379myself,master-016153651620001connected0-5460
1379d2b20f26ab13d53068d276ec5d988b7a0273100.64.122.197:6379@16379slave1895cb4b9c31b848666c61000e502f55a29a8255016153651630002connected
e83446c368839c5fdccf5f70e3b1004eb67cb651100.107.55.68:6379@16379master-016153651640143connected10923-16383