Kubernetes实践：CRD开发
发布于 2020-11-20 · 本文总共 10765 字 · 阅读大约需要
31 分钟
Custom Resource Definition
CRD是Kubernetes为提高可扩展性，让开发者去自定义资源（如Deployment，StatefulSet等）的一种方法；
Kubernetes Operator 是一种封装、部署和管理 Kubernetes 应用的方法。
Operator=CRD+Controller；
Controller：监听CRD实例（以及关联的资源）的CRUD事件，然后执行相应的业务逻辑；
历史
Kubernetes 1.7之后，提供了CRD（CustomResourceDefinitions）的自定义资源二次开发能力来扩展kubernetes API，通过此扩展，可以向kubernetes API中增加新类型，会比修改kubernetes的源代码或者是创建自定义的API server来的更加的简洁和容易，并且不会随着kuberntetes内核版本的升级，而出现需要代码重新merge的需要，以及兼容性方面的问题； 这一功能特性的提供大大提升了kubernetes的扩展能力；
实践
CRD
安装kubebuild
wget https://github.com/kubernetes-sigs/kubebuilder/releases/download/v2.3.1/kubebuilder_2.3.1_linux_arm64.tar.gz
Kubebuild
kubebuilder init –domain qiuwenqi.com –license apache2 –owner “qwq”
[root@qwq-test01 virtualmobilephone]# kubebuilder init --domain qiuwenqi.com --license apache2 --owner "qiuwenqi"
Writing scaffold for you to edit...
Get controller runtime:
$ go get sigs.k8s.io/controller-runtime@v0.5.0
Update go.mod:
$ go mod tidy
Running make:
$ make
/home/qiuwenqi/GO/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
go build -o bin/manager main.go
Next: define a resource with:
$ kubebuilder create api
kubebuilder create api –group infra –version v1 –kind MobilePhone
[root@qwq-test01 mobilephone]# kubebuilder create api --group infra --version v1 --kind MobilePhone
Create Resource [y/n]
invalid input "", should be [y/n]y
Create Controller [y/n]
y
Writing scaffold for you to edit...
api/v1/mobilephone_types.go
controllers/mobilephone_controller.go
Running make:
$ make
/home/qiuwenqi/GO/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
go: finding github.com/onsi/gomega v1.8.1
go: finding github.com/onsi/ginkgo v1.11.0
go: finding github.com/hpcloud/tail v1.0.0
go: finding golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7
go: finding gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7
go build -o bin/manager main.go
kubebuilder create api –group infra –version v1 –kind VirtualMobilePhone
[root@qwq-test01 virtualmobilephone]# kubebuilder create api --group infra --version v1 --kind VirtualMobilePhone
Create Resource [y/n]
y
Create Controller [y/n]
y
Writing scaffold for you to edit...
api/v1/virtualmobilephone_types.go
controllers/virtualmobilephone_controller.go
Running make:
$ make
/home/qiuwenqi/GO/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
go build -o bin/manager main.go
install kustomize
[root@qwq-test01 mobilephone]# go install sigs.k8s.io/kustomize
go: downloading github.com/go-openapi/spec v0.19.3
go: downloading github.com/emicklei/go-restful v2.9.5+incompatible
go: downloading github.com/ghodss/yaml v1.0.0
go: extracting github.com/ghodss/yaml v1.0.0
go: extracting github.com/emicklei/go-restful v2.9.5+incompatible
go: extracting github.com/go-openapi/spec v0.19.3
go: downloading github.com/go-openapi/swag v0.19.5
go: downloading github.com/go-openapi/jsonpointer v0.19.3
go: downloading github.com/go-openapi/jsonreference v0.19.3
go: extracting github.com/go-openapi/jsonpointer v0.19.3
go: extracting github.com/go-openapi/jsonreference v0.19.3
go: extracting github.com/go-openapi/swag v0.19.5
go: downloading github.com/PuerkitoBio/purell v1.1.1
go: downloading github.com/mailru/easyjson v0.7.0
go: extracting github.com/PuerkitoBio/purell v1.1.1
go: downloading github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578
go: extracting github.com/mailru/easyjson v0.7.0
go: extracting github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578
go: finding github.com/ghodss/yaml v1.0.0
go: finding github.com/go-openapi/spec v0.19.3
go: finding github.com/emicklei/go-restful v2.9.5+incompatible
go: finding github.com/go-openapi/jsonpointer v0.19.3
go: finding github.com/go-openapi/jsonreference v0.19.3
go: finding github.com/go-openapi/swag v0.19.5
go: finding github.com/mailru/easyjson v0.7.0
make install
[root@qwq-test01 mobilephone]# make install
/home/qiuwenqi/GO/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/mobilephones.infra.qiuwenqi.com created
make run
[root@qwq-test01 mobilephone]# make run
/home/qiuwenqi/GO/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
/home/qiuwenqi/GO/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
go run ./main.go
2020-11-20T14:34:32.307+0800 INFO controller-runtime.metrics metrics server is starting to listen {"addr": ":8080"}
2020-11-20T14:34:32.308+0800 INFO setup starting manager
2020-11-20T14:34:32.308+0800 INFO controller-runtime.manager starting metrics server {"path": "/metrics"}
2020-11-20T14:34:32.308+0800 INFO controller-runtime.controller Starting EventSource {"controller": "mobilephone", "source": "kind source: /, Kind="}
2020-11-20T14:34:32.409+0800 INFO controller-runtime.controller Starting Controller {"controller": "mobilephone"}
2020-11-20T14:34:32.409+0800 INFO controller-runtime.controller Starting workers {"controller": "mobilephone", "worker count": 1}
kubectl get crd
[root@qwq-test01 ~]# kubectl get crd
NAME CREATED AT
mobilephones.infra.qiuwenqi.com 2020-11-20T06:33:07Z
kubectl apply -f config/samples/infra_v1_mobilephone.yaml
[root@qwq-test01 mobilephone]# kubectl apply -f config/samples/infra_v1_mobilephone.yaml
mobilephone.infra.qiuwenqi.com/mobilephone-sample created
config/samples/infra_v1_mobilephone.yaml
[root@qwq-test01 mobilephone]# cat config/samples/infra_v1_mobilephone.yaml
apiVersion: infra.qiuwenqi.com/v1
kind: MobilePhone
metadata:
name: mobilephone-sample
spec:
# Add fields here
foo: bar
kubectl get mobilephone.infra.qiuwenqi.com
[root@qwq-test01 mobilephone]# kubectl get mobilephone.infra.qiuwenqi.com
NAME AGE
mobilephone-sample 3m13s
sample-controller
build
git clone https://github.com/kubernetes/sample-controller.git
cd sample-controller
go build -o sample-controller .
start
root@qwq-test04:~/go/src/sample-controller# ./sample-controller -kubeconfig=$HOME/.kube/config
I1124 17:46:25.449691 3457216 controller.go:115] Setting up event handlers
I1124 17:46:25.449844 3457216 controller.go:156] Starting Foo controller
I1124 17:46:25.449872 3457216 controller.go:159] Waiting for informer caches to sync
E1124 17:46:25.461746 3457216 reflector.go:138] pkg/mod/k8s.io/client-go@v0.0.0-20201114085741-77eda6a9395b/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Foo: failed to list *v1alpha1.Foo: the server could not find the requested resource (get foos.samplecontroller.k8s.io)
E1124 17:46:26.747439 3457216 reflector.go:138] pkg/mod/k8s.io/client-go@v0.0.0-20201114085741-77eda6a9395b/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Foo: failed to list *v1alpha1.Foo: the server could not find the requested resource (get foos.samplecontroller.k8s.io)
E1124 17:46:29.412360 3457216 reflector.go:138] pkg/mod/k8s.io/client-go@v0.0.0-20201114085741-77eda6a9395b/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Foo: failed to list *v1alpha1.Foo: the server could not find the requested resource (get foos.samplecontroller.k8s.io)
E1124 17:46:34.014518 3457216 reflector.go:138] pkg/mod/k8s.io/client-go@v0.0.0-20201114085741-77eda6a9395b/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Foo: failed to list *v1alpha1.Foo: the server could not find the requested resource (get foos.samplecontroller.k8s.io)
E1124 17:46:43.133875 3457216 reflector.go:138] pkg/mod/k8s.io/client-go@v0.0.0-20201114085741-77eda6a9395b/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Foo: failed to list *v1alpha1.Foo: the server could not find the requested resource (get foos.samplecontroller.k8s.io)
E1124 17:47:04.727122 3457216 reflector.go:138] pkg/mod/k8s.io/client-go@v0.0.0-20201114085741-77eda6a9395b/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Foo: failed to list *v1alpha1.Foo: the server could not find the requested resource (get foos.samplecontroller.k8s.io)
I1124 17:47:32.150103 3457216 controller.go:164] Starting workers
I1124 17:47:32.150153 3457216 controller.go:170] Started workers
I1124 17:48:07.041822 3457216 controller.go:228] Successfully synced 'default/example-foo'
I1124 17:48:07.042464 3457216 event.go:291] "Event occurred" object="default/example-foo" kind="Foo" apiVersion="samplecontroller.k8s.io/v1alpha1" type="Normal" reason="Synced" message="Foo synced successfully"
I1124 17:48:07.070607 3457216 controller.go:228] Successfully synced 'default/example-foo'
I1124 17:48:07.070678 3457216 event.go:291] "Event occurred" object="default/example-foo" kind="Foo" apiVersion="samplecontroller.k8s.io/v1alpha1" type="Normal" reason="Synced" message="Foo synced successfully"
I1124 17:48:08.081110 3457216 controller.go:228] Successfully synced 'default/example-foo'
I1124 17:48:08.081254 3457216 event.go:291] "Event occurred" object="default/example-foo" kind="Foo" apiVersion="samplecontroller.k8s.io/v1alpha1" type="Normal" reason="Synced" message="Foo synced successfully"
I1124 17:48:08.689052 3457216 controller.go:228] Successfully synced 'default/example-foo'
I1124 17:48:08.689141 3457216 event.go:291] "Event occurred" object="default/example-foo" kind="Foo" apiVersion="samplecontroller.k8s.io/v1alpha1" type="Normal" reason="Synced" message="Foo synced successfully"
I1124 17:48:32.194876 3457216 controller.go:228] Successfully synced 'default/example-foo'
I1124 17:48:32.194969 3457216 event.go:291] "Event occurred" object="default/example-foo" kind="Foo" apiVersion="samplecontroller.k8s.io/v1alpha1" type="Normal" reason="Synced" message="Foo synced successfully"
create crd
root@qwq-test04:~/go/src/sample-controller# kubectl create -f artifacts/examples/crd.yaml
customresourcedefinition.apiextensions.k8s.io/foos.samplecontroller.k8s.io created
root@qwq-test04:~/go/src/sample-controller#
root@qwq-test04:~/go/src/sample-controller# kubectl get crd
NAME CREATED AT
foos.samplecontroller.k8s.io 2020-11-24T09:47:13Z
virtualmobilephones.infra.qiuwenqi.com 2020-11-24T07:40:13Z
get crd
root@qwq-test04:~/go/src/sample-controller#
root@qwq-test04:~/go/src/sample-controller# kubectl create -f artifacts/examples/example-foo.yaml
foo.samplecontroller.k8s.io/example-foo created
root@qwq-test04:~/go/src/sample-controller# kubectl get foos.samplecontroller.k8s.io
NAME AGE
example-foo 2m13s
代码结构
clientset（自定义资源对象的客户端）；
listers（用来提供对于 GET/List 资源对象的请求提供只读缓存层）；
informers（List/Get 资源对象，还可以监听事件并触发回调函数。
FaaS
在当今云原生技术快速发展的时代，可能在不久的将来，Operator 模式可能也会被淘汰。 因为 Operator 也需要开发者关注一些部署的细节，让开发者真正只关注在自己的业务逻辑，“业务代码” 变成 “服务” 完全对开发者透明， 可能需要比 Kubernetes 更上层的框架 - FaaS框架。
FaaS(Function as a service):用户只要写自己的业务函数，向 Kubernetes 提交业务函数，FaaS 框架将业务函数变成 Deployment，变成 Pod，变成 Service。
Mem
知名分布式项目的Operator汇总：https://github.com/operator-framework/awesome-operators
refs
https://github.com/kubernetes/sample-controller
https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/
https://sq.163yun.com/blog/article/174980128954048512
https://github.com/kubernetes-sigs/kubebuilder
https://zhuanlan.zhihu.com/p/52367044
https://blog.csdn.net/github_35614077/article/details/98749285
https://hazelcast.org/blog/build-your-kubernetes-operator-with-the-right-tool/
https://github.com/answer1991/articles/blob/master/Kubernetes-is-the-next-generation-os.md
https://book.kubebuilder.io/cronjob-tutorial/gvks.html
https://github.com/operator-framework/awesome-operators