在k8s中安装jenkins并实现动态生成 jenkins slave
安装jenkins
1、创建一个命名空间
$ kubectl create namespace kube-ops
2、为jenkins创建pvc（也可以使用存储类创建）
apiVersion: v1
kind: PersistentVolume
metadata:
name: opspv
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Delete
nfs:
server: 192.168.1.244
path: /data/k8s
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: opspvc
namespace: kube-ops
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 2Gi3、创建jenkins需要的rbac权限
apiVersion: v1
kind: ServiceAccount
metadata:
name: jenkins2
namespace: kube-ops
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: jenkins2
rules:
- apiGroups: ["extensions", "apps"]
resources: ["deployments"]
verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
- apiGroups: [""]
resources: ["services"]
verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get","list","watch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: jenkins2
namespace: kube-ops
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: jenkins2
subjects:
- kind: ServiceAccount
name: jenkins2
namespace: kube-ops也可以为ServiceAccoun绑定一个系统现有的 cluster-admin 集群角色权限
4、创建jenkins pod
$ docker pull docker.io/jenkins/jenkins:lts
$ docker pull cnych/jenkins:jnlp6
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: jenkins2
namespace: kube-ops
spec:
template:
metadata:
labels:
app: jenkins2
spec:
terminationGracePeriodSeconds: 10
serviceAccount: jenkins2
containers:
- name: jenkins
image: docker.io/jenkins/jenkins:lts
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
name: web
protocol: TCP
- containerPort: 50000
name: agent
protocol: TCP
resources:
limits:
cpu: 1000m
memory: 1Gi
requests:
cpu: 500m
memory: 512Mi
livenessProbe:
httpGet:
path: /login
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 5
failureThreshold: 12
readinessProbe:
httpGet:
path: /login
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 5
failureThreshold: 12
volumeMounts:
- name: jenkinshome
subPath: jenkins2
mountPath: /var/jenkins_home
env:
- name: LIMITS_MEMORY
valueFrom:
resourceFieldRef:
resource: limits.memory
divisor: 1Mi
- name: JAVA_OPTS
value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvision
er.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Duser.timezone=Asia/Shanghai
securityContext:
fsGroup: 1000
volumes:
- name: jenkinshome
persistentVolumeClaim:
claimName: opspvc
---
apiVersion: v1
kind: Service
metadata:
name: jenkins2
namespace: kube-ops
labels:
app: jenkins2
spec:
selector:
app: jenkins2
type: NodePort
ports:
- name: web
port: 8080
targetPort: web
nodePort: 30003
- name: agent
port: 50000
targetPort: agent$ kubectl apply -f jenkins2.yaml
$ kubectl get pod -n kube-ops
jenkins2-76644dbc9b-llcsp 0/1 Running 0 #不能正常启动
$ kubectl describe pod jenkins2-76644dbc9b-llcsp -n kube-ops
$ kubectl logs -f jenkins2-76644dbc9b-llcsp -n kube-ops
5、在nfs服务器上修改jenkins持久目录的权限并重新创建jenkins pod
$ chown -R 1000 /data/k8s/jenkins2 #在192.168.1.244上
$ kubectl delete -f jenkins2.yaml
$ kubectl apply -f jenkins2.yaml
$ kubectl get pod -n kube-ops
jenkins2-76644dbc9b-llcsp 1/1 Running 0
为什么是1000？
上述镜像的Dockerfile文件中定义的是：user=jenkins group=jenkins uid=1000 gid=1000
Dockerfile文件的地址：
https://github.com/jenkinsci/docker/blob/master/Dockerfile
ARG user=jenkins
ARG group=jenkins
ARG uid=1000
ARG gid=1000
ARG http_port=8080
ARG agent_port=50000
ARG JENKINS_HOME=/var/jenkins_home
$ kubectl get svc -n kube-ops
jenkins2 NodePort 10.105.121.176
http://192.168.1.243:30003
初始密码在nfs服务器上
$ cat /data/k8s/jenkins2/secrets/initialAdminPassword
在jenkins上创建kubernetes云
enkins Master 和 Jenkins Slave 以 Pod 形式运行在 Kubernetes 集群的 Node 上，Master 运行在其中一个节点，并且将其配置数据存储到一个 Volume 上去，Slave 运行在各个节点上，并且它不是一直处于运行状态，它会按照需求动态的创建并自动删除
这种方式的工作流程大致为：当 Jenkins Master 接受到 Build 请求时，会根据配置的 Label 动态创建一个运行在 Pod 中的 Jenkins Slave 并注册到 Master 上，当运行完 Job 后，这个 Slave 会被注销并且这个 Pod 也会自动删除，恢复到最初状态。
1、安装插件
安装kubernetes plugin， 点击 Manage Jenkins -> Manage Plugins -> Available -> Kubernetes plugin
2、增加kubernetes云
点击 Manage Jenkins —> Configure System —> (拖到最下方)Add a new cloud —> 选择 Kubernetes，然后填写 Kubernetes 和 Jenkins 配置信息----连接测试
name：kubernetes
Kubernetes 地址：https://kubernetes.default.svc.cluster.local
Kubernetes 命名空间：kube-ops
Jenkins 地址：http://jenkins2.kube-ops.svc.cluster.local:8080（jenkins2是svc）
3、添加pod模板
添加pod模板----Kubernetes Pod Template
名称：jnlp
命名空间：kube-ops
标签列表：dongyali-jnlp
4、添加容器模板
添加容器----Container Template
名称：jnlp
Docker 镜像：cnych/jenkins:jnlp6（Jenkins 版本在 2.176.x以下的镜像名字去掉6）
工作目录：/home/jenkins/agent
运行的命令：清空
命令参数：清空
5、添加两个卷
添加卷----Host Path Volume
主机路径：/var/run/docker.sock
挂载路径：/var/run/docker.sock
主机路径：/root/.kube
挂载路径：/root/.kube
6、可能需要配置ServiceAccount
$ kubectl get sa -n kube-ops
jenkins2 1 14h
点击添加卷下面的高级----Service Account----jenkins2
7、用shell测试 Kubernetes 动态生成 jenkins slave
新建任务----名字----自由风格
通用----勾选限制项目的运行节点----标签表达式：dongyali-jnlp
构建----执行shell----输入如下内容----保存立即构建
echo "测试 Kubernetes 动态生成 jenkins slave"
echo "==============docker in docker==========="
docker info
echo "=============kubectl============="
kubectl get pods观察 Kubernetes 集群中 Pod 的变化：
$ kubectl get pod -n kube-ops
jenkins2-76644dbc9b-llcsp 1/1 Running 0 3h59m
jnlp-tl1km 1/1 Running 0 44s
当任务运行完毕，jnlp这个slave pod就会自动消失。
8、用pipeline测试 Kubernetes 动态生成 jenkins slave
新建任务----名字----流水线
在流水线脚本中输入如下内容----保存立即构建
node('dongyali-jnlp') {
stage('Clone') {
echo "1.Clone Stage"
}
stage('Test') {
echo "2.Test Stage"
}
stage('Build') {
echo "3.Build Stage"
}
stage('Deploy') {
echo "4. Deploy Stage"
}
}