Running web based on k8s: dynamic and static separation of nginx+tomcat based on NFS

Posted by bigmichael on Wed, 19 Jan 2022 22:39:43 +0100

Image of nginx based on alpine:

Note: the user shell type created based on alpine must be bin/bash, or the su -c command cannot be executed #/ bin/bash script;

<P>pwd
<P>/data/dockerfile</P>

To build a base image:

Build warehouse file

# vim repositories 
http://mirrors.aliyun.com/alpine/v3.8/main
http://mirrors.aliyun.com/alpine/v3.8/community

Write Dockerfile

# vim Dockerfile
FROM k8s-harbor.taozi.net/pub-image/alpine:v3.8.1
LABEL maintainer="lijian/20210117"
COPY repositories /etc/apk/repositories
RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && apk update && apk --no-cache add iotop gcc libgcc libc-dev libcurl libc-utils pcre-dev zlib-dev libnfs make pcre pcre2 zip unzip net-tools pstree wget libevent libevent-dev iproute2 openssl-dev

Create build sh:

vim build.sh
#!/bin/bash
docker build -t k8s-harbor.taozi.net/pub-image/alpine-base:V3.8.1 .
docker push k8s-harbor.taozi.net/pub-image/alpine-base:V3.8.1

Create nginx base image:

Enter container command:

<P>docker run -it --rm k8s-harbor.taozi.net/pub-image/alpine-base:V3.8.1 sh </P>

Prepare relevant documents:

mkdir /data/dockerfile/web/nginx/1.18.0-alpine/ -pv
cd 1.18.0-alpine/
wget http://nginx.org/download/nginx-1.18.0.tar.gz
Remove the from the package nginx.conf,And modify:
vim nginx.conf
#Add on the second line:
daemon off;

Write Dockerfile:

vim Dockerfile 

FROM k8s-harbor.taozi.net/pub-image/alpine-base:V3.8.1
LABEL maintainer="lijian/20210117"
ADD nginx-1.18.0.tar.gz /usr/local/src
RUN cd /usr/local/src/nginx-1.18.0 && ./configure --prefix=/apps/nginx \
--user=nginx \
--group=nginx \
--with-http_ssl_module \
--with-http_v2_module \
--with-http_realip_module \
--with-http_stub_status_module \
--with-http_gzip_static_module \
--with-pcre \
--with-stream \
--with-stream_ssl_module && make && make install && ln -s /apps/nginx/sbin/nginx /usr/bin/ && addgroup -g 2022 -S nginx && adduser -s /sbin/nologin -S -D -u 2022 -G nginx nginx
COPY nginx.conf /apps/nginx/conf/nginx.conf
RUN chown -R nginx.nginx  /apps/nginx/
EXPOSE 80 443
CMD ["nginx"]

Create build sh:

vim  build.sh 
#!/bin/bash
docker build -t k8s-harbor.taozi.net/pub-image/nginx-base:1.18.0 .
docker push k8s-harbor.taozi.net/pub-image/nginx-base:1.18.0

Create nginx business image:

Prepare files and web files

mkdir /data/dockerfile/app/nginx-app1/ -p
cd /data/dockerfile/app/nginx-app1/
mkdir html
echo Tish is nginx-app1 > html/index.html
tar zcfv html.tar.gz html

Write Dkocerfile:

vim Dockerfile 
FROM k8s-harbor.taozi.net/pub-image/nginx-base:1.18.0
LABEL maintainer="lijian/20210117"
ADD html.tar.gz /data/nginx/
COPY nginx.conf /apps/nginx/conf/nginx.conf
RUN chown -R nginx.nginx /data/nginx
CMD ["nginx"]

Build mirror script:

vim  build.sh 
#!/bin/bash
docker build -t k8s-harbor.taozi.net/pub-image/nginx-app1:v1 .
docker push k8s-harbor.taozi.net/pub-image/nginx-app1:v1

Start container:

<P>docker run -d -p 8011:80 k8s-harbor.taozi.net/pub-image/nginx-app1:v1 </P>

Operating nginx in k8s environment:

Create a yaml file for the namespace:

mkdir /data/k8s-01-data/yml/linux-61/nginx-app1 -pv
vim linux61.yaml 
kind: Namespace
apiVersion: v1
metadata:
  name: linux61

Create yaml file for nginx-app1:

vim nginx.yaml 
kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    app: linux61-nginx-deployment-lable
  name: linux61-nginx-deployment
  namespace: linux61
spec:
  replicas: 1
  selector: 
    matchLabels:
      app: linux61-nginx-selector
  template:
    metadata:
      labels:
        app: linux61-nginx-selector
    spec:
      containers:
      - name: linux61-nginx-container
        image: k8s-harbor.taozi.net/apps/nginx-app1:v1
        #command: ["/apps/tomcat/bin/run_tomcat.sh"]
        #imagePullPolicy: IfNotPresent
        imagePullPolicy: Always
        ports:
        - containerPort: 80
          protocol: TCP
          name: http
        - containerPort: 443
          protocol: TCP
          name: https
        env:
        - name: "password"
          value: "123456"
        - name: "age"
          value: "18"
        resources: 
          limits:
            cpu: 500m
            memory: 1Gi
          requests:
            cpu: 200m
            memory: 200Mi

---

kind: Service
apiVersion: v1
metadata:
  labels:
    app: linux61-nginx-svc
  name: linux61-nginx-svc
  namespace: linux61
spec:
  type: NodePort
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
    nodePort: 30001
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
    nodePort: 30443
  selector:
    app: linux61-nginx-selector

Run nginx-app1 using k8s:

<P>kubectl apply -f nginx.yaml </P>

Login using browser

Making tomcat image based on alpine:

Create a jdk image (directly use the underlying image alpine):

Write Dkocerfile:

vim Dockerfile
FROM k8s-harbor.taozi.net/pub-image/alpine-base:V3.8.1
LABEL maintainer="lijian/20210117"
RUN apk add --no-cache bash openjdk8-jre-base=8.275.01-r0 && rm -rf /var/cache/apk/*

Build mirror script:

vim  build.sh 
#!/bin/bash
docker build -t k8s-harbor.taozi.net/pub-image/jdk-base:8.275.01-r0 .
docker push k8s-harbor.taozi.net/pub-image/jdk-base:8.275.01-r0

Create tomcat basic image:

Make startup script:

cat  > run_tomcat.sh << EOF
#!/bin/bash
su - www -c "/apps/tomcat/bin/catalina.sh start"
su - www -c "tail -f /etc/hosts"
EOF
chmod a+x run_tomcat.sh

Create test page:

mkdir ROOT
echo "this is tomcat-app-base"> ROOT/index.jsp
tar czfv ROOT.tar.gz ROOT/

Modify page file path:

The server of the binary file XML, take it out and modify it

vim server.xml
appBase="/d

Write Dkocerfile:

vim Dockerfile
FROM k8s-harbor.taozi.net/pub-image/jdk-base:8.275.01-r0
LABEL maintainer="lijian"
ENV LANG en_US.UTF-8
ENV TERM xterm
ENV TOMCAT_MAJOR_VERSION 8
ENV TOMCAT_MINOR_VERSION 8.5.64
ENV CATALINA_HOME /apps/tomcat
ENV APP_DIR /webapps
ADD apache-tomcat-8.5.64.tar.gz /apps
RUN  ln -s /apps/apache-tomcat-8.5.64 /apps/tomcat && addgroup  -g 2021 -S www && adduser  -s /bin/bash -S -D  -u 2021 -G www www
ADD server.xml /apps/tomcat/conf/server.xml
ADD run_tomcat.sh /apps/tomcat/bin/run_tomcat.sh
ADD ROOT.tar.gz /data/tomcat/webapps
RUN chown -R www.www /apps/ /data/tomcat/
EXPOSE 8080
CMD ["/apps/tomcat/bin/run_tomcat.sh"]

Build mirror script:

cat > build.sh << EOF
#!/bin/bash
docker build -t k8s-harbor.taozi.net/pub-image/tomcat:8.5.73 .
docker push k8s-harbor.taozi.net/pub-image/tomcat:8.5.73
EOF

Test:

Build tomcat business image:

Enter the business directory of tomcat:

# pwd 
/data/dockerfile/app/tomcat-app1

Create test page:

mkdir ROOT
echo "this is tomcat-app-11111"> ROOT/index.jsp
tar czfv ROOT.tar.gz ROOT/

Write Dkocerfile:

FROM k8s-harbor.taozi.net/pub-image/tomcat:8.5.73
LABEL maintainer="lijian"
ADD ROOT.tar.gz /data/tomcat/webapps
ADD server.xml /apps/tomcat/conf/server.xml
RUN chown -R www.www /apps/  /data/tomcat/
EXPOSE 8080
CMD ["/apps/tomcat/bin/run_tomcat.sh"]

Build mirror script:

cat > build.sh << EOF
#!/bin/bash
docker build -t k8s-harbor.taozi.net/apps/tomcat-app1:v1 .
docker push k8s-harbor.taozi.net/apps/tomcat-app1:v1
EOF

test

Running tomcat in k8s environment:

Create the yaml file of tomcat-app1:

vim tomcat-app1.yaml 
kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    app: linux61-tomcat-app1-deployment-lable
  name: linux61-tomcat-app1-deployment
  namespace: linux61
spec:
  replicas: 1
  selector: 
    matchLabels:
      app: linux61-tomcat-app1-selector
  template:
    metadata:
      labels:
        app: linux61-tomcat-app1-selector
    spec:
      containers:
      - name: linux61-tomcat-app1-container
        image: k8s-harbor.taozi.net/apps/tomcat-app1:v1
        #command: ["/apps/tomcat/bin/run_tomcat.sh"]
        #imagePullPolicy: IfNotPresent
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
          protocol: TCP
          name: http
        env:
        - name: "password"
          value: "123456"
        - name: "age"
          value: "18"
        resources: 
          limits:
            cpu: 500m
            memory: 1Gi
          requests:
            cpu: 200m
            memory: 200Mi

---

kind: Service
apiVersion: v1
metadata:
  labels:
    app: linux61-tomcat-app1-svc
  name: linux61-tomcat-app1-svc
  namespace: linux61
spec:
  type: NodePort
  ports:
  - name: http
    port: 8080
    protocol: TCP
    targetPort: 8080
    nodePort: 30002
  selector:
    app: linux61-tomcat-app1-selector

Run nginx-app1 using k8s:

<P>kubectl apply -f tomcat-app1.yaml </P>

Login using browser

nginx+tomcat in k8s realizes dynamic and static separation:

Rebuild the business image of nginx and add upstream:

Modify the configuration file of nginx:

vim nginx.conf
#The upstream statement block is within the http statement block, and the url cannot be underlined:
upstream  tomcat-app1 { 
        server  linux61-tomcat-app1-svc.linux61.svc.taozi.local:8080 weight=1 fail_timeout=5s max_fails=3;
}
server {
......
#The location statement block is within the server statement block
location ~* \.jsp$ {
proxy_pass  http://tomcat-app1;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;  
proxy_set_header X-Real-IP  $remote_addr;
}
......
}

Rebuild mirror:

bash build.sh

Delete the original nginx pod:

kubectl delete linux61-nginx-app1-deployment-5bbc8745d7-w45g4  pod -n linux61

The node will pull the image again, and the reconstruction is successful:

You can also use yaml file to delete the original pod and create a new one in Shuyong yaml file:

kubectl delete -f nginx.yaml
kubectl apply -f nginx.yaml

Experiment with web pages:

Dynamic and static separation based on NFS:

Prepare nfs server: host 192.168.37.106

yum install nfs-util -y
mkdir /data/linux61/{static,images} -pv
echo "this is nginx-app1-static" > /data/linux61/static/index.html
echo "this is tomcat-app1-images" > /data/linux61/images/index.jsp
 stay /etc/exports Add the following line to the file:
vim /etc/exports
/data/linux61/                 192.168.37.0/24(rw,no_root_squash) 
systemctl enable --now nfs
 Experiment with mount and read-write functions on other hosts:
mount -t nfs 192.168.37.106:/data/linux61  /mnt
cp /etc/hosts /mnt/

Modify the yaml file of nginx-app1:

Note: #volumes belongs to deployment spec.template. The level of spec is one level higher than volumeMounts: and you should pay attention to the format when writing

cd /data/k8s-01-data/yml/linux-61/nginx-app1/
#volumeMounts and resources: both belong to deployment spec.template. Sub options in spec.containers
        volumeMounts:
        - name: linux61-static
          mountPath: /data/nginx/html
          readOnly: false
#volumes belong to deployment spec.template. The level of spec is one level higher than volumeMounts: and you should pay attention to the format when writing
      volumes:
      - name: linux61-static
        nfs:
          server: 192.168.37.106
          path: /data/linux61/static

yaml file to which nginx is applied:

<P>kubectl apply -f nginx-app1.yaml </P>

Modify the yaml file of tomcat-app1:

vim tomcat-app1.yaml
#Add the following at the end of the statement block of containers:
        volumeMounts:
        - name: linux61-images
          mountPath: /data/tomcat/webapps/ROOT/images
         # mountPath: /data/tomcat/webapps/ROOT
          readOnly: false
      volumes:
      - name: linux61-images
        nfs:
          server: 192.168.37.106
          path: /data/linux61/image

Start a new pod: kubectl apply -f tomcat-app1.yaml

Test:

Mount relationship notes:

First mount

It is equivalent to the coincidence of two images files. The original ROOT file index JSP has not changed. The files in the images directory are nfs files;

Relationship of the second mount

For the directory in the container, the images file under nfs coincides with the ROOT file, and the name is subject to ROOT, but the file inside is already a file under nfs, but the original ROOT file is overwritten by the mount relationship;

Commands used:

kubectl  get service -A  -o wide
kubectl  get  pods -A  -o wide
kubectl  get  nodes-A -o wide
kubectl  get deployment  -A 
kubectl  get deployment -n linux61 -o wide 
kubectl  describe svc linux61-nginx-app1-svc -n linux61  #Query the details of an svc
kubectl  describe deployment -n linux61                #Query the deployment details of a namespace
kubectl  create -f tomcat-app1.yaml
kubectl  apply  -f tomcat-app1.yaml
kubectl  delete -f tomcat-app1.yaml
kubectl  create -f tomcat-app1.yaml  --save-config --record
kubectl  apply -f tomcat-app1.yaml  --record #Recommended command
kubectl  exec -it linux61-tomcat-app1-deployment-6bccd8f9c7-g76s5 sh  -n linux61   #Enter a pod
kubectl  logs  linux61-tomcat-app1-deployment-6bccd8f9c7-g76s5   -n linux61  #Query the log of a pod
kubectl  delete pods  linux61-tomcat-app1-deployment-6bccd8f9c7-g76s5   -n linux61