Introduction: iLogtail is the collection part of Alibaba cloud's simple log service, also known as "SLS". It is used to collect telemetry data, such as logs, tracking and indicators, and has been officially open source
(https://github.com/alibaba/il... ). This paper introduces the simplest process of how to install, configure and use ilogtail in K8S environment, so as to help users quickly get started with ilogtail log collection using precompiled version.
Preparation before use
- Alicloud log service is enabled and a Project is created (see the previous section getting started with ilogtail - collecting host environment logs to SLS for specific steps)
- Prepare a K8S cluster with public network access, and the server architecture is X86-64.
Create log configuration
1. Jump to the log service console (sls.console.aliyun.com) and click the project created in the previous section.
2. After entering the Project query page, click the "magnifying glass" icon in the left sidebar to expand the logstore management interface, and click "+" to pop up the right sidebar of "create logstore". Follow the prompts to configure. Enter the logstore name and click "confirm".
3. After the logstore is created successfully, cancel the data access wizard. Click the "Cube" button in the left sidebar and select "machine group" in the pop-up "resource" floating layer. In the left column of the expanded "machine group", click the "square grid" icon in the upper right corner, and select "create machine group" in the pop-up floating layer.
4. Configure according to the prompts in the sidebar of "create machine group", select "user defined ID" for "machine group ID", and the recommendations of "name", "machine group Topic" and "user defined ID" are consistent. "User defined ID" is one of the most important configurations. In this tutorial, "my-k8s-group" is used, which will be used again later when installing ilogtail. Click OK to save the machine group.
5. Click the "magnifying glass" icon in the left sidebar again to expand the logstore management interface, click the "expand down" icon of the logstore created in step 2 to pop up the "configure logstore" menu. Click the "+" button of "logtail configuration".
6. Search for "kube" in the pop-up "quick access data" dialog box and select "Kubernertes file". Click continue in the pop-up prompt box.
7. In the "Kubernertes file" configuration interface, directly select "use existing machine group".
8. Jump to the "machine group configuration" interface, select the machine group created in step 4, click ">" to add it to the "application machine group", and then click "next".
9. In ilogtail configuration, only modify the two required items "configuration name" and "log path", and click "next" to confirm.
10. Complete index configuration. This step does not modify any options. Click next to complete the configuration.
At this point, the entire log configuration has been completed. Please keep the page open.
Installing ilogtail
1. Log in to the central control computer of K8S cluster. Edit the ConfigMap YAML of ilogtail.
$ vim alicloud-log-config.yaml
Paste the following content in Vim and save it (note that modify the fields prompted in the comment, lines 7-11).
apiVersion: v1
kind: ConfigMap
metadata:
name: alibaba-log-configuration
namespace: kube-system
data:
log-project: "my-project" #Change to actual project name
log-endpoint: "cn-wulanchabu.log.aliyuncs.com" #Modify to actual endpoint
log-machine-group: "my-k8s-group" #You can customize the machine group name
log-config-path: "/etc/ilogtail/conf/cn-wulanchabu_internet/ilogtail_config.json" #Modify CN Wulanchabu to the actual project region
log-ali-uid: "*********" #Change to alicloud UID
access-key-id: "" #This tutorial is not available
access-key-secret: "" #This tutorial is not available
cpu-core-limit: "2"
mem-limit: "1024"
max-bytes-per-sec: "20971520"
send-requests-concurrency: "20"2. Calculate alicloud log config Yaml's sha256 hash and edit ilogtail's daemon set yaml.
$ sha256sum alicloud-log-config.yaml f370df37916797aa0b82d709ae6bfc5f46f709660e1fd28bb49c22da91da1214 alicloud-log-config.yaml $ vim logtail-daemonset.yaml
Paste the following content in Vim and save it (Note: modify the fields prompted in the comment, lines 21 and 25).
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: logtail-ds
namespace: kube-system
labels:
k8s-app: logtail-ds
spec:
selector:
matchLabels:
k8s-app: logtail-ds
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
k8s-app: logtail-ds
kubernetes.io/cluster-service: "true"
version: v1.0
annotations:
checksum/config: f370df37916797aa0b82d709ae6bfc5f46f709660e1fd28bb49c22da91da1214 #It must be modified to alicloud log config hash of yaml
spec:
containers:
- name: logtail
image: registry.cn-wulanchabu.aliyuncs.com/log-service/logtail:latest #It can be modified to the region closest to k8s cluster
resources:
limits:
cpu: 2
memory: 1024Mi
requests:
cpu: 100m
memory: 256Mi
livenessProbe:
httpGet:
path: /liveness
port: 7953
scheme: HTTP
initialDelaySeconds: 30
periodSeconds: 60
securityContext:
privileged: true
env:
- name: HTTP_PROBE_PORT
value: "7953"
- name: "ALIYUN_LOGTAIL_CONFIG"
valueFrom:
configMapKeyRef:
name: alibaba-log-configuration
key: log-config-path
- name: "ALIYUN_LOGTAIL_USER_ID"
valueFrom:
configMapKeyRef:
name: alibaba-log-configuration
key: log-ali-uid
- name: "ALIYUN_LOGTAIL_USER_DEFINED_ID"
valueFrom:
configMapKeyRef:
name: alibaba-log-configuration
key: log-machine-group
- name: "ALICLOUD_LOG_ACCESS_KEY_ID"
valueFrom:
configMapKeyRef:
name: alibaba-log-configuration
key: access-key-id
- name: "ALICLOUD_LOG_ACCESS_KEY_SECRET"
valueFrom:
configMapKeyRef:
name: alibaba-log-configuration
key: access-key-secret
- name: "ALICLOUD_LOG_DOCKER_ENV_CONFIG"
value: "true"
- name: "ALICLOUD_LOG_ECS_FLAG"
value: "false"
- name: "ALICLOUD_LOG_DEFAULT_PROJECT"
valueFrom:
configMapKeyRef:
name: alibaba-log-configuration
key: log-project
- name: "ALICLOUD_LOG_ENDPOINT"
valueFrom:
configMapKeyRef:
name: alibaba-log-configuration
key: log-endpoint
- name: "ALICLOUD_LOG_DEFAULT_MACHINE_GROUP"
valueFrom:
configMapKeyRef:
name: alibaba-log-configuration
key: log-machine-group
- name: "ALIYUN_LOG_ENV_TAGS"
value: "_node_name_|_node_ip_"
- name: "_node_name_"
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: "_node_ip_"
valueFrom:
fieldRef:
fieldPath: status.hostIP
# resource limit for logtail self process
- name: "cpu_usage_limit"
valueFrom:
configMapKeyRef:
name: alibaba-log-configuration
key: cpu-core-limit
- name: "mem_usage_limit"
valueFrom:
configMapKeyRef:
name: alibaba-log-configuration
key: mem-limit
- name: "max_bytes_per_sec"
valueFrom:
configMapKeyRef:
name: alibaba-log-configuration
key: max-bytes-per-sec
- name: "send_request_concurrency"
valueFrom:
configMapKeyRef:
name: alibaba-log-configuration
key: send-requests-concurrency
volumeMounts:
- name: sock
mountPath: /var/run/
- name: root
mountPath: /logtail_host
readOnly: true
mountPropagation: HostToContainer
terminationGracePeriodSeconds: 30
tolerations:
- operator: "Exists"
hostNetwork: true
dnsPolicy: "Default"
volumes:
- name: sock
hostPath:
path: /var/run/
- name: root
hostPath:
path: /3. Apply YAML configuration and create ConfigMap and DaemonSet.
$ kubectl apply -f alicloud-log-config.yaml configmap/alibaba-log-configuration created $ kubectl apply -f logtail-daemonset.yaml daemonset.apps/logtail-ds created
4. Wait for 1 minute and check whether DeamonSet operates normally
$ kubectl get -f logtail-daemonset.yaml
At this time, the console should print a message similar to the following, indicating that the installation is successful
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE logtail-ds 3 3 3 3 3 <none> 2m1s
Report log and view
1. Create a Pod for continuously generating logs.
$ vim demo-pod.yaml
Paste the following in Vim and save it (note that you may need to modify the fields prompted in the comment, lines 8-9).
apiVersion: v1
kind: Pod
metadata:
labels:
name: demo-pod
name: demo-pod
spec:
# imagePullSecrets: # Comment out to enable specific image pull secret
# - name: myregistrykey # repleace it to specific registry key containers
containers:
- image: busybox
imagePullPolicy: IfNotPresent
name: demo-pod
command: ["/bin/sh"]
args: ["-c", "while true; do echo $(date) >>/tmp/demo.log; sleep 10; done"]
resources: {}
securityContext:
capabilities: {}
privileged: false
terminationMessagePath: /dev/termination-log
dnsPolicy: ClusterFirst
restartPolicy: AlwaysApply YAML configuration and create Pod
$ kubectl apply -f demo-pod.yaml pod/demo-pod created
Return to the Web console, click "query log" in the configuration completion interface to jump to the log query interface. Click the "magnifying glass" icon on the left side of the page, select logstore, click the "eye" icon, try to adjust the Shard and time range in the "consumption preview" sidebar on the left, and click preview to view the reported log.
What's Next
You can go to the next section to learn how to use iLogtail - iLogtail local deployment scheme
Understand the container collection principle "collect container standard output through DaemonSet console"
You can also learn more usage of ilogtail collection:
- Collect container text log through DaemonSet console
- Collection container log
- How to get the Label and environment variables of a container
Original link
This article is the original content of Alibaba cloud and cannot be reproduced without permission.