26 data analysis cases -- the fourth station: web server log data collection based on Flume and Kafka
- Python: Python 3.x；
- Hadoop2.7.2 environment;
Extraction code: kohe
Step 1: install and start the httpd server
[root@master ~]# yum -y install httpd [root@master ~]# cd /var/www/html/ [root@master html]# vi index.html #Enter the following Hello Flume [root@master html]# service httpd start
The result is accessed through the browser. Indicates successful startup.
Check whether there is log generation.
[root@master html]# cd /var/log/httpd/ [root@master httpd]#cat access_log
The result is.
Step 2: configure Flume.
Enter the / usr/local/flume/conf directory and create a directory named access_log-HDFS.properties. Set access under the monitoring / var/log/httpd / directory_ Log file and send the contents to kafka through port 9092.
[root@master httpd]# cd /usr/local/flume/conf/ [root@master conf]# vi access_log-HDFS.properties #The contents of the configuration file are as follows. a1.sources = s1 a1.sinks = k1 a1.channels = c1 # Describe/configure the source a1.sources.s1.type = exec a1.sources.s1.command = tail -f /var/log/httpd/access_log a1.sources.s1.channels=c1 a1.sources.s1.fileHeader = false a1.sources.s1.interceptors = i1 a1.sources.s1.interceptors.i1.type = timestamp #Kafka sink configuration a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink a1.sinks.k1.topic = cmcc a1.sinks.k1.brokerList = master:9092 a1.sinks.k1.requiredAcks = 1 # Use a channel which buffers events in memory a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 # Bind the source and sink to the channel a1.sources.s1.channels = c1 a1.sinks.k1.channel = c1
Step 3: configure kafka
First, ensure that the zookeeper and kafka processes are in normal state. If they are closed, you can start them with the following command.
[root@master ~]# /usr/local/zookeeper/bin/zkServer.sh start [root@master ~]# cd /usr/local/kafka/bin/ [root@master bin]# ./kafka-server-start.sh -daemon /usr/local/kafka/config/server.properties
Step 4: write code to consume kafka data using Python
The PyKafkafka module is used here to create a customer named kafkacustomer Py file is used to consume kafka data from port 9092. topic needs to be formulated when consuming data. The code is as follows.
[root@master ~]# vim kafkacustomer.py from pykafka import KafkaClient client = KafkaClient(hosts="192.168.0.10:9092") topic = client.topics['cmcc'] //Specify the consumption data from the topic cmcc consumer = topic.get_simple_consumer( consumer_group="tpic", reset_offset_on_start=True ) for message in consumer: //Traverse the received content if message is not None: //If the information is not empty print(message.offset, message.value) //Data results
Step 5: start the project
Start Flume root directory and use access_ log-HDFS. The properties configuration file starts data collection. After data collection is started, kafka's topic name is cmcc automatically created.
[root@master ~]# cd /usr/local/flume/ [root@master flume]# bin/flume-ng agent --name a1 --conf conf --conf-file conf/access_log-HDFS.properties -Dflume.root.logger=INFO,console
Run kafkacustomer. Com using Python 3 Py file. Whenever the page is refreshed, the log file will be generated in real time, and collected by flume and sent to kafka.
[root@master ~]python customerkafka.py
Follow up cases are continuously updated
01 HBase crown size query system based on Python
02 civil aviation customer value analysis based on Hive
03 analysis of pharmacy sales data based on python
04 web server log data collection based on Flume and Kafka
05 Muke network data acquisition and processing
06 Linux operating system real-time log collection and processing
07 medical industry case - Analysis of dialectical association rules of TCM diseases
08 education industry case - Analysis of College Students' life data
10 entertainment industry case - advertising revenue regression prediction model
11 network industry case - website access behavior analysis
12 retail industry case - real time statistics of popular goods in stores
13 visualization of turnover data
14 financial industry case - financial data analysis based on stock information of listed companies and its derivative variables
15 visualization of bank credit card risk data
Operation analysis of 16 didi cities
17 happiness index visualization
18 employee active resignation warning model
19 singer recommendation model
202020 novel coronavirus pneumonia data analysis
Data analysis of 21 Taobao shopping Carnival
22 shared single vehicle data analysis
23 face detection system
24 garment sorting system
25 mask wearing identification system
26 imdb movie data analysis