I used Python to make a data visualization analysis of the comments on the White Snake 2 and the green snake

Posted by nay4 on Sun, 19 Sep 2021 15:16:37 +0200

Hello, I'm spicy.

Finally, we are going to start the series of articles on data analysis. Compared with crawlers, the technical dimension has risen to a higher level. The output of my article will update the series of practical projects and the series of detailed explanation and summary of knowledge points in two columns respectively, with the goal of realizing 100 cases of crawler and data analysis practical projects in the short term.

Libraries involved:

Pandas - data processing Pyecharts - data visualization jieba - word segmentation collections - data statistics

Visualization part:

Line chart line histogram Bar Pie chart Pie Calendar chart Calendar cloud chart WordCloud map Geo

White Snake 2: robbed by green snake

*Plot introduction:*

On July 23, 2021, white snake 2: green snake robbery was released in the mainland. It mainly tells that in the late Southern Song Dynasty, Xiaobai was finally pressed under Leifeng Tower by the sea of France in order to save Xu Xian. Xiaoqing is accidentally driven into the strange fantasy of Shura city by Fahai. In several crises, Xiaoqing was rescued by the mysterious masked boy. Xiaoqing took the idea of going out to rescue Xiaobai. After suffering and growing up, she found a way to leave with the masked boy

Execution link Notebook

Install third party packages

!pip install pyecharts
!pip install pandas
!pip install numpy

Import third party packages

import pandas as pd
import numpy as np
from pyecharts.charts import * 
from PIL import Image
from collections import Counter
from pyecharts import options as opts # Visual configuration item
from pyecharts.commons.utils import JsCode # Used to run js code
from pyecharts.globals import ThemeType,SymbolType,ChartType # Visual theme style

Read data

df = pd.read_excel("./White Snake 2.xlsx")
df.head(10)  # View the first 10 lines
iduser namecityscorecommentComment time
01142669584Qitong glutinous riceguest5.0The plot is very attractive. Watching an animated cartoon surprised me2021-08-31 23:56:30
11142662178LnV14610189Xining5.0Strong picture sense!2021-08-31 23:36:00
21142666877Alo861902585Guangzhou5.0And a very good connection, wonderful2021-08-31 23:34:41
31142660216Y. Xi'an4.0The characters in the picture don't say that you can always believe in chasing light. The plot is smooth and the overall rhythm is OK. It is recommended to watch -!2021-08-31 23:30:56
41142669423I want to see the moon for youFengtai5.0Yes, although Xiaoqing and Xiaobai's obsession is far fetched (OK). If you can elaborate on the obsession of Niu Mo, the plot will be more perfect2021-08-31 23:27:28
51142669422Ah, Ka, wow, ahsunshine5.0It's good. It feels more and more attractive2021-08-31 23:27:12
61142669404Lfz9696Yongzhou4.5OK, very good2021-08-31 23:23:30
71142666812TenacityGuangzhou4.0The movie was ok, except that a man next door kept shaking his legs.2021-08-31 23:22:23
81142661206CQE579669148Urumqi5.0Take a good look, recommend2021-08-31 23:16:22
91142668420Fat suonaIli 5.0The plot is a little incomprehensible, but the animation effects are great! The plot is very moving.2021-08-31 23:06:36

Data cleaning

Missing value view

df.isnull().sum()
id      0
 user name     1
 city      0
 score      0
 comment      0
 Comment time    0
dtype: int64

Check and find that there is a real situation

The user name is missing, and the data of other columns is complete. Fill in the blank value with "unknown":

df['user name'].fillna('unknown', inplace=True)
df.isnull().sum()

Pyecharts data visualization

Score grade distribution

# Linear gradient
color_js = """new echarts.graphic.LinearGradient(0, 0, 1, 0,
    [{offset: 0, color: '#009ad6'}, {offset: 1, color: '#ed1941'}], false)"""
​
df_star = df.groupby('score')['comment'].count()
df_star = df_star.sort_values(ascending=True)
x_data = [str(i) for i in list(df_star.index)]
y_data = df_star.values.tolist()
b1 = (
    Bar()
    .add_xaxis(x_data)
    .add_yaxis('',y_data,itemstyle_opts=opts.ItemStyleOpts(color=JsCode(color_js)))
    .reversal_axis()
    .set_series_opts(label_opts=opts.LabelOpts(position='right'))    
    .set_global_opts(
        yaxis_opts=opts.AxisOpts(name='Rating'),
        xaxis_opts=opts.AxisOpts(name='people/second'),
        title_opts=opts.TitleOpts(title='Score grade distribution',pos_left='45%',pos_top="5%"),
        legend_opts=opts.LegendOpts(type_="scroll", pos_left="85%",pos_top="28%",orient="vertical")
    )
)
​
df_star = df.groupby('score')['comment'].count()
x_data = [str(i) for i in list(df_star.index)]
y_data = df_star.values.tolist()
p1 = (
    Pie(init_opts=opts.InitOpts(width='800px', height='600px'))
    .add(
    '',
    [list(z) for z in zip(x_data, y_data)],
    radius=['10%', '30%'],
    center=['65%', '60%'],
    label_opts=opts.LabelOpts(is_show=True),
    ) 
    .set_colors(["blue", "green", "#800000", "red", "#000000", "orange", "purple", "red", "#000000", "orange", "purple"])
    .set_series_opts(label_opts=opts.LabelOpts(formatter='score{b}: {c} \n ({d}%)'),position="outside")   
)
​
b1.overlap(p1)
b1.render_notebook()

The score of 5.0 reached 56%, more than half of the audience received five-star praise, and more than four-star praise reached 85%. It seems that everyone still highly recognizes this animation.

Distribution of daily comments from August 1, 2021 to August 31, 2021:

# Set style
# The style of the loaded js code is mainly color and theme
color_js = """new echarts.graphic.LinearGradient(0, 1, 0, 0,
    [{offset: 0, color: '#009ad6'}, {offset: 1, color: '#ed1941'}], false)"""
​
area_color_js = (
    "new echarts.graphic.LinearGradient(0, 0, 0, 1, "
    "[{offset: 0, color: '#eb64fb'}, {offset: 1, color: '#3fbbff0d'}], false)"
)
​
# Set parameters
linestyle_dic = { 'normal': {
                    'width': 2,  
                    'shadowColor': '#696969', 
                    'shadowBlur': 10,  
                    'shadowOffsetY': 10,  
                    'shadowOffsetX': 10,  
                    }
                }
​
# Transfer time format
df['Comment time'] = pd.to_datetime(df['Comment time'], format='%Y/%m/%d %H:%M:%S')
# Daily Comments
df['Comment time'] = pd.to_datetime(df['Comment time'], format='%Y/%m/%d %H:%M:%S')
df_day = df.groupby(df['Comment time'].dt.day)['comment'].count()  # Get the number of comments according to the comment time (count)
day_x_data = [str(i) for i in list(df_day.index)] # x axis
day_y_data = df_day.values.tolist() # Output to list y-axis
 
line1 = (
    Line(init_opts=opts.InitOpts(bg_color=JsCode(color_js))) # Linear visualization
    .add_xaxis(xaxis_data=day_x_data) # Add x-axis data
    .add_yaxis(  # Add y-axis data
        series_name="",  # y-axis name
        y_axis=day_y_data, # data
        is_smooth=True,
        is_symbol_show=True,
        symbol="circle",
        symbol_size=6,
        linestyle_opts=opts.LineStyleOpts(color="#fff"), # Configure y axis
        label_opts=opts.LabelOpts(is_show=True, position="top", color="white"), # y-axis label
        itemstyle_opts=opts.ItemStyleOpts(
            color="red", border_color="#fff", border_width=3
        ),
        tooltip_opts=opts.TooltipOpts(is_show=False),
        areastyle_opts=opts.AreaStyleOpts(color=JsCode(area_color_js), opacity=1),
    )
    .set_global_opts(
        title_opts=opts.TitleOpts(
            title="Daily comments in August",
            pos_top="5%",
            pos_left="center",
            title_textstyle_opts=opts.TextStyleOpts(color="#fff", font_size=16),
        ),
        xaxis_opts=opts.AxisOpts(
            type_="category",
            boundary_gap=True,
            axislabel_opts=opts.LabelOpts(margin=30, color="#ffffff63"),
            axisline_opts=opts.AxisLineOpts(is_show=False),
            axistick_opts=opts.AxisTickOpts(
                is_show=True,
                length=25,
                linestyle_opts=opts.LineStyleOpts(color="#ffffff1f"),
            ),
            splitline_opts=opts.SplitLineOpts(
                is_show=True, linestyle_opts=opts.LineStyleOpts(color="#ffffff1f")
            ),
        ),
        yaxis_opts=opts.AxisOpts(
            type_="value",
            position="left",
            axislabel_opts=opts.LabelOpts(margin=20, color="#ffffff63"),
            axisline_opts=opts.AxisLineOpts(
                linestyle_opts=opts.LineStyleOpts(width=2, color="#fff")
            ),
            axistick_opts=opts.AxisTickOpts(
                is_show=True,
                length=15,
                linestyle_opts=opts.LineStyleOpts(color="#ffffff1f"),
            ),
            splitline_opts=opts.SplitLineOpts(
                is_show=True, linestyle_opts=opts.LineStyleOpts(color="#ffffff1f")
            ),
        ),
        legend_opts=opts.LegendOpts(is_show=False),
    )
)
line1.render_notebook()

The number of comments per day peaked on August 1 (data excluding July), and the number of comments gradually decreased with the passage of time, which is also in line with the general law of film viewing.

Comments per hour

The statistics is the sum of comments per hour and day in the 31 days from August 1, 2021 to August 31, 2021 (if you are interested, you can view the distribution of 24-hour film reviews on a day separately and filter by date)

df_hour = df.groupby(df['Comment time'].dt.hour)['comment'].count()
hours_x_data = [str(i) for i in list(df_hour.index)]
hours_y_data = df_hour.values.tolist()
 
line1 = (
#     Line(init_opts=opts.InitOpts(bg_color=JsCode(color_js)))
    Line(init_opts=opts.InitOpts(width='1000px', height='400px'))
    .add_xaxis(xaxis_data=hours_x_data)
    .add_yaxis(
        series_name="",
        y_axis=hours_y_data,
        is_smooth=True,
        is_symbol_show=True,
        symbol="circle",
        symbol_size=6,
        linestyle_opts=opts.LineStyleOpts(color="#fff"),
        label_opts=opts.LabelOpts(is_show=True, position="top", color="white"),
        itemstyle_opts=opts.ItemStyleOpts(
            color="red", border_color="#fff", border_width=3
        ),
        tooltip_opts=opts.TooltipOpts(is_show=False),
        areastyle_opts=opts.AreaStyleOpts(color=JsCode(area_color_js), opacity=1),
    )
    .set_series_opts(
        linestyle_opts=linestyle_dic,label_opts=opts.LabelOpts(font_size=12, color='red' ),
        markpoint_opts=opts.MarkPointOpts(
            data=[opts.MarkPointItem(type_="max",itemstyle_opts=opts.ItemStyleOpts(
            color="#06FFD7", border_width=3)), 
            opts.MarkPointItem(type_="min",itemstyle_opts=opts.ItemStyleOpts(
            color="#06FFD7", border_width=3))],
            symbol_size=[65, 50],
            label_opts=opts.LabelOpts(position="inside", color="red", font_size=10)
            ),
    )
    .set_global_opts(
        title_opts=opts.TitleOpts(
            title="Comments per hour",
            pos_top="5%",
            pos_left="center",
            title_textstyle_opts=opts.TextStyleOpts(color="#EB1934", font_family='STKaiti', font_size=20),
        ),
        xaxis_opts=opts.AxisOpts(
            type_="category",
            boundary_gap=True,
            axislabel_opts=opts.LabelOpts(margin=30, color="#EB1934"),
            axisline_opts=opts.AxisLineOpts(
                is_show=False, 
                linestyle_opts=opts.LineStyleOpts(color="#EB1934")
            ),
            axistick_opts=opts.AxisTickOpts(
                is_show=False,
                length=25,
                linestyle_opts=opts.LineStyleOpts(color="#EB1934"),
            ),
        ),
        yaxis_opts=opts.AxisOpts(
            type_="value",
            position="left",
            axislabel_opts=opts.LabelOpts(is_show=False, margin=20, color="#EB1934"),
            axisline_opts=opts.AxisLineOpts(
                is_show=False,
                linestyle_opts=opts.LineStyleOpts(width=2, color="#EB1934")
            ),
            axistick_opts=opts.AxisTickOpts(
                is_show=False,
                length=10,
                linestyle_opts=opts.LineStyleOpts(color="#EB1934"),
            ),
            splitline_opts=opts.SplitLineOpts(
                is_show=False, linestyle_opts=opts.LineStyleOpts(color="#EB1934")
            ),
        ),
        legend_opts=opts.LegendOpts(is_show=False),
        graphic_opts=[
            opts.GraphicImage(
                graphic_item=opts.GraphicItem(
                    id_="logo", z=-10, bounding="raw", origin=[50, 100]
                ),
                graphic_imagestyle_opts=opts.GraphicImageStyleOpts(
                    image="./12.jpg",
                    width=1000,
                    height=400,
                    opacity=0.3,
                ),
            )
        ],
    )
)
# line1.render_notebook()
# The background map can be displayed locally, but the platform only displays the line chart without background. You can copy the code to the local operation
Image.open("./2.png")

From the perspective of hour distribution, we generally choose to comment from the afternoon to the evening. Especially after 17:00, we are still more dedicated during working hours. The peak of the second comment is 22:00, which is a time when young people stay up late are more active, and the work and rest time of young partners is relatively late.

3.4 comments per day of the week

The statistics is the sum of comments on each day of the week from August 1, 2021 to August 31, 2021:

​

# Add field 'week'
dic = {1:'Monday',2:'Tuesday',3:'Wednesday',4:'Thursday',5:'Friday',6:'Saturday',7:'Sunday'}
df['week'] = df['Comment time'].dt.dayofweek+1
df['week'] = df['week'].map(dic)
df.head(5

)
​
 
iduser namecityscorecommentComment timeweek
01142669584Qitong glutinous riceguest5.0The plot is very attractive. Watching an animated cartoon surprised me2021-08-31 23:56:30Tuesday
11142662178LnV14610189Xining5.0Strong picture sense!2021-08-31 23:36:00Tuesday
21142666877Alo861902585Guangzhou5.0And a very good connection, wonderful2021-08-31 23:34:41Tuesday
31142660216Y. Xi'an4.0The characters in the picture don't say that you can always believe in chasing light. The plot is smooth and the overall rhythm is OK. It is recommended to watch -!2021-08-31 23:30:56Tuesday
41142669423I want to see the moon for youFengtai5.0Yes, although Xiaoqing and Xiaobai's obsession is far fetched (OK). If you can elaborate on the obsession of Niu Mo, the plot will be more perfect2021-08-31 23:27:28Tuesday
# Comments per day of the week
dic = {1:'Monday',2:'Tuesday',3:'Wednesday',4:'Thursday',5:'Friday',6:'Saturday',7:'Sunday'}
df['week'] = df['Comment time'].dt.dayofweek+1
df1 = df.sort_values('week',ascending=True)
df_week = df1.groupby(['week'])['comment'].count()
week_x_data = [dic[i] for i in list(df_week.index)]
week_y_data = df_week.values.tolist()
 
line1 = (
    Line(init_opts=opts.InitOpts(bg_color=JsCode(color_js)))
    .add_xaxis(xaxis_data=week_x_data)
    .add_yaxis(
        series_name="",
        y_axis=week_y_data,
        is_smooth=True,
        is_symbol_show=True,
        symbol="circle",
        symbol_size=6,
        linestyle_opts=opts.LineStyleOpts(color="#fff"),
        label_opts=opts.LabelOpts(is_show=True, position="top", color="white"),
        itemstyle_opts=opts.ItemStyleOpts(
            color="red", border_color="#fff", border_width=3
        ),
        tooltip_opts=opts.TooltipOpts(is_show=False),
        areastyle_opts=opts.AreaStyleOpts(color=JsCode(area_color_js), opacity=1),
    )
    .set_global_opts(
        title_opts=opts.TitleOpts(
            title="Comments per day of the week",
            pos_top="5%",
            pos_left="center",
            title_textstyle_opts=opts.TextStyleOpts(color="#fff", font_size=16),
        ),
        xaxis_opts=opts.AxisOpts(
            type_="category",
            boundary_gap=True,
            axislabel_opts=opts.LabelOpts(margin=30, color="#ffffff63"),
            axisline_opts=opts.AxisLineOpts(is_show=False),
            axistick_opts=opts.AxisTickOpts(
                is_show=True,
                length=25,
                linestyle_opts=opts.LineStyleOpts(color="#ffffff1f"),
            ),
            splitline_opts=opts.SplitLineOpts(
                is_show=True, linestyle_opts=opts.LineStyleOpts(color="#ffffff1f")
            ),
        ),
        yaxis_opts=opts.AxisOpts(
            type_="value",
            position="left",
            axislabel_opts=opts.LabelOpts(margin=20, color="#ffffff63"),
            axisline_opts=opts.AxisLineOpts(
                linestyle_opts=opts.LineStyleOpts(width=2, color="#fff")
            ),
            axistick_opts=opts.AxisTickOpts(
                is_show=True,
                length=15,
                linestyle_opts=opts.LineStyleOpts(color="#ffffff1f"),
            ),
            splitline_opts=opts.SplitLineOpts(
                is_show=True, linestyle_opts=opts.LineStyleOpts(color="#ffffff1f")
            ),
        ),
        legend_opts=opts.LegendOpts(is_show=False),
    )
)
line1.render_notebook()

From the data distribution of each day of the week, Mondays and Sundays are the active periods for comments. It is very interesting. The beginning and end of the week start in the break and end in the leisure.

3.5 calendar chart

times = [x.strftime('%Y-%m-%d') for x in list(pd.date_range('20210801', '20210831'))]
data = [[times[index],day_y_data[index]] for index,item in enumerate( day_y_data)]
Cal = (
    Calendar(init_opts=opts.InitOpts(width="800px", height="500px"))
    .add(
        series_name="Distribution of daily comments in August",
        yaxis_data=data,
        calendar_opts=opts.CalendarOpts(
             pos_top='20%',
             pos_left='5%',
             range_="2021-08",
             cell_size=40,
             # Mm / DD / yy label style settings
             daylabel_opts=opts.CalendarDayLabelOpts(name_map="cn",
                                                     margin=20,
                                                     label_font_size=14,
                                                     label_color='#EB1934', 
                                                     label_font_weight='bold'
                                                    ),
             monthlabel_opts=opts.CalendarMonthLabelOpts(name_map="cn",
                                                         margin=20,
                                                         label_font_size=14,
                                                         label_color='#EB1934', 
                                                         label_font_weight='bold',
                                                         is_show=False
                                                        ),
             yearlabel_opts=opts.CalendarYearLabelOpts(is_show=False),
        ),
        tooltip_opts='{c}',
    )
    .set_global_opts(
        title_opts=opts.TitleOpts(
            pos_top="2%", 
            pos_left="center", 
            title=""
        ),
        visualmap_opts=opts.VisualMapOpts(
            orient="horizontal", 
            max_=2000,
            pos_bottom='10%',
            is_piecewise=True,
            pieces=[{"min": 1200},
                    {"min": 800, "max": 1200},
                    {"min": 500, "max": 800},
                    {"min": 300, "max": 500},
                    {"min": 80, "max": 300},
                    {"max": 80}],
            range_color=["#F5F5F5", "#FFE4E1", "#FFCC99", "#F08080", "#CD5C5C", "#990000"]
        ),
        legend_opts=opts.LegendOpts(is_show=True,
                                    pos_top='5%',
                                    item_width = 50,
                                    item_height = 30,
                                    textstyle_opts=opts.TextStyleOpts(font_size=16,color='#EB1934'),
                                    legend_icon ='path://path://M621.855287 587.643358C708.573965 540.110571 768 442.883654 768 330.666667 768 171.608659 648.609267 42.666667 501.333333 42.666667 354.057399 42.666667 234.666667 171.608659 234.666667 330.666667 234.666667 443.22333 294.453005 540.699038 381.59961 588.07363 125.9882 652.794383 21.333333 855.35859 21.333333 1002.666667L486.175439 1002.666667 1002.666667 1002.666667C1002.666667 815.459407 839.953126 634.458526 621.855287 587.643358Z'
                                   ),
    )
)
Cal.render_notebook()

3.6 role heat

Main characters: Xiaobai, Xiaoqing, Xu Xian, Fahai, Sima, sister sun, leader of Niutou Gang, masked man, owner of Baoqing workshop and scholar

roles=['Xiaobai','indigo plant','Xu Xian','Fahai','Sima','Sister sun','Niutou sect leader','Masked man','Baoqing workshop owner','scholar']
content=''.join([str(i) for i in list(df['comment'])])
roles_num=[]
for role in roles:
    count=content.count(role)
    roles_num.append((role,count))
roles_num=pd.DataFrame(roles_num)
roles_num.columns=['name','Number of occurrences']
roles_num
nameNumber of occurrences
0Xiaobai1523
1indigo plant2683
2Xu Xian239
3Fahai396
4Sima112
5Sister sun20
6Niutou sect leader1
7Masked man3
8Baoqing workshop owner101
9scholar4
# Linear gradient
color_js = """new echarts.graphic.LinearGradient(0, 1, 0, 0,
    [{offset: 0, color: '#FFFFFF'}, {offset: 1, color: '#ed1941'}], false)"""

roles_num=roles_num.sort_values(by='Number of occurrences',ascending=False)
roles_num=roles_num.reset_index(drop=True)
b2 = (
        Bar()
        .add_xaxis(list(roles_num['name']))
        .add_yaxis('frequency', list(roles_num['Number of occurrences']),itemstyle_opts=opts.ItemStyleOpts(color=JsCode(color_js)))
        .set_global_opts(title_opts=opts.TitleOpts(title='Frequency distribution of film review roles',pos_top='2%',pos_left = 'center'),
            legend_opts=opts.LegendOpts(is_show=False),
            yaxis_opts=opts.AxisOpts(name="frequency",name_location='middle',name_gap=50,name_textstyle_opts=opts.TextStyleOpts(font_size=16)))

    )
b2.render_notebook()

3.7 geographical distribution of visitors

cities = df['city'].values.tolist()
data = Counter(cities).most_common(80)
geo = (
    Geo(init_opts=opts.InitOpts(width="1000px", height="600px", bg_color="#404a59"))
    .add_schema(maptype="china", 
                itemstyle_opts={
                  'normal': {
                      'shadowColor': 'rgba(0, 0, 0, .5)', 
                      'shadowBlur': 5, 
                      'shadowOffsetY': 0, 
                      'shadowOffsetX': 0, 
                      'borderColor': '#fff'
                  }
              }
               )
    .add("Number of comments", data,type_=ChartType.HEATMAP,)
    .set_series_opts(label_opts=opts.LabelOpts(is_show=False))
    .set_global_opts(
       title_opts=opts.TitleOpts(title="Geographical distribution",pos_top="2%", pos_left="center",
                                 title_textstyle_opts=opts.TextStyleOpts(color="#fff", font_size=16)),
       legend_opts=opts.LegendOpts(is_show=False),
       visualmap_opts=opts.VisualMapOpts(
            is_show=True,
            is_piecewise=True,
            min_ = 0,
            max_ = 500,
            split_number = 5,
            series_index=0,
            pos_bottom='5%',
            pos_left='5%',
            textstyle_opts=opts.TextStyleOpts(color="#fff"),
            pieces=[
                {'max':500, 'min':401, 'label':'401-500', 'color': '#990000'},
                {'max':400, 'min':301, 'label':'301-400', 'color': '#CD5C5C'},
                {'max':300, 'min':201, 'label':'201-300', 'color': '#F08080'},
                {'max':200, 'min':101, 'label':'101-200', 'color': '#FFCC99'},
                {'max':100, 'min':0, 'label':'0-100', 'color': '#FFE4E1'},
               ],
            ),
    )
)
geo.render_notebook()

From the geographical distribution map, the audience is mainly distributed in Beijing, Tianjin, Shanghai, Chongqing, Sichuan, Guangdong, Yunnan and other places.

Industry data: add to get PPT template, resume template, industry classic book PDF.
Interview question bank: the classic and hot real interview questions of large factories over the years are continuously updated and added.
Learning materials: including Python, crawler, data analysis, algorithm and other learning videos and documents, which can be added and obtained
Communication plus group: the boss points out the maze. Your problems are often encountered by others. Technical assistance and communication.

Topics: Python Data Analysis pandas pyecharts