Analysis of Douban comments on TV series of procedural ape

Posted by sriusa on Mon, 21 Feb 2022 07:44:37 +0100

The original text was posted on the TesterHome forum by xinxi, Original link


During the Spring Festival holiday, I chased a TV series "perfect partner", starring the goddess "Gao Yuanyuan". There are about 45 episodes. I feel that the overall plot is OK, and the acting skills and appearance of the goddess are still online

General plot introduction:

Chen Shan is a female lawyer in the Securities Department of a famous law firm. At the age of nearly 40, she has encountered a bottleneck in her rise. Therefore, she devoted herself to her work and had no time to take into account the emotional needs of her husband and daughter.
Her husband sun Lei worked in a state-owned enterprise. In order to support his wife's career, he undertook the important task of taking care of his family and almost gave up his career pursuit. Over the past decade, the marriage model of strong women and weak men has been running well and gradually out of balance. Unconsciously, it has been overwhelmed. At this time, investment banking elite Lin Qingkun appeared, and the big order in his hand became the target of Chen Shan and his competitors.
Facing the dual crisis of marriage and workplace, Chen Shan tried hard to deal with both at the same time, but it was more and more counterproductive. After many tests, sun Lei and Chen Shan,
Finally, it is found that their original heart for each other is still there. As long as they cherish and support each other, they are the most perfect partners for each other.

At that time, the play was also very bloody in some places, including praise and scolding. I want to analyze a wave through the comments of netizens.

data sources

You can find thousands of comments on Douban. It should be enough to climb down

Douban address

View request

Take a look at the data interface and data format

data format

On the h5 page, a comment is composed of avatar, score and comment.

When viewing the returned data, it does not return a standard kv structure data, but a lump of html structure, and then the front end will render it.

Copy curl

The purpose of copying curl is to facilitate debugging scripts through postman

Using header

Reuse header header to get data in script


Dependency Library


pip3 install jieba (participle)  
pip3 install httpx (Send network request)
pip3 install logzero (journal)
pip3 install beautifulsoup4 (analysis html)
pip3 install wordcloud (Generate word cloud)
pip3 install pyecharts (Generate chart)

Get comments circularly, query the data by the number of start s, get the corresponding span tag data through BeautifulSoup, and finally get class="short" is the comment

def get_info_by_comments():
    Get comments
    Page_INDEX = 0
    IS_RUNNING = True
    while IS_RUNNING:
        url = HOST + "/subject/35196566/comments?percent_type=&start={}&limit=20&status=P&sort=new_score&comments_only=1".format(
        Page_INDEX = Page_INDEX + 1
        r = httpx.get(url, headers=headers, timeout=10)
        if 'html' in r.text:
            html = r.json()['html']
            soup = BeautifulSoup(html, 'html.parser')
            for k in soup.find_all('span'):
                if 'class="short"' in str(k):
                    comment = str(k.string).encode()
                    write_content_to_file(FilePath, comment, is_cover=False)
                    write_content_to_file(FilePath, "\n".encode(), is_cover=False)
            IS_RUNNING = False

Get comments

Use the script to save the comments offline in the local file for subsequent analysis

Word cloud generated from comments

Generate word cloud script

with open(CommentsFilePath, 'r') as f:
    r_info = f.readlines()

sentence = ''.join(r_info)
# Use jieba to segment sentences
font_path = "/System/Library/fonts/PingFang.ttc"
word = jieba.cut(sentence)
words = " ".join(word)
# Generate the nd array of the picture and pass in the picture path
im = imageio.imread('test.jpeg')
wc = wordcloud.WordCloud(width=600, height=800, background_color='white', font_path=font_path, mask=im,
                         contour_width=1, contour_color='black')

jieba will be used here for word segmentation. wordcloud generates word cloud by the number of words after segmentation

Get score

I also want to see how the scores are distributed

The script to get the score is as follows:

if 'html' in r.text:
    html = r.json()['html']
    soup = BeautifulSoup(html, 'html.parser')
    for k in soup.find_all('span'):
        if k.get('class') != None:
            # Find score
            if 'allstar' in str(k.get('class')):
                title = str(k.get('title')).encode()

Generate scoring pie chart through pyechards

with open(STARFilePath, 'r') as f:
    r_info = f.readlines()

QITA = 0

for i in r_info:
    STAR = i.replace("\n", "")
    if STAR == "Poor":
        JIAOCHA = JIAOCHA + 1
    elif STAR == "it 's not bad":
        HAIXING = HAIXING + 1
    elif STAR == "recommend":
        TUIJIAN = TUIJIAN + 1
    elif STAR == "Very bad":
        HENCHA = HENCHA + 1
    elif STAR == "recommend strongly":
        LIJIAN = LIJIAN + 1
        QITA = QITA + 1
pie = (
        .add("", [('Very bad', HENCHA), ('Poor', JIAOCHA), ('it 's not bad', HAIXING), ('recommend', TUIJIAN), ('recommend strongly', LIJIAN)])
        .set_global_opts(title_opts=opts.TitleOpts(title="Score distribution"))

pie.render(path=RootPath + "/star_pie.html")

Get comment time and generate chart

I also want to see the time period of netizens' comments, according to the hour dimension

The data obtained is to seconds, and the time processing is as follows:

1. Filter by hour

2. Sorted by time

comment_time_list = []
with open(TIMEFilePath, 'r') as f:
    r_info = f.readlines()
for i in r_info:
    comment_time = i.replace("\n", "").split(":")[0] + ":00:00"  # Filter by hour

comment_dict = {}
comment_dict_list = []
for i in comment_time_list:
    if i not in comment_dict.keys():
        comment_dict[i] = comment_time_list.count(i)

for k in comment_dict:
    comment_dict_list.append({"time": k, "value": comment_dict[k]})

comment_dict_list.sort(key=lambda k: (k.get('time', 0)))
# print(comment_dict_list) # Sorted by time
time_list = []
value_list = []
for c in comment_dict_list:

c = (
    Bar(init_opts=opts.InitOpts(width="1600px", height="800px"))
        .add_yaxis("quantity", value_list, category_gap=0, color=Faker.rand_color())
        .set_global_opts(title_opts=opts.TitleOpts(title="Comment time-histogram"),
                         xaxis_opts=opts.AxisOpts(name="Submission time", axislabel_opts={"rotate": 45}))


The line chart has no smooth curve set

Line chart and set smooth curve

Get the most words
Also want to see what are the "most words"

1. Use jieba for word segmentation

2. Filtered modal particle

Including modal particles, which is not accurate. Some words have no meaning

Remove modal particles and filter some commonly used modal particles

import jieba

def stopwordslist():
    Create a stop word list
    stopwords = [line.strip() for line in open('chinese_stopword.txt', encoding='UTF-8').readlines()]
    return stopwords

def seg_depart(sentence):
    Chinese word segmentation of sentences
    :param sentence: 
    print("Participle in progress")
    out_str_list = []
    sentence_depart = jieba.cut(sentence.strip())
    stopwords = stopwordslist()
    outstr = ''
    # De stop word
    for word in sentence_depart:
        if word not in stopwords:
            if word != '\t':
                outstr += word
                outstr += " "
    return out_str_list


Through the analysis of a TV drama review, we can get:

1. python Network Request exercise

2. python common data structure exercises

3. Chart Library of python

4. Common algorithm exercises

Finally, after recommending this play, the male host 35 + gives up his career for his wife's career, takes care of his children, and balances family and career

The word "family" appears most in the last pie chart, which is associated with the high pressure of our internet work and often work overtime and ignore the company of our family

I wish you all work overtime and spend more time with your family~

Topics: Python