Data acquisition and fusion technology_ Experiment 3

Posted by BlueKai on Wed, 10 Nov 2021 02:11:19 +0100

Code cloud link: Experiment 3

1. Operation ①

1.1 operation content

  • Content: specify a website and crawl all the pictures in the website, such as China Meteorological Network( ). Single thread and multi thread crawling are used respectively. (the number of crawling pictures is limited to the last 4 digits of the student number)
  • Output information: output the downloaded Url information on the console, store the downloaded image in the images sub file, and give a screenshot.

1.2 problem solving ideas

1.2.1 observe and analyze the main page and find that the picture is under the img tag and the sub website is in the href of the a tag

1.2.2 while traversing the website, compile spider function to crawl some pictures from each website

soup = BeautifulSoup(data, "lxml")
images ="img")
links ="a")
if len(images) >= 5:
    for i in range(5):
        if images[i]["src"].startswith("http") and images[i]["src"] not in imgs:
    if len(imgs) >= 121 or deep == 0:
for link in links:
    if link["href"].startswith("http"):

1.2.3 single thread downloading pictures

def download():
    global imgs
    for i in range(121):
        request.urlretrieve(imgs[i],r"D:\data acquisition\demo\practice\3\3.1\images\The first"+str(i+1)+"Zhang Tu.jpg")
        print("downloaded ", imgs[i])

1.2.4 multi thread downloading pictures

def download_threads():
    global imgs
    for i in range(121):
        T = threading.Thread(target=download, args=(imgs[i], i+1))

1.2.5 results

1.3 experience

  • Single thread downloads pictures in order, which is very simple and intuitive, and the speed is slow. Multi thread does not necessarily execute in order, but the download speed is fast
  • Familiar with the traversal of the website and consolidated the understanding of multithreaded crawler

2. Operation ②

2.1 operation contents

Request: use the sketch framework to reproduce the job ①. Output information: the same as operation ①

2.2 problem solving ideas

2.2.1 set the unique parameter Src in item_ URL to store the picture address and write mySpider

class mySpider(scrapy.Spider):
    name = "mySpider"
    count = 0
    imgs = []
    def start_requests(self):
        url = ""
        yield scrapy.Request(url=url, callback=self.parse, meta={"deep":3})

    def parse(self, response):
            deep = response.meta["deep"]
            if deep <= 0 or self.count >= 121:
            data = response.body.decode()
            selector = scrapy.Selector(text=data)
            images = selector.xpath("//img/@src").extract()
            n = 0
            for i in images:
                if i.startswith("http") and i not in self.imgs:
                    item = DemoItem()
                    item["src_url"] = i
                    self.count += 1
                    n += 1
                    if n > 5:
                    yield item
            links = selector.xpath("//a/@href").extract()
            for link in links:
                if link.startswith("http"):
                    url = response.urljoin(link)
                    yield scrapy.Request(url=url, callback=self.parse, meta={"deep":deep-1})

        except Exception as err:

2.2.2 download pictures in pipelines

class DemoPipeline:
    count = 1
    def process_item(self, item, spider):
            if self.count <= 121:
                with open("./images/The first" + str(self.count) + "Zhang Tu.jpg", "wb") as f:
                    img = requests.get(item["src_url"]).content
                    self.count += 1
        except Exception as err:
        return item

2.2.3 modifying setting

BOT_NAME = 'demo'

SPIDER_MODULES = ['demo.spiders']
NEWSPIDER_MODULE = 'demo.spiders'


   'demo.pipelines.DemoPipeline': 300,

2.2.4 output results

2.3 experience

  • After reviewing the scratch crawler framework and learning how to use xpath, I feel that skilled xpath will be more convenient than CSS
  • Understand the parameter transmission mechanism of parse in scratch

Operation ③

3.1 operation contents

  • Requirements: crawl the Douban movie data, use scene and xpath, store the content in the database, and store the pictures in the imgs path.
    All college information is stored in the database, and the process recording Gif of browser F12 debugging and analysis is added to the blog.

  • Output information:

    Serial number Movie title director performer brief introduction Film rating Film cover
    1 The Shawshank Redemption Frank delabond Tim Robbins Want to set people free 9.7 ./imgs/xsk.jpg

3.2 problem solving ideas

3.2.1 analyze the web page content and find the location of the required content

3.2.2 write movieItem

class MovieItem(scrapy.Item):
    rank = scrapy.Field()
    name = scrapy.Field()
    director = scrapy.Field()
    actor = scrapy.Field()
    score = scrapy.Field()
    img = scrapy.Field()
    desp = scrapy.Field()

3.2.3 compiling spider

Crawl to page 10

    def start_requests(self):
        for i in range(10):
            url = "" + str(i * 25)
            yield scrapy.Request(url=url,callback=self.parse)

parse function

    def parse(self, response):
            data = response.body.decode()
            selector = scrapy.Selector(text=data)
            # Get each movie information
            movies = selector.xpath("//li/div[@class='item']")
            for m in movies:
                rank = m.xpath("./div[@class='pic']/em/text()").extract_first()
                image = m.xpath("./div[@class='pic']/a/img/@src").extract_first()
                name = m.xpath("./div[@class='info']//span[@class='title']/text()").extract_first()
                members = m.xpath("./div[@class='info']//p[@class='']/text()").extract_first()
                desp = m.xpath("./div[@class='info']//p[@class='quote']/span/text()").extract_first()
                score = m.xpath("./div[@class='info']//span[@class='rating_num']/text()").extract_first()
                item = MovieItem()
                item['rank'] = rank
                item['name'] = name
                director ='director:(.*?)\s main', members).group(1)
                actor ='to star:(.*)', members)
                item['director'] = director
                #actor may be empty because there are cartoons without actors
                if actor == None:
                    item['actor'] = "null"
                    item["actor"] =
                item['desp'] = desp
                item['score'] = score
                item['img'] = image
                yield item

        except Exception as err:

3.2.4 compiling database classes

class MovieDB:
    def openDB(self):
        self.con = sqlite3.connect("movies.db")
        self.cursor = self.con.cursor()
            self.cursor.execute("create table movies (Rank int,Name varchar(32),Director varchar(32),"
                                "Actors varchar(64),Description varchar(64),Score varchar(8),ImgPath varchar(64))")
            self.cursor.execute("delete from movies")

    def closeDB(self):

    def insert(self, Rank,Name,Director,Actors,Description,Score,ImgPath):
            self.cursor.execute("insert into movies (Rank,Name,Director,Actors,Description,Score,ImgPath) "
                                "values (?,?,?,?,?,?,?)", (Rank,Name,Director,Actors,Description,Score,ImgPath))
        except Exception as err:

3.2.5 compiling pipelines

class MoviespiderPipeline:
    def open_spider(self, spider):
        self.db = MovieDB()

    def process_item(self, item, spider):
        path = r"./images/"+item['name']+".jpg"
        url = item['img']
        img = requests.get(url).content
        with open(path,"wb") as f:
            print("The first"+item['rank']+"Page cover downloaded successfully")
        print("The first" + item['rank'] +'Movie data inserted successfully')
        return item

    def close_spider(self, spider):
        print("End crawling")

3.2.6 modifying setting
3.2.7 output results

3.3 experience

  • More proficient in the use of scratch and xpath