Problems with Scrapy crawling image custom image file names

Posted by objNoob on Wed, 05 Jan 2022 08:29:48 +0100

catalogue

Problem: only one picture is downloaded. The name of the picture is the name of the last picture, but the content is not the content of the last picture. When printing, the relevant information of the last picture appears multiple times.

Relevant information of corresponding documents:

Items file related content:

Spider file content:

Settings file content:

Relevant contents of pipelies file:

At first, I wrote this. I took the last paragraph of the image path as the file name. There was no big problem

But I don't think it looks good. I plan to use their original names as file names

The final writing method is not to pass in the entire item, but the corresponding field content in the item object

Problem: only one picture is downloaded. The name of the picture is the name of the last picture, but the content is not the content of the last picture. When printing, the relevant information of the last picture appears multiple times.

Relevant information of corresponding documents:

Items file related content:

# Define here the models for your scraped items
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.html

import scrapy


class FabiaoqingItem(scrapy.Item):
    # define the fields for your item here like:
    images = scrapy.Field()
    image_urls = scrapy.Field()
    # pass

Spider file content:

import scrapy
from first.items import FabiaoqingItem

class FabiaoqingSpider(scrapy.Spider):
    name = 'fabiaoqing'
    # allowed_domains = ['https://www.fabiaoqing.com/']
    start_urls = ['https://www.fabiaoqing.com/biaoqing']

    def parse(self, response):
        # Get item object
        item = FabiaoqingItem()
        # Get all expression pack information
        imgs = response.css('.tagbqppdiv')
        for img in imgs:
            # title
            title = img.css('a::attr(title)').extract()
            # route
            src = img.css('a img::attr(data-original)').extract()
            # Extract the contents of the list into a string
            title = ''.join(title)
            src = ''.join(src)
            # print(title, src)
            # Encapsulate the obtained content into the item object
            item['images'] = title
            item['image_urls'] = src
            # Submit item to pipeline
            yield item


# Highlight: during the implementation process, no error is reported and no result is given. Find a warning:
# WARNING: Disabled FabiaoqingPipeline: ImagesPipeline requires installing Pillow 4.0.0 or later
# This means that the version of pilot 4.0.0 or above is not installed in the system. You need to install it manually: PIP install pilot

Settings file content:

# Scrapy settings for first project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#     https://docs.scrapy.org/en/latest/topics/settings.html
#     https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#     https://docs.scrapy.org/en/latest/topics/spider-middleware.html

# Console log output level
import random

# Directory where picture files are saved
IMAGES_STORE = './images'
# Limit log output level
LOG_LEVEL = 'ERROR'

BOT_NAME = 'first'

SPIDER_MODULES = ['first.spiders']
NEWSPIDER_MODULE = 'first.spiders'

# Crawl responsibly by identifying yourself (and your website) on the user-agent
# USER_AGENT = 'first (+http://www.yourdomain.com)'
# User agent list

USER_AGENT_LIST = [
    # Edge
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36 Edg/96.0.1054.62',
    # Firefox
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:95.0) Gecko/20100101 Firefox/95.0',
    # Chrome
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'
]
# Random user agent
USER_AGENT = random.choice(USER_AGENT_LIST)

# Obey robots.txt rules
# ROBOTSTXT_OBEY = True
# Compliance with robots protocol
ROBOTSTXT_OBEY = False

# Configure maximum concurrent requests performed by Scrapy (default: 16)
# CONCURRENT_REQUESTS = 32

# Configure a delay for requests for the same website (default: 0)
# See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
# DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
# CONCURRENT_REQUESTS_PER_DOMAIN = 16
# CONCURRENT_REQUESTS_PER_IP = 16

# Disable cookies (enabled by default)
# COOKIES_ENABLED = False

# Disable Telnet Console (enabled by default)
# TELNETCONSOLE_ENABLED = False

# Override the default request headers:
# DEFAULT_REQUEST_HEADERS = {
#   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
#   'Accept-Language': 'en',
# }

# Enable or disable spider middlewares
# See https://docs.scrapy.org/en/latest/topics/spider-middleware.html
# SPIDER_MIDDLEWARES = {
#    'first.middlewares.FirstSpiderMiddleware': 543,
# }

# Enable or disable downloader middlewares
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
# DOWNLOADER_MIDDLEWARES = {
#    'first.middlewares.FirstDownloaderMiddleware': 543,
# }

# Enable or disable extensions
# See https://docs.scrapy.org/en/latest/topics/extensions.html
# EXTENSIONS = {
#    'scrapy.extensions.telnet.TelnetConsole': None,
# }

# Configure item pipelines
# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
# Project pipeline
ITEM_PIPELINES = {
    # The lower the value, the higher the priority
    # 'first.pipelines.FirstPipeline': 300,
    # 'first.pipelines.ToMysqlPipeline': 301,
    'first.pipelines.FabiaoqingPipeline': 302,
}

# Enable and configure the AutoThrottle extension (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/autothrottle.html
# AUTOTHROTTLE_ENABLED = True
# The initial download delay
# AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
# AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
# AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
# AUTOTHROTTLE_DEBUG = False

# Enable and configure HTTP caching (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
# HTTPCACHE_ENABLED = True
# HTTPCACHE_EXPIRATION_SECS = 0
# HTTPCACHE_DIR = 'httpcache'
# HTTPCACHE_IGNORE_HTTP_CODES = []
# HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'

Relevant contents of pipelies file:

At first, I wrote this. I took the last paragraph of the image path as the file name. There was no big problem

import scrapy
from scrapy.pipelines.images import ImagesPipeline


class FabiaoqingPipeline(ImagesPipeline):
    """Download and publish the expression package pipeline class of love network
    """

    def get_media_requests(self, item, info):
        """Request for each emoticon packet
        """
        return [scrapy.Request(url=item['image_urls'])]

    def file_path(self, request, response=None, info=None, *, item=None):
        """Path of picture saving (sub path)
        """

        # Take the last segment of the download path as the file name
        img_name = request.url.split('/')[-1]
        print(img_name)
        return img_name

    def item_completed(self, results, item, info):
        """take item Pass to next pipe class
        """
        return item

Print results:
ab4b8062gy1gxlavqgxgbj20fb0fbmxu.jpg
008h4Hg2gy1gxb1gzxuz0g309d095nin.gif
ceeb653ely8gxvsn09kndj20g30eodge.jpg
ceeb653ely8gxx3gluh1jg208c06v74o.gif
006APoFYly8gxxjk5ks1fg30c80c8q4r.gif
ceeb653ely8gxvsdnb5orj20g30eojs0.jpg
ceeb653ely8gx8ywsjfthg20e80e84qp.gif
ceeb653ely8gx44pqmmhhj206o06omx9.jpg
006APoFYly8gwzyusrfg7j30oo0lgjt6.jpg
006C7PHRly1gw2df87078g30900904k8.gif
003MWcpMly8gv33zza8pzj608c08g74j02.jpg
ceeb653ely8gx8zv4cgxug2064064mzt.gif
ceeb653ely8gwa4c5uhbrj20j80j6aau.jpg
ceeb653ely8gwx3sekvz5j20j20j275b.jpg
ceeb653ely8gwc75zwiupj20dw0dwq3h.jpg
ceeb653ely8gwbv0zgjt3j20k00k0aao.jpg
ceeb653ely8gvwshx6ufng207u06u74m.gif
006APoFYly8gvxgba126cg308c05a0yc.gif
006GJQvhgy1gvvwkivvpqj328g28g18h.jpg
a6653c0dly1gvw1qv571hg206o062an5.gif
6958e69dgy1gvz2qe32vwj20go0gpgnn.jpg
006APoFYly8gvz9fi8a6uj30hs0hj0ty.jpg
006APoFYly8gwkftnnywag308m0aqqv8.gif
0078qrzmgy1gwdxp8p79sg303m03m74f.gif
006APoFYly8gxd8ufvm23j30j60j6jsa.jpg
ceeb653ely8gw2o2b5f61j20jz0j5dgr.jpg
ceeb653ely8gwlimzamsqg20a00a07nc.gif
ceeb653ely8gw2xy677q7g208w06ogtx.gif
003MWcpMly8gv903csftwg60670674ht02.gif
006Mi9iRgy1guvla2fgdog60ez0g0q9102.gif
0073Cjx6gy1gw262v18waj30th139gqr.jpg
ceeb653ely8gw335ge6jaj20d90hswfj.jpg
006APoFYly8gw12oxpu2yg301e01etca.gif
006APoFYly8gw13bu7ztvj30hs0hswfe.jpg
006APoFYly8gx8ngr1wc0g303403476m.gif
ceeb653ely8gx80qxpxahg20db0dbasj.gif
0068Lfdegy1gwio2wr0rjj30u00u0jv1.jpg
006APoFYly8gw2hau44pdg306o06o74u.gif
ceeb653ely8gx95ui88qwg209806kaac.gif
ceeb653ely8gx90naj760j20ix0njdh5.jpg
006APoFYly8gx901bj2i1g30790797wh.gif
006APoFYly8gx8nbk4948g307k05d4qp.gif
ceeb653ely8gx7ewc23ttg206o06ijtp.gif
ceeb653ely8gvzktgrq7tj206o06oq2y.jpg
0068Lfdegy1gvvi5yqolwj30u00u0djv.jpg

Open the folder and have a look

45 in total, no problem

But I don't think it looks good. I plan to use their original names as file names

Let's delete the previous pictures first

return [scrapy.Request(url=item['image_urls'], meta={'item': item})]

import re
import scrapy
from scrapy.pipelines.images import ImagesPipeline


class FabiaoqingPipeline(ImagesPipeline):
    """Download and publish the expression package pipeline class of love network
    """

    def get_media_requests(self, item, info):
        """Request for each emoticon packet
        """
        # return [scrapy.Request(url=item['image_urls'])]
        # Pass in meta to prepare for subsequent customization file names
        return [scrapy.Request(url=item['image_urls'], meta={'item': item})]

    def file_path(self, request, response=None, info=None, *, item=None):
        """Path of picture saving (sub path)
        """

        # Take the last segment of the download path as the file name
        # img_name = request.url.split('/')[-1]
        # print(img_name)
        # return img_name

        # Custom file name
        item = request.meta['item']     # Get item object from meta
        img_name = item['images']
        # The names of some expression packs contain special characters, so they cannot be saved and need to be changed
        img_name = re.sub(r'[\/:*?"<>|\n]', '', img_name)
        # Because some are gif and some are jpg, the file suffix cannot be written dead
        img_type = request.url.split('.')[-1]
        img_path = img_name + '.' + img_type
        print(img_path)
        return img_path

    def item_completed(self, results, item, info):
        """take item Pass to next pipe class
        """
        return item

Print results:
I'll fucking kill you.jpg
 Your father is here.gif
 Call me baby, you won't be alone in the new year.jpg
2022 Must be rich(Panda head expression pack).gif
 It's 2022.gif
 Call me wife for the new year, and you won't be alone.jpg
 Your father is here. Your father is here.gif
 See you next time when you have insomnia.jpg
 Although I roll, I'm still a vegetable.jpg
 Patrick Star Q elasticのAss, run.gif
 You want to break my defense.jpg
 Knock on your head.gif
 Why not? No. 1 in Internet surfing.jpg
 Very yellow vegetable dog.jpg
 Finger line!(Specify row).jpg
 Silent egg(Sweat expression bag).jpg
 I really want to fit in with you, but I have to work overtime.gif
 Panda head expression pack of the last hero League mobile game.gif
 congratulations!You've been visited by a trapped cat!From now on, no matter what you're doing, you'll be sleepy!.jpg
 Cheers to our speechless.gif
 You're so cute, I must knock, so I'm cute.jpg
 Baby mouth a happy s la(Kulomi expression pack).jpg
 Wife, open the door. It's me. I'm back. Open the door? Didn't drink? Guilty Xiugou knocks on the door.gif
 Dripping sweat little yellow face Christmas clothes Christmas hat expression bag.gif
2022 Must be able to love panda head expression bag.jpg
 Your eyes are full of you.jpg
 Fart attack.gif
 Young master and young lady get up.gif
3d Refer to human expression pack warning once.gif
 Tickle your ass dynamic picture expression pack.gif
 Da baa color.jpg
 The person pointed out did three anal lifting exercises, right now.jpg
 Cat probe observation GIF Animation expression pack.gif
 Look, I'm angry again.jpg
 Wordless sweat expression pack.gif
Ji Bad guy bad guy big bad guy.gif
 No one likes me.jpg
 The leaping legs jumped up GIF Dynamic graph.gif
1)60 There's something in that(Wechat chat box expression pack).gif
 Berry thing(Fruit homophonic expression pack).jpg
 Poggy crying emoticon pack.gif
 Hug charging.gif
 Sad frog gives you a punch.gif
 From time to time, the gang came to visit white prostitutes(Group chat expression pack).jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.jpg
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif
 Mom! Big mouth cat expression bag.gif

It can be seen that a very strange phenomenon. The first 44 pictures are normal, but there are a lot of repetitions in 45 pictures. More importantly, only 2 pictures have been downloaded. Let's have a look

The file name is the name of the last picture, and the file name does not correspond to the content

The final writing method is not to pass in the entire item, but the corresponding field content in the item object

return [scrapy.Request(url=item['image_urls'], meta={'images': item['images']})]

import re
import scrapy
from scrapy.pipelines.images import ImagesPipeline


class FabiaoqingPipeline(ImagesPipeline):
    """Download and publish the expression package pipeline class of love network
    """

    def get_media_requests(self, item, info):
        """Request for each emoticon packet
        """
        # return [scrapy.Request(url=item['image_urls'])]
        # Pass in meta to prepare for subsequent customization file names
        # return [scrapy.Request(url=item['image_urls'], meta={'item': item})]
        return [scrapy.Request(url=item['image_urls'], meta={'images': item['images']})]

    def file_path(self, request, response=None, info=None, *, item=None):
        """Path of picture saving (sub path)
        """

        # Take the last segment of the download path as the file name
        # img_name = request.url.split('/')[-1]
        # print(img_name)
        # return img_name

        # # Custom file name
        # item = request.meta['item']     # Get item object from meta
        # img_name = item['images']
        # # The names of some expression packs contain special characters, so they cannot be saved and need to be changed
        # img_name = re.sub(r'[\/:*?"<>|\n]', '', img_name)
        # # Because some are gif and some are jpg, the file suffix cannot be written dead
        # img_type = request.url.split('.')[-1]
        # img_path = img_name + '.' + img_type
        # print(img_path)
        # return img_path

        # That's it. Strange
        img_name = request.meta['images']
        # The names of some expression packs contain special characters, so they cannot be saved and need to be changed
        img_name = re.sub(r'[\/:*?"<>|\n]', '', img_name)
        # Because some are gif and some are jpg, the file suffix cannot be written dead
        img_type = request.url.split('.')[-1]
        img_path = img_name + '.' + img_type
        print(img_path)
        return img_path

    def item_completed(self, results, item, info):
        """take item Pass to next pipe class
        """
        return item

Print results:
I'll fucking kill you.jpg
 Your father is here.gif
 Call me baby, you won't be alone in the new year.jpg
2022 Must be rich(Panda head expression pack).gif
 It's 2022.gif
 Call me wife for the new year, and you won't be alone.jpg
 Your father is here. Your father is here.gif
 See you next time when you have insomnia.jpg
 Although I roll, I'm still a vegetable.jpg
 Patrick Star Q elasticのAss, run.gif
 You want to break my defense.jpg
 Knock on your head.gif
 Why not? No. 1 in Internet surfing.jpg
 Very yellow vegetable dog.jpg
 Finger line!(Specify row).jpg
 Silent egg(Sweat expression bag).jpg
 I really want to fit in with you, but I have to work overtime.gif
 Panda head expression pack of the last hero League mobile game.gif
 congratulations!You've been visited by a trapped cat!From now on, no matter what you're doing, you'll be sleepy!.jpg
 Cheers to our speechless.gif
 You're so cute, I must knock, so I'm cute.jpg
 Baby mouth a happy s la(Kulomi expression pack).jpg
 Wife, open the door. It's me. I'm back. Open the door? Didn't drink? Guilty Xiugou knocks on the door.gif
 Dripping sweat little yellow face Christmas clothes Christmas hat expression bag.gif
2022 Must be able to love panda head expression bag.jpg
 Your eyes are full of you.jpg
 Fart attack.gif
 Young master and young lady get up.gif
3d Refer to human expression pack warning once.gif
 Tickle your ass dynamic picture expression pack.gif
 Da baa color.jpg
 The person pointed out did three anal lifting exercises, right now.jpg
 Cat probe observation GIF Animation expression pack.gif
 Look, I'm angry again.jpg
 Wordless sweat expression pack.gif
Ji Bad guy bad guy big bad guy.gif
 No one likes me.jpg
 The leaping legs jumped up GIF Dynamic graph.gif
1)60 There's something in that(Wechat chat box expression pack).gif
 Berry thing(Fruit homophonic expression pack).jpg
 Poggy crying emoticon pack.gif
 Hug charging.gif
 Sad frog gives you a punch.gif
 From time to time, the gang came to visit white prostitutes(Group chat expression pack).jpg
 Mom! Big mouth cat expression bag.jpg
 I'll fucking kill you.jpg
 I'll fucking kill you.jpg
 Call me baby, you won't be alone in the new year.jpg
 Call me baby, you won't be alone in the new year.jpg
2022 Must be rich(Panda head expression pack).gif
2022 Must be rich(Panda head expression pack).gif
 It's 2022.gif
 It's 2022.gif
 Your father is here.gif
 Your father is here.gif
 Look, I'm angry again.jpg
 Look, I'm angry again.jpg
Ji Bad guy bad guy big bad guy.gif
Ji Bad guy bad guy big bad guy.gif
 The leaping legs jumped up GIF Dynamic graph.gif
 The leaping legs jumped up GIF Dynamic graph.gif
 No one likes me.jpg
 No one likes me.jpg
 Da baa color.jpg
 Da baa color.jpg
 See you next time when you have insomnia.jpg
 See you next time when you have insomnia.jpg
 You want to break my defense.jpg
 You want to break my defense.jpg
 I really want to fit in with you, but I have to work overtime.gif
 I really want to fit in with you, but I have to work overtime.gif
 Dripping sweat little yellow face Christmas clothes Christmas hat expression bag.gif
 Dripping sweat little yellow face Christmas clothes Christmas hat expression bag.gif
 Silent egg(Sweat expression bag).jpg
 Silent egg(Sweat expression bag).jpg
 Very yellow vegetable dog.jpg
 Very yellow vegetable dog.jpg
 Why not? No. 1 in Internet surfing.jpg
 Why not? No. 1 in Internet surfing.jpg
2022 Must be able to love panda head expression bag.jpg
2022 Must be able to love panda head expression bag.jpg
 Baby mouth a happy s la(Kulomi expression pack).jpg
 Baby mouth a happy s la(Kulomi expression pack).jpg
 Your father is here. Your father is here.gif
 Your father is here. Your father is here.gif
 Call me wife for the new year, and you won't be alone.jpg
 Call me wife for the new year, and you won't be alone.jpg
 You're so cute, I must knock, so I'm cute.jpg
 You're so cute, I must knock, so I'm cute.jpg
 Tickle your ass dynamic picture expression pack.gif
 Tickle your ass dynamic picture expression pack.gif
 Although I roll, I'm still a vegetable.jpg
 Although I roll, I'm still a vegetable.jpg
 The person pointed out did three anal lifting exercises, right now.jpg
 The person pointed out did three anal lifting exercises, right now.jpg
 Finger line!(Specify row).jpg
 Finger line!(Specify row).jpg
 congratulations!You've been visited by a trapped cat!From now on, no matter what you're doing, you'll be sleepy!.jpg
 congratulations!You've been visited by a trapped cat!From now on, no matter what you're doing, you'll be sleepy!.jpg
 From time to time, the gang came to visit white prostitutes(Group chat expression pack).jpg
 From time to time, the gang came to visit white prostitutes(Group chat expression pack).jpg
 Berry thing(Fruit homophonic expression pack).jpg
 Berry thing(Fruit homophonic expression pack).jpg
 Your eyes are full of you.jpg
 Your eyes are full of you.jpg
1)60 There's something in that(Wechat chat box expression pack).gif
1)60 There's something in that(Wechat chat box expression pack).gif
 Wordless sweat expression pack.gif
 Wordless sweat expression pack.gif
 Cat probe observation GIF Animation expression pack.gif
 Cat probe observation GIF Animation expression pack.gif
 Sad frog gives you a punch.gif
 Sad frog gives you a punch.gif
 Panda head expression pack of the last hero League mobile game.gif
 Panda head expression pack of the last hero League mobile game.gif
 Knock on your head.gif
 Knock on your head.gif
 Cheers to our speechless.gif
 Cheers to our speechless.gif
 Young master and young lady get up.gif
 Young master and young lady get up.gif
 Patrick Star Q elasticのAss, run.gif
 Patrick Star Q elasticのAss, run.gif
 Poggy crying emoticon pack.gif
 Poggy crying emoticon pack.gif
 Hug charging.gif
 Hug charging.gif
3d Refer to human expression pack warning once.gif
3d Refer to human expression pack warning once.gif
 Fart attack.gif
 Fart attack.gif
 Wife, open the door. It's me. I'm back. Open the door? Didn't drink? Guilty Xiugou knocks on the door.gif
 Wife, open the door. It's me. I'm back. Open the door? Didn't drink? Guilty Xiugou knocks on the door.gif

The result of this printing is even more strange. Basically, there is a repetition in each

But! What's more strange is that the result is no problem. Let's have a look

There are 45 items in total. There is no problem in the quantity. The file name is also user-defined. It is their original name and there is no problem in the content. It is one-to-one correspondence, which is very inexplicable!

Obviously, it is a problem from top to bottom, but the result is no problem.

In a sense, the problem has been solved, but it has not been completely solved.

I hope Daniel can solve my doubts. Thank you very much!

Topics: Python

Programmer Think

Problems with Scrapy crawling image custom image file names

Problem: only one picture is downloaded. The name of the picture is the name of the last picture, but the content is not the content of the last picture. When printing, the relevant information of the last picture appears multiple times.

Relevant information of corresponding documents:

Items file related content:

Spider file content:

Settings file content:

Relevant contents of pipelies file:

At first, I wrote this. I took the last paragraph of the image path as the file name. There was no big problem

But I don't think it looks good. I plan to use their original names as file names

The final writing method is not to pass in the entire item, but the corresponding field content in the item object

Hot Topics