Programmer Think

From crawling one page of data to crawling all data Let's talk about the general process of static web crawler first   Data loading method By clicking on the second page, you can find that there are more at the back of the website? start=25 field This part is called "query string". The query string is transmitted to the server as a ...

preface As a new generation of contemporary youth, should they be more or less able to make short videos? Haha, what about the creators of contemporary we media~ When making videos, how much do you need some funny sounds? Or strange sounds? Music, etc~ How slow it is to download one by one, let's use python to realize batch download tod ...

Hello, I'm Ou K. The May Day holiday is coming. There is plenty of time for welfare (five-day holiday) this year. How do you want to play such a long holiday? Play like this? Still playing like this? In this issue, we will briefly analyze the distribution of popular scenic spots and national travel in China through the sales of ti ...

catalogue First, simulate the browser's environmental preparation. 1. Introduction to selenium 2.Selenium installation 3. Install WebDriver (1) Installing chromedriver (2) Add chromedriver to environment variable 2, Example: use Selenium operation browser to obtain QQ music list data 1. Start the browser 2. Use webdriver to open qq mus ...

catalogue 1, Installation and documentation address 2, Send GET request Add headers and query parameters response.text and response Difference of content 3, Send POST request 4, Use agent 5, Cookies 6, session 7, Handling untrusted SSL certificates   Although the urllib module in Python's standard library already contains most of th ...

catalogue Downloader Middleware 1, process_request(self, request, spider) 2, process_response(self, request, response, spider) 3, Random request header Middleware setting.py  middlewares.py httpbin.py 4, ip proxy pool Middleware 1. Purchasing agent 2. Using ip proxy pool 3. Exclusive agent pool   Downloader Middleware Downloader ...

Data storage Two data storage methods are mainly introduced: Stored in files, including text files and csv files Stored in databases, including MySQL relational database and mongoDB database Store to txt title = "First text" # W create write W + create read + Write # r read r + read + Write # A write a + read write attach with open(r'C:\Users ...

Ctrip's automatic login is still a little troublesome. Let's look at the official website first:   Needless to say, you must locate the label first, locate it in the red box, jump through click(), and come to the following page:   Here, first locate the tag to the place where you enter the user name and password, and then use send_keys() can ...

Due to an introductory course in Python next semester So I've been groping for myself during the winter vacation. After all, I can't drop out at that time. It's also a water credit On a whim recently, I plan to try climbing Baidu translation After a day's work, the liver finally came out Don't talk too much and start it directly (the environmen ...

Welcome to "Python from zero to one", where I will share about 200 Python series articles, take you to learn and play, and see the interesting world of Python. All articles will be explained in combination with cases, codes and the author's experience. I really want to share my nearly ten years of programming experience with you. I ho ...