Programmer Think

1. Understand scratch_ splash？ scrapy_splash is a component of scratch The loading of js data by scratch Splash is implemented based on Splash.Splash is a Javascript rendering service.The final response obtained by using scratch splash is equivalent to the web page source code after the browser is fully rendered. splash official document ...

How do crawlers deal with streaming loading pages? Please collect these 100 lines of code! In the previous article, the academic committee made an article Take photos of the website , if you find that the screen capture page is a streaming page, what if you cut off part of it? The so-called streaming loading page, the page height is consta ...

  sometimes when we grab a page with requests, the results may be different from those seen in the browser: the normally displayed page data can be seen in the browser, but not in the results obtained with requests. This is because requests obtain the original HTML document, while the pages in the browser are generated after JavaScript pro ...

Due to the characteristics of its dynamic explanatory language, python runs code much slower than java and c + +. Especially when doing scientific computing, the disadvantages of Python are more prominent due to the billions and billions of operations. The solution is always more difficult than the difficulty. numba is a powerful tool to solve ...

From the case of crawling Taobao and saving it as a csv file. Forced by the video upload of station B, station C failed to pass the review, so it can only be directly linked to the text version. Basic write crawler: https://www.bilibili.com/video/BV1CW411C7ZM?spm_id_from=333.999.0.0 Advanced object oriented write crawler: https://www.bilibili.c ...

Crawl the historical net value of private placement network and crack the encrypted value Recently, I crawled the historical net value of private placement online and wrote some experiences in the process of crawling. There are many difficulties above. For example, if selenium is directly used, anti crawling will be detected, and the crawling ...

Foreword Share a case of using some crawler technology to simply crawl articles from media web pages as needed and save them to the local designated folder for reference only. In the learning process, do not visit the website frequently and cause web page paralysis. Do what you can!!! Crawling demand Crawl address: Construction Archives - c ...

Blog Text and Source Download: Python Crawl Doubles + Data Visualization Preface At my sister's invitation, I saw the Python crawler a while ago. I have to say that Python's grammar is really concise and graceful, readable, close to natural language, and very suitable for beginners of programming. Before you start, explain what crawls are: ...

Song search element Netease cloud music website is: https://music.163.com/ The idea is to enter a song name after entering, click the search button, and capture the search request through the developer debugging tool. The captured data information is as follows:   All song related information is in the result. Each a tag (hyperlink) carri ...

How to call Python programs using Java This article introduces how to call python methods in java for your reference. The combination of Java and python may be used in actual engineering projects, which will involve a problem, that is, how to call the written Python script with Java program. There are three methods to implement. The specific m ...