proxy-list
awesome-web-scraping
Our great sponsors
proxy-list | awesome-web-scraping | |
---|---|---|
5 | 6 | |
2,204 | 6,308 | |
- | - | |
4.0 | 5.1 | |
about 1 year ago | 18 days ago | |
Makefile | ||
MIT License | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
proxy-list
- Proxylist Sources
-
Where do you find proxies for proxychains?
Fetch them from this Repo daily https://github.com/clarketm/proxy-list. I created a cron Job which Pulls the repository and then Updates my config.
-
Does anyone know how do use Jdownloader 2 to bypass MEGA dl limits?
surfshark doesn't provide proxies, and if they did it would be one domain. https://github.com/clarketm/proxy-list might be worth a try
-
Help scraping StockX / Goat / eBay
There is a daily update list here https://github.com/clarketm/proxy-list/blob/master/proxy-list-raw.txt
-
Is it normal that NOT EVEN ONE proxy from github or other sites ... work ??
def filterOut_workingProxies(proxies): #FILTER OUT WORKING PROXIES workingProxies = [] #for i in range(10): for i in range(len(proxies)): p = proxies[i] print(f"testing: {p}...") proxy = { 'http' : 'http://'+p, #ERK:::https://stackoverflow.com/a/61466680 'https' : 'https://'+p} try: resp = requests.get("https://www.google.com", proxies=proxy, timeout=3) print(resp.status_code) if resp.status_code == 200 or str(resp.status_code)[0]=='2': print("+++++ WROKING\n") workingProxies.append(proxy) except: pass return workingProxies [...] #DAILY UPDATING PROXY LINKS: #CONST urls_github = ["https://github.com/ShiftyTR/Proxy-List/blob/master/proxy.txt", "https://github.com/clarketm/proxy-list/blob/master/proxy-list-raw.txt"] url_fplNet = "https://free-proxy-list.net"
awesome-web-scraping
-
Ask HN: LinkedIn sent me a cease and desist for my Chrome extension. Help?
>I can scrape linkedin with a python script. That doesn't mean linkedin can shut down python.
Well said!
Also, what about copy-and-paste? The last time I checked, data could be highlighted in the browser, copied, and pasted...
Does that mean that LinkedIn can shut down the copy-and-paste capability of your browser and/or operating system?
What about "Save Page As..." functionality (the ability of a browser to save a page offline?)
Can LinkedIn shut down "Save Page As..." ?
Also, what about the Print Screen (take a screen snapshot) capabilities of your operating system?
Can LinkedIn shut down that?
Finally, there's literally oodles of software that can be used for web scraping; what follows below is just one non-canonical list:
https://github.com/lorien/awesome-web-scraping
Is LinkedIn going to shut down all of that, at the same time?
Anyway, an excellent point about Python!
- Awesome-web-scraping – List of libraries, tools and APIs for web scraping
-
How does webscraping a website work and putting the data into my website?
Because at least for the scraping part there are open-source and paid services that will probably get you the data today if you need it (unless these are some really hard-to-scrape websites you're targeting) But if you are keen on learning yourself just scroll down this subreddit you will find many guides users shared along the years...
-
Russian Flag in Readme
E.g. how would a Ukrainian dev feel having his project showcased in this list, under the Russian flag?
[0] https://github.com/lorien/awesome-web-scraping/issues/136
- A central repository for scrapping scripts
What are some alternatives?
mubeng - An incredibly fast proxy checker & IP rotator with ease.
Proxyman - Modern. Native. Delightful Web Debugging Proxy for macOS, iOS, and Android ⚡️
fresh-proxies - fresh-proxies
awesome-micropython - A curated list of awesome MicroPython libraries, frameworks, software and resources.
Proxy-List - Free proxy list UPDATED HOURLY! -- for api visit
TabNine - AI Code Completions
Awesome-Warez - All your base are belong to us!
proxy-list - A list of free, public, forward proxy servers. UPDATED DAILY!
cookiecutter-poetry-pypackage - Cookiecutter template for poetry managed python package
Sneaks-API - A StockX, FlightClub, Goat, and Stadium Goods API all in one. This sneaker API allows users to search sneakers and track and compare prices while providing additional info such as product links and images
syntax-highlighter - Syntax Highlighter extension for Visual Studio Code (VSCode). Based on Tree-sitter.