Spidey
TumblThree
Our great sponsors
| Spidey | TumblThree | |
|---|---|---|
| 2 | 4 | |
| 11 | 565 | |
| - | 1.2% | |
| 9.5 | 8.0 | |
| 11 days ago | 3 days ago | |
| C# | C# | |
| Apache License 2.0 | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Spidey
-
I need data from a website. It is viable to create an API that scrapes the website and returns the data on an endpoint?
Didn't get a chance to reply earlier but depending on what you're trying to do, you might want a web crawler. I have a crawler on Github that I built for scraping in instances where someone doesn't have an API. If you go this route, I suggest doing it as a background task and go off cached data.
-
Recursion needed in small crawler
This may be overkill but I have library out there for building web crawlers. Spidey is the library. I'm not suggesting you use it but you could look at it for ideas. It uses a multithreaded, producer/consumer approach that avoids recursion and stack overflow issues. Use a queue, pull from the queue for each url, push new urls on when you find them. Do need to optimize my code a bit more but if it helps at all. But your issue is most likely the fact that you're finding a link to the page you are currently on. HashSet or List of found URLs would solve the issue.
TumblThree
-
Twitter archive not working?
There's a github project called TumblThree that can pull tweets and media that have been publicly posted to any account you give it. It can't do private accounts or DMs and it doesn't pull quote tweets, but it gives you a backup of everything publicly available.
- Way to automatically download images from liked Tumblr posts?
-
Is there a good list of up-to-date data archiving tools for different websites?
For Tumblr, I've found, but not tried TumblThree. It looks like it's built for Windows.
-
Archiving entire tumblr?
Presumably https://github.com/TumblThreeApp/TumblThree
What are some alternatives?
scrapyrt - HTTP API for Scrapy spiders
tumblr-utils - Utilities for dealing with Tumblr blogs, Tumblr backup
scrapy-sanoma-kuntavaalit2021 - Fetch Sanoma kuntavaalit 2021 data [Moved to: https://github.com/raspi/scrapy-kuntavaalit2021-sanoma]
SingleFileZ - Web Extension to save a faithful copy of an entire web page in a self-extracting ZIP file
SCrawler - 🏳️🌈 Media downloader from any sites, including Twitter, Reddit, Instagram, Threads, Facebook, OnlyFans, YouTube, Pinterest, PornHub, XHamster, XVIDEOS, ThisVid etc.
reddit-save - A Python tool for backing up your saved and upvoted posts on reddit to your computer.
XClipper - XClipper is a clipboard manager for Windows & Android which helps to track clipboard activities and makes it easier to interact with them.
Uno Platform - Build Mobile, Desktop and WebAssembly apps with C# and XAML. Today. Open source and professionally supported.
ReactiveUI - An advanced, composable, functional reactive model-view-viewmodel framework for all .NET platforms that is inspired by functional reactive programming. ReactiveUI allows you to abstract mutable state away from your user interfaces, express the idea around a feature in one readable place and improve the testability of your application.
Feed-Manager-WPF-App - RSS Feed Downloader and local file manager in a modern WPF Metro Design and MVVM Design Pattern.
cy-client - Unofficial cyberdrop album downloader client
UnamDownloader - A Free Silent (Hidden) Open Source Downloader (Binder) - Includes Windows Defender Bypass - Build Native - Unam Downloader