handlefinder VS nitter

MorningBot: https://github.com/bnkc/morningbot HandleFinder: https://github.com/bnkc/handlefinder ThreeSigma: https://www.threesigma.ai

I found myself scrolling through Github’s “trending” repos, looking for some coding inspiration. Within the next hour, I stumbled across something called The Sherlock Project. Interesting, It had over 35k stars, must be pretty popular.

I quickly cloned the repo and started toying around with it. It didn’t take me long to realize the power of this tool. All I had to do was insert a username, and Wa Lah! I was looking at every social media website that was associated with the username. Not only that but direct links to the accounts.

I immediately wanted to turn this into a web app so that everyone could use it. My first challenge was that this was a CLI tool, so I got to work. The Sherlock project makes about 400 requests to various site s to check if your username exists. This was going to be tough... I noticed they were using requests.FutureSession to multithread the result.

My first thought was maybe I could collect all the requests into a list and then dump it back to the frontend with a get request, but the issue here was that the results could take a couple minutes to gather, and no-one is gonna sit and wait that long.

Ok, so how do I continuously report out data to the frontend as the requests get full-filled? The answer? Web-sockets. Not just any web-socket though, a multithreaded web-socket. After ALOT of trial and error I finally got something working. The Issue now though was that it wouldn't run in production due to a multiprocessing error: Daemonic processes are not allowed to have children.

The hell did that mean? back to the drawing board. Eventually I learned that you cant use the standard multiprocessing library for this kind of thing, you had to use billiard. Bam! It worked. I quickly hacked together a simple frontend, configured the web socket, and results were pouring in.

Time to post on reddit. I quickly sent out a post or two and eagerly awaited peoples responses. There was an issue though. The more people flooded the site, the slower it got until it was completely unusable. Dang. How do I fix this? I removed the post quickly and got back to work.

Turns out, the web-socket is considered a "long running request" as it makes 400 external requests. Maybe I could use celery to offload this process to a worker and queue it up. I started working on it and realized this was a little out of my skill range.

I then decided to take a look at the logs where I hosted the code and what do i find? CPU, Memory, and bandwidth all reaching a staggering 100% usage. Welp, I found my issue. I did some rework of my codebase and it started running a little faster.

I then realized the true issue. I was using the free tier of Render that only allowed for one instance of my app...duh.

Needless to say, I learned to take it slow, build tests for my code, and be patient with results.

What do you guys think? Any hard lessons learned in coding? What were your takeaways?

Here is also a link to the repo: https://github.com/bnkc/handlefinder

Here is a link to the repo: https://github.com/bnkc/handlefinder

Don't use Youtube without going through a proxy like Invidious [1] or Newpipe

Don't use {site} Search without going through a proxy like SearxNG [2]

Don't use TwiXXer without going through a proxy like Nitter - this has gotten more difficult lately but it still works as long as you feed the daemon some registered accounts. Video does not work at the moment but that seems to be fixable.

Don't use Reddit without going through a proxy like libreddit [4]

Start noticing the pattern? Maybe it is time to start producing promotional posters:

The only thing to come between you and ADS could be a proxy / ADS. I'ts just not worth the risk

ADS / New rules for a sane net / Sane net protects you, your partner and your community

A proxy here and a filter there, ADS nowhere

The more you tighten your grip, ${site}, the more viewers will slip through your fingers

[1] https://github.com/iv-org/invidious

[2] https://github.com/searxng/searxng

[3] https://github.com/zedeus/nitter

[4] https://github.com/libreddit/libreddit

https://nitter.esmailelbob.xyz/liuzhuang1234/status/17601959...

(bit of trial and error from https://github.com/zedeus/nitter/wiki/Instances)

I assume this is why Nitter got slammed and is on the verge of breaking.

https://github.com/zedeus/nitter/issues/1154

The creator of the project left this comment an hour ago:

https://github.com/zedeus/nitter/issues/1155#issuecomment-19...

(HN removed the deep link to the specific comment from the title)

Most of the time you can just replace twitter.com with nitter.net or nitter.it but when these instances are down or blocked you can use other instances listed here: https://github.com/zedeus/nitter/wiki/Instances

But as the other commenter said Libredirect is the way to go: https://libredirect.github.io/

Use a Youtube proxy like Invidious [1], problem solved and you get to subscribe to channels without telling the Beast about your interests. Add Sponsorblock (which supports Invidious) to get rid of any in-stream advertising which remains and you'll be transported back to those hallowed times of yore when men were men, women were women and advertising was something you found in newspapers. Youtube will try to make this harder just like Xitter is trying to make it harder to use proxies like Nitter [2].

[1] https://github.com/iv-org/invidious

[2] https://github.com/zedeus/nitter

njal.la restored nitter.net, and replied with their reasons: https://github.com/zedeus/nitter/issues/1150#issuecomment-18...

tbh I can totally understand why they acted this way.