pygooglenews VS oldweb-today

I have ~4k start in 2 Python libraries. Both help fetch live news articles. Links below.

These were my first libraries.

I took the approach of promoting them as any other product. You have to "sell" your code. Even if it's 100% free.

In my opinion, the most important thing is DEMO. Just make a GIF where you showcase what your software does:

* 80% of engineers won't even bother to read the description

No one will spend their precious time trying to get through your code.

[0] https://github.com/kotartemiy/newscatcher Programmatically collect normalized news from (almost) any website.

[1] https://github.com/kotartemiy/pygooglenews If Google News had a Python library

I wrote a pygooglenews package for news mining out of google news.

I'm curious where the data get fetched from. The Author mentions that Mozilla Readability and SimplePie are used.

Readability to parse the content. SimplePie to fetch the data (I assume). Dat from RSS feeds?

In case you want to make something similar, I recently wrote a blog on where you could get news data for free [1]

(self-promo) I'd recommend to take a look at my Python package to mine news data from Google News [2]. Also, in 3 days we're releasing an absolutely free News API [3] that will support ~50-100k top stories per day.

[1] https://blog.newscatcherapi.com/an-ultimate-list-of-open-sou...

[2] https://github.com/kotartemiy/pygooglenews

[3] https://newscatcherapi.com/free-news-api

I wrote a python package to scrape google news headlines at scale: https://github.com/kotartemiy/pygooglenews

Write libraries AND services, where it makes sense.

I wrote a Python library to scrape google news [0]

We also have it as a service [1]

Want to know why? Because devs who can't pay won't pay. Businesses who can pay will rather pay for a service (API in our case), and not care about maintaining it.

[0] https://github.com/kotartemiy/pygooglenews

[1] https://newscatcherapi.com/google-news-api

You can use this Python project to scrape Google News (https://github.com/kotartemiy/pygooglenews) If you look at the code you can get a feel of how to call Google News directly.

https://github.com/kotartemiy/pygooglenews https://pypi.org/project/google-play-scraper/

There are flat-file CMSes (content management systems) like Grav: https://getgrav.org/

I guess, in some vague/broad sense, config-as-code systems also implement something similar? Maybe even OpenAPI schemas could count to some degree...?

In the old days, the "semantic web" movement was an attempt to make more webpages both human- and machine-readable indefinitely by tagging them with proper schema: https://en.wikipedia.org/wiki/Resource_Description_Framework. Even Google was on board for a while, but I guess it never saw much uptake. As far as I can tell it's basically dead now, both because of non-semantic HTML (everything as a React div), general laziness, and LLMs being able to parse things loosely.

-------------

Side thoughts...

Philosophically, I don't know that capturing raw data alone as files is really sufficient to capture the nuances of any particular experience, or the overall zeitgeist of an era. You can archive Geocities pages, but that doesn't really capture the novelty and indie-ness of that era. Similarly, you can save TikTok videos, but absent the cultural environment that created them (and a faithful recreation of the recommendation algorithm), they wouldn't really show future archaeologists how teenagers today lived.

I worked for a natural history museum for a while, and while we were there, one of the interesting questions (well, to me anyway) was whether our web content was in and of itself worth preserving as a cultural artifact -- both so that future generations can see what exhibits were interesting/apropos for the cultures of our times, but also so they could see how our generation found out about those exhibitions to begin with (who knows what the Web will morph into 50 years later). It wasn't enough to simply save the HTML of our web pages, both because they tie into various other APIs and databases (like zoological collections) and because some were interactive experiences, like games designed to be played with a mouse (before phones were popular), or phone chatbots with some of our specimens. To really capture the experience authentically would've required emulating not just our tech stacks and devices, among other things.

Like for the earlier Geocities example, sure you could just save the old HTML and render it with a modern browser, but that's not the same as something like https://oldweb.today/?browser=ns3-mac#http://geocities.com/ , which emulates the whole OS and browser too. And that still isn't the same as having to sit in front of a tiny CRT and wait minutes for everything to download over a 14.4k modem, only to be interrupted when mom had to make a call.

I guess that's a longwinded of critiquing "file over app": It only makes sense for things that are originally files/documents to begin with. Much of our lives now are not flat docs but "experiences" that take much more thought and effort to archive. If the goal is truly to preserve that posterity, it's not enough to just archive their raw data, but to develop ways to record and later emulate entire experiences, both technological and cultural. It ain't easy!

1. Anything that can emulate Windows 95

2. HotDog, Dreamweaver, and/or Frontpage for HTML editing. They each have their strengths & weaknesses and work relatively well together. Normally I'd lay out pages in Dreamweaver or Frontpage and then clean up the HTML with HotDog. https://en.wikipedia.org/wiki/HotDog. Upload them with BulletproofFTP (http://www.bpftp.com/), an ancient predecessor of Filezilla.

3. Not sure if any of the old hosts are still around. Maybe Neocities.org comes close to the Geocities of old, or else you can get barebones and very cheap but reliable web hosting from https://www.nearlyfreespeech.net/

Get AOL dialup: https://getonline.aol.com/dialup and a 56k modem off eBay, no cheating with megabit connections.

You can emulate old browsers online: https://oldweb.today/

People generally weren't self-hosting their websites back then because phone lines weren't very reliable, and the dialup modems of the day (like today's cable modems) generally had faster download than upload speeds. After the BBS era, commercial internet hosts started becoming pretty common, but they usually used expensive ISDN or T1 lines.

Apache was still in its infancy then, and Microsoft IIS was common.

---------

Overall, I don't think the network stack is really THAT far removed from what we have today. The basics of HTTP were pretty fleshed out already (it was more the HTTPS and DNS security extensions that really evolved), along with better compression and parallelization protocols in HTTP2/3. And of there's been a huge amount of backend server optimizations, caching, reverse proxies, etc. But the old network stack would still work today and you can still run a basic barebones http daemon and firewall the same way you could back then, open up port 80, and watch your box get pwned by the bots. Hell, there are probably still zombie bots leftover from that era running on someone's basement PC, just casually scanning the internet day in and day out, waiting for its chance...

What's changed a lot more (IMO) is more the language of the web itself. HTML was very basic then, as was JS and CSS. Making websites without modern CSS, in particular, is very painful... requiring a lot of nested tables, frames (remember those? they're different from iframes!), etc. A lot of the pain was abstracted away to backend scripting languages (Perl, ASP, Coldfusion) or frontend extensions (Java, Shockwave Flash) and people didn't really work in vanilla HTML/JS very often (because they were so weak by then). Plain HTML is for your basic Astro Marmalade site with a bunch of animated gifs and whatever, but probably real ecommerce sites (such as it were) wouldn't be written in plain HTML :)

I like https://oldweb.today ... actually emulates old OS/browser combinations and proxies stuff from archive.org

Another option to what you are looking for https://oldweb.today/

Good news! You can go back in time https://oldweb.today/

https://web.archive.org/ does not have a screenshot from that month. https://oldweb.today doesn't work either because it pulls up a broken version (I tried all of the browsers it allows on the left). I tried looking online for another site to be able to pull an archive but was unsuccessful.

I need to see what gravitytransformation.com/macro-calculator looked like in September of 2019. https://web.archive.org/ does not have a screenshot from that month. https://oldweb.today doesn't work either.

https://oldweb.today/ gives you that experience, using some of the same building blocks.