spotify-playlist-archive VS help-scraper

There's a Spotify Playlist Archive on GitHub, but sadly, the earliest version it has archived, is from 2021-12-21.

Cool project!

I maintain an archive of Spotify playlists, scraped daily via GitHub Actions: https://github.com/mackorone/spotify-playlist-archive.

If you're willing to share, I'd love to hear about your experience with the Spotify API. What endpoints do you use? How often do you call them? How do you handle rate limiting? Has Spotify reached out regarding storing playlist data (being in violation of their terms of service)? Any other lessons or advice?

Additionally, I wonder if there's any opportunity for collaboration here. Perhaps you can use the archive to fill out your dataset? Maybe it can act as a cache for your system's scraping needs? Perhaps our Web UIs could link to each other or be combined somehow (https://spotifyplaylistarchive.com/)?

> Currently we keep track of 18378 Spotify playlists

How did you find all of these playlists? Querying Spotify for "Spotify"-owned playlists gives me less than 5000 results. I'd love to add more playlists to my archive but I don't have good discovery mechanisms, especially for non-Spotify-owned playlists.

I looked for it here: https://github.com/mackorone/spotify-playlist-archive/blob/main/playlists/pretty/05dTMGk8MjnpQg3bKuoXcc.md but either it's not there or it has a different name.

Here ya go: https://github.com/mackorone/spotify-playlist-archive/blob/main/playlists/pretty/37i9dQZF1DWTtTyjgd08yp.md

Certainly give u/mackorone project a look (https://github.com/mackorone/spotify-playlist-archive), as he has been doing it longer than i have and has cumulative playlist for every list being archived, which i don't. im only doing what im doing as im not a programmer, and feel alot of people who want this aren't, so having CSV spreadsheets (which include the URI link for each song) that can easily be converted back to a playlist on Spotify, it makes it easier for anyone to jump in and get use out of it!

RapCavier

Hi folks! A few years ago, I built system for archiving public Spotify playlists: https://github.com/mackorone/spotify-playlist-archive. For the technically inclined: it's a GitHub repository of Markdown playlist files, updated once a day by a Travis CI bot.

this site has all the songs from the pollen playlist in the past in one page so might be worth a shot looking here https://github.com/mackorone/spotify-playlist-archive/blob/master/playlists/cumulative/POLLEN.md

This is really impressive.

I've been running a scraper against the output of the "aws --help" CLI commands for a few months, to try and get a better feel for how often AWS changes - I call this "help scraping". The answer is it changes a LOT - there are updates to their APIs every single day.

Here's the commit log of changes I've tracked so far: https://github.com/simonw/help-scraper/commits/main/aws

My AWS tracker has caught a ton of a activity - they really do ship new feature in that tool every day of the week: https://github.com/simonw/help-scraper/commits/main/aws

My scraper against the GitHub GraphQL schema catches some interesting details too: https://github.com/simonw/help-scraper/commits/main/github