memphis VS xidel

Hey check out the first good issues in the Memphis.dev open source

Try Memphis.dev blog or you can check out Github

- https://github.com/memphisdev/memphis

Argo is a collection of open-source tools for Kubernetes to run workflows, manage clusters, and do GitOps easily. Memphis is an open-source next-generation alternative to traditional message brokers.

I have dealt with the same problem. It depends what are the cycle intervals. If for example, it's every few minutes maybe it's worth keeping a machine on the cloud and the DB on that machine. in my opinion, it's a little bit expensive. The thing that worked for me is to run once a day a lambda function and store the data on a message broker. You should take a look at memphis.dev which is open-source and very easy to work with.

Image Source

Join our 2K stargazers on Github and try Memphis out, I am sure you are going to be surprised 📷 https://github.com/memphisdev/memphis-broker

You could try Xidel[1]. It supports JSON, XML and HTML using XPath/XQuery 3.1

It has some extensions to the standard that are pretty nice (JSONiq, CSS selectors, html “template” matching), but you can limit it to just standard XPath/XQuery if you like.

I recommend getting the nightly v .99 build if you give it a try, the stable .98 version is pretty old and I’ve had no issues with .99

1. https://www.videlibri.de/xidel.html

Windows binary of Xidel (https://github.com/benibela/xidel) a commandline tool to download and extract data from HTML pages

Xidel for extraction and pagination

Download an entire subreddit to JSON Lines with Xidel:

I believe that you can't read from a url with batch alone. you'll need a helper app like xidel https://www.videlibri.de/xidel.html

I have not played with this, but it looks like xidel might allow you to do this in Bash. jc also has a URL string parser that could be used in such a script.

I built Xidel for such authenticated requests: