parser VS Cloudbox

Thoroughly scraping is challenging, especially in an environment where you don’t have (or want) a JavaScript runtime.

For content extraction, I found the approach the Postlight library takes quite neat. It scores individual html nodes based on some heuristics (text length, link density, css classes). It the selects the nodes with the highest score. [1] I ported it to Swift for a personal read later app.

[1] https://github.com/postlight/parser

I've been working on an enhanced reader mode extension for the last few months. I found that Mercury Reader's parser tool is useful for extracting content. If that's not exactly what you're looking for, readibility is another good option. It's a library used inside Firefox's reader moder that you can use in any project.

I currently have it turned off while I search for better sources, but I have a VM that runs a custom cron script that combines a custom RSS reader, podfox, mercury-parser, and coqui-ai to generate audio podcasts from RSS news feeds. I should probably clean it up and release the script/setup process. With a few tweaks and some AI text-to-speech and a little machine learning audio processing you can get a really good podcast experience from text posts.

It looks like Relay could be updated to convert it locally though, since the parser that it uses appears to be open source.

https://github.com/postlight/mercury-parser The only one I need, shit's too good

An alternative would be to extract the plain text from the article's page with either some "readability" API or a library like Mercury Parser: https://github.com/postlight/mercury-parser

I haven’t directly compared them, but I have also found mercury parser (https://github.com/postlight/mercury-parser) to be very reliable.

Since it turns a website into very plain (X)HTML it‘s fairly easy to use it to make a browsing proxy or automatically produce epub files for e-readers, which is what I do.

Make sure that at the end of the process you have the node and npm executables installed - the http.webpage integration uses the Mercury Parser API to convert web pages to Markdown.

Good luck! Those HTML issues you're coming across are tough and so varied across the web!

I was working with Mercury Parser (pluggable parsing for different sites) in the past.

https://github.com/postlight/mercury-parser

https://docs.saltbox.dev or https://cloudbox.works

I have just got my first seedbox and im trying to get it setup, first thing i would like to do is get plex working with my existing libraries stored on Google Drive. I have followed all the guides ( How to mount Google Drive for Plex usage | RapidSeedbox: Help Center & Google Drive API Client ID and Client Secret · Cloudbox/Cloudbox Wiki · GitHub )

2023-03-06 08:29:39,951 - INFO - CONFIG [140352731267072]: Using default setting --config=config/config.json 2023-03-06 08:29:39,951 - INFO - CONFIG [140352731267072]: Using default setting --logfile=autoscan.log 2023-03-06 08:29:39,951 - INFO - CONFIG [140352731267072]: Using default setting --loglevel=INFO 2023-03-06 08:29:39,951 - INFO - CONFIG [140352731267072]: Using default setting --queuefile=queue.db 2023-03-06 08:29:39,951 - INFO - CONFIG [140352731267072]: Using default setting --cachefile=cache.db 2023-03-06 08:29:39,970 - INFO - CONFIG [140352731267072]: Using default setting --config=config/config.json 2023-03-06 08:29:39,970 - INFO - CONFIG [140352731267072]: Using default setting --logfile=autoscan.log 2023-03-06 08:29:39,970 - INFO - CONFIG [140352731267072]: Using default setting --loglevel=INFO 2023-03-06 08:29:39,971 - INFO - CONFIG [140352731267072]: Using default setting --queuefile=queue.db 2023-03-06 08:29:39,971 - INFO - CONFIG [140352731267072]: Using default setting --cachefile=cache.db ___ __ / | __ __/ /_____ ______________ _____ / /| |/ / / / __/ __ \/ ___/ ___/ __ `/ __ \ / ___ / /_/ / /_/ /_/ (__ ) /__/ /_/ / / / / /_/ |_\__,_/\__/\____/____/\___/\__,_/_/ /_/ 2023-03-06 08:29:40,097 - INFO - AUTOSCAN [140352731267072]: ######################################################################### # Title: Plex Autoscan # # Author: l3uddz # # URL: https://github.com/l3uddz/plex_autoscan # # -- # # Part of the Cloudbox project: https://cloudbox.works # ######################################################################### # GNU General Public License v3.0 # ######################################################################### 2023-03-06 08:29:40,102 - INFO - PLEX [140352731267072]: Requesting section info from Plex... 2023-03-06 08:29:40,119 - ERROR - PLEX [140352731267072]: Issue encountered when attempting to list detailed sections info. Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 703, in urlopen httplib_response = self._make_request( File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 449, in _make_request six.raise_from(e, None) File "", line 3, in raise_from File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 444, in _make_request httplib_response = conn.getresponse() File "/usr/lib/python3.10/http/client.py", line 1374, in getresponse response.begin() File "/usr/lib/python3.10/http/client.py", line 318, in begin version, status, reason = self._read_status() File "/usr/lib/python3.10/http/client.py", line 287, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 489, in send resp = conn.urlopen( File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 787, in urlopen retries = retries.increment( File "/usr/local/lib/python3.10/dist-packages/urllib3/util/retry.py", line 550, in increment raise six.reraise(type(error), error, _stacktrace) File "/usr/local/lib/python3.10/dist-packages/urllib3/packages/six.py", line 769, in reraise raise value.with_traceback(tb) File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 703, in urlopen httplib_response = self._make_request( File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 449, in _make_request six.raise_from(e, None) File "", line 3, in raise_from File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 444, in _make_request httplib_response = conn.getresponse() File "/usr/lib/python3.10/http/client.py", line 1374, in getresponse response.begin() File "/usr/lib/python3.10/http/client.py", line 318, in begin version, status, reason = self._read_status() File "/usr/lib/python3.10/http/client.py", line 287, in _read_status raise RemoteDisconnected("Remote end closed connection without" urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/autoscan/plex.py", line 25, in show_detailed_sections_info resp = requests.get('%s/library/sections/all?X-Plex-Token=%s' % ( File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 73, in get return request("get", url, params=params, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 587, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 701, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 547, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

Plex + https://cloudbox.works/ on a server in a friendly country. Fully automatic once you set it up.

Swizzin, Cloudbox or /r/plexshares are the easiest ways.

If you're willing to put in the work, I recommend taking a look at Cloudbox.

Biggest established tooling projects are Cloudbox (Docker), and Swizzin' (not Docker).

See also Cloudbox, and Swizzin'. The latter is a more complete, alibeit not Dockery, soln.

You'd probably prefer to look at something like Cloudbox then, which is a kind of Dockerized version of the same thing. Also pretty popular.