DownloadNet VS beaker

For anyone who uses Chrome and wants to view their archived pages in the browser as if they were still online (URL and everything intact), and also full-text search through their browsing history that was archived (like AB plans to add in future, I think, right nikki?) you can check out DownloadNet: https://github.com/dosyago/DownloadNet

You can have multiple archives, and even use a mode where you only archive pages you bookmark rather than everything.

This does look cool. It reminds me of a recent discovery I made. The other day, while trying to recover some disk space, I found a giant file on my hard disk. It turned out to be a nine-hour screen recording from almost a year ago. I had no idea it existed, so I must’ve accidentally left the screen recording on. Watching it was fascinating; it was like a window into my thought process at that time. You could see how I was researching something online. It was almost like a play-by-play, akin to re-watching a sports performance – very instructive and surprisingly useful.

In a similar vein to what you’ve done, but focusing specifically on web browsing, I’ve created a tool called ‘DownloadNet.’ It archives for offline use and fully indexes every page you visit. Additionally, it can be configured to archive only the pages you bookmark, offering another mode of operation. It’s an open-source tool, so feel free to check it out: https://github.com/dosyago/DownloadNet

Given that I directly work in this space I found the article's synthesis of a range of ideas about browser innovation to be highly relevant.

More generally, the article is actually extremely interesting and examines a bunch of ideas worthy of consideration if you're interested in the future of web browsing.

Perhaps none of the ideas are new in isolation, but it's encouraging that people are doing this foundational conceptual work and imagining where a synthesis of them would go.

Despite being interesting somehow on the page it was not so easy to read. Here's a summary of key ideas:

Stagnation in Browser Evolution: Berjon notes that despite being central to the web's architecture, browsers haven't changed much in their fundamental design for a long time. They have undergone incremental changes but the core concept remains largely the same as it was decades ago.

Reimagining Browsers: He suggests that to increase user agency—a principle that the web should empower users—we need to consider major overhauls to what a browser is and how it operates.

Integration of Search and Social: Berjon challenges the traditional separation of browsers, search engines, and social platforms. He advocates for an integrated approach where the browser encompasses these functions, aligning more closely with users' experiences and expectations.

Shift From Client to Agent: The author proposes rethinking the browser not just as a client for retrieving documents but as an "agent" that provides a variety of services, potentially including server-like functions, to empower users.

User Agency and Personal Data Servers: By incorporating elements such as Personal Data Servers (PDS), users could manage their own data and services like recommendations, identity, and subscriptions, which currently rely on third-party providers.

Tab Management: Berjon critiques the use of tabs, suggesting that they are an ineffective method for organizing and interacting with web content, and advocates for better UI solutions.

Business Models: He delves into the financial aspects of browsers, highlighting the significant profits derived from setting search engine defaults. Berjon argues for reinvestment of these profits into the web as a public good and for developing business models that truly benefit user agency.

Potential for Change: Despite the challenges, Berjon is optimistic about the possibility of change, noting that there is room for product differentiation and that financial incentives can drive innovation in the browser space.

I found the one about User Agency and Personal Data Servers particularly fascinating. I've been exploring the idea of a federated search engine, where a person curates their own search through their browsing history (and ultimately could share it socially), in DownloadNet: https://github.com/dosyago/DownloadNet

And my company has been developing a platform for building extended and customized browsing experiences and delivering them anywhere. It's my hope that BrowserBox will play a part in the future direction of the browser as user agent. It's open source so if you care about the future of the web, get involved: https://github.com/BrowserBox/BrowserBox :)

If you're interested in utilizing your history information for something in your intentional interests, consider saving an archive of pages you browse to make a search engine you can query back through later.

You can save the full content for indexing with full text search, and you can even export archives as tarballs by zipping up the directory. Many people find this a useful way to "mine" their own browser history to create a curated search engine aligned with your interests. Or simply to save the pages they browse for review offline--either to save bandwidth, or just because they're actually "offline"--at a remote site, or on an airplane.

Everything is saved in a fully interactive way. Personally tho, I find search the most useful feature. Also, we're open source so if you want to get involved, please do so!

https://github.com/dosyago/DiskerNet

If you want full-text-search with archiving check out my project, DiskerNet. https://github.com/dosyago/DiskerNet --> also well done on LinkWarden! Looks like a great product! :)

For archiving, look into https://github.com/dosyago/DiskerNet

It's real next gen thinking on this topic.

As for the featured tool wayback... If HN readers can't figure out what it does after reading docs, its likely the thinking behind it is equally unclear.

For me, the peak of decentralization efforts were Beaker Browser [1] and Stealth [2].

But one project didn't make enough money and the author of the other one got doxxed into oblivion, so I guess we can't have nice things.

A peer to peer browser has so much potential, I wish somebody else might give it a try.

[1] https://github.com/beakerbrowser/beaker

[2] https://github.com/tholian-network/stealth

I wanted to mention Beaker Browser, but sadly, it's been archived: https://github.com/beakerbrowser/beaker/blob/master/archive-...

AT proto has some significant similarities to Matrix:

* Both are work by self-authenticating git-style replication of Merkle trees/DAGs

* Both define strict data schemas for extensible sets of events (Matrix uses JSON schema - https://github.com/matrix-org/matrix-spec/tree/main/data/eve... and OpenAPI; AT uses Lexicons)

* Both use HTTPS for client-server and server-server traffic by default.

* Both are focused on decentralised composable reputation - e.g. https://matrix.org/blog/2020/10/19/combating-abuse-in-matrix... on the Matrix side, or https://paulfrazee.medium.com/the-anti-parler-principles-for... on the bluesky side, etc.

* Both are designed as big-world communication networks. You don't have the server balkanisation that affects ActivityPub.

* Both eschew cryptocurrency systems and incentives.

There are some significant differences too:

* Matrix aspires to be the secure communication layer for the open web.

* AT aspires (i think) to be an open decentralised social networking protocol for the internet.

* AT has portable identity by default. We've been working on this on Matrix (e.g. MSC1228 - https://github.com/matrix-org/matrix-spec-proposals/pull/122... and MSC2787 - https://github.com/matrix-org/matrix-spec-proposals/blob/nei...) and have a new MSC (and implementation on Dendrite) in progress right now which combines the best bits of MSC1228 & MSC2787 into something concrete, at last. In fact the proto-MSC is due to emerge today.

* AT is proposing a asymmetrical federation architecture where user data is stored on Personal Data Servers (PDS), but indexing/fan-out/etc is done by Big Graph Servers (BGS). Matrix is symmetrical and by default federates full-mesh between all servers participating in a conversation, which on one hand is arguably better from a self-sovereignty and resilience perspective - but empirically has created headaches where an underpowered server joins some massive public chatroom and then melts. Matrix has improved this by steady optimisation of both protocol and implementation (i.e. adding lazy loading everywhere - e.g. https://matrix-org.github.io/synapse/latest/development/syna...), but formalising an asymmetrical architecture is an interesting different approach :)

* AT is (today) focused on for public conversations (e.g. prioritising big-world search and indexing etc), whereas Matrix focuses both on private and public communication - whether that's public chatrooms with 100K users over 10K servers, or private encrypted group conversations. For instance, one of Matrix's big novelties is decentralised access control without finality (https://matrix.org/blog/2020/06/16/matrix-decomposition-an-i...) in order to enforce access control for private conversations.

* Matrix also provides end-to-end encryption for private conversations by default, today via Double Ratchet (Olm/Megolm) and in the nearish future MLS (https://arewemlsyet.com). We're also starting to work on post quantum crypto.

* Matrix is obviously ~7 years older, and has many more use cases fleshed out - whether that's native VoIP/Video a la Element Call (https://element.io/blog/introducing-native-matrix-voip-with-...) or virtual worlds like Third Room (https://thirdroom.io) or shared whiteboarding (https://github.com/toger5/TheBoard) etc.

* AT's lexicon approach looks to be a more modular to extend the protocol than Matrix's extensible event schemas - in that AT lexicons include both RPC definitions as well as the schemas for the underlying datatypes, whereas in Matrix the OpenAPI evolves separately to the message schemas.

* AT uses IPLD; Matrix uses Canonical JSON (for now)

* Matrix is perhaps more sophisticated on auth, in that we're switching to OpenID Connect for all authentication (and so get things like passkeys and MFA for free): https://areweoidcyet.com

* Matrix has an open governance model with >50% of spec proposals coming from the wider community these days: https://spec.matrix.org/proposals

* AT has done a much better job of getting mainstream uptake so far, perhaps thanks to building a flagship app from day one (before even finishing or opening up the protocol) - whereas Element coming relatively late to the picture has meant that Element development has been constantly slowed by dealing with existing protocol considerations (and even then we've had constant complaints about Element being too influential in driving Matrix development).

* AT backs up all your personal data on your client (space allowing), to aid portability, whereas Matrix is typically thin-client.

* Architecturally, Matrix is increasingly experimenting with a hybrid P2P model (https://arewep2pyet.com) as our long-term solution - which effectively would end up with all your data being synced to your client. I'd assume bluesky is consciously avoiding P2P having been overextended on previous adventures with DAT/hypercore: https://github.com/beakerbrowser/beaker/blob/master/archive-.... Whereas we're playing the long game to slowly converge on P2P, even if that means building our own overlay networks etc: https://github.com/matrix-org/pinecone

I'm sure there are a bunch of other differences, but these are the ones which pop to the top of my head, plus I'm far from an expert in AT protocol.

It's worth noting that in the early days of bluesky, the Matrix team built out Cerulean (https://matrix.org/blog/2020/12/18/introducing-cerulean) as a demonstration to the bluesky team of how you could build big-world microblogging on top of Matrix, and that Matrix is not just for chat. We demoed it to Jack and Parag, but they opted to fund something entirely new in the form of AT proto. I'm guessing that the factors that went into this were: a) wanting to be able to optimise the architecture purely for social networking (although it's ironic that ATproto has ended up pretty generic too, similar to Matrix), b) wanting to be able to control the strategy and not have to follow Matrix's open governance model, c) wanting to create something new :)

From the Matrix side; we keep in touch with the bluesky team and wish them the best, and it's super depressing to see folks from ActivityPub and Nostr throwing their toys in this manner. It reminds me of the unpleasant behaviour we see from certain XMPP folks who resent the existence of Matrix (e.g. https://news.ycombinator.com/item?id=35874291). The reality is that the 'enemy' here, if anyone, are the centralised communication/social platforms - not other decentralisation projects. And even the centralised platforms have the option of seeing the light and becoming decentralised one day if we play our parts well.

What would be really cool, from my perspective, would be if Matrix ended up being able to help out with the private communication use cases for AT proto - as we obviously have a tonne of prior art now for efficient & audited E2EE private comms and decentralised access control. Moreover, I /think/ the lexicon approach in AT proto could let Matrix itself be expressed as an AT proto lexicon - providing interop with existing Matrix rooms (at least semantically), and supporting existing Matrix clients/SDKs, while using AT proto's ID model and storing data in PDSes etc. Coincidentally, this matches work we've been doing on the Matrix side as part of the MIMI IETF working group to figure out how to layer Matrix on top of other existing protocols: e.g. https://datatracker.ietf.org/doc/draft-ralston-mimi-matrix-t... and https://datatracker.ietf.org/doc/draft-ralston-mimi-matrix-m... - and if I had infinite time right now I'd certainly be trying to map Matrix's CS & SS APIs onto an AT proto lexicon to see what it looks like.

TL;DR: I think AT proto is cool, and I wish that open projects saw each other as fellow travellers rather than competitors.

it sounds a lot like you're reinventing what Beaker Browser had built on top of DAT, except that it could do more. For example, they made a distributed Twitter clone as a proof of concept, but folks actually started using it. Definitely included blogging stuff.

Really cool stuff around taking sites and things other folks had built and using them as a basis for your new thing.

https://github.com/beakerbrowser/beaker/

As a long time patchwork user —April 2017 for the win…— that just recently quit, I could see how the multitude of half finished clients, deprecated functionality would get to that outcome.

SSB is dead, other than the few trying to make a go financially at it, via either crowdfunding, NLnet grants, or VC.

I've reverted to Web 1.0 blogging, with none of the bs that is consistent with using a archived client, focus on trying to fit a database into a mobile app — without regard to front end functionality.

> When I look at Beaker, I think it was probably 50% easy. The initial demo took 2 weeks: 20%. It was a full website editor in about 2 months: 30%. The feedback was great: 50%. The users didn't stick: 50%. We got invited to talks which increased exposure: 51%. A few niche communities took an interest: 53%. Folks liked it enough to donate via OpenCollective and Patreon: 54%. You get the idea. Notably absent is "usage and retention went through the roof: 80%" and then "usage continued to grow for years: 100%."

Everything that pfrazee wrote here about Beaker Browser at https://github.com/beakerbrowser/beaker/blob/master/archive-... is true for ssb.

I'm sad to see this go, a remnant of another web which could have been. I actually spent a lot of time playing with Beaker and hacking it up for my own purposes.

We actually had a discussion a few years ago where I made a suggestion about change to the default behavior. At the time, you made a perfectly valid response and declined my suggestion, but I'm curious if your thinking is the same today, given how things played out: https://github.com/beakerbrowser/beaker/issues/1444

Beaker, Hybercore

Disclosure: It's in Romanian, no cookies, no JS, no trackers

Beaker Browser https://beakerbrowser.com/ seems dead, loved the concept but it's no longer updated

Now that you've asked, nope, didn't found anything with a clear future on the "Web3" side of the internet. Vast majority make use of crypto/blockchain and IMHO blockchain is anything but not decentralization.