ratarmount
brotli
ratarmount | brotli | |
---|---|---|
10 | 26 | |
637 | 13,156 | |
- | 0.8% | |
9.1 | 8.1 | |
16 days ago | 7 days ago | |
Python | TypeScript | |
MIT License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
ratarmount
- Ratarmount: Access large archives as a filesystem efficiently
- Show HN: Rapidgzip – Parallel Gzip Decompressing with 10 GB/S
- Ratarmount: Random Access Tar Mount
-
Ask HN: Most interesting tech you built for just yourself?
This is basically the same reason why I started with ratarmount (https://github.com/mxmlnkn/ratarmount) but the focus was more on runtime performance and random access and as the name suggests it started out with access to recursive tar archives. The current version should also work for your use case with recursive zips.
-
Looking for advice uploading data while at uni. I need to split the data i need to upload to carry it with me
As an added complication this would need to work under windows (i need onenote and that's win only :/ ) ; this alone makes the majority of solutions that i came up with impossible. One way could've been splitting the data onto various tar files and then mounting those with rartarmount but...linux only :( .
-
How Much Faster Is Making a Tar Archive Without Gzip?
Pragzip actually decompress in parallel and also access at random. I did a Show HN here: https://news.ycombinator.com/item?id=32366959
indexed_gzip https://github.com/pauldmccarthy/indexed_gzip can also do random access but is not parallel.
Both have to do a linear scan first though. The implementations however can do the linear scan on-demand, i.e., they scan only as far as needed.
bzip2 works very well with this approach. xz only works with this approach when compressed with multiple blocks. Similar is true for zstd.
For zstd, there also exists a seekable variant, which stores the block index at the end as metadata to avoid the linear scan. indexed_zstd offers random access to those files https://github.com/martinellimarco/indexed_zstd
I wrote pragzip and also combined all of the other random access compression backends in ratarmount to offer random access to TAR files that is magnitudes faster than archivemount: https://github.com/mxmlnkn/ratarmount
-
Ratarmount – Fast transparent access to archives through FUSE
Or via the experimental AppImage I created this week:
wget -O ratarmount 'https://github.com/mxmlnkn/ratarmount/releases/download/v0.10.0/ratarmount-manylinux2014_x86_64.AppImage'
-
Hop: 25x faster than unzip and 10x faster than tar at reading individual files
I've recently been looking into this same issue because I analyse a lot of data like sosreports or other tar/compressed data from customer systems. Currently I untar these onto my zfs filesystem which works out OK because it has zstd compression enabled but I end up decompressing and recompressing which is quite expensive as often the files are GBs or more compressed.
But I've started using a tool called "ratarmount" (https://github.com/mxmlnkn/ratarmount) which creates an index once (and something I could automate our upload system to generate in advance, but you can also just process it lcoally) and then lets you fuse mount the file. This works pretty great with the only exception that I can't create scratch files inside the directory layout which in the past I'd wanted to do.
I was surprised how hard a problem to solve it is to get a bundle file format that is indexable and compressed with a good and fast compression algorithm which mostly boils down to zstd at this point.
While it works quite well, especially with gzip and bzip2, sadly the zstd and xz (and some other compression formats) don't allow for decompressing only parts of a file by default, even though it's possible the default tools aren't doing it. The nitty gritty details are summarised here:
-
Is there a way to accelerate extracting .tar contents?
Well, you could try to skip extraction and access the tar archive using ratarmount, and stack overlayfs on top to allow writing, but that will have an impact on compilation time.
brotli
-
Node.js vs Angular: Navigating the Modern Web Development Landscape
Using tools like Brotli, you can boost your application’s load time. You can use the ngUpgrade library to mix AngularJS and Angular components to enhance runtime performance, bringing in hybrid applications that can be used with techniques like ahead-of-time (AOT) compilation, aiding in faster browser rendering.
-
Jpegli: A New JPEG Coding Library
JPEGLI = A small JPEG
The suffix -li is used in Swiss German dialects. It forms a diminutive of the root word, by adding -li to the end of the root word to convey the smallness of the object and to convey a sense of intimacy or endearment.
This obviously comes out of Google Zürich.
Other notable Google projects using Swiss German:
https://github.com/google/gipfeli high-speed compression
Gipfeli = Croissant
https://github.com/google/guetzli perceptual JPEG encoder
Guetzli = Cookie
https://github.com/weggli-rs/weggli semantic search tool
Weggli = Bread roll
https://github.com/google/brotli lossless compression
Brötli = Small bread
-
Compression efficiency with shared dictionaries in Chrome
The brotli repo on github has a dictionary generator: https://github.com/google/brotli/blob/master/research/dictio...
I have a hosted version of it on https://use-as-dictionary.com/ to make it easier to experiment with.
-
The Full-Stack development experience
An additional element that we can finally remove from our stack is the minification of JavaScript and CSS files. Thanks to algorithms like brotli (with a very Swiss flavour) we no longer need to minify and compress our files before distributing them. Cloudflare, Nginx, or Apache will take care of everything for us.
-
Brotli vs. GZIP vs. Zopfli: Comparing JavaScript Compression Techniques.
As you navigate the intricate world of JavaScript compression and web development, having a trusted partner by your side can make all the difference. That's where Coding Crafts comes in. At Coding Crafts, we take pride in being a top-tier software development company in USA. Our team of experts specializes in web development, optimization, and everything in between. As the best IT company in USA, we are dedicated to delivering cutting-edge solutions that drive performance and efficiency. Our expertise extends to choosing the right compression technique for your web application, ensuring that your website performs optimally. In conclusion, the choice of JavaScript compression technique depends on various factors, including your specific goals, browser and server support, and performance requirements. Whether you opt for Brotli, GZIP, or Zopfli, Coding Crafts is here to provide the guidance and expertise you need to enhance your web application's performance and user experience. For more information on how Coding Crafts can assist you with your web development and optimization needs, contact us today. Resources "Brotli - GitHub Repository": https://github.com/google/brotli "Zopfli - Google Developers": https://developers.google.com/speed/articles/zopfli "Introduction to GZIP Compression - MDN Web Docs": https://developer.mozilla.org/en-US/docs/Web/HTTP/Overview#gzip_compression "Brotli vs. GZIP vs. Zopfli: Which Compression Method is Best?" - KeyCDN Blog: https://www.keycdn.com/blog/brotli-vs-gzip-vs-zopfli
-
Framer Update: 2x Faster Sites
We serve your site from a global cache location close to your visitors to make sure your site loads fast. In addition, we use an advanced HTML and text compression algorithm called Brotli. Compressed content is now cached, so we can send it directly to your visitors instead of compressing each request individually. In our tests this often improves loading speed by up to 2x, which will have a very positive impact on your Lighthouse scores like LCP. This will be especially noticeable on larger sites, so you can scale your site without worry.
-
How Much Faster Is Making a Tar Archive Without Gzip?
For anyone who wants to try this, zstd -T0 uses all your threads to compress, and https://github.com/facebook/zstd has a lot more description. Brotli, https://github.com/google/brotli, is another modern format with some good features for high compression levels and Content-Encoding support in web browsers. You might also want to play with the compression level (-1 to -11 or more, zstd's --fast=n).
One reason these modern compressors do better is not any particular mistake made defining DEFLATE in the 90s, but that new algos use a few MB of recently seen data as context instead of 32KB, and do other things impractical in the 90s but reasonable on modern hardware. The new algorithms also contain logs of smart ideas and have fine-tuned implementations, but that core difference seems important to note.
-
Troubling Launching Duckstation
It seems to be using a lib called brotli - https://github.com/google/brotli. Can you compile from source?
-
2000’s Winamp/WMP-ish skins?
(++) Play Youtube videos and playlistsRelease date: Feb 3 20223dyd, [email protected] libraries: brotli https://github.com/google/brotli (1.0.7)
-
Stop delaying. Share knowledge on a blog built with Eleventy.
Alright, the minifying is done. What else? Did you know you can serve HTML, CSS and JS compressed? A lot of websites still use gzip, but there’s also Brotli. Brotli is specifically made for the web and compresses a lot better than gzip in most cases.
What are some alternatives?
tarindexer - python module for indexing tar files for fast access
Snappy - A fast compressor/decompressor
asar - Simple extensive tar-like archive format with indexing
LZ4 - Extremely Fast Compression algorithm
PyFilesystem2 - Python's Filesystem abstraction layer
zstd - Zstandard - Fast real-time compression algorithm
pixz - Parallel, indexed xz compressor
LZMA - (Unofficial) Git mirror of LZMA SDK releases
InstaPy - 📷 Instagram Bot - Tool for automated Instagram interactions
ZLib - A massively spiffy yet delicately unobtrusive compression library.
icoextract - Extract icons from Windows PE files (.exe/.dll)
zlib-ng - zlib replacement with optimizations for "next generation" systems.