ParlAI VS lrzip

Do you have https://github.com/facebookresearch/ParlAI installed? Looks like they're doing something weird with their flake8 config.

I would suspect similar to blenderbot2 from meta and parl.ai.

There's quite a few open-source Reinforcement Learning challenges that you can explore with modest amounts of compute in order to build some experience training RL models, for example the Nethack Learning Environment, Atari, Minigrid, etc. For me personally, I had only worked in NLP / dialogue for years but got into RL by implementing Random Network Distillation models for NetHack. It's a fun area that definitely has its own unique challenges vs other domains. -AM

I am intrigued by the model because it develops long term memory that it can access for future conversations which you can see in more detail on the model cards.

Continue reading | Check out the paper, project, github link and reference article.

I have tried to use parl.ai in the past. I actually wanted to play with blenderbot 1.0. I kinda hate this library because it isn't exactly quick and easy to learn. I ended up using the Huggingface version.

You probably meant to link this: https://github.com/facebookresearch/ParlAI/blob/main/project...

lrzip

Long Range ZIP or LZMA RZIP

https://github.com/ckolivas/lrzip

"A compression utility that excels at compressing large files (usually > 10-50 MB). Larger files and/or more free RAM means that the utility will be able to more effectively compress your files (ie: faster / smaller size), especially if the filesize(s) exceed 100 MB. You can either choose to optimise for speed (fast compression / decompression) or size, but not both."

7zip and XZ are almost always the best in any comparison. (They use the same algorithm.) Occasionally something new comes allong that may be bettyer, but it fades away... Like lrzip. https://lkml.org/lkml/2011/6/4/23 https://github.com/ckolivas/lrzip

For example lrzip has an intense "dupe hunting" mode and takes days for large content, but does compress very well once it's done (and expansion is fast). I use it on long term storage backups and disk images and junk. Completely incompatible with streaming, unlike chunk-based like gzip or deflate or etc, although unpacking can stream such as searching or verifying a tarfile archive. But the original source has to be file-based so seeking for the hunting can work across the entire file-as-a-block.

Best I know of for that is something like lrzip still, but even then it's probably not state of the art. https://github.com/ckolivas/lrzip

It'll also take a hell of a long time to do the compression and decompression. It'd probably be better to do some kind of chunking and deduplication instead of compression itself simply because I don't think you're ever going to have enough ram to store any kind of dictionary that would effectively handle so much data. You'd also not want to have to re-read and reconstruct that dictionary to get at some random image too.

There's also lrzip for large files: https://github.com/ckolivas/lrzip