smart_open VS RE2

We’re using smart_open, which is an amazing library that lets you open objects in S3 (and other cloud object stores) as if they’re files on your filesystem. It’s obviously critical that we’re able to seek to an arbitrary position in an S3 file without first downloading the whole thing. We’ll assume you’re using Poetry, but you should be able to follow along with any other package manager:

If you didn’t follow along with the first article in this series, you should be able to follow this article with your own dataset as long as you install smart_open and Meadowrun. smart_open is an amazing library that lets you open objects in S3 (and other cloud object stores) as if they’re files on your filesystem, and Meadowrun makes it easy to run your Python code on the cloud.

I see that you're primarily looking into Python work, so I'd recommend `smart_open` as a nice, compact way to get started.

https://github.com/RaRe-Technologies/smart_open

You want smart_open. It gives you a (more complete) file-like interface to many different storage systems, including s3. You can read and seek as needed.

See also smart_open: https://github.com/RaRe-Technologies/smart_open which might be more user-friendly? Never used it myself but it was on HN before. Discussion on their bugtracker: https://github.com/RaRe-Technologies/smart_open/issues/579

Looking at the benchmark where C++ is worst compared to other languages, it's depending on the library used. I would guess if they used Google's re2 Regex library instead of Boost's, the result would be different.

https://github.com/google/re2

https://github.com/greensoftwarelab/Energy-Languages/blob/ma...

That page says it just includes "some of the most common special characters", and following the link to the Examples page in turn includes a link to the full list.

Python uses backtracking, so this probably isn't O(n), especially with the ability to choose the dictionary.

But with there are non-backtracking matchers which would make this O(n). Here's re2 from https://github.com/google/re2 :

  >>> import re2

RE2 is a Google regular expression library

google's regex library: https://github.com/google/re2

I am currently working with Regex, specifically Re2, and was wondering if there is a real difference between the above expressions for repeated sub-regex.

I ported Google RE2, a regular expression library, to SPVM as Resource::Re2, and created SPVM::Regex, a wrapper for it.

I searched for I found that there is a Perl compatible regular expression called Google RE2. It is written in C++, and with Google RE2, I can use Perl-compatible regular expressions as a library.

Yes, but there is an interesting clarification here. RE2 has used the "caching" approach documented in the Ruby bug ticket linked for quite some time (since its birth?): https://github.com/google/re2/blob/954656f47fe8fb505d4818da1...

It is mentioned only briefly in Cox's article on regex matching in the wild. Look for the word "bitstate": https://swtch.com/~rsc/regexp/regexp3.html

I didn't know Perl had implemented this trick too.

The paper[1] cited in the Ruby bug ticket was published very recently. When I first read the Ruby bug ticket, I immediately wondered how they sidestepped the memory use problem. The paper's abstract seems to suggest there is some technique for doing so, as it rebuffs the idea of doing "full" memoization. Alas, I do not have access the paper. (Which is fucking ridiculous.)

[1]: https://ieeexplore.ieee.org/abstract/document/9519427