How can I efficiently search for a specific string in a large text file using C#?

1. ripgrep

   1 366 52,481 7.6 Rust

   ripgrep recursively searches directories for a regex pattern while respecting your gitignore

   

   Right. The "generic SIMD" algorithm is one I'm quite familiar with and have implemented. It's what ripgrep uses for example, although it's a little smarter than "just take the first and last bytes." ripgrep tries to guess at which bytes are the best to pick to maximize throughput by reducing false positives in the initial candidate scan. You can see the implementation here: https://github.com/BurntSushi/memchr/tree/master/src/memmem

2. InfluxDB

   www.influxdata.com featured

   InfluxDB – Built for High-Performance Time Series Workloads. InfluxDB 3 OSS is now GA. Transform, enrich, and act on time series data directly in the database. Automate critical tasks and eliminate the need to move data externally. Download now.

   

3. Apache Lucene

   2 7 2,309 9.2 C#

   Apache Lucene.NET

   

4. StreamRegex

   3 1 20 5.7 C#

   A .NET Standard 2.1+ Library to perform string parsing operations on Streams and StreamReaders. Includes Extensions for Regex.

   

5. nlc

   4 2 9 10.0 C#

   Line counter written in C# targeting .NET 6

   

6. fnlc

   5 2 13 1.3 C#

   A line-counter written in C# and using Intrinsics

   

7. Microsoft.IO.RecyclableMemoryStream

   6 10 2,057 6.6 C#

   A library to provide pooling for .NET MemoryStream objects to improve application performance.

   

   Another suggestion to try, there is a tool provided by Microsoft called Microsoft.IO.RecyclableMemoryStream which greatly reduces the amount of memory to garbage collect when streaming large amounts of data.

8. rust-memchr

   7 32 1,079 6.6 Rust

   Optimized string search routines for Rust.

   

   Right. The "generic SIMD" algorithm is one I'm quite familiar with and have implemented. It's what ripgrep uses for example, although it's a little smarter than "just take the first and last bytes." ripgrep tries to guess at which bytes are the best to pick to maximize throughput by reducing false positives in the initial candidate scan. You can see the implementation here: https://github.com/BurntSushi/memchr/tree/master/src/memmem

9. SaaSHub

   www.saashub.com featured

   SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

   

Suggest a related project