Effecient way for creating partition in large file to feed in to a node worker thread

This page summarizes the projects mentioned and recommended in the original post on /r/node

SurveyJS - Open-Source JSON Form Builder to Create Dynamic Forms Right in Your App
With SurveyJS form UI libraries, you can build and style forms in a fully-integrated drag & drop form builder, render them in your JS app, and store form submission data in any backend, inc. PHP, ASP.NET Core, and Node.js.
surveyjs.io
featured
InfluxDB - Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
featured
  • csv-parser

    Streaming csv parser inspired by binary-csv that aims to be faster than everyone else

  • I was using the [csv-parser](https://github.com/mafintosh/csv-parser) library to handle csv parsing in node. The file can be huge ranging from 50,000 to 500,000 lines, maybe even larger. I had to perform some computations on the csv, after this is submitted to the server for that I was thinking of dividing the csv into chunks which I could then provide to worker threads for performing the computation. The worker thread would get the number of lines to skip and then start reading the lines after that up to a particular limit. I create a read stream and pass the csv-parser in with the option of number of lines to skip. I tried to perform some benchmarks on it but could find no visible benefits between skipping lines and not skipping lines. Even if I read the whole file it was sometimes faster than reading the ending 30,000 lines.

  • SurveyJS

    Open-Source JSON Form Builder to Create Dynamic Forms Right in Your App. With SurveyJS form UI libraries, you can build and style forms in a fully-integrated drag & drop form builder, render them in your JS app, and store form submission data in any backend, inc. PHP, ASP.NET Core, and Node.js.

    SurveyJS logo
NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts

  • Advanced RAG with guided generation

    2 projects | dev.to | 18 Apr 2024
  • Lezer: A Parsing System for CodeMirror, Inspired by Tree-Sitter

    9 projects | news.ycombinator.com | 24 Mar 2024
  • Creating excerpts in Astro

    4 projects | dev.to | 14 Mar 2024
  • Parsing PDFs in Node.js

    5 projects | dev.to | 12 Mar 2024
  • How to Create a Real-time Public Transportation Schedule App

    4 projects | dev.to | 1 Mar 2024