Our great sponsors
-
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
python-remotezip
Python module to access single members of a zip archive without downloading the full content from a remote web server.
-
BrowserBoxPro
Discontinued :cyclone: BrowserBox is Web application virtualization via zero trust remote browser isolation and secure document gateway technology. Embed secure unrestricted webviews on any device in a regular webpage. Multiplayer embeddable browsers, open source! [Moved to: https://github.com/BrowserBox/BrowserBox]
https://github.com/mafintosh/csv-parser/pull/121 https://github.com/mafintosh/csv-parser/pull/151 https://github.com/mafintosh/csv-parser/issues/218
The author of the library probably has learned, the hard way many many lessons (and probably also decided to prioritize some of the requested issues / feature requests along the way).
The above is not meant as a ding on the project itself and I am sure it is used successfully by many people. The point here is that your claim that you can easily write a csv parser in 200 lines of code does not hold water. It's anything but easy and you should use a battle tested library and not reinvent the wheel.
The EDGAR API bulk data is similar in nature, albeit in JSON instead.[1]
[1]: https://github.com/andrewmcwattersandco/programming-language...
The EDGAR API bulk data is similar in nature, albeit in JSON instead.[1]
[1]: https://github.com/andrewmcwattersandco/programming-language...
You might find a lot of interesting tools in the Datasette ecosystem. Data dashboarding for SQLite with all sorts of import and export and visualization plugins.
https://datasette.io/
qsv [1] also has a sqlp command which lets you run Polars SQL queries (even on multiple files). Here I'll send the csv data from stdin (represented by -) and then (optionally) pipe the output to the table command for formatting. The shape of the result is also printed to stderr (the (4, 2) below).
[1] https://github.com/jqnatividad/qsv
$ echo 'Name,Department,Salary
Agreed! People are surprised but CSV files (while I would use them for everything) work great in low-tech environment, or when files must be produced by many different actors with different technical levels.
One concrete example is the French standard to describe Electrical Vehicles Charge Points, which is made of 2 parts (static = position & overall description, dynamic = current state, occupancy etc). Both "files" are just CSV files:
https://github.com/etalab/schema-irve
Both sub-formats are specified via TableSchema (https://frictionlessdata.io/schemas/table-schema.json).
Files are produced directly by electrical charge points operators, which can have widely different sizes & technicality, so CSV works nicely in that case.
Also, when the server allows the range requests for zip files, the zip files are huge and one needs just a few files, one can actually download just the "central directory" and the data of the needed compressed files without downloading the whole file:
https://github.com/gtsystem/python-remotezip
I'm using this type of simple approach to build a SaaS right now. We need to spin up many VPS and provision them, and the fastest way to do that is with rsync and ssh.
But we didn't stop there: this SaaS for our open source browser product is entirely built like this^0: behind the scenes it's a collection of bash scripts that implement and execute the business operation of the SaaS.
So basically, it's a command-line interface to the SaaS. Think of it this way, say I didn't have a website, with login, and "click a button to open a browser", but instead people would write me letters, send me cheques, or call me on the phone. Then I can serve their requests manually, at the command line.
The reason I made it like this was:
- clear separation between thin web front-end and actual business logic
- nice command-line interface (options, usage, help, clear error messages) to business logic for maintenance and support to jump on and fix things
- inheritance of operating system permissions and user process isolation
- highly testable implementation
Maybe this is dumb, but I really like it. To me it's an architecture and approach that makes sense.
I'm sure this is not new, and I think a lot of good quality operations must be built via this way. I highly align with the author's stance of the composition of a few simple command line tools to get the job done.
Perhaps we can call this "unix driven development", or "unix-philosophy backend engineering"
0: https://github.com/dosyago/BrowserBoxPro (saas coming soonish)