sqlite-parquet-vtable

A SQLite vtable extension to read Parquet files (by cldellow)
Add to my DEV experience Suggest topics
Source Code
sqlite-parquet-vtable Reviews
Suggest alternative
Edit details
Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
Sponsored

Sqlite-parquet-vtable Alternatives

Similar projects and alternatives to sqlite-parquet-vtable

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a better sqlite-parquet-vtable alternative or higher similarity.
Suggest an alternative to sqlite-parquet-vtable

sqlite-parquet-vtable reviews and mentions

Posts with mentions or reviews of sqlite-parquet-vtable. We have used some of these posts to build our list of alternatives and similar projects. The last one was on 2022-06-24.
  • Universal Database
    1 project | /r/learnpython | 10 Jul 2022
    Sqlite3 has parquet extension that support parquet (https://github.com/cldellow/sqlite-parquet-vtable) as virtual table. I use sqlite3 a lot, for work and personally. It's really good, but I do have issue with large datasets, mainly due VACUUM operations. Insertion rate drops significantly when single table hits around 20M rows. Indexing is important for your query speed, but it'll impact your write speed.
  • Show HN: Easily Convert WARC (Web Archive) into Parquet, Then Query with DuckDB
    3 projects | news.ycombinator.com | 24 Jun 2022
    Well there's a virtual table extension to read parquet files in SQLite. I've not tried it myself. https://github.com/cldellow/sqlite-parquet-vtable
  • One-liner for running queries against CSV files with SQLite
    20 projects | news.ycombinator.com | 21 Jun 2022
    /? sqlite arrow

    - "Comparing SQLite, DuckDB and Arrow with UN trade data" https://news.ycombinator.com/item?id=29010103 ; partial benchmarks of query time and RAM requirements [relative to data size] would be

    - https://arrow.apache.org/blog/2022/02/16/introducing-arrow-f... :

    > Motivation: While standards like JDBC and ODBC have served users well for decades, they fall short for databases and clients which wish to use Apache Arrow or columnar data in general. Row-based APIs like JDBC or PEP 249 require transposing data in this case, and for a database which is itself columnar, this means that data has to be transposed twice—once to present it in rows for the API, and once to get it back into columns for the consumer. Meanwhile, while APIs like ODBC do provide bulk access to result buffers, this data must still be copied into Arrow arrays for use with the broader Arrow ecosystem, as implemented by projects like Turbodbc. Flight SQL aims to get rid of these intermediate steps.

    > - One cannot create a trigger on a virtual table.

    Just posted about eBPF a few days ago; opcodes have costs that are or are not costed: https://news.ycombinator.com/item?id=31688180

    > - One cannot create additional indices on a virtual table. (Virtual tables can have indices but that must be built into the virtual table implementation. Indices cannot be added separately using CREATE INDEX statements.)

    It looks like e.g. sqlite-parquet-vtable implements shadow tables to memoize row group filters. How does JOIN performance vary amongst sqlite virtual table implementations?

    > - One cannot run ALTER TABLE ... ADD COLUMN commands against a virtual table.

    Are there URIs in the schema? Mustn't there thus be a meta-schema that does e.g. nested structs with portable types [with URIs], (and jsonschema, [and W3C SHACL])?

    /? sqlite arrow virtual table

    - sqlite-parquet-vtable reads parquet with arrow for SQLite virtual tables https://github.com/cldellow/sqlite-parquet-vtable :

      $ sqlite/sqlite3
  • Show HN: WarcDB: Web crawl data as SQLite databases
    3 projects | news.ycombinator.com | 19 Jun 2022
    https://github.com/cldellow/sqlite-parquet-vtable

    But for my use case virtual would be too complicated.

  • A note from our sponsor - WorkOS
    workos.com | 23 Apr 2024
    The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning. Learn more →

Stats

Basic sqlite-parquet-vtable repo stats
4
261
10.0
almost 3 years ago

cldellow/sqlite-parquet-vtable is an open source project licensed under Apache License 2.0 which is an OSI approved license.

The primary programming language of sqlite-parquet-vtable is C++.

Popular Comparisons

  1. sqlite-parquet-vtable VS duckdb
  2. sqlite-parquet-vtable VS WarcDB
  3. sqlite-parquet-vtable VS zsv
  4. sqlite-parquet-vtable VS sqlite_protobuf
  5. sqlite-parquet-vtable VS visidata
  6. sqlite-parquet-vtable VS datasette
  7. sqlite-parquet-vtable VS qhs
  8. sqlite-parquet-vtable VS tidy-viewer
  9. sqlite-parquet-vtable VS csvq
  10. sqlite-parquet-vtable VS octosql

Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com