sqlite-utils-jq
cloudquery
sqlite-utils-jq | cloudquery | |
---|---|---|
2 | 102 | |
8 | 5,609 | |
- | 1.1% | |
3.7 | 10.0 | |
9 months ago | about 5 hours ago | |
Python | Go | |
- | Mozilla Public License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
sqlite-utils-jq
-
Welcome to Datasette Cloud
There are a few things you can do here.
SQLite is great at JSON - so I often dump JSON structures in a TEXT column and query them using https://www.sqlite.org/json1.html
I also have plugins for running jq() functions directly in SQL queries - https://datasette.io/plugins/datasette-jq and https://github.com/simonw/sqlite-utils-jq
I've been trying to drive the cost of turning semi-structured data into structured SQL queries down as much as possible with https://sqlite-utils.datasette.io - see this tutorial for more: https://datasette.io/tutorials/clean-data
This is also an area that I'm starting to explore with LLMs. I love the idea that you could take a bunch of messy data, tell Datasette Cloud "I want this imported into a table with this schema"... and it does that.
I have a prototype of this working now, I hope to turn it into an open source plugin (and Datasette Cloud feature) pretty soon. It's using this trick: https://til.simonwillison.net/gpt3/openai-python-functions-d...
-
SQLite Functions for Working with JSON
Since SQLite supports custom SQL functions, you can add JQ support to it pretty easily.
I just threw together a plugin for my sqlite-utils CLI tool that adds a jq() function here:
https://github.com/simonw/sqlite-utils-jq
Use it like this:
sqlite-utils memory "select jq(:doc, :expr) as result" \
cloudquery
-
We might want to regularly keep track of how important each server is
Check out CloudQuery - https://github.com/cloudquery/cloudquery for an easy cloud asset inventory.
-
Cloud asset tracking
There both do something like what you're looking for.... https://github.com/cloudquery/cloudquery https://github.com/openraven/magpie
-
Show HN: Nango – Open unified API for product integrations
Unified API is a holly grail but as many said quite difficult to abstract every use case in a scalable way that won't break. At CloudQuery (https://github.com/cloudquery/cloudquery) we focus solely on the ELT use-case(Founder/Maintainer here).
-
Welcome to Datasette Cloud
Congrats!! How does it compare to the ELT space and the modern data stack where you have ingestion/storage/visualization layers decoupled?
Asking as the founder of CloudQuery (https://github.com/cloudquery/cloudquery), Saw Datasette quite a few times around data exploration but curious to hear about the most popular use-cases of Datasette!
-
Launch HN: PeerDB (YC S23) – Fast, Native ETL/ELT for Postgres
Congrats!! We also focus on performance at CloudQuery (https://github.com/cloudquery/cloudquery) by using Golang, gRPC and still trying to be abstract enough to support different databases :)
In any case good luck!
-
airbyte VS cloudquery - a user suggested alternative
2 projects | 2 Jun 2023
CloudQuery for ETL
2 projects | 2 Jun 2023Another ELT framework that's an alternative to Airbyte
-
meltano VS cloudquery - a user suggested alternative
2 projects | 2 Jun 2023
Another alternate ELT
-
RDS to S3 Options
Check out CloudQuery, we have PostgreSQL source connectors and S3 destination that supports parquet (Disclaimer: Maintainer and founder here)
-
Cloudquery, Resoto, Steampipe, or Airbyte?
Hello! Im Yevgeny, Founder & maintainer at CloudQuery . We've built CloudQuery as an open source high performance ELT framework so you should get pretty good results syncing all your cloud assets from high number of accounts (we have users syncing more than 10K Azure subscription and thousands of AWS accounts concurrently).
What are some alternatives?
pyjq - A Python binding for ./jq
steampipe - Zero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No DB required.
sqlitebson - BSON extension for sqlite
steampipe-mod-aws-compliance - Run individual controls or full compliance benchmarks for CIS, PCI, NIST, HIPAA and more across all of your AWS accounts using Powerpipe and Steampipe.
sqlite-utils-litecli - Interactive shell for sqlite-utils using litecli
cloud-custodian - Rules engine for cloud security, cost optimization, and governance, DSL in yaml for policies to query, filter, and take actions on resources
sqlite-utils - Python CLI utility and library for manipulating SQLite databases
cloudsploit - Cloud Security Posture Management (CSPM)
litecli - CLI for SQLite Databases with auto-completion and syntax highlighting
cartography - Cartography is a Python tool that consolidates infrastructure assets and the relationships between them in an intuitive graph view powered by a Neo4j database.
grist-core - Grist is the evolution of spreadsheets.
opencspm - Open Cloud Security Posture Management Engine