steampipe
metriql
Our great sponsors
steampipe | metriql | |
---|---|---|
146 | 7 | |
6,379 | 284 | |
2.4% | 0.4% | |
9.7 | 1.9 | |
5 days ago | about 1 year ago | |
Go | Kotlin | |
GNU Affero General Public License v3.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
steampipe
- Steampipe: Dynamically query APIs, code and more with SQL
-
Cloud Tools You Probably Haven't Heard Of
Steampipe is a tool for querying cloud APIs and other data sources using SQL in a zero-ETL manner.
-
Show HN: Query Your Sheets with SheetSQL
Readers may also enjoy Steampipe [1], an open source CLI to live query Google Sheets [2] and 140+ other services with SQL (e.g. AWS, GitHub, etc). It uses Postgres Foreign Data Wrappers under the hood and supports joins etc across the services. (Disclaimer - I'm a lead on the project.)
1 - https://github.com/turbot/steampipe
-
Osquery: An sqlite3 virtual table exposing operating system data to SQL
be mindful of its AGPLv3 https://github.com/turbot/steampipe/blob/v0.21.8/LICENSE (AFAIK v0.4.3 is the last MIT release https://github.com/turbot/steampipe/blob/v0.4.3/LICENSE ) and the actual providers are Apache 2 <https://github.com/turbot/steampipe-plugin-aws/blob/v0.131.0...> (but I don't know if provider drift makes them compatible with 0.4 or not)
iasql seems to be AWS only, but good for them for taking this on:
-
How to run an AWS CIS v3.0 assessment in CloudShell
In a prior post I showed how to install Steampipe in AWS CloudShell to instantly query over 460+ resource types from your AWS APIs using SQL, and another post on how to use the Steampipe AWS Compliance mod to assess over 25+ security benchmarks across your AWS accounts.
- Git Query Language
- Query Cloud and SaaS APIs with SQL
-
Cutting down AWS cost by $150k per year simply by shutting things off
Readers may find Steampipe's [1] AWS Thrifty Mod [2] useful. It will automatically scan multiple accounts and regions for 50 cost saving opportunities - many of which are looking for over-provisioned or unused resources. For example, it's crazy how much you can save by doing things like just converting your EBS volumes to the newer gp3 type. Combine with Flowpipe [3] to automate checks and actions. It's all open source and extensible.
1 - https://github.com/turbot/steampipe
- FLaNK Weekly 08 Jan 2024
-
Zero-ETL for Postgres: Live-query cloud APIs with 100 open source FDWs
Steampipe [1] is an open source project [2] that includes an embedded Postgres to instantly query cloud, code & more with SQL. This release expands our plugin ecosystem [3] to be a full Zero-ETL platform. Steampipe plugins can now run natively in your own Postgres as Foreign Data Wrappers [4], as SQLite extensions [5] or as simple data export tools [6]. Please give it a try, we'd love your feedback and contributions!
1 - https://steampipe.io
metriql
-
Getting started with a metrics store
Some of the companies that operate in space are Cube Dev; Transform(currently acquired by dbt); metriql. See more companies at https://www.moderndatastack.xyz/companies/metrics-store.
-
Launch HN: Hydra (YC W22) – Query Any Database via Postgres
Presto is pretty successful but its focus is to be distributed query engine, not a proxy layer for the existing query engines. We use Trino ( formerly Presto) as our query layer and do something similar to Hydra at Metriql [1] with a fairly different use-case. Data people provide a semantic layer with the mecrics and expose them to 18+ downstream tools.
[1]: https://metriql.com
-
How do you separate ML from analytics in your data pipeline?
This is why metrics store tooling have started appearing recently (e.g. TransformData, SuperGrain, Metriql, dbt Metrics) - to solve the problem of this table / metric disorganization across an org's data landscape.
-
Open source Business intelligence platform made with Python
We're using Superset to enable our analysts to explore our clients' SEM/SEO/analytics data. It also posts alerts to Slack when, say, the daily session count of a website isn't what was expected given the historical data.
Yeah, it's a little rough to get going, but once it is, we've found it to be a really powerful (and actively developed!) BI tool. It's even better with dbt + MetriQL [0], which can automatically sync Superset's dataset metadata directly with properties you set up in dbt.
Adding custom visualizations is much harder than it should be, but they're very much aware of that, and working to address it. Their Slack community is super-helpful, too.
[0]: https://metriql.com
-
Show HN: Low-Code Metrics Store
As a current Looker power-user, this looks really solid.
One thing I’m not sure about though: can you use the metrics outside of the native tool, and if so how?
That is, I see Looker as a BI tool, not a metrics layer, since you mainly use the metrics you define inside Looker, not in other tools. On the other hand, something like MetriQL[0] is a pure metrics layer that can supposedly be used anywhere.
Is this both? If so, some better documentation around how to use the metrics layer would be helpful (or maybe I just didn’t look in the right place).
[0] https://metriql.com/
-
Notes on the Perfidy of Dashboards
3. Define metrics in one place on top of your data models and expose the metrics to all the data tools. (This layer is new, and we're tapping it at https://metriql.com)
-
Launch HN: Evidence (YC S21) – Web framework for data analysts
We use BSL license and metriql is free with a single database target. If you want to connect multiple dbt projects in a single deployment, you need to go through the sales cycle.
We work with ETL vendors that use metriql to make revenue with our BI tool integrations so we picked BSL license to be able to structure our business model in a way that you should be required to pay only if you're reselling metriql to your customers.
You can find the license here: https://github.com/metriql/metriql
What are some alternatives?
cloudquery - The open source high performance ELT framework powered by Apache Arrow
cube.js - 📊 Cube — The Semantic Layer for Building Data Applications
cloud-custodian - Rules engine for cloud security, cost optimization, and governance, DSL in yaml for policies to query, filter, and take actions on resources
evidence - Business intelligence as code: build fast, interactive data visualizations in pure SQL and markdown
inspec-aws - InSpec AWS Resource Pack https://www.inspec.io/
mlcraft - Synmetrix – open source semantic layer / Boost your LLM precision
steampipe-mod-github-sherlock - Interrogate your GitHub resources with the help of the world's greatest detectives: Powerpipe + Steampipe + Sherlock.
examples - Example apps and instrumentation for Honeycomb
embedded-postgres-binaries - Lightweight bundles of PostgreSQL binaries with reduced size intended for testing purposes.
csv-metabase-driver - A CSV metabase driver
dockertest - Write better integration tests! Dockertest helps you boot up ephermal docker images for your Go tests with minimal work.
Multicorn - Data Access Library