airbyte VS sqlfluff

AgentCloud uses Airbyte to build data pipelines, which allow us to split, chunk, and embed data from over 300 data sources, including Postgres.

I'l also give a shout-out to Airbyte (https://airbyte.com/), with which I've had some limited success with integrating Salesforce to a local database. The particular pull for Airbyte is that we can self-host the open source version, rather than pay Fivetran a significant sum to do this for us.

It's an immature tool, so I don't yet know that I can claim we've spent _less_ than Fivetran on the additional engineering and ops time, but it feels like it has potential to do so once stabilized.

Link to GitHub -->

When it says "available in open source", is that under the main airbyte repo's licensing [1], hence primarily licensed under the Elastic License v2 and therefore not typically considered open source by many?

Airbyte has previous of advertising their offering as open source while not really being as per the OSD[2]. This has been raised with them previously but without response [3][4]. They've also been extending their use of ELv2, recently moving many of their existing MIT licensed connectors to be ELv2 [5].

[1] https://github.com/airbytehq/airbyte/blob/master/LICENSE

As possible solution, I can suggest Airbyte(https://airbyte.com/). it's more performant than generic python script.

Airbyte, an open-source data integration engine that offers hundreds of connectors with data warehouses and databases, has gained popularity for its seamless integration and data syncing capabilities. Xata's integration with Airbyte offers a streamlined data ingestion process from any Airbyte input source directly into your Xata database.

You can try and contribute some new connectors/operators for workflow managers like Airflow or Airbyte

Repo : https://github.com/sqlfluff/sqlfluff

I'm surprised sqlfluff hasn't been mentioned yet. Perhaps not a comprehensive list, but it's worked for everything I've thrown at it. There's an ANSI keyword list [0], and then dialect-specific lists for everything from DB2 [1] to Snowflake [2].

[0]: https://github.com/sqlfluff/sqlfluff/blob/main/src/sqlfluff/...

It has tons of annoying quirks, but I couldn't imagine running a DBT project without it: https://github.com/sqlfluff/sqlfluff

Move SQL queries to sql files and read from files (Use sqlfluff to lint the code https://github.com/sqlfluff/sqlfluff)

SQLFluff is a good CLI tool for this and includes support for jinja and dbt. I don't think there's a VSCode plugin for it yet.

This linter can really enforce some best practices https://github.com/sqlfluff/sqlfluff

A list of best practices:

BigQuery SQL and SQLFluff

There's a SQL parser/linter called SQLFluff that my team uses for our CI/CD. I've made a few pull requests to fix the parser for the particular SQL dialect we used, and my college compiler classes definitely helped.