DataflowTemplates
bigquery-utils
DataflowTemplates | bigquery-utils | |
---|---|---|
4 | 6 | |
1,092 | 1,031 | |
1.1% | 1.0% | |
9.8 | 6.1 | |
3 days ago | 6 days ago | |
Java | Java | |
Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
DataflowTemplates
-
Which Database to use for rest api
Google provide a Dataflow template for copying from BigQuery to Datastore, see this stack overflow answer.
- Sync Postgres to BigQuery, possible? How?
-
New to GCP - need help designing pipeline from production Heroku Postgres to BigQuery
Ah, looks like the template default appends new rows. If I want to overwrite the table, looks like I might be able to just replace this line in the template code to WRITE_TRUNCATE (see here). Cool!
-
Tricky Dataflow ep.1 : Auto create BigQuery tables in pipelines
However, learning to use Apache Beam, which is the open source framework behind Dataflow, is no bed of roses: The official documentation is sparse, GCP-provided templates don't work out-of-the-box, and the Javadoc is, well, a javadoc.
bigquery-utils
-
Swirl: An open-source search engine with LLMs and ChatGPT to provide all the answers you need 🌌
Using the Galaxy UI, knowledge workers can systematically review the best results from all configured services including Apache Solr, ChatGPT, Elastic, OpenSearch, PostgreSQL, Google BigQuery, plus generic HTTP/GET/POST with configurations for premium services like Google's Programmable Search Engine, Miro and Northern Light Research.
-
Modern data stack: scaling people and technology at FINN
Data Transformations: This phase involves modifying and integrating tables to generate new tables optimized for analytical use. Consider this example: you want to understand the purchasing behavior of customers aged between 20-30 in your online shop. This means you'll need to join product, customer, and transaction data to create a unified table for analytics. These data preparation tasks (e.g., joining fragmented data) for analysis are essentially what "Data Transformations" entail. At FINN, technologies utilized in this phase include BigQuery as a data warehouse, dbt for data transformation, and a combination of GitHub Actions and Datafold for quality assurance.
-
Running Transformations on BigQuery using dbt Cloud: step by step
Introduction In today's data-driven world, transforming raw data into valuable insights is crucial. This process, however, often involves complex tasks that demand efficiency, scalability, and reliability. Enter dbt Cloud—a powerful tool that simplifies data transformations on Google BigQuery. In this article, we'll take you through a step-by-step guide on how to run BigQuery transformations using dbt Cloud. Let's dive in!
-
Do I need a cloud computing–based data cloud company
You'll want to evaluate what BigQuery has to offer and see if it makes sense for you to move over.
-
I used ChatGPT to get an Internship
Watch the introductory videos on BigQuery on the Google Cloud Platform website (https://cloud.google.com/bigquery)
-
Wrangling BigQuery at Reddit
Within the audit logs you can find BigQueryAuditMetadata details in the protoPayload.metadataJson submessage in the Cloud Logging LogEntry message. GCP has offered several versions of BigQuery audit logs so there are both older “v1” and newer “v2” versions. The v1 logs report API invocations and live within the protoPayload.serviceData submessage while the v2 logs report resource interactions like which tables were read from and written to by a given query or which tables expired. The v2 data lives in a new field formatted as a JSON blob within the BigQueryAuditMetadata detail inside the protoPayload.metadataJson submessage. In v2 logs the older protoPayload.serviceData submessage does exist for backwards compatibility but the information is not set or used. We scrape details from the JobChange object instead. We referenced the GCP bigquery-utils Git repo for how to use INFORMATION_SCHEMA queries and audit logs queries.
What are some alternatives?
janusgraph - JanusGraph: an open-source, distributed graph database
solr - Apache Solr open-source search software
pgsink - Logically replicate data out of Postgres into sinks (files, Google BigQuery, etc)
swirl-search - Swirl is an open-source search platform that uses AI to search multiple content and data sources simultaneously and return AI-ranked results. And provides summaries of your answers from searches using LLMs. It's a one-click, easy-to-use Retrieval Augmented Generation (RAG) Solution.
professional-services - Common solutions and tools developed by Google Cloud's Professional Services team. This repository and its contents are not an officially supported Google product.
dataproc-templates - Dataproc templates and pipelines for solving simple in-cloud data tasks
yauaa - Yet Another UserAgent Analyzer
spark-bigquery-connector - BigQuery data source for Apache Spark: Read data from BigQuery into DataFrames, write DataFrames into BigQuery tables.