DataflowTemplates
professional-services
DataflowTemplates | professional-services | |
---|---|---|
4 | 8 | |
1,092 | 2,729 | |
1.1% | 0.9% | |
9.8 | 9.1 | |
3 days ago | 9 days ago | |
Java | Python | |
Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
DataflowTemplates
-
Which Database to use for rest api
Google provide a Dataflow template for copying from BigQuery to Datastore, see this stack overflow answer.
- Sync Postgres to BigQuery, possible? How?
-
New to GCP - need help designing pipeline from production Heroku Postgres to BigQuery
Ah, looks like the template default appends new rows. If I want to overwrite the table, looks like I might be able to just replace this line in the template code to WRITE_TRUNCATE (see here). Cool!
-
Tricky Dataflow ep.1 : Auto create BigQuery tables in pipelines
However, learning to use Apache Beam, which is the open source framework behind Dataflow, is no bed of roses: The official documentation is sparse, GCP-provided templates don't work out-of-the-box, and the Javadoc is, well, a javadoc.
professional-services
-
What necessary principles need to be added to PubSub when connect it from Cloud Run?
I just did this for Pubsub2Inbox (added Cloud Run support), you can see Terraform example here.
-
Is there a repository of GCP script examples?
The Google Cloud Profession Services has published a bunch of scripts and tools. They also publish Cloud Foundations Toolkit which is a set of reference terraform and deployment manager templates for common cloud infrastructure.
-
My GCP feature requests for 2022
Hey, for the last topic, you can always use my Pubsub2Inbox tool (now with MS Graph support): https://github.com/GoogleCloudPlatform/professional-services/tree/main/tools/pubsub2inbox
-
Cloud Run pause in the middle of entrypoint
I'm fairly new to Cloud Run and containers, but did a bit of Cloud function. I'm trying to use this repo to gather data from our Cloud Storage to make a Data Studio page that runs from Cloud run. Link here.
-
Best way to trigger an archive function
Are you downloading the file from a bucket and reuploading it to another? You can just copy it as a metadata operation. See an example here: https://github.com/GoogleCloudPlatform/professional-services/pull/663
-
GCP terraform
Terraform examples for GCP from Google’s professional services team: https://github.com/GoogleCloudPlatform/professional-services
-
Pubsub2Inbox: swiss army knife Cloud Function for Pub/Sub messages
gcs2bq for creating GCS dashboards: https://github.com/GoogleCloudPlatform/professional-services/tree/main/tools/gcs2bq
What are some alternatives?
janusgraph - JanusGraph: an open-source, distributed graph database
Competitive-Programming
pgsink - Logically replicate data out of Postgres into sinks (files, Google BigQuery, etc)
python-docs-samples - Code samples used on cloud.google.com
yauaa - Yet Another UserAgent Analyzer
workflows-samples - This repository contains samples for Cloud Workflows.
debezium-examples - Examples for running Debezium (Configuration, Docker Compose files etc.)
bigquery-schema-generator - Generates the BigQuery schema from newline-delimited JSON or CSV data records.
migrate - Database migrations. CLI and Golang library.
example-quote-generator-app - A simple web application using a React front-end and a Python back-end API, both secured using ZITADEL.
bigquery-utils - Useful scripts, udfs, views, and other utilities for migration and data warehouse operations in BigQuery.