DataflowTemplates
Cloud Dataflow Google-provided templates for solving in-Cloud data tasks (by GoogleCloudPlatform)
professional-services
Common solutions and tools developed by Google Cloud's Professional Services team. This repository and its contents are not an officially supported Google product. (by GoogleCloudPlatform)
| DataflowTemplates | professional-services | |
|---|---|---|
| 4 | 8 | |
| 1,092 | 2,729 | |
| 1.1% | 0.9% | |
| 9.8 | 9.1 | |
| 3 days ago | 9 days ago | |
| Java | Python | |
| Apache License 2.0 | Apache License 2.0 |
The number of mentions indicates the total number of mentions that we've tracked plus the number of user suggested alternatives.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
DataflowTemplates
Posts with mentions or reviews of DataflowTemplates.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2021-04-05.
-
Which Database to use for rest api
Google provide a Dataflow template for copying from BigQuery to Datastore, see this stack overflow answer.
- Sync Postgres to BigQuery, possible? How?
-
New to GCP - need help designing pipeline from production Heroku Postgres to BigQuery
Ah, looks like the template default appends new rows. If I want to overwrite the table, looks like I might be able to just replace this line in the template code to WRITE_TRUNCATE (see here). Cool!
-
Tricky Dataflow ep.1 : Auto create BigQuery tables in pipelines
However, learning to use Apache Beam, which is the open source framework behind Dataflow, is no bed of roses: The official documentation is sparse, GCP-provided templates don't work out-of-the-box, and the Javadoc is, well, a javadoc.
professional-services
Posts with mentions or reviews of professional-services.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2022-04-29.
-
What necessary principles need to be added to PubSub when connect it from Cloud Run?
I just did this for Pubsub2Inbox (added Cloud Run support), you can see Terraform example here.
-
Is there a repository of GCP script examples?
The Google Cloud Profession Services has published a bunch of scripts and tools. They also publish Cloud Foundations Toolkit which is a set of reference terraform and deployment manager templates for common cloud infrastructure.
-
My GCP feature requests for 2022
Hey, for the last topic, you can always use my Pubsub2Inbox tool (now with MS Graph support): https://github.com/GoogleCloudPlatform/professional-services/tree/main/tools/pubsub2inbox
-
Cloud Run pause in the middle of entrypoint
I'm fairly new to Cloud Run and containers, but did a bit of Cloud function. I'm trying to use this repo to gather data from our Cloud Storage to make a Data Studio page that runs from Cloud run. Link here.
-
Best way to trigger an archive function
Are you downloading the file from a bucket and reuploading it to another? You can just copy it as a metadata operation. See an example here: https://github.com/GoogleCloudPlatform/professional-services/pull/663
-
GCP terraform
Terraform examples for GCP from Google’s professional services team: https://github.com/GoogleCloudPlatform/professional-services
-
Pubsub2Inbox: swiss army knife Cloud Function for Pub/Sub messages
gcs2bq for creating GCS dashboards: https://github.com/GoogleCloudPlatform/professional-services/tree/main/tools/gcs2bq
What are some alternatives?
When comparing DataflowTemplates and professional-services you can also consider the following projects:
janusgraph - JanusGraph: an open-source, distributed graph database
Competitive-Programming
pgsink - Logically replicate data out of Postgres into sinks (files, Google BigQuery, etc)
python-docs-samples - Code samples used on cloud.google.com
yauaa - Yet Another UserAgent Analyzer
workflows-samples - This repository contains samples for Cloud Workflows.
debezium-examples - Examples for running Debezium (Configuration, Docker Compose files etc.)
bigquery-schema-generator - Generates the BigQuery schema from newline-delimited JSON or CSV data records.
migrate - Database migrations. CLI and Golang library.
example-quote-generator-app - A simple web application using a React front-end and a Python back-end API, both secured using ZITADEL.
bigquery-utils - Useful scripts, udfs, views, and other utilities for migration and data warehouse operations in BigQuery.
DataflowTemplates vs janusgraph
professional-services vs Competitive-Programming
DataflowTemplates vs pgsink
professional-services vs python-docs-samples
DataflowTemplates vs yauaa
professional-services vs workflows-samples
DataflowTemplates vs debezium-examples
professional-services vs bigquery-schema-generator
DataflowTemplates vs migrate
professional-services vs example-quote-generator-app
DataflowTemplates vs bigquery-utils