yauaa
DataflowTemplates
Our great sponsors
| yauaa | DataflowTemplates | |
|---|---|---|
| 2 | 4 | |
| 728 | 1,089 | |
| - | 1.6% | |
| 9.7 | 9.8 | |
| 4 days ago | 2 days ago | |
| Java | Java | |
| Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
yauaa
- December 5, 2022: FLiP Stack Weekly
-
How to Parse User Agents with a Snowflake UDF in dbt
UDF code in the main project: https://github.com/nielsbasjes/yauaa/blob/master/udfs/snowflake/src/main/java/nl/basjes/parse/useragent/snowflake/ParseUserAgent.java
DataflowTemplates
-
Which Database to use for rest api
Google provide a Dataflow template for copying from BigQuery to Datastore, see this stack overflow answer.
- Sync Postgres to BigQuery, possible? How?
-
New to GCP - need help designing pipeline from production Heroku Postgres to BigQuery
Ah, looks like the template default appends new rows. If I want to overwrite the table, looks like I might be able to just replace this line in the template code to WRITE_TRUNCATE (see here). Cool!
-
Tricky Dataflow ep.1 : Auto create BigQuery tables in pipelines
However, learning to use Apache Beam, which is the open source framework behind Dataflow, is no bed of roses: The official documentation is sparse, GCP-provided templates don't work out-of-the-box, and the Javadoc is, well, a javadoc.
What are some alternatives?
Apache Flink - Apache Flink
janusgraph - JanusGraph: an open-source, distributed graph database
flink-faker - A data generator source connector for Flink SQL based on data-faker.
pgsink - Logically replicate data out of Postgres into sinks (files, Google BigQuery, etc)
snowflake-kafka-connector - Snowflake Kafka Connector (Sink Connector)
professional-services - Common solutions and tools developed by Google Cloud's Professional Services team. This repository and its contents are not an officially supported Google product.
nussknacker - Low-code tool for automating actions on real time data | Stream processing for the users.
debezium-examples - Examples for running Debezium (Configuration, Docker Compose files etc.)
nifi-extracttext-processor - Apache NiFi Custom Processor Extracting Text From Files with Apache Tika
migrate - Database migrations. CLI and Golang library.
Presto - The official home of the Presto distributed SQL query engine for big data
bigquery-utils - Useful scripts, udfs, views, and other utilities for migration and data warehouse operations in BigQuery.