janusgraph VS DataflowTemplates

First, you need to choose a specific graph database platform to work with, such as Neo4j, OrientDB, JanusGraph, Arangodb or Amazon Neptune. Once you have selected a platform, you can then start working with graph data using the platform's query language.

JanusGraph

There were a couple years where it was mostly abandoned. Good to see this solid graph database being well maintained. the milestones[1] mostly show a lot of upgrading libraries, some enhancements/features sprinkled in, but for a while Janus was nearly abandoned.

Maintenance re-started in 2017, with IBM & Google stepping up to back it[2].

[1] https://github.com/JanusGraph/janusgraph/milestones?state=cl...

[2] https://architecht.io/google-ibm-back-new-open-source-graph-...

QOMPLX partnered with the graph database experts at Expero to implement their system with JanusGraph, which uses Scylla as an underlying fast and scalable storage layer. We had the privilege to learn from their use case at Scylla Summit this January, which we share with you today.

At the time of writing this article, JanusGraph's latest release page on GitHub says that JanusGraph is compatible with Cassandra 3.11.0. So I will get this version's download link from the Apache Cassandra archive, and I will use the wget shell command to download it to my /opt directory. Then I will use the tar command to extract the contents of the downloaded archive.

Google provide a Dataflow template for copying from BigQuery to Datastore, see this stack overflow answer.

Ah, looks like the template default appends new rows. If I want to overwrite the table, looks like I might be able to just replace this line in the template code to WRITE_TRUNCATE (see here). Cool!

However, learning to use Apache Beam, which is the open source framework behind Dataflow, is no bed of roses: The official documentation is sparse, GCP-provided templates don't work out-of-the-box, and the Javadoc is, well, a javadoc.