amazon-neptune-samples VS plume

repo

The following tutorial will help to transform data into Gremlin data load format.

ETL Process for Transforming and Loading data into Amazon Neptune

I get what you're saying which is why in the past two years (part of my MSc) I've been developing https://github.com/plume-oss/plume which exports CPGs with a pluggable graph database backend. The idea is that there is an interface that accepts the DiffGraph objects that build the AST, CFG, etc. and an implementing class would write the graph database specific approach to storing these results.

We have one for TinkerGraph, Neo4j, TigerGraph, and Neptune which compared to OverflowDb, the rest are way too slow. We implemented the ODB driver in the same way to avoid bias.

It is important to note that Plume runs Scala 2.13/3.1.1 (since this is what ODB and the CPG project use) and Neo4j in-mem was on 2.12 (until Nov 21) so I only measure remote Neo4j.

As I've been developing I've approached the ShiftLeft team to assist and we are in the experimenting phase to write up our results with comparing how ODB does on different programs and versus other graph DBs. Currently my first write up is the approach for incremental CPG building and data flow querying and the second being this graph DB comparison.

The TLDR of our multi-graph comparison is that remote DBs take way too long due to network overhead of moving data between one machine to the next. Another issue is that other databases use way too much memory and Neo4j/TinkerGraph are the worst offenders here. Simple programs produce multi-million edge graphs with a few hundred thousand nodes which is only "acceptable" in a couple of seconds if you think of a business use-case like analysing a program within a CI/CD pipeline for example.

On the other hand, I do see the utility in having the other databases supported if time/memory constraints are not an issue e.g. I have one user of Plume using TinkerGraph backend for certain compatibility Tinker has with other tools and I can imagine Neo4j having similar third-party support.

I think OverflowDB is a great database and the query language the team has extended on top of it is really useful and you should definitely give it a shot and/or contribute to it's development further to include other features other than speed/memory efficiency/query capabilities.

If there is enough interest from the ideas that Plume has carried across then I'm sure the team may integrate some Plume's capabilities to keep the multi-graph database notion across while maintaining it for the latest changes.