scipipe VS formkiq-core

https://scipipe.org - A pipeline tool for shell commands by a declarative flow-based API in Go

Github link: https://github.com/scipipe/scipipe

There are many pipeline tools for shell commands, but a majority has one or more limitations in their API which makes certain complex pipelines impossible or really hard to write.

We were pushing the limits of all the tools we tried, so developed our own, and implemented it in Go, with a declarative API for defining the data flow dependencies, instead of inventing yet another DSL. This has allowed us great flexibility in developing also complex pipelines, e.g. combining parameter sweeps nested with cross-validation implemented as workflow constructs.

SciPipe is also unique in providing an audit report for every single output of the workflow, in a structured JSON format. A helper tool allows converting these reports to either an HTML report, a PDF, or a Bash script that will generate the one accompanying output file from scratch.

An extra cool things is that, because the audit reports live alongside output files, if you run a scipipe workflow that uses files generated by another scipipe workflow, it will pick up also all the history for the input files generated by this earlier workflow, meaning that you get a 100% complete audit report, even if your analysis spans multiple workflows!

We may want to get in touch with each other. We have an Open Core document management platform that runs in AWS; I'm not sure about your roadmap, but there may be something there that's of use: https://github.com/formkiq/formkiq-core

We are in the weird position of building something (a document management platform) that can be for SaaS bootstrappers (especially our free version), but our main marketing and sales efforts are for larger orgs, whether for SaaS or internal use.

We have a free offering, FormKiQ Core (https://github.com/formkiq/formkiq-core) that will also include this AI classification, so it's possible to use that as a foundation for creating a custom ECM system.

I think that makes sense to me (biased as I am). I wonder if Milvus (mentioned in another comment) can handle some of this, or if a dedicated EDMS is required. We have created an Open Core EDMS that could provide the document management functionality running using AWS: https://github.com/formkiq/formkiq-core

Our product is FormKiQ, and while our free version, FormKiQ Core, handles all of the standard functionality, we want to get our enterprise modules out there for startups that could use them.

If you are okay with hosting in your own cloud, you can try FormKiQ Core: https://github.com/formkiq/formkiq-core

I'm definitely curious as to what software is available in this space. Our company, FormKiQ, is wading into this AI processing of emails into matters right now, and I don't know that I've personally seen anything with the flexibility you need on the market. We may be too robust for what you need, but I'm always looking out to see if there is a simpler solution we can provide if enough people have the same problem.

If hosted in-house can be swapped out by an AWS account your organization owns and controls, you can try FormKiQ Core: https://github.com/formkiq/formkiq-core. It covers all of the expected standard functionality of a document management system, with an emphasis on flexibility and integration.

But if you decide to create such a system, you may want to evaluate if our Open Core document management platform might save you time on your MVP: https://github.com/formkiq/formkiq-core