condenser VS profiles

[1]: https://github.com/TonicAI/condenser

[1]: https://github.com/TonicAI/condenser

This may be overkill but we have used Tonic for this: https://www.tonic.ai/

A tool like https://www.tonic.ai/ might help.

I attended a presentation a month or so ago where a co-worker was advocating for tonic. I've never used it myself, but, definitely looks as though it would cover your bases. I do agree with onomazein though that this should really be handled on the infrastructure side if at all possible. Dev's, testers, anyone else should be able to pull from an already anonymized location, but, but, someone else should be responsible for setting up the environment and the initial synchronization.

So I built a gitlab pipeline to create a backup of this upstream db without downtime using various SQL utils. This archive is then staged into an image so the data will unpack and load on startup. I then used a data subsetter called condenser to create datasets for certain use cases. Now devs can load reliable dev data quicker, test against data that QA uses but within their unique envs (local and preview) and create datasets for their own use cases.

Now, the reason for the asterisk in the last section is that it is obviously a little more complicated than simply hardcoding the news sources into the very software itself. OSINTer works by collecting information and then store it in a database, until it's needed by the CTI researcher, which means that when scraping websites we want to be able to filter out unnecessary information and clutter like ads, layout specific sections and other parts of the website which isn't directly related to the news story. To do this, OSINTer makes use of a Domain Specific Language or DSL created by me. All of that sounds rather fancy, but what it all translates to is that OSINTer takes in a series of files in a simple and structured JSON format, which describes which websites to scrape, and which parts of these websites to keep. This also means that it is not only very fast to add new news-sources (approx 5 mins) but also that if OSINTer where to be used for trend research in a completely different area, it would be possible to switch out these JSON files and have OSINTer collect some completely different data. Within the context of OSINTer, are this DSL called profiles and can be found at https://gitlab.com/osinter/profiles