CKAN VS kuwala

We use Kubernetes as our deployment platform. Any feedback on one of these open source data catalogs ? - https://atlas.apache.org/#/ - https://opendatadiscovery.org/ - https://open-metadata.org/ - https://marquezproject.github.io/marquez/ - https://datahubproject.io/ - https://www.amundsen.io/ - https://ckan.org/ - https://magda.io/

Ckan

We publish all our data on the [Data Portal](https://data.nhm.ac.uk), a Museum project that's been running since 2014. Instead of MediaWiki it runs on an open-source Python framework called [CKAN](https://ckan.org), which is designed for hosting datasets - though we've had to adapt it in various ways so that it can handle such large amounts of data.

What are the 5 hottest dbt Repositories one should star on Github 2022?

dbt is a software framework that sits in the middle of the ELT process. It represents the transformative layer after loading data from an original source. Dbt combines SQL with software engineering principles.

Here are my top5!

- Lightdash (https://github.com/lightdash/lightdash): Lightdash converts dbt models and makes it possible to define and easily visualize additional metrics via a visual interface.

- ⏎ re_data (https://github.com/re-data/re-data): Re-Data is an abstraction layer that helps users monitor dbt projects and their underlying data. For example, you get alerts when a test failed or a data anomaly occurs in a dbt project.

- evidence (https://github.com/evidence-dev/evidence): Evidence is another tool for lightweight BI reporting. With Evidence, you can build simple reports in "medium style" using SQL queries and Markdown.

- Kuwala (https://github.com/kuwala-io/kuwala): With Kuwala, a BI analyst can intuitively build advanced data workflows using a drag-drop interface on top of the modern data stack without coding. Behind the Scenes, the dbt models are generated so that a more experienced engineer can customize the pipelines at any time.

- fal ai (https://github.com/fal-ai/fal): Fal helps to run Python scripts directly from the dbt project. For example, you can load dbt models directly into the Python context which helps to apply Data Science libraries like SKlearn and Prophet in the dbt models.

Kuwala ( https://github.com/kuwala-io/kuwala ) Kuwala is a data workspace that consolidates the Modern Data Stack and makes it usable for BI analysts and Engineers. Even though dbt is originally targeted at BI Analysts, dbt is mainly used by Engineers. This shifts a large amount of pipeline engineering effort to the IT department. With Kuwala, a BI analyst can intuitively build advanced data workflows using a drag-drop interface on top of the modern data stack without coding. Consequently, the BI Analyst can work more iteratively and maintain the complete workflow from source to metrics in a dashboard. Under the hood and Behind the Scenes, the dbt models are generated so that a more experienced engineer can customize the pipelines at any time. In addition, engineers can easily convert dbt models into reusable “drag and drop” components.

- 🧱 Kuwala: With Kuwala, a BI analyst can intuitively build advanced data workflows using a drag-drop interface on top of the modern data stack without coding. Behind the Scenes, the dbt models are generated so that a more experienced engineer can customize the pipelines at any time.

In 2020 data teams spent 79% of their time on data gathering and cleaning. The enormous amount of data and missing data talents lead to a huge failure rate of 85% in data science projects. Kuwala wants to provide a tool that combines the best open source projects in an UI that is easy to handle by analysts and flexible enough for engineers to adjust the tool to edge cases and internal needs. We believe in Open Source and so we are! Everything we do is published on our Github Repo: https://github.com/kuwala-io/kuwala or https://kuwala.io. And now we search you to help us to built out our hosted version of Kuwala.

We're currently using it as a PoC to combine POI with population data. You can find it on our GitHub.

Yes, exactly. Looking at a graph schema is very intuitive. We're currently using Neo4j to model different POI sources (OSM, Google, etc.) with population and other sources. So trying to model real-world entities. E.g., you can express "this shop is in that shopping mall in this district where 3214 people between 14-25 are living." It's open-source on GitHub. Would be happy to get your feedback. :) https://github.com/kuwala-io/kuwala

Also you can easily run it locally for any country you like. It's just a few commands: https://github.com/kuwala-io/kuwala/tree/master/kuwala-pipelines/population-density

Here's the link to the README issue: https://github.com/kuwala-io/kuwala/issues/20

👾 https://github.com/kuwala-io/kuwala/tree/master/kuwala-pipelines/population-density 📖 https://medium.com/kuwala-io/querying-the-most-granular-demographics-dataset-62da16b441a8 💬 https://join.slack.com/t/kuwala-community/shared_invite/zt-l5b2yjfp-pXKFBjbnl7_P3nXtwca5ag