lightdash VS echarts

> YAML, pivoting being done in the frontend, no symmetric aggregates

(one of the maintainers of Lightdash) You touched on some of our most interesting problems here! Would be especially interested to hear about what you liked / didn't like about symmetric aggregates in Looker and how you find dev with YAML. If you have an idea of how you'd like these to look in Lightdash, the team would be really open to making that a reality.

For pivoting in the backend, this is coming! Issue here: https://github.com/lightdash/lightdash/issues/2907

What are the 5 hottest dbt Repositories one should star on Github 2022?

dbt is a software framework that sits in the middle of the ELT process. It represents the transformative layer after loading data from an original source. Dbt combines SQL with software engineering principles.

Here are my top5!

- Lightdash (https://github.com/lightdash/lightdash): Lightdash converts dbt models and makes it possible to define and easily visualize additional metrics via a visual interface.

- ⏎ re_data (https://github.com/re-data/re-data): Re-Data is an abstraction layer that helps users monitor dbt projects and their underlying data. For example, you get alerts when a test failed or a data anomaly occurs in a dbt project.

- evidence (https://github.com/evidence-dev/evidence): Evidence is another tool for lightweight BI reporting. With Evidence, you can build simple reports in "medium style" using SQL queries and Markdown.

- Kuwala (https://github.com/kuwala-io/kuwala): With Kuwala, a BI analyst can intuitively build advanced data workflows using a drag-drop interface on top of the modern data stack without coding. Behind the Scenes, the dbt models are generated so that a more experienced engineer can customize the pipelines at any time.

- fal ai (https://github.com/fal-ai/fal): Fal helps to run Python scripts directly from the dbt project. For example, you can load dbt models directly into the Python context which helps to apply Data Science libraries like SKlearn and Prophet in the dbt models.

Lightdash ( https://github.com/lightdash/lightdash ) Lightdash converts dbt models and makes it possible to define and easily visualize additional metrics via a visual interface. The front end helps to understand and extend the underlying SQL queries. Lightdash also visualizes business metrics and makes them shareable with the data team. It is also possible to integrate all data into another visualization tool.

- ⚡️ Lightdash: Lightdash converts dbt models and makes it possible to define and easily visualize additional metrics via a visual interface.

Visualization / Analysis: Lightdash, Superset

Ingestion / ETL: Airbyte, Singer, Jitsu Transformation: dbt Orchestration: Airflow, Dagster Testing: GreatExpectations Observability: Monosi Reverse ETL: Grouparoo, Castled Visualization: Lightdash, Superset

1) An integration with Metabase Cloud is on our roadmap for Q1! We'd love to integrate with Lightdash, but they don't have a public API just yet[1].

2) Several of our customers use us to alert on schema changes in Postgres, specifically so they can get ahead of application database changes that will end up in the warehouse, so you're definitely not alone! Here's a link on how to connect postgres: https://docs.metaplane.dev/docs/postgres

That's an excellent stack and one we kept front and center when building out Metaplane, so definitely let us know if you have any feedback or suggestions here!

[1]: https://github.com/lightdash/lightdash/issues/632

I would recommend lightdash which is essentially an open source Looker clone https://github.com/lightdash/lightdash

import puppeteer from "puppeteer"; import fs from "node:fs"; import { storiesTpl, storiesArgs, generOptions, generOptionsWithFn, } from "./template.mjs"; const ECHARTS_BASE_URL = "https://echarts.apache.org/examples/en/index.html"; function capitalizeFirstLetter(str) { if (!str || str.length === 0) { return ""; } str = str.toLowerCase(); return str.charAt(0).toUpperCase() + str.slice(1); } (async function () { const browser = await puppeteer.launch(); const page = await browser.newPage(); // Navigate the page to a URL await page.goto(ECHARTS_BASE_URL); // Set screen size await page.setViewport({ width: 1080, height: 1024 }); // Type into search box // const examples = await page.type([".example-list-panel"]); const searchResultSelector = ".example-list-panel > div"; const results = await page.$$(searchResultSelector); for (const element of results) { // gener namespace const ele = await element.$(".chart-type-head"); const title = await ele.evaluate((el) => el.textContent); let namespace = title.split(" ").filter(Boolean); namespace = namespace.slice(0, namespace.length - 1); namespace = namespace .map((item) => item.replace("\n", "").replace("/", "")) .filter(Boolean) .join(""); console.log(`${namespace} start`); const instances = await element.$$(".row .example-list-item"); const components = []; for (const instance of instances) { // title const titleElement = await instance.$(".example-title"); const subTitle = await titleElement.evaluate((el) => el.textContent); const titles = subTitle .split(" ") .filter(Boolean) .map((item) => item .replace(/\+/g, "") .replace(/\(/g, "") .replace(/\)/g, "") .replace(/-/g, "") ); const title = titles.map((item) => capitalizeFirstLetter(item)).join(""); const link = await instance.$(".example-link"); const newPagePromise = new Promise((resolve) => { browser.on("targetcreated", async (target) => { if (target.type() === "page") { const targetPage = await target.page(); const url = await targetPage.url(); if (url.includes("editor")) { resolve(targetPage); } } }); }); await link.click(); const newPage = await newPagePromise; await newPage.setViewport({ width: 40000, height: 20000 }); await newPage.waitForSelector(".ace_text-layer"); await new Promise((resolve) => { setTimeout(() => { resolve(); }, 3000); }); let content = await newPage.evaluate( () => document.querySelector(".ace_text-layer").innerText ); content = content .replace(/\[\]/g, "[] as any") .replace(//g, "") .replace(/var/g, "let"); let options; if (content.includes("myChart")) { options = generOptionsWithFn({ options: content }); } else { options = generOptions({ options: content }); } components.push({ options, title }); await newPage.close(); } const args = components .filter(({ options }) => { if (options.includes("$")) return false; return true; }) .map(({ options, title }) => storiesArgs({ options: options, name: title }) ) .join("\r\n"); const scripts = storiesTpl({ namespace: `Charts/${namespace}`, components: args, }); fs.writeFileSync(`./bots/assests/${namespace}.stories.ts`, scripts); console.log(`${namespace} end`); } })();

I want to know where my money goes. I like to look at stacked-area (or column) charts of the categories of spending. To make this work I have some software I made ~20 years ago that does double-entry book-keeping. At the end of the month, I import statements from financial service providers (eg: Wells Fargo, Chase, PayPal, Stripe, etc). Lots of stuff is repeat purchases (eg: Shell Gas) and my software automatically categorises. Some transactions I have to categorise manually. Each category / vendor becomes an expense-account and my banks and CCs exist as assets and liabilities.

Once the import and reconciliation is done I pull up a my column chart that shows where the money went -- and can compare over time -- see a full year of movement. I've been through various charting libraries with it and most recently moved to ECharts[0] -- so I'm planning to expand with Treemap and Sankey style visuals.

The import process, which I do monthly takes maybe an hour. I'm importing from like 5 bank accounts, 3 payment processors, 4 CC providers. The part that takes the longest is signing into their slow sites, navigating past pop-up/interstitial, getting to their download page and waiting for it to download. Loads of these sites (WF, Chase) have been "modernised" and have some real bullshit UI/UX going on -- lags, no keyboard, elements jump around, forms can't remember state, ctrl+click won't open in a new page cause that damned link isn't actually a link but some nested monster of DIVs with 19 event listeners on each one -- and somehow still all wrong.

I think the most-best feature would be to have some tool automatically get all my transactions from all these providers into one common format. Gimmee some JSON with like 10 commonly-named fields for the normal stuff and then 52 other BS fields that each provider likes to add (see a PayPal CSV for example). Does that exist and I just don't know?

[0] https://echarts.apache.org/

Coded mainly in Perl and Gnuplot, recently extended by Python Pandas and JavaScript Tabulator and ECharts

Dashboards have a lot of charts for different forms and data. This is another common requirement. This template recommends Apache ECharts, a powerful, easy-to-use, and flexible JavaScript visualization library.

We have developed an open source graphics library for react native APP, which is based on Apache ECharts and uses RNSVG or RNSkia for rendering in a way that is almost identical to using it in the web, and can satisfy most graphics situations. The project source code is available at https://github.com/wuba/wrn-echarts .

apache-echarts for charting system (it has 3d chart anyway)

How are you importing the definition? Assuming you are using "apache/echarts" and not some other lib named "echarts", you should be able to import DatasetModel directly and let tree shaking trim out what you're not using.

ECharts -- open source js lib for enterprise-grade charts