Top 23 Data Open-Source Projects

• sheetjs

  0 24,178 7.1 JavaScript

  :green_book: SheetJS Community Edition -- Spreadsheet Data Toolkit

• metabase

  0 23,468 9.6 Clojure

  The simplest, fastest way to get business intelligence and analytics to everyone in your company :yum:

• react-query

  4 16,815 9.5 TypeScript

  ⚛️ Hooks for fetching, caching and updating asynchronous data in React

  Latest mention: What does this error mean and how do I resolve it? | reddit.com/r/typescript | 2021-01-14

  The specific error in the useMutation() source code seems to be that they're doing Function Overloading in TS, and they're misidentifying MutationKey as your intended argument but you're trying to provide a function that returns a promise, a MutationFunction, so that specific error message is (arguably) a mistake by TS. Perhaps the MutationFunction generics could be set, to help TS figure out your types.

• data

  6 14,407 7.1 Jupyter Notebook

  Data and code behind the articles and graphics at FiveThirtyEight

  Latest mention: Sometimes you really do get what you deserve | reddit.com/r/instantkarma | 2021-01-12

  First let's take a look at criminals indicted in each administration. Data here.

• awesome-bigdata

  0 9,615 5.2

  A curated list of awesome big data frameworks, ressources and other awesomeness.

• semantic

  1 7,889 9.5 Haskell

  Parsing, analyzing, and comparing source code across many languages

  Latest mention: New Haskell Foundation to Foster Haskell Adoption, Raises 200k USD | news.ycombinator.com | 2020-12-27

  PostgREST is written in Haskell and quite popular: https://postgrest.org/en/v7.0.0/

  Shellcheck has become a very important tool to many people writing shell scripts: https://github.com/koalaman/shellcheck

  GitHub's semantic analysis is written in Haskell: https://github.com/github/semantic

  I'm not sure why Pandoc doesn't count.

  I think it is definitely the case that most examples of successful Haskell are technical programs of interest to hackers. I don't think that's much of a problem myself.

• OpenRefine

  0 7,794 9.5 Java

  OpenRefine is a free, open source power tool for working with messy data and improving it

• countly-server

  0 4,660 9.8 JavaScript

  Countly helps you get insights from your application. Available self-hosted or on private cloud.

• tabulator

  0 3,754 9.1 JavaScript

  Interactive Tables and Data Grids for JavaScript

• Bogus

  0 3,634 7.4 C#

  :card_index: A simple and sane fake data generator for C#, F#, and VB.NET. Based on and ported from the famed faker.js.

• react-refetch

  0 3,386 0.3 JavaScript

  A simple, declarative, and composable way to fetch data for React components

• mimesis

  0 3,192 8.2 Python

  Mimesis is a high-performance fake data generator for Python, which provides data for a variety of purposes in a variety of languages.

• ckan

  0 2,850 9.4 Python

  CKAN is an open-source DMS (data management system) for powering data hubs and data portals. CKAN makes it easy to publish, share and use data. It powers catalog.data.gov, europeandataportal.eu/data, data.humdata.org among many other sites.

• akshare

  0 2,813 9.6 Python

  AkShare is an elegant and simple financial data interface library for Python, built for human beings! 开源财经数据接口库

• awesome-json-datasets

  0 2,186 0.7 JavaScript

  A curated list of awesome JSON datasets that don't require authentication.

• scio

  2 2,047 9.4 Scala

  A Scala API for Apache Beam and Google Cloud Dataflow.

  Latest mention: ELT, Data Pipeline | dev.to | 2021-01-01

  To counter the above mentioned problem, we decided to move our data to a Pub/Sub based stream model, where we would continue to push data as it arrives. As fluentd is the primary tool being used in all our servers to gather data, rather than replacing it we leveraged its plugin architecture to use a plugin to stream data into a sink of our choosing. Initially our inclination was towards Google PubSub and Google Dataflow as our Data Scientists/Engineers use Big Query extensively and keeping the data in the same Cloud made sense. The inspiration of using these tools came from Spotify’s Event Delivery – The Road to the Cloud. We did the setup on one of our staging server with Google PubSub and Dataflow. Both didn't really work out for us as PubSub model requires a Subscriber to be available for the Topic a Publisher streams messages to, otherwise the messages are not stored. On top of it there was no way to see which messages are arriving. During this the weirdest thing that we encountered was that the Topic would be orphaned losing the subscribers when working with Dataflow. PubSub we might have managed to live with, the wall in our path was Dataflow. We started off with using SCIO from Spotify to work with Dataflow, there is a considerate lack of documentation over it and found the community to be very reserved on Github, something quite evident in the world of Scala for which they came up with a Code of Conduct for its user base to follow. Something that was required from Dataflow for us was to support batch write option to GCS, after trying our hand at Dataflow to no success to achieve that, Google's staff at StackOverflow were quite responsive and their response confirmed that it was something not available with Dataflow and streaming data to BigQuery, Datastore or Bigtable as a datastore was an option to use. The reason we didn't do that was to avoid high streaming cost to these services to store data, as majority of our jobs from the data team are based on batched hourly data. The initial proposal to the updated pipeline is shown below.

• stats

  0 1,884 3.4 Go

  A well tested and comprehensive Golang statistics library package with no dependencies.

• lens

  0 1,716 6.8 Haskell

  Lenses, Folds, and Traversals - Join us on freenode #haskell-lens

• kea

  0 1,690 8.1 JavaScript

  Production Ready State Management for React

• kiba

  1 1,517 4.8 Ruby

  Data processing & ETL framework for Ruby

  Latest mention: Ruby 3.0.0 Released | news.ycombinator.com | 2020-12-24

  in ruby you have the async gems from https://github.com/socketry/async

  i dont know for data manipulation but i guess also ruby has it. maybe this https://github.com/thbar/kiba

• gofakeit

  1 1,388 7.8 Go

  Random fake data generator written in go

  Latest mention: Gofakeit v6. Now supports concurrency and crypto/rand | news.ycombinator.com | 2021-01-18

  The new v6 release now supports the ability to have localized rand that allows for concurrent generating of random data. Crypto/rand has now been integrated as well as if you have a custom rand that fulfills the math/rand source64 interface you can use it with gofakeit. Let me know your thoughts.

  https://github.com/brianvoe/gofakeit

• cubes

  0 1,379 0.0 Python

  Light-weight Python OLAP framework for multi-dimensional data analysis

• CodeSearchNet

  1 1,346 7.0 Jupyter Notebook

  Datasets, tools, and benchmarks for representation learning of code.

  Latest mention: Speedtyper.dev: Type racing for programmers | reddit.com/r/webdev | 2021-01-16

  https://github.com/github/CodeSearchNet#downloading-data-from-s3

Index