rel8
prosto
Our great sponsors
rel8  prosto  

2  8  
104  59  
    
7.9  5.5  
7 days ago  6 months ago  
Haskell  Python  
GNU General Public License v3.0 or later  MIT License 
Stars  the number of stars that a project has on GitHub. Growth  month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
rel8
prosto

Excel 2.0 – Is there a better visual data model than a grid of cells?
One idea is to use columns instead of cells. Each column has a definition in terms of other columns which might also be defined in terms of other columns. If you change value(s) in some source column then these changes will propagate through the graph of these column definitions. Some fragments of this general idea were implemented in different systems, for example, Power BI or Airtable.
This approach was formalized in the conceptoriented model of data which relies on two basic elements: mathematical functions and mathematical sets. In contrast, most traditional data models rely on only sets. Functions are implemented as columns. The main difficulty in any formalization is how to deal with columns in multiple tables.
This approach was implemented in the Prosto data processing toolkit: https://github.com/asavinov/prosto

Show HN: Query any kind of data with SQL powered by Python
Having Python expressions within a declarative language is a really good idea because we can combine low level logic of computations of values with high level logic of set processing.
A similar approach is implemented in the Prosto data processing toolkit:
https://github.com/asavinov/prosto
Although Prosto is viewed as an alternative to MapReduce by relying on functions, it also supports Python UserDefined Functions in its ColumnSQL:

NoCode SelfService BI/Data Analytics Tool
Most of the selfservice or nocode BI, ETL, data wrangling tools are am aware of (like airtable, fieldbook, rowshare, Power BI etc.) were thought of as a replacement for Excel: working with tables should be as easily as working with spreadsheets. This problem can be solved when defining columns within one table: ``ColumnA=ColumnB+ColumnC, ColumnD=ColumnAColumnE`` we get a graph of column computations* similar to the graph of cell dependencies in spreadsheets.
Yet, the main problem is in working multiple tables: how can we define a column in one table in terms of columns in other tables? For example: ``Table1::ColumnA=FUNCTION(Table2::ColumnB, Table3::ColumnC)`` Different systems provided different answers to this question but all of them are highly specific and rather limited.
Why it is difficult to define new columns in terms of other columns in other tables? Short answer is that working with columns is not the relational approach. The relational model is working with sets (rows of tables) and not with columns.
One generic approach to working with columns in multiple tables is provided in the conceptoriented model of data which treats mathematical functions as firstclass elements of the model. Previously it was implemented in a data wrangling tool called Data Commander. But them I decided to implement this model in the *Prosto* data processing toolkit which is an alternative to mapreduce and SQL:
https://github.com/asavinov/prosto
It defines data transformations as operations with columns in multiple tables. Since we use mathematical functions, no joins and no groupby operations are needed and this significantly simplifies and makes more natural the task of data transformations.
Moreover, now it provides *ColumnSQL* which makes it even easier to define new columns in terms of other columns:
https://github.com/asavinov/prosto/blob/master/notebooks/col...

Show HN: Hamilton, a Microframework for Creating Dataframes
Hamilton is more similar to the Prosto data processing toolkit which also relies on column operations defined via Python functions:
https://github.com/asavinov/prosto
However, Prosto allows for data processing via column operations in many tables (implemented as pandas data frames) by providing a columnoriented equivalents for joins and groupby (hence it has no joins and no groupbys which are known to be quite difficult and require high expertise).
Prosto also provides ColumnSQL which might be simpler and more natural in many use cases.
The whole approach is based on the conceptoriented model of data which makes functions firstclass elements of the model as opposed to having only sets in the relational model.

Against SQL
One alternative to SQL (type of thinking) is ColumnSQL [1] which is based on a new data model. This model is relies on two equal constructs: sets (tables) and functions (columns). It is opposed to the relational algebra which is based on only sets and set operations. One benefit of ColumnSQL is that it does not use joins and groupby for connectivity and aggregation, respectively, which are known to be quite difficult to understand and error prone in use. Instead, many typical data processing patterns are implemented by defining new columns: link columns instead of join, and aggregate columns instead of groupby.
More details about "Why functions and columnorientation" (as opposed to sets) can be found in [2]. Shortly, problems with setorientation and SQL are because producing sets is not what we frequently need  we need new columns and not new table. And hence applying set operations is a kind of workaround due the absence of column operations.
This approach is implemented in the Prosto data processing toolkit [0] and ColumnSQL[1] is a syntactic way to define its operations.
[0] https://github.com/asavinov/prosto Prosto is a data processing toolkit  an alternative to mapreduce and joingroupby
[1] https://prosto.readthedocs.io/en/latest/text/columnsql.html ColumnSQL (work in progress)
[2] https://prosto.readthedocs.io/en/latest/text/why.html Why functions and columnorientation?
 Functions matter – an alternative to SQL and mapreduce for data processing

NoSQL Data Modeling Techniques
> This is closer to the way that humans perceive the world — mapping between whatever aspect of external reality you are interested in and the data model is an order of magnitude easier than with relational databases.
One approach to modeling data based on mappings (mathematical functions) is the conceptoriented model [1] implemented in [2]. Its main feature is that it gets rid of joins, groupby and mapreduce by manipulating data using operations with functions (mappings).
> Everything is prejoined — you don’t have to disassemble objects into normalised tables and reassemble them with joins.
One old related general idea is to assume the existence of universal relation. Such an approach is referred to as the universal relation model (URM) [3, 4].
[1] A. Savinov, Conceptoriented model: Modeling and processing data using functions, Eprint: arXiv:1911.07225 [cs.DB], 2019 https://www.researchgate.net/publication/337336089_Concepto...
[2] https://github.com/asavinov/prosto Prosto Data Processing Toolkit: No joingroupby, No mapreduce
[3] https://en.wikipedia.org/wiki/Universal_relation_assumption
[4] R. Fagin, A.O. Mendelzon and J.D. Ullman, A Simplified Universal Relation Assumption and Its Properties. ACM Trans. Database Syst., 7(3), 343360 (1982).

Feature Processing in Go
(Currently, it is not actively developed and the focus is moved to a similar project  https://github.com/asavinov/prosto  also focused on data preprocessing and feature engineering)
What are some alternatives?
opaleye
gofeatureprocessing  🔥 Fast, simple sklearnlike feature processing for Go
fquery  A graph query engine
Preql  An interpreted relational query language that compiles to SQL.
capepython  Collaborate on privacypreserving policy for data science projects in Pandas and Apache Spark
Optimus  :truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, DaskcuDF, Vaex and PySpark
hamilton  A scalable general purpose microframework for defining dataflows. You can use it to create dataframes, numpy matrices, python objects, ML models, etc.
PostgreSQL  Mirror of the official PostgreSQL GIT repository. Note that this is just a *mirror*  we don't work with pull requests on github. To contribute, please see https://wiki.postgresql.org/wiki/Submitting_a_Patch
jrutil