opaleye
prosto
opaleye | prosto | |
---|---|---|
9 | 9 | |
602 | 90 | |
- | - | |
8.3 | 3.6 | |
6 days ago | almost 3 years ago | |
Haskell | Python | |
BSD 3-clause "New" or "Revised" License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
opaleye
-
What's your favorite Database EDSL/library in Haskell?
If you ever have any questions about Opaleye I'm happy to help. Feel free to open an issue to ask about anything any time.
-
Persistent vs. beam for production database
Sounds like Opaleye isn't on your list of choices, but if it is then feel free to ask me any questions, any time by filing an issue (I'm the Opaleye maintainer).
-
How to build a large-scale haskell backend for a photo sharing app (some questions)
Opaleye is Posgres-only, and Postgres does such a good job of optimizing queries that performance issues basically don't arise. I have a long-standing invitation to improve Opaleye's query generation as soon as anyone can produce a repeatable example of a poorly-performing query. In Opaleye's eight years, no one ever has. There's a thread where two reports have come close, but it's still not clear that that's simply due to using a six year old version of Postgres.
- What are things that the Haskell scene lacks the most?
-
Out of memory when building product-profunctors
Nice! Well done. If you have any more questions about product-profunctors or Opaleye then please let me know. It's best to ask by [opening an issue](https://github.com/tomjaguarpaw/haskell-opaleye/issues/new).
- Embedded Pattern Matching
- How to simply do opaleye field type conversion
-
Against SQL
The only way out that I can see is to design embedded domain specific languages (EDSLs) that inherit the expressiveness, composability and type safety from the host language. That's what Opaleye and Rel8 (Postgres EDSLs for Haskell do. Haskell is particularly good for this. The query language can be just a monad and therefore users can carry all of their knowledge of monadic programming to writing database queries.
This approach doesn't resolve all of the author's complaints but it does solve many.
Disclaimer: I'm the author of Opaleye. Rel8 is built on Opaleye. Other relational query EDSLs are available.
[1] https://github.com/tomjaguarpaw/haskell-opaleye/
-
Combining Deep and Shallow Embedding of Domain-Specific Languages
For an example of how this plays out in practice observe Opaleye's MaybeFields (generously contributed by Shane and /u/ocharles at Circuithub). The definition is essentially identical to Optional from the paper. Instead of a specialised typeclass Inhabited we use the ProductProfunctor NullSpec (which happens to conjure up an SQL NULL, but it could be any other witness).
prosto
-
Show HN: PRQL 0.2 – Releasing a better SQL
> Joins are what makes relational modeling interesting!
It is the central part of RM which is difficult to model using other methods and which requires high expertise in non-trivial use cases. One alternative to how multiple tables can be analyzed without joins is proposed in the concept-oriented model [1] which relies on two equal modeling constructs: sets (like RM) and functions. In particular, it is implemented in the Prosto data processing toolkit [2] and its Column-SQL language. The idea is that links between tables are used instead of joins. A link is formally a function from one set to another set.
[1] Joins vs. Links or Relational Join Considered Harmful https://www.researchgate.net/publication/301764816_Joins_vs_...
[2] https://github.com/asavinov/prosto data processing toolkit radically changing how data is processed by heavily relying on functions and operations with functions - an alternative to map-reduce and join-groupby
-
Excel 2.0 – Is there a better visual data model than a grid of cells?
One idea is to use columns instead of cells. Each column has a definition in terms of other columns which might also be defined in terms of other columns. If you change value(s) in some source column then these changes will propagate through the graph of these column definitions. Some fragments of this general idea were implemented in different systems, for example, Power BI or Airtable.
This approach was formalized in the concept-oriented model of data which relies on two basic elements: mathematical functions and mathematical sets. In contrast, most traditional data models rely on only sets. Functions are implemented as columns. The main difficulty in any formalization is how to deal with columns in multiple tables.
This approach was implemented in the Prosto data processing toolkit: https://github.com/asavinov/prosto
-
Show HN: Query any kind of data with SQL powered by Python
Having Python expressions within a declarative language is a really good idea because we can combine low level logic of computations of values with high level logic of set processing.
A similar approach is implemented in the Prosto data processing toolkit:
https://github.com/asavinov/prosto
Although Prosto is viewed as an alternative to Map-Reduce by relying on functions, it also supports Python User-Defined Functions in its Column-SQL:
- No-Code Self-Service BI/Data Analytics Tool
-
Show HN: Hamilton, a Microframework for Creating Dataframes
Hamilton is more similar to the Prosto data processing toolkit which also relies on column operations defined via Python functions:
https://github.com/asavinov/prosto
However, Prosto allows for data processing via column operations in many tables (implemented as pandas data frames) by providing a column-oriented equivalents for joins and groupby (hence it has no joins and no groupbys which are known to be quite difficult and require high expertise).
Prosto also provides Column-SQL which might be simpler and more natural in many use cases.
The whole approach is based on the concept-oriented model of data which makes functions first-class elements of the model as opposed to having only sets in the relational model.
-
Against SQL
One alternative to SQL (type of thinking) is Column-SQL [1] which is based on a new data model. This model is relies on two equal constructs: sets (tables) and functions (columns). It is opposed to the relational algebra which is based on only sets and set operations. One benefit of Column-SQL is that it does not use joins and group-by for connectivity and aggregation, respectively, which are known to be quite difficult to understand and error prone in use. Instead, many typical data processing patterns are implemented by defining new columns: link columns instead of join, and aggregate columns instead of group-by.
More details about "Why functions and column-orientation" (as opposed to sets) can be found in [2]. Shortly, problems with set-orientation and SQL are because producing sets is not what we frequently need - we need new columns and not new table. And hence applying set operations is a kind of workaround due the absence of column operations.
This approach is implemented in the Prosto data processing toolkit [0] and Column-SQL[1] is a syntactic way to define its operations.
[0] https://github.com/asavinov/prosto Prosto is a data processing toolkit - an alternative to map-reduce and join-groupby
[1] https://prosto.readthedocs.io/en/latest/text/column-sql.html Column-SQL (work in progress)
[2] https://prosto.readthedocs.io/en/latest/text/why.html Why functions and column-orientation?
- Functions matter – an alternative to SQL and map-reduce for data processing
-
NoSQL Data Modeling Techniques
> This is closer to the way that humans perceive the world — mapping between whatever aspect of external reality you are interested in and the data model is an order of magnitude easier than with relational databases.
One approach to modeling data based on mappings (mathematical functions) is the concept-oriented model [1] implemented in [2]. Its main feature is that it gets rid of joins, groupby and map-reduce by manipulating data using operations with functions (mappings).
> Everything is pre-joined — you don’t have to disassemble objects into normalised tables and reassemble them with joins.
One old related general idea is to assume the existence of universal relation. Such an approach is referred to as the universal relation model (URM) [3, 4].
[1] A. Savinov, Concept-oriented model: Modeling and processing data using functions, Eprint: arXiv:1911.07225 [cs.DB], 2019 https://www.researchgate.net/publication/337336089_Concept-o...
[2] https://github.com/asavinov/prosto Prosto Data Processing Toolkit: No join-groupby, No map-reduce
[3] https://en.wikipedia.org/wiki/Universal_relation_assumption
[4] R. Fagin, A.O. Mendelzon and J.D. Ullman, A Simplified Universal Relation Assumption and Its Properties. ACM Trans. Database Syst., 7(3), 343-360 (1982).
-
Feature Processing in Go
(Currently, it is not actively developed and the focus is moved to a similar project - https://github.com/asavinov/prosto - also focused on data preprocessing and feature engineering)
What are some alternatives?
esqueleto - Bare bones, type-safe EDSL for SQL queries on persistent backends.
Preql - An interpreted relational query language that compiles to SQL.
mywatch
Optimus - :truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
HDBC - Haskell Database Connectivity
mito - The mitosheet package, trymito.io, and other public Mito code.
database-migrate - database-migrate haskell library to assist with migration for *-simple sql backends.
rel8 - Hey! Hey! Can u rel8?
HongoDB - A Simple Key Value Store
spyql - Query data on the command line with SQL-like SELECTs powered by Python expressions
squeal-postgresql - Squeal, a deep embedding of SQL in Haskell
hamilton - A scalable general purpose micro-framework for defining dataflows. THIS REPOSITORY HAS BEEN MOVED TO www.github.com/dagworks-inc/hamilton