ferocity VS jOOQ

The advantage of the static import DSL is that it is more composable: anyone else can define operators that interoperate with my operators without the complexity of the scheme linked above not to mention more generality. This goes not just for the low level operators like "filter" that you might want to supplement but the many more operators you can write that are implemented out of mine.

I prototyped an API that works in the same direction as streams by having something like

https://paulhoule.github.io/pidove/apidocs/com/ontology2/pid...

in that it wraps all Iterables returned by my methods and has not just the teardown facility but also all the operators attached as instance methods which lets you write the chaining style you ask for that I know is in demand.

If I was going to go any further on pidove it would have involved more use of code generation and this system

https://github.com/paulhoule/ferocity

which was supposed to be a code generator for writing code generators, and it could code generate stubs that would let you write expression trees in Java as S-expressions and build them up into methods and either compile the code to real Java source code or execute the methods by evaluating the expression tree in place.

Like common LISP you can write syntactic macros in that that metalanguage because an Expression> can be evaluated at compile time, one of quite a few concepts like "quoting" that I encountered in that spike.

The idea was ferocity would get to the point where it synthesizes a more complete and perfect pidove.

but you can't have an overload that takes two differently parameterized expressions, this lets you write Java code in a lisp-like syntax that can be metaprogrammed on:

https://github.com/paulhoule/ferocity/blob/main/ferocity-std...

that project got me thinking a lot about the various ways types manifest in Java and I made the discovery that when you add Expression types you practically discover an extended type systems where there are many things that are implied by the existence of ordinary Java types.

I find this alien to my point of view. On the other hand, my side projects aren't driven by FOMO but are more like the "special interests" of autistic people.

Most of the time I have three side projects going on, maybe two of which are really getting the attention they deserve and one that is languishing. (See my profile to see about my current three.) Occasionally I get inspired to spend 1-4 weekends on some sudden inspiration, of which

https://github.com/paulhoule/pidove

came to completion but

https://github.com/paulhoule/ferocity

probably won't. The project I'm working the hardest on now is something that I was baffled that it didn't exist 18 years ago but felt compelled to do something out because of the Twitterinsanity last December and it turned out the technological conditions right now make it the perfect time to work on.

I have hacked off and on on this

https://github.com/paulhoule/ferocity

which lets you write extended Java in a lispy syntax. It generates stubs for the standard library and other packages you choose, unerasing types by putting them into method names. It works pretty well with IDEs but there are still problems w/ type erasure such that some kinds of type checking can't be done by the compiler working directly on the lispy Java, probably I wouldn't implement newer features such as pattern matching that are dependent on type inference to work, though lambda definitions are feasible if you give specific types.

The 'extended' bit was almost discovered instead of invented in that it is pretty obvious that you need quote and eval functions such that you can write lispy Java programs that manipulate Java expressions. Said expressions can be evaled at runtime with a primitive interpreter or incorporated into classes that are compiled w/ Javac. The motivation of the thing was to demonstrate Java-embedded-in-Java (an ugly kind of homoiconicity) and implement syntactic macros from Java which I think that prototype proves is possible but there is a lot more to be done on it to be really useful. Enough has been implemented in it right now in that you can use it to write the code generator that builds stubs. It might be good for balls-to-the-walls metaprogramming in Java but I think many will think it combines all the worse features of Java and Lisp.

There is this project

https://github.com/paulhoule/ferocity/

which I might finish up when I'm done with the report I'm writing. It is possible to create Java expression trees with trees of static method calls that look a lot like S-expressions and stick them together into statements, methods and classes.

You should be able to do the same tricks people do with LISP macros and it could work code generation miracles but it would have that "LISP curse" problem in spades.

The plan is to generate a code generator that is sufficient to generate the full DSL implementation (ferocity0) and use that to generate the full implementation (ferocity.)

I have some tests for ferocity0 writing .java files to get fed to javac and for ferocity0 running expression trees with a primitive interpreter. Already the type system is enriched over the type system because interpreted ferocity0 can handle Java expressions as a type at run time so you get issues like quoting and unquoting in LISP.

Internal or external?

I think Java is just fine for internal DSLs, see

https://www.jooq.org/

I was also hacking on this project

https://github.com/paulhoule/ferocity/

which was about making Java homoiconic. Namely in ferocity you can write

   Expression literal = of("Hello World");

You could process that Expression in a few different ways, for instance you could evaluate the tree using reflection or you could turn the tree into source code and compile it with javac.

Going down that path I found it was more about discovery rather than invention. That is, you will discover the issues that come up around quoting in LISP and develop some answers to them. Also for the exercise to be interesting at you will get into an execution model that is a bit bigger than the original language : for the metaprogramming to be fun at all you want to be able to write Expressions that manipulate Expressions so you end up introducing types that don't really exist in the base language. These are almost trivial to handle using the primitive interpreter style but could be more of a challenge to compile to source code.

Some source code is here

https://github.com/paulhoule/ferocity

It's been a long time since I worked on it and I need to see if I have more notes explaining the plan behind it, but the idea was to develop "ferocity0" which was a version of the DSL that was good enough to build stubs for some of the language and a large part of the standard library, and then use code generation techniques to create "ferocity1" which would be like "ferocity0" but would cover 100% of the Java language.

My current side project is being sidelined by supply chain problems so I might go back into that one.

> 2. ORMs do not hide SQL nastiness.

This is certainly true!

I mean: ORMs are now well known to "make the easy queries slightly more easy, while making intermediate queries really hard and complex queries impossible".

I think the are of ORMs is over. It simply did not deliver.

If a book on SQL is --say-- 100 pages, a book on Hibernate is 400 pages. So much to learn just to make the easy queries slightly easier to type? Just not worth it.

I prefer jooq any day over ORMs. And dont get me started over what tools like Hasuna have to offer.

There are also some languages (forgot the names) that are SQL-done-right. Select in the back, more type safe, more logic, more in the same steps as the query gets executed. These need to be adopted by PG and MySQL and we're good to go. (IMHO)

https://www.jooq.org/

https://hasura.io/

See: https://github.com/jOOQ/jOOQ/issues/5916

I've worked on a few systems that were stored procedure heavy with Microsoft SQL server, not so much because I am a .NET guy, some of these came when I was working on super-random projects for a consulting company.

My take is that it is a lot like building a system that has an API over the database except that instead of writing in API in, say, Java and exposing it through an http API with, say, JAX-RS, you are writing the API in stored procedures and accessing it through JDBC or ODBC or the native API of the database.

It seems very possible to build some thin layer that uses metadata to make an http API over stored procedures.

I'd say that systems like that can work very well.

The basic challenge is maintaining version control over your scripts, my coworkers were rubyists on my first such project and built a system inspired by migrations in Ruby on Rails where we wrote up and down migration scripts for every database change. I carried that approach to other projects where the people had less discipline to begin with. There is a little awkwardness there that the "down" script that reverts a procedure to a previous version has a cut-and-pasted copy of the old version of the stored procedure, if I had to do it over again I'd make something where each version of a stored proc is in a numbered file and the "migrations" just say "upgrade ABC proc to version 7" or maybe you could make something that looks into the VCS and finds the old version.

From what I've read, PL/SQL from Oracle looks a lot better than the Transact-SQL language in SQL server but I've never done a major project with Oracle. Most places I've worked at recently use PostgreSQL and I think this would be a viable architecture for that.

One area where it seems to be a hassle is with the "query builder" pattern, for instance where I work now we have a very complex search form with a huge number of options that builds a SQL query. I think you can do that kind of thing with what they call "Dynamic SQL", see

https://www.postgresql.org/docs/15/plpgsql-statements.html#P...

but it seems preferable to do that kind of thing with a real programming language, particularly if you have tools like

https://www.jooq.org/

I have a bachelor's in computer science, took a databases class in college (which I did poorly in), and worked as a backend developer for two years, but I always struggled with SQL. I can do the basic SELECT * FROM table WHERE column = 1 but when the SQL statement gets long with lots of joins I couldn't understand it and relied on another programmer for help. When I need to build a website myself I end up going with MongoDB because that allows me to write code instead of writing SQL. That being said, instead of doing all the data processing stuff in the backend, I'd like to try doing as much of it as I can in the database and learn some programmimg language that is SQL-esque (for my next personal project). I know Scala and am very comfortable doing functional programming stuff like List(1,2,3).filter(_.isEven()) to get the even numbers in a list or writing List(1,2,3).reduce(_+_) to apply the addition operation on all the numbers in a list. I know the Big Data framework Apache Spark is written in and works with Scala, but I really want to learn something that works with a traditional database that runs on one centralized server and not have to worry about the distributed computing MapReduce paradigm stuff (also installing the Big Data ecosystem on my personal computer is a pain). Like I want to try building something with a traditional database like Oracle, SQL Server, etc. Something I find really helpful is having IDE code error highlighting and auto-completion and the ability to run a static analysis code quality checker tool, which I can do with Scala code, but I don't know of a way to have those things with traditional SQL Strings sent from the backend to the database. I know of things like Java's JOOQ or C#'s LINQ, but I don't want to use one of those, I want to use something in the database that is database specific and pushes as much data processing as possible into the database. I heard of languages like PL/SQL, T-SQL, and PL/pgSQL to add procedural control to SQL, making it like a real programming language. On Wikipedia I found this list of PL/SQL editors but there are so many choices and I don't know which one to pick (it has to not cost me money and I would love one that has auto-completion with error highlighting and suggestions or maybe some sort of graphical query builder tool that gives me choices of what to put to compose my statement). I saw someone on Reddit say that more recently they added the ability to add database triggers in various programming languages. I also heard mentions online of this thing called "dbt" which adds software engineering tools like version control to SQL, but I don't know if that can help me get around my difficulty with SQL or if it is something I would want to use. Any advice would be appreciated.

Spring Boot, ili ako bi nesto vise lightweight u Javi Spark + jOOQ

> I've been doing ORM on Java since Hibernate was new, and it has always sucked.

Have you ever looked at something like myBatis? In particular, the XML mappers: https://mybatis.org/mybatis-3/dynamic-sql.html

Looking back, I actually quite liked it - you had conditionals and ability to build queries dynamically (including snippets, doing loops etc.), while still writing mostly SQL with a bit of XML DSL around it, which didn't suck as much as one might imagine. The only problem was that there was still writing some boilerplate, which I wasn't the biggest fan of.

Hibernate always felt like walking across a bridge that might collapse at any moment (one eager fetch away from killing the performance, or having some obscure issue related to the entity mappings), however I liked tooling that let you point towards your database and get a local set of entities mapped automatically, even though codegen also used to have some issues occasionally (e.g. date types).

That said, there's also projects like jOOQ which had a more code centric approach, although I recall it being slightly awkward to use in practice: https://www.jooq.org/ (and the autocomplete killed the performance in some IDEs because of all the possible method signatures)

More recently, when working on a Java project, I opted for JDBI3, which felt reasonably close to what you're describing, at the expense of not being able to build dynamic queries as easily, as it was with myBatis: https://jdbi.org/

That said, with the multi-line string support we have in Java now, it was rather pleasant regardless: https://blog.kronis.dev/tutorials/2-4-pidgeot-a-system-for-m...

I don't think there's a silver bullet out there, everything from lightweight ORMs, to heavy ORMs like Hibernate, or even writing pure SQL has drawbacks. You just have to make the tradeoffs that will see you being successful in your particular project.

Have a look at jooq - I know this has been used to rewrite SQL from one dialect to another, so it MUST be capable of collating code activity metrics. Look here. Otherwise, you might want to look into writing your own parser. ANTLR has a T-SQL dialect parser script here.

Feature requests go here: https://github.com/jOOQ/jOOQ/issues/new/choose. Looking forward!

You may have the fortune of being familiar with the jOOQ library for Java/JVM apps, that can generate domain models based on your database schema, as well as methods on these classes to perform queries. In case you're not, here an example Postgres schema: