Presto VS HikariCP

Presto: Presto is an open-source distributed SQL query engine that enables querying data from various sources. It provides fast and interactive analytics capabilities, supporting a wide range of data formats and integration with different storage systems.

The rise of Big Data projects, real-time self-service analytics, online query services, and social networks, among others, have enabled scenarios for massive and high-performance data queries. In response to this challenge, MPP (massively parallel processing database) technology was created, and it quickly established itself. Among the open-source MPP options, Presto (https://prestodb.io/) is the best-known option. It originated in Facebook and was utilized for data analytics, but later became open-sourced. However, since Teradata has joined the Presto community, it offers support now.

We have some of this functionality in Presto (https://github.com/prestodb/presto), but it takes fair bit of work to implement it for all the different backends.

Note that glibc has a similar problem in multithreaded contexts. It strands unused memory in thread-local pools, which grows your memory usage over time like a memory leak. We got lower memory usage that didn't grow over time by switching to jemalloc.

Example of this: https://github.com/prestodb/presto/issues/8993

I look at the grammar. Here is preto's grammar which is mostly similar to other sql engines: https://github.com/prestodb/presto/blob/master/presto-parser/src/main/antlr4/com/facebook/presto/sql/parser/SqlBase.g4

export MVNW_VERBOSE=true git clone https://github.com/prestodb/presto.git cd presto bash ./mvnw clean install

more specifically, recently started learning about Presto [paper], and have been diving deeper into [source] code.

> are they using a connection pooler

We use Hikari [1] an in-process connection pooler. We didn't opt for pgbouncer at al, because we didn't want to add the extra infra yet.

> since what they did in code can be natively done with PgBouncer, PgCat, et al.

Can you point me to a reference I could look at, about doing a major version upgrade with PgBouncer et al? My understanding that we would still need to write a script to switch masters, similar to what we wrote.

> all the active connections

The active connections we were referring too were websocket connections, we haven't had problems with PG connections.

Right now the algorithm we use to find affected queries and notify websockets starts to falter when the number of active websocket connections get too high. We're working on improving it in the coming weeks. I'll update the essay to clarify.

> I did feel for them here:

Thank you! That part was definitely the most frustrating.

[1] https://github.com/brettwooldridge/HikariCP

// Create a DataSource object, for example using [HikariCP](https://github.com/brettwooldridge/HikariCP) // Kapper is un-opinionated about which pooler, if any you use. val dataSource = HikariDataSource().apply { jdbcUrl = "jdbc:PostgreSQL://localhost:5432/mydatabase" username = "username" password = "password" } // The Kapper API is exposed as an extension of the java.sql.Connection interface: dataSource.connection.use { connection -> // Do database stuff }

No contexto específico estava sendo falado sobre o Hikari Connection Pool. Mas, se o Hikari é um Connection Pool, o que seria um "Pool"?

Looks like HikariCP is also awaiting fixes for this https://github.com/brettwooldridge/HikariCP/pull/2055

I have used hikari and exposed to do this in the past with postgres, although other dialects are supported.

I found one suggestion from the author of HikariCP on how to address this, which I implemented and it worked. However, there are additional classes involved, and it feels a little clunky and hard to follow.

Not really, you can change some things in spring boot but doing so will typically trigger a refresh which is less reliable than restarting but still causes a large performance hit. You could probably do it with hikari if you really needed to but it's inadvisable to build your application around this mechanic.