Apache Hive
HikariCP
Our great sponsors
Apache Hive | HikariCP | |
---|---|---|
14 | 33 | |
5,284 | 19,310 | |
1.2% | - | |
9.7 | 6.5 | |
7 days ago | 1 day ago | |
Java | Java | |
Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Apache Hive
-
Apache Iceberg as storage for on-premise data store (cluster)
Trino or Hive for SQL querying. Get Trino/Hive to talk to Nessie.
-
In One Minute : Hadoop
Hive, A data warehouse infrastructure that provides data summarization and ad hoc querying.
-
DeWitt Clause, or Can You Benchmark %DATABASE% and Get Away With It
Apache Drill, Druid, Flink, Hive, Kafka, Spark
-
Apache Spark, Hive, and Spring Boot — Testing Guide
In this article, I'm showing you how to create a Spring Boot app that loads data from Apache Hive via Apache Spark to the Aerospike Database. More than that, I'm giving you a recipe for writing integration tests for such scenarios that can be run either locally or during the CI pipeline execution. The code examples are taken from this repository.
- Apache Hive in the vein!
-
Jinja2 not formatting my text correctly. Any advice?
ListItem(name='Apache Hive', website='https://hive.apache.org/', category='Interactive Query', short_description='Apache Hive is a data warehouse software project built on top of Apache Hadoop for providing data query and analysis. Hive gives an SQL-like interface to query data stored in various databases and file systems that integrate with Hadoop.'),
-
Understanding SQL Dialects
Apache Hive takes in a specific SQL dialect and converts it to map-reduce.
-
The Data Engineer Roadmap 🗺
Apache Hive
-
Open Source SQL Parsers
Apache Calcite is a popular parser/optimizer that is used in popular databases and query engines like Apache Hive, BlazingSQL and many others.
-
Word-Aligned Bloom Filters
> whether this would really work out in most workloads
> just because it keeps the cache-lines hotter and less likely to be evicted.
Okay, so keeping cache for a bloom filter problem is real - but the real force evicting memory out of the cache line is the next row-group you read + all the other stuff you have to do when you implement this in a database product.
So the two things I work with, Apache Hive and Apache Impala switched to a blocked bloom filter at different points in time.
Hive BloomKFilter - https://github.com/apache/hive/blob/master/storage-api/src/j...
Impala/Kudu one - https://github.com/apache/impala/blob/master/be/src/kudu/uti...
The C++ one also has an AVX specialization, while the Java one relies on the JVM to do it (not always) - https://github.com/apache/impala/blob/master/be/src/kudu/uti...
We ran a lot of trivial benchmarks and several benchmarks where the shuffle-join (not sort-merge, this is just a partitioned hash join) generates a bloom filter (a semijoin) before sending rows out and the 1-cache line version won out when the bloom filter went slightly over the 1 Million + 5% rate [1].
The regular bloom filter went from (38ns -> 108ns for 1k -> 1m items), while the BloomK stuck at (27ns) despite making room for a million times more items in the bloom. The bloom-1 (which is the 64bit version) underperformed on accuracy (was ~2x faster at 16ns per op, but worse at filtering out items).
[1] - https://github.com/prasanthj/bloomfilter/tree/master/benchma...
HikariCP
-
Java virtual threads caused a deadlock in TPC-C for PostgreSQL
Looks like HikariCP is also awaiting fixes for this https://github.com/brettwooldridge/HikariCP/pull/2055
-
Writing to db
I have used hikari and exposed to do this in the past with postgres, although other dialects are supported.
-
A Tale of Two Connection Pools
I found one suggestion from the author of HikariCP on how to address this, which I implemented and it worked. However, there are additional classes involved, and it feels a little clunky and hard to follow.
-
Should I use diesel ORM if performance is the most important thing?
Whatever you choose, presuming your app is database heavy, I highly recommend spending time on DB schema design to make all queries as short as possible, avoid relying on transactions, and keep your connection pool tiny. For reference: https://github.com/brettwooldridge/HikariCP/wiki/About-Pool-Sizing
-
Art of README
You reminded me of the HikariCP library and it’s documentation: Clear and simple with references to other libraries trying to accomplish the same thing. It is not in the closure space though.
-
Java SQL code generator. SQL and OOP united finally.
There's a reason pretty much any system will use a connection pool (Hikari is the de facto standard). Yours should to.
-
How is the market for Kotlin developers where you live?
HikariCP to pool and conserve relational database connections, which are a scarce and valuable commodity
-
Recording HikariCP Metrics in Clojure
HikariCP is a popular JDBC connection pool, battle-tested and exhibiting good performance. A Clojure wrapper does the things discussed in this post here. However, our project uses HikariCP directly.
-
Retry logic for optimistic locking and transparent failover 🐘🚀
YBDemo is a simple Java program that creates an HikariCP connection pool from the hikari.properties file in the current directory, and takes SQL statements to execute as lines from stdin. There's no multi-line statement: each line is a thread, executing the statement in a loop. The goal is to make it easy to run a demo with concurrent threads by providing the set of queries in a simple way, interactive or though file redirection. Only the first column of the first row is displayed, I use row_to_json() or json_agg() to format a larger result into one value. The thread stops if no row is returned, I use RETURNING to get a row from DML.
-
JDBC Tutorial Part 3: Using Database Connection Pools
Connection pools are configurable, and different implementations have different parameters that you can adjust. For example, you might want to configure the maximum number of connection objects that the pool maintains or the maximum lifetime of a connection in the pool. Refer to the pool implementation documentation for more information.
What are some alternatives?
c3p0 - a mature, highly concurrent JDBC Connection pooling library, with support for caching and reuse of PreparedStatements.
spring-boot-r2dbc - An example implementation of Spring Boot R2DBC REST API with PostgreSQL database.
superset - Apache Superset is a Data Visualization and Data Exploration Platform
Vibur DBCP - Vibur DBCP - concurrent and dynamic JDBC connection pool
JDBI - The Jdbi library provides convenient, idiomatic access to relational databases in Java and other JVM technologies such as Kotlin, Clojure or Scala.
Flyway - Flyway by Redgate • Database Migrations Made Easy.
jOOQ - jOOQ is the best way to write SQL in Java
Presto - The official home of the Presto distributed SQL query engine for big data
H2 - H2 is an embeddable RDBMS written in Java.
ObjectBox Java (Kotlin, Android) - Java and Android Database - fast and lightweight without any ORM
FlexyPool - FlexyPool adds metrics and failover strategies to a given Connection Pool, allowing it to resize on demand.
Apache Phoenix - Mirror of Apache Phoenix