pg_hexedit VS TimescaleDB

I am the author of a tool that generates wxHexEditor tags and annotations for Postgres relation files -- pg_hexedit:

https://github.com/petergeoghegan/pg_hexedit

I've invested quite a lot of effort in it, and it would be nice to have support for multiple hex editors. That was anticipated to some degree:

https://github.com/petergeoghegan/pg_hexedit#supporting-othe...

I understand why you favor a declarative template format for describing files with tags -- that probably scales really nicely. What I'm doing is pretty grotty, but works surprisingly well in practice. I procedurally generate a description of each file in a shell script, and then open the file in wxHexEditor. I'm generating huge XML files, which is slow, but there are simple workarounds to get acceptable performance.

wxHexEditor doesn't support declarative tags. But even if it did I might not want to use them; the on-disk format of PostgreSQL is much more complicated than most file formats, and isn't supposed to be consumed by third party tools. Writing a C program that uses the struct definitions from the server itself makes the complexity quite manageable -- the tool is basically feature complete, even though I haven't spent a huge amount of time on it.

That said, it would be great if I could adapt pg_hexedit to a hex editor that had some kind of "best of both worlds" support for tags - tags that can be generated lazily and on-demand, when a portion of the file needs to be drawn or redrawn. This would be easy to adapt to -- Postgres relation files always consist of a series of 8KiB blocks/pages. My tool can easily generate tags for any single block without knowing any special context or having any expensive-to-generate state -- I just need a block number (i.e. an 8KiB-aligned byte offset).

For those of you who don’t know, TimescaleDB is a PostgreSQL extension for high-performance real-time analytics on time series and event data. It is available as an open-source extension or fully managed on Timescale Cloud.

If you are interested in Timescale, you can try Timescale for free or check out our GitHub, community Slackand join discussions with thousands of Timescale users.

After diving into the world of Large Language Models (LLMs) and getting comfortable with their capabilities, I decided to take the next step and work on some projects. I started looking for interesting project ideas, and that’s when I discovered an exciting opportunity: a hackathon organized by TimeScale, featuring an Open Source AI Challenge in collaboration with pgai and Ollama on dev.to

View on GitHub

Unfortunately, there’s no magic formula for setting work_mem. It depends on your system’s available memory, workload, and query patterns. The TimescaleDB Team has a tool to autotune and the topic is widely discussed. Here are some excellent resources to guide you:

Timescale is an extension for PostgreSQL that transforms it into an efficient time series database. It optimizes time series data processing using features like intelligent chunking, hypertables, and IMMV with aggregates. Hypertables automatically partition the data by time, but to the user, the table still appears as a regular table.

TimescaleDB (PostgreSQL) for primary database (can be configured for self hosted psql as well)

Yes, that ratio is very small.

I built a very simple, custom syslog solution, a syslog-ng server writing directly to a TimescaleDB hypertable (https://www.timescale.com/), and I am getting a 30x compression ratio.

We are thrilled to team up with Timescale to bring the community our newest challenge. We think you'll like this one.