hatchet VS pyllms

Hatchet (https://hatchet.run) | New York City | Full-time

We're hiring a founding engineer to help us with development on our open-source, distributed task queue: https://github.com/hatchet-dev/hatchet.

We recently launched on HN, you can check out our launch here: https://news.ycombinator.com/item?id=39643136. We're two second-time YC founders in this for the long haul and we are just wrapping up the YC W24 batch.

As a founding engineer, you'll be responsible for contributing across the entire codebase. We'll compensate accordingly and with high equity. It's currently just the two founders + a part-time contractor. We're all technical and contribute code.

Stack: Typescript/React, Go and PostgreSQL.

To apply, email alexander [at] hatchet [dot] run, and include the following:

1. Tell us about something impressive you've built.

2. Ask a question or write a comment about the state of the project. For example: a file that stood out to you in the codebase, a Github issue or discussion that piqued your interest, a general comment on distributed systems/task queues, or why our code is bad and how you could improve it.

Can you explain why you chose every function to take in context? https://github.com/hatchet-dev/hatchet/blob/main/python-sdk/...

This seems like a lot of boiler plate to write functions with to me (context I created http://github.com/DAGWorks-Inc/hamilton).

Hello HN, we're Gabe and Alexander from Hatchet (https://hatchet.run), we're working on an open-source, distributed task queue. It's an alternative to tools like Celery for Python and BullMQ for Node.js, primarily focused on reliability and observability. It uses Postgres for the underlying queue.

Why build another managed queue? We wanted to build something with the benefits of full transactional enqueueing - particularly for dependent, DAG-style execution - and felt strongly that Postgres solves for 99.9% of queueing use-cases better than most alternatives (Celery uses Redis or RabbitMQ as a broker, BullMQ uses Redis). Since the introduction of SKIP LOCKED and the milestones of recent PG releases (like active-active replication), it's becoming more feasible to horizontally scale Postgres across multiple regions and vertically scale to 10k TPS or more. Many queues (like BullMQ) are built on Redis and data loss can occur when suffering OOM if you're not careful, and using PG helps avoid an entire class of problems.

We also wanted something that was significantly easier to use and debug for application developers. A lot of times the burden of building task observability falls on the infra/platform team (for example, asking the infra team to build a Grafana view for their tasks based on exported prom metrics). We're building this type of observability directly into Hatchet.

What do we mean by "distributed"? You can run workers (the instances which run tasks) across multiple VMs, clusters and regions - they are remotely invoked via a long-lived gRPC connection with the Hatchet queue. We've attempted to optimize our latency to get our task start times down to 25-50ms and much more optimization is on the roadmap.

We also support a number of extra features that you'd expect, like retries, timeouts, cron schedules, dependent tasks. A few things we're currently working on - we use RabbitMQ (confusing, yes) for pub/sub between engine components and would prefer to just use Postgres, but didn't want to spend additional time on the exchange logic until we built a stable underlying queue. We are also considering the use of NATS for engine-engine and engine-worker connections.

We'd greatly appreciate any feedback you have and hope you get the chance to try out Hatchet.

This is a great question, thanks for asking.

We are testing workflows internally that use orchestration software like Hatchet/Temporal to allow the framework to robustly handle 100s of GBs of upload data from parsing to chunking to embedding to storing [1][2]. The goal is to build durable execution at each step, because even steps like PDF extraction can be expensive / time consuming. We are targeting an prelim. release of these features in < 1 month.

Logging is built natively into the framework with postgres or sqlite options. We ship a GUI that leverages these logs and the application flow to allow developers to see queries, search results, and RAG completions in realtime.

We are planning on adding more features here to help with evaluation / insight as we get further feedback.

On the A/B, slow rollout, and analytics side, we are still early but suspect there is a lot of value to be had here, particularly because human feedback is pretty crucial in optimizing any RAG system. Developer feedback will be particularly important here since there are a lot of paths to choose between.

[1] https://hatchet.run/

Absolutely -- just created an issue if you'd like to follow along or provide feedback!

Kagi | Full & Part Time | Remote | http://kagi.com

Kagi is building a user-centric search engine, free from ads and tracking.

Our primary language is Crystal, and we are always interested in talking to developers who share our values. Some of our roles are listed below, but feel free to reach out even if you don't see a match.

https://help.kagi.com/kagi/company/hiring-kagi.html

You can reach me directly at [email protected]

If you're looking for alternative search, I have to mention Kagi (https://kagi.com/). Not free, but totally worth it to filter out results like geeksforgeeks, tutorialspoint, w3schools etc.

Tip: use bang searches from the browser address bar by setting your default search engine to DuckDuckGo (or https://kagi.com/)

Shout out to the Kagi search engine. There are no ads. The only incentive is to be good enough to earn your money.

All 3 of your examples are add-driven free platforms for finding content that the platform didn't produce. Free platforms are incentivized to overwhelm you with options, forcing you to look at everything, including ads. Like a grocery store or Ikea, they don't want you to make a quick in-and-out visit. They want you to look at everything they have in hopes that you'll be impulsive.

You want pretty much the opposite: a tool. Tools let you quickly and efficiently accomplish a task. No distractions. When you're done, you're done.

The paths to salvation that I see:

* Pay for tools that work well when they're available. I can't speak for them myself, but I know people who swear by <https://kagi.com/> for ad-free web searches.*

* Put time and effort into your own tools. For example, you can setup and run your own search engine. Although, if you're willing to put in that effort you might put effort into hijacking and cleaning up the interfaces of free platforms instead.

* Vote and campaign for interoperability laws to enable others to put effort into hijacking and cleaning up bad interfaces.

*Surprisingly, interfaces don't get much better when you pay for video and music streaming services. I think that comes down to how most people use them. Most people don't open Netflix knowing what they want to watch. They open Netflix knowing that they're going want to watch something, and hoping to find something entertaining.

I've been using Kagi [0] since a few months and I am extremely surprised how well it works. With DDG it took a few months and then I just added !g everywhere because I never found what I was looking for. With Kagi I almost never do that, every now and then I think I am not able to find what I expect and then I add !g, so far it hasn't given me more then what Kagi gives me.

So, I am wondering how much money you would really need. Because if Kagi can do it, why can't Bing do it?

[0]: https://kagi.com