speakeasy VS fern

I’ve built conviction that code generation only gets useful in the long term when it is entirely deterministic, or filtered through humans. Otherwise it is almost always technical debt. Hence LLM code generation products are a cool toy, but no sensible teams will use them without an amazing “Day 2” workflow.

As an example, in my day job (https://speakeasyapi.dev), we sell code generation products using the OpenAPI specification to generate downstream artefacts (language SDKs, terraform providers, markdown documentation). The determinism makes it useful — API updates propagate continuously from server code, to specifications, then to the SDKs / providers / docs site. There are no breaking changes because the pipeline is deterministic and humans are in control of the API at the start. The code generation itself is just a means to an end : removing boilerplate effort and language differences by driving it from a source of truth (server api routes/types). Continuously generated, it is not debt.

We’ve put a lot of effort into trying to make an LLM agent useful in this context. However giving them control of generated code directly means it’s hard to keep the “no breaking changes”, and “consistency” restrictions that’s needed to make code generation useful.

The trick we’ve landed on to get utility out of an LLM in a code generation task, is to restrict it to manipulating a strictly typed interface document, such that it can only do non-breaking things to code (e.g. adjust comments / descriptions / examples) by making changes through this interface.

In my mind the analagous behaviour would be if the golang checksum database added in license terms that stated "you need to abide by a BSL to use data from this service". What that actually would mean is so nebulous that it feels threatening.

[0] Source: https://registry.terraform.io/v1/providers/airbytehq/airbyte...

[1] Source: https://github.com/airbytehq/terraform-provider-airbyte/tree... gzipped : ~300 resources, ~300 data sources

(NB: in airbyte's case the TF Provider was generated from a ~150Kb OpenAPI spec via https://speakeasyapi.dev: implying docs could be compressed even more)

I'm working on a company https://speakeasyapi.dev/ with the goal helping companies in this ecosystem get great production quality client sdks, terraform providers, cli(s) and all the developer surfaces you may want supported for our API. We also manage the spec and publishing workflow for you so all you have to do is build your API and we'll do the rest.

Feel free to email me at [email protected] or join our slack (https://join.slack.com/t/speakeasy-dev/shared_invite/zt-1cwb...) . We're in open beta and working with a few great companies already and we'd be happy for you to try out the platform for free!

Hi all I am a founding engineer for a API Experience company called Speakeasy - speakeasyapi.dev and we have recently released a Client SDK Generator for APIs using OpenAPI 3.0.X documents (soon to support 3.1). The generator will generate idiomatic Golang SDKs (along with other languages) that feel natural to use, easy to mock, and just work. The generator is free to use and can be run via a standalone golang built CLI with no external dependencies that can be easily installed as a binary or via homebrew (mac & linux). Check it out here https://github.com/speakeasy-api/speakeasy. If you have any questions or want to get in touch to see how Speakeasy can help you improve your APIs, just let me know!

The generator has been battle tested on thousands of APIs and we are sharing the results in our github repo. If you want to try it out on your own, download the CLI or brew install and get started in minutes:

Lots of these have been popping up lately, they all seem really good.

https://buildwithfern.com/

Fern | https://buildwithfern.com | Founding Backend Engineer | $160k + equity | On-site NYC | Full-time

At Fern, we're creating the modern developer experience platform. We work with developer-focused companies to generate SDKs & API documentation. We're looking for a Founding Backend Engineer to help us scale with our users. You'll join a small team (3 of us) and will be a product owner who designs, builds, and ships weekly.

Learn more at https://www.buildwithfern.com/careers

Fern (YC W23) | Founding Engineer | New York City | $130k-$160k + 0.5-1.0% equity | Full Time | Open Source | https://buildwithfern.com

REST APIs underpin the internet but are still painful to work with. They are often untyped, unstandardized, and out-of-sync across multiple sources of truth. With Fern, we aim to bring great developer experiences to REST APIs.

Our stack is Next.js + Vercel, Express (Node.js) + FastAPI (Python), Postgres DB + Prisma ORM, and AWS CDK. We're open source: https://www.github.com/fern-api/fern

We closed a Seed this year from top-tier US investors, including Y Combinator, Abhinav Asthana (Postman CEO), Arash Ferdowsi (Dropbox co-founder), and Ian McCrystal (Stripe's Head of Docs).

Learn more: https://www.buildwithfern.com/careers

Thank you for your encouraging words and insights!

There are indeed popular DSLs and code to openapi solutions out there. Many of which are easy to plug in to the openapi-stack libraries btw!

I guess I personally always found it frustrating to try to control the generated OpenAPI output using additional tooling and ended up preferring yaml + a visualisation tool as the api design workflow. (e.g. swagger editor)

But something like https://buildwithfern.com, or using zod as substitute for json schema may indeed be worth a try as a step before emitting openapi.

Fern (YC W23) | Founding Engineer | New York City | $125k-$175k + equity | Full Time | Open Source | https://buildwithfern.com

REST APIs underpin the internet but are still painful to work with. They are often untyped, unstandardized, and out-of-sync across multiple sources of truth. With Fern, we aim to bring great developer experiences to REST APIs.

Our stack is Next.js + Vercel, Express (Node.js) + FastAPI (Python), Postgres DB + Prisma ORM, and AWS CDK.

We closed a Seed this year from top-tier US investors, including Y Combinator, Abhinav Asthana (Postman CEO), Arash Ferdowsi (Dropbox co-founder), and Ian McCrystal (Stripe's Head of Docs).

Apply by emailing [email protected]

Hi HN! Langfuse is OSS observability and analytics for LLM applications (repo: https://github.com/langfuse/langfuse, 2 min demo: https://langfuse.com/video; try it yourself: https://langfuse.com/demo)

Langfuse makes capturing and viewing LLM calls (execution traces) a breeze. On top of this data, you can analyze the quality, cost and latency of LLM apps.

When GPT-4 dropped, we started building LLM apps – a lot of them! [1, 2] But they all suffered from the same issue: it’s hard to assure quality in 100% of cases and even to have a clear view of user behavior. Initially, we logged all prompts/completions to our production database to understand what works and what doesn’t. We soon realized we needed more context, more data and better analytics to sustainably improve our apps. So we started building a homegrown tool.

Our first task was to track and view what is going on in production: what user input is provided, how prompt templates or vector db requests work, and which steps of an LLM chain fail. We built async SDKs and a slick frontend to render chains in a nested way. It’s a good way to look at LLM logic ‘natively’. Then we added some basic analytics to understand token usage and quality over time for the entire project or single users (pre-built dashboards).

Under the hood, we use the T3 stack (Typescript, NextJs, Prisma, tRPC, Tailwind, NextAuth), which allows us to move fast + it means it's easy to contribute to our repo. The SDKs are heavily influenced by the design of the PostHog SDKs [3] for stable implementations of async network requests. It was a surprisingly inconvenient experience to convert OpenAPI specs to boilerplate Python code and we ended up using Fern [4] here. We’re fans of Tailwind + shadcn/ui + tremor.so for speed and flexibility in building tables and dashboards fast.

Our SDKs run fully asynchronously and make network requests in the background. We did our best to reduce any impact on application performance to a minimum. We never block the main execution path.

We've made two engineering decisions we've felt uncertain about: to use a Postgres database and Looker Studio for the analytics MVP. Supabase performs well at our scale and integrates seamlessly into our tech stack. We will need to move to an OLAP database soon and are debating if we need to start batching ingestion and if we can keep using Vercel. Any experience you could share would be helpful!

Integrating Looker Studio got us to first analytics charts in half a day. As it is not open-source and does not work with our UI/UX, we are looking to switch it out for an OSS solution to flexibly generate charts and dashboards. We’ve had a look at Lightdash and would be happy to hear your thoughts.

We’re borrowing our OSS business model from Posthog/Supabase who make it easy to self-host with features reserved for enterprise (no plans yet) and a paid version for managed cloud service. Right now all of our code is available under a permissive license (MIT).

Next, we’re going deep on analytics. For quality specifically, we will build out model-based evaluations and labeling to be able to cluster traces by scores and use cases.

Looking forward to hearing your thoughts and discussion – we’ll be in the comments. Thanks!

[1] https://learn-from-ai.com/

[2] https://www.loom.com/share/5c044ca77be44ff7821967834dd70cba

[3] https://posthog.com/docs/libraries

[4] https://buildwithfern.com/

You can recommend it in what context, from openapi (as they claim https://github.com/fern-api/fern#starting-from-openapi ) or from their ... special ... definition schema?

For those wanting less talk, moar code: https://github.com/fern-api/fern-java/blob/0.4.2-rc3/example... -> https://github.com/fern-api/fern-java/blob/0.4.2-rc3/example...

I'm one of the builders of an open source project (buildwithfern.com) to improve client codegen. One of the learnings I've had is that the quality of OpenAPI specs varies widely (like really widely). We wrote a linter that suggests improvements to your OpenAPI before you run the code generators and that's been really helpful for generating idiomatic clients.

You can try Fern for free: https://buildwithfern.com