langchainjs VS instructor

You will find the below code in langchainjs/src/utils/openai.ts

At line 11 in langchain-openai/src/utils/headers.ts

In this article, we will review deno.json file in lanchainjs source code. we will look at:

LangChain.js on GitHub

AI: Langchainjs

The current implementation uses the node-redis client, but I wanted to try out iovalkey client. I am not a JS/TS expert, but it was simple enough to port the existing implementation.You can refer to the code on GitHub

import { GithubRepoLoader } from "langchain/document_loaders/web/github"; import ignore from "ignore"; const loader = new GithubRepoLoader( "https://github.com/langchain-ai/langchainjs", { recursive: false, ignorePaths: ["*.md", "yarn.lock"] } ); const docs = await loader.load(); console.log(docs.slice(0, 3));

*** all code examples are using LangChain TS on the main branch on September 22nd, 2023 (roughly version 0.0.153).

At the time of the prototype's development, the Langchain GitHub loader sent one request per file to fetch the repository sequentially, leading to prolonged download times. In our case about 2 minutes for the insights.opensauced.pizza repository. This issue was later resolved in hwchase17/langchainjs#2224, enabling parallel requests for faster retrieval.

For those looking for an even easier way to manage structured outputs, the Instructor library is a fantastic tool. Instructor is built on top of Pydantic and aims to simplify the process of getting structured data like JSON from LLMs.

I’ve recently come across the instructor library, and I have to say, I’m pretty impressed. The concept of structuring unstructured data is both powerful and, dare I say, a bit magical. The idea that you can take data that’s all over the place and somehow impose order on it—well, that’s just my kind of wizardry.

We will start with generating some synthetic data for our app. For that we will use Virat’s processed AirBnB 2023 10k filings dataset and generate synthetic data for the sub-step (expanding the keyword into a query). As this dataset contains triplets of question, context and answer, we will do the inverse of the sub-step: generate a keyword query from the provided question. To do that, we will use Instructor with the OpenAI API to generate the keyword query.

If you look into the instructor code(https://github.com/jxnl/instructor/blob/06a49e7824729b8df1f7...). Here is the core code snippet they use:

```

instructor-go is a port of the popular Python package https://github.com/jxnl/instructor.

This implementation uses `jsonschema` and Go struct tags to send data and schema information to the model to return the appropriate response schema.

Currently, OpenAI and Anthropic are supported, and you can see all examples of different capabilities here: https://github.com/instructor-ai/instructor-go/tree/main/exa....

This is in early development and would love some feedback.

Thanks for checking it out!

The Structured project started as a Go conversion of Instructor, but it is a more general-purpose library. It is designed to be extremely easy to use and set up.