tiktoken
graphql-ws
tiktoken | graphql-ws | |
---|---|---|
32 | 21 | |
9,980 | 1,669 | |
6.4% | - | |
6.7 | 7.9 | |
about 1 month ago | 12 days ago | |
Python | TypeScript | |
MIT License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
tiktoken
- FLaNK AI - 01 April 2024
-
GPT-3.5 crashes when it thinks about useRalativeImagePath too much
Their tokenizer is open source: https://github.com/openai/tiktoken
Data files that contain vocabulary are listed here: https://github.com/openai/tiktoken/blob/9e79899bc248d5313c7d...
-
How fast is JS tiktoken?
OpenAI's refference tokeniser - https://github.com/openai/tiktoken
-
Anthropic announces Claude 2.1 – 200k context, less refusals
ChatGPT presumably adds them as special tokens to the cl100k_base tokenizer, as they demo in the tiktoken documentation: https://github.com/openai/tiktoken#extending-tiktoken
-
What is the best way to get an approximate number of tokens for a piece of text?
I want to measure the approximate number of tokens in a piece of text to understand if I will need to modify it before passing it into the context of an OpenAI API call. Tiktoken can do this, but I'm not sure if it's overkill to use that library just for this simple task. I don't need to actually tokenize the text, I just need an approximate count (e.g. within like 1% of the text's actual token length for text that represents the visible text on a webpage).
-
Show HN: LLaMA tokenizer that runs in browser
https://platform.openai.com/tokenizer or the official python library tiktoken https://github.com/openai/tiktoken or this JS port of tiktoken https://github.com/dqbd/tiktoken
-
Made a GPT-3.5-Turbo and GPT-4 Tokenizer
It's built on top of the tiktoken library and is basically just a lambda function in the backend.
- AiPrice - an API for calculating OpenAI tokens and pricing
-
Anyone able to explain what happened here?
"All" is a single token in OpenAI's tiktoken Tokenizer, unrelated to the token for capital "A". Even lowercase "all" is a distinct token from "All" or "ALL."
-
Which lib is the tokenizer page using to calculate the tokens?
check tiktoken
graphql-ws
- FLaNK AI - 01 April 2024
-
Websocket with socket.io or GraphQL subscriptions
When using a GraphQL over Websocket protocol you can execute all operation kinds (query, mutation and subscription) over that transport as it is a bidirectional protocol.
-
Graph-ql subscriptions without Apollo
The GraphQL over WebSocket solution definitely requires an additional library on the client for the complexity of the protocol (namely graphql-ws). The readme of that project describes how you can connect to a GraphQL over WebSocket compliant server.
-
Issues with deploy
10:29:46 AM: Build ready to start 10:29:47 AM: build-image version: d2c6dbeac570350a387d832f64bc980dc964ad65 (focal) 10:29:47 AM: build-image tag: v4.8.0 10:29:47 AM: buildbot version: d7330f24833f29d0263d28116347ab83094a2561 10:29:47 AM: Fetching cached dependencies 10:29:48 AM: Failed to fetch cache, continuing with build 10:29:48 AM: Starting to prepare the repo for build 10:29:48 AM: No cached dependencies found. Cloning fresh repo 10:29:48 AM: git clone https://github.com/josephmasongsong/josephmasongsong-v2 10:29:49 AM: Preparing Git Reference refs/heads/main 10:29:49 AM: Parsing package.json dependencies 10:29:50 AM: Starting build script 10:29:50 AM: Installing dependencies 10:29:50 AM: Python version set to 2.7 10:29:51 AM: Downloading and installing node v16.15.0... 10:29:51 AM: Downloading https://nodejs.org/dist/v16.15.0/node-v16.15.0-linux-x64.tar.xz... 10:29:51 AM: Computing checksum with sha256sum 10:29:52 AM: Checksums matched! 10:29:54 AM: Now using node v16.15.0 (npm v8.5.5) 10:29:54 AM: Started restoring cached build plugins 10:29:54 AM: Finished restoring cached build plugins 10:29:54 AM: Attempting ruby version 2.7.2, read from environment 10:29:56 AM: Using ruby version 2.7.2 10:29:56 AM: Using PHP version 8.0 10:29:56 AM: No npm workspaces detected 10:29:56 AM: Started restoring cached node modules 10:29:56 AM: Finished restoring cached node modules 10:29:57 AM: Installing NPM modules using NPM version 8.5.5 10:29:59 AM: npm WARN EBADENGINE Unsupported engine { 10:29:59 AM: npm WARN EBADENGINE package: '[email protected]', 10:29:59 AM: npm WARN EBADENGINE required: { npm: 'please-use-yarn', yarn: '>= 1.19.1' }, 10:29:59 AM: npm WARN EBADENGINE current: { node: 'v16.15.0', npm: '8.5.5' } 10:29:59 AM: npm WARN EBADENGINE } 10:30:16 AM: npm WARN deprecated [email protected]: See https://github.com/lydell/source-map-url#deprecated 10:30:16 AM: npm WARN deprecated [email protected]: See https://github.com/lydell/source-map-resolve#deprecated 10:30:17 AM: npm WARN deprecated [email protected]: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details. 10:30:17 AM: npm WARN deprecated [email protected]: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details. 10:30:17 AM: npm WARN deprecated [email protected]: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details. 10:30:20 AM: npm WARN deprecated [email protected]: Package moved to @redux-devtools/extension. 10:30:21 AM: npm WARN deprecated [email protected]: The querystring API is considered Legacy. new code should use the URLSearchParams API instead. 10:30:25 AM: npm WARN deprecated [email protected]: The `subscriptions-transport-ws` package is no longer maintained. We recommend you use `graphql-ws` instead. For help migrating Apollo software to `graphql-ws`, see https://www.apollographql.com/docs/apollo-server/data/subscriptions/#switching-from-subscriptions-transport-ws For general help using `graphql-ws`, see https://github.com/enisdenjo/graphql-ws/blob/master/README.md 10:30:30 AM: npm WARN deprecated [email protected]: No longer maintained. Use [lru-cache](http://npm.im/lru-cache) version 7.6 or higher, and provide an asynchronous `fetchMethod` option. 10:30:31 AM: npm WARN deprecated @types/[email protected]: This is a stub types definition. vfile-message provides its own type definitions, so you do not need this installed. 10:30:36 AM: npm WARN deprecated [email protected]: The querystring API is considered Legacy. new code should use the URLSearchParams API instead. 10:30:40 AM: npm WARN deprecated [email protected]: Please see https://github.com/lydell/urix#deprecated 10:30:41 AM: npm WARN deprecated [email protected]: https://github.com/lydell/resolve-url#deprecated 10:30:44 AM: npm WARN deprecated [email protected]: This SVGO version is no longer supported. Upgrade to v2.x.x. 10:31:11 AM: npm ERR! code E401 10:31:11 AM: npm ERR! Incorrect or missing password. 10:31:11 AM: npm ERR! If you were trying to login, change your password, create an 10:31:11 AM: npm ERR! authentication token or enable two-factor authentication then 10:31:11 AM: npm ERR! that means you likely typed your password in incorrectly. 10:31:11 AM: Creating deploy upload records 10:31:11 AM: npm ERR! Please try again, or recover your password at: 10:31:11 AM: npm ERR! https://www.npmjs.com/forgot 10:31:11 AM: npm ERR! 10:31:11 AM: npm ERR! If you were doing some other operation then your saved credentials are 10:31:11 AM: npm ERR! probably out of date. To correct this please try logging in again with: 10:31:11 AM: Failed during stage 'building site': Build script returned non-zero exit code: 1 (https://ntl.fyi/exit-code-1) 10:31:11 AM: npm ERR! npm login 10:31:11 AM: npm ERR! A complete log of this run can be found in: 10:31:11 AM: npm ERR! /opt/buildhome/.npm/_logs/2022-06-09T17_29_57_939Z-debug-0.log 10:31:11 AM: Error during NPM install 10:31:11 AM: Build was terminated: Build script returned non-zero exit code: 1 10:31:11 AM: Failing build: Failed to build site 10:31:11 AM: Finished processing build request in 1m24.040513333s
-
GraphQL Subscriptions: Why we use SSE/Fetch over Websockets
There is already a JavaScript implementation for server + client with support for multiplexing from the same maintainer of graphql-ws: graphql-sse
-
how it subscription work on the apollo server?
Actually WebSocket is not the only protocol for executing GraphQL subscriptions. There is also a graphql-over-sse protocol, which is much more lightweight, and uses Server Sent Events. A popular implementation of it is the graphql-sse library, which is maintained by the same person as graphql-ws. The setup and API of both libraries is pretty similar.
-
Advice on using graphql-ws with Apollo/React
That said, I feel so close to success that I can't quite give up. I am using graphql-ws as I understand this to be the new standard implementation of GraphQL Subscriptions. However, of course, Apollo subscriptions have not been updated to implement this. So, looking at the Recipes over on the graphql-ws documentation I have found the guide for using Apollo client. However as I am new to Apollo, it is using a lot of terms I am unfamiliar with.
-
GraphQL Subscriptions and Mikro-Orm in 2021
Okay but seriously, if you've fallen down the rabbit hole of Apollo docs pointing you towards one library (subscription-transport-ws) which then points you to another (graphql-ws) , and so on and so forth, then hopefully this helps pull you out.
-
How does a client know if the server managing its subscription goes offline? (Multiple instances)
The most common protocol used today for subscriptions is the Apollo websocket protocol.
-
Graphql Subscriptions w/o apollo
GraphQL subscriptions are just web sockets. You can check graphql-ws package if you want something without Apollo dependency. Though I don't understand why you want to not depend on it in the first place.
What are some alternatives?
tokenizer - Pure Go implementation of OpenAI's tiktoken tokenizer
subscriptions-transport-ws - :arrows_clockwise: A WebSocket client + server for GraphQL subscriptions
daath-ai-parser - Daath AI Parser is an open-source application that uses OpenAI to parse visible text of HTML elements.
apollo-server - 🌍 Spec-compliant and production ready JavaScript GraphQL server that lets you develop in a schema-first way. Built for Express, Connect, Hapi, Koa, and more.
CLIP - CLIP (Contrastive Language-Image Pretraining), Predict the most relevant text snippet given an image
graphql-sse - Zero-dependency, HTTP/1 safe, simple, GraphQL over Server-Sent Events Protocol server and client.
skypilot - SkyPilot: Run LLMs, AI, and Batch jobs on any cloud. Get maximum savings, highest GPU availability, and managed execution—all with a simple interface.
GraphQL Kotlin - Libraries for running GraphQL in Kotlin
bricks - Open-source natural language enrichments at your fingertips.
ws - Simple to use, blazing fast and thoroughly tested WebSocket client and server for Node.js
terminal-copilot - A smart terminal assistant that helps you find the right command.
pothos - Pothos GraphQL is library for creating GraphQL schemas in typescript using a strongly typed code first approach