deepirtools VS transformers

Agreed, item response theory (IRT) would be a principled approach to use. I recently released a Python package called DeepIRTools that could be helpful here. It uses a deep learning approach to fit IRT models and provides a method for determining the latent dimensionality (i.e., how many "components" to retain). To get the dimension-reduced data (called embeddings in deep learning, factor scores in IRT, and components in PCA), you would just call model.scores(data).

There were actually multiple bugs which impacted long context benchmarks and general inference - I helped fix some of them.

1. RMS norm was 1e-6, but should be 1e-5 - see https://github.com/huggingface/transformers/pull/37418

2. Llama 4 Scout changed RoPE settings after release - conversion script for llama.cpp had to be fixed. See https://github.com/ggml-org/llama.cpp/pull/12889

3. vLLM and the Llama 4 team found QK Norm was normalizing across entire Q & K which was wrong - accuracy increased by 2%. See https://github.com/vllm-project/vllm/pull/16311

If you see https://x.com/WolframRvnwlf/status/1909735579564331016 - the GGUFs I uploaded for Scout actually did better than inference providers by +~5% on MMLU Pro. https://docs.unsloth.ai/basics/tutorial-how-to-run-and-fine-... has more details

import os from pathlib import Path from tempfile import mkdtemp from dotenv import load_dotenv from langchain_core.prompts import PromptTemplate from langchain_docling.loader import ExportType from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline # Import pipeline from langchain_community.llms import HuggingFacePipeline def _get_env_from_colab_or_os(key): try: from google.colab import userdata try: return userdata.get(key) except userdata.SecretNotFoundError: pass except ImportError: pass return os.getenv(key) load_dotenv() # https://github.com/huggingface/transformers/issues/5486: os.environ["TOKENIZERS_PARALLELISM"] = "false" HF_TOKEN = _get_env_from_colab_or_os("HF_TOKEN") print(f"The value of HF_TOKEN is: '{HF_TOKEN}'") FILE_PATH = ["https://arxiv.org/pdf/2408.09869"] # Docling Technical Report EMBED_MODEL_ID = "sentence-transformers/all-MiniLM-L6-v2" GEN_MODEL_ID = "mistralai/Mixtral-8x7B-Instruct-v0.1" # Added modifications - Mistral 7B Instruct v0.1 tokenizer = AutoTokenizer.from_pretrained(GEN_MODEL_ID, token=HF_TOKEN) model = AutoModelForCausalLM.from_pretrained(GEN_MODEL_ID, token=HF_TOKEN) # Create the text-generation pipeline pipeline = pipeline("text-generation", model=model, tokenizer=tokenizer) # Initialize Langchain LLM using the pipeline llm = HuggingFacePipeline(pipeline=pipeline) ### END of added modifications EXPORT_TYPE = ExportType.DOC_CHUNKS QUESTION = "Which are the main AI models in Docling?" PROMPT = PromptTemplate.from_template( "Context information is below.\n---------------------\n{context}\n---------------------\nGiven the context information and not prior knowledge, answer the query.\nQuery: {input}\nAnswer:\n", ) TOP_K = 3 MILVUS_URI = str(Path(mkdtemp()) / "docling.db")

Transformers by Hugging Face

The first step is to prepare your data for fine-tuning. This usually involves tokenizing your text and converting it into a format that the LLM can understand. Here's an example using the Transformers library:

pip install torch torchvision torchaudio einops timm pillow pip install git+https://github.com/huggingface/transformers pip install git+https://github.com/huggingface/accelerate pip install huggingface_hub huggingface_hub[hf_xet]

Qwen 3 is coming shortly as well https://github.com/huggingface/transformers/pull/36878

That said none of the recent string of releases has done much yet to smash a wall, they've just met the larger proprietary models where they were.

Huggingface's transformers library supports something similar to this. You set a minimum length, and until that length is reached, the end of sequence token has no chance of being output.

https://github.com/huggingface/transformers/blob/51ed61e2f05...

S1 does something similar to put a lower limit on its reasoning output. End of thinking is represented with the <|im_start|> token, followed by the word 'answer'. IIRC the code dynamically adds/removes <|im_start|> to the list of suppressed tokens.

Both of these approaches set the probability to zero, not something small like you were suggesting.

Transformers like Llama use rotary embeddings which are applied in every single attention layer

https://github.com/huggingface/transformers/blob/222505c7e4d...

!pip install torch torchvision torchaudio einops timm pillow !pip install git+https://github.com/huggingface/transformers !pip install git+https://github.com/huggingface/accelerate !pip install git+https://github.com/huggingface/diffusers !pip install huggingface_hub !pip install sentencepiece bitsandbytes protobuf decord ffmpeg-python imageio opencv-python