Fine Tuning Mistral 7B on Magic the Gathering Draft

• peft

  26 13,783 9.7 Python

  🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.

  

There is not a lot of great content out there making this clear, but basically all that matters for basic fine tuning is how much VRAM you have -- since the 3090 / 4090 have 24GB VRAM they're both pretty decent fine tuning chips. I think you could probably fine-tune a model up to ~13B parameters on one of them with PEFT (https://github.com/huggingface/peft)

• llm-foundry

  37 3,710 9.7 Python

  LLM training code for Databricks foundation models

  

Related comment from gwern: https://news.ycombinator.com/item?id=38438859

Also - why qlora rather than a full finetune? Using LambdaLabs, It'd cost roughly the same as your quote. Cheaper I think if you're willing to gamble with fp8: https://github.com/mosaicml/llm-foundry/tree/main/scripts/tr.... And fewer hyperparameters to tune as well

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• mage

  117 1,757 10.0 Java

  Magic Another Game Engine (by magefree)

  

Yeah, that surprised me too, given that https://github.com/magefree/mage is open source and pretty actively developed.

• MtgMatrix

  1 3 6.9 Python

  Use semantic search to browse similar and related cards in Magic: The Gathering. Powered by Langchain.

Excellent, thank you for posting this!

I was actually just looking into fine-tuning an LLM for Magic: The Gathering this week -- I've been building a small card-similarity browser using semantic embeddings of cards to find functionally or flavorfully similar cards.

I've just been using InstructorXL, but either Instructor doesn't have enough innate knowledge of the game, or else I need to work on better prompts, but so far I've tried 9 different prompts, and none of them seem to perform very well for generating embeddings:

https://github.com/HanClinto/MtgMatrix/blob/main/data/create...

So my next step was to try and download a dataset of similar cards (I have some ideas on this), and I was trying to see if I could use this to do triplet-loss training of a large embedding model or something.

Aaaaand, that's as far as I've gotten. I haven't actually figured out _how_ to hook all of that up, but your post is extremely inspirational for me. Thank you for posting this!!

Suggest a related project