Fine Tuning Mistral 7B on Magic the Gathering Draft

• peft

  26 15,689 9.6 Python

  🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.

  

  There is not a lot of great content out there making this clear, but basically all that matters for basic fine tuning is how much VRAM you have -- since the 3090 / 4090 have 24GB VRAM they're both pretty decent fine tuning chips. I think you could probably fine-tune a model up to ~13B parameters on one of them with PEFT (https://github.com/huggingface/peft)

• Scout Monitoring

  www.scoutapm.com featured

  Free Django app performance insights with Scout Monitoring. Get Scout setup in minutes, and let us sweat the small stuff. A couple lines in settings.py is all you need to start monitoring your apps. Sign up for our free tier today.

  

• llm-foundry

  37 3,957 9.7 Python

  LLM training code for Databricks foundation models

  

  Related comment from gwern: https://news.ycombinator.com/item?id=38438859

  Also - why qlora rather than a full finetune? Using LambdaLabs, It'd cost roughly the same as your quote. Cheaper I think if you're willing to gamble with fp8: https://github.com/mosaicml/llm-foundry/tree/main/scripts/tr.... And fewer hyperparameters to tune as well

• mage

  117 1,836 10.0 Java

  Magic Another Game Engine (by magefree)

  

  Yeah, that surprised me too, given that https://github.com/magefree/mage is open source and pretty actively developed.

• MtgMatrix

  1 3 6.9 Python

  Use semantic search to browse similar and related cards in Magic: The Gathering. Powered by Langchain.

  Excellent, thank you for posting this!

  I was actually just looking into fine-tuning an LLM for Magic: The Gathering this week -- I've been building a small card-similarity browser using semantic embeddings of cards to find functionally or flavorfully similar cards.

  I've just been using InstructorXL, but either Instructor doesn't have enough innate knowledge of the game, or else I need to work on better prompts, but so far I've tried 9 different prompts, and none of them seem to perform very well for generating embeddings:

  https://github.com/HanClinto/MtgMatrix/blob/main/data/create...

  So my next step was to try and download a dataset of similar cards (I have some ideas on this), and I was trying to see if I could use this to do triplet-loss training of a large embedding model or something.

  Aaaaand, that's as far as I've gotten. I haven't actually figured out _how_ to hook all of that up, but your post is extremely inspirational for me. Thank you for posting this!!

Suggest a related project