BooookScore
A package to generate summaries of long-form text and evaluate the coherence of these summaries. Official package for our ICLR 2024 paper, "BooookScore: A systematic exploration of book-length summarization in the era of LLMs". (by lilakk)
FABLES
By mungg
| BooookScore | FABLES | |
|---|---|---|
| 1 | 2 | |
| 72 | 28 | |
| - | - | |
| 7.0 | 8.5 | |
| about 2 months ago | about 1 month ago | |
| Python | Python | |
| MIT License | MIT License |
The number of mentions indicates the total number of mentions that we've tracked plus the number of user suggested alternatives.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
BooookScore
Posts with mentions or reviews of BooookScore.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2024-04-09.
-
Evaluating faithfulness and content selection of LLMs in book-length summaries
With a link to https://arxiv.org/pdf/2310.00785.pdf - which then links to another GitHub repository, https://github.com/lilakk/BooookScore which has a bunch of prompts in https://github.com/lilakk/BooookScore/tree/main/prompts
Which makes me think that this original paper isn't evaluating LLMs so much as it's evaluating that one particular prompting technique for long summaries.
Gemini Pro 1.5 has 1m token context length, which should remove the need for weird hierarchical summary tricks. I wonder how well it would score?
FABLES
Posts with mentions or reviews of FABLES.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2024-04-09.
-
Evaluating faithfulness and content selection of LLMs in book-length summaries
As far as I can tell, this study was almost entirely about fiction: https://github.com/mungg/FABLES/blob/main/booklist.md lists 26 books, only one of which is classified as non-fiction.
I would imagine that summaries of non-fiction books are evaluated quite differently from summaries of fiction.
I've been trying to figure out what prompts they used. The https://github.com/mungg/FABLES GitHub repo says this:
Summary -- (str) Entire book summarized