node-gtk VS marker

I'll say it because it kinda saddens me, but contributing to gnome is not a fun experience. Even though I would be normally super excited about such a request and would be happy to contibute to a FOSS project that I like (I did https://github.com/romgrk/web-toolkit and https://github.com/romgrk/node-gtk after all), my experience with many of the long term contributors to gnome has simply been too disheartening. You guys should think about why so many people are put off from contributing to gnome.

Look, all I'm saying is GTK needs to do better in terms of documentation, it has been needing it for years. I've worked on nodejs bindings for years and I've more than once stopped worked on it altogether simply out of frustration due to the absence of proper documentation for the whole ecosystem. Please, don't take all this as an attack, I'd be happy to help. But I think you're limiting the project by applying restrictions that chase those who could help.

Congrats on the launch! How do you guys compare with Datalab with regards to accuracy?

https://www.datalab.to/

How does it compare to Datalab/Marker https://github.com/datalab-to/marker ? We evaluated many PDF->MD converters and this one performed the best, though it is not perfect.

I usually convert to markdown from PDF local laws when I need them as a reference for the functional specification. That way its easier to pinpoint to exact section of the thing.

Its not easy to convert general law to markdown, it involved online converters and manual fixes. Currently experimenting with marker [1] on local LLM hardware and so far it is the best out there.

[1]: https://github.com/VikParuchuri/marker

> with LLM as a judge

For anyone else interested, prompt is here [0]. The model used was gemini-2.0-flash-001.

Benchmarks are hard, and I understand the appeal of having something that seems vaguely deterministic rather than having a human in the loop, but I have a very hard time accepting any LLM-judged benchmarks at face value. This is doubly true when we're talking about something like OCR which, as you say, is a very hard problem for computers of any sort.

I'm assuming you've given this some thought—how did you arrive at using an LLM to benchmark OCR vs other LLMs? What limitations with your benchmark have you seen/are you aware of?

[0] https://github.com/VikParuchuri/marker/blob/master/benchmark...

I'm a fan of the team of Allen AI and their work. Unfortunately, the benchmarking of olmocr against marker (https://github.com/VikParuchuri/marker) is quite flawed.

Throughput - they benchmarked marker API cost vs local inference cost for olmocr. In our testing, marker locally gets 20 - 120 pages per second on an H100 (without custom kernels, etc). Olmocr in our testing gets between .4 (unoptimized) and 4 (sglang) pages per second on the same machine.

Accuracy - their quality benchmarks are based on win rate with only 75 samples - which are different between each tool pair. The samples were filtered down from a set of ~2000 based on opaque criteria. They then asked researchers at Allen AI to judge which output was better. When we benchmarked with our existing set and LLM as a judge, we got a 56% win rate for marker across 1,107 documents. We had to filter out non-English docs, since olmocr is English-only (marker is not).

Hallucinations/other problems - we noticed a lot of missing text and hallucinations with olmocr in our benchmark set. You can see sample output and llm ratings here - https://huggingface.co/datasets/datalab-to/marker_benchmark_... .

You can see all benchmark code at https://github.com/VikParuchuri/marker/tree/master/benchmark... .

Happy to chat more with anyone at Allen AI who wants to discuss this. I think olmocr is a great contribution - happy to help you benchmark marker more fairly.

Very cool, any plans for a dockerized API of market similar to what Unstructured released? I know you have a very attractively priced serverless offering (https://www.datalab.to) but having something to develop against locally would be great (for those of us not in the Python world).