wtf_wikipedia VS duckling

This may finally be a solution for scraping wikipedia and turning it into structured data. (Or do we even need structured data in the post-AI age?)

Mediawiki is notorious for being hard to parse:

* https://github.com/spencermountain/wtf_wikipedia#ok-first- - why it's hard

* https://techblog.wikimedia.org/2022/04/26/what-it-takes-to-p... - an entire article about parsing page TITLES

* https://osr.cs.fau.de/wp-content/uploads/2017/09/wikitext-pa... - a paper published about a wikitext parser

For the reasons others have said I don't see it replacing 'traditional' scraping soon. But I am looking forward to it replacing current methods of extracting data from the scraped content.

I've been using Duckling [0] for extracting fuzzy dates and times from text. It does a good job but I needed a custom build with extra rules to make that into a great job. And that's just for dates, 1 of 13 dimensions supported. Being able to use an AI that handles them with better accuracy will be fantastic.

Does a specialised model trained to extract times and dates already exist? It's entity tagging but a specialised form (especially when dealing with historical documents where you may need Gregorian and Julian calendars).

[0] https://github.com/facebook/duckling

Ah, sorry: https://github.com/facebook/duckling

To understand what relative distance and direction is indicated from the reference point, I'd look into something like Facebook & Wit.AI's Duckling, and a custom classifier to identify if it's on the reference point ("corner of"), or some distance from ("200 meters southwest"). If you can parse out a distance and direction, then it's all logic to plot the point.

It also powers the backend of Wit.ai which FB owns. Wit's open-source entity parser, duckling, is written entirely in Haskell. https://github.com/facebook/duckling

Our RandomPicForLater intent will have one slot called reminderTime and will be of type @duckling.time. Duckling is a library that extracts entities from text, and it is one of the tools used in JAICP for this purpose. Entity types in Duckling are called dimensions and there's a number of them built in, among them is Time which suits us perfectly since we need to ask users when they want us to schedule a post for and then parse a text input into a datetime object.

I'm working on a pretty interesting project right now and I'm having different results depending on the build tool used: with cabal, the test suite fails but it passes with stack.

Try downloading the v0.2.0.0 release, extracting it somewhere, opening that location in powershell, and running these commands:

Duckling (https://github.com/facebook/duckling) is a library for parsing text into structured data.

If you really want high precision, Duckling is a good project to check out https://github.com/facebook/duckling