pudzu-packages VS wikiextractor

Tools: Python and pudzu-charts.

Like last time, the plots were generated using Python and pillar.

A hacky Python script to scrape the data from yougov and generate the chart using this hacky library that I wrote.

Generated using Python and pudzu. Flag images from Wikipedia. Afghanistan's flag is the outgoing flag, though the likely new flag of the Islamic Emirate of Afghanistan has the same aspect ratio.

Medal (and population) data downloaded from Wikipedia. Graph generated in Python using pudzu-charts.

The annual fatality rates were calculated by dividing the number of leaders who died in office by the number of years that office has existed for. For French presidents this omits the Second Empire period (1852-1870), while for German chancellors it omits the Allied Occupation (1945-49) though it also starts in 1867 with the North German Confederation. The data is all from Wikipedia, and the chart was plotted using Python and pudzu.

Larger text corpora: each plot is generated from around 450MB of Wikipedia article text (or as much as is available), extracted using wikiextractor.

Methodology: extracted 100MB of article texts from each of the different Wikipedias using https://github.com/attardi/wikiextractor, and counted the character prevalences using Python. The similarity measure is just the sum of the absolute differences in character prevalences (so a lower score means more similar): e.g. if language A has distribution {A: 0.5, B: 0.3, C: 0.2} and language B has distribution {A: 0.8, B: 0.2} then their similarity is |0.5-0.8|+|0.3-0.2|+|0.2-0.0|=0.6. The final chart was generated using graphviz and pillar.

With the help of wikiextractor, i was able to query it and process the dump. However, when i start inspecting it, some articles are empty. For example AccessibleComputing should not be empty, but the dump gave: ``` AccessibleComputing 0 10 854851586 2021-01-23T15:15:01Z Elli shel wikitext text/x-wiki #REDIRECT [[Computer accessibility]]