wikipedia-word-frequency VS hierarchical-attention-networks

I have a project that uses words from Wikipedia (specifically, a list of frequently-used words on Wikipedia).

This might help, generated based on wikipedia data, including Ukrainian https://github.com/IlyaSemenov/wikipedia-word-frequency/tree/master/results

For this analysis, I'm going to use the word frequency of Wikipedia from 2021-08-20. I'm choosing this, because it has 2,676,542 unique words. Most of those are non-words, but regardless, they're still unique, and would work for passphrases.

Yeah, if the language doesn't have a preexisting list, you might need to generate one yourself from some large corpus of documents (e.g., Wikipedia using this tool). Btw it looks like a small one is available here for Armenian.

Have you considered trying to modify the code to be more memory efficient instead? You're just performing a word count here, you don't need to have all of your data in memory at the same time to achieve that. Hell, you could even use a pre-computed word frequency if you can find one that's suitable for your task. For example, here's one computed against english wikipedia: https://github.com/IlyaSemenov/wikipedia-word-frequency

I am following the code as given here: https://github.com/tqtg/hierarchical-attention-networks/blob/master/train.py I am not able to figure out why this error is being thrown and how to resolve it. Please help me resolve it, stuck on this for a while. I have also attached the full stacktrace below

I am using the data and code from the following link: https://github.com/tqtg/hierarchical-attention-networks/blob/master/data_prepare.py on Google Colab, but it says RAM exhausted.