scispacy VS spacy-models

https://github.com/medspacy/medspacy and https://allenai.github.io/scispacy/ should get you most of the way there

The ** pip install -r requirement.txt ** failed with lot of errors while installing a dependency ** nmslib ** with an error ** clang compiler does not support '-march=native' **. Since I already had the Xcode installed on my Mac the version of C was 11 but somehow the mentioned command was failing, since the said argument was not supported on the C version. With a lot of digging around I finally found a solution mentioned on SciSpacy GitHub page.

(C:\Users\nikhizzz\AppData\Local\conda\conda\envs\tensorflowspyder) C:\Users\nikhizzz>conda install -c conda-forge spacyFetching package metadata .............Solving package specifications: .Package plan for installation in environment C:\Users\nikhizzz\AppData\Local\conda\conda\envs\tensorflowspyder:The following NEW packages will be INSTALLED: blas: 1.0-mkl cymem: 1.31.2-py35h6538335_0 conda-forge dill: 0.2.8.2-py35_0 conda-forge msgpack-numpy: 0.4.4.2-py_0 conda-forge murmurhash: 0.28.0-py35h6538335_1000 conda-forge plac: 0.9.6-py_1 conda-forge preshed: 1.0.0-py35h6538335_0 conda-forge pyreadline: 2.1-py35_1000 conda-forge regex: 2017.11.09-py35_0 conda-forge spacy: 2.0.12-py35h830ac7b_0 conda-forge termcolor: 1.1.0-py_2 conda-forge thinc: 6.10.3-py35h830ac7b_2 conda-forge tqdm: 4.29.1-py_0 conda-forge ujson: 1.35-py35hfa6e2cd_1001 conda-forgeThe following packages will be UPDATED: msgpack-python: 0.4.8-py35_0 --> 0.5.6-py35he980bc4_3 conda-forgeThe following packages will be DOWNGRADED: freetype: 2.7-vc14_2 conda-forge --> 2.5.5-vc14_2Proceed ([y]/n)? yblas-1.0-mkl.t 100% |###############################| Time: 0:00:00 0.00 B/scymem-1.31.2-p 100% |###############################| Time: 0:00:00 1.65 MB/smsgpack-python 100% |###############################| Time: 0:00:00 5.37 MB/smurmurhash-0.2 100% |###############################| Time: 0:00:00 1.49 MB/splac-0.9.6-py_ 100% |###############################| Time: 0:00:00 0.00 B/spyreadline-2.1 100% |###############################| Time: 0:00:00 4.62 MB/sregex-2017.11. 100% |###############################| Time: 0:00:00 3.31 MB/stermcolor-1.1. 100% |###############################| Time: 0:00:00 187.81 kB/stqdm-4.29.1-py 100% |###############################| Time: 0:00:00 2.51 MB/sujson-1.35-py3 100% |###############################| Time: 0:00:00 1.66 MB/sdill-0.2.8.2-p 100% |###############################| Time: 0:00:00 4.34 MB/smsgpack-numpy- 100% |###############################| Time: 0:00:00 0.00 B/spreshed-1.0.0- 100% |###############################| Time: 0:00:00 0.00 B/sthinc-6.10.3-p 100% |###############################| Time: 0:00:00 5.49 MB/sspacy-2.0.12-p 100% |###############################| Time: 0:00:10 7.42 MB/s(C:\Users\nikhizzz\AppData\Local\conda\conda\envs\tensorflowspyder) C:\Users\nikhizzz>python -VPython 3.5.3 :: Anaconda custom (64-bit)(C:\Users\nikhizzz\AppData\Local\conda\conda\envs\tensorflowspyder) C:\Users\nikhizzz>python -m spacy download enCollecting en_core_web_sm==2.0.0 from https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.0.0/en_core_web_sm-2.0.0.tar.gz#egg=en\_core\_web\_sm==2.0.0 Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.0.0/en_core_web_sm-2.0.0.tar.gz (37.4MB) 100% |################################| 37.4MB ...Installing collected packages: en-core-web-sm Running setup.py install for en-core-web-sm ... doneSuccessfully installed en-core-web-sm-2.0.0 Linking successful C:\Users\nikhizzz\AppData\Local\conda\conda\envs\tensorflowspyder\lib\site-packages\en_core_web_sm --> C:\Users\nikhizzz\AppData\Local\conda\conda\envs\tensorflowspyder\lib\site-packages\spacy\data\en You can now load the model via spacy.load('en')(C:\Users\nikhizzz\AppData\Local\conda\conda\envs\tensorflowspyder) C:\Users\nikhizzz>

Well the medium model is using Glove (common crawl) for word vectors. There are only 685K keys so depending on the corpus you are working with, its possible lots of the words you are interested in don't have a corresponding vector and end up as zero vectors. Spacy Document/Span vectors are simply averages of the word vectors. So the higher performance of phrases may simply be because there is a higher chance of non Out of Vocabulary (OOV) words. So less chance of a zero vector.

spaCy vs transformers isn't really a good comparison. You can plug a variety of things into spaCy's NLP pipelines, including Huggingface's transformer models. spaCy 3, in particular, has pre-built models with Huggingface's transformers, like en_core_web_trf.