# Creating Vocab from the Input TextString

**URL:** <https://discourse.cogstack.org/t/creating-vocab-from-the-input-textstring/205>\
**Category:** MedCAT\
**Created:** [May 22, 2023, 11:59pm UTC](https://discourse.cogstack.org/t/creating-vocab-from-the-input-textstring/205 "2023-05-22T23:59:44Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sen](https://avatars.discourse-cdn.com/v4/letter/s/cc9497/32.png) [@Sen](https://discourse.cogstack.org/u/Sen)\
**Post date:** [May 22, 2023, 11:59pm UTC](https://discourse.cogstack.org/t/creating-vocab-from-the-input-textstring/205/1 "2023-05-22T23:59:44Z")

</div>

I have created SNOMET\_CDB.DAT as mentioned in

[https://htmlpreview.github.io/?https://github.com/CogStack/MedCATtutorials/blob/main/notebooks/specialised/Preprocessing\_SNOMED\_CT.html](https://htmlpreview.github.io/?https://github.com/CogStack/MedCATtutorials/blob/main/notebooks/specialised/Preprocessing_SNOMED_CT.html)

I am curious on how best to create a vocab. Can somebody point to the code that creates vocab from a input text file.  
The following is not helpful for me.

> **[Google Colaboratory](https://colab.research.google.com/github/CogStack/MedCATtutorials/blob/main/notebooks/introductory/Part_3_1_Building_a_Concept_Database_and_Vocabulary.ipynb)**

---

<div class="post-metadata">

**Author:** ![patrickj](https://avatars.discourse-cdn.com/v4/letter/p/e36b37/32.png) [@patrickj](https://discourse.cogstack.org/u/patrickj)\
**Post date:** [May 25, 2023, 8:55am UTC](https://discourse.cogstack.org/t/creating-vocab-from-the-input-textstring/205/2 "2023-05-25T08:55:22Z")

</div>

Hi Sen

Did you have any luck with this? I have the same question.

The following article:

[https://www.researchgate.net/publication/351357469\_Multi-domain\_Clinical\_Natural\_Language\_Processing\_with\_MedCAT\_the\_Medical\_Concept\_Annotation\_Toolkit](https://www.researchgate.net/publication/351357469_Multi-domain_Clinical_Natural_Language_Processing_with_MedCAT_the_Medical_Concept_Annotation_Toolkit)

says that “We have compiled our own VCB by scraping Wikipedia and enriching it with words from UMLS. Only the Wikipedia VCB is made public, but the full VCB can be built with scripts provided in the MedCAT repository ([GitHub - CogStack/MedCAT: Medical Concept Annotation Tool](https://github.com/CogStack/MedCAT)).”

However, I can’t find the scripts.

---

<div class="post-metadata">

**Author:** ![Sen](https://avatars.discourse-cdn.com/v4/letter/s/cc9497/32.png) [@Sen](https://discourse.cogstack.org/u/Sen)\
**Post date:** [May 25, 2023, 1:17pm UTC](https://discourse.cogstack.org/t/creating-vocab-from-the-input-textstring/205/3 "2023-05-25T13:17:32Z")

</div>

Thanks @patrickj . Now i know i am not alone 🙂@anthony.shek we would be much helped with any guidance.

---

<div class="post-metadata">

**Author:** ![anthony.shek](https://avatars.discourse-cdn.com/v4/letter/a/b782af/32.png) [@anthony.shek](https://discourse.cogstack.org/u/anthony.shek)\
**Post date:** [May 25, 2023, 5:45pm UTC](https://discourse.cogstack.org/t/creating-vocab-from-the-input-textstring/205/4 "2023-05-25T17:45:46Z")

</div>

Hey @Sen @patrickj

Unfortunately we do not share the exact way we do it. However you can use the following code below to create your own. Just replace the comments with your own code. Good Luck! 😃

```auto
from medcat.vocab import Vocab
import os

vocab = Vocab()

# the vocab.txt file need to be in the tab sep format: <token>\t<word_count>\t<vector_embedding_separated_by_spaces>
# Vector embedding can be created from Word2Vec, you can also use transformer embeddings calculated from transformer libraries and packages such as BERT.
# embeddings of 300 dimensions is standard

vocab.add_words('vocab_data.txt', replace=True)
vocab.make_unigram_table()
vocab.save("vocab.dat")

```

---

<div class="post-metadata">

**Author:** ![patrickj](https://avatars.discourse-cdn.com/v4/letter/p/e36b37/32.png) [@patrickj](https://discourse.cogstack.org/u/patrickj)\
**Post date:** [May 25, 2023, 7:21pm UTC](https://discourse.cogstack.org/t/creating-vocab-from-the-input-textstring/205/5 "2023-05-25T19:21:53Z")

</div>

Thanks for the reply Anthony

---

<div class="post-metadata">

**Author:** ![Sen](https://avatars.discourse-cdn.com/v4/letter/s/cc9497/32.png) [@Sen](https://discourse.cogstack.org/u/Sen)\
**Post date:** [May 26, 2023, 3:35am UTC](https://discourse.cogstack.org/t/creating-vocab-from-the-input-textstring/205/6 "2023-05-26T03:35:32Z")

</div>

Thanks for the hand @anthony.shek. I am able to get the vocab built up.  
@patrickj are your able to get at it? Let me know if you want a hand.