Farhan Naqvi
What are special tokens in Tokenization?

What are special tokens in Tokenization?

Farhan Naqvi's photo
Farhan Naqvi
·

3 min read

Special tokens are additional tokens added during the tokenization process to serve specific purposes in natural language processing tasks. These tokens are not derived from the original text but are inserted to provide additional information or perform certain operations. Here's what each type of special token typically represents:

  1. Beginning of Sequence (BOS): This token indicates the beginning of a sequence. It is often used in tasks such as language modeling or sequence generation to signal the model to start generating text.

  2. End of Sequence (EOS): This token marks the end of a sequence. It is used in tasks like language modeling to indicate the completion of text generation.

  3. Padding Token (PAD): In machine learning tasks involving sequence data, input sequences often need to have uniform lengths for efficient processing (e.g., in mini-batch training). Padding tokens are inserted to standardize the length of input sequences by filling empty spaces in shorter sequences.

  4. Unknown Token (UNK): This token represents words or tokens that are not present in the vocabulary used for tokenization. When encountering out-of-vocabulary words during tokenization, they are replaced with the unknown token to ensure the model can still process them.

  5. Mask Token (MASK): This token is used in tasks like masked language modeling, where a portion of the input sequence is masked out, and the model is trained to predict the masked tokens based on the context provided by the surrounding tokens.

These special tokens play a crucial role in training and using natural language processing models effectively.

Examples

<BOS> & <EOS>

from tokenizers import Tokenizer, models, pre_tokenizers

# Initialize the tokenizer with a character-based model
tokenizer = Tokenizer(models.CharBPETokenizer())

# Define the special tokens
special_tokens = ["<BOS>", "<EOS>", "<PAD>", "<UNK>", "<MASK>"]

# Add the special tokens to the tokenizer
tokenizer.add_special_tokens(special_tokens)

# Tokenize a sentence
sentence = "This is an example sentence."
encoded = tokenizer.encode(sentence)

# Print the tokens with special tokens
tokens_with_special = [tokenizer.id_to_token(token_id) for token_id in encoded.ids]
print(tokens_with_special)

#OUTPUT
['<BOS>', 'T', 'h', 'i', 's', 'Ġis', 'Ġan', 'Ġexample', 'Ġsentence', '.', '<EOS>']

<MASK>

In encoder-only models like BERT, the <MASK> token is used during training for a task called Masked Language Modeling (MLM). During MLM, a certain percentage of tokens in the input sequence are randomly replaced with the <MASK> token. The model is then trained to predict the original tokens from the context provided by the surrounding tokens.

Using the example : "The quick brown fox jumps over the lazy dog."

During training, we randomly mask some tokens:

  • Original: "The quick brown fox jumps over the lazy dog."

  • Masked: "The quick brown <MASK> jumps over <MASK> lazy dog."

The model is then trained to predict the masked tokens based on the surrounding context. For example, given the masked token "<MASK>", the model should predict the original token "fox" based on the context provided by "brown" and "jumps".

This process helps the model learn meaningful representations of the input sequence and improves its ability to understand and generate text.

AIgenaiTokenization