Embedding (NLP)

Embedding is a fundamental technology in machine learning and natural language processing that transforms discrete or complex objects (such as words, sentences, or images) into numerical vector representations of a fixed dimension. These vectors, or embeddings, are positioned in a multi-dimensional space such that semantically similar objects are located close to each other.

Definition and Concept

Formally, an embedding is a mapping function $f:X\to {ℝ}^d$, where $X$ is the original space of objects (e.g., a vocabulary of words), and ${ℝ}^d$ is a multi-dimensional vector space (the embedding space) with dimension $d$. The dimension $d$ is significantly smaller than the dimension of the original space, which makes these representations dense.

The theoretical basis for word vector representations is distributional semantics, which posits that a word's meaning is determined by the contexts in which it appears. That is, words that occur in similar contexts have similar meanings and, therefore, close vector representations.

Core Principles of Embeddings

• Fixed dimensionality: All objects are mapped to vectors of the same length, regardless of the size of the original data (e.g., the length of a sentence).
• Semantic proximity: The distance between vectors (often measured by cosine similarity) reflects the semantic similarity of the original objects.
• Support for mathematical operations: The vectors preserve semantic relationships, allowing for algebraic operations to be performed on them. The classic example is: $\text{vector}(\text{“king”})-\text{vector}(\text{“man”})+\text{vector}(\text{“woman”})\approx \text{vector}(\text{“queen”})$.

History and Development

Early Approaches (1980s–2000s)

The first ideas for vector representations emerged in the 1980s as part of neural network research. Early methods were based on statistical analysis of word co-occurrence.

The Word2Vec Era (2013)

A revolutionary moment was the development of Word2Vec in 2013 by a Google team led by Tomas Mikolov. Word2Vec proposed two efficient and computationally inexpensive architectures for training word embeddings:

• CBOW (Continuous Bag of Words): Predicts the central word based on its surrounding context.
• Skip-gram: Predicts context words from the central word.

Word2Vec became the first popular implementation of vector representations, largely due to its open-source code and high performance.

Development of Alternative Approaches

After Word2Vec, other significant static embedding models emerged:

• GloVe (2014): A model developed at Stanford University that uses global word co-occurrence statistics to train vectors.
• FastText (2015): A model from Facebook that considers word morphology by representing each word as a sum of its character n-gram vectors. This allows for the creation of embeddings even for words not present in the training vocabulary (Out-of-Vocabulary words).

The Era of Transformers and Contextual Embeddings (2018–Present)

A breakthrough occurred with the advent of the Transformer architecture and the BERT model (2018). This led to the emergence of contextual embeddings, where a word's vector representation depends on its context of use. Unlike static representations, where the word "key" would have the same vector in the sentences "door key" and "musical key," contextual models generate different embeddings for each case.

Types of Embeddings

By Level of Representation

• Word embeddings: The basic type, where each word is represented by a separate vector (Word2Vec, GloVe).
• Sentence and document embeddings: Represent entire phrases, sentences, or documents with a single vector (PV-DM, PV-DBOW).
• User and item embeddings: Used in recommendation systems to represent user interests and item characteristics.

By Contextuality

• Static embeddings: Each word is assigned a single, fixed vector, regardless of context (Word2Vec, GloVe, FastText).
• Contextual embeddings: Generate different representations for the same word depending on its surroundings. Key examples include:
  • BERT: A bidirectional model based on Transformers.
  • ELMo: A bidirectional LSTM model.
  • RoBERTa, DistilBERT, ALBERT: Improved variants of BERT.

By Modality

• Text embeddings: The most common type, including representations of words, sentences, and documents.
• Visual embeddings: Representations of images for computer vision tasks.
• Multimodal embeddings: Combine different data types (text, images, audio) into a single vector space. An example is ImageBind, which can link data from six modalities.

Architectures and Training Methods

Classical Methods

• One-hot encoding: The simplest method, where each word is encoded as a vector the size of the vocabulary with a one in the corresponding position. Disadvantages: high sparsity and a lack of semantic information.
• Matrix factorization: Dimensionality reduction methods (e.g., LSA) applied to word co-occurrence matrices.

Neural Network Architectures

• Shallow neural networks: The CBOW and Skip-gram architectures in Word2Vec use two-layer neural networks for efficient training.
• Transformers: An architecture that revolutionized the field thanks to the attention mechanism.

Modern Approaches

• Masked language modeling: BERT uses the task of predicting masked words to learn contextual representations.
• Contrastive learning: Methods that maximize the similarity of semantically close (positive) pairs and minimize the similarity of distant (negative) pairs.

Applications of Embeddings

Embeddings have found wide application in various fields:

Natural Language Processing

• Search and information retrieval: Improving the quality of semantic search, allowing documents to be found by meaning rather than keywords.
• Text classification: Vector representations serve as input data for classifiers, increasing their accuracy.
• Sentiment analysis: Determining the emotional tone of texts while considering context.
• Machine translation: Improving the understanding of the semantics of the source and target languages.

Recommendation Systems

User and item embeddings form the basis of personalized recommendation systems, including:

• Collaborative filtering: based on embeddings of user behavior.
• Content-based filtering: using embeddings of item features.

Computer Vision

• Image classification and retrieval: Convolutional neural networks and Vision Transformers create image embeddings for classification and searching for visually similar images.

Bioinformatics and Medicine

• Medical data analysis: Analyzing clinical records and disease diagnosis.
• Molecular representations: Creating embeddings to predict the properties of chemical compounds.

Technical Aspects and Optimization

• Dimensionality optimization methods: Matryoshka Representation Learning (MRL) is an innovative approach that allows obtaining embeddings of different dimensions from a single model by concentrating important information at the beginning of the vector.
• Embedding quantization: Reducing the precision of number representation (e.g., to 8 or 4 bits) to speed up computations and save memory.
• Database integration: Modern vector databases (e.g., Milvus, Pinecone) and extensions for traditional DBMSs (e.g., pgvector for PostgreSQL) enable efficient storage and retrieval of embeddings.

External links

• Word embedding — Wikipedia

See also

• Token (LLM)
• Tokenization (NLP)
• Retrieval-augmented generation (RAG)
• Hybrid retrieval

Bibliography

• Mikolov, T. et al. (2013). Efficient Estimation of Word Representations in Vector Space. arXiv:1301.3781.
• Mikolov, T. et al. (2013). Distributed Representations of Words and Phrases and their Compositionality. arXiv:1310.4546.
• Pennington, J.; Socher, R.; Manning, C. (2014). GloVe: Global Vectors for Word Representation. PDF.
• Bojanowski, P. et al. (2017). Enriching Word Vectors with Subword Information. arXiv:1607.04606.
• Joulin, A. et al. (2017). Bag of Tricks for Efficient Text Classification. arXiv:1607.01759.
• Peters, M. E. et al. (2018). Deep Contextualized Word Representations. ACL Anthology.
• Devlin, J. et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805.
• Reimers, N.; Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. arXiv:1908.10084.
• Kusupati, A. et al. (2022). Matryoshka Representation Learning. arXiv:2205.13147.
• Radford, A. et al. (2021). Learning Transferable Visual Models From Natural Language Supervision. PMLR 139.
• Girdhar, R. et al. (2023). ImageBind: One Embedding Space To Bind Them All. CVPR 2023.