How vector embeddings work in RAG: dimensionality, metrics, and semantic spaces

Computers do not understand the conceptual meaning of words, sentences, or technical skills. They only understand numbers. For decades, text search relied on lexical indexing—counting exact keyword frequencies using algorithms like TF-IDF and BM25. While fast, lexical matching fails when a job seeker searches for "frontend engineer" and an open vacancy specifies "React developer".

Vector embeddings solve this fundamental limitation. An embedding model takes arbitrary text—a word, a paragraph, or an entire resume—and transforms it into a fixed-length array of floating-point numbers called a dense vector representation. In this coordinate space, geometric distance directly mirrors semantic meaning.

When looking at vector embeddings explained through geometry, phrases that express similar underlying intent land near each other in space, enabling machines to understand relationships, analogies, and conceptual synonyms.

• Dense vectors capture semantic intent rather than literal character strings.
• High-dimensional coordinate geometry groups conceptual synonyms naturally together.
• Enables cross-lingual, semantic, and fuzzy concept retrieval across millions of documents.

Once two pieces of text are converted into dense vector representations, how does a vector database determine how closely related they are? Search engines use three primary mathematical metrics.

Vector dimensionality represents a constant engineering balancing act between semantic precision and infrastructure cost. Early embedding models used 384 or 768 dimensions. Contemporary models frequently output 1,536 or 3,072 dimensions per chunk.

Higher dimensions capture finer semantic nuance, but they quadruple RAM requirements and disk storage while slowing down approximate nearest neighbor (ANN) search across millions of vectors.

Modern architectures overcome this dilemma through Matryoshka Representation Learning (MRL). MRL trains embedding models such that the most critical semantic information is packed into the first 256 or 512 dimensions. Developers can truncate 1,536-dimensional vectors to 512 dimensions, reducing vector database storage costs by 66% while preserving 98% of retrieval accuracy.

A critical mistake in enterprise RAG architectures is treating embeddings as static, interchangeable strings. Embedding spaces are model-specific coordinate systems.

If you embed your document repository using an OpenAI model and subsequently embed incoming user queries using an open-source BGE or Cohere model, the resulting vectors exist in completely separate, non-overlapping universes. The cosine similarity scores will be random noise.

Furthermore, when upgrading an embedding model from version 2 to version 3, teams cannot perform a rolling update on the fly. The entire database must be re-embedded in a dual-write migration before switching search traffic over.

• Never mix vectors generated from different embedding models in the same search collection.
• Always record the embedding model name and dimension version in document chunk metadata.
• Plan dual-write migration pipelines whenever upgrading embedding model versions in production.