SAP HANA Cloud Vector Engine Guide: Semantic Search, RAG and HNSW

Original educational visual created for SAP Latest.

SAP HANA Cloud Vector Engine lets an application store embeddings beside enterprise data and rank semantically similar records with SQL. This makes it useful for semantic search, product matching, support knowledge retrieval and retrieval-augmented generation (RAG) without moving every business attribute into a separate vector database.

Key idea: A vector is not the source of truth. Keep the original text, business key, authorization attributes and embedding metadata together so every retrieved chunk can be verified and cited.

What a vector embedding represents

An embedding converts content into an array of numbers. Items with similar meaning should be close in the embedding space. SAP HANA Cloud provides vector data types such as REAL_VECTOR and HALF_VECTOR, along with similarity functions including COSINE_SIMILARITY and L2DISTANCE. The vector dimension depends on the embedding model.

End-to-end semantic search and RAG flow

1. Extract text from approved business documents or records.
2. Split content into chunks that preserve useful context.
3. Create document embeddings and store them with source metadata.
4. Create a query embedding using the same compatible model.
5. Rank candidates with cosine similarity or L2 distance.
6. Apply authorization and business filters before returning context.
7. Send only the selected context to the answer-generation step and return citations.

Design a vector-ready table

CREATE COLUMN TABLE KB_CHUNK (
  CHUNK_ID       BIGINT GENERATED BY DEFAULT AS IDENTITY,
  DOCUMENT_ID    NVARCHAR(80)  NOT NULL,
  CHUNK_NO       INTEGER       NOT NULL,
  CONTENT        NCLOB         NOT NULL,
  SOURCE_URL     NVARCHAR(500),
  BUSINESS_UNIT  NVARCHAR(40),
  VALID_FROM     DATE,
  MODEL_ID       NVARCHAR(100) NOT NULL,
  CONTENT_HASH   NVARCHAR(64)  NOT NULL,
  EMBEDDING      REAL_VECTOR,
  PRIMARY KEY (CHUNK_ID)
);

Why store MODEL_ID and CONTENT_HASH? They make re-embedding controlled. If the source text or embedding model changes, you can identify exactly which rows need to be regenerated.

Create embeddings correctly

When using SAP HANA Cloud’s VECTOR_EMBEDDING function, document chunks use the DOCUMENT text type and user searches use QUERY. The exact model and version available depend on your HANA Cloud configuration and release, so do not hard-code an example model without checking your system.

UPDATE KB_CHUNK
   SET EMBEDDING = VECTOR_EMBEDDING(
         CONTENT,
         'DOCUMENT',
         :model_and_version )
 WHERE EMBEDDING IS NULL
   AND MODEL_ID = :model_and_version;

Semantic search with cosine similarity

WITH QUERY_VECTOR AS (
  SELECT VECTOR_EMBEDDING(
           :question,
           'QUERY',
           :model_and_version ) AS V
    FROM DUMMY
)
SELECT TOP 8
       K.DOCUMENT_ID,
       K.CHUNK_NO,
       K.CONTENT,
       K.SOURCE_URL,
       COSINE_SIMILARITY(K.EMBEDDING, Q.V) AS SCORE
  FROM KB_CHUNK AS K
  CROSS JOIN QUERY_VECTOR AS Q
 WHERE K.BUSINESS_UNIT = :business_unit
   AND K.MODEL_ID = :model_and_version
 ORDER BY SCORE DESC;

The business filter is not optional. Similarity ranking must never bypass user authorization, data residency or document validity rules.

Vector index for larger datasets

SAP HANA Cloud supports HNSW vector indexes for approximate nearest-neighbor search. Use the same similarity function in the index and the query. Indexing can improve retrieval speed on larger datasets, but it introduces a recall-versus-latency trade-off, so benchmark with representative queries.

CREATE HNSW VECTOR INDEX KB_CHUNK_VIDX
  ON KB_CHUNK (EMBEDDING)
  SIMILARITY FUNCTION COSINE_SIMILARITY;

Benchmark first: Exact search may be sufficient for small or heavily filtered datasets. Add an HNSW index only after measuring volume, latency, recall and memory behavior in your own workload.

Chunking strategy that works

• Split on headings, paragraphs or business sections instead of arbitrary character counts.
• Keep tables, steps and warning notes with their explanatory context.
• Use a small overlap only when a concept crosses chunk boundaries.
• Store section title, document version, language and source link as metadata.
• Remove duplicate or boilerplate text before embedding.
• Evaluate retrieval with real questions and known relevant documents.

RAG safety and governance

• Authorization before generation: retrieve only records the user may see.
• Prompt-injection defense: treat retrieved content as untrusted data, not instructions.
• Citations: return source document, section and URL with the answer.
• Freshness: expire or re-embed changed documents based on hash and version.
• Model tracking: never compare vectors created by incompatible models.
• Observability: log query, filters, retrieved IDs, scores, model version and user feedback.

Performance checklist

1. Apply selective relational filters before or together with vector ranking.
2. Return only the columns and number of chunks required by the application.
3. Measure P50, P95 and P99 latency with production-like concurrency.
4. Track retrieval recall separately from answer quality.
5. Consider HALF_VECTOR only after validating memory savings and accuracy impact.
6. Rebuild or tune the index when data distribution and volume change materially.

Common mistakes

• Mixing models: document and query vectors must be compatible.
• No metadata filters: pure similarity can return the wrong company, language or validity period.
• Huge chunks: they dilute meaning and waste context tokens.
• No citations: users cannot verify the generated answer.
• Indexing too early: an index cannot fix poor chunking or irrelevant embeddings.
• Using similarity as confidence: a high score is not proof that the answer is correct.

Interview questions

1. What is the difference between cosine similarity and L2 distance?
   Cosine similarity compares vector direction; L2 measures Euclidean distance. The best choice depends on the model and evaluation results.
2. Why store relational data with vectors?
   It enables governed filters, joins, lineage and business context alongside semantic ranking.
3. What is HNSW?
   A graph-based approximate nearest-neighbor index designed for fast vector retrieval at scale.
4. Why use DOCUMENT and QUERY embedding types?
   The embedding service can optimize representations for stored content versus search intent.
5. How do you evaluate a RAG system?
   Measure retrieval recall/precision, answer faithfulness, citation accuracy, latency, cost and user feedback.

Hands-on exercise

Create a small knowledge table with 20 approved SAP support notes or internal troubleshooting articles. Chunk by heading, embed every chunk, and prepare 10 real questions with expected relevant sources. Compare exact cosine search with an HNSW index, then record recall@5 and P95 latency. Add a business-unit filter and verify that unauthorized content never appears.

Official SAP references

• SAP HANA Cloud Vector Engine overview
• CREATE VECTOR INDEX statement
• VECTOR_EMBEDDING function
• SAP HANA Cloud data exploration tools tutorial

Quick takeaway: Good vector search is a data-engineering and governance problem first. Strong chunking, metadata, authorization and evaluation matter more than adding an index or a larger model.