Jewish American Heritage Month

Posted By admin on January 4, 2024

This post is co-authored by@adieldar(Principal Data Scientist, Microsoft)

In the world of AI & data analytics, vector databases are emerging as a powerful tool for managing complex and high-dimensional data.

In this article, we will explore the concept of vector databases, the need for vector databases in data analytics, and how Azure Data Explorer (ADX)aka Kusto can be used as a vector database.

Vector databases store and manage data in the form of vectors that are numerical arrays of data points. Vector databases allow manipulating and analyzing set of vectors at scale using vector algebra and other advanced mathematical techniques.

The use of vectors allows for more complex queries and analyses, as vectors can be compared and analyzed using advanced techniques such as vector similarity search, quantization and clustering.

Traditional databases are not well-suited for handling high-dimensional data, which is becoming increasingly common in data analytics. In contrast, vector databases are designed to handle high-dimensional data, such as text, images, and audio, by representing them as vectors.

This makes vector databases particularly useful for tasks such as machine learning, natural language processing, and image recognition, where the goal is to identify patterns or similarities in large datasets

Vector similarity is a measure of how different (or similar) two or more vectors are.Vector similarity search is a technique used to find similar vectors in a dataset.

In vector similarity search, vectors are compared using a distance metric, such as Euclidean distance or cosine similarity. The closer two vectors are, the more similar they are.

Embeddings are a common way of representing data in a vector format for use in vector databases. An embedding is a mathematical representation of a piece of data, such as a word, text document or an image, that is designed to capture its semantic meaning.

Embeddings are created using algorithms that analyze the data and generate a set of numerical values that represent its key features. For example, an embedding for a word might represent its meaning, its context, and its relationship to other words.

Lets take an example.

Below two phrases are represented as vectors after embedding with a model.

Phrase 1

Phrase 2

(Image credits OpenAI)

Embeddings that are numerically similar are also semantically similar. For example, as seen in the following chart, the embedding vector of canine companions say will be more similar to the embedding vector of woof than that of meow.

(Image credits OpenAI)

they can be created using standard python packages (eg. spaCy, sent2vec, Gensim), but Large Language Models (LLM) generate highest quality embeddings for semantic text search. Thanks to OpenAI and other LLM providers, we can now use them easily. You just send your text to an embedding model in Azure Open AI and it generates a vector representation which can be stored for analysis.

At the core of Vector Similarity Search is the ability to store, index, and query vector data.

ADX is a cloud-based data analytics service that enables users to perform advanced analytics on large datasets in real-time. It is particularly well-suited for handling large volumes of data, making it an excellent choice for storing and searching vectors.

ADX supports a special data type called dynamic, which can store unstructured data such as arrays and property bags. Dynamic data type is perfect for storing vector values. You can further augment the vector value by storing metadata related to the original object as separate columns in your table.

Furthermore, we have added a new user-defined function series_cosine_similarity_fl to perform vector similarity searches on top of the vectors stored in ADX.

ADX as a vector database

Lets say you want to run semantic searches on top of Wikipedia pages.

We will generate vectors for tens of thousands of Wikipedia pages by embedding them with an Open AI model and storing the vectors in ADX along with some metadata related to the page.

Demo scenario

Now we want to search wiki pages with natural language queries to look for the most relevant ones. We can achieve that by the following steps:

Semantic search flow

Lets run some queries:

This query calculates similarity score for thousands of vectors in the table within seconds and returns the top n results.

Search query 1:places where we worship

Result:

Search query 2: unfortunate events in history

Result:

If youd like to try this demo, head to the azure_kusto_vector GitHub repository and follow the instructions.

The Notebook in the repo will allow you to -

You can start by spinning up your own free Kusto cluster within seconds -https://aka.ms/kustofree

We look forward to your feedback and all the exciting things you build with vectors & ADX!

Read more from the original source:

Azure Data Explorer for Vector Similarity Search

Category: Holocaust
Tags: