faiss is a framework that provides efficient similarity search and clustering for dense vectors. The following is the demo provided on the official website

# 1. First, build training data and test data
import numpy as np

d = 64  # dimension
nb = 100000  # database size
nq = 10000  # nb of queries
np.random.seed(1024)  # make reproduciable
xb = np.random.random(nb, d).astype("float32")  # The training data of [10000,64] and the shape is the query data of [10000,63]
xb[:, 0] += np.arange(nb) / 1000.
xq = np.random.random((nq, d)).astype('float32')
xq[:, 0] += np.arange(nq) / 1000.

# 2. Create an index. faiss creates an index and preprocesses vectors to improve query efficiency. faiss provides a variety of index methods. Here, choose the simplest index for violent retrieval of L2 distance: indexFlatL2
# When creating an index, you must specify the dimension d of the vector, and most indexes need to be trained. indexFlat2 skips this step.
import faiss

index = faiss.IndexFlatL2(d)  # build the index
print(index.is_trained)

# 3. After the index is created and trained (if necessary), you can execute the add and search methods. Add method usually adds samples during training, and search is to find similar vectors
index.add(xb)  # ad vectors to the index
print(index.ntotal)

# Pass in search vector to find similar vector

k = 4  # we want to seee 4 nearest neighbor
D, I = index.search(xq, k)  # Actual search represents the last four vectors of each query vector, D represents the distance dimension with similar vectors, and I represents the ID of similar users

print(I[:5])  # neighbors of the 5 first queries
print(D[-5:])  # neighbors of the 5 last queries

Speed up search

If too many vectors need to be stored, the speed of violent search index IndexFlat2 is very slow. The index indexvfflat (inverted file) of the search method is accelerated. In fact, K-means is used to establish clustering centers,
Then, by querying the nearest cluster center, and then comparing all vectors in the cluster, we can get similar vectors
When creating indexVFFlat, you need to specify another index as the quantizer to calculate the distance or similarity. The add method needs to be trained first
Parameter introduction:
faiss.METRIC_L2: faiss defines two measures of similarity, namely faiss METRIC_ INNER_ PRODUCT. One is the Euclidean distance, a vector inner product
nlist: number of cluster centers
k: Find the most similar K vectors
index.nprobe: find the number of cluster centers. The default is 1

nlist = 100  # Number of cluster centers
k = 4
quantizer = faiss.IndexFlatL2(d)  # the other index
index = faiss.indexIVFFlat(quantizer, d, nlist,
                           faiss.METRIC_L2)  # here we specify METRIC_L2,by default it performs inner-product search
assert not index.is_trained
index.train(xb)
assert index.is_trained

index.add(xb)  # add may be a bit slower as well
D, I = index.search(xq, k)  # actual search
print(I[-5:])  # neighbors of the 5 last queries

index.nporbe = 10  # default nprobe is 1, try a few more
D, I = index.search(xq, k)
print(I[-5:])

Method to reduce memory: use disk to store inverted indexes

The indexes indexFlatL2 and indexIVFFlat we saw above will store all vectors in memory. In order to meet the demand of large amount of data, faiss provides a compression algorithm based on product quantizer to encode the vector size to the specified number of bytes. At this time, the stored vector is compressed, and the query distance is also approximate.

nlist = 100
m = 8
k = 4
quantizer = faiss.IndexFlatL2(d)  # this remains the same

index = faiss.IndexIVFPQ(quantizer, d, nlist, m, 8)  # 8 specifies that each sub_vector is encoded as 8 bits

index.train(xb)
index.add(xb)
D, I = index.search(xb[:5], k)  # sanity check
print(I)
print(D)
index.nprobe = 10  # make comparable with experiment above
D, I = index.search(xq, k)
print(I[-5:])

Simplified index representation

From the indexIVFFlat and indexIVVFPQ above, we can see that their construction needs to provide another index first. Similarly, faiss also provides pca, lsh and other methods. Sometimes they are used in combination, which will be more troublesome to construct the index. Faiss provides the way to construct the index through string expression. For example, the following expression can represent the above instance of creating indexIVFPQ

index = faiss.index_factory(d, "IVF100,PQ8")

reference resources:
Faiss tutorial details
faiss technology accumulation