How to Use N2 to Find Nearest Neighbors in Large Datasets
N2
Lightweight approximate Nearest Neighbor algorithm library written in C++ (with Python/Go bindings).
N2 stands for two N's, which comes from 'Approximate Nearest Neighbor Algorithm'.
Why N2 Was Made
Before N2, there has been other great approximate nearest neighbor libraries such as Annoy and NMSLIB. However, each of them had different strengths and weaknesses regarding usability, performance, and etc. So, N2 has been developed aiming to bring the strengths of existing aKNN libraries and supplement their weaknesses.
Features
- Lightweight library which runs fast with large datasets.
- Good performance in terms of index build time, search speed, and memory usage.
- Supports multi-core CPUs for index building.
- Supports a mmap feature by default to efficiently process large index files.
- Supports Python/Go bindings.
Supported Distance Metrics
| Metric | Definition | d(p, q) |
| "angular" | 1 - cosθ | 1 - {sum(p i · q i) / sqrt(sum(p i · p i) · sum(q i · q i))} |
| "L2" | squared L2 | sum{(p i - q i) 2} |
| "dot" | dot product | sum(p i · q i) |
N2 supports three distance metrics. For "angular" and "L2", d (distance) is defined such that the closer the vectors are, the smaller d is. However for "dot", d is defined such that the closer the vectors are, the larger d is. You may be wondering why we defined and implemented "dot" metric as plain dot product and not as (1 - dot product). The rationale for this decision was to allow users to directly interpret the d value returned from Hnsw search function as a dot product value.
Quickstart
- Install N2 with pip.
$ pip install n2- Here is a python code snippet demonstrating how to use N2.
import numpy as np
from n2 import HnswIndex
N, dim = 10240, 20
samples = np.arange(N * dim).reshape(N, dim)
index = HnswIndex(dim)
for sample in samples:
index.add_data(sample)
index.build(m=5, n_threads=4)
print(index.search_by_id(0, 10))
# [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]Full Documentation
Visit n2.readthedocs.io for full documentation. The documentation site explains the following contents in detail.
- Installation Guide
- how to build from source, etc.
- User Guide with Basic Examples
- Benchmark
- detailed explanation of how we performed benchmark experiemnts.
Performance
- Here are the results of our benchmark experiments.
- You can also see benchmarks of various ANN libraries in Python at ann-benchmarks.com. Note that N2 version 0.1.6 is used in ann-benchmarks.com (last checked on October 8th, 2020) and we are continuing our efforts to improve N2 performance.
Index Build Time
Search Speed
Memory Usage
References
- Y. A. Malkov and D. A. Yashunin, "Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs," CoRR, vol. abs/1603.09320, 2016. [Online]. Available: http://arxiv.org/abs/1603.09320
- NMSLIB: https://github.com/nmslib/nmslib
- Annoy: https://github.com/spotify/annoy
Download Details:
Author: kakao
Source Code: https://github.com/kakao/n2
License: Apache-2.0 license
