After tearing down a dedicated server (Qdrant) and a Postgres extension (pgvector), I wanted the third camp: a vector database that is a file. LanceDB is that camp — connect() opens a path in-process, storage is the versioned Lance columnar format, and the index is IVF-PQ, not an HNSW graph. I read lancedb v0.30.0 at a5288de and lance v8.0.0 at 15f2ff5. Every claim points at a line.
The bet: a file, not a server
The entire client/embedded decision is one branch. ConnectBuilder::execute (connection.rs:910-932) checks uri.starts_with("db") — that goes to the LanceDB Cloud HTTP client — and everything else (a local path, s3://, gs://, az://) falls through to ListingDatabase, which runs inside your process. There is no daemon to start and none gets started for you: ListingDatabase opens an object store directly and, for local paths, just create_dir_alls the directory (listing.rs:550-560). A “table” is a <name>.lance directory under that root (listing.rs:218-234), opened as a NativeTable in-process (listing.rs:1198-1211). I grepped the shipped library for process or socket machinery: the only Command::spawn in src is a test utility (test_utils/connection.rs:96-101), gated behind #[cfg(test)] (lib.rs:183-184); there is no TcpListener, no bind, no listen.
Two honest nuances. First, that branch is a prefix match, not a scheme match: connect("db_data") — a perfectly plausible local directory name — matches starts_with("db"), routes to the remote branch, and errors out for lack of an API key and region (connection.rs:876-881); there’s no local fallback. The Python sync binding checks startswith("db://") properly (__init__.py:207), so this bites Rust users, not Python ones. Second, the remote path is genuinely separate code: it’s behind the remote cargo feature (lib.rs:179-180, stub error otherwise at connection.rs:901-907), a reqwest client pointed at https://{db}.{region}.api.lancedb.com (client.rs:426-428). The Rust crate ships with default=[] — remote is compile-time opt-in (Cargo.toml:111) — but the Python and Node bindings compile it in by default (python/Cargo.toml:49, nodejs/Cargo.toml:43), so for pip/npm users the embedded/cloud split is decided at runtime, by the URI alone.
And to be precise about “no server”: an s3:// URI still does network I/O — as a storage client. The compute (scans, index search, commits) stays in your process. “No server” means exactly “no LanceDB server.”
Storage: a commit is a new manifest file
If there’s no server, who owns consistency? The format does. A Lance dataset’s state at any version is one manifest file — a snapshot pointing at fragments and their data files (table.proto:36-47) — and a commit means writing the next manifest, e.g. _versions/{N}.manifest (commit.rs:93-110; a V2 naming scheme also exists, so “e.g.”). The write is create-only: RenameCommitHandler uses rename_if_not_exists and maps AlreadyExists to CommitConflict (commit.rs:1417-1468); the object-store handler uses PutMode::Create (commit.rs:1476-1532). Two writers racing to version N+1: one wins, one gets a conflict. Version numbers only go up — new_from_previous increments (manifest.rs:202-231, transaction.rs:2332-2334) — so versioning is append-only at the snapshot level.
The data itself is columnar: a fragment’s data files each hold a subset of columns for the same rows (fragment.rs:474-506, table.proto:308-380), and column metadata is laid out for individual reads, i.e. column projection (file2.proto:161-163). Your embeddings get no special home — a vector is just a FixedSizeList column (datatypes.rs:223-237) next to your metadata columns.
The consequences fall out mechanically. Deletes don’t rewrite data files; they write a new soft-tombstone deletion file (fragment.rs:1812-1880, stored under _deletions/, table.proto:436-465). Updates and compaction do rewrite fragments, but always land as a new manifest version (transaction.rs:400-424, :343-350). And because committed versions are never mutated, you get time travel for free: checkout_version reopens any past snapshot (dataset.rs:433). Old versions stick around until you clean them up — auto-cleanup exists but is strictly opt-in via a config setting, so by default history costs storage until an explicit cleanup.
The index: IVF everywhere, PQ for compression
Here’s where the comparison really splits. Qdrant and pgvector both bet on HNSW — a navigable graph over full-precision vectors, with a memory-resident inclination. Lance bets on IVF: partition the vectors into k-means cells, search only a few cells. And it’s not one option among many — every one of the seven vector index types is IVF_*: IVF_FLAT, IVF_SQ, IVF_PQ, IVF_HNSW_SQ, IVF_HNSW_PQ, IVF_HNSW_FLAT, IVF_RQ (lib.rs:137-146). The index builder enforces it: if stages[0] isn’t Ivf, it’s an error (vector.rs:524-529).
A query computes distances from the query vector to all centroids, sorts, and takes the top nprobes partitions (kmeans.rs:1305-1329, ivf.rs:1182-1192). The default isn’t a fixed small probe count — it starts at minimum_nprobes = 1 and, if k results don’t materialize, expands up to all partitions (scanner.rs:1587-1588; maximum_nprobes defaults to every partition, ivf.rs:1189).
Inside a partition, PQ is the compression that makes disk-first work: each vector is split into sub-vectors (default 16, 8 bits each, builder.rs:56-66), each quantized to a codebook code (pq.rs:43-54). At query time the engine builds a lookup table of query-to-codeword distances (distance.rs:22-55) and approximates each distance by summing table entries over the codes — asymmetric distance computation, and the original vectors are never reconstructed (distance.rs:124-159).
And HNSW? It exists here — but only inside an IVF partition, as a sub-index for searching within a cell. Ask the HNSW sub-index to pick partitions and it literally panics: find_partitions is unimplemented!("only for IVF") (hnsw/index.rs:186-192). In Qdrant and pgvector, the HNSW graph is the whole index; in Lance, the graph is at most a tenant inside an IVF cell. IVF_PQ is the canonical family member — the legacy Vector enum value is literally an alias for "IVF_PQ" (lib.rs:139,163) — though I’ll scope one thing precisely: in the lance Python dataset API there is no default index type at all; index_type is a required argument (dataset.py:3733-3736, whitelist of the seven at dataset.py:3451-3463). What the lancedb product layer defaults to is outside what I verified here.
LanceDB’s embedded path is a file-format pipeline: open a Lance table in-process, keep vectors as columns, partition with IVF, compress with PQ, and return nearest rows.
Where does this fit?
Two axes, and LanceDB stakes out a new position on both.
| Qdrant | pgvector | LanceDB | |
|---|---|---|---|
| Exists as | Dedicated server | Postgres extension | Embedded library + file format |
| Index family | HNSW graph (build-time payload links) | HNSW graph (query-time iterative scan) | IVF-first, PQ compression; HNSW only inside a cell |
| Where data lives | Engine-owned storage | Postgres pages, WAL-logged | Lance files on local disk or object store |
| Consistency story | Server coordinates | Postgres MVCC | Create-only manifest commits, versioned snapshots |
The first axis is what the vector database exists as: a server you operate, an extension inside the database you already run, or a file format your process opens. The second is the index family: HNSW graphs lean memory-resident and full-precision; IVF-PQ compresses aggressively and partitions coarsely, which is exactly the shape that survives on disk and object stores. LanceDB’s two bets are the same bet twice — no resident process, no resident graph.
When would you reach for it?
Reach for LanceDB when your data already lives on S3 or local disk, you don’t want to operate a server, and your workload looks embedded, batch, or analytical — an indexing pipeline, a notebook, a desktop app, a lambda that opens a dataset and searches it. You get versioned, time-travelable storage and vector search from a pip install, and the columnar format means your metadata and embeddings share one file layout.
Go in knowing the trade-offs. Concurrent writers resolve by commit conflict, not by a coordinating server — the create-only manifest write means racing writers get CommitConflict, which is correct but is not a multi-writer OLTP story. Low-latency online serving with hot caches and admission control is where the server engines answer. Index building is explicit — you create the index; nothing maintains a graph incrementally behind your back the way a server engine does. And in Rust, mind the db prefix: name a local directory db_anything and connect() will try to take you to the cloud.
Methodology & scope
I read lance v8.0.0 at 15f2ff5 and lancedb v0.30.0 at a5288de — the connection/database layer in lancedb, and the commit path, table format protos, and vector index crates in lance. Three headline claims — embedded routing with no daemon, append-only manifest versioning over a columnar format, and the IVF-first/PQ-compressed index family — were each run through an adversarial claim-verifier at these commits and confirmed.
One correction happened during verification, and it’s worth stating: my original claim was “only a db:// scheme routes to remote.” That was refuted — the code matches the prefix db (connection.rs:910-932) — and the corrected wording is what you read above.
Scope caveats, stated plainly:
- This is structural analysis, not benchmarks. Recall, latency, and memory behavior of IVF-PQ vs. HNSW depend on your data,
nprobes, and PQ parameters; I measured nothing here. - “No server” means no LanceDB daemon; object-store URIs still do network I/O as a storage client, and
connect_namespace("rest")is an explicit opt-in network path. - The HNSW comparisons to Qdrant and pgvector are sourced from my earlier teardowns of those repos, not re-verified in this one.
- “No default index type” is scoped to the lance Python dataset API; the lancedb product layer may supply its own default, which I did not verify.
