feat: add the explain_plan function

[nuvic]

It's useful to see the underlying query plan for debugging purposes. This exposes LanceScanner's explain_plan function. Addresses #1288

[github-actions]

ACTION NEEDED

Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

[nuvic]

Hi @wjones127, I've added the explain_plan function to the python package. Just wanted to check that this is the correct approach before I continue. Thanks

[wjones127]

I was thinking explain_plan would go on LanceQueryBuilder, so users could call it on their exact calls to make a query. If you do it like this, users have to figure out how to translate their parameters in this query.

There is a method _execute_query, where the plan is executed. You would do the same thing in there, but call explain_plan() after the scanner() call:

lancedb/python/python/lancedb/table.py

Lines 1670 to 1684 in 2021d5a

	return ds.scanner(
	columns=query.columns,
	filter=query.filter,
	prefilter=query.prefilter,
	nearest={
	"column": query.vector_column,
	"q": query.vector,
	"k": query.k,
	"metric": query.metric,
	"nprobes": query.nprobes,
	"refine_factor": query.refine_factor,
	},
	with_row_id=query.with_row_id,
	batch_size=batch_size,
	).to_reader()

[nuvic]

Thanks, that makes more sense. Added to LanceQueryBuilder

[wjones127]

There's a few more changes you'll need to make to get the doctests to pass, but otherwise looks good.

[wjones127]

For the example to work, you'll need to create the table:

Suggested change

	>>> import lancedb
	>>> db = lancedb.connect("./.lancedb")
	>>> table = db.open_table("my_table")
	>>> import lancedb
	>>> db = lancedb.connect("./.lancedb")
	>>> db.create_table("my_table", [{"vector": [99, 99]}])
	>>> table = db.open_table("my_table")

[wjones127]

To make the example more informative and get the example to pass, you should print the output. Also, you can use ... as a wildcard so you don't have to write it all out.

Suggested change

	>>> plan = table.search(query).explain_plan(True)
	>>> print(plan)
	>>> table.search(query).explain_plan(True)
	>>> print(plan)
	Projection: fields=[i, s, vec, _distance]
	Take: columns=\"_distance, _rowid, vec, i, s\"
	SortExec: TopK(fetch=5), expr=...
	KNNIndex: name=..., k=5, deltas=1
	ScalarIndexQuery: query=i > 10

To run the doctests locally, you can run:

pytest --doctest-modules python/lancedb

[nuvic]

Oh I see. Okay I've updated the examples so they pass the doctest
2aabbbc
f3b9624

[nuvic]

I'm trying to figure out how to add this to the nodejs sdk. Looking at nodejs/src, there doesn't seem to be a Scanner exposed there, so it's not as simple as the python solution.
Since we want to put explain_plan in the Query data structure (nodejs/src/query.rs), we need to get access to a dataset ref there, to get to the scanner. What do you recommend?

[wjones127]

I'm trying to figure out how to add this to the nodejs sdk. Looking at nodejs/src, there doesn't seem to be a Scanner exposed there, so it's not as simple as the python solution.

This is true. Although you've only implemented for the legacy Python sync API, which wraps pylance. In the future, we'll also want to add to the new Python async API. (Eventually the legacy Python sync API will be replaced by wrapping the async API, and LanceDB will no longer depend on pylance.) The async API also doesn't have a scanner directly exposed.

Since we want to put explain_plan in the Query data structure (nodejs/src/query.rs), we need to get access to a dataset ref there, to get to the scanner. What do you recommend?

You will need to add a explain_plan method to the Rust Query structure here:

lancedb/nodejs/src/query.rs

Line 33 in 56b4fd2

impl Query {

And that should call an explain_plan method you'll have to add to to the ExecutableQuery trait in this file:

lancedb/rust/lancedb/src/query.rs

Line 430 in 56b4fd2

pub trait ExecutableQuery {

Doing this for Python async API will be very similar to this.

[nuvic]

Thanks for the explanation above. I've now implemented the method for the python async API and the nodejs API. Could you take a look?

[wjones127]

I feel like this replicates some (but not all) of the logic from create_plan above. Could we instead pull this out to some common function (build_plan?) and then both create_plan and explain_plan can be built off of those?