Feature Request: IndexLookUp PushDown

When processing queries with indexes, TiDB needs to scan the index to find the row id and then scan the row data via row-ids. This is index lookup. IndexLookup requires 2 RPC from TiDB to TiKV, and causes the latency to be high. If the data and index are in the same TiKV instance, we can push down index lookup to TiKV, saving half of the RPCs. This will save lots of TiDB/TiKV CPU/network resources, and reduce latency.

RFC: #62615

Development Tasks

Proto

• Tipb support IndexLookUp push down tipb#375

TiDB

• Add hint INDEX_LOOKUP_PUSH_DOWN to parser #62713
• Make SelectResult to read multiple outputs from one dist task request. #63318
• planner: PhysicalIndexLookUpReader support IndexLookup push down. #63611
• executor: support to execute pushdown in IndexLookupReader #63745
• tests: small fix of test of index_lookup_pushdown.test #63906
• executor: IndexLookupReader push down support partition table #64594
  • *: support for partition table and common handle for index lookup push down #64839
• executor: IndexLookupReader push down support common handle #64595
  • *: support for partition table and common handle for index lookup push down #64839
• Add metrics for index lookup push-down #64485
• planner: fix hint index_lookup_pushdown may affect other hint even pushdown is not allowed #64520
• next-gen support index lookup push down #64664
• *: add bazel_pushdowntest #64704
• store: revert some unexpected comment in mockstore #64732
• test: optimize the test for TopN in index lookup pushdown. #64867
• *: support system variable tidb_index_lookup_pushdown_policy and hint NO_INDEX_LOOKUP_PUSHDOWN #64932
• *: fix index lookup pushdown error for partition table + uncomitted rows #65157
• executor: IndexLookupReader push down support keep order
• support multi-value index

TiKV

• IndexLookup push-down sub tasks in TiKV

release-8.5

• *: cherry-pick IndexLookup pushdown to release 8.5 #64714
• *: fix index lookup pushdown error for partition table + uncomitted rows #65162