[MOD-6057] Improve and elaborate FT.PROFILE documentation (#2532)

* MOD-12263 vector iterator details

Added detailed notes on vector iterator profiles, including vector search modes and batch execution statistics.

* MOD-6056 rename “counter” and “size”

* MOD-12816 update ResultProcessor GILTime

* MOD-12414 Add internal cursor reads to profile documentation

* add missing iterators

* update RP section

should we remove vector filter?

* fix example with new names

* remove vector filter

* refine internal cursor description

* fix GIL entreies & optimizer

* add versions and format disclaimer + Shard ID field

* fix version disclaimer

* DWD copy-edit pass

---------

Co-authored-by: alon <[email protected]>
Co-authored-by: David W. Dougherty <[email protected]>

Author: 3 people

content/commands/ft.profile.md

@@ -87,6 +87,11 @@ is the query string, sent to `FT.SEARCH` or `FT.AGGREGATE`.
 
 ## Return
 
+{{< note>}}
+This page contains the up-to-date profile output as of RediSearch v2.8.33, v2.10.26, v8.2.7, and v8.4.3. 
+The output format itself may differ between RediSearch versions, and RESP protocol versions.
+{{< /note >}}
+
 `FT.PROFILE` returns a two-element array reply. The first element contains the results of the provided `FT.SEARCH` or `FT.AGGREGATE` command.
 The second element contains information about query creation, iterator profiles, and result processor profiles.
 Details of the second element follow in the sections below.
@@ -97,19 +102,22 @@ This section contains query execution details for each shard.
 When more than one shard is in play, the shards will be labeled `Shard #1`, `Shard #2`, etc.
 If there's only one shard, the label will be omitted.
 
-| Returned field name      | Definition |
-|:--                       |:--         |
-| `Total`&nbsp;`profile`&nbsp;`time`     | The total run time (ms) of the query. Normally just a few ms. |
-| `Parsing`&nbsp;`time`           | The time (ms) spent parsing the query and its parameters into a query plan. Normally just a few ms. |
-| `Pipeline`&nbsp;`creation`&nbsp;`time` | The creation time (ms) of the execution plan, including iterators, result processors, and reducers creation. Normally just a few ms for `FT.SEARCH` queries, but expect a larger number for `FT.AGGREGATE` queries. |
-| `Warning`                | Errors that occurred during query execution. |
+| Returned field name      | Definition                                                                                                                                                                                                            |
+|:--                       |:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `Shard ID`               | String containing the unique shard ID. In Redis Open Source, this identifier is denoted as the node ID, and is received from the `RedisModule_GetMyClusterID()` API. (Available as of RediSearch v8.4.3)                               |
+| `Total`&nbsp;`profile`&nbsp;`time`     | The total run time (ms) of the query. Normally just a few ms.                                                                                                                                                         |
+| `Parsing`&nbsp;`time`           | The time (ms) spent parsing the query and its parameters into a query plan. Normally just a few ms.                                                                                                                   |
+| `Pipeline`&nbsp;`creation`&nbsp;`time` | The creation time (ms) of the execution plan, including iterators, result processors, and reducers creation. Normally just a few ms for `FT.SEARCH` queries, but expect a larger number for `FT.AGGREGATE` queries.   |
+| `Total`&nbsp;`GIL`&nbsp;`time` | In multi-threaded deployments, the total time (ms) the query spent holding and waiting for the Redis Global Lock (referred to as GIL) during execution. Note: this value is only valid in the shards profile section. |
+| `Warning`                | Errors that occurred during query execution.                                                                                                                                                                          |
+| `Internal`&nbsp;`cursor`&nbsp;`reads` | The number of times the coordinator fetched result batches from a given shard during a distributed `AGGREGATE` query in cluster mode. Includes the initial request plus any subsequent batch fetches.                 |
 
 ### Iterator profiles
 
-This section contains index iterator information, including `Type`, `Query Type`, `Term` (when applicable), `Time` (in ms), `Counter`, `Child iterator`, and `Size` information.
-Each iterator represents an executor for each part of the query plan, nested per the execution plan. The operation types mentioned below (`UNION`, etc) should match up with the provided query.
+This section contains index iterator information, including `Type`, `Query Type`, `Term` (when applicable), `Time` (ms), `Number of reading operations`, `Child iterator`, and `Estimated number of matches` information.
+Each iterator represents an executor for each part of the query plan, nested per the execution plan. The operation types mentioned below (for example, `UNION`) should match up with the provided query.
 
-Inverted-index iterators also include the number of elements they contain. Hybrid vector iterators return the top results from the vector index in batches, including the number of batches.
+Inverted-index iterators also include the number of elements they contain. Vector iterators include additional profiling information. See [Notes on Vector iterator profile](#notes-onvector-iterator-profile) below.
 
 Iterator types include:
 
@@ -121,19 +129,49 @@ Iterator types include:
 * `TAG` with `Term`
 * `NUMERIC` with `Term`
 * `VECTOR`
+* `METRIC - VECTOR DISTANCE`
+* `GEO` with `Term`
 * `EMPTY`
 * `WILDCARD`
 * `OPTIONAL`
+* `OPTIMIZER` with `Optimizer mode` - Query optimization wrapper for `FT.SEARCH` queries sorted by a numeric field. Enabled by default in [DIALECT 4]({{< relref "/develop/ai/search-and-query/advanced-concepts/dialects##dialect-4-deprecated" >}}) or explicitly with `WITHOUTCOUNT`.
+* `ID-LIST` - Iterator over specific document IDs (appears when using `INKEYS`)
+
+**Notes on `Number of reading operations` and `Estimated number of matches`**
+
+`Number of reading operations` is the number of times an iterator was interacted with. A very high value in comparison to others is a possible warning flag. `NUMERIC` and `Child iterator` types are broken into ranges, and `Number of reading operations` will vary depending on the range. For `UNION`, the sum of the reading operations in child iterators should be equal to or greater than the child iterator's reading operations.
+
+`Estimated number of matches` is the size of the document set. `Number of reading operations` should always be equal to or less than `Estimated number of matches`.
+
+#### Notes on Vector iterator profile {#notes-onvector-iterator-profile}
+
+For vector queries, the iterator profile includes additional information specific to vector search execution.
 
-**Notes on `Counter` and `Size`**
+#### Vector search mode
 
-Counter is the number of times an iterator was interacted with. A very high value in comparison to others is a possible warning flag. `NUMERIC` and `Child interator` types are broken into ranges, and `Counter` will vary depending on the range. For `UNION`, the sum of the counters in child iterators should be equal or greater than the child iterator’s counters.
+The `Vector search mode` field indicates the execution strategy used for both `VECTOR` and `METRIC - VECTOR DISTANCE` iterator types:
 
-`Size` is the size of the document set. `Counter` should always be equal or less than `Size`.
+| Mode | Description |
+|:--   |:--          |
+| `STANDARD_KNN`              | Pure KNN vector search without filters. |
+| `RANGE_QUERY`               | Range-based vector search. Used with `VECTOR_RANGE` queries. |
+| `HYBRID_ADHOC_BF`           | Ad-hoc brute force for filtered queries. Iterates through results that pass the filter and calculates distances on-the-fly.  |
+| `HYBRID_BATCHES`            | Batch-based filtered search. Retrieves top vectors from the index in batches and checks which ones pass the filter. |
+| `HYBRID_BATCHES_TO_ADHOC_BF`| Dynamic mode switching. Starts with batch-based search but switches to ad-hoc brute force if batch iterations yield insufficient results. |
+
+#### Batch execution statistics
+
+For queries using batch modes (`HYBRID_BATCHES` or `HYBRID_BATCHES_TO_ADHOC_BF`), additional batch statistics are included:
+
+| Returned field name | Definition |
+|:--                  |:--         |
+| `Batches number`    | Total number of batch iterations executed during the search. |
+| `Largest batch size`| The maximum batch size used during execution. Batch sizes may vary dynamically based on the ratio of estimated matching documents to total index size. |
+| `Largest batch iteration (zero based)` | The iteration number (0-based) when the largest batch occurred. |
 
 ### Result processor profiles
 
-Result processors form a powerful pipeline in Redis Query Engine. They work in stages to gather, filter, score, sort, and return results as efficiently as possible based on complex query needs. Each processor reports `Time` information, which represents the total duration (in milliseconds, or ms) spent by the processor to complete its operation, and `Counter` information, which indicates the number of times the processor was invoked during the query.
+Result processors form a powerful pipeline in Redis Query Engine. They work in stages to gather, filter, score, sort, and return results as efficiently as possible based on complex query needs. Each processor reports `Time` information, which represents the total duration (in milliseconds, or ms) spent by the processor to complete its operation, and `Results processed` information, which indicates the number of times the processor was invoked during the query.
 
 | Type            | Definition |
 |:--              |:--         |
@@ -142,9 +180,13 @@ Result processors form a powerful pipeline in Redis Query Engine. They work in s
 | `Scorer`          | The `Scorer` processor assigns a relevance score to each document based on the query’s specified scoring function. This function could involve factors like term frequency, inverse document frequency, or other weighted metrics. |
 | `Sorter`          | The `Sorter` processor arranges the query results based on a specified sorting criterion. This could be a field value (e.g., sorting by price, date, or another attribute) or by the score assigned during the scoring phase. It operates after documents are fetched and scored, ensuring the results are ordered as required by the query (e.g., ascending or descending order). `Scorer` results will always be present in `FT.SEARCH` profiles. |
 | `Loader`          | The `Loader` processor retrieves the document contents after the results have been sorted and filtered. It ensures that only the fields specified by the query are loaded, which improves efficiency, especially when dealing with large documents where only a few fields are needed. |
+| `Threadsafe-Loader` | The `Threadsafe-Loader` processor safely loads document contents when the query runs in a background thread (relevant for multi-threaded deployments). It acquires the Redis Global Lock (referred to as GIL) to access document data. Reports an additional `GIL-Time` field representing the time (ms) spent holding the GIL and waiting to acquire it. |
 | `Highlighter`     | The `Highlighter` processor is used to highlight matching terms in the search results. This is especially useful for full-text search applications, where relevant terms are often emphasized in the UI. |
-| `Paginator`       | The `Paginator` processor is responsible for handling pagination by limiting the results to a specific range (e.g., LIMIT 0 10).It trims down the set of results to fit the required pagination window, ensuring efficient memory usage when dealing with large result sets. |
-| `Vector`&nbsp;`Filter`   | For vector searches, the `Vector Filter` processor is sometimes used to pre-process results based on vector similarity thresholds before the main scoring and sorting. |
+| `Pager/Limiter`       | The `Pager/Limiter` processor is responsible for handling pagination by limiting the results to a specific range (e.g., LIMIT 0 10).It trims down the set of results to fit the required pagination window, ensuring efficient memory usage when dealing with large result sets. |
+| `Counter`         | The `Counter` processor counts the total number of matching results. Used when running queries with `LIMIT 0 0` to return only the count. |
+| `Grouper`         | The `Grouper` processor groups results by field values. Used with `GROUPBY` in `FT.AGGREGATE` queries. |
+| `Projector`       | The `Projector` processor applies field transformations. Used with `APPLY` in `FT.AGGREGATE` queries. |
+| `Network` | Collects and merges results from shards. Appears as the root processor in the coordinator's result processor chain for `FT.AGGREGATE` in cluster mode. |
 
 ### Coordinator
 
@@ -206,7 +248,7 @@ Here's an example of running the `FT.PROFILE` command for a compound query.
          2) INTERSECT
          3) Time
          4) "0"
-         5) Counter
+         5) Number of reading operations
          6) (integer) 6
          7) Child iterators
          8)  1) Type
@@ -215,7 +257,7 @@ Here's an example of running the `FT.PROFILE` command for a compound query.
              4) UNION
              5) Time
              6) "0"
-             7) Counter
+             7) Number of reading operations
              8) (integer) 6
              9) Child iterators
             10)  1) Type
@@ -224,7 +266,7 @@ Here's an example of running the `FT.PROFILE` command for a compound query.
                  4) UNION
                  5) Time
                  6) "0"
-                 7) Counter
+                 7) Number of reading operations
                  8) (integer) 4
                  9) Child iterators
                 10)  1) Type
@@ -233,37 +275,37 @@ Here's an example of running the `FT.PROFILE` command for a compound query.
                      4) kids
                      5) Time
                      6) "0"
-                     7) Counter
+                     7) Number of reading operations
                      8) (integer) 4
-                     9) Size
+                     9) Estimated number of matches
                     10) (integer) 4
                 11)  1) Type
                      2) TEXT
                      3) Term
                      4) +kid
                      5) Time
                      6) "0"
-                     7) Counter
+                     7) Number of reading operations
                      8) (integer) 4
-                     9) Size
+                     9) Estimated number of matches
                     10) (integer) 4
             11)  1) Type
                  2) TEXT
                  3) Term
                  4) small
                  5) Time
                  6) "0"
-                 7) Counter
+                 7) Number of reading operations
                  8) (integer) 2
-                 9) Size
+                 9) Estimated number of matches
                 10) (integer) 2
          9)  1) Type
              2) UNION
              3) Query type
              4) TAG
              5) Time
              6) "0"
-             7) Counter
+             7) Number of reading operations
              8) (integer) 6
              9) Child iterators
             10)  1) Type
@@ -272,44 +314,44 @@ Here's an example of running the `FT.PROFILE` command for a compound query.
                  4) new
                  5) Time
                  6) "0"
-                 7) Counter
+                 7) Number of reading operations
                  8) (integer) 4
-                 9) Size
+                 9) Estimated number of matches
                 10) (integer) 10
             11)  1) Type
                  2) TAG
                  3) Term
                  4) used
                  5) Time
                  6) "0"
-                 7) Counter
+                 7) Number of reading operations
                  8) (integer) 4
-                 9) Size
+                 9) Estimated number of matches
                 10) (integer) 8
    6) 1) Result processors profile
       2) 1) Type
          2) Index
          3) Time
          4) "0"
-         5) Counter
+         5) Results processed
          6) (integer) 3
       3) 1) Type
          2) Scorer
          3) Time
          4) "0"
-         5) Counter
+         5) Results processed
          6) (integer) 3
       4) 1) Type
          2) Sorter
          3) Time
          4) "0"
-         5) Counter
+         5) Results processed
          6) (integer) 3
       5) 1) Type
          2) Loader
          3) Time
          4) "0"
-         5) Counter
+         5) Results processed
          6) (integer) 3
 {{< / highlight >}}
 </details>
@@ -366,32 +408,32 @@ Here's an example of running the `FT.PROFILE` command for a vector query.
          2) VECTOR
          3) Time
          4) "0"
-         5) Counter
+         5) Number of reading operations
          6) (integer) 3
    6) 1) Result processors profile
       2) 1) Type
          2) Index
          3) Time
          4) "0"
-         5) Counter
+         5) Results processed
          6) (integer) 3
       3) 1) Type
          2) Metrics Applier
          3) Time
          4) "0"
-         5) Counter
+         5) Results processed
          6) (integer) 3
       4) 1) Type
          2) Sorter
          3) Time
          4) "0"
-         5) Counter
+         5) Results processed
          6) (integer) 3
       5) 1) Type
          2) Loader
          3) Time
          4) "0"
-         5) Counter
+         5) Results processed
          6) (integer) 3
 {{< /highlight >}}
 </details>