# Symbol index

Link code in the docs to the symbols a compiler resolved, from indexes your project's CI produces.

The symbol index connects the docs to the code. With it, every name in a code excerpt links to the definition the
compiler resolved, every symbol gets a page with its signature, documentation, definition and references, and a guide
page can declare which symbols it explains.

docsp never runs a compiler. Your project's CI builds the code, runs the indexers and hands docsp their output. docsp
reads those files, checks them against the source checkout, and fails the build when they do not match.

## Inputs

| Index type | Produced by | Gives |
|---|---|---|
| `scip` | A SCIP indexer: `scip-clang` for C and C++, `rust-analyzer scip` for Rust | Definitions, references, kinds, signatures, documentation. |
| `nm` | `nm -P` on each object file and archive of the build | Links between files across languages, through the linker: which object uses a name another object defines. |
| `tla` | The SANY XML exporter of the TLA+ tools (`tla2sany.xml.XMLExporter`) | Definitions and references in TLA+ specifications. |

Other SCIP indexers, such as `scip-go` and `scip-python`, write the same format and are read by the same decoder.
Their symbols get ids by the generic rule below. They have not been tested with real projects yet.

Typical CI commands, run in the build environment of the code (check each indexer's documentation for the options of
your version):

```terminal
$ rust-analyzer scip .
$ scip-clang --compdb-path=build/compile_commands.json
$ mkdir -p nm && cd build && find . -name '*.o' -o -name '*.a' | while read -r f; do nm -P "$f" > "../nm/$(echo "${f#./}" | sed 's|/|__|g').nm"; done
```

The first two write `index.scip`. The `nm` loop writes one file per object or archive, named after its path in the
build tree with `/` replaced by `__`, which is the naming docsp expects.

## Configuration

```json title="project.json"
{
  "symbols": {
    "repo": "https://github.com/acme/engine",
    "commit": "3f1c2b9e8d7a6f5e4d3c2b1a0f9e8d7c6b5a4f3e",
    "sourceRoot": "../../engine",
    "strict": true,
    "indexes": [
      { "type": "scip", "paths": ["index/rust.scip", "index/cpp.scip"], "pathPrefix": "native" },
      { "type": "nm", "paths": ["index/nm"], "pathPrefix": "native" },
      { "type": "tla", "paths": ["index/tla"], "pathPrefix": "specs" }
    ]
  }
}
```

| Field | Effect |
|---|---|
| `repo` | Repository URL, used in source links. |
| `commit` | Full hash of the commit the indexes were made from. |
| `sourceRoot` | Top folder of the source checkout, relative to the project folder. |
| `sourceLink` | Link template. Default `{repo}/blob/{commit}/{path}#L{line}`, which fits GitHub. Links to a whole file drop the part from the last `#`, `?` or `&` before `{line}`. |
| `strict` | `true` makes an ambiguous name in a hand-written code block an error instead of a warning. |
| `indexes[].type` | `scip`, `nm` or `tla`. |
| `indexes[].paths` | Index files or folders, relative to the project folder. A folder contributes its `.scip`, `.nm` or `.xml` files. |
| `indexes[].pathPrefix` | Prepended to the paths found in the index to make repository paths: the SCIP project root; for `nm`, the source folder that the build tree mirrors; the folder of the TLA+ modules. |

## Checks against the source

The build stops with an error when:

- `sourceRoot` is not the top folder of a git checkout;
- the checkout is not at `commit`;
- a file docsp reads is not tracked at that commit, or has uncommitted changes;
- a path from an index or a snippet leaves the checkout;
- the text at a definition in the index is not the symbol's name (its display name, or the last part of its SCIP
  symbol when the index gives none). This catches an index made from another commit or a wrong `pathPrefix`.

The build log reports how many definitions were checked and why any were not, for example whole-file module
definitions that span several lines, or TLA+ infix and prefix operators that do not start with their name. A SCIP
document that does not say how columns are counted is read as UTF-8 bytes, and the log says so.

### commit: "auto"

`"commit": "auto"` takes the checkout's current HEAD and skips the commit, tracked-file and uncommitted-change
checks. TLA+ definitions that do not match are left out instead of failing. It exists for example sites that index
their own repository, such as the `spark` project in the docsp repository. Do not use it for real projects: the
links would point at a commit that may not match the indexes.

One project has one commit. A `::snippet` on a page of an old docs version reads the same source as the current
version.

## Symbol ids and URLs

A symbol id is `<lang>/<qualified name>`, without package manager, package name or version, so ids survive releases.

| Language | Rule | Example |
|---|---|---|
| Rust | Starts with the crate name: the package name with `-` as `_`, or the target name for tests and binaries. Impl blocks read as `Type` or `<Type as Trait>`. | `rust/engine::queue::Queue::push` |
| C, C++ | The SCIP descriptors joined with `::`. A method disambiguator from the indexer (C++ overloads) stays as `name(disambiguator)`. | `cpp/engine::Event::emit` |
| TLA+ | `tla/<Module>!<name>` | `tla/Queue!Enqueue` |
| Other SCIP languages | The language of the SCIP document in lower case, then the descriptors joined with `::`. | |

Local symbols (parameters, local variables) have no id.

Every symbol defined in the index gets a page at `/<project>/sym/<lang>/<path>/`, where `<path>` is the qualified
name with `::` and `!` as `/`, and other characters outside `A-Z a-z 0-9 _ . ~ -` as `-`. `/<project>/sym/` lists every symbol by file. Two ids that would share a URL fail the
build.

A symbol page shows the kind, signature and documentation, the definition and its references as source links at the
commit, the page that explains the symbol, and, from `nm`, the files whose objects use it. The source of the
definition and of the first eight references is rendered in the page with symbol links, in an expandable block with
a link to the code host.

Symbol pages are in the search index under Symbols. A symbol whose name is exactly the query comes first.

## Claiming symbols

A guide that explains a symbol lists it in its frontmatter:

```yaml
---
title: The job queue
symbols: [rust/engine::queue::Queue, rust/engine::queue::Queue::push]
---
```

Links to a claimed symbol then open the guide instead of the symbol page, and the symbol page links back to it. One
page per version may claim a symbol. An id that is not in the index fails the build, with a suggestion when a symbol
of the same name exists.

## How code gets its links

**In `::snippet` blocks**, every token covered by an occurrence in the index links to the page that claims the
symbol, otherwise to its symbol page. Symbols from outside the index (the standard library, dependencies) have no
link. This is exact: it uses the compiler's resolution.

**In hand-written fenced blocks** of a language that the index covers, docsp falls back to names. A name links when
exactly one function, method, type or constant of that language has it. Fields, variables and locals never match.
Comments and strings are skipped.

:::warning[The name fallback only sees names.]
`x.close()` links to the only `close` in the index even when `x` is of a type the index does not cover, and a
parameter named like a function links to that function. A name that matches several symbols is not linked and gives a
warning, or an error with `strict`. Where precision matters, use `::snippet`.
:::

## nm and linker links

A name defined in one object and undefined (`U`) in another links the second object's source file to the symbol,
when the definition is:

- a Rust function marked `#[no_mangle]` (or `#[unsafe(no_mangle)]`) with that name, or
- a C or C++ symbol with that name defined in the source file of the defining object.

Strong definitions (`T`, `D`, `R`, `B`, `G`, `S`, `i`, `u`) win over weak and common ones (`W`, `V`, `C`). Two strong
definitions of a linked name fail the build, since a linker would refuse them.

Object files map to sources by their name. docsp turns `__` back into `/`, drops `CMakeFiles/<target>.dir/` and `.o`
(or `.obj`), then prepends `pathPrefix`. With `"pathPrefix": "native"`:

```text
CMakeFiles__app.dir__src__event.cpp.o.nm  ->  native/src/event.cpp
```

A mapped source missing from the commit fails the build.

## Current limits

- **ELF only.** Mach-O `nm` prefixes every name with `_`, which docsp does not strip, so names do not match.
- **Archive members.** A member of an archive maps to a source only when exactly one file in the commit has its name
  without `.o`. References from other members (Rust codegen units, duplicate file names, and every member with
  `commit: "auto"`) are left out.
- **C++ is not tested on real data yet.** The C and C++ path is covered by unit tests with synthetic indexes.
- **Other SCIP languages** load with generic ids and are untested.
- **TLA+ highlighting** uses a small grammar shipped with docsp.

`packages/docsp/scripts/scip-stats.mjs <file.scip>` prints the counts docsp decodes from an index, to compare with
other readers.

## Try it

The `spark` project in `examples/site/` of the docsp repository uses a synthetic SCIP index of a small Rust library,
`nm` output for a C program that calls it, and a TLA+ specification. Build the example with `pnpm build` and open
`/spark/guides/counter/`.
