Skip to content
docsp
On this page

Symbol index

Link code in the docs to the symbols a compiler resolved, from indexes your project's CI produces.

The symbol index connects the docs to the code. With it, every name in a code excerpt links to the definition the compiler resolved, every symbol gets a page with its signature, documentation, definition and references, and a guide page can declare which symbols it explains.

docsp never runs a compiler. Your project's CI builds the code, runs the indexers and hands docsp their output. docsp reads those files, checks them against the source checkout, and fails the build when they do not match.

Inputs

Index type Produced by Gives
scip A SCIP indexer: scip-clang for C and C++, rust-analyzer scip for Rust Definitions, references, kinds, signatures, documentation.
nm nm -P on each object file and archive of the build Links between files across languages, through the linker: which object uses a name another object defines.
tla The SANY XML exporter of the TLA+ tools (tla2sany.xml.XMLExporter) Definitions and references in TLA+ specifications.

Other SCIP indexers, such as scip-go and scip-python, write the same format and are read by the same decoder. Their symbols get ids by the generic rule below. They have not been tested with real projects yet.

Typical CI commands, run in the build environment of the code (check each indexer's documentation for the options of your version):

Terminal
$ rust-analyzer scip .
$ scip-clang --compdb-path=build/compile_commands.json
$ mkdir -p nm && cd build && find . -name '*.o' -o -name '*.a' | while read -r f; do nm -P "$f" > "../nm/$(echo "${f#./}" | sed 's|/|__|g').nm"; done

The first two write index.scip. The nm loop writes one file per object or archive, named after its path in the build tree with / replaced by __, which is the naming docsp expects.

Configuration

JSONproject.json
{
  "symbols": {
    "repo": "https://github.com/acme/engine",
    "commit": "3f1c2b9e8d7a6f5e4d3c2b1a0f9e8d7c6b5a4f3e",
    "sourceRoot": "../../engine",
    "strict": true,
    "indexes": [
      { "type": "scip", "paths": ["index/rust.scip", "index/cpp.scip"], "pathPrefix": "native" },
      { "type": "nm", "paths": ["index/nm"], "pathPrefix": "native" },
      { "type": "tla", "paths": ["index/tla"], "pathPrefix": "specs" }
    ]
  }
}
Field Effect
repo Repository URL, used in source links.
commit Full hash of the commit the indexes were made from.
sourceRoot Top folder of the source checkout, relative to the project folder.
sourceLink Link template. Default {repo}/blob/{commit}/{path}#L{line}, which fits GitHub. Links to a whole file drop the part from the last #, ? or & before {line}.
strict true makes an ambiguous name in a hand-written code block an error instead of a warning.
indexes[].type scip, nm or tla.
indexes[].paths Index files or folders, relative to the project folder. A folder contributes its .scip, .nm or .xml files.
indexes[].pathPrefix Prepended to the paths found in the index to make repository paths: the SCIP project root; for nm, the source folder that the build tree mirrors; the folder of the TLA+ modules.

Checks against the source

The build stops with an error when:

  • sourceRoot is not the top folder of a git checkout;
  • the checkout is not at commit;
  • a file docsp reads is not tracked at that commit, or has uncommitted changes;
  • a path from an index or a snippet leaves the checkout;
  • the text at a definition in the index is not the symbol's name (its display name, or the last part of its SCIP symbol when the index gives none). This catches an index made from another commit or a wrong pathPrefix.

The build log reports how many definitions were checked and why any were not, for example whole-file module definitions that span several lines, or TLA+ infix and prefix operators that do not start with their name. A SCIP document that does not say how columns are counted is read as UTF-8 bytes, and the log says so.

commit: "auto"

"commit": "auto" takes the checkout's current HEAD and skips the commit, tracked-file and uncommitted-change checks. TLA+ definitions that do not match are left out instead of failing. It exists for example sites that index their own repository, such as the spark project in the docsp repository. Do not use it for real projects: the links would point at a commit that may not match the indexes.

One project has one commit. A ::snippet on a page of an old docs version reads the same source as the current version.

Symbol ids and URLs

A symbol id is <lang>/<qualified name>, without package manager, package name or version, so ids survive releases.

Language Rule Example
Rust Starts with the crate name: the package name with - as _, or the target name for tests and binaries. Impl blocks read as Type or <Type as Trait>. rust/engine::queue::Queue::push
C, C++ The SCIP descriptors joined with ::. A method disambiguator from the indexer (C++ overloads) stays as name(disambiguator). cpp/engine::Event::emit
TLA+ tla/<Module>!<name> tla/Queue!Enqueue
Other SCIP languages The language of the SCIP document in lower case, then the descriptors joined with ::.

Local symbols (parameters, local variables) have no id.

Every symbol defined in the index gets a page at /<project>/sym/<lang>/<path>/, where <path> is the qualified name with :: and ! as /, and other characters outside A-Z a-z 0-9 _ . ~ - as -. /<project>/sym/ lists every symbol by file. Two ids that would share a URL fail the build.

A symbol page shows the kind, signature and documentation, the definition and its references as source links at the commit, the page that explains the symbol, and, from nm, the files whose objects use it. The source of the definition and of the first eight references is rendered in the page with symbol links, in an expandable block with a link to the code host.

Symbol pages are in the search index under Symbols. A symbol whose name is exactly the query comes first.

Claiming symbols

A guide that explains a symbol lists it in its frontmatter:

YAML
---
title: The job queue
symbols: [rust/engine::queue::Queue, rust/engine::queue::Queue::push]
---

Links to a claimed symbol then open the guide instead of the symbol page, and the symbol page links back to it. One page per version may claim a symbol. An id that is not in the index fails the build, with a suggestion when a symbol of the same name exists.

In ::snippet blocks, every token covered by an occurrence in the index links to the page that claims the symbol, otherwise to its symbol page. Symbols from outside the index (the standard library, dependencies) have no link. This is exact: it uses the compiler's resolution.

In hand-written fenced blocks of a language that the index covers, docsp falls back to names. A name links when exactly one function, method, type or constant of that language has it. Fields, variables and locals never match. Comments and strings are skipped.

Warning:

The name fallback only sees names. x.close() links to the only close in the index even when x is of a type the index does not cover, and a parameter named like a function links to that function. A name that matches several symbols is not linked and gives a warning, or an error with strict. Where precision matters, use ::snippet.

A name defined in one object and undefined (U) in another links the second object's source file to the symbol, when the definition is:

  • a Rust function marked #[no_mangle] (or #[unsafe(no_mangle)]) with that name, or
  • a C or C++ symbol with that name defined in the source file of the defining object.

Strong definitions (T, D, R, B, G, S, i, u) win over weak and common ones (W, V, C). Two strong definitions of a linked name fail the build, since a linker would refuse them.

Object files map to sources by their name. docsp turns __ back into /, drops CMakeFiles/<target>.dir/ and .o (or .obj), then prepends pathPrefix. With "pathPrefix": "native":

Text
CMakeFiles__app.dir__src__event.cpp.o.nm  ->  native/src/event.cpp

A mapped source missing from the commit fails the build.

Current limits

  • ELF only. Mach-O nm prefixes every name with _, which docsp does not strip, so names do not match.
  • Archive members. A member of an archive maps to a source only when exactly one file in the commit has its name without .o. References from other members (Rust codegen units, duplicate file names, and every member with commit: "auto") are left out.
  • C++ is not tested on real data yet. The C and C++ path is covered by unit tests with synthetic indexes.
  • Other SCIP languages load with generic ids and are untested.
  • TLA+ highlighting uses a small grammar shipped with docsp.

packages/docsp/scripts/scip-stats.mjs <file.scip> prints the counts docsp decodes from an index, to compare with other readers.

Try it

The spark project in examples/site/ of the docsp repository uses a synthetic SCIP index of a small Rust library, nm output for a C program that calls it, and a TLA+ specification. Build the example with pnpm build and open /spark/guides/counter/.

↑ ↓ to move↵ to openSearching docsp