Faster JetBrains IDEs with shared indexes

If you develop with IntelliJ IDEA, PyCharm, GoLand, or other JetBrains IDEs, it’s likely you’ve waited for “indexing” to complete after opening a project. While this may be annoying, it’s necessary for IntelliJ and other heavy-weight IDEs to have features such as code search, highligiting, refactoring, and code completion.

Waiting for an IDE to finish indexing a project might not be a big problem for many workflows. After the first load, indexes are cached and subsequent runs are faster. However, indexing time can be a huge blocker for developers, especially in these cases:

• large projects (monorepos, many dependencies, monolithic applications)

• running old/slow machines (indexing is CPU-intensive)

• ephemeral developer workspaces (containers, remote IDEs)

In this post, we’ll cover how shared indexes can significantly reduce IDE load times, share some examples, and a one-line command to generate these for your project. (Historically, shared indexes have been difficult to set up)

How indexing works

Indexing works by traversing the project’s codebase to create a “virtual map” of classes, methods, and objects for future lookups. After the index is generated, it is cached on your device for later use.

Indexing a codebase will likely take the longest the first time you open it on your machine. When the codebase changes, such as pulling code or switching branches, your indexes will “update,” but significantly faster than the first time.

Shared indexes ⚡

Shared indexes make it possible to host pre-generated indexes for others to download, significantly improving loading speeds across your team. These remote indexes work in conjunction with local indexing to ensure your IDE always has up-to-date information on the codebase.

Generating JetBrains shared indexes for your project

JetBrains has a guide for creating shared indexes, but it involves many steps, including downloading custom tooling and uploading indexes to a CDN. It also lacks instructions for automating this process, to generate indexes in CI, for example.

Using a Docker container to generate shared indexes makes it simple to try locally or automate with cron/CI:

cd your_large_codebase/

# generate shared indexes
docker run -it --rm \
  -v "$(pwd)":/var/project \
  -v "$HOME"/indexes-output:/shared-index \
  -e INDEXES_CDN_URL=https://cdn.myserver.com/project \
    -u "$(id -u):$(id -g)" \
  bencdr/indexer:idea-2021.3

After generating indexes, you can upload the output folder to your CDN, or a local server. You can also use shared indexes without a CDN by using a network share or even your local filesystem for testing. Check out my GitHub repo for details: https://github.com/bpmct/jetbrains-indexer

Benchmarking shared indexes

I tested indexing time for some popular projects on my 2019 MacBook Pro. To benchmark your own projects, File → Invalidate Caches in your IDE will allow you to opt in/out of downloading shared indexes to simulate first launching your project.

Project	Language(s)	Local indexing 🐌	With shared indexes ⚡	Improvement %
kubernetes/kubernetes	Go	2m 40s	22s	727%
cdr/code-server	Typescript	2m 30s	34s	441%
Coder internal monorepo	Go & Typescript	3m 20s	32s	625%
jetbrains/intellij-community	Java	6m 30s	2m 15s	288%

These times were averaged across two test runs. Your mileage will vary depending on network speeds, device performance, etc.

Remote development & shared indexes in JetBrains

Recently, JetBrains released remote development support, making it simple to develop from powerful, remote workspaces. On-demand workspaces have a lot of benefits, such as faster onboarding and better reproducibility. However, first-time indexing happens much more frequently, since, after all, workspaces are meant to be ephemeral.

Shared indexes work with Coder, our remote development platform. Coder supports all JetBrains IDEs locally, or via the web browser. If you don’t want to host a CDN for shared indexes, you can include them in the workspace image, so everything loads in a snap ⚡

If you’d like to learn more about Coder, you can request a demo or try it for free.

References

• Talk: Indexing, or How We Made Indexes Shared and Fast
• JetBrains Remote Development
• JetBrains docs: Indexing
• GitHub: idea-shared-index-dockerfile