Article 2 of 2 in the series How to write company-mode backends
By Austin Bingham

In the first article in this series we looked at how to define the simplest company-mode backend. [1] This backend drew completion candidates from a predefined list of options, and allowed you to do completion in buffers in fundamental mode. The main purpose of that article was to introduce the essential plumbing of a company-mode backend.

In this article we'll expand upon the work of the first, adding some useful UI elements like annotations and metadata. We'll also implement a rough form of fuzzy matching, wherein candidates will be presented to the user when they mostly match the prefix. After this article you'll know almost everything you need to know about writing company-mode backends, and you'll be in a great position to learn the rest on your own.

Most of what we'll be doing in the article revolves around handling completion candidate "metadata", data associated in some way with our completion candidates. In practice this kind of data covers things like documentation strings, function signatures, symbols types, and so forth, but for our purposes we'll simply associate some biographical data with the names in our completion set sample-completions.

company-mode provides affordances for displaying metadata as part of the completion process. For example, if your backend is showing completions for function names, you could display the currently-selected function's signature in the echo area. We'll develop a backend that displays a sentence about the selected candidate in the echo area, and we'll also display their initials as an annotation in the candidate selection popup menu.

Adding more data to our completion candidates

First we need to add some metadata to our existing completion candidates. To do this we'll use Emacs text properties. ((Text properties allow you to associate arbitrary data with strings. You can read about them here. Specifically, we use the special read syntax for text properties.)) For each completion candidate we define an :initials property containing their initials and a :summary property containing a one-sentence summary of the candidate. [2] To add these properties, update sample-completions to look like this:

(defconst sample-completions
  '(#("alan" 0 1
      (:initials
      "AMT"
      :summary
      (concat "Alan Mathison Turing, OBE, FRS (/ˈtjʊərɪŋ/ "
              "tewr-ing; 23 June 1912 – 7 June 1954) was a "
              "British mathematician, logician, cryptanalyst, "
              "philosopher, pioneering computer scientist, "
              "mathematical biologist, and marathon and ultra "
              "distance runner.")))
    #("john" 0 1
      (:initials
      "JVN"
      :summary
      (concat "John von Neumann (/vɒn ˈnɔɪmən/; December 28, "
              "1903 – February 8, 1957) was a Hungarian and "
              "American pure and applied mathematician, physicist, "
              "inventor and polymath.")))
    #("ada" 0 1
      (:initials
      "AAK"
      :summary
      (concat "Augusta Ada King, Countess of Lovelace (10 December "
              "1815 – 27 November 1852), born Augusta Ada Byron "
              "and now commonly known as Ada Lovelace, was an "
              "English mathematician and writer chiefly known for "
              "her work on Charles Babbage's early mechanical "
              "general-purpose computer, the Analytical Engine.")))
    #("don" 0 1
      (:initials
      "DEK"
      :summary
      (concat "Donald Ervin Knuth (/kəˈnuːθ/[1] kə-nooth; born "
              "January 10, 1938) is an American computer "
              "scientist, mathematician, and Professor Emeritus "
              "at Stanford University.")))))

Attaching properties like this is a very convenient way to store metadata for completion candidates. Of course in a real backend you probably wouldn't have a hard-coded list of candidates, and you'd be fetching them dynamically from a server, database, or external process. In that case, you'd need to also dynamically fetch the metadata you want and attach it to the candidate strings you serve through your backend. In the end, text properties work well in this context because they transparently transport the metadata - which company-mode doesn't know about - with the completion strings that company-mode definitely knows about.

Adding completion menu annotations

This change by itself doesn't really do anything, of course. All we've done is add properties to some strings, and we need to instruct company-mode on how to actually use them for display. The first way we'll use this metadata, then, is to add a small annotation to each entry in the popup menu used for candidate selection. To add this annotation, we need to update company-sample-backend to respond to the annotation command. This command should resolve to the annotation you want to use for the given candidate. Typically this means calling a function taking the completion candidate string arg and returning the annotation string.

First let's define a function that takes a completion candidate string and returns an annotation. Remember that our candidate strings store their metadata as text properties, so fundamentally this function simply needs to extract a property. For the annotation, we'll extract the :initials property and return it (prefixed with a blank.) That function looks like this:

(defun sample-annotation (s)
  (format " [%s]" (get-text-property 0 :initials s)))

Next we need to update our backend to respond to the annotation command like this:

(defun company-sample-backend (command &optional arg &rest ignored)
  (interactive (list 'interactive))``

  (case command
    (interactive (company-begin-backend 'company-sample-backend))
    (prefix (and (eq major-mode 'fundamental-mode)
                (company-grab-symbol)))
    (candidates
    (remove-if-not
      (lambda (c) (string-prefix-p arg c))
      sample-completions))
    (annotation (sample-annotation arg))))

In the last line we tell the backend to call sample-annotation with the candidate string to produce an annotation.

Now when we do completion we see the candidates' initials in the popup menu:

candidates-initials

Displaying metadata in the echo area

Where the annotation command adds a small annotation to the completion popup menu, the meta backend command produces text to display in the echo area. [3] The process for producing the metadata string is almost exactly like that of producing the annotation string. First we write a function that extracts the string from the candidate text properties. Then we wire that function into the backend through the meta command.

As you've probably guessed, the function for extracting the metadata string will simply read the :summary property from a candidate string. It looks like this:

(defun sample-meta (s)
  (get-text-property 0 :summary s))

The changes to the backend look like this:

(defun company-sample-backend (command &optional arg &rest ignored)
  (interactive (list 'interactive))

  (case command
    (interactive (company-begin-backend 'company-sample-backend))
    (prefix (and (eq major-mode 'fundamental-mode)
                (company-grab-symbol)))
    (candidates
    (remove-if-not
      (lambda (c) (string-prefix-p arg c))
      sample-completions))
    (annotation (sample-annotation arg))
    (meta (sample-meta arg))))

As before, in the last line we associate the meta command with our sample-meta function.

Here's how the metadata looks when displayed in the echo area:

Screen Shot 2014-11-03 at 12.02.10 PM

Fuzzy matching

As a final improvement to our backend, let's add support for fuzzy matching. This will let us do completion on prefixes which don't exactly match a candidate, but which are close enough. [4] For our purposes we'll implement a very crude form of fuzzy matching wherein a prefix matches a candidate if the set of letters in the prefix is a subset of the set of letters in the candidate. The function for performing fuzzy matching looks like this:

(defun sample-fuzzy-match (prefix candidate )
  (cl-subsetp (string-to-list prefix)
              (string-to-list candidate)))

Now we just need to modify our backend a bit. First we need to modify our response to the candidates command to use our new fuzzy matcher. Then we need to respond to the no-cache command by returning true. [5] Here's how that looks:

(defun company-sample-backend (command &optional arg &rest ignored)
  (interactive (list 'interactive))

  (case command
    (interactive (company-begin-backend 'company-sample-backend))
    (prefix (and (eq major-mode 'fundamental-mode)
                (company-grab-symbol)))
    (candidates
    (remove-if-not
      (lambda (c) (sample-fuzzy-match arg c))
      sample-completions))
    (annotation (sample-annotation arg))
    (meta (sample-meta arg))
    (no-cache 't)))

As you can see, we've replaced string-prefix-p in the candidates response with sample-fuzzy-match, and we've added (no-cache 't).

Here's how our fuzzy matching looks in action:

Screen Shot 2014-11-03 at 12.23.45 PM

That's all, folks

We've seen how to use Emacs' text properties to attach metadata to candidate strings. This is a really useful technique to use when developing company-mode backends, and one that you'll see used in real-world backends. With that metadata in place, we've also seen that it's very straightforward to tell your backend to display annotations in popup menus and metadata in the echo area. Once you've got the basic techniques under your belt, you can display anything you want as part of completion.

There are still more aspects to developing company-mode backends, but with what we've covered in this series you can get very far. More importantly, you know the main concepts and infrastructure for the backends, so you can learn the rest on your own. If you want to delve into all of the gory details, you'll need to read the company-mode source code, and specifically the documentation for company-backends. [6]

For an example of a fairly full-featured backend implementation that's currently in use (and under active development), you can see the emacs-ycmd project. [7] Happy hacking!

[1]The first article in this series..
[2]The summaries are simply the first sentences of the respective Wikipedia articles.
[3]The Emacs manual entry on the echo area.
[4]Fuzzy matching is commonly used for completion tools because it addresses common cases where users transpose characters, accidentally leave characters out, or consciously leverage fuzzy matching for increased speed.
[5]The details of why this is the case are murky, but the company-mode source code specifically states this. The same source also says that we technically should be implementing a response to match, but that doesn't seem to affect this implementation.
[6]The company-mode project page.
[7]The emacs-ycmd project is on github. In particular, see company-ycmd.el.
Category: Programming
Tags: Emacs