Search index

Generate a search index, to be used by the JavaScript search widget in a Jekyll site. It also adds a tag which can be used to add the search widget to your site, optionally allowing the Jekyll site to specify a template for each search result.

http://search-fdroid-website-pserwylo.surge.sh/fdroid-website/

The (relatively small) website changes required to support this are at pserwylo/fdroid-website/search.

Technologies:

• lunr.js is the search library which is used.
  • jekyll-fdroid uses lunr to generate an index of each package name + summary.
  • This index is serialized and combined with the original list of package information (name + summary + icon + packageName).
  • The index is loaded via AJAX when the website is viewed.
  • The generation of the index is done in Ruby, but it embeds the v8 JavaScript engine. This is done with MIT licensed code from jekyll-lunr-js-search which has been adapted for our requirements (i.e. indexing package names and summaries, instead of arbitrary pages in a Jekyll site).
• Awesomplete is the library used to build the drop-down/autocomplete widget for searching.
  • Chose it because it doesn't have a dependency on jQuery, and seems well supported and does what we require.
  • It is a bit opinionated about the style applied to the dropdown, but I'm sure we could override if it got annoying.
• Mustache.js is used to render templates in JavaScript.
  • We are unable to use Liquid, because that is for Ruby and build time.
  • However the results list is generated at runtime via JavaScript.
  • Mustache is a commonly used and lightweight templating engine, with similar markup to Liquid.
  • jekyll-fdroid contains a default template which is what is used by the F-Droid website, but it can be overriden.

Requirements:

• mod_deflate on the Apache2 server.
  • Without this, we are serving an index which is 746KiB large. With it, we are serving 193.5KiB.
  • For example, see this compression test for my surge.sh site
  • Note that gitlab.io doesn't use compression and so it serves the large index. This is noticeable when viewing the home page.
  • Thankfully the index is cached by the browser, so it is only slow on the first load.

Limitations:

• English language stemmer
  • Stemmers are the code which truncates "Running", "Runner", "Runners" all to "Run", so that we don't need to keep a separate entry for each, and also so that searching for "Runner" will return results for "Running".
  • There are multi-lingual stemmers available in the lunr-languages plugins, but only for a selection of languages.

Future improvements:

• Include multi-lingual indexes (see the discussion on stemmers under "Limitations" above.
• Benchmark how much more space is required to index the descriptions too.
  • I suspect it may start to make the index size quite a bit larger and may not be worth it though.