Speeding Up Search on Big WordPress Sites with Elasticsearch

Speeding Up Search on Big WordPress Sites with Elasticsearch

Large sites with a lot of content can easily get bogged down when people try to find content.

Every time someone uses the “Search” box on your site, they’re using MySQL queries to the database, eating up resources and slowing everything down.

Check out our post 27 Solutions to 14 Forgotten WordPress Page Speed Problems for more on that.

Fortunately, installing a dedicated search engine like Elasticsearch can speed up your site, and allow your visitors to quickly search for and find the awesome content that might otherwise be hidden away on your site.

In this article, we’ll look at how to set up an Elasticsearch server, and two WordPress plugins that can help you manage Elasticsearch from the WordPress admin panel: ElasticPress from 10up, and Fantastic Elasticsearch.

Want to learn more? Check out “27 Solutions to 14 Forgotten WordPress Page Speed Problems.”Read more

Why Replace WordPress Search?

The built-in search function in WordPress is really just a direct query into the MySQL database. It works fine for a site with 100 posts of a few hundred words, but if your site is posting a dozen articles a day on a variety of topics, searching for a single word or short phrase in a database table will take time.

Your site may not be the same size as a globe-spanning news service with dozens of contributors, but it may be large enough where time matters. Time your site visitors don’t want to waste.

Why Choose Elasticsearch?

Elasticsearch is a popular dedicated search tool used by giants like the Wikimedia Foundation, Facebook, Mozilla and Netflix. A site that runs on people’s questions, Quora, also uses Elasticsearch.

How Elasticsearch works. Image credit: Software Engineering Daily.

Elasticsearch simplifies and enhances the Apache Lucene search engine. As with most software that simplifies things for the user, there’s a lot of complexity behind it. The speed comes from the underlying structure of the system. All your content gets stored in “nodes” that can sit on any number of different servers.

As the size of the site’s content grows, nodes can then be stored into clusters. At the other end, content in a node gets an index composed of “shards” of content. When someone searches your site, Elasticsearch checks the index for the search term, not the entire database each time.

Identifying an Elasticsearch Server Host

ElasticPress needs a server to store its information so before you install any plugins you should put an Elasticsearch server in place. This may be the most difficult piece of the puzzle.

Check with your web host first to see if they support Elasticsearch. Most Linux distributions include an Elasticsearch package.

You can get a Docker image for Elasticsearch too. See Building WordPress on Docker With Windows, Linux and OS X to learn how to set up Docker.

You can also choose from a variety of cloud solutions. Amazon Web Services offers up to 750 hours per month of Elasticsearch services for free. QBox’s basic plan offers an hourly search plan for a little over US 5 cents.

Lastly, if you have a gigantic site with lots of traffic, 10up wants you to know they have ElasticPress.io to serve your needs. This requires a fairly big monthly fee, but may well be worth the expense.

Installing Elasticsearch 5.3

If you host your own instance of Elasticsearch, you will need to install the system on your server.

The engine components are written in Java, so you will need a 64-bit Java Development Kit (JDK), version 8 or later. You can get the get the latest official distribution here. OpenJDK is an open source option that will also work:

You will need the bulkier Java Development Kit because it comes with a Java Virtual Machine (JVM) that the more common Java Runtime Environment (JRE) lacks. Elasticsearch does all its work inside the JVM.

Elasticsearch is actually a package of these components, known collectively as the Elastic Stack:

  • Logstash: collects the data
  • Elasticsearch: The basic search engine
  • Beats: transports the search data
  • Kibana: The user interface that manages the system
  • Elasticsearch Hadoop: the engine that manages Big Data
  • X-Pack: A bunch of premium services from Elasti.co. Basic Monitoring is free of charge, but Security and other stuff will cost you.

Fortunately, all of these come in a single package, either as an archive (.zip or tar.gz) or Linux installation package (RPM or DEB).

Next, get Elasticsearch here.

Elasticsearch version 5.6 introduced a graphical installer for Windows that greatly simplifies the install. Sadly, at the time of this writing the ElasticPress plugin only supports Elasticsearch 5.3 and earlier. So you will have to use the command line installer to get Elasticsearch working with your WordPress site.

The following steps will install Elasticsearch from the Windows command line. The Linux packages should manage this for you.

Use your favorite archive tool (PKZip, 7Zip, WinZip and the like) to extract the Elasticsearch pack to a directory you have access to on the server volume you want to use. This will be the Home or installation directory. Be aware that the volume you choose must have at least 15% of its total space free to successfully install Elasticsearch.

Open a command line and point it to the installation directory. Type this command:

.\bin\Elasticsearch.bat

The installer will begin configuring the volume and identify an IP address for a localhost environment (likely 127.0.0.1:9200)

To run Elasticsearch from the Windows command line, type the same Elasticsearch.bat command you used to install.

To confirm it’s running:

curl -XGET 'localhost:9200/?pretty'

To install the Elasticsearch service, so it runs in the background, type:

Elasticsearch-service install

Start the service with:

Elasticsearch-service start

The Elasticsearch online documentation is very helpful if you run into problems.

Installing ElasticPress

Next up is installing ElasticPress. Use the standard process to install and activate ElasticPress. When you’re done, the plugin leaves a message at the top of the Plugin admin page indicating that you have a “quick setup process” to complete before your new search engine is running on your site.

That setup process is just pointing ElasticPress to your Elasticsearch server. The plugin will connect Elasticsearch to your WordPress database in the background. Get this from your searching host, if you didn’t install the package yourself.

Otherwise, use the IP address for localhost (127.0.0.1:9200) that the installer identified in the last section. Your search capabilities will now use Elasticsearch.

Enter your host server’s IP address to start ElasticPress.

Using Fantastic Elasticsearch Plugin

This plugin is a useful alternative to ElasticPress, allowing you a bit more flexibility on when to use Elasticsearch instead of standard WordPress search. If you’re willing to trade off some speed for cost reductions for cloud server space and traffic, Fantastic ES is the way to go.

Once you’ve connected your site to the Elasticsearch server, identify the areas of your site that get the most search traffic. Then use the plugin to finetune what content you want to use this tool with. Go to Settings > ElasticSearch > WordPress Integration. You can choose specific category archives, custom post types, and tags.

Fantastic Elasticsearch lets you choose from a variety of categories and post types.

If you’re a search expert, there are many other settings that Fantastic Elasticsearch can configure for you.

Conclusion

Elasticsearch is an excellent solution to the problem of topic searches taking a long time to complete. Depending on your comfort level with storing data in the cloud, and finding an affordable solution for your storage, both ElasticPress and Fantastic Elasticsearch can simplify setup and management of Elasticsearch in WordPress.

Let’s hope that ElasticPress will soon support the simpler graphical installation of Elasticsearch 5.6.

Michael McCallister
What issues have you had with readers finding your site's content? What's your experience with Elasticsearch? Have we missed your favorite search plugin? Let us know!