Building a Full Stack AutoComplete Widget with Java and Kendo UI

Anyone that has done a web search is familiar with autocomplete behavior. As you type characters, the search engine suggests several - potentially hilarious - complete queries. While autocomplete controls are common, actually developing them has a number of non-trivial technical challenges.

Where do you get the options from? Do you specify them on the client or load them server-side? Do you load all the options at once or as the user types?

Kendo UI's AutoComplete widget gives you the hooks to make all these implementations possible. In this article we'll build a few Java-backed AutoComplete widgets to explore the best way to structure the autocomplete controls you need to build. We'll start with a small hardcoded list, and scale all the way up a server filtered list with a million options.

Note: The finished examples are available in this GitHub repository if you'd like to follow along.

Getting Started

Let's start with the basics. The simplest Kendo UI AutoComplete you can build takes a hardcoded list of options in JavaScript directly.

<input id="autocomplete">
<script>
    $( "#autocomplete" ).kendoAutoComplete({
        dataSource: [ "One", "Two", "Three" ]
    });
</script>

If you prefer the (excuse the pun) autocompletion and validation niceties that Kendo UI's JSP wrappers provide, you can use them to hardcode options as well.

<kendo:autoComplete name="autocomplete">
    <kendo:dataSource data="<%= new String[]{ "One", "Two", "Three" } %>"></kendo:dataSource>
</kendo:autoComplete>

Note: When using the JSP wrappers, make sure you have Kendo UI's jar on your classpath and have the taglib imported correctly. If you're confused by this, checkout our getting started documentation that walks you through the setup.

The rest of this example will use the JSP wrapper syntax, but keep in mind that everything can be done in JavaScript directly if that's your preference. To see the JavaScript syntax to use, refer to AutoComplete's API documentation.

Ramping Up The Options

While hardcoding the options works well for very simple cases, it's not practical or maintainable for large lists.

To show a bigger and more realistic use case, let's build an autocomplete for selecting countries (United States, Canada, etc). Since there are ~200 countries, hardcoding the full list in a single JavaScript or JSP file would be a mess.

To implement this, we'll start by altering our data source. Instead of hardcoding options, we'll use the read option to specify a server-side endpoint to retrieve the data from.

<kendo:autoComplete name="country">
    <kendo:dataSource>
        <kendo:dataSource-transport read="/App/countries"/>
    </kendo:dataSource>
</kendo:autoComplete>

Here, once the user types a single character, Kendo UI performs an AJAX request to /App/countries and expects a JSON formatted array of strings to be returned.

Therefore, the next step is adding something to listen at /App/countries. For simplicity, we'll implement this with a basic Servlet.

Note: You can absolutely implement this with a more robust MVC library such as Struts or Spring. I'm sticking with a Servlet here since it's universal.

First, we'll add the following to the app's deployment descriptor (web.xml).

<servlet>
    <servlet-name>CountryServlet</servlet-name>
    <servlet-class>com.countries.CountryServlet</servlet-class>
</servlet>
<servlet-mapping>
    <servlet-name>CountryServlet</servlet-name>
    <url-pattern>/countries</url-pattern>
</servlet-mapping>

This tell our app to invoke the Servlet class com.countries.CountryServlet when a request matching the URL pattern /countries is received. Next we have to create com.countries.CountryServlet, which is shown below.

package com.countries;

import com.google.gson.Gson;
import java.io.IOException;
import java.io.PrintWriter;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

public class CountryServlet extends HttpServlet {

    public static final String[] countries = new String[]{
        "Afghanistan",
        "Albania",
        ...
        "Zimbabwe"
    };

    @Override
    protected void doGet(HttpServletRequest request, HttpServletResponse response)
            throws ServletException, IOException {

        response.setContentType("application/json");

        Gson gson = new Gson();
        try (PrintWriter out = response.getWriter()) {
            out.print(gson.toJson(countries));
        }
    }
}

A couple of things to note here:

• Setting the content type to application/json tells the browser that this response is JSON and not HTML, an image, etc.
• Google's gson library is used to serialize a Java object (in this case a String[]) into a formatted JSON string. Any JSON serialization library will work. I went with gson because I find it easy to use and well documented.
• This setup assumes that the application's context root is /App. The context root defaults to the name of the project.

And that's it. With this approach the autocomplete does not load any data until the user starts typing. Once the user types a single character, the widget makes a GET AJAX request to retrieve all countries from /App/countries. When the AJAX call completes, the widget filters through the options client side as the user types. This workflow is shown below.

The key here is: although the data is returned by the server, the filtering is done on the client. Since there are only ~200 options here this isn't a problem - JavaScript can loop over 200 options quickly.

But what if we had more? Filtering through hundreds of options on the client is fine, but what if there were thousands, tens of thousands, or even - as we're about to see - MILLIONS.

REALLY Ramping Up The Options

For our next example we're going to go big by building an autocomplete for the top million sites on the web. Where are we going to get the data from? Alexa, a web metrics site, conveniently offers a CSV of the top million sites on the web that is updated daily.

We'll start by placing this CSV file in a data directory in our application's WebContent.

Next, we'll setup the same structure we used for the domain autocomplete. First a kendo:autoComplete tag that invokes /App/sites.

<kendo:autoComplete name="site">
    <kendo:dataSource serverFiltering="true">
        <kendo:dataSource-transport read="/App/sites"/>
    </kendo:dataSource>
</kendo:autoComplete>

Then a Servlet to listen at /App/sites.

<servlet>
    <servlet-name>SiteServlet</servlet-name>
    <servlet-class>com.sites.SiteServlet</servlet-class>
</servlet>
<servlet-mapping>
    <servlet-name>SiteServlet</servlet-name>
    <url-pattern>/sites</url-pattern>
</servlet-mapping>

This Servlet is more complex than our country Servlet; it must load data from the CSV file instead of returning a hardcoded list. To clean up our code, we'll abstract the CSV parsing into a separate SiteService class. With this difference in mind, the Servlet looks pretty similar to our previous example. It returns a JSON serialized array of Strings to show in the autocomplete.

package com.sites;

import com.google.gson.Gson;
import java.io.IOException;
import java.io.PrintWriter;
import java.util.List;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

public class SiteServlet extends HttpServlet {

    @Override
    protected void doGet(HttpServletRequest request, HttpServletResponse response)
            throws ServletException, IOException {
        List<String> sites = SiteService.load();

        response.setContentType("application/json");

        Gson gson = new Gson();
        try (PrintWriter out = response.getWriter()) {
            out.print(gson.toJson(sites));
        }
    }

    @Override
    public void init() throws ServletException {       
        String path = this.getServletContext().getRealPath("data/top-1m.csv");
        SiteService.build(path);
    }
}

The init() call gets a reference to our CSV file and calls SiteService.build() to parse it. The implementation of SiteService is shown below. Don't worry about the details here; we're concerned with building an autocomplete and not necessarily how to parse a CSV file in Java. Just know that at the end of build(), the sites variable contains a List full of one million sites as strings.

package com.sites;

import java.io.BufferedReader;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;

public class SiteService {

    public static List<String> sites = new ArrayList();

    public static void build(String path) {
        BufferedReader reader = null;
        String line;

        try {
            reader = new BufferedReader(new FileReader(path));
            while ((line = reader.readLine()) != null) {
                String[] strings = line.split(",");
                sites.add(strings[1] + " (Rank " + strings[0] + ")");
            }
        } catch (FileNotFoundException e) {
            throw new RuntimeException("Could not find file", e);
        } catch (IOException e) {
            throw new RuntimeException("An IO error occurred reading the file.", e);
        } finally {
            if (reader != null) {
                try {
                    reader.close();
                } catch (IOException e) {
                    throw new RuntimeException("An I/O error occurred closing the file", e);
                }
            }
        }
    }

    public static List<String> load() {
        return sites;
    }
}

This works, but there's one major problem with this example. While this Servlet does return a JSON encoded array of one million sites, it instantly seizes up the browser that requested it. As it turns out, despite its modern processing power, asking JavaScript to parse through one million options is still a bit much.

When the processing becomes this intensive, it makes sense to move the filtering from the client to the server. And Kendo UI makes it easy to do just that.

Moving Filtering To The Server

To move filtering to the server, first we must set the data source's serverFiltering option to true. This tells the widget to treat the data returned from the server as filtered.

<kendo:autoComplete name="site">
    <kendo:dataSource serverFiltering="true">
        <kendo:dataSource-transport read="/App/sites"/>
        </kendo:dataSource-transport>
    </kendo:dataSource>
</kendo:autoComplete>

Now, the widget will send a whole bunch of information to the server as the user types. For example the following request parameters are sent after the user types an "a".

These filters tell the server how to filter data - in this case, to get items that start with an "a" in a case insensitive manner. This information is great for more complex widgets - such as the a grid - but it's overkill for a simple widget like autocomplete. All the server needs to know is what the user typed.

Fortunately, we can configure how data is sent to the server using the data source's transport.parameterMap option. Simply put, the parameter map is a JavaScript function that converts Kendo UI's formats into a format more suitable to the server receiving the request. The parameter map we'll use is shown below.

<kendo:autoComplete name="site">
    <kendo:dataSource serverFiltering="true">
        <kendo:dataSource-transport read="/App/sites">
            <kendo:dataSource-transport-parameterMap>
                function( options ) {
                    return { filter: options.filter.filters[ 0 ].value };
                }
            </kendo:dataSource-transport-parameterMap>
        </kendo:dataSource-transport>
    </kendo:dataSource>
</kendo:autoComplete>

Note how options.filter.filters[ 0 ].value corresponds to the query string parameter that contained "a" in our previous example.

This function tells Kendo UI: instead of sending several complex filters, just send a filter parameter set to the string that the user typed.

Now we need to change our Servlet to retrieve this parameter and pass it on to our service. We'll by adding the code below to the Servlet.

String filter = request.getParameter("filter");
List<String> sites = SiteService.load(filter);

Then, we'll change the service's load() method to actually do the filtering.

public static List<String> load(String startswith) {
    List<String> matches = new ArrayList();
    startswith = startswith.toLowerCase();
    for (String country : sites) {
        if (country.toLowerCase().startsWith(startswith)) {
            matches.add(country);
        }
    }
    return matches;
}

Note: While this example gets its data from a CSV file, you can easily adapt the code to work with any database or ORM setup. It's easy to build a WHERE clause when you only have one filter string to use.

We're almost there. This example now loads in modern browsers, but it's extremely sluggish and still crashes older browsers. Why?

A single character can still match tens of thousands of results with such a large dataset. And even though JavaScript is no longer filtering these requests, it still has to loop over them and add an <li> for each into the autocomplete's menu.

Therefore we'll make one final adjustment: setting the AutoComplete widget's minLength option to 3.

<kendo:autoComplete name="site" minLength="3">
    <kendo:dataSource serverFiltering="true">
        <kendo:dataSource-transport read="/App/sites">
            <kendo:dataSource-transport-parameterMap>
                function( options ) {
                    return { filter: options.filter.filters[ 0 ].value };
                }
            </kendo:dataSource-transport-parameterMap>
        </kendo:dataSource-transport>
    </kendo:dataSource>
</kendo:autoComplete>

This requires the user to type three characters before seeing results. Because the maximum number of matches is now in the hundreds rather than the tens of thousands, our autocomplete functions fine in all browsers.

While forcing the user to type three characters before getting feedback is not ideal, it's a necessary optimization when dealing with enormous data sets such as this.

As another option, you can consider returning a subset of the data for shorter queries. For example consider Google. The search engine has an enormous number of options it could show you when you type "a", but it limits the results to the top four or five.

Let's add this behavior to our example. The following alternation to the load() method returns the first 100 matches if the filter is less than three characters.

public static List<String> load(String startswith) {
    List<String> matches = new ArrayList();
    startswith = startswith.toLowerCase();
    for (String country : sites) {
        if (country.toLowerCase().startsWith(startswith)) {
            matches.add(country);
        }
        if (startswith.length() < 3 && matches.size() == 100) {
            return matches.subList(0, 100);
        }
    }

    return matches;
}

This approach works well if you have some means of ranking the options. In our case, since the sites are ranked from 1 to one million, this is perfect; the user is more likely to type a site towards the top of the rankings anyways.

After all of this, we finally have a functioning autocomplete with our epic data set.

You can checkout the full source of these examples at https://github.com/tjvantoll/kendo-java-autocomplete. The code runs surprisingly quickly for such a large data set. Even if you're not interested in the implementation, an autocomplete of the top million sites is a lot of fun to filter through.

Conclusion

In this article we built several increasingly large autocomplete controls using Kendo UI's AutoComplete widget. We started with a simple hardcoded list, moved onto a server driven list, then finally switched the filtering itself to the server.

We also saw how easy it was to integrate Kendo UI's JSP wrappers and a Java backend into the autocomplete process.

Do you have any other Java + Kendo UI integrations you'd like us to discuss? Let us know in the comments and we'll see what we can do.