Matchbook Example "Game", 2x iOS and 2x Android clients

Matchbook is a lightweight and platform-agnostic matchmaking solution, intended for use in mobile applications (think games that require near-real-time, relatively-low-latency, persistent communications between two or more client devices).

At its core is a server component, which provides a JSON-based webservice allowing clients to find, create, and join matches. The server also acts as a proxy/relay when necessary, allowing client devices to tunnel through any firewalls that might exist between them.

In addition to the server component, Matchbook includes prebuilt SDK’s for both Java and Objective-C. These SDK’s are intended to support the development of native applications that make use of the Matchbook webservice on Android and iOS devices, respectively.

I could go on at length, but it’s simpler to just link to the project on Github (Matchbook is open-source, and permissively licensed, naturally):

https://github.com/adam-roth/matchbook

Note that Google is currently building comparable functionality into Google Play, although their realtime communications API is currently only available on Android (iOS support is under development).

And although Apple already has realtime and turn-based gaming API’s for iOS, they natually have no intention of inviting Android devices to the party. Ever.

So let the record show that Matchbook got there first.

Also note that the method described in the other article for getting your hands on an instance of ‘Unsafe‘ no longer works. Using the latest JDK, the following code will not work (you’ll get compiler errors for even trying to import sun.misc.Unsafe):

private static Unsafe getUnsafeInstance() throws SecurityException, NoSuchFieldException, IllegalArgumentException,
    IllegalAccessException {
    Field theUnsafeInstance = Unsafe.class.getDeclaredField("theUnsafe");
    theUnsafeInstance.setAccessible(true);
    return (Unsafe) theUnsafeInstance.get(Unsafe.class);
 }

Here is a variant that still works:

	public static Object getUnsafe() {
		try {
			Class unsafeClass = Class.forName("sun.misc.Unsafe");
			Field unsafeField = unsafeClass.getDeclaredField("theUnsafe");
			unsafeField.setAccessible(true);
			Object unsafeObj = unsafeField.get(unsafeClass);
			
			return unsafeObj;
		}
		catch (Exception e) {
			return null;//UNSAFE_ERROR_CODE;
		}
	}

Note that as importing sun.misc.Unsafe now causes a compiler error, it is only possible to return an Object reference to the ‘Unsafe‘ instance. This means that any method that you want to call on it must be done through reflection, which is tedious at best. Luckily, I’ve taken care of this tedious bit for you, and created a wrapper that provides the same public API as sun.misc.Unsafe and uses reflection to delegate method calls to the real ‘Unsafe‘ instance. The source code for this utility is quite long and not very interesting, so I’m not going to post it here. Instead you can use this direct link to download a copy.

Here is a simple example of how to use this class to perform some basic tasks:

		System.out.println("Testing sun.misc.Unsafe...");
		double[] averages = {-777.0, -777.0, -777.0};
		UnsafeUtil util = new UnsafeUtil();
		System.out.println("addressSize=" + util.addressSize() + ", pageSize=" + util.pageSize());
		
		long memPointer = util.allocateMemory(1024);
		long memPointer2 = util.allocateMemory(1024);
		System.out.println("1K memory blocks at addresses:  " + memPointer + ", " + memPointer2);
		
		//valid copy
		util.copyMemory(memPointer, memPointer2, 1024);
		
		//invalid copy
		//util.copyMemory(memPointer, memPointer2, 1024000);
		
		int result = util.getLoadAverage(averages, 1);
		System.out.println("getLoadAverage:  Result=" + result + ", averages=" + averages[0] + ", " + averages[1] + ", " + averages[2]);

Do note that if you uncomment the “invalid copy” line then running this code will very likely crash your JVM. As noted above, you should use this utility with caution, particularly if you are running it in an environment shared by other Java code.

And why do all this when Sun (now Oracle) clearly does not want people mucking around with this class? Because ultimately power belongs in the hands of developers. All developers, not just the select few chosen to work on privileged system code.

But suffice to say, if you want to install the Adobe AIR runtime on an emulated Android device for testing or development purposes without having to wade through a ton of fuss and nonsense, you have two basic options. The first is described briefly here, and basically involves installing the latest Flash/Flex/AIR SDK from the official website and then grabbing the AIR runtime package from ‘<AIR_SDK_ROOT>/runtimes/air/android/emulator/Runtime.apk‘. This file is the AIR runtime that must be installed on the emulator for any apps built with Adobe AIR to function. And this method works fine if you don’t mind downloading and installing the entire AIR SDK (90 MB) just to grab this one file.

You other option is to use this direct link to download just the AIR runtime package (6.1 MB). Note that this is version 2.7 of the AIR runtime, and has been tested with an emulator running Android 3.1 only. I cannot vouch for it working with any other configurations.

In either case, once you have gotten a hold of the the runtime .apk file, installing it on your emulator is a relatively simple process. All you need is the ‘adb‘ utility that is included in the “Android SDK Platform Tools” package (note that this package is not the same as the similarly named “Android SDK Tools” package). If you don’t have this package installed yet, then use your Android configuration manager to install it. Then simply navigate to ‘<ANDROID_SDK_ROOT>/platform-tools‘ and run the following command (while your Android emulator is running):

adb install /path/to/your/AIR/runtime.apk

This will install the AIR runtime on your emulated device, and you can now install and run AIR-based Android apps on your emulator. Quite simple, really. Which only doubles my confusion with respect to why this simple process seems to be so poorly documented online.

As this example code builds upon the OverridableHttpRequest class discussed in the previous post, you will need that class (or your own comparable implementation) before you get started. Once you have that, the code for the filter implementation is not very complex:

public class InputSanitizerFilter implements Filter {
	private static final Logger LOG = Logger.getLogger(InputSanitizerFilter.class);
	
	private static final String BANNED_INPUT_CHARS = ".*[^a-zA-Z0-9\\@\\'\\,\\.\\/\\(\\)\\+\\=\\-\\_\\[\\]\\{\\}\\^\\!\\*\\&\\%\\$\\:\\;\\? \\t]+.*";
	
	public static final String QUARANTINE_ATTRIBUTE_NAME = "filter.quarantined.params";
	public static final String SUSPICIOUS_REQUEST_FLAG_NAME = "filter.suspicious.request";

	@Override
	public void destroy() {
		//no work necessary
	}

	@Override
	@SuppressWarnings("unchecked")
	public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException {
		//wrap the original request and set up an empty quarantine map
		OverridableHttpRequest newRequest = new OverridableHttpRequest((HttpServletRequest)request);
		Map<String, String> quarantine = new HashMap<String, String>();
		newRequest.setAttribute(QUARANTINE_ATTRIBUTE_NAME, quarantine);
		
		//inspect each parameter, and move any suspicious ones into thw quarantine area
		Enumeration<String> names = request.getParameterNames();
		while (names.hasMoreElements()) {
			String name = names.nextElement();
			String value = request.getParameter(name);
			if (value.matches(BANNED_INPUT_CHARS)) {
				//uh-oh, found something that doesn't look right, quarantine it and make sure the request is flagged as suspicious
				LOG.warn("Removing potentially malicious parameter from request:  " + name);
				quarantine.put(name, value);
				newRequest.removeParameter(name);
				newRequest.setAttribute(SUSPICIOUS_REQUEST_FLAG_NAME, "true");
			}
		}
		
		//done, send the modified request on down the chain
		chain.doFilter(newRequest, response);
	}

	@Override
	public void init(FilterConfig arg0) throws ServletException {
		//no work necessary
	}
}

Basically there is a regex that defines the set of disallowed characters (anything that is not a letter, number, standard punctuation character, or whitespace…this example will also exclude any parameters that contain newlines as the webapp that it is used in does not contain any inputs that can accept newlines, but you can obviously modify it to your liking), and every parameter in the request is inspected to see if it contains one or more banned character(s). Any parameter that is found to include banned characters is copied into the ‘quarantine‘ map, and then removed as a parameter (so calling ‘request.getParameter(“badParam”)‘ will return null). This will prevent application code from ever seeing the suspect parameter(s), unless it explicitly goes looking inside of the quarantine map (which you might do if you want to allow users to have special characters in their password, for instance).

Also, if one or more parameters are quarantined, the request is flagged as suspicious by setting an attribute named ‘filter.suspicious.request‘. This gives code downstream from the filter a quick way to see if the request has been modified by the filter in any way, and allows the application to make its own decisions about whether or not it wants to venture into the quarantine area to inspect the potentially unsafe data. For instance:

if (request.getAttribute(InputSanitizerFilter.SUSPICIOUS_REQUEST_FLAG_NAME) != null) {
    Map<String, String> quarantine = (Map<String, String>)request.getAttribute(InputSanitizerFilter.QUARANTINE_ATTRIBUTE_NAME);
    for (String key : quarantine.keySet()) {
        System.out.println("Quarantined parameter:  name=" + key + ", value=" + quarantine.get(key));
    }
}

…or, with the right frameworks in place you might easily create something like an ‘@IgnoresQuarantine‘ annotation that can be applied selectively to business methods to mark a parameter (or set of parameters) as not subject to quarantine on calls to that method so that you never need to manually go looking through the quarantine area (after implementing the annotation and its backing logic). But that’s just slightly outside of the scope for today.

For example, say that you have a Filter that you are using to sanitize input parameters and guard against things like XSS attacks by ensuring that obviously invalid values like “<script>alert(‘hacked!’);</script>” are filtered out. Wouldn’t it be great if you could implement your Filter such that when it finds one or more forbidden values it flags the request as potentially malicious (using a request attribute), removes any parameters that contain potentially unsafe data, and relocates the potentially unsafe data to a predetermined quarantine area (also accessible via a request attribute)? This would protect your webapp code from ever receiving a malicious parameter, while still allowing parts of the code that might permit seemingly malicious parameters (for instance, there is no reason to prevent a user from registering with a password of “<script>alert(‘hacked!’);</script>” if that is what they want to use, particularly if you are hashing user passwords like you should be) to still access them if desired by going through the quarantine area.

Of course, two-thirds of the functionality described above can be implemented without being able to override request parameters. With the standard API you can certainly check for potentially malicious parameters, set an attribute if you find any, and copy their values into a quarantine area. But what you cannot do is remove the parameters from the request, so any application code that directly accesses a request parameter value may still be at risk, particularly if its author forgets to check to see if the request has been flagged as suspect. The real beauty of being able to override parameter values is that you can do things like completely prevent a malicious parameter from ever being visible to your application code unless your application code goes out of its way to look for it (and if you do that, and you do it incorrectly, then that’s your own fault).

Anyways, the code to enable this kind of functionality is a fairly straightforward (if tedious) exercise in writing an HttpServletRequest wrapper and then overriding a few choice methods (and adding a couple new ones):

import java.io.BufferedReader;
import java.io.IOException;
import java.io.UnsupportedEncodingException;
import java.security.Principal;
import java.util.Collections;
import java.util.Enumeration;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Locale;
import java.util.Map;
import java.util.Set;

import javax.servlet.RequestDispatcher;
import javax.servlet.ServletInputStream;
import javax.servlet.http.Cookie;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpSession;

public class OverridableHttpRequest implements HttpServletRequest {
        
	private HttpServletRequest wrappedRequest;
	private Map<String, String> newParams;
	private Set<String> removedParams;

	public OverridableHttpRequest(HttpServletRequest requestToWrap) {
		this.wrappedRequest = requestToWrap;
		this.newParams = new HashMap<String, String>();
		this.removedParams = new HashSet<String>();
	}

	// these things we add so that params can be overridden
	public void setParameter(String name, String value) {
		this.removedParams.remove(name);
		this.newParams.put(name, value);
	}

	public void removeParameter(String name) {
		this.newParams.remove(name);
		this.removedParams.add(name);
	}

	// these things we need to override so that the correct state is exposed through the standard API
	@SuppressWarnings("rawtypes")
	@Override
	public Enumeration getParameterNames() {
		Set<String> result = new HashSet<String>();
		Enumeration requestParams = this.wrappedRequest.getParameterNames();
		while (requestParams.hasMoreElements()) {
			Object param = requestParams.nextElement();
			if (!removedParams.contains(param)) {
				result.add((String) param);
			}
		}
		result.addAll(newParams.keySet());

		return Collections.enumeration(result);
	}

	@Override
	public String[] getParameterValues(String arg0) {
		//NOTE:  not strictly to spec
		String[] result = new String[1];
		result[0] = this.getParameter(arg0);

		return result;
	}

	@Override
	public String getParameter(String arg0) {
		if (removedParams.contains(arg0)) {
			return null;
		}
		if (newParams.containsKey(arg0)) {
			return newParams.get(arg0);
		}
		return this.wrappedRequest.getParameter(arg0);
	}

	@SuppressWarnings("rawtypes")
	@Override
	public Map getParameterMap() {
		Map<String, String[]> result = new HashMap<String, String[]>();
		for (Object key : this.wrappedRequest.getParameterMap().keySet()) {
			result.put((String)key, (String[])this.wrappedRequest.getParameterMap().get(key));
		}
		for (String key : this.newParams.keySet()) {
			result.put(key, new String[] {this.newParams.get(key)});
		}
		for (String key : this.removedParams) {
			result.remove(key);
		}
		
		return result;
	}

	// these things we should probably override but don't right now
	@Override
	public String getRequestURI() {
		// FIXME: should return a modified URI based upon current state
		return this.wrappedRequest.getRequestURI();
	}

	@Override
	public StringBuffer getRequestURL() {
		// FIXME: should return a modified URL based upon current state
		return this.wrappedRequest.getRequestURL();
	}

	@Override
	public String getQueryString() {
		// FIXME: should return a modified String based upon current state
		return this.wrappedRequest.getQueryString();
	}

	// everything else just passes through
	@Override
	public Object getAttribute(String arg0) {
		return this.wrappedRequest.getAttribute(arg0);
	}

	@SuppressWarnings("rawtypes")
	@Override
	public Enumeration getAttributeNames() {
		return this.wrappedRequest.getAttributeNames();
	}

	@Override
	public String getCharacterEncoding() {
		return this.wrappedRequest.getCharacterEncoding();
	}

	@Override
	public int getContentLength() {
		return this.wrappedRequest.getContentLength();
	}

	@Override
	public String getContentType() {
		return this.wrappedRequest.getContentType();
	}

	@Override
	public ServletInputStream getInputStream() throws IOException {
		return this.wrappedRequest.getInputStream();
	}

	@Override
	public String getLocalAddr() {
		return this.wrappedRequest.getLocalAddr();
	}

	@Override
	public String getLocalName() {
		return this.wrappedRequest.getLocalName();
	}

	@Override
	public int getLocalPort() {
		return this.wrappedRequest.getLocalPort();
	}

	@Override
	public Locale getLocale() {
		return this.wrappedRequest.getLocale();
	}

	@SuppressWarnings("rawtypes")
	@Override
	public Enumeration getLocales() {
		return this.wrappedRequest.getLocales();
	}

	@Override
	public String getProtocol() {
		return this.wrappedRequest.getProtocol();
	}

	@Override
	public BufferedReader getReader() throws IOException {
		return this.wrappedRequest.getReader();
	}

	@SuppressWarnings("deprecation")
	@Override
	public String getRealPath(String arg0) {
		return this.wrappedRequest.getRealPath(arg0);
	}

	@Override
	public String getRemoteAddr() {
		return this.wrappedRequest.getRemoteAddr();
	}

	@Override
	public String getRemoteHost() {
		return this.wrappedRequest.getRemoteHost();
	}

	@Override
	public int getRemotePort() {
		return this.wrappedRequest.getRemotePort();
	}

	@Override
	public RequestDispatcher getRequestDispatcher(String arg0) {
		return this.wrappedRequest.getRequestDispatcher(arg0);
	}

	@Override
	public String getScheme() {
		return this.wrappedRequest.getScheme();
	}

	@Override
	public String getServerName() {
		return this.wrappedRequest.getServerName();
	}

	@Override
	public int getServerPort() {
		return this.wrappedRequest.getServerPort();
	}

	@Override
	public boolean isSecure() {
		return this.wrappedRequest.isSecure();
	}

	@Override
	public void removeAttribute(String arg0) {
		this.wrappedRequest.removeAttribute(arg0);
	}

	@Override
	public void setAttribute(String arg0, Object arg1) {
		this.wrappedRequest.setAttribute(arg0, arg1);
	}

	@Override
	public void setCharacterEncoding(String arg0)
			throws UnsupportedEncodingException {
		this.wrappedRequest.setCharacterEncoding(arg0);
	}

	@Override
	public String getAuthType() {
		return this.wrappedRequest.getAuthType();
	}

	@Override
	public String getContextPath() {
		return this.wrappedRequest.getContextPath();
	}

	@Override
	public Cookie[] getCookies() {
		return this.wrappedRequest.getCookies();
	}

	@Override
	public long getDateHeader(String arg0) {
		return this.wrappedRequest.getDateHeader(arg0);
	}

	@Override
	public String getHeader(String arg0) {
		return this.wrappedRequest.getHeader(arg0);
	}

	@SuppressWarnings("rawtypes")
	@Override
	public Enumeration getHeaderNames() {
		return this.wrappedRequest.getHeaderNames();
	}

	@SuppressWarnings("rawtypes")
	@Override
	public Enumeration getHeaders(String arg0) {
		return this.wrappedRequest.getHeaders(arg0);
	}

	@Override
	public int getIntHeader(String arg0) {
		return this.wrappedRequest.getIntHeader(arg0);
	}

	@Override
	public String getMethod() {
		return this.wrappedRequest.getMethod();
	}

	@Override
	public String getPathInfo() {
		return this.wrappedRequest.getPathInfo();
	}

	@Override
	public String getPathTranslated() {
		return this.wrappedRequest.getPathTranslated();
	}

	@Override
	public String getRemoteUser() {
		return this.wrappedRequest.getRemoteUser();
	}

	@Override
	public String getRequestedSessionId() {
		return this.wrappedRequest.getRequestedSessionId();
	}

	@Override
	public String getServletPath() {
		return this.wrappedRequest.getServletPath();
	}

	@Override
	public HttpSession getSession() {
		return this.wrappedRequest.getSession();
	}

	@Override
	public HttpSession getSession(boolean arg0) {
		return this.wrappedRequest.getSession(arg0);
	}

	@Override
	public Principal getUserPrincipal() {
		return this.wrappedRequest.getUserPrincipal();
	}

	@Override
	public boolean isRequestedSessionIdFromCookie() {
		return this.wrappedRequest.isRequestedSessionIdFromCookie();
	}

	@Override
	public boolean isRequestedSessionIdFromURL() {
		return this.wrappedRequest.isRequestedSessionIdFromURL();
	}

	@SuppressWarnings("deprecation")
	@Override
	public boolean isRequestedSessionIdFromUrl() {
		return this.wrappedRequest.isRequestedSessionIdFromUrl();
	}

	@Override
	public boolean isRequestedSessionIdValid() {
		return this.wrappedRequest.isRequestedSessionIdValid();
	}

	@Override
	public boolean isUserInRole(String arg0) {
		return this.wrappedRequest.isUserInRole(arg0);
	}

}

So the new methods being added here are ‘removeParameter(String name)‘ and ‘setParameter(String name)‘, which do pretty much what their name implies. If you are familiar with the standard ‘removeAttribute(String name)‘ and ‘setAttribute(String name)‘ methods, then you should feel right at home with these new additions. They simply let you manipulate request parameters in a way that’s identical to how you can already manipulate request attributes.

One minor deviation from the Servlet specification that is worth noting is that I have overridden ‘getParameterValues(String name)‘ such that it only returns the first value associated with a given parameter. This means that if for some reason your webapp uses URL’s like “http://mysite.com/api?user=bob&user=jane&user=paul” then you will only see “bob” as a value for the ‘user’ parameter. In practice I have not ever come across a web application that intentionally relied on a single parameter name having multiple values associated with it, and if you are designing your web application in such a way then you should probably just stop and pick a less confusing pattern. I see no value in a feature that allows a single parameter to have multiple values that you only get to see if you use a different API method to get them (‘getParameterValues‘ instead of ‘getParameter‘), and so I have removed this feature from the implementation. If someone can come up with a solid justification for having such a feature, I will add it back in.

Also, left as an exercise is overriding ‘getRequestURI()‘, ‘getRequestURL()‘, and ‘getQueryString()‘ to return the correct values based upon the modified request state. It’s fairly rare to have application code that depends upon the values of these calls, so in most cases you will not need to do this.

In any case, to make use of the OverridableHttpRequest class, you can do the following:

public class ExampleFilter implements Filter {
	@Override
	public void init(FilterConfig filterConfig) throws ServletException {
		//do initialization things here
	}

	@Override
	public void doFilter(ServletRequest request, ServletResponse response,
			FilterChain filterChain) throws IOException, ServletException {

		HttpServletRequest httpReq = (HttpServletRequest) response;
		OverridableHttpRequest newRequest = new OverridableHttpRequest(httpReq);
     
		//do work and modify the request as desired
		newRequest.removeParameter("someBadXssParam");

		//pass the modified request on to the webapp, anyone downstream will see 
		//the modified state with no 'someBadXssParam' in it
		filterChain.doFilter(newRequest, response);
	}

	@Override
	public void destroy() {
		//do shutdown things here
	}
}

Simple, but powerful. I just wish the Servlet spec included this kind of functionality out of the box so that it wouldn’t be necessary to implement a complete HttpServletRequest wrapper just to add a couple of basic mutator methods.

I’ve always been mildly amused at the continually growing use of CAPTCHA images, or more accurately, at their ever-increasing complexity. It seems that the only truly effective CAPTCHA’s are the ones that even human beings can barely decipher. But more interesting to me is the fact that these distorted snippets of letters and numbers have become a sort of de-facto Turing test. If you can determine what the characters are, then you are human; otherwise you are not. For whatever reason, these images have become a symbolic line in the sand separating man from machine, and by exploring ways to cross this line we may move ever so slightly closer towards the creation of true artificial-intelligence.

So let’s examine a very basic CAPTCHA image, one that was used in a popular online-forum distribution before it was cracked long ago:

This CAPTCHA works on the principle of contrast. Human beings can discern distinct regions in an otherwise noisy image so long as each distinct region meets some minimum contrast level above/below that of the background noise. This kind of image can be difficult to decipher computationally, because pulling out coherent regions from amongst the background noise requires contextual understanding of large portions of the image at once, which is generally a difficult thing to accomplish programmatically. That isn’t to say it can’t be done, however.

A human being looking at this image is able to recognize that there is some threshold created by the background noise that has been introduced, above which an element is part of the encoded data, and below which an element is simply part of the background noise and should be discarded. Once that is done discerning the text becomes a simple matter of discarding everything below the noise threshold, and keeping everything above. So let’s see if we can code it. First, we need a way to determine the noise threshold:

        //init, determine the average color intensity of the image
        int average = 0;
        for (int row = 0; row < image.getHeight(); row++) {
                for (int column = 0; column < image.getWidth(); column++) {
                        int color = image.getRGB(column, row) & 0x000000FF;  //only need the last 8 bits
                        average += color;
                }
        }
        average /= image.getWidth() * image.getHeight();

This bit of code determines the average color intensity of the entire image (216 / 255 in this case). Because this CAPTCHA is in grayscale it only needs to look at a single component of the pixel color, but colorized CAPTCHA images could be processed in a similar fashion by computing the intensity using the full RGB value. In any case, now we have a basic threshold that we can use for determining which parts of the image contain valuable data, and which parts contain only noise. We can do that like so:

        //first pass, mark all pixels as WHITE or BLACK
        for (int row = 0; row < image.getHeight(); row++) {
                for (int column = 0; column < image.getWidth(); column++) {
                        int color = image.getRGB(column, row) & 0x000000FF;  //only need the last 8 bits
                        if (color <= average * .70 ) {
                                image.setRGB(column, row, BLACK);
                                darkRegion = true;
                        }
                        else if (color < .85 * average && darkRegion && row < image.getHeight() - 1 
                                && (image.getRGB(column, row + 1) & 0x000000FF) < .85 * average) {
                                image.setRGB(column, row, BLACK);
                        }
                        else if (color < .85 * average && ! darkRegion && row < image.getHeight() - 1 && column > 0 
                                && column < image.getWidth() - 1 
                                &&  (((image.getRGB(column, row + 1) & 0x000000FF) < color) 
                                        || ((image.getRGB(column + 1, row) & 0x000000FF) < color) 
                                        || ((image.getRGB(column - 1, row) & 0x000000FF) < color))) {
                                image.setRGB(column, row, BLACK);
                                darkRegion = true;
                        }
                        else {
                                image.setRGB(column, row, WHITE);
                                darkRegion = false;
                        }
                }
        }

Note that this code assumes that darker pixels are part of the data and lighter pixels are part of the background noise, because that is how the input CAPTCHA is set up. A smarter approach would be to look at the number of pixels falling above the noise threshold and the number of pixels falling below, and then keep whichever group is smaller. For a CAPTCHA like this one to be effective, there must be more noise than data, so it follows that the data that you’re looking for will always be in the smaller group of pixels.

In any case, what the above code does is traverse the image, and turn any pixels that appear to be noise white, and any pixels that appear to be data black. Note that it includes some rudimentary region-detection code, owing to the fact that we expect our data pixels to be tightly clustered together in distinct regions. So when the code encounters a pixel that it considers to be part of the data, it also lowers the selection criteria for the next pixel because there is a strong possibility that the next pixel will also be data. This helps prevent false-negatives from erroneously dropping out valuable pieces of data. Let’s take a peek at what our CAPTCHA image looks like at this point:

It’s not perfect, but it is definitely improved. We have successfully removed all of the background noise from the image, but unfortunately we have also removed some pieces of the actual data. The data that is left is all in the right place, however, so perhaps we can amplify and/or reconstruct it:

                //second pass, eliminate horizontal gaps
                for (int row = 0; row < image.getHeight(); row++) {
                        for (int column = 0; column < image.getWidth(); column++) {
                                int color = image.getRGB(column, row) & 0x000000FF;  //only need the last 8 bits
                                if (color == 255) {
                                        consecutiveWhite++;
                                }
                                else {
                                        if (consecutiveWhite < 3 && column > consecutiveWhite) {  
                                                for (int col = column - consecutiveWhite; col < column; col++) {
                                                        image.setRGB(col, row, BLACK);
                                                }
                                        }
                                        consecutiveWhite = 0;
                                }
                        }
                }
                consecutiveWhite = 0;
                
                //third pass, eliminate vertical gaps
                for (int column = 0; column < image.getWidth(); column++) {
                        for (int row = 0; row < image.getHeight(); row++) {
                                int color = image.getRGB(column, row) & 0x000000FF;  //only need the last 8 bits
                                if (color == 255) {
                                        consecutiveWhite++;
                                }
                                else {
                                        if (consecutiveWhite < 2 && row > consecutiveWhite) {
                                                for (int r = row - consecutiveWhite; r < row; r++) {
                                                        image.setRGB(column, r, BLACK);
                                                }
                                        }
                                        consecutiveWhite = 0;
                                }
                        }
                }

This code fills in any small vertical and horizontal runs of white pixels with black pixels, the rationale being that any small group of white pixels that is surrounded on either end by black pixels is virtually guaranteed to be part of the data that was erroneously discarded. Again we can take a peek at our result:

Getting better, but we’re not quite there yet. Our characters are much more distinct, but there is still some missing data. A fair bit of the missing data is now contained in small regions of white pixels that are actually encapsulated within our characters. Filling them in is a relatively simple matter:

                //fourth pass, attempt to fill regions
                for (int row = 0; row < image.getHeight(); row++) {
                        for (int column = 0; column < image.getWidth(); column++) {
                                if (image.getRGB(column, row) == WHITE) {
                                        int height = countVerticalWhite(image, column, row);
                                        int width = countHorizontalWhite(image, column, row);
                                        int area = width * height;
                                        if ((area <= 12) || (width == 1) || (height == 1)){
                                                image.setRGB(column, row, BLACK);
                                        }
                                }
                        }
                }
                
                //fifth pass repeats the fourth
                for (int row = 0; row < image.getHeight(); row++) {
                        for (int column = 0; column < image.getWidth(); column++) {
                                if (image.getRGB(column, row) == WHITE) {
                                        int height = countVerticalWhite(image, column, row);
                                        int width = countHorizontalWhite(image, column, row);
                                        int area = width * height;
                                        if ((area <= 12) || (width == 1) || (height == 1)){
                                                image.setRGB(column, row, BLACK);
                                        }
                                }
                        }
                }

Here we check, for each white pixel, how many adjacent white pixels exist both vertically and horizontally. This gives us a rough estimate of the size of the current region of white pixels. If the size is too small, then the code assumes that the white pixel is actually supposed to be part of the data, and turns it black. Note that the algorithm is methodical in its approach, in that when it detects a small region of white pixels, it toggles only the initial pixel that it tested in that region. This toggling will reduce the region-size reported for any adjacent white pixels, increasing the likelihood that they will be toggled as well on the next iteration, which is why two passes of the same algorithm are applied. And yes, I know having the same code repeated twice is poor coding style, but for illustrative purposes it gets the job done. Anyways, we now have:

Many of the gaps are now filled in, and the text is starting to look fairly legible. There are now, however, a few spurious black pixels that have cropped up along the edges of the characters. We could go back and refine the previous step, but instead let’s just prune out these outliers:

                //sixth pass, clear any false-positive
                for (int row = 0; row < image.getHeight(); row++) {
                        for (int column = 0; column < image.getWidth(); column++) {
                                if (image.getRGB(column, row) != WHITE) {
                                        if (countBlackNeighbors(image, column, row) < 3) {
                                                image.setRGB(column, row, WHITE);
                                        }
                                }
                        }
                }

This pruning step removes any black pixels that are bordered by 3 or fewer black pixels. This is a fairly strict threshold, and will have the effect of smoothing/rounding out corners (i.e. some legitimate data will be discarded), but it will also clear out any spurious black pixels that exist in the image. Now our image looks like so:

The letters have taken on a softer, more rounded quality. They also happen to look vaguely reminiscent of what you might get if you were to scan a text document using an older scanner. Which is worth mentioning because we will eventually be feeding our cleaned-up CAPTCHA image to an optical-character-recognition program that is designed to process just this sort of data. First, however, our characters are all misaligned. We’ve come this far, so we might as well fix the alignment issue while we’re at it:

                //now find the characters
                List<CharacterBox> characters = new ArrayList<CharacterBox>();
                int totalCharWidth = 10;
                int maxCharHeight = 0;
                for (int column = 0; column < image.getWidth(); column++) {
                        int highestBlack = countVerticalWhite(image, column, 0);
                        if (highestBlack < image.getHeight()) {
                                totalCharWidth += 5; //5 px spacing in between chars
                                CharacterBox box = new CharacterBox();
                                box.setX(column);
                                while (column < image.getWidth() && countVerticalWhite(image, column, 0) < image.getHeight()) {
                                        int currentBlack = countVerticalWhite(image, column, 0);
                                        if (currentBlack < highestBlack) {
                                                highestBlack = currentBlack;
                                        }
                                        column++;
                                }
                                box.setWidth(column - box.getX());
                                box.setY(highestBlack - 5);
                                box.setHeight(image.getHeight() - highestBlack + 5); //can trim this later
                                if (box.getHeight() > maxCharHeight) {
                                        maxCharHeight = box.getHeight();
                                }
                                totalCharWidth += box.getWidth();
                                characters.add(box);
                        }
                }

Here we simply compute a bounding box for each distinct region of black pixels (i.e. each character), plus some additional padding so that our output image will draw nicely. Speaking of output image, we can now create it by positioning our characters in correct alignment with each other in a new image, like so:

                //output a new image with aligned characters
                BufferedImage dst = new BufferedImage (totalCharWidth, maxCharHeight,
                                                           BufferedImage.TYPE_INT_BGR);
                for (int column = 0; column < dst.getWidth(); column++) {
                        for (int row = 0; row < dst.getHeight(); row++) {
                                dst.setRGB(column, row, WHITE);
                        }
                }
                int xPos = 5;
                int yPos = 0;
                for (CharacterBox box : characters) {
                        for (int oldY = box.getY(); oldY < box.getY() + box.getHeight(); oldY++) {
                                for (int oldX = box.getX(); oldX < box.getX() + box.getWidth(); oldX++) {
                                        dst.setRGB(xPos + (oldX - box.getX()), yPos + (oldY - box.getY()), image.getRGB(oldX, oldY));
                                }
                        }
                        xPos += box.getWidth() + 5;
                }
                ImageIO.write(dst, "png", new File(OUTPUT));

Now we have the following:

The characters are nicely aligned and uniformly spaced. We now have something that is suitable for sending into a character-recognition program. For this example we use tesseract, a free and open-source OCR program that provides a good level of accuracy. We can send our output to tesseract like so:

                Process tesseractProc = Runtime.getRuntime().exec(TESSERACT_BIN + " " + OUTPUT + " " + TESSERACT_OUTPUT);
                tesseractProc.waitFor();

This invokes tesseract on our output image, and it writes its results to a text file located at ‘TESSERACT_OUTPUT‘. In this case, the text file contains the following:

IKEECL

…which is 100% correct.

Using a handful of very simple image filtering loops based around a brief examination of how a human being would approach the image, and some existing OCR software, the CAPTCHA has been defeated. Of course, this only works for this one specific style of CAPTCHA, but the basic approach of reducing noise, amplifying data, and isolating characters should be broadly applicable to a wide range of different CAPTCHA styles. The challenge lies not in breaking the CAPTCHA, but in devising an algorithm that can attempt to break any number of different CAPTCHA styles dynamically and with a success rate comparable to that of a human being. It needs a way to determine, from the CAPTCHA image itself, what kind of noise exists and how it should best be removed. That is the real challenge, and it’s beyond the scope of this article.

Note that for the sake of preserving some sense of brevity I’ve left out the implementation of some minor utility functions and variable declarations and the like. In general, you can assume that a function (or variable) does what its name implies. If, however, you would like a complete copy of the source-code used, you can download it using this link (zipped Eclipse project).

Note that in order to get it to run you will also need to install tesseract on your system, and edit the values at the start of the Java code to point at your local tesseract installation.