Published

I once shared with a friend how mistakes as small as 3 letters can take down an entire application.

Whether it’s a website, mobile app or bespoke system, the running and delivery of web application requires a lot of moving parts operating in conjunction with each other. A server receives requests which it forwards to some server-side language to perform some action which often includes querying a database. Then you have third-party services such as APIs, intrusion prevention system and social networks. All these parts need to be properly configured so they can communicate with each other and in a world of typos that’s where the danger lies; mistype a command and you can be left staring at a wall of text hunting for the culprit perfectly hidden in plain sight.

My smallest configuration mistake used to be 3 characters long — I mistyped 3 letters and the connection died. Today I beat my record with a typo that took down an application because of a single character.

So we’ve got an application that’s a little bit dated. It’s running fine on production but our client alerted us to some minor bugs they kept encountering which we could not duplicate on our development environment. It should be noted that both environments were compatible but not identical. And yes, we all know about Docker but for various reasons it was not an option in this case.

Given the situation, we decided to set up another development environment that was as close as possible to production. This necessitated manually acquiring and setting up everything we needed which was no small feat given how technology quickly forgets older versions making them harder to locate. But we managed to gather what we needed and after some quick testing were able to install our application on our new (old) development environment.

Except that it would not run.

We were using Apache HTTP server and our application required .htaccess files be enabled and the RewriteEngine be turned on. Nothing out of the ordinary there except the log showed this error:

Options FollowSymLinks and SymLinksIfOwnerMatch are both off, so the RewriteRule directive is also forbidden due to its similar ability to circumvent directory restrictions : /PATH/TO/FILE

Basically the server is saying “Hey, we know you wanna re-write the request, probably because your application needs it, but you’ve turned off FollowSymLinks and SymLinksIfOwnerMatch in your options which makes things a little bit sus. So we’re gonna go ahead and stop the re-write.”

The thing is we had already turned on both FollowSymLinks and SymLinksIfOwnerMatch in our configuration file. Heck, we eventually turned it on in several different places because we just could not work out where the problem lay. We went as far as to re-install and re-configure the Apache HTTP server from scratch to give ourselves a clean slate and that’s when we saw the light. You see, in order for our application to run Apache needed to hand over control to a server-side language. So we checked the configuration for the hand-over process and we found it.

Here’s what the configuration looked like at first:

Options ExecCGI

And here is what it should have looked like:

Options +ExecCGI

Do you see it? Do you see the one missing character? That’s all it took and our application ran fine ever since.

Basically what was happening was we were over-writing the previous Option values that had been set earlier. By adding the innocuous plus sign, we were instead adding a new value to the list. And to be fair, Apache knew this would trip people up and included the following message in the main configuration file:

# The Options directive is both complicated and important.  Please see
# http://httpd.apache.org/docs/2.4/mod/core.html#options
# for more information.

I’ve heard people say the Apache’s confusing configuration is what pushed people to use NGINX instead. And when Caddy was released developers started using that because NGINX was confusing by comparison. I suppose it’s a good sign for developer experience if applications are becoming less confusing.