When Tomcat, HTTPS, and Excel collide

One of the main reasons that we started publishing a blog at YD was to try to give back to the community when we Figured Stuff Out. Here’s one such story about a strange interaction among Excel, Tomcat, and HTTPS.

Buggy Behavior

When our users tried to open an .xls file (via Internet Explorer into Excel) that originated from our application, they would receive the popup below stating “The Web site you want to view requests identification. Please choose a certificate.” Even more unfortunately, Excel would display the popup 3-20 times before they could get to the data at all.

Certificate Error Image

Needless to say, this was annoying our customers.

We were totally confused and at the time we could find no explanation of this behavior on the web. No other content was exhibiting this sort of problem while originating from our application. HTTPS via Apache Server was working fine. We only had two other clues:

  • We could not recreate the problem by connecting to Tomcat via plain HTTP. (Makes sense given the message.) Of course, unencrypted HTTP was not an option in production.
  • If you chose to Save the file when prompted rather than to Open it, you could use it later with no annoying popups.

Wait, WebDAV?!

So we started some desperation debugging and fired up Ethereal (now Wireshark) to sniff the packets coming from Excel. We then noticed that Excel was sending several OPTIONS requests (one for each sub-directory of the file output) back to our server once Excel received the file. Using cURL, the most incriminating request/response looked like this:

user@server:~> curl -i -X OPTIONS https://app.example.com/APP/
HTTP/1.1 403
Date: Wed, 20 Jun 2007 15:36:26 GMT
Server: Apache/1.3.34 Ben-SSL/1.57 (Unix) mod_jk/1.2.23
Allow: GET, HEAD, POST, PUT, DELETE, TRACE, OPTIONS
Transfer-Encoding: chunked
Content-Type: text/plain

At this point, our guess was that Excel was trying to do some WebDAV “cleverness” by checking whether the file was some sort of WebDAV resource. Excel seemed to descend the “directory tree” of the .xls file looking for any WebDAV features listed in the OPTIONS. (We guess.)

Admittedly, despite the cURL output above, we don’t really support PUT and DELETE. Also, 403 is an inappropriate status to be returning for an OPTIONS request. On the other hand, our allowed HTTP request types are pretty standard, and didn’t include any of those fancy WebDAV ones.

We found that the same request directed to the normal URL of our application gave a more sensible response:

user@server:~> curl -i -X OPTIONS https://app.example.com/APP/main
HTTP/1.1 200 OK
Date: Wed, 20 Jun 2007 15:36:32 GMT
Server: Apache/1.3.34 Ben-SSL/1.57 (Unix) mod_jk/1.2.23
Allow: GET, HEAD, POST, TRACE, OPTIONS
Content-Length: 0
Content-Type: text/plain

Altogether, something fishy was going on…

Tomcat Trickery

Why the two different OPTIONS responses? A quick look at our web-app configuration (web.xml) showed us that we had nothing mapped to the root web-application path, https://app.example.com/APP/. All normal traffic to our application was intentionally sent to https://app.example.com/APP/main instead. Therefore, requests to https://app.example.com/APP/ were sent to the default servlet defined in $CATALINA_HOME/conf/web.xml.

We hadn’t changed the default server servlet since installing Tomcat, so the requests were being handled by org.apache.catalina.servlets.DefaultServlet, which is by default set to “readOnly” mode. We tried looking in the
source code for DefaultServlet
. The code was specifying a 403 error for PUT and DELETE when in “readOnly” mode. Unable to glean much more information from the source code, we then made the logical leap that something similar must be happening for OPTIONS. (Of course, it also doesn’t make sense for OPTIONS to report that PUT and DELETE are allowable when in readOnly mode, but that is a complaint for another time.)

Finally, a Fix

This left us with two options to eliminate the 403 response:

  1. Write our own default servlet that returned a 200 status and the correct request methods (POST, GET, OPTIONS, HEAD, maybe TRACE) for the OPTIONS call.
  2. Provide a servlet mapping to the root web-app directory that then gave correct OPTIONS output.

We opted for the second path. (No need to add extra jar files and configuration to every deployed Tomcat instance. Plus, the DefaultServlet was not terribly extensible.) We ended up redirecting all requests to the web-app directory to our standard servlet, which already had correct OPTIONS behavior.

Hope that helps y’all.

Pizza Box Management

A sterotypical “enterprise software” project: a team of 50 consultants imported to do the work of ten, politics everywhere, a series of death marches, unclear requirements for replacement of a legacy system that no one understands, and too many managers all looking for a reason to exist. The top customer suit calls in representatives from the consulting firms doing the work (of course, there’s not just one contractor – that would be too efficient) and delivers a lecture on hard work. If the developers would just work harder, he claims, they could catch up with the (completely unrealistic) schedule. “You can’t be working nights and weekends like you should,” he says. “I don’t see any pizza boxes when I arrive in the morning!”

The story is true, and I was there when it happened. Brits are supposed to be masters of irony, but this chap must have had his detector switched off. I guess he would have been happy to breeze in at 0900, tie immaculately pressed, and get a bleary-eyed “hello” from a group of unkempt developers covered in tomato-sauce stains after an all-nighter – and the contrast was completely lost on him. And that’s not to mention the defects those developers would surely have been creating during their crunch time.

Thankfully, at youDevise we don’t practise pizza-box management. Developers sign up for a reasonable set of tasks in each iteration and leave themselves enough slack for learning new skills or removing productivity drains – for example, some of us are learning Scala and others are trying out Selenium RC.

Some folks do like working in the evenings or from home, but we have a firm policy against checkins after midnight and discourage overtime of any kind. On the occasions where a late-night or weekend task is required, we ensure that the person affected takes comp time to catch up on rest (this affects operations staff particularly).

By the way, back to that consulting project – the next time we did order pizza for an evening session, I took the boxes upstairs to the suit’s desk and piled them on his chair. The pile stretched most of the way to the ceiling and was deeply satisfying. We didn’t get any more lectures about hard work for the rest of the project.