- TDD didn't help. By now it should be obvious that TDD is a complete waste of time because it didn't uncover this obvious integration issue. There's no justification for TDD.
- The Unit Testing framework didn't help. It's a completely blown unit. Unit testing is oversold as a technology.
- Reliance on "testing" is stupid. There's no point in even attempting to "test" software, since it still broke when we tried to deploy it. Testing simply doesn't uncover enough problems.
Clearly, we need a Bold New Process to solve and prevent problems like this.
- Hundreds of unit tests pass.
- A suite of a half-dozen "integration" scripts (over a dozen TestCases) pass.
- Real Apache deployment fails because I couldn't figure out how to get mod_wsgi to pass the HTTP Authorization header. Even though it's clearly and simply documented. [I was busy focusing on Apache; mod_wsgi solves the problem handily.]
What I did was copy a page from AWS and put the digested authentication information in a query string. In one sense, this is a huge change to the API's -- it's visible. In another sense this is a minor tweak to the application.
- I could extend the test script for the auth.rest module to include the new authentication-via-query-string mechanism. Having tests that failed made is really easy to refactor and subclass until the tests passed. Then I could refactor some more to simplify the resulting modules.
- I could rerun the unittest suite, including the various "integration" tests (tests that had everything by Apache) to be sure everything still worked. Believe it or not, there were actual problems uncovered by this. Specifically, some tests didn't properly use the web services API library. The library had changed, but was mostly backwards compatible, so the tests had continued to work. The latest round of changes broke backwards compatibility, and some tests now failed.
- Despair did not set in. There were issues: sales folks were in total panic because the whole "house of cards" architecture had collapsed. A working test suite makes a compelling case that the application -- generally -- is still sound. We're just stumbling on an Apache deployment issue. In one sense, it's a "show stopper", but in another sense it's just a Visible But Minor (VBM™) hurdle.