Testing and embracing change

My client can only be described as "Agile curious" rather than gung-ho disciples of our lord, Kent Beck. So, it was nice to see an email circulated on the company-wide technical mailing lists asking for help in writing automated regression tests. The author asked about the wisdom of serializing the output of all tests into XML and comparing them between runs.

I almost choked on my coffee. I worked on a project that did this last year and it was not pretty. So, I emailed him how using serialized objects in tests had worked for us:

1. It was fragile. Even minor refactoring of the Java objects would break the tests.

2. It didn’t scale. As the number of XML files that needed to be maintained increased, productivity declined. (At one point, we were maintaining about 20 megs of serialization XML.)

3. Java developers tend not to like maintaining large XML files as their Java code changes - so some refactoring of the XML was, shall we say, less than diligent...

4. Serializing Sets (or any other collection where the order in which the elements were serialized was non-deterministic) could break the tests. For a time, some developers were hacking the problem domain classes to make the order deterministic just so the tests would pass.

To this last point, he replied that XML Unit could handle this non-determinism. However, I think the first three points were more than enough to frighten me from doing that again.