Discussion

The purpose of these scenes is not to break as many algorithms as possible, it is to exercise these algorithms. Passing these tests doesn't mean all real environments are easily and accurately rendered. All of these scenes could be made more difficult. As the area of the diffuse secondary luminaire in the caustic scene gets smaller, many algorithms get much less efficient. If the mirror cube in that scene is replaced by a sphere, some approaches break. These simple changes show where different algorithms break in a quick and easy manner.

One of the obvious things missing from the description of the test scenes is the runtime it took to generate the images. The scenes we present are not realistic. Algorithms should probably not be optimized for these cases. Some of the scenes create situations where efficiency issues show up, such as the 101 lights in the emission test. This can serve as a good reminder of some of the common assumptions on scenes.

There are a range of useful levels of adoption for a database of test scenes. These can be characterized as:

• Everyone uses these models, and agreed upon benchmark images are regularly used for testing and comparing accuracy.
• The repository grows with the addition of test scenes from other work, allowing comparisons between algorithms, but no single set of scenes becomes standard.
• The community uses the test names and similar models to discuss strengths and weaknesses of existing and new algorithms.

The first is pretty optimistic, both in assuming a universal standard and in assuming that this set of models captures every aspect worth testing in current and future algorithms. The second seems much more likely, and becomes especially useful with a maintained repository of models and data. Even the last is beneficial, however, as it allows a much less ambiguous way of describing cases where algorithms do or do not work.