These are some preliminary ideas:
We can create some types of "recognition-based benchmarks". For example, give 1,000 + 1 images. Among the 1,000 images, one of them is the closest to that particular image. The "closest" will be defined by a committee but should be commonly agreed by most people. If 1,000 are too few, we can easily increase that to much more. ILSVRC (http://www.image-net.org/challenges/LSVRC/2013/#data) has a lot of images.
The participants are divided into two groups:
- customer hardware
- commodity hardware.
We may further divide the groups to
- with network connections (can offload computation).
- without network connections (cannot offload computation).