Experimental design

Good experiments are build upon good experiment design. Before you start experimenting, walk by the topics below.

Intended for: BSc, MSc, PhD

1. Choose test environments (unless your work is fully theoretical).  

Take this step seriously, because once you start working on an environment it will take quite some time to get results, and this time should be worth it. In general, you want the environment to be:

2. Choose baselines. 

When you experiment with you new method, you want to find out whether it works better than alternative available approaches (a 'baseline'). 

3. Choose your evaluation criteria. 

Although the evaluation criterion is often straightforward, do not forget to think about it in advance (or your experiments will be worthless). 

4. Keep comparisons fair. 

Make sure to keep the comparison between methods fair. This is very important, and (unfortunately) very often ignored. 

5. Decide how to challenge your method (optional). 

Apart from comparing your method to other approaches (baseline), you also want to figure out how your method works. Two types of questions/approaches are common: 

TO DO: Type of experiment: proof of concept, insight, comparison. 

6. Choose your visualizations. 

A final crucial consideration is how you will visualize your experiments. Think about this in advance, because you will have to log the right data! (Although often new ideas about visualization come up during your experimentation cycles.)