WalT: a Reproducible Testbed for Reproducible Network Experiments

Experiments involving wireless networks are difficult to run. We have a choice between using specialized platforms (like ORBIT, w-iLab.t, IoT-LAB, and others) or setting up some ad hoc testbeds. Specialized platforms make the experimentation task easier and offer good support for experiment repeatability. However, they only offer operating conditions that are far away from real-world deployments: nodes are usually distributed on a regular grid in one large room, and reproducibility in varying environmental conditions is very limited. As the performance of wireless networks strongly depends on their topology, environment, and operating conditions, validating protocols in artificial conditions may not lead to meaningful results, and reproducibility is de facto limited. Moreover, researchers do not have physical access to the equipment so debugging is tedious and some measurements like fine-grain energy consumption may be impossible. Ad hoc testbeds can help validating a given protocol, but they do not scale and it is difficult to reproduce results since most of the time a precise specification is missing.

We have developed WalT, a reproducible platform to run reproducible experiments. Our goal was threefold: (i) easily setup a platform to develop, debug and make preliminary validation, (ii) allow deployment and control of experiments on a larger scale, (iii) provide a way for others to deploy exactly the same experiment in a different environment, to challenge reproducibility and also guarantee a suitable starting point and adequate repeatability when comparing with other proposals. It follows an approach that lies somewhere in between specialized platforms with rigid and limited topology, and ad hoc testbeds. WalT nodes are single-board computers on which users can deploy their OS (filesystem, kernel) packaged as a docker image for easy customization and sharing.

A WalT platform provides:

• full remote control over nodes: rebooting, remote shell sessions, deploying OS images,
• management of node OS images: clone from the docker hub, modify locally, and publish images,
• log management: means to collect, store, and query experiment logs and event traces are provided,
• automated discovery of the platform topology.

WalT presents the following advantages:

• it can be easily replicated at the required deployment places,
• it is cost-effective, built from inexpensive components running open source software,
• it is lightweight, it can serve for a portable demo or as a development platform in an office without a tedious deployment in a whole building,
• it is adaptable, you can use it for sensor networks or experiments with IEEE 802.11,
• it is easy to install and use.

Besides supporting reproducible experiments, WalT also enables the emergence of reproducible platforms-researchers can set up their WalT platforms to validate the results of others at different places or environments.

All WalT resources are publicly available: forge project, sources, and docker hub.