With the rise in popularity of the Internet of Things in all kinds of different application scenarios, various middleware solutions have appeared with different use-cases and optimizations in mind. The design space for any specific deployment is thus increasingly large, but little objective support exists to help choose the best middleware for each use-case. From this stems the need to evaluate how different IoT middleware solutions perform in different use-cases. Measuring the performance of IoT middleware in a way that 1) provides common ground among experiments, and 2) makes it easier to integrate new IoT middleware in the benchmark is not straightforward. In this paper, we propose a generic architecture for comparing the performance of publish/subscribe middleware, develop a tool that implements this architecture, and show the benefits in time and effort that can be reaped from our approach. We further validate our approach by using the architecture and tool to benchmark different middleware solutions, taking lessons from the changes necessary to support new middleware, and attempting to quantify the effort through lines of code and to qualitatively assess code structure similarity.