The Internet of Things (IoT) emerged simultaneously in various research fields and application domains, leading to the appearence of a multitude of hardware, software technologies, horizontal platforms and data models. Lost in this diversity, adoption decisions are made to a large extent based on familiarity with a technology, or because there is a broad community support for it. In this context, bechmarking different technologies at different horizontal levels would provide a more solid justification for the adoption of a specific technology in a specific context. In this paper, we reflect on our previous work on benchmarking publish-subscribe IoT platforms as a middleware for IoT applications, reporting lessons learned and identifying challenges, thus contributing to the discussion on open topics and relevant downstream work.