Abstract
Dear Editor,
I thank you for your letter and in many ways support it, in that I think more attention needs to be drawn to the many good things our industry does and the care that many take to think about modelling. It is good to see debate.
I will try to answer each of your points, but not necessarily in the order you presented them.
This paper is about literacy, not about model validity, either at the level of the software or the modeller. I can quite understand that having ones ‘literacy’ questioned could be challenging; however, we have applied the term in a similar way as we see Energy Literacy applied when studying energy savings in the home. In such studies people are asked to rank the annual energy consumption of items. This is a common and well-established method. We mean no more by it than that. When doing such work interesting questions include: (a) whether the ranking is consistent with the expected energy use of the items; (b) whether people rank in approximately the same order as each other; and (c) whether a group of professionals working in the field have a different ranking to the public/random. When we started the work, we thought we knew what the answer would be – professionals would rank the items in roughly the same order and that order would be roughly the same as the model. We were very surprised at the results, particularly as the building was such a simple one. As emphasised at the end of your letter, you say you are most concerned about the methodology that lies behind the paper; however, in the letter you only hint at ways we could have improved what we did, not I think found any fatal issues. I think we have presented what we did honestly and explained it well and set out our definition of literacy clearly. But I agree, given more resources and time we could have dug deeper. The paper would be very easy to replicate. All that is needed is a monitored building and a survey. We would encourage others to do so. It would have been interesting to have used other software; however, we used the one most people in the sample used. I would be surprised if the order changed that much using a different engine, but as we have not done this I cannot be sure. I think it would be rather worrying all round, and the industry in deep trouble, if the order were heavily influenced by the package. R2 is still a common measure, possibly the most common, so I think we are reasonably safe in using it, even though it is not perfect. Looking at the statistical element of the paper, I know many papers, possibly most, have taken less care with the statistics. The use of clustering analysis/kappa values for one shows this. We used a real monitored building to ensure our model was reasonable and validated, but we understand no model is perfect – however, if a thermal simulation of a simple house based on a real building with real monitored data is that unreliable, the industry again would have to question itself. One can find many papers out there that do not take this much effort, many in journals or conference proceedings I would suggest you know well. One hundred and eight people answered the questionnaire; we have all seen others publish work with small samples – very few have used more. We had a psychologist on the team as we were undertaking a survey and working with human subjects, again not all teams make this effort. I am happy to apologise for the possibly negative vibes in the press; however, none of us can control the press, nor would I suggest should we seek to do so. The paper does not aim to criticise modellers in any way, it criticises people like me, who are part of the educational chain experienced by those now modelling. I am glad you support us in the suggestion that improvements could be made to building physics education. As I am one of the educators, I am very much pointing the finger at myself. Being I was as surprised as anyone at the results, I have truly been kicked in the teeth, and feel humbled. Although any list of parameters could be considered arbitrary, I would suggest they do a reasonable job of covering the space from those that might be expected to have a great impact and those little; it was this range we were after. The question about using a house rather than a commercial building that most modellers would be accustomed to is a good one. This is something we agonised over for weeks. The house meant zones were less of an issue; we could monitor the building at reasonable cost and set the problem out simply in the questionnaire. It would be fantastic to see others with more resources tackle a commercial building in the same way. We certainly do not think what we have identified is ‘where modelling is going wrong’. We think we have identified one under-discussed reason for concern, amongst many. We also think that it might be easier than some others to set right, but cannot prove it. Although it is true that many modellers are not brought to the table until late in the day, this is not always the case, and many of the best firms are much more integrated than this, especially those working on very low energy designs. This is where literacy (as we define it) is key. In early design meetings, it is simply not possibly to respond with ‘let me go and model that and I will have the answer for you next week’. Rapid diagnostic skill is key at the early stage, and it is the ability to deliver this, which is what the paper is all about. It is not about the standardisation of models, or ensuring all modellers produce the same results when using software. It is about what building models know and can rapidly access to bring to the table, or act on when modelling in a time constrained commercial environment. We would have liked to have discussed the background to the commercial setting more, the history of modelling, the legislative process around the world and the efforts we are all going to improve things, but the paper was already far longer than journals normally accept. I was rather surprised by the suggestion that the paper might not be robust, nor good science, and that Building Services Engineering Research and Technology (BSERT) had failed to ensure robust peer review. A lot of care and effort went into this work, beyond what is often found, and as I think I have laid out above, the work can hold its head high compared to other published works in terms of methodology and analysis.