CONCLUDING COMMENTS

The second phase of the testing programme has been successfully completed. In studying the results generated in Phase 2 it is important to note the following points:

1. The results generated and comments made only refer to the software actually used in the trials. This should not simply be taken to mean the product name but also the release number and version number of the software.
2. Only the SMARTFIRE SP participated in Phase 2.
3. Only three test cases were selected for Phase 2, these were the radiation test case 2000-1-5, the Steckler room fire case 2000-2-1 and the LPC fire case 2000-2-5.
4. While the Phase 1 simulations did not make use of the most sophisticated physics models available in each of the SPs, the Phase 2 simulations are intended to explore the benefits of using more sophisticated physics models and finer computational meshes.
5. The series of trials undertaken in this project should not be considered to be definitive. They have been selected as a basis for exploring the potential of the benchmarking process. It is intended that additional tests should be added to the suite of test cases.

From the Phase 2 analysis it has been demonstrated that the SMARTFIRE SP is capable of producing improved results over those predicted in the Phase 1 testing regime. The Phase 1 testing regime was essential to allow comparison between different computer codes without the bias of the user or specialist features that may exist in one code and not another. In Phase 2 the user was free to perform the test using the best modelling features available in the code to best represent the scenario being modelled. In this way it was hoped to demonstrate that in addition to achieving a common minimum standard of performance, the SP’s were also capable of achieving improved agreement with the experimental or theoretical results.

Predictions for the radiation test case (2000-1-5) using the SMARTFIRE multi-ray radiation model with 24 rays, showed considerable improvement over the results generated in Phase 1. The results from this simulation indicate the greater inherent accuracy that the multi-ray radiation model has over the simpler six-flux model. It is important to note that the greater degree of accuracy offered by the multi-ray model may not manifest itself in producing more accurate fire predictions. Whether or not the multi-ray radiation model will make a significant difference in a fire simulation depends on the nature of the case being examined.

In the Phase 1 simulations, all the SPs predictions for the Steckler room fire case (2000-2-1) failed to accurately reproduce the measured temperatures, but successfully captured the overall trends. The results for Phase 2 showed that considerable improvement could be achieved by a more sophisticated treatment of the wall boundary conditions and more accurately representing the material properties. While further improvement could be achieved through the use of the multi-ray model and mesh refinement, these were insignificant in comparison.

In Phase 1, it was not possible to generate converged solutions of the LPC-007 case (i.e. 2000-2-5) beyond 300s. This was thought due to the nature of the boundary conditions selected for Phase 1. In Phase 2, with a more sophisticated treatment of the wall boundary conditions - which included a heat loss calculation - it was possible to generate converged solutions for the entire duration of the experiment. While errors in the numerical predictions persisted, the numerical predictions were able to reproduce most of the observed trends in the experimental results.

In studying the outcome of the Phase 2 test cases, it is clear that by activating sophisticated physics models, the SP tested was capable of generating improved predictions in all of the cases examined. While this may seem an intuitively obvious result, it is a necessary demonstration of the capability of the fire modelling tool that this can be done in a measurable and reproducible manner.

Furthermore, these results should not be treated in isolation but taken within the context of the Phase 1 findings. A significant conclusion from the Phase 1 predictions was that within the limits of the Phase 1 testing regime and taking into consideration experimental inconsistencies and errors, all three SPs were capable of producing reasonable engineering approximations to the experimental data, both for the simple CFD and fire cases. With the completion of the Phase 2 testing, this statement is somewhat strengthened - at least for the SP tested in Phase 2.

The concept and testing protocols developed as part of this project have been shown to be a valuable tool in providing a verifiable method of benchmarking and gauging the basic and advanced capabilities of CFD based fire models on a level playing field. To further improve the capabilities of the approach, it is recommended that additional test cases in the two categories, theoretical and experimental, be developed and several of the fire cases be refined.

In addition, a modification to the testing procedures is suggested that would reduce the burden and cost of performing the testing by the test organisation. While all of the test cases using all of the codes were run by a single organisation – in this case FSEG – the code developers also were requested to run an independent selection of the test cases as specified. This was necessary to verify that the results produced in this report are a true and fair representation of the capabilities of the various software products under the specified test conditions. This has proven to be quite useful as it brings the developers into the benchmarking process and it eliminates issues concerning fairness and biased reporting of results. However, if this process is to become a mandatory requirement, the testing organisation will have a considerable amount of work to do if it is to run every software product and its various upgrades through each of the test cases. In order to reduce the cost of testing, it is suggested that the test organisation should only perform the random testing and require the software developers to run and submit all of the test cases.

It is finally recommended that the principles and procedures developed in this project be adopted in some form as a quality measure of fire modelling software that is intended for use in the U.K. for design purposes.