ZFS Validation & QA
Sydney Vanda and John Salinas of Intel will give a talk on ZFS Validation & QA at the 2016 OpenZFS Developer Summit:
One of the challenges of creating ZFS features and releases is how to know if they are stable for end users. Earlier this year, ZFSonLinux landed the ZFS test suite from Solaris that Delphix was using for Illumos development. In order to both land new features and create releases, it is important that adequate tests exists to validate both the new functionality as well as existing code. One of the challenges in ZFSonLinux is that tests that work in OpenZFS are disabled. However, many of these tests can be enabled as they do work, while others require minor Linux fixes to make them work. This presentation will show how some tests have been modified so ZoL can start to make full use of the test suite. A few people at Intel have been looking into the test suite and evaluating tests, adding patches for improvements as well as Linux-equivalencies, and working on integrating the test fixes back into upstream ZoL.
Throughout this process, many notable discoveries have been made about the test suite that are worth discussing to the greater community. This includes the test suite (designed for testing production software) currently only being able to run tests using loop devices, for which our team has been working on a patch for real disk support in Linux. The difficulty in landing new features such as this is what to do with intermittent failures on the build bots and also test inconsistencies. A list will be presented to show what tests have intermittent failures and how to test these by hand to see if they are really issues or not. Some sort of standardization for the test suite might be useful for future development, and can include such things as configuration files, abstraction of functions for test set-up (setting individual disk partitioning and device directory for example), adding early failures for system commands not present, and also overall test validation and usefulness (noticing many areas of discrepancies and also false positives).
The goal of this presentation is help improve the quality of ZFS by porting existing tests, adding new tests, and improving tests that lack validation steps. Another area of concern is adding support for multiple platforms. One challenge that OpenZFS faces is that even it gets behind what is in the Delphix OS. This presentation will show how we have audited to see where both Open ZFS and ZoL are behind and how we can make them current. Overall, reconciling OpenZFS and ZoL, and just knowing what is in OpenZFS, Illumos, and ZoL is the big challenge that needs to be addressed.