An ISO Standard Guards the Ada Hen House
Ada has become the first programming language to establish the process for testing compilers’ implementation as an international standard. Ada compilers can be labeled as “certified as conforming to standard Ada” (informally called “Validated Ada Compilers”) only if they follow a specific route of testing and certification. In October 1999, the International Organization of Standards (ISO) approved the prescribed path in Ada: Conformity Assessment of a Language Processor (ISO/IEC-18009:1999).
The standard provides other programming languages with a compiler certification model to follow. If the C community, for example, needs to distinguish between compilers that translate standard C and those that permit nonstandard extensions or that fail to handle the entire language, the new ISO standard process for testing Ada compilers has blazed a trail to that goal.
The Ada community will not notice much change in the testing procedures except in the vocabulary (for instance, validation versus conformity assessment) and in a streamlined bureaucracy. The new standard codifies existing practices for testing Ada compilers that tool vendors have followed since the early eighties.
The Ada community is well-qualified to pioneer the standard. Ada is unique in insisting that software tools’ conformity to the language’s international standard be an integral part of marketing compilers. Facilities all over the world have tested standard Ada implementations for over 15 years.
The language’s designers realized that compiler testing is critical to Ada’s success. Testing compilers for translating “proper Ada” actually predates the first Ada standard of 1983. The designers intended the language-initially used in large embedded systems–to be dependable, reusable, portable, maintainable, and legible. Many different programmers working toward the same goal had to simultaneously write, reuse, and integrate software components. After being fielded, the systems were assumed to have a long life during which they would port to many different hardware systems.
Ada 95 continues this emphasis on high reliability. The language is used for most modern aircraft fly-by-wire controls and new air traffic control systems, as well as rail transportation systems and satellites. Ada has been used in the Chunnel and the subway systems in Paris, Hong Kong, London, and New York; for manufacturing Volvos in Sweden; and for controlling steel mills in West Virginia. The language is used for smaller applications, such as network switching systems and e-commerce applications, where adaptability and time to market are important factors.
To realize its purpose, Ada must execute reliably across platforms and national borders. In other words, it must be standardized. If Ada compilers translated dialects, not only would safety-critical verification of the software be difficult, but also the software would not be reusable, universally legible, or as maintainable. Today, the Ada Conformity Assessment Authority (ACAA) safeguards the procedures for testing Ada compilers’ translations against the ISO standard.
A history of Ada conformity assessment
The original big user of Ada, the US Department of Defense, first tested compilers’ conformity to standard Ada in 1984. In October 1998, the DoD handed conformance testing over to an industry group, the Ada Resource Association. The ARA consists of numerous Ada compiler and tool vendors who work together to promote and support Ada’s use in the commercial and government marketplace.
Having Ada vendors controlling the process of testing Ada compilers is very much like having the foxes guarding the hen house. The vendors could easily decide to make the process less rigorous, reducing its value for Ada users. Therefore, to safeguard the process, the Ada community agreed to make the testing process an international standard. This standard, ISO/IEC-18009, outlines the basic conformity assessment process. The standard does not specify such details as what to test or how to distribute the tests, but it does specify what is allowed and disallowed in testing.
Why is a separate authority needed?
The conformity assessment standard defines an independent agency–the ACAA–to manage the testing process. Although the Ada tool vendors finance it through the ARA, the ACAA’s real boss is the ISO, and its charges are the testing laboratories. Independent laboratories, or Ada Conformity Assessment Laboratories (ACALs), test the compilers. Because compiler vendors pay testing fees, they could put financial pressure on a lab to successfully complete tests. If a laboratory certifies a processor that does not meet the Ada standard, the testing becomes useless for users. The ACAA reduces this risk by ensuring that labs use the same detailed procedure. The ACAA and testing labs cooperate to develop the detailed procedures, with the ACAA as the final arbiter. This lets laboratories compete on the basis of price and service.
The ACAA enforces consistency by verifying that the ACALs follow the procedures for each completed testing, maintaining the test suite, and handling test disputes. Thus, the ACAA ensures that all labs use exactly the same tests. The ACAA also maintains the single, common list of successfully tested compilers.
The Ada Conformity Assessment Test Suite (ACATS) includes both positive tests, which check that the language’s features work as defined by the Ada 95 standard, and negative tests, which check that the compiler rejects illegal Ada code. It is freely available to everyone from many sources, including the ACAA’s Web site, www.ada-auth.org/~acats.
The value of Ada conformity assessment
Ada users know that conformity assessment is the only objective presale assurance that a compiler implements Ada correctly. They consistently state that independent third-party testing is the most important feature of Ada conformity assessment. Although vendors might have a strong incentive to fudge results, the testing laboratory has verified that the compiler passes the tests. The Ada conformity assessment standard strengthens this advantage by providing an agency (the ACAA) to police the laboratories, ensuring comparable results from different testing laboratories. Moreover, the ACAA and the testing laboratories are independent organizations, minimizing the possibility of collusion. Finally, the free availability of the test suite and test reports also makes test results more reliable, because any interested party can repeat some or all of the testing. This provides yet another disincentive to cheating.
Because the test suite (ACATS) is freely available, all Ada vendors use it for regression testing and to judge the quality of their implementations even before they contract with a lab for testing. This inevitably improves the quality of Ada compilers because the test suite detects many problems, which the vendor eliminates, long before users are affected.
The common list of successfully tested compilers serves two purposes. Not only can users verify the testing of a particular compiler, they can also access the actual test reports. This is a more reliable way to determine a compiler’s test status than relying on a vendor’s information. Second, the list is complete and lets users find all compiler vendors that target a certain processor. Because not all compiler vendors are well-known, the list provides an Ada project manager with the most choices.
The Ada conformity assessment process resembles the open source movement in that the test source code and documentation are freely available, while testing services cost money. The new standard goes further by providing free maintenance to users of the process. No charge is made to a vendor or user that requests a test modification or disputes a test’s results; the ACAA bears the cost. Thus, users face no political, financial, or monolithic corporate obstacles in participating in the maintenance of tests and procedures. Consequently, the test suite is more likely to reflect real users’ needs.
Status
Ada compiler testing continues unabated. Over 50 conformity assessments have been performed in the last two years. The test suite now contains more than 3,600 tests covering the full breadth of Ada 95. The suite expands existing tests and adds new ones as user and vendor needs evolve. New tests focus on recent corrections to the Ada standard-the most likely areas for processor errors.
Software engineers have always been confident of the quality of Ada compilers because the compilers were independently verified. The new ISO standard assures them that the procedure is protected against weakening by vendors. The standard is the product of 15 years of evolution of the process of Ada conformity assessment. The process has evolved to benefit everyone: users, vendors, and testing laboratories.
As demand for reliable software grows, other languages will need to establish procedures for testing conformity of compilers to their standard. Others could adapt the model and experience of the Ada language for other languages, especially where a strong central authority exists, as is the case with Java.
by Randy Brukardt
Randall Brukardt is the manager and Technical Agent of the Ada Conformity Assessment Authority. He has been involved with the Ada language for nearly twenty years, having been lead designer for a popular PC Ada compiler, an Ada 9x distinguished reviewer, and now is one of the editors for the Ada 95 standard. He still is Director of Technical Operations at R.R. Software, Inc, leading development of their compiler and Windows products. He occasionally finds time for travel and photography. Contact him at agent@ada-auth.org.
©2000 IEEE
Personal use of this material is permitted. However, permission to reprint or republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.
