The Java Community Process(SM) Program - Community Resources

What is the Appeals Process?

The process defined by the Spec Lead that allows implementors of the Specification to challenge one or more tests defined by the Specification's TCK. The appeals process should identify:

Who can make challenges to the TCK.
What challenges to the TCK may be submitted.
How these challenges are submitted.
How and by whom challenges are addressed.
How accepted challenges to the TCK are managed.

What is the scope of the TCK?

TCKs are not trivial to produce, so Begin Early! Define the level of assertion coverage desired and plan adequate resources and time. Creating a good TCK will require nearly the same amount of effort and time as the RI.

What is needed for Test Development?

Several levels of compatibility testing coverage:

Signature - checks completeness
Method - checks basic functionality
Assertion - checks required behavior 75 % assertion coverage is a good practical target.
If started too late, you may not even achieve 50%.
Must have signature coverage.
Strive for 100% method coverage as minimum.
Goal should always be to provide as much assertion coverage as possible.

What can you get Free of Charge?

The Sample TCK Coverage Document (TCD) for JCP 2.6, which includes an overview of the TCK Documentation, a description of means used to validate TCK quality, a description of means used to validate TCK coverage, test coverage numbers and justification for the adequacy of the TCK.

Example for a TCD:

TCK Coverage Document for JSR-xxx:
Wombat API (Optional Package for J2SE)
TCK Components
User's Guide
Compatibility Rules
Configuration instructions
Test Suite
API Tests 
Signature Test
Framework code
Configuration Interview
JavaTest TM Harness

Terminology of Metrics
Assertion: A specific statement of functionality or behavior derived from a specification.
A testable assertion is one that can be validated in an implementation by testing.
Test: A binary application (or script) comprised of one or more Test Cases.
Test Case: A single set of test inputs, execution conditions, and expected results
developed to verify an implementation's conformance with a specific assertion.
Assertion Breadth Coverage: Ratio of all assertions tested by at least one test case to
the total number of testable assertions defined by the specification.
Assertion Depth Coverage: For all assertions tested by test cases, the ratio of these
test cases to the total number of test cases needed to completely test the assertions
(assertions may require multiple test cases to adequately verify conformance to the
specification).
Method Coverage: Ratio of methods directly exercised by test cases to the total number
of methods defined by the specification.

Coverage
Total testable assertions: 426 as identified through the use of the Java CTT Spec
Trac tool and hand markup (hand markup was required for the two referenced specifications)
Tests: 304 tests (comprising 627 test cases)
Assertion Breadth Coverage:  70% as measured by Spec Trac and 65% estimated for both
referenced specifications
Assertion Depth Coverage: Approx. 50% (estimated by sampling)
Assertion Breadth Coverage varies by package and ranges from 30% to 95%
Assertion Breadth Coverage of the Wombat data exchange protocol is estimated to be 90%
Method Coverage (static): 94% as measured by the Java CTT API Coverage Tool
Method Coverage (runtime against the RI): 97% as measured by XYZ code coverage utility
API Signature Coverage: 100% for all defined public and protected members and validated
by the Java CTT Sig Test tool

Quality Assurance
TCK was run using representative configurations of the Reference Implementation on
the following platforms:
Red Hat Linux/Sun J2SE v1.3.1 and 1.4.2
Windows 98, 2000, XP/Sun J2SE v1.3.1, 1.4.2
Solaris 9/Sun J2SE 1.4.1
Mac OS 10.2.6/Apple J2SE v1.4.2
Code quality was demonstrated through use of code reviews and inspections
User's Guide was constructed from the Java CTT TCK User's Guide template
Documentation instructions and Configuration Interviews were exhaustively verified
and tested

Justification of Adequacy
As the first major release of this specification, the coverage is good and consistent
with the coverage of similar JSRs at this level of spec maturity. Coverage variance
is due to effects of late spec changes and the availability/cost of test development
resources. Emphasis was given to covering critical spec areas as identified by the
Spec Lead and Expert Group. We plan to increase coverage in weak areas during the
next major specification update release.

Sun's Test Development Tools. Please check http://jcp.org/en/resources/tdk for more information.

The PMO welcomes suggestions and requests from the Spec Leads for improvements of this guide and the process: pmo@jcp.org