Introduction

This report is a first draft version of deliverable D1.1b for the validation unit contract ELTA/0209/VAL-1, and is intended to be read as a parallel report to deliverable D1.1a which describes the validation of lexica. The theoretical framework underlying this work was presented in and ; the present document assumes an understanding of the issues discussed in those reports.

As noted in those reports, and elsewhere, the word validation is generally used (synonymously with evaluation) to define the process by which the usefulness or applicability of a given resource for a specific purpose may be determined. It may well be that a resource suitable for one purpose is of no use for another; this affects in particular resources such as language corpora which typically have many users, and may have applications often entirely unanticipated by their original creators. Rather than attempt to define validation procedures appropriate to many different applications or scenarios, the approach taken here is to define objective criteria by which data essential to such an evaluation can be assembled, by reference to a check list of identifiable characteristics. The intended outcome from a validation process should thus be an accurate description of a corpus, using standardized terminology; it is up to the individual user to determine the extent to which a corpus is fit for a given purpose.

We use the word corpus throughout as a short form for written language resource, recognizing however that many corpora include both spoken and written materials. There is consequently some overlap between the procedures and requirements described here and those defined in .

The bulk of this report consists of a checklist of features and properties which the evaluator should investigate in the resource under consideration. In some cases, specific recommendations are made; for the most part however, the only requirement is that the evaluation should consistently attempt to ascertain whether or not information about the feature concerned is available, and if so, what it is.

Evaluation can (and probably should) be performed both as a part of the creation of a resource, and subsequently by an independent assessor. As noted above, the creator of a resource may have a different set of objectives from subsequent users of it, which suggests that it is probably a good idea not to rely solely on evaluations carried out during resource production. On the other hand, only the creator of a resource may be able to supply some of the less obviously apparent information required for a complete description of a resource. The two stages are thus complementary, and wherever possible both should be attempted.

An appendix to the report contains a summary checklist of questions which may be used to guide the evaluation process.

Scope of Corpus Evaluation

The model proposed here derives from that underlying most pre-existing corpus standardization work (most notably TEI and its applications in Eagles and CES). As descriptive standards, these standards largely address issues about corpus content, which may be thought of as the identifiable intellectual content of a resource, considered independently from any particular physical instantiation of it. Since ELRA is adsditionally concerned with the deposit and distribution of physical corpora, we have found it necessary to extend the descriptive model to include consideration of corpus components, that is constituents and status of the carrier media used for a specific instance of the written resource.

Under this latter heading we consider such aspects as the way the corpus is or may be stored (on which media); where alternate versions exists, which of these are to be regarded as canonical; what components make up the complete resource (disks, manuals, documentation etc); how they are to be unpacked or installed, etc.

When considering the content of a corpus, it is convenient to distinguish: metadata: descriptive information about the corpus contents and included with it structural components identified within the corpus interpretive annotation provided with the corpus

The distinction between structural and interpretive reflects the fact that most people seem to categorize an analysis such as this is a paragraph (structural) differently from the formally equivalent judgement this is a noun (interpretive). A similar distinction underlies the notion of level of annotation as exemplified by (inter alia) the Corpus Encoding Specification (Ide 1998), where the distinction is further justified by the observation that the addition of so-called structural markup is generally easier to automate than that of interpretive markup, since the latter (almost) invariably requires human judgement and knowledge, while the former rarely does. Particularly in the case of textual markup, interpretative judgements tend to be more controversial than structural ones, if only because the latter relate to aspects of a text which are accepted as intrinsic to its substance by the community of text readers. Structural interpretations form part of the contracts of literacy (Snow and Ninio, 1986) which form the precondition of a text's recognition as meaningful by the members of a particular community of readers.

Markup, whether introduced into a corpus to identify structural or interpretive components, can only properly be validated with reference to an abstract model of some kind. For structural features, the model will comprise textual components and features which may be either entirely intuitive and common sense based, or expressed in terms of some consensus-based model such as that of the TEI. Interpretative markup may be similarly theory-free (see, for example, Leech 1993, but it is more customary to define it with reference to some explicitly stated analytic model, and hence to facilitate both automatic validation of the corpus itself (to check that it is valid in its own terms) and comparison of two corpora using different markup schemes derived from a common abstract model.

At the technical level, structural markup is generally embedded within the corpus, while interpretive markup need not be. There is an increasing use of stand off markup, which represents particular kinds of annotation as distinct documents comprising pointers into a corpus and associated annotations of various kinds. Such annotation maps require careful validation with special purpose software.

Corpus Components

Corpora may be instantiated using a variety of media (disks, CD, DVD, tape) etc. or online. It is essential that the validator can ascertain whether or not the media delivered form a complete and integral copy of the corpus. To make best use of a corpus, it is also generally necessary to have access to supporting metadata, code books, explanatory documentation, licence agreements, etc. As far as possible all such supporting material should be included in digital form along with the corpus itself. Where it is supplied in paper form, this should also be clearly indicated.

To facilitate this essential step of the validation, we propose that each corpus should be accompanied by a corpus manifest. This is a list of all the physical components which should be assessed together, and which together constitute the corpus and the support environment needed to make best of use it, indicating for each one its type (paper document, computer file, audio or video recording, etc.) its carrier (computer file name and location, document title etc.) its status (integral part of corpus, descriptive metadata, associated annotation, documentation, etc.) for digital components, the storage format (character encoding, binary format, record structure, etc.)

Where more than one version of a given component is supplied, their status should be clearly indicated. For example, it is sometimes convenient to include different versions of a computer file using different levels of encoding, or different sets of conventions. In such a case, the corpus creator should select one such version as primary and indicate that the others are alternates, so as to maintain corpus integrity.

Where a corpus consists of very many computer files they will generally be organized into a hierarchic structure of some kind, and may be supplied in compressed form. Any software needed to decompress or to make accessible the corpus must also be specified in the corpus manifest.

The digital version of certain supporting documents (e.g. word-processed files, spreadsheets, databases, audio or video files etc.) may be supplied in non-standard or proprietary formats which may require the use of specific software. As far as possible all such dependencies on external software should be avoided: the primary components of a corpus must be processable on any platform, using generic standard interfaces and open architectures.

The following formats are recommended for processing of the most commonly used media types: text filesXML or SGML conforming to a standard or supplied DTD or schema audioMP3, WAV videoMpeg, Quicktime image filesPNG, JPG

Where XML is used for text files, the character encoding should be represented in UTF-16, UTF-8 or (exceptionally) one of the ISO 8859 character sets, as indicated in the encoding attribute of the XML declaration. If different files use different encodings, each should contain such a declaration. Where SGML is used, an appropriate SGML declaration defining the character encoding employed must be included in the corpus manifest.

Where XML or SGML is used for text files, a DTD or Schema must be supplied or referenced, against which the text files can be validated. For written corpus resources, the most appropriate DTD is likely to be an application of the TEI, such as the XCES, but any adequately documented vocabulary may be employed. Both the DTD instance and its associated documentation should be supplied in the corpus manifest.

Text files supplied as alternates to corpus components such as documentation may use any appropriate format, though open formats such as HTML or XML are to be preferred above proprietary word processor or database formats. In particular, where database or spreadsheet data is used to document some part of a corpus, a printout of the material should be suppled and the data should be exported into some open format such as XML or comma-delimited files.

Metadata

By "metadata" we mean...

Multiple levels of metadata may be associated with a corpus. For example, some information may relate to the corpus as a whole (for example, its title, the purpose for which it was created, its distributor, etc); other information may relate only to individual components of it (for example, the bibliographic description of an individual source text), or to groups of such components (for example, a taxonomic classification).

In the following lists, we have supplied the TEI/XCES element corresponding with the topic in question. This is not meant to imply that only corpora conforming to TEI/XCES standards can be validated, of course, but rather to add precision to the topics addressed. That said, obviously a corpus which is conformant to those standards will be much simpler to validate than one which supplies equivalent information in some other manner.

Validation of Structural Information

Validation of Information about Interpretive Annotation