What is metadata and why do you need it?

Metadata is usually defined as data about data. The word appears only six times in the 100 million word British National Corpus (BNC), in each case as a technical term from the domain of information processing. However, all of the material making up the British National Corpus predates the whole-hearted adoption of this word by the library and information science communities for one very specific kind of data about data: the kind of data that is needed to describe a digital resource in sufficient detail and with sufficient accuracy for some agent to determine whether or not that digital resource is of relevance to a particular enquiry. This so-called discovery metadata has become a major area of concern with the expansion of the World Wide Web and other distributed digital resources, and there have been a number of attempts to define standard sets of metadata for specific subject domains, for specific kinds of activity (for example, digital preservation) and more generally for resource discovery. The most influential of the generic metadata schemes has been the Dublin Core Metadata Initiative (DCMI), which (in the year after the BNC was first published), proposed 15 metadata categories which it was felt would suffice to describe any digital resource well enough for resource discovery purposes. For the linguistics community, more specific and structured proposals include those of the Text Encoding Initiative (TEI), the Open Language Archive Community (OLAC), and the ISLE Metadata Initiative (IMDI).

These and other initiatives have as a common goal the definition of agreed sets of metadata categories which can be applied across many different resources, so that potential users can assess the usefulness of those resources for their own purposes. The theory is that in much the same way that domestic consumers expect to find standardized labelling on their grocery items (net weight in standard units, calorific value per 100 grams, indication of country of origin, etc.), so the user of digital resources will expect to find a standard set of descriptors on their data items. While there can be no doubt that any kind of metadata is better than none, and that some metadata categories are of more general interest than others, it is far less clear on what basis or authority the definition of a standard set of metadata descriptors should proceed. Digital resources, particularly linguistic corpora, are designed to serve many different applications, and their usefulness must thus be evaluated against many different criteria. A corpus designed for use in one context may not be suited to another, even though its description suggests that it will be.

Nevertheless, it is no exaggeration to say that without metadata, corpus linguistics would be virtually impossible. Why? Because corpus linguistics is an empirical science, in which the investigator seeks to identify patterns of linguistic behaviour by inspection and analysis of naturally occurring samples of language. A typical corpus analysis will therefore gather together many examples of linguistic usage, each taken out of the context in which it originally occurred, like a laboratory specimen. Metadata restores and specifies that context, thus enabling us to relate the specimen to its original habitat. Furthermore, since language corpora are constructed from pre-existing pieces of language, questions of accuracy and authenticity are all but inevitable when using them: without metadata, the investigator has no way of answering such questions. Without metadata, the investigator has nothing but disconnected words of unknowable provenance or authenticity.

In many kinds of corpus analysis, the objective is to detect patterns of linguistic behaviour which are common to particular groups of texts. Sometimes, the analyst examines occurrences of particular linguistic phenomena across a broad range of language samples, to see whether certain phenomena are more characteristic of some categories of text than others. Alternatively, the analyst may attempt to characterize the linguistic properties or regularities of a particular pre-defined category of texts. In either case, it is the metadata which defines the category of text; without it, we have no way of distinguishing or grouping the component texts which make up a large heterogenous corpus, nor even of talking about the properties of a homogenous one.

Scope and representation of metadata

Many different kinds of metadata are of use when working with language corpora. In addition to the simplest descriptive metadata already mentioned, which serves to identify and characterize a corpus regarded as a digital resource like any other, we discuss below the following categories of metadata, which are of particular significance or use in language work: editorial metadata, providing information about the relationship between corpus components and their original source () analytic metadata, providing information about the way in which corpus components have been interpreted and analysed ( descriptive metadata, providing classificatory information derived from internal or external properties of the corpus components () administrative metadata, providing documentary information about the corpus itself, such as its title, its availability, its revision status, etc. (this section)

In earlier times, it was customary to provide corpus metadata in a free standing reference manual, if at all. It is now more usual to present all metadata in an integrated form, together with the corpus itself, often using the same encoding principles or markup language. This greatly facilitates both automatic validation of the accuracy and consistency with which such documentation is provided, and also facilitates the development of more human-readable and informative software access to the contents of a corpus.

A major influence in this respect has been the Text Encoding Inititiative (TEI), which in 1994 first published an extensive set of Guidelines for the Encoding of Machine Readable Data. (TEI P1). These recommendations have been widely adopted, and form the basis of most current language resource standardization efforts. A key feature of the TEI recommendations was the definition of a specific metadata component known as the TEI Header. This has four major parts, derived originally from the International Standard Bibliographic Description (ISBD), which sought to extend the well-understood principles of print bibliography to the (then!) new world of digital resources: a file description, identifying the computer fileIn ISBD, the term computer file is used to refer to any computer-held object, such as a language corpus, or a component of one. itself and those responsible for its authorship, dissemination or publication etc., together with (in the case of a derived text such as a corpus) similar bibliographic identification for its source; an encoding description, specifying the kinds of encoding used within the file, for example, what tags have been used, what editorial procedures applied, how the original material was sampled, and so forth; a profile description, supplying additional descriptive material about the file not covered elsewhere, such as its situational parameters, topic keywords, descriptions of participants in a spoken text etc. a revision description, listing all modifications made to the file during the course of its development as a distinct object.

The TEI scheme expressed its recommendations initially as an application of the Standard Generalized Markup Language (SGML: ISO 8879). More recently, it has been re-expressed as an application of the current de facto standard language of the internet: the W3C's extensible markup language (XML), information on which is readily available elsewhere.

The scope of this article does not permit exhaustive discussion of all features of the TEI Header likely to be of relevance to corpus builders or users, but some indication of the range of metadata it supports is provided by the summary below (). For full information, consult the online version of the TEI Guidelines (), or the Corpus Encoding Standard (). Dunlop 1995 and Burnard 1999 describe the use of the TEI Header in the construction of the BNC

Editorial metadata

Because electronic versions of a non-electronic original are inevitably subject to some form of distortion or translation, it is important to document clearly the editorial procedures and conventions adopted. In creating and tagging corpora, particularly large ones assembled from many sources, many editorial and encoding compromises are necessary. The kind of detailed text-critical attention possible for a smaller literary text may be inappropriate, whether for methodological or financial reasons. Nevertheless, users of a tagged corpus will not thank the encoder if arbitrary editorial changes have been silently introduced, with no indication of where, or with what regularity. Such corpora can actively mislead the unwary or partially informed user.

A conscientious corpus builder should therefore take care to consider making explicit in the corpus markup at least the following kinds of intervention: addition or omission where the encoder has supplied material not present in the source, or (more frequently in corpus work) where material has been omitted from a transcription or encoding. correctionwhere the source material is judged erroneous (for example, misprints) but the encoder wishes to preserve the original error, or simply to indicate that it has been corrected. normalizationwhere, although not considered erroneous, the source material exhibits a variant form which the encoder wishes to replace by a standardized form, either retaining the original, or silently.

The explicit marking of material missing from an encoded text may be of considerable importance as a means of indicating where non-linguistic (or linguistically intractable) items such as symbols or diagrams or tables have been omitted: <gap desc="diagram"/> Such markup is useful where the effort involved in a more detailed transcription (using more specific elements such as figure or table, or even detailed markup such as SVG or mathml) is not considered worthwhile. It is also useful where material has been omitted for sampling reasons, so as to alert the user to the dangers of using such partial transcriptions for analysis of text-grammar features: <div type="chapter"> <gap extent="100 sentences" cause="sampling strategy"/> <s>This is not the first sentence in this chapter.</s>

As these examples demonstrate, the tagging of a corpus text encoded in XML is itself a special and powerful form of metadata, instructing the user how to interpret and reliably use the data. As a further example, consider the following hypothetical case. In transcribing a spoken English text, a word that sounds like `skuzzy' is encountered by a transcriber who does not recognize this as one way of pronouncing the common abbreviation `SCSI' (small computer system interface). The transcriber T1 might simply encode his or her uncertainty by a tag such as <unclear extent="two syllables" resp="T1" desc="sounds like skuzzy"/> or even <gap extent="two syllables" cause="unrecognizable word">

Alternatively, the transcriber might wish to allow for the possibility of `skuzzy' as a lexical item while registering doubts as to its correctness, to propose a correct spelling for it, or simply to record that the spelling has been corrected from an unstated deviant form. This range of possibilities might be represented in a number of ways, some of which are shown here: <sic>skuzzy</sic> <corr>SCSCI</corr> <choice> <sic>skuzzy</sic> <corr>SCSI</corr> </choice> The first of these encodings enables the encoder to signal some doubt about the authenticity of the word. The second enables the encoder to signal that the word has been corrected, without bothering to record its original form. The third provides both the dubiously authentic form and its correction, indicating that a choice must be made between them.

This same method might be applied to the treatment of apparent typographic error in printed originals, or (with slightly different tagging since normalization is not generally regarded as the same kind of thing as correction) to the handling of regional or other variant forms. For example, in modern British English, contracted forms such as `isn't' exhibit considerable regional variation, with forms such as `isnae', `int' or `ain't' being quite orthographically acceptable in certain contexts. An encoder might thus choose any of the following to represent the Scots form `isnae': <reg>isn't</reg> <orig>isnae</orig> <choice> <reg>isn't</reg> <orig>isnae</orig> </choice>

Which choice amongst these variant encodings will be appropriate is a function of the intentions and policies of the encoder: these, and other aspects of the encoding policy, should be stated explicitly in the corpus documentation, or the appropriate section of the encoding description section of a TEI Header.

Analytic metadata

A corpus may consist of nothing but sequences of orthographic words and punctuation, sometime known as plain text. But, as we have seen, even deciding on which words make up a text is not entirely unproblematic. Texts have many other features worthy of attention and analysis. Some of these are structural features such as text, text subdivision, paragraph or utterance divisions, which it is the function of a markup system to make explicit, and concerning which there is generally little controversy. Other features are however (in principle at least) recognizable only by human intelligence, since they result from an understanding of the text.

Corpus-builders do not in general have the leisure to read and manually tag the majority of their materials; detailed distinctions must therefore be made either automatically or not at all (and the markup should make explicit which was the case!). In the simplest case, a corpus builder may be able reliably to encode only the visually salient features of a written text such as its use of italic font or emphasis, or by applying probabilistic rules derived from other surface features such as capitalization or white space usage.

At a later stage, or following the development of suitably intelligent tools, it may be possible to review the elements which have been marked as visually highlighted, and assign a more specific interpretive textual function to them. Examples of the range of textual functions of this kind include quotation, foreign words, linguistic emphasis, mention rather than use, titles, technical terms, glosses, etc.

The performance of such tools as morpho-syntactic taggers may occasionally be improved by pre-identification of these, and of other kinds of textual features which are not normally visually salient, such as names, addresses, dates, measures, etc. It remains debatable whether effort is better spent on improving the ability of such tools to handle arbitrary text, or on improving the performance of pre-tagging tools. Such tagging has other uses however: for example, once names have been recognized, it becomes possible to attach normalized values for their referents to them, thus facilitating development of systems which can link all references to the same individual by different names. This kind of named entity recognition is of particular interest in the development of message understanding and other NLP systems.

The process of encoding or tagging a corpus is best regarded as the process of making explicit a set of more or less interpretive judgments about the material of which it is composed. Where the corpus is made up of reasonably well understood material (such as contemporary linguistic usage), it is reasonably easy to distinguish such interpretive judgments from apparently objective assertions about its structural properties, and hence convenient to represent them in a formally distinct way. Where corpora are made up of less well understood materials (for example, in ancient scripts or languages), the distinction between structural and analytic properties becomes less easy to maintain. Just as, in some models of cognition at least, a text triggers meaning but does not embody it, so a text triggers multiple encodings, each of equal formal validity, if not utility.

Linguistic annotation of almost any kind may be attached to components at any level from the whole text to individual words or morphemes. At its simplest, such annotation allows the analyst to distinguish between orthographically similar sequences (for example, whether the word `Pat' at the beginning of a sentence is a proper name, a verb, or an adjective), and to group orthographically dissimilar ones (such as the negatives `not' and `-n't'). In the same way, it may be convenient to specify the base or lemmatized version of a word as an alternative for its inflected forms explicitly, (for example to show that `is', v`was' `being' etc. are all forms of the same verb), or to regularize variant orthographic forms, (for example, to indicate in a historical text that `morrow', `morwe' and `morrowe' are all forms of the same token). More complex annotation will use similar methods to capture one or more syntactic or morphological analyses, or to represent such matters as the thematic or discourse structure of a text.

Corpus work in general requires a modular approach in which basic text structures are overlaid with a variety of such annotations. These may be conceptualized as operating as a series of layers or levels, or as a complex network of descriptive pointers, and a variety of encoding techniques may be used to express them (for example, XML or RDF schemas, annotation graphs, standoff markup...).

Descriptive metadata

The social context within which each of the language samples making up a corpus was produced, or received, is arguably at least as significant as any of its intrinsic linguistic properties, if indeed the two can be entirely distinguished. In large mixed corpora such as the BNC, it is of considerably more importance to be able to identify with confidence such information as the mode of production or publication or reception, the type or genre of writing or speech, the socio-economic factors or qualities pertaining to its producers or recipients, and so on. Even in smaller or more narrowly focussed corpora, such variables and a clear identification of the domain which they are intended to typify are of major importance for comparative work.

At the very least, a corpus text should indicate its provenance, (i.e. the original material from which it derives) with sufficient accuracy that the source can be located and checked against its corpus version. Existing bibliographic descriptions are easily found for conventionally published materials such as books or articles and the same or similar conventions should be applied to other materials. In either case, the goal is simple: to provide enough information for someone to be able to locate an independent copy of the source from which the corpus text derives. Because such works have an existence independent of their inclusion in the corpus, it is possible not only to verify but also to extend their descriptive metadata.

For fugitive or spoken material, where the source may not be so easily identified and is less likely to be preserved independently of the corpus, this is less feasible. It is correspondingly important that the metadata recorded for such materials should be as all inclusive as feasible. When transcribing spoken material, for example, such features as the place and time of recording, the demographic characteristics of speakers and hearers, the social context and setting etc. are of immense value to the analyst, and cannot easily be gathered retrospectively.

Where interpretative categories or descriptive taxonomies have been applied, for example in the definition of text types or genres, these must also be documented and defined if the user is to make full use of the material.

To record the classification of a particular text, one or more of the following methods may be used: a list of descriptive keywords, either arbitrary or derived from some specific source, such as a standard bibliography a reference to one or more of internally-defined categories, declared in the same way as other analytic metadata, each defined as unstructured prose, or as a more structured set of situational parameters.

Despite its apparent complexity, a classificatory mechanism of this kind has several advantages over the kind of fixed classification schemes implied by simply assigning each text a fixed code, chiefly as regards flexibility and extensibility. As new ways of grouping texts are identified, new codes can be added. Cross classification is built into the system, rather than being an inconvenience. More accurate and better targetted enquiries can be posed, in terms of the markup. Above all, because the classification scheme is expressed in the same way as all the other encoding in the corpus, the same enquiry system can be used for both.

It will rarely be the case that a corpus uses more than one reference or segmentation scheme. However, it will often be the case that a corpus is constructed using more than one editorial policy or sampling procedure and it is almost invariably the case that each corpus text has a different source or particular combination of text-descriptive features or topics.

To cater for this variety, the TEI scheme allows for contextual information to be defined at a number of different levels. Information relating, either to all texts, or potentially to any number of texts within a corpus should be held in the overall corpus header. Information relating either to the whole of a single text, or to potentially any of its subdivisions, should be held in a single text header. Information is typically held in the form of elements whose names end with the letters decl (for declaration), and have a specific type. Examples include <editorialDecl> for editorial policies, <classDecl> for text classification schemes, and so on.

The following rules define how such declarations apply:a single declaration appearing only in the corpus header applies to all texts a single declaration appearing only in a text header applies to the whole of that text, and over-rides any declaration of the same type in a corpus header where multiple declarations of the same type are given in a corpus header, individual texts or text components may specify those relevant to them by means of a linking attribute

As a simple example, here is the outline of a corpus in which editorial policy E1 has been applied to texts T1 and T3, while policy E2 applies only to text T2: <teiCorpus> <teiHeader> <!-- ... --> <editorialDecl id="E1"> ... </editorialDecl> <editorialDecl id="E3"> ... </editorialDecl> <!-- ... --> </teiHeader> <tei.2 id="T1"> <teiHeader> <!-- no editorial declaration supplied --> </teiHeader> <text id="T1" decls="E1"> ... </text> </tei.2> <tei.2> <teiHeader> <editorialDecl id="E2"> ... </editorialDecl> </teiHeader> <text id="T2"> ... </text> </tei.2> <tei.2 id="T3"> <teiHeader> <!-- no editorial declaration supplied --> </teiHeader> <text id=T1 decls="E1"> ... </text> </tei.2> The same method may be applied at lower levels, with the decls attribute being specified on divn class elements, if all the possible declarations are specified within a single header.

A similar method may be used to associate text descriptive information with a given text, (though not with part of a text). Corpus texts are generally selected in order to represent particular classifications, or text types, but the taxonomies from which those classifications come are widely divergent across different corpora.

Finally, we discuss briefly the methods available for the classification of units of a text more finely grained than the complete text. These are of particular importance for transcriptions of spoken language, in which it is often of particular importance to distinguish, for example, speech of women and men, or speech produced by speakers of different socio-economic groups. Here the key concept is the provision of means by which information about individual speakers can be recorded once for all in the header of the texts they speak. For each speaker, a set of elements defining a range of such variables as age, social class, sex etc. can be defined in a <participant> element. The identifier of the participant is then used as the value for a who attribute supplied on each <u> element enclosing an utterance by the participant concerned. To select utterances by speakers according to specified participant criteria, the equivalent of a relational join between utterance and participant must be performed, using the value of this identifier.

The same method may be applied to select speech within given social contexts or settings, given the existence in the header of a <settingDesc> element defining the various contexts in which speech is recorded, which can be referenced by the decls attribute attached to an element enclosing all speech recorded in a particular setting.

Metadata categories for language corpora: a summary

As we have noted, the scope of metadata relevant to corpus work is extensive. In this final section, we present an overview of the kinds of data about data which are regarded as most generally useful.

Multiple levels of metadata may be associated with a corpus. For example, some information may relate to the corpus as a whole (for example, its title, the purpose for which it was created, its distributor, etc); other information may relate only to individual components of it (for example, the bibliographic description of an individual source text), or to groups of such components (for example, a taxonomic classification).

In the following lists, we have supplied the TEI/XCES element corresponding with the topic in question. This is not meant to imply that all corpora should conform to TEI/XCES standards, but rather to add precision to the topics addressed.

Conclusions

Metadata plays a key role in organizing the ways in which a language corpus can be meaningfully processed. It records the interpretive framework within which the components of a corpus were selected and are to be understood. Its scope extends from straightforward labelling and identification of individual items to the detailed representation of complex interpretive data associated with their linguistic components. As such, it is essential to proper use of a language corpus.

Bibliography Burnard, L.(1999) in . I.2 pp. 31-51. Cambridge, Mass: MIT Press. Dunlop, D. (1995) in Ide, Nancy and Jean Veronis (1995) Kluwer 19950-7923-3704-2 Sperberg-McQueen, C.M. and Burnard, L. (1994) Chicago and Oxford: ACH-ALLC-ACL Text Encoding Initiative. van den Heuvel, Henk, Louis Boves and Eric Sanders (2000). Available from Ide, Nancy (coordinator) (1998) Available from Gibaldi, Joseph (1998) (2nd ed). Lamport, L. (1986) . Addison-Wesley.