What is metadata and why do you need it?

Metadata is usually defined as data about data. The word appears only six times in the 100 million word British National Corpus (BNC), in each case as a technical term from the domain of information processing. However, all of the material making up the British National Corpus predates the whole-hearted adoption of this word by the library and information science communities. Since the BNC was first published in 1994, metadata has come to be used most frequently for one very specific kind of data about data: the kind of data that is needed to describe a digital resource in sufficient detail and with sufficient accuracy for some agent to determine whether or not that digital resource is of relevance to a particular enquiry. This so-called discovery metadata has become a major area of concern with the expansion of the World Wide Web and other distributed digital resources, and there have been a number of attempts to define standard sets of metadata for specific subject domains, for specific kinds of activity (for example, digital preservation) and more generally for resource discovery. The most influential of the generic metadata schemes has been the Dublin Core Metadata Initiative (DCMI), which (in 1995, the year after the BNC was first published), proposed 15 metadata categories which it was felt would suffice to describe any digital resource well enough for resource discovery purposes. For the linguistics community, more specific and structured proposals include those of the Text Encoding Initiative (TEI), the Open Language Archive Community (OLAC), and the ISLE Metadata Initiative (IMDI).

These and other initiatives have as a common goal the definition of agreed sets of metadata categories which can be applied across many different resources, so that potential users can assess the usefulness of those resources for their own purposes. The theory is that in much the same way that domestic consumers expect to find standardized labelling on their grocery items (net weight in standard units, calorific value per 100 grams, indication of country of origin, etc.), so the user of digital resources will expect to find a standard set of descriptors on their data items. While there can be no doubt that some information, however limited, about a resource is more useful than none, and that some metadata categories are of more general interest than others, it is far less clear on what basis or authority the definition of a standard set of metadata descriptors should proceed. Digital resources, particularly linguistic corpora, are designed to serve many different applications, and their usefulness must thus be evaluated against many different criteria. A corpus designed for use in one context may not be suited to another, even though its description suggests that it will be.

Nevertheless, it is no exaggeration to say that without metadata, corpus linguistics would be virtually impossible. Why? Because corpus linguistics is an empirical science, in which the investigator seeks to identify patterns of linguistic behaviour by inspection and analysis of naturally occurring samples of language. A typical corpus analysis will therefore gather together many examples of linguistic usage, each taken out of the context in which it originally occurred, like a laboratory specimen. Metadata can restore that context by supplying information about it, thus enabling us to relate the specimen to its original habitat. Furthermore, since language corpora are constructed from pre-existing pieces of language, questions of accuracy and authenticity are all but inevitable when using them: without metadata, the investigator has no way of answering such questions. Without metadata, the investigator has nothing but disconnected words of unknowable provenance or authenticity.

In many kinds of corpus analysis, the objective is to detect patterns of linguistic behaviour which are common to particular groups of texts. Sometimes, the analyst examines occurrences of particular linguistic phenomena across a broad range of language samples, to see whether certain phenomena are more characteristic of some categories of text than others. Alternatively, the analyst may attempt to characterize the linguistic properties or regularities of a particular pre-defined category of texts. In either case, it is the metadata which defines the category of text; without it, we have no way of distinguishing or grouping the component texts which make up a large heterogenous corpus, nor even of talking about the properties of a homogenous one.

Scope and representation of metadata

Many different kinds of metadata are of use when working with language corpora. In addition to the simplest descriptive metadata already mentioned, which serves to identify and characterize a corpus regarded as a digital resource, we discuss below the following categories of metadata, which are of particular significance or use in language work: editorial metadata, providing information about the relationship between corpus components and their original source () analytic metadata, providing information about the way in which corpus components have been interpreted and analysed () descriptive metadata, providing classificatory information derived from internal or external properties of the corpus components () administrative metadata, providing documentary information about the corpus itself, such as its title, its availability, its revision status, etc. (this section)

In earlier times, it was customary to provide corpus metadata in a free-standing reference manual if at all. Early corpora such as the Brown or LOB were always accompanied by a large A4 volume of typescript. It is now more usual to present all metadata in an integrated form, together with the corpus itself, often using the same encoding principles or markup language. This facilitates automatic validation of the accuracy and consistency of the documentation, simplifies the development of user-friendly access software the data, and helps ensure that corpus and metadata are kept together, and can be distributed as a single unit.

A major influence in this respect has been the Text Encoding Inititiative (TEI), which in 1994 first published an extensive set of Guidelines for the Encoding of Machine Readable Data. (TEI P1). These recommendations have been widely adopted, and form the basis of most current language resource standardization efforts. A key feature of the TEI recommendations was the definition of a specific metadata component known as the TEI Header.

The TEI Header was first thought of as a kind of electronic title page, which could be prefixed to a computer file (or a collection of such files) to supply the same kind of information as is provided by the title page and other front matter of a conventional book. Thus, it has four major parts, derived originally from the International Standard Bibliographic Description (ISBD): a file description, identifying the computer file itself and those responsible for its authorship, dissemination or publication etc., together with (in the case of a derived text such as a corpus) similar bibliographic identification for its source; an encoding description, specifying the kinds of encoding used within the file, for example, what tags have been used, what editorial procedures applied, how the original material was sampled, and so forth; a profile description, supplying additional descriptive material about the file not covered elsewhere, such as its situational parameters, topic keywords, descriptions of participants in a spoken text etc. a revision description, listing all modifications made to the file during the course of its development as a distinct object.

In this way, the TEI sought to extend the well-understood principles of print bibliography to the (then!) new world of digital resources. The TEI recommendations, initially expressed as an application of the Standard Generalized Markup Language (SGML: ISO 8879), proved very influential, and have since been re-expressed as an application of the current de facto standard language of the internet: the W3C's extensible markup language (XML), information on which is readily available elsewhere.

The scope of this article does not permit exhaustive discussion of all features of the TEI Header likely to be of relevance to corpus builders or users, but some indication of the range of metadata it supports is provided by the summary below (). For full information, consult the online version of the TEI Guidelines (), or the Corpus Encoding Standard (), which is a specialization of them for corpus work. Dunlop 1995 and Burnard 1999 describe the use of the TEI Header in the construction of the BNC.

Editorial metadata

Because electronic versions of a non-electronic original are inevitably subject to some form of distortion or translation, it is important to document clearly the editorial procedures and conventions adopted. In creating and tagging corpora, particularly large ones assembled from many sources, many editorial and encoding compromises are necessary. The kind of detailed text-critical attention possible for a smaller literary text may be inappropriate, whether for methodological or financial reasons. Nevertheless, users of a tagged corpus will not thank the encoder if arbitrary editorial changes have been silently introduced, with no indication of where, or with what regularity. Corpora encoded in such a way can mislead the unwary or partially informed user.

A conscientious corpus builder should therefore take care to consider making explicit in the corpus markup at least the following kinds of intervention: addition or omission where the encoder has supplied material not present in the source, or (more frequently in corpus work) where material has been omitted from a transcription or encoding. correctionwhere the encoder has corrected material in the source which is judged erroneous (for example, misprints); normalizationwhere, although not considered erroneous, the source material exhibits a variant form which the encoder has replaced by a standardized form.

The encoder may simply record the fact that such interventions have taken place by making a note of this in the corpus header, possibly describing their scope and nature. Alternatively, assuming that the corpus uses a sufficiently powerful markup language, each such intervention may be explicitly signalled within the encoded text. In the latter case, it may be possible to retain both original and corrected (or normalized) form, so that the corpus user can decide for themselves on whether or not to accept the intervention. We give some simple examples below.

The explicit marking of material missing from an encoded text may be of considerable importance as a means of indicating where non-linguistic (or linguistically intractable) items such as symbols or diagrams or tables have been omitted: <gap desc="diagram"/> Such markup is useful where the effort involved in a more detailed transcription (using more specific elements such as figure or table, or even detailed markup such as SVG or mathml) is not considered worthwhile. It is also useful where material has been omitted for sampling reasons, so as to alert the user to the dangers of using such partial transcriptions for analysis of text-grammar features: <div type="chapter"> <gap extent="100 sentences" cause="sampling strategy"/> <s>This is not the first sentence in this chapter.</s>

As these examples demonstrate, the tagging of a corpus text encoded in XML is itself a special and powerful form of metadata, instructing the user how to interpret and reliably use the data. For example, in transcribing a spoken English text, a word that sounds like `skuzzy' is encountered by a transcriber who does not recognize this as one way of pronouncing the common abbreviation `SCSI' (small computer system interface). The transcriber might simply encode his or her uncertainty by marking an omission in the following way: <gap extent="two syllables" cause="unrecognizable word">

Alternatively, the transcriber might wish to allow for the possibility of `skuzzy' as a lexical item while registering doubts as to its correctness: <sic>skuzzy</sic>

Now consider the case where the transcriber finds in the source something that clearly reads wierd stuff. Again, the transcriber can simply flag that this is probably an error: <sic>wierd</sic> stuff Or they might decide both to correct the error and also to record that they have done so: <corr>weird</corr> stuff.

Corrections of orthographic error like this help the corpus user find word forms even when they happen to have been mis-spelled. On the other hand, such corrections are a little annoying for the corpus user who is interested in the study of orthographic error itself. For such users, an ideal encoding would preserve both the error and its correction, perhaps like this: <choice> <sic>wierd</sic> <corr>weird</corr> </choice> stuff

The same range of possibilities might be needed in the treatment of handling of historical, regional, or other kinds of variant forms. For example, in modern British English, contracted forms such as `isn't' exhibit considerable regional variation, with forms such as `isnae', `int' or `ain't' being quite orthographically acceptable in certain contexts. An encoder might thus choose any of the following to represent the Scots form `isnae': <reg>isn't</reg> <orig>isnae</orig> <choice> <reg>isn't</reg> <orig>isnae</orig> </choice>

Which of these different encoding styles will be appropriate is a function of the intentions and policies of the encoder: these, and other aspects of the encoding policy, should be stated explicitly in the corpus documentation, or the appropriate section of the encoding description section of a TEI Header.

Analytic metadata

A corpus may consist of nothing but sequences of orthographic words and punctuation, sometime known as plain text. But, as we have seen, even deciding on which words make up a text is not entirely unproblematic. Texts have many other features worthy of attention and analysis. Some of these are structural features such as text, text subdivision, paragraph or utterance divisions, which it is the function of a markup system to make explicit, and concerning which there is generally little controversy. Other features are however (in principle at least) recognizable only by human intelligence, since they result from an understanding of the text.

Corpus-builders do not in general have the leisure to read and manually tag the majority of their materials; detailed distinctions must therefore be made either automatically or not at all (and the markup should make explicit which was the case!). In the simplest case, a corpus builder may be able reliably to encode only the visually salient features of a written text such as its use of italic font or emphasis. In documents produced by modern word processors particular combinations of such features may be encoded in the document as style markers, which can easily be automatically converted to a more semantically useful markup. Similarly, a more explicit markup (for example, of sentences) might be derived by the application of probabilistic rules derived from surface features such as punctuation, capitalization, and white space usage.

At a later stage, or following the development of suitably intelligent tools, it may be possible to review the elements which have been marked as visually highlighted, and assign a more specific interpretive textual function to them. Examples of the range of textual functions of this kind include quotation, foreign words, linguistic emphasis, mention rather than use, titles, technical terms, glosses, etc.

The performance of such tools as morpho-syntactic taggers may occasionally be improved by pre-identification of these, and of other kinds of textual features which are not normally visually salient, such as names, addresses, dates, measures, etc. It remains debatable whether effort is better spent on improving the ability of such tools to handle any text, or on improving the performance of pre-tagging tools. Such tagging has other uses however: for example, once names have been recognized, it becomes possible to attach normalized values for their referents to them, thus facilitating development of systems which can link all references to the same individual by different names. This kind of named entity recognition is of particular interest in the development of message understanding and other Natural Language Processing (NLP) systems.

The process of encoding or tagging a corpus is best regarded as the process of making explicit a set of more or less interpretive judgments about the material of which it is composed. Where the corpus is made up of reasonably well understood material (e.g. contemporary newspaper texts), it is reasonably easy to distinguish such interpretive judgments from apparently objective assertions about its structural properties, and hence convenient to represent them in a formally distinct way. Where corpora are made up of less well understood materials (for example, in ancient scripts or languages), the distinction between structural and analytic properties becomes less easy to maintain. Just as, according to some theories, a text triggers meaning but does not embody it, so a text triggers multiple encodings, each of equal formal validity, if not utility.

Linguistic annotation of almost any kind may be attached to components at any level from the whole text to individual words or morphemes. At its simplest, such annotation allows the analyst to distinguish between orthographically similar sequences (for example, whether the word `Pat' at the beginning of a sentence is a proper name, a verb, or an adjective), and to group orthographically dissimilar ones (such as the negatives `not' and `-n't'). In the same way, it may be convenient to specify the base or lemmatized version of a word as an alternative for its inflected forms explicitly, (for example to show that `is', `was' `being' etc. are all forms of the same verb), or to regularize variant orthographic forms, (for example, to indicate in a historical text that `morrow', `morwe' and `morrowe' are all forms of the same word). More complex annotation will use similar methods to capture one or more syntactic or morphological analyses, or to represent such matters as the thematic or discourse structure of a text.

Corpus work requires a modular approach in which basic text structures are overlaid with a variety of such annotations. These may be thought of as a distinct layers or levels, or as a complex network of descriptive pointers, and a variety of encoding techniques may be used to express them. Ideas from mathematics, formal language theory, and computer science have been particularly influential in the development of techniques for this purpose, for example in RDF or annotation graphs; most such techniques rely on the use of XML as their basic means of expression however. We discuss some of the implications of this in the next section.

Descriptive metadata

The social context (that is, the place, time, and participants) within which each of the language samples making up a corpus was produced or received is arguably at least as significant as any of its intrinsic linguistic properties — if indeed the two can be entirely distinguished. In large corpora which sample language characteristic of many different social contexts such as the BNC, it is of considerably more importance to be able to identify with confidence such information as the mode of production or publication or reception, the type or genre of writing or speech, the social class or occupation, gender, or age of the producers or recipients of the speech, and so on. Even in smaller or more narrowly focussed corpora, such variables and a clear identification of the domain which they are intended to typify are of major importance for comparative work.

At the very least, a corpus text should indicate its provenance, (i.e. the original material from which it derives) with sufficient accuracy that the source can be located and checked against its corpus version. Existing bibliographic descriptions are easily found for conventionally published materials such as books or articles and the same or similar conventions should be applied to other materials. In either case, the goal is simple: to provide enough information for someone to be able to locate an independent copy of the source from which the corpus text derives. Because such works have an existence independent of their inclusion in the corpus, it is possible not only to verify but also to extend their descriptive metadata.

For fugitive or spoken material, where the source may not be so easily identified and is less likely to be preserved independently of the corpus, this is less feasible. It is correspondingly important that the metadata recorded for such materials should be as extensive as feasible. When transcribing spoken material, for example, such features as the place and time of recording, the demographic characteristics of speakers and hearers, the social context and setting etc. are of immense value to the analyst, and cannot easily be gathered retrospectively.

The text-type or genre labels used in a given corpus may sometimes be drawn from an open ended set, but it is also convenient for them to be taken from a predefined set of values, or taxonomy. Sometimes both approaches may be taken: for example in the BNC, each text is associated with an open ended set of descriptive keywords relating to its subject matter and also with one of a set of pre-defined domain codes. Thus, text B1G in the BNC baby corpus, which is an extract from a textbook on Geographical Information Systems, contains (amongst other things) the following information in its header: <catRef target="alltim3 acad wriase0 wridom3 wrista2 "/> <classCode scheme="DLee">W ac soc science</classCode> <keywords scheme="COPAC"> <term>Geography - Methodology - Addresses, essays, lectures</term> <term> Geographical information systems.</term> <term> Geography - Computer programs</term> </keywords> The first line here indicates how the text is classified according to the classification defined for the whole corpus, and consists of a series of codes (alltim3, acad, etc.) each of which is further defined in the corpus header. The second line indicates how the text was classified in a scheme defined by David Lee for the BNC as a whole, again using predefined codes such as W for written, ac for academic prose etc. The remaining part of the example however shows how the source text is classified by the COPAC (a major UK online library catalogue), using a sequence of descriptive cataloguing terms.

When a corpus is constructed according to a pre-defined set of selection criteria, as was the BNC, it is essential to provide both definitions of the criteria concerned and an indication of which criteria apply to each text, but even where this is not the case, documentation and definition of any classification scheme used is essential if the user is to make full use of the material.

It will rarely be the case that a corpus uses more than one reference or segmentation scheme. However, it will often be the case that a corpus is constructed using more than one editorial policy or sampling procedure and it is almost invariably the case that each corpus text has a different source or particular combination of text-descriptive features or topics. To cater for this variety, the TEI scheme allows for contextual information to be defined at a number of different levels. Information relating, either to all texts, or potentially to any number of texts within a corpus is held in the overall corpus header, while information relating either to the whole of a single text, or to potentially any of its subdivisions, should be held in a single text header.

It is also often necessary to classify textual components smaller than the whole of a text. For example, in transcriptions of spoken language, it is often desirable to identify speech produced by particular individuals, for example to distinguish the speech of women and men, or of members of different socio-economic groups. Here the key concept is the provision of a means by which information about individual speakers can be recorded once for all in the header of the texts they speak. For each speaker, a set of elements defining a range of such variables as age, social class, sex etc. might be defined and grouped together within a <person> element, like the following: student male 15-20 instructor female 30-35 ]]> Within the body of the text, each utterance can then identify its speaker using the identifiying code given as the value of the id attribute above: Good morning class I didn't do it ]]> The who attribute supplied on each <u> element is sufficient to identify which speaker is concerned. To select utterances by speakers according to specified participant criteria (for example to find all male speech, or all speech by an instructor in a specific age group), the equivalent of a relational join between utterance and participant must be performed, using the value of this identifier. This method simplifies the encoding of the text, since there is no need to supply (say) age or sex information for each utterance, and also makes it extensible: if a new category of information becomes available about a given speaker, it need only be added to the speaker element for it to be usable in queries across the whole existing corpus.

The same method might be used to select speech within particular social contexts or settings, given the existence in the header of a <settingDesc> element defining the various contexts in which speech is recorded, which can be referenced by the decls attribute attached to an element enclosing all speech recorded in a particular setting. For example, a text or corpus header might contain entries like the following: Southside Cafe, South Quad Instructors Room, Regius Building ]]> while each conversation transcribed in the corpus might be marked as a distinct div element like this: <div where="SCA"> <u who="T1">Skinny cap no sugar please</u> <u who="XX">You got it</u> </div> As before, the identifier SCA can be used to associate the content of this div element with the metadata describing it in the setting element, so that an XML query engine can answer questions such as is the phrase skinny cap used in formal or informal sitations?

Metadata categories for language corpora: a summary

As we have noted, the scope of metadata relevant to corpus work is extensive. In this final section, we present an overview of the kinds of data about data which are regarded as most generally useful.

Multiple levels of metadata may be associated with a corpus. For example, some information may relate to the corpus as a whole (for example, its title, the purpose for which it was created, its distributor, etc); other information may relate only to individual components of it (for example, the bibliographic description of an individual source text), or to groups of such components (for example, a taxonomic classification).

In the following lists, we have supplied the TEI element corresponding with the topic in question. This is not meant to imply that all corpora should conform to the TEI Recommendations, but simply to give examples taken from a widely used implementation of the the topics addressed.

Conclusions

Metadata plays a key role in organizing the ways in which a language corpus can be meaningfully processed. It records the interpretive framework within which the components of a corpus were selected and are to be understood. Its scope extends from straightforward labelling and identification of individual items to the detailed representation of complex interpretive data associated with their linguistic components. As such, it is essential to proper use of a language corpus.

Bibliography Burnard, L.(1999) in . I.2 pp. 31-51. Cambridge, Mass: MIT Press. Dunlop, D. (1995) in Ide, Nancy and Jean Veronis (1995) Kluwer 19950-7923-3704-2 Sperberg-McQueen, C.M. and Burnard, L. (1994) Chicago and Oxford: ACH-ALLC-ACL Text Encoding Initiative. van den Heuvel, Henk, Louis Boves and Eric Sanders (2000). Available from Ide, Nancy (coordinator) (1998) Available from Gibaldi, Joseph (1998) (2nd ed). Lamport, L. (1986) . Addison-Wesley.