[Paper delivered at the ACH-ALLC International Humanities Computing Conference, University of Virginia, 9-13 June 1999]
Michael Fraser
Oxford University
The Higher Education Funding bodies in the UK recently called for bids to develop subject-based faculty 'hubs' or gateways which locate, catalogue, and give access to digital resources suitable for use in Higher Education teaching and research as part of the new Resource Discovery Network. The new faculty hubs will further develop the design and purpose of the existing centrally-funded gateways, amongst which ADAM (Art, Design and Media), IHR-INFO (History), and SOSIG (Social Sciences) currently have some remit for humanities disciplines.
The Humanities Computing Unit has been invited to submit a bid to develop the proposed Humanities Hub of the new Resource Discovery Network. The proposal draws upon existing work relating to subject gateways within Oxford, in particular the HumBul Gateway for the Humanities and, on a smaller scale, the Computer-Assisted Theology gateway, as well as other gateways within the UK and beyond.
This paper will focus on a particular issue which lies at the core of any subject-based gateway, the criteria by which resources are selected for inclusion within the gateway. Subject gateways explicitly state or at the very least imply a concern that the resources catalogued are quality-assured, an assurance based on human intervention. But what does quality mean in this context? Against what criteria and with what authority can an individual resource be deemed fit for inclusion and therefore deemed fit for purpose?
Gateways tend to fall into two basic types. To a large extent the HumBul Gateway and the Computer-Assisted Theology gateway demonstrate both types. The Theology gateway was developed by an individual enthusiast with a keen interest in the possibilities offered by the Internet for teaching and research and with specific subject expertise. Gateways of this type are numerous on the Internet and indeed many of the existing gateways to humanities subjects fall into this category. For the purposes of this discussion such gateways may be termed amateur gateways since their development is often dependant on one or two individuals, often without formal institutional support and frequently presented with little information about selection criteria, intended audience, available metadata, consistent classification, advanced searching and so on. What these gateways can offer however, is a subject practitioner's view of the Internet with evaluative as well as descriptive annotation for each linked resource; they derive their authority from the recognised expertise of the subject specialist. The second type of gateway, one which to a large extent the new HumBul strives to be, and which may be termed the professional gateway, are fewer in number (certainly for the humanities). The professional gateway is identified by institutional and sometimes national support, developed by a specific project team, and constructed along the lines of an advanced library catalogue (often drawing its cataloguers from amongst subject librarians). The professional gateway offers durability, structured and easily retrievable data. On the other hand there is a tendency to hide from the end-user the evaluative judgments made about individual resources held within the virtual collection despite publication of the criteria by which resources are selected for inclusion. Both types of gateway contend with the inherent tension between trying to be a digital library (cataloguing and dissemination) and something like an academic reviews journal (discovery and evaluation).
The EU-funded DESIRE Project, "Selection Criteria for Quality Controlled Information Gateways", developed and published a list of quality selection criteria designed as a reference point for subject-gateways (see <http://www.ukoln.ac.uk/metadata/desire/quality/title-page.html>). The criteria presented were arranged under five headings which may be summarized as relating to: audience, content, design, maintenance or durability, and comparison with related resources. Under each heading a number of sub-categories contain a series of questions to be considered by resource contributors to subject gateways. The categories are comprehensive and the questions detailed. The application of this criteria is intended to highlight quality and limit quantity. The ADAM and SOSIG Gateways, for example, either explicitly draw attention to this particular set of criteria or have developed a similar set for their own resource contributors.
Leaving aside the issue of whether such a comprehensive approach to selection criteria actually serves the purpose for which it was designed, it is significant to note that the wide range of questions which are required to be answered satisfactorily before a resource can be admitted into the catalogue bear little or no relation to the metadata available to the end-user. For the most part the user of gateways such as the two mentioned above are presented with fairly sparse metadata consisting of title, subject, description and so on. Whilst the cataloguer is forced to make evaluative judgements about resources, the user has little notion of what these judgements might have been. The mere fact that a resource has been included within the gateway is, it seems, assumed to be enough. Descriptions are short and objective, and rarely is an indication given even to the contributor's identity or authority for making such judgments.
A fundamental question which underlies this paper is whether there is a need for detailed quality assurance at all when the effort might be better expended on more comprehensive, factual, metadata to assist the searching and delivery of a gateway's holdings. The combination of professional cataloging and amateur evaluation appears to be successfully provided by services like the Internet Movie Database and to some extent by commercial ventures like Amazon.Com. Both these databases, however, concern themselves with offline media available to their users only with some additional effort. The Internet subject gateway, of course, catalogues resources sharing the same digital medium as itself. It is intrinsic to an Internet gateway to not only point away from itself but to actually take the user to those resources using the same mode of delivery. One might argue that providing reviews of Internet resources is a superfluous activity given that the function of a gateway is to take the user to the objects which they might inspect for themselves, a task which neither the Internet Movie Database nor Amazon.Com can undertake.
On the other hand, as this paper will argue, given that academic subject gateways have an additional role of providing access to digital resources suitable for teaching and research the Internet offers something which the offline media cannot: a full integration of the resource catalogue, resource evaluation, and the resources themselves. It is only the combination of all four fundamental elements, discovery, evaluation, cataloguing and dissemination, which moves us towards a gateway which is subject-based, academic, and Internet integrated, a combination which lies at the core of the proposed Humanities Hub.
The short presentation intends to raise for discussion a number of issues related to the selection of networked resources for inclusion within a subject-based Internet Gateway. The issues and points I intend to raise can be treated as a supplement to the abstract on page x. The context out of which this discussion arises is the recent proposal by the UK higher education funding bodies (namely the JISC) to fund the development of a Resource Discovery Network comprising a number of faculty hubs and an over-arching network centre (or executive body). It may be of interest to some that the RDN centre is being run by a triumvirate, one of whose members is Dan Greenstein the director of that other subject-based network, the Arts & Humanities Data Service.
In sum, the Resource Discovery Network will develop a series of subject-based Internet gateways. Rather than being called gateways they will be hubs. I must confess that when I first heard this I considered it yet another example of the British driving on the left when everyone else was on the right. However, in a considered moment of spin-doctoring I now see that hubs have the potential to offer much more to the user than gateways which, by definition, are meant to facilitate speedy access through them rather than being a centre of activity in themselves.
One of the hubs has been designated for the humanities and I am pleased to informally announce that Oxford University's Humanities Computing Unit has been successful in its bid to host the humanities hub which will continue the HCU's history of providing access to humanities internet resources through the HumBul gateway.
Some might think that the UK is coming a little late to the creation of Internet gateways given that gateways and ever-more complex search engines abound. Think how fortunate it might have been if JISC had funded its development when Yahoo was but a simple directory. However, it is evident given the continuous discussion about locating and evaluating Internet resources that the existing gateways have not solved the problems and probably never will do. The question which underlies this presentation is what can a new subject gateway learn from the last five years and how might it be sufficiently different to find its own niche within the academic market at least?
In theory subject-based gateways should be uncomplicated. They are not hybrid in the sense of mixing media. They are delivered using the same medium as the resources which they catalogue. Whereas the user of the online library catalogue or the online film database, or the online bookstore has to make that extra effort to obtain the item catalogued within the database, for the user of the subject gateway the catalogued item is but a click away.
In reality, the humanities hub, like other subject gateways, has to face similar issues concerning collection policies as the library world. The Resource Discovery Network aims to provide access to high quality resources. What is 'high quality'? The answer, I suspect the same as that given in my colleagues' presentations, is of course, 'It depends'. Whatever the answer, the key, I believe, is making explicit the evaluation of any resource which makes it into the hub. There are two levels on which this may operate. The first is to make explicit the selection criteria used to evaluate resources, the second is to make explicit the judgements given on any given resource.
With regard to the publication of selection criteria a gateway might learn much from the library world where increasingly collection development policies together with selection criteria are made publicly available (particularly true for public libraries). The EU-funded DESIRE Project included amongst its remit the development of selection criteria for the inclusion of quality-assured resources. These criteria have been incorporated into the collection policies of the SOSIG and ADAM gateways. However, the evaluation criteria have more to do with the evaluation of the resource as a publication (ease of navigation, meta-information, etc.) rather than the evaluation of its intellectual content.
It is interesting to note that the selection criteria found within the library collection development policies often specify a very similar set of criteria. A core element, however, is the expectation that librarians will make use of subject expertise as part of the selection process. Subject expertise within the public library sector includes published reviews. Subject expertise within the academic library includes the recommendations and needs of teaching and research staff. In this respect Subject Gateways should be no different. However, whilst book reviews abound and books and other more traditional library resources have been fully integrated into teaching and research, Web resources have few reviews in either printed or online publications and even after ten years of trying (in the UK at least) digital resources still present obstacles to integration into teaching at least.
The online library catalogue and the library collection development policy offers a framework for the selection and description of resources. In any case many of the existing subject gateways have been created as within the library community as an extension of the traditional library service and many make some use of library classification systems, standards and protocols.
However, library catalogues also differ from the subject gateway in other respects. A library catalogue describes resources collected by the library. The collection of resources for the library conforms to its (usually) declared collection policy. For the most part the library catalogue describes resources already held by the library. This, however, is changing as increasingly libraries integrate the catalogue of their local holdings with catalogues of digital holdings served from elsewhere. The overall purpose of the library catalogue is to facilitate access to resources through factual metadata. Rarely, does the library catalogue contain evaluative information within its records. Any sense of quality assurance belongs within the collection policy, the means by which resources are obtained for inclusion in the library and subsequently catalogued.
In the abstract I have made a distinction between what I have termed amateur and professional gateways. One of the distinguishing marks is that whilst amateur gateways are often flat HTML files with little searchable (or much browsable) metadata, they often make explicit the evaluative judgements made about a resources through annotation and descriptive links. Professional gateways on the other hand increasingly make explicit the selection criteria within collection development policies but do not continue the process by making explicit the evaluative judgements made about resources within the metadata delivered to the end-user.
The Humanities Hub, whilst accepting that a collection development policy is essential, also believes that the judgements arrived at by the selectors should be made explicit within the metadata attached to each resource. The Hub should be akin to a reviews journal. A foundation stone of quality assurance in higher education is the peer-review system. The concept of peer-review underlies the academic publication process and indeed, in the present discussions about the future of academic publication it is peer review which remains at the forefront (it is the publisher's name which can take a backwards step).The fruits on one's career as an academic are placed under the scrutiny of one's academic colleagues not only within the process which leads to publication but also in the post-publication phase. Book reviews in particular (not so many post-publication journal article reviews) can determine the overall verdict.
One medium term aim of the Hub is to find some mechanisms by which the classification and description of resources (for which librarians tend to have the expertise) can be combined with the evaluation of resources for teaching and research purposes (for which the subject experts have the expertise). It is not enough to state that selectors will take heed of existing reviews because these barely exist for academic resources (scholarly email forums seem the closest we get). The closest parallel I can think of at this point is browsing in a second-hand bookshop and selecting a book from the shelf in which someone has pasted a review of the book. Why should anyone want to do this (and they certainly used to do it frequently if bookshops in Oxford are anything to go by - and no, it wasn't simply the reviewer who did it). What is attractive about having an evaluation of the very book in your hands which presumably was once in someone's bookcase? If it is attractive to keep book reviews with their subjects then so too with Web resources.
At this point I would like to flag the possibility of a publication devoted to reviewing Web resources for humanities higher education with all the crucial elements required of an academic journal - in particular, the emphasis on peer-review. In theory, reviews of academic Web sites should benefit the Hub with a richer set of evaluative metadata and benefit the authors of the Web site in the same manner in which book reviews are meant to be of benefit. In practical terms one might ask how reviewers might be rewarded since one can hardly offer them a copy of the Web resource for retention (unless a fee-based access resource in the first place). The Hub will consider, amongst other things, paying a fee to reviewers in lieu of receiving a tangible resource. Comments on this are welcome.
The Hub might have something further to learn from commercial sites like Amazon.com and the Internet Movie Database (to name but two). Both encourage the participation of users and most online shops make every attempt to win repeat visits through a combination of registration, customization, and user feedback. The latter is particularly developed in Amazon and the Movie Database which encourages users to supply reviews of books and films. The HumBul Gateway in its current form also endeavours to encourage comments from visitors on particular resources. However, such free expression is commentary and should not be confused with peer-review. In any case, there are few free comments in HumBul and few unsolicited reviews in Amazon.
The final decision about the quality of a resource and its fitness for purpose lies with the end-user, in particular the member of the community addressed by the subject gateway. This paper asks only that the judgement of the practitioner, in the case of the Humanities Hub the academic research and teacher, be included within the selection loop, preferably at the moment of inclusion but failing that, fully integrated within the available metadata as soon as possible thereafter.