Collections Building: Project, Resource and Collections Evaluation Tools and Methods

S. Buhr, B. DeFelice, & K. Kastens


A high quality, relevant, useful collection of earth system science educational materials has been a highly desired characteristic of DLESE from its inception. For a collection which is community driven like DLESE, it is critical to have a variety of methods of evaluating and assessing the collection as a whole and the resources within it, to ensure the desired high quality.
This session will introduce participants to three different approaches to the evaluation and assessment of DLESE content:

1. The DLESE Evaluation Toolkit is concerned with providing a broad array of tools to evaluate earth system education learning resources and projects.
2. The DLESE Community Review project provides a mechanism for formal review of individual resources in the DLESE collections.
3. The DLESE Collections Assessment project considers the DLESE collections as a whole, and through comparing the collection content to collection use, provides ideas about areas of the collection that need to be developed.

This session will be useful for people concerned about developing high quality learning resources and collections.

The DLESE Collections Assessment Project:
"Collections Assessment" provides a comparison of the scope and balance of an existing library collection with the scope and balance of the collection desired by the users and builders of the collection. It can help show where there are gaps between the content of the actual collection and what people expect and seek in the collection. In DLESE's case, the definition of the desired collection began with the crafting of the Scope Statement (http://www.dlese.org/documents/policy/collection_policy.html). The Collections Assessment team is now developing techniques for capturing a more nuanced view of the "desired" collection by looking at user requests for resources via the search and browse capabilities of the DLESE Discovery System. Recommendations for metrics for describing the scope and balance of the actual collection are also being developed, based on the parameters included in the DLESE required and recommended metadata. The audience for the DLESE Collections Assessment are collection builders deciding what to gather, resource developers deciding what to develop, and funders deciding what development efforts to fund. DLESE's Collections Assessement effort is lead by Barbara DeFelice (Barbara.DeFelice@dartmouth.edu) of Dartmouth College.
The participants will have an opportunity to share ideas about the kinds of materials they need in the DLESE collection, to learn more about the DLESE collection content, and to learn about the kinds of use and collection data that are most helpful in collection project evaluation.
At the workshop, DeFelice will first lead a brief review of the collection scope statement, and will show data concerning the actual DLESE Collection and its growth over the last several months. The participants will brainstorm about the most important learning resource types, learning contexts and topics to them. How well does the DLESE actual collection depth map to their top concerns? Then, participants will disperse to individual computers, and try searches that are important to them. How well did the collections support these requests? Were you able to discover a substantial gap or thin spot in the Collection, a spot at which the actual DLESE collection does not match your personal desired collection? Then the group will reassemble, and look at data about the DLESE community's desired collection, based on several months of requests in the search and browse functions of the Discovery System. How does this data compare to what participants just did in the preceding exercises? The participants will help develop a list of the most important use and content data elements for DLESE collections as a whole or for their collections projects.

The DLESE Community Review System:
The Community Review System is aimed at selecting the "best" resources from the DLESE Broad Collection for inclusion in the DLESE Reviewed Collection. The criteria for admission to the Reviewed Collection are scientific accuracy, pedagogical effectiveness, ease of use for teacher and learner, quality of documentation, importance or significance of content, ability to motivate or inspire learners, and robustness as a digital resource. The Community Review System combines two types of reviews: reviews delivered via a Web-based recommendation engine from educators in the DLESE community who have used the resource in their classroom, and specialist reviews mediated by an Editorial Review Board. Development of the Community Review System is lead by Kim Kastens (kastens@ldeo.columbia.edu) of Lamont-Doherty Earth Observatory, Columbia University. The testbed for the Community Review System is at: http://dlese.ldeo.columbia.edu
At the workshop, Kastens will present an overview of how the Community Review System works, the premises on which it was designed, and how it interfaces with the DLESE Discovery System. Participants will then disperse to individual computers to try reviewing a resource through the Community Review System, and to view examples of the kinds of reports that the CRS provides to the resource creator, to the Reviewed Collection editors, and to the public. The group will then reconvene to critique what they have seen. The developers are actively seeking feedback on the useability of the web-based recommendation engine, the kinds of information provided to the resource creator and the public, and the ways in which the DLESE Discovery System interacts with the Community Review System.

The DLESE Evaluation Toolkit:
The DLESE Evaluation Toolkit project seeks to develop an Earth systems education evaluation resource for geoscience educators. The resource will be grounded in the needs of Earth system educators, and will provide a virtual home for a geoscience education evaluation community. The goals for the project are to: Establish a forum and community for evaluation dialogue within DLESE, provide a robust collection of evaluation resources useful for Earth systems educators, and to disseminate the resources through the DLESE infrastructure and through professional society workshops and proceedings. The DLESE Evaluation Toolkit project is lead by Susan Buhr (sbuhr@cires.colorado.edu) , Cooperative Institute for Research in Environmental Sciences (CIRES), University of Colorado, Boulder. The beta test site for the Evaluation Toolkit is at: http://beta1.dpc.ucar.edu/evaltoolkit/overview.html
Participants are invited to come to the workshop with a project idea (either planned or existing) that they would like to evaluate. Buhr will give an overview of the purposes and process of project evaluation. Then participants will disperse to individual computers and use the Evaluation Toolkit to begin designing an evaluation plan and to find the resources they would need to actually implement an evaluation. The group will reconvene to discuss their challenges in project evaluation, and provide feedback about additional features or resources they would like to see included in the Evaluation Toolbook.