the true

UCMP Teacher Website on Evolution: Summary of Evaluation Methods and Findings

Rockman et al (REA), an independent research and evaluation group with expertise in technology interventions in education, conducted a thorough, multi-component evaluation of WWW.evolution. Over the course of the website’s development, the evaluation explored multiple aspects of the website and its constituent elements, ranging from its appeal and utility, to its ability to help users understand new concepts and acquire strategies for and confidence in teaching evolution. The evaluation included both formative elements intended to help the design team create an appealing, user-friendly, and useful website, and summative elements designed to explore the impact of the site on its target populations.

Evaluation Methodology

A national database of prospective evaluation participants was generated by aggregating contact lists from UCMP and Rockman et al (REA). The database contained entries for 246 individuals at the time of the first evaluation activity (Fall 2001), and grew to contain 402 entries by the time of the final evaluation activity (Spring 2005). This pool was comprised of formal and informal educators who had previously attended one or more UCMP-sponsored workshops or short courses on evolution at a regional or national conference and expressed interest in learning more about evolution, as well as educators who were referred by state science teacher associations, science and science education organizations, their school districts, or teacher colleagues that agreed to advertise the trial program and the opportunity to participate in its evaluation. The evaluation activities were thus promoted both among teachers who were highly motivated to teach evolution, as well as teachers who may have been less likely to teach evolution. The recruitment pool also included educators who, rather than being referred by an organization or colleague, volunteered independently to participate in evaluation activities after browsing the UCMP website.

REA staff emailed potential participants an invitation to complete a brief online survey, hosted on REA’s server, to gather demographic information about their educational background, amount of teaching experience, level of expertise in teaching evolution, and grade levels and subjects taught. The educational levels taught by the volunteer pool included elementary, middle and high school, and college.

With the exception of the web site log file analysis, recruitment for each evaluation activity was carried out by email invitations from REA staff. For two of the evaluation activities, namely the Fall 2001 online survey and Fall 2003 section review/impact surveys, REA staff initially targeted all participants in the database, in an effort to maximize the sample size.

In general, selection of participants was guided by sample specifications that were defined by UCMP staff and approved by REA researchers. UCMP staff were interested in the selection of samples that would represent, to the greatest extent possible:

• A range of geographic areas in the U.S. (e.g., West, Midwest, South, Northeast); • A range of grade level(s) taught—elementary (K-5), middle school (6-8), high school (9-12), and college, with an emphasis on middle and high school; • Varying amounts of expertise in evolutionary concepts, based on the subject of the participants’ degree or certification, as follows: • Novice - any non-science subject, such as English, Education, Special Education, etc. • Intermediate - Science Education, Composite Science, Secondary Science, etc. • Expert - Biology, Zoology, Genetics, Plant Breeding, etc.

Recruitment for some activities purposefully included or excluded individuals with prior exposure to the website. For example, for the pre/post assessments of teacher knowledge, REA staff initially sent invitations to a selection of 200 educators, drawn