necessary in core academic areas. One important area that has still gone relatively overlooked is the integration of vocational guidance in K-12 schools. According to Schwager (2001), employers are not satisfied with the level of education that students have upon graduation, and they feel that there is a gap between what people are learning in school and the tasks they are expected to perform at work. The same sentiment is also true from students’ point of view. A recent study found that high school students feel they are getting too little career guidance (Farris State University Career Insitutue for Education and Workforce Development, 2002). Of the students surveyed, 51% reported that no one at school had helped them at all with their career exploration. A majority of those responding (78%) said their parents were their top adult influence on career planning. However, 70% also reported that they had spent three hours or less over the past few months discussing career choices with their parents. In addition, parents have a fairly narrow frame of reference when it comes to jobs and job training. This report is of concern because it shows that students are not provided the necessary information to make decisions about what to do upon completing a high school education.

According to Maxwell and Rubin (2001) educators should go beyond being concerned about the lack of guidance at the high school level. Students should have an integrated curriculum where they get introduced to different vocational options early in the educational process. In a response to this concern it is proposed that the Career Education Responsive to Every Student (CERES) curriculum be used and evaluated in K-12 programs (Project CERES, 2002). CERES is a comprehensive guidance curriculum for grades K through 12. It is designed to work with teachers’ existing curriculum providing a means of infusing career education into basic skills instruction.

The CERES program was developed in California public schools in response to community and employer concerns that too many students were completing their education without a positive orientation to the world-of-work. Many could not demonstrate minimum competency in basic reading, writing, math, listening, and speaking, as well as academic thinking, problem solving, and comprehension, and did not demonstrate skills related to working on teams or serving customers. CERES was developed to provide students with opportunities to acquire workplace skills. The idea behind this project was that students would meet four main objectives such as being able to identify and acquire positive work habits; acquire knowledge of economic/free enterprise principles; acquire knowledge of diverse occupations including job training preparation and job duties; and apply basic skills to career decisions and actions, including job-seeking and job-retention.

The CERES program is widely used throughout the U.S. and Puerto Rico. A review over a five-year period (1993-97) showed that the program has been adopted by approximately 800 public, and 35 non-public schools. In an attempt to evaluate the results of the CERES program seven program schools and four control schools were studied over a two-year period. The results indicated that students, at all grade levels, performed significantly better than students in comparable test sites lacking the program. Although the evaluation did provide support for the use of the CERES program there are still many questions that could be asked in evaluating a career curriculum. This article will provide a comprehensive model for evaluating the CERES program.

As mentioned above, several questions about the use of the CERES program remain unanswered. The first of these is the degree to which teachers are integrating CERES materials into the classrooms. If the program has been properly implemented it is desirable to see if it effectively attains its objectives. If the objectives are being met the next question is whether those objectives are attaining the overall goal of having high school graduates that are more familiar with career choices and who are better prepared for the next step in their career, whether that is a job or further education. If the program is successful in attaining its overall desired goal, a final question is whether there are any side effects. Side effects could be higher levels of self-efficacy, or an over burdened curriculum that takes away from other required lessons. Answering these questions will provide guidance to improving the CERES program, as well as overall information about the outcomes of the program on post high school success.

In evaluating the CERES program schools will be used as units of analysis. Overall it is proposed here for discussion purposes that 15 schools that have adopted the CERES program and 100 schools that have not adopted the program throughout California will be used as the sample.

The first portion of the evaluation involves the extent to which the CERES program is being implemented and used in school curriculum. A qualitative evaluation using an interview technique where teachers are directly asked specific questions about how they use the program in their classroom will be conducted. The drawback to a qualitative evaluation is that it takes up time and resources. To solve this problem, only 5 of the 15 schools will be randomly selected for this part of the evaluation. A comparison of the two groups will be made to see if there are major differences. Within each of the five schools two teachers will be selected randomly from each grade, and will each participate in 30-minute individual interviews with the evaluator. Specifically, the schools will be compared on the outcome variables (discussed later in this article) to see if there are significant differences.

The second portion of the evaluation will answer the question of whether the program is meeting its objectives. The CERES program has eight
well-defined objectives.

Self-awareness referring to the ability to acquire knowledge about one’s self-physical characteristics, interests, aptitudes, abilities and attitudes.
Attitude development is defined as the development of positive attitudes toward-self and others, learning, working, leisure and individual and cultural differences.
Career awareness and exploration enhances the development of awareness of career options and opportunities and the ability to explore them in relation to interests, aptitudes and values.
Consumer competencies refer to the ability to acquire knowledge and skills to make wise consumer decisions.
Educational awareness is defined by the ability to recognize that all educational experiences are integrated with total career development and preparation.
Economic awareness is the ability to develop an awareness of the economic system.
Career planning and decision making is the ability to acquire decision-making skills and apply them to personal career development.
Career preparation is the ability to develop competencies needed for entry into the next educational or occupational levels.

The extent to which the CERES program meets each of these objectives will be tested by using all 15 randomly selected schools that have used the CERES program. A questionnaire will be passed out to students from random classes throughout the schools to measure the extent to which they perceive they have acquired each of the eight abilities. The questionnaire will ask the students to indicate the extent to which they acquired each ability on a scale of one to five (1 = not at all, 5 = to a great extent). Demographic information about the student’s grade level and age will also be collected.

The third portion of this evaluation will determine the extent to which the CERES program has accomplished its ultimate goal of having students graduate from high school who are prepared to enter the world of work or continue in their education.

In an effort to further evaluate CERES this proposal recommends that a random comparison group design be used. In this particular evaluation the treatment group will be schools that have implemented the program and the control group will be schools that have not.

The criterion group is used in the first step of analysis by estimating counterfactual weights for each variable thought to affect the outcome measure other than the program. A regression analysis will be used to examine the four outcome variables.

Research reviewed for this proposed evaluation indicates the possible moderator variables are; average socioeconomic status, size of school, teacher to student ratio, and location of school. Once the estimates for the betas in the regression are calculated, these estimates will be applied to the mean scores for the experimental group on all the possible confounding variables. This provides an estimate score for the experimental group that is stable because it was calculated based on a large sample of the population. This process controls for any pre-intervention differences on the outcome of interest, and is preferred over simply calculating regression weights with the experimental group because it takes into account experimental group bias. The analysis provides an estimated outcome, which can be compared to the actual outcome of the schools on each of the response variables.

For the present proposed evaluation twelfth grade students at both schools using CERES, and those that are not, will complete a questionnaire designed to measure vocational readiness. The items ask participants to rank how they feel on a scale of 1 to 5 (1 = very inaccurate, 5 = very accurate). In addition to the self-report questionnaire, archival data from the same schools will be collected from RAND California (http://ca.rand.org) and Internet sites that rate high schools on several different categories. These outcome variables will also be used in the random comparison regression equation to control for the same confounding variables discussed previously.

These data will be analyzed by comparing the expected outcomes to the actual outcomes. To control for possible type-1 error a conservative estimate will be collected. This will be done because schools that have decided to implement the CERES program could have something inherently different than schools that have not implemented the program. The final effect sizes for each of the outcome variables will be calculated based on these conservative estimates.

The final question this evaluation will address is in regard to any side effects the
CERES program might have. These data will be collected through phone interviews with vocational and career guidance counselors who will be asked to describe any unforeseen side effects both positive and negative.

The means for the eight items on the survey will be compiled to measure the extent to which the program is meeting each of its objectives. Demographic characteristics will be used to control for moderator effects.

In conclusion, this is a proposal for a comprehensive evaluation that adds to the previous evaluation of the CERES program. This article describes the suggested specific steps to be taken in performing an evaluation that tests the implementation, the theory, the results, and the side effects of the CERES program. This thorough evaluation will provide additional information about the usefulness of the program and will help educators decide if the CERES program can add to the K through 12 curriculum. If your school uses CERES or would like to and would be willing to participate in an evaluation study such as that described here please contact EdITS.

CAREER EDUCATION RESPONSIVE TO EVERY STUDENT

EdITS / P.O. Box 7234 / San Diego, CA 92167 / 800-416-1666 / 619-222-1666 / Fax 619-226-1666
customerservice@edits.net.