Michie, M. (1995). Evaluating teacher perceptions of programs at a field study centre. Science Teachers Association of the Northern Territory Journal, 14, 82-92.
Formal evaluations of teacher perceptions of programs at a field study centre are examined as an indication of the suitability of the programs and their effectiveness in promoting environmental education. Evaluation enables initiation of appropriate changes where these are necessary. The procedure of carrying out an evaluation and its outcomes at one field study centre are examined.
Programs offered by educational centres of various types in Australia have seldom been evaluated formally to determine the effectiveness of their learning experiences. The 'warm fuzzies' when students leave the centre, the letters they send and comments by teachers indicate to the staff that things must be going fine. Many centres lack the resources to undertake thorough evaluations. Much evaluation of their programs results in little more than retrospective judgement by the staff members concerned.
This paper reports on two evaluations that were conducted at the Channel Island Field Study Centre (CIFSC) in 1989 and 1991. The rationale for the evaluations is explained in the context of the field study centre:
CIFSC was opened in 1987 and is located on an island in Darwin Harbour. It is normally accessed by road as it is connected to the mainland by a bridge and is 50 km from the city centre. The development of the power station on Channel Island since 1983 lead to the opening of the field study centre there. A variety of programs are offered, in marine studies (mangroves, coral reef, rocky shoreline), terrestrial habitat studies, geoscience (sedimentary rocks and processes), social history (leprosarium), energy (power station) and environmental impact. Some visits incorporate programs from more than one area. These programs are offered at primary and secondary levels. As well, there are visits from tertiary institutions, mainly in education and science. There has only been one member of staff and he is secondary science trained.
Physically, the field study centre includes a large demountable building left on the power station site; there is a large classroom space, kitchenette and toilets, and areas for displays and library. It is air-conditioned and provides some relief during the warmer parts of the year. It is remote from the mangroves and other marine habitats, and these areas are often visited without going to the building.
The main aim of the evaluation was to find out about teacher perceptions of the learning outcomes, programs and resources of the field study centre. It was an opportunity to determine if the learning outcomes and materials were appropriate, particularly for primary students. The evaluation also provided information relevant to deciding future directions for the development of the field study centre. The need to evaluate the programs was determined primarily because of the diversity of programs and user groups. In such a situation it was difficult to ensure that programs being offered were what the teacher wanted. Often on-the-spot responses can be misleading, and the evaluation enabled the teachers to examine the visit in the perspective of the whole of the learning program.
The 1989 evaluation coincided with a change in the centre's funding arrangements, but was not a requirement of either the previous or present funding body. Additional questions in the 1991 evaluation were designed to determine the effects of changes implemented since, and in some cases as a result of, the 1989 evaluation.
Webb (1980, 1989) has carried out some research on field study centres in Australia. She evaluated the "effectiveness of established Field Studies Centres in relating the outcomes of their programs to primary school pupils' understanding and appreciation of environmental issues" (Webb 1980, p.84). In a subsequent study (Webb 1989) she looked at the role of field study centres in environmental education and one of her suggestions was that they be renamed environmental education centres. McIntyre (1990) looked at the background to evaluation of outdoor education programs, giving some examples of the techniques used. He examined several aspects of evaluation, including methodology, design and data collection techniques.
Webb (1980) considered the acquisition of knowledge and the change of attitudes in students exposed to an field study program rather than the perceptions of teachers. The 1989 CIFSC evaluation predates McIntyre's publication and the 1991 evaluation follows a similar pattern to the first, but the techniques used in both CIFSC studies are similar to those he outlined (McIntyre, 1990).
The evaluation instruments
The instruments used in both evaluations were self-administered questionnaires. The original instrument for the 1989 evaluation consisted of thirteen questions covering a wide range of areas of interest. For the 1991 evaluation some of the questions were expanded and a fourteenth question added. The questions are shown in Table 1; an asterisk beside a question indicates that it was added in 1991. Questions included both limited (yes/no) and open-ended (write-in) responses.
Questionnaires were sent to teachers who had visited the field study centre in the previous two years. Respondents were given two weeks to return the questionnaire, although responses received after that time were included in the analysis. Respondents could be identified and thus it was possible to match them to details of their visit to the field study centre. Completion of the questionnaire was entirely voluntary.
In the 1989 evaluation, 24 (49%) of the 49 questionnaires distributed were returned. In the 1991 evaluation, 75 questionnaires were distributed and 31 (41%) responses were received.
Numerical responses to both evaluations are shown in Table 1; they have been converted to percentages for the number of actual respondents. Written responses to each of the open-ended questions were compiled and considered with regard for their significance to the aims of the evaluation. Some of these are quoted or referred to in the text. A number of issues were identified from the responses and these have been given special consideration (see below).
Q1. The purpose of this question was to determine how teachers had originally heard about CIFSC. This question was expanded for the second evaluation to include attendance at inservices and the range of publications; emphasis was placed on these areas as a result of the 1989 evaluation.
The results show that colleagues appear to be the main source of information, and it is reasonable to suggest that teachers who have a good experience will recommend the field study centre to their colleagues. Information sent to schools is also important, but how this is distributed in schools is unknown. Field Notes, a quarterly newsletter, was read by most respondents. There may be confusion between the handbook and other brochures.
Inservice education or training is not a prerequisite to any field study centre visit. The author feels that where it has been offered it gives the best background to CIFSC's philosophy and resources.
Q2. This question was included to find out how teachers perceived the field study centre and what it offered. Responses were generally in terms of the objectives of the science or social education curricula, rather than environmental education. Most responses of CIFSC's offerings were stated in terms of "hands-on" learning experiences in the natural environment. The original purpose of CIFSC was to provide science-type field studies but the emphasis was changed during the early stages.
Q3. Teachers were asked how their visit match their expectations in terms of the program offered and the facilities available. Most answers here were positive in both areas, with some negative comments on matters beyond the control of the education officer (although not necessarily the teacher), such as large numbers of students, variation in the numbers of animals from time to time, and lack of power at the demountable during a series of visits in a very hot period. Because of natural cycles there are occasionally times when few animals are seen in the mangroves; unfavorable comments seem to indicate that there is a lack of understanding of the natural cycles or an expectation of a Disneyland-type display.
Several respondents equated the field study centre with the building, not realising that it extends to wherever the educational program had been carried out. The perception that the areas which are visited are more truly the field study centre needs to be enhanced.
Q4/Q5. These questions were used to find out how teachers had programmed the class visit into their programs. The majority used their visit to the field study centre as part of their general classwork, with a few using it either as an introduction (although some may have used 'Channel Island' as a separate theme) or conclusion. Table 2 shows the range of responses to Q5 indicating how well they do this and the variety of classroom activities a visit can generate.
Q6. The purpose of this question was to determine whether the materials used on the visit and the experiences gained were appropriate. Much effort has gone into the preparation of materials which students use at the field study centre, and ensuring that the students have appropriate experiences. The responses indicate in general this is the case and that teachers appreciate this, particularly as much of the development of materials has been with the assistance of teachers.
Q7. This question, which asked which activity was the most valuable, caused some confusion because there was only one focus to the visit for many groups. Some teachers responded "all"; other comments included "reinforce theory in the field" and "usually (students are) not able to study habitats in such detail and were fascinated with what they saw".
Q8. The responses to this question, asking what was liked least about the visit, are listed in Table 3(a), and most of them related to environmental factors which are beyond human control, although there were disparaging remarks about students ("whingeing kids"). Other responses were "never had a bad experience" and "need to think of excuses to go". The solution to proposals relating to the students' physical environment should be sought through effective planning and by programming the visit during the appropriate season.
Q9. Here teachers were given an opportunity to suggest any changes or improvements and these are listed in Table 3(b). Many of the responses were constructive and have been or are being put into effect, particularly those relating to education.
Q10. The original question here asked how teachers perceived the programs as inspiring conservation. In both evaluations, the great majority agreed that the programs inspired conservation. Some teachers disagreed but cross-referencing showed that often they were making short visits to the power station. Some teachers of senior students also made a negative response, and it would appear that the academic side of the programs at this level tends to override the conservation aspect. One teacher was against visiting the coral reef because of the fragility of the environment, and another felt that "children pay lip-service to what they think you want".
Two new questions were included here in the 1991 evaluation. The first arose from Webb's (1989) suggestion that field study centres should be renamed centres for environmental education. The majority of respondents agreed that the site was a centre for environmental education; a name change was not implicit in the questionnaire. One respondent disagreed but made no comment.
The final part of the question related to aspects of environmental education beyond the field study centre. Respondents were more concerned with issues about habitats rather than lifestyle; one response asserted that it was "better to concentrate on Channel Island".
Q11. This question was designed to determine how effective communications were between the education officer and teacher. As the field study centre is distant from the client group, the education officer is located closer to Darwin. Most respondents considered communications satisfactory, although many would have had to leave a message and be contacted later. Some teachers would prefer to have the worksheets beforehand and/or classroom visits.
Q12. Teachers were asked if their students enjoy the visit, and most did, although many (or at least their teachers) find the environment physically uncomfortable, particularly prior to the wet season. Primary students seem to enjoy their visits more whereas secondary students are more likely to complain about the physical conditions (weather, need to walk, lack of water and shade). This is also seen from letters written by students after and comments during or after their visits. Primary students often comment positively about being able to "explore" the island.
Q13. Teachers were asked if there was any value in bring their class back for a second visit. Most teachers agreed and would examine a different aspect if they were to make return visits, although some were interested in consolidation and extension work. However, in reality this has hardly ever occurred, primarily due to the cost of transport.
Q14. In the 1991 evaluation, teachers were asked if they would return within the next twelve months, to determine if future use could be predicted. The overall 41% response to the questionnaire suggest the responses are of little use in this respect, and this is confirmed by the low number of returning teachers.
As a result of the 1989 evaluation five issues relating to the organisation of visits to the field study centre were identified. These were passed on to teachers by inclusion in Field Notes and in later information ( e.g. handbooks and information sheets) about the field study centre. These issues are discussed here in much the way they were originally circulated back to teachers.
1. Numbers. A problem in some schools is the need to justify the expense of an excursion by bringing more than one class group at a time. However working with larger groups is difficult and often some students (and occasionally teachers) feel that they aren't getting their money's worth. More frequently it's an excuse for students to "goof off". At other times it will mean that the groups won't be able to visit some areas of the island, particularly the boardwalk.
2. Timing of visits. The best visits are the ones that aren't rushed, so that there is time for students to carry out their investigation fully. Short visits give little leeway for discovery and these visits have to be more structured. Unfortunately the education officer cannot ensure that the right animals are in the right place at the right time nor hold back the tide.
3. Worksheets and assessment. The education officer can prepare worksheets for particular visits but they are not written usually as assessment items. Firstly, most of the worksheets are written to focus attention; some things are only seen at particular times of the year. Secondly, they aren't written to keep the kids busy; being busy thwarts observation. Thirdly, if they are for assessment let him know and they can be directed that way.
Rather, assessment should be of what the students learned from the excursion rather than what they did. If you want some follow-up activities for assessment then let the education officer know.
4. Relevance. Visits to CIFSC are not picnics or holidays. They are working days and students are expected to be doing fieldwork related to their class work. (Hopefully they will enjoy the experience.) Some teachers seem to think that they have to examine every facet of Channel Island in one visit: even if this was possible it isn't desirable.
5. Discipline. Discipline remains the responsibility of the classroom teacher. However, the education officer is a teacher and can maintain effective discipline, and teachers should be willing to pass over control. This will allow the teacher to observe what their students are learning, although they should maintain peripheral discipline (e.g. if students move away from the group or get off task when engaged in field work).
What do the evaluations indicate?
Comparison of the numeric responses to the two evaluations indicated that the centre was seen to have improved its programs between 1989 and 1991. Virtually all questions with a numeric response show an increase from the 1989 to 1991 evaluations.
Negative responses by teachers to some questions indicate areas which need to be examined more closely. These, together with the responses to Q.9 (changes and improvements), are the main areas where the evaluations indicated ongoing development.
1. Effective communication is necessary to ensure teacher support and student participation. Brochures need to be directed towards teachers rather than just schools, especially the CIFSC handbook which contains valuable information about visits. Also, teachers need to be informed about the materials to be used prior to the visit.
The practice here has been changed so that wherever possible the education officer visits the teacher to discuss the outcomes of the excursion, the materials to be used, and to any other arrangements. Background information (handbook or information sheet) is also supplied as well as discussion physical conditions likely to be encountered.
Another strategy which has been tried wherever possible is for the education officer to visit the class prior to their excursion, to discuss the background to the excursion. This allows him to discuss other aspects, including learning outcomes and the students' expectations. Often a video, The Conservation Story (PAWA, 1987), is shown; this gives students an overall impression of what they will find at Channel Island, and its history. Although this has not been tested, it appears that the familiarity established allows learning to proceed more smoothly and with fewer discipline problems. Novelty-reducing treatment has been used with students visiting science museums and included a slide/tape program designed to substitute for an actual visit (Kubota and Olstad, 1991).
A number of attempts have been made to get information into schools, through the principals or the librarians. This remains a difficult area; the implication from the evaluations is that teachers who have already visited should be targeted to pass information on to their colleagues.
2. Programs need to be more explicitly conservation oriented and identifiable as such; this applies to power station visits in particular as well as all other visits regardless of academic level.
The problem which is perceived here involves the conflicting definitions of environmental education. By adopting an education for the environment philosophy there is no area of the teaching at Channel Island which should not be conservation-based.
3. Extension and general environmental education programs should be supported but not at the expense of field programs.
4. The centre's infrastructure needs to be developed to allow for more effective use of the area; this includes completion of boardwalks and pathways and provision of interpretative materials.
5. Teachers should be encouraged to use the facilities when the climate is most suitable, but should not disregard features which can only be visited in the hotter months ( e.g. coral reef, monsoon vine forest). They should be informed about the need for appropriate preparation at these times.
Acknowledgment: The assistance of Anne Richards (Evaluation, Research and Assessment Branch, Northern Territory Department of Education) in the design of the evaluation instruments is gratefully acknowledged.
Kubota, C.A., and Olstad, R.G. (1991). Effects of novelty-reducing preparation on exploratory behavior and cognitive learning in a science museum setting. Journal of Research in Science Education, 28(3), 22 -234.
McIntyre, N. (1990). Programme evaluation in outdoor education. In K. McRae, ed., Outdoor and Environmental Education: diverse purposes and practices. Macmillan, Melbourne, 204-231.
PAWA. (1987). The Conservation Story. Power and Water Authority, Darwin (video).
Webb, J.B. (1980). A Survey of Field Studies Centres in Australia. Australian National Parks and Wildlife Service, Special Publication, 4, Canberra.
Webb, J.B. (1989). A Review of Field Study Centres in Eastern Australia. Australian National Parks and Wildlife Service, Occasional Paper, 17, Canberra.
Table 1. Summary of questions in both 1989 and 1991 evaluations, with results for numerical responses. An asterisk indicates questions which were not included in the 1989 evaluation.
|1(a)||Indicate how you found out about the Channel Island Field Study Centre?|
|- through inservice||13||30|
|- *preservice at DIT/NTU||*||13|
|- from brochures sent to schools||47||37|
|- from colleagues||54||57|
|- other (please state)||29||13|
|*(b)||Indicate if you have attended an inservice:|
|- specifically about CIFSC||*||13|
|- that visited CIFSC as part of the program||*||16|
|- where you took part in a workshop on CIFSC||*||10|
|*(c)||Indicate which of the following field study centre publications you have read.|
|- CIFSC brochure||*||73|
|- "Field Notes"||*||77|
|- CIFSC Handbook||*||27|
|- Occasional Papers||*||37|
|- other (please state)||*||13|
|2||What did you perceive the field study centre to be and what did you think it offered?|
|3||Did your visit match your expectations in terms of:|
|4||Which of the following teaching situations was appropriate to your visit?|
|- as an introduction to a unit of work||13||37|
|- part of a unit of work||70||77|
|- an end point||4||20|
|- not related to school lessons at that time*||*||7|
|5||What follow-up work was done with the students after the visit?|
|6(a)||Were the materials appropriate?|
|6(b)||Were the experiences appropriate|
|7||Which activity was most valuable?|
|8||What did you like least about the visit?|
|9||Have you any suggestions for changes or improvement?|
|10(a)||Do you think the program inspired conservation of the environment?|
|*(b)||Do you think that the Channel Island Field Study Centre can justify being called "a centre for environmental education"?|
|*(c)||Do you think that the field study centre should be involved in other aspects of environmental education in schools?|
|11||Were pre-visit communications and information adequate?|
|12||Did your students enjoy the visit?|
|13||Would another visit to the centre with the same group of students be of value?|
|*14||Do you expect that you will make another visit to the field study centre in the next 12 months?|
Table 2. Classroom activities generated by visits to Channel Island (responses to Question 5).
|Learning outcomes||Classroom Activity|
|General outcomes||completion of worksheets;recapitulation of points learned; further excursions; video and worksheet; follow-up visit by education officer; none|
|Language outcomes||letters; language/texts; reports and recounts; discussions; projects; narrative writing; incorporated into case study; class book; photographic display; summary; refinement of learning from books; illustrations; story writing; library research|
|Science and social education outcomes||research on mangrove environments; analysis of garbage collection; investigated properties of salt and fresh water; displays of rocks and shells; lessons on ecosystems; discussion of effects of adaptations observed; research on energy production; study of sea life; experiments on sediments; a unit on rock types; maps; work on natural features of NT|
Table 3. Areas where the evaluations indicated need for changes.
(a) Responses to Question 8, about what was least liked about the visit.
|sandflies||lack of time|
|heat and sun and time outside causes difficulties||the heat|
|never had a bad experience||need to think for more excuses to go|
|absence of plant names||whingeing kids|
|little interaction/discussion with students||unfinished boardwalk|
|talks too long for age of children||too hot - better in the dry|
|lack of accessible information||damaged shoes|
|climb down to reef||enclosed classroom|
(b) Responses to Question 9 suggesting changes and improvements
|Educational||more models and displays; more information prior to visit; microphone on nature walks; some children need time to explore before settling down to work; activity sheet to do as we walk around; more classes to be made aware of the excellent facilities; examples of organisms; keep sending out information and having inservices; video as summary; ongoing data collection; more interactions; comparative studies|
|Physical environment||located away from power station; water cooler & cups; discovery trails; toilets and water on track; outside shade and seating; complete boardwalk to open water; name tags on plants; precise sketches of geological formations; cool water bubblers across the island; continuation of boardwalk|
Michael Michie's Home Page