Green Plan Banner

Green Plan Evaluation
Executive Summary

 

Download Green Plan Evaluation Report [267 KB pdf]

 


 

Background

The Canada-Ontario Agreement of the Agricultural Component of the Green Plan, an equally-shared Canada-Ontario program, was designed to encourage and assist farmers with the implementation of appropriate farm management practices within the framework of environmentally sustainable agriculture. It was delivered over a five-year period from 1992 to 1997. Under the Agreement, the federal component was delivered by Agriculture and Agri-Food Canada's Market and Industry Services Branch, from the Ontario regional office in Guelph.

The three objectives of the Green Plan were:

  1. Conserving and enhancing the natural resources that agriculture uses and shares.
  2. Minimizing the impact of the agri-food sector on environmental resources required by non-agriculture sector.
  3. Protecting the agri-food sector from the environmental impacts caused by sectors and factors external to agriculture.

The Agriculture Green Plan in Ontario consisted of seven program areas, namely, Technology Transfer, Research, Best Management Practices, Rural Conservation Clubs, Wetlands/Woodlands/Wildlife, Stewardship Information Bureau and Environmental Farm Plans.

 


 

Evaluation Procedures

The program evaluation was conducted using an Evaluation Framework specified by the client. A work plan was developed and followed in conducting the data collection procedures.

The collection of data for the evaluation involved: a review of records, files and reports; and the use of 14 questionnaires with a total of 314 respondents. The personal, telephone and group interviews and mail surveys which were conducted are outlined below in the Data Collection Summary.

 


 

Data Collection Summary

Programs
Questionnaires GP TT BMP Research WWW CC EFP SIB Total
Committees 14p               14
Manager & Staff   2p 3p 5p 2p 8t 4p
10t
4p 2p 40
BMP Writers     2p           2
Researchers       10t         10
WWW Participants
Group
        35m
3g 28
      35
3-28
CC Type A
Participants
Group
          5t
18m
2g 11
    5
18
2-11
EFP Prog. Reps
Technical
Committee
            12t
9t
65m
  12
9
65
SIB Extension
Farmers
              24t
45m
24
45
Code: p - personal m - mail
t - telephone g - groups

The data were tabulated and analyzed to answer the various evaluation questions identified by the evaluation framework.


 

Evaluation Findings

Although the focus of the evaluation was the Green Plan, it became apparent that data would have to be collected, not about the Green Plan per se, but about each of the seven programs. The responses of those individuals who designed, delivered or participated in each program were presented for each evaluation issue. Only the overall summary for each evaluation issue is outlined below. For further details of responses regarding each program, see the body of the report.

The general findings for each evaluation issue were as follows:

1.0 Rationale

1.1 Were the program goals and objectives of the seven programs consistent with Green Plan environmental objectives?

A large majority of respondents had a copy of or claimed they were aware of their program goals. They believed their program goals were clearly specified, consistent with Green Plan goals and were appropriate. The consistency of responses indicates an unusually high awareness of and support for the program goals.

The program goals identified for the seven program areas were consistent with the Green Plan environmental objectives. We also determined that a high degree of awareness of these goals exists throughout the program delivery staff.

1.2 Were the operational objectives clearly specified and consistent with program goals?

The respondents overwhelming believed that the operational objectives of their work plan were clearly specified and consistent with program goals. They also claimed a high awareness of their work plans.

1.3 Was there a continuing need for each of the programs?

A need exists for programs which provide original research on agronomic and environmental problems as well as a system which communicates this and existing information to farmers and landowners. Farmers need both technical and management data. While the specific programs have not been evaluated, it is obvious that there are a very large number of landowners who have not yet adopted many of the known means of enhancing the environment. The EFP, BMPs, Conservation Clubs Type B and the WWW projects were innovative and demonstrated new ways to encourage farmers and landowners to better manage their soil and water resources.

 


2.0 Objective Achievement

2.1 Were the expected operational outputs, measures and targets clearly specified?

The operational outputs and measures were clearly specified but the actual numbers of participants were only specified for the Technology Transfer and EFP programs.

2.2 Were the program activities carried out as planned?

Based upon the evidence available, primarily self-reports, the programs were generally carried out as planned. A number of program delivery problems were identified but most did not jeopardize the overall success of the programs. The SIB was an exception in that it was not carried out as initially planned.

2.3 Within each program area, was there flexibility in responding to program needs and making adjustments?

A majority of respondents interviewed from each program believed there was adequate flexibility in responding to project needs and in making necessary adjustments.

2.4 How likely is it that the programs will achieve their objectives by March 31, 1997?

A relatively wide range of opinions were found as to the percentage of the program objectives expected to be achieved by the end of the program. About one-fifth of the respondents did not know or would not make an estimate. Most believed 71% or more of the objectives would be achieved. The most optimistic were the researchers, all of whom believed over 90% of their objectives would be achieved. The members of the three Committees were less optimistic than the others as half of those who made an estimate said 70% or less of the objectives would be achieved.

2.5 Has involvement with Green Plan activities introduced environmental considerations into normal business and program decision making by farmers?

The responses of the program participants provide very strong evidence that the farmers interviewed and those farmers observed by the program planning and delivery staff consider environmental issues when planning their farm activities. The consistently high level of consideration of environmental issues shown by the various respondents, especially by the farmers, indicates this objective is being achieved.

2.6 To what extent did participation in Green Plan activities influence participants' practices?

Participation in Green Plan programs had a major influence on the adoption of environmentally sustainable agricultural practices among participants. Among the Conservation Clubs and WWW participants, approximately 90% said the program influenced their adoption of new practices. The reports of over 80% of the Conservation Clubs staff and EFP local county committee members confirmed that their programs had also influenced farmer adoption of new practices.

 


3.0 Program Design and Delivery

3.1 How effective was the delivery approach of each program?

The various programs were delivered in different manners, some directly by AAFC, such as Research and Rural Conservation Clubs, some by third parties such as WWW, EFP and SIB. The others, Technology Transfer and BMPs, while partly financed as third party delivery activities, were completed by committees made up of AAFC and OMAFRA staff with the help of consultants.

Alternatives to various programs were proposed. The majority of respondents did not believe their program could be delivered at a lower cost. The Committee members suggested reducing the cost of the Research, EFP and BMP programs.

The interviewees, including managers, staff and participants, generally believed that their program delivery approach and the actual on-site delivery were quite effective. In most cases, a few individuals dissented. One of the most frequent complaints was that there was either too much administration or the administrative costs were too high.

 

3.2 Did the priority settings and fund allocation process lead to optimum environmental benefits within the given budget of $25.6 million?

While we believe the priority setting and fund allocation are reasonable, there is no empirical means of determining, at a reasonable cost, the optimum environmental benefits. Additional funding of research on adoption and diffusion theory and training in extension methods was discussed.

The failure of the EFP program to meet their participation objectives and under utilization of grants could be considered the single largest error in allocation. These monies were later allocated to other programs mainly related to ground water management. The SIB was almost terminated early in the program but a decision was made to continue the Bureau. Closure would have had relatively limited impact on either the overall program achievements or budget.

The Green Plan evaluation activities were not optimized because they were: late in starting; disjointed by the use of three consulting teams and changes in the evaluation design; underfunded; and, in the case of the benchmark survey, extremely rushed. The result is three evaluation studies which lack continuity because they each address very different issues and do not facilitate detailed measurement of the environmental benefits of the Green Plan or the seven programs.

 

3.3 Did the farmers have access to sufficient technical and economic information to determine the feasibility of adopting sustainable agricultural or environmental practices?

On the basis of the relatively consistent comments of the respondents, we conclude that the programs provided adequate technical information on various innovations. The generally perceived inadequacy of economic information is symptomatic of a long standing emphasis on the biological and technical characteristics of innovations at the expense of the economic costs and benefits. Greater emphasis should be given to the profitability of innovations as a means of encouraging their adoption.

 

3.4 Were there barriers to the adoption of sustainable agricultural and environmental practices which were not addressed by Green Plan?

The Green Plan programs did not address all barriers to adoption. The areas most frequently ignored were economic in nature. The lack of information on economic returns or implications of the practice, the cost of investing and the sources of financing were all cited by the respondents interviewed. The results are consistent with those found for the previous question.

 

3.5 What are the most appropriate indicators of an environmental program's success?

The respondents were able to differentiate between indicators of the success of an environmental program and those which are indicators of environmental improvements at the farm level. The answers provided tend to overlap quite a lot with slightly more specific innovations being identified as on-farm indicators.

The most frequently cited indicators of general and on-farm successes were: the level of adoption of specific innovations; measures of improvements in the environment such as water quality and achievement of program objectives. There was no consensus among the respondents on either question. The most obvious indicators appear to be measures of the extent to which a new practice or behaviour has been adopted or tried by the farmers and measures of changes in the environment.

 


4.0 Other Impacts and Effects

4.1 What unintended impacts and effects resulted from Green Plan activities?

Many unintended program effects were identified by the respondents. Others were apparent to the evaluation consultants from reading various reports and meeting with program personnel. Most of the unintended impacts were positive in nature and flow from the success of specific programs or activities. Some could have been anticipated but were not identified in advance because they were not objectives of the specific programs.

We believe the following positive impacts have occurred:

  1. New Working Relationships Were Established
    These developed within the agricultural industry, within rural communities, between rural and urban groups and between landowners and various government agencies and non-government agencies. These occurred within the three Accord Committees, during several programs and as a result of specific projects.
  2. Existing Organizations Were Empowered
    Third party delivery provided existing organizations with financial benefits and prestige. Rural Conservation Clubs and WWW projects gave local groups the opportunity to undertake useful projects. The OFA and OSCIA gained experience and credibility. This empowerment helped establish the socio/political environment which made the Agricultural Adaptation Council possible.
  3. New Groups Were Created
    The Rural Conservation Clubs and WWW Clubs brought people together at both the organizational and operational levels. A few of these will continue to undertake similar or new projects.
  4. New Information Sources
    The web site established at London by AAFC provides a new scientific source of environmental information which is available to rural and urban populations.
  5. International Recognition
    The EFP and BMPs have been recognized by and materials have been purchased by other Canadian provincial and foreign agencies.
  6. New Environmental Policies
    The on-farm fuel storage policies and regulations have been rewritten as the result of the EFP program.

The negative unintended impacts tended to be related to project failure or to be very situation specific. The impacts were relatively minor such as unhappiness with a contractor, excess geese and planting trees improperly.

 


5.0 Alternative Delivery Systems

5.1A Is there an alternative management structure which would have been more efficient or effective?

The management system was, as stated by the respondents, excessively complex, diffuse, slow and cumbersome. The system of three Committees plus an administrative officer, but not a manager was designed to provide high level involvement by the many stakeholders.

We believe the program would have benefited from the designation of a single program administrator who would have had primary responsibility for managing the Green Plan.

 

5.2 Is there an alternative delivery mechanism?

The opinions of the respondents regarding third party delivery were mixed with many people providing responses which indicate a limited understanding of all the potential cost factors or the level of actual costs. One cannot determine the incremental costs of third party delivery until such time as benchmark studies have been conducted on government delivery costs.

The extent to which third party delivery is effective depends upon the type of program being delivered and who the third party is. Third party delivery works best when the clients identify more closely with the third party than the first party. It also works better when the program is primarily action oriented, requires on-site delivery and is of intermediate technical requirements.

One consideration must be the degree of accountability. The first party needs to develop precise, measurable objectives which are agreed to by the delivery agency. The work activities of the third party should be monitored to assure performance. Accountability should be maintained in all situations even those of a political nature.

 


6.0 Future Directions

6.1 What are the environmental priorities of the agricultural sector?

Overall, the subjects or issues receiving the greatest number of mentions by the respondents were: soil conservation, management and quality; water quality, both surface and ground; reduced pesticide use; waste management; and the protection of natural areas, habitats and the environment.

6.2 What are the respective roles of farmers, farm organizations and governments in responding to the agricultural sector priorities?

The respondents perceived the most important role of each to be:

AAFC Research and development, program funding and policy development
OMAFRA Technical expertise, extension services, education, research and funding
Farm Organizations Lobbying, education, program delivery and promotion
Farmers The adoption of environmentally sustainable practices, participate in protecting the environment and informing themselves and society of environmental issues
 

 


 

Conclusions

  1. The Green Plan can generally be considered a success. A body of new knowledge was created and several innovative means of delivering information to farmers were demonstrated.
  2. The Green Plan facilitated the development of new alliances, relationships and partnerships.
  3. The management structure would have been stronger and more effective if a single manager had been responsible for overall management control.
  4. Third party and multiple party program delivery procedures were generally successful. The relative merits and costs of third party delivery cannot be determined until benchmark studies have been completed. Which types of programs can best be delivered to what clients under which circumstances should be determined.
  5. There is a need to provide training programs for individuals who will manage future third party program delivery projects.
  6. Additional work should be undertaken to capture information regarding the experiences of those people who delivered the Rural Conservation Clubs and WWW program.
  7. Problems associated with the evaluation function were identified.
 

Recommendations

  1. Future multi-program government initiatives should consider the designation of a full time program manager.
  2. Third party program delivery should be investigated to ascertain comparative costs from existing records.
  3. Suggestions for maintaining the accountability of third party delivery agencies are provided.
  4. A study to determine the management and extension experiences of the Rural Conservation Clubs Type B projects and the WWW projects should be conducted.
  5. A training program for third party program delivery staff should be developed.
  6. Greater priority be given to research on technology transfer theory and extension training. Pertinent related diffusion, adoption and communication literature should be posted on the AAFC website.
  7. The program evaluation function should be revised by giving this activity higher priority, having the same evaluator over the full time span of the program, maintain consistent procedures, increasing the level of financing and increasing awareness of evaluation procedures.
 

 

Green Plan Home

 

Created: 16 November, 1996 13:26:59
Last Updated: Wednesday, May 11, 2011 04:19:47 PM