Ontaro - Canada Logos

NSCP Banner
 

Evaluation Assessment


TABLE OF CONTENTS

 

 NSCP Home Page

 

 

INTRODUCTION

In recent years, soil and water conservation has taken on increased importance in Canada's agri-food industry. Public concern for the environment has increased awareness of soil degradation issues. In November, 1986, soil and water conservation was defined as one of five elements of a long-term National Agriculture Strategy endorsed by the federal and provincial Ministers of Agriculture. The strategy defined a federal-provincial partnership and called for the negotiation of joint agreements to finance and coordinate agricultural soil and water conservation and development programs.

In December, 1987, the Prime Minister announced the $75 million National Soil Conservation Program. Of this total, $45 million is being provided by the Department of Western Economic Diversification, and $30 million by Agriculture Canada. The allocation of these funds, by province, was determined on the basis of estimates of the rates of soil degradation across the country. Subject to federal-provincial Accords/Agreements, federal allocations are being matched equally by the provinces.

The NSCP has two major objectives:

  • to encourage the implementation of the most appropriate soil resource management and use, within practical economic limits and according to soil capability, in order to sustain the long-term productivity of the soil; and

  • to encourage economic diversification in Western Canada.

In addition, the NSCP included a provision for complementary soil conservation activities to be undertaken to secure and enhance waterfowl habitat in Western Canada.

An extension to the NSCP announced in the Green Plan, the Environmental Sustainability Initiatives under Special Income Assistance Program, and Permanent Cover Program II under FSAM are all indications of continued government commitment for soil conservation and land management activities.

An evaluation of the NSCP has been requested by senior management of Agriculture Canada to measure the extent of implementation and the appropriateness of soil resource management and use initiatives, and to provide information for the development of future conservation initiatives.

This Evaluation Assessment is an examination of the extent to which the NSCP can be evaluated.

Assessment Approach

Two principal methods were used to gather information for the Evaluation Assessment: consultation with an Advisory Committee and senior Agriculture Canada officials; and a review of internal files and documents.

An Advisory Committee was established to provide advice on the implementation of the Evaluation Assessment. Composed of members from across the country, the Committee included provincial, producer group, and Agriculture Canada representatives. The Advisory Committee provided information on programs and suggestions on issues to be considered in the evaluation study.

A review of internal files and documents was carried out in order to gain a broad understanding of the program and how it was set up. The review included the Accords and Agreements under which NSCP was carried out, the accompanying work plans and annual updates, as well as internal Agriculture Canada files.

PROGRAM DESCRIPTION

The National Soil Conservation Program is carried out jointly, by the federal and provincial governments. Subject to the consideration of five basic elements, provinces were allowed to direct funds to meet individual provincial interests. The five basic program elements eligible for funding under NSCP are:

  • Awareness -- This element consists of local, regional and national awareness elements. It can include provision of materials such as films, pamphlets, school teaching aids and other materials; provision of speakers bureaus; and Soil Conservation Week materials.

  • Research -- Research encompasses both basic and applied research into specific problems such as compaction and cropping systems and will be conducted at federal and provincial research stations and at universities.

  • Soil Survey and Monitoring -- The survey and monitoring element is designed to provide better estimates of soil quality and rates of degradation for the agricultural areas of Canada, as well as various characteristics of agricultural soils.

  • On-Farm Technical and Financial Assistance -- This element is intended to supply the technical and financial assistance to allow farmers and farm groups to develop and demonstrate soil conservation techniques. Assistance is short term, aimed at helping farmers to develop financially viable soil conservation practices. Long term assistance is not contemplated.

  • Permanent Cover [or "Land Use Adjustment"] -- This last element is aimed at converting marginal lands subject to severe degradation from annual crop production to permanent vegetative cover. Although permanent cover was initially targeted only for the Prairie Provinces, it has also been included in Ontario's program.

Given that each Agreement was driven by provincial priorities, provincial sub-programs vary by province, ranging from a low of three components in Newfoundland to a maximum of eleven in Alberta. In general all elements were covered in each province but combinations of the five elements were common.

All provinces with the exception of Newfoundland 1 signed formal Agreements with work plans outlining the activities to be undertaken within the province. The Agreements are managed by a Management Committee of an equal number of federal and a provincial officials, usually co-chaired by a federal and provincial member. In most cases, an Implementation Committee was appointed to deal more directly with the implementation of the Agreement.

_________________________

1

Given that only $100,000 was allocated to Newfoundland, the province decided not to sign an formal Agreement, instead the program was based on an exchange of letters between federal & provincial officials.

The following table shows the distribution of the federal contributions by province, matched in each case with provincial funds. The total of federal and provincial funds is $150 million.

Federal Contributions by Province

Province Amount Negotiated
($ millions)
Date Signed
B.C. 2.5  August 1990
Alberta 17.4  July 1989
Saskatchewan 27.0  July 1989
Manitoba  9.0  August 1989
Ontario 11.1 April 1990
Quebec 5.3  November 1988
P.E.I. 0.9  August 1989
N.B. 1.2  August 1989
N.S. 0.5  March 1990
Newfoundland 0.1  April 1989 2
Total 75.0   
2  Letters of Agreement only

 

A separate document has been prepared which contains program profiles by province These profiles more fully identify the provincial programs, explain the elements and provide the allocation of funds by sub-program. Copies of this document may be requested from Program Evaluation Division.

 

Related Programs

Federal Programs with soil conservation objectives include the following:

Environmental Sustainability Initiatives under the Special Income Assistance Program (1990) - Fourteen million dollars was allocated to environmental sustainability initiatives which included a variety of issues including soil and water conservation.

Permanent Cover Program II and Land Management Assistance Program under Farm Support and Adjustment Measures (FSAM-I) (1991) - Fifty million dollars was provided for PCP II and $22 million for LMAP. The PCP II program is a continuation of the PCP originally under NSCP providing funds for the conversion of marginal lands to permanent cover in the prairie region. The Land Management Adjustment Program encourages the conversion of annually cropped land to permanent cover in eastern Canada and British Columbia.

Green Plan (1990) - An amount of $150 million has been allocated under the heading of Achieving Environmental Sustainability in Agriculture. Of this amount $127.9 million will be delivered through federal-provincial agreements on a cost-shared basis.

 

PROVINCIAL EVALUATIONS

Evaluation requirements vary by province.

In British Columbia, Ontario and Nova Scotia there were no specific provisions for an evaluation in the Agreement, but evaluation was identified as a sub-activity in the workplan. The British Columbia Management Committee prepared an evaluation framework in which the identified issues were specific to the sub-programs. They recommended establishing base-line data early in the program (1991) and suggested that particular attention be paid to Producer Conservation Organizations (PCO). No information is currently available on NSCP evaluation work undertaken in Ontario and Nova Scotia, although Ontario has survey results for other conservation initiatives.

Each of the Quebec and New Brunswick Agreements contained a clause in the Agreement which required the Management Committee to carry out an Evaluation Framework within one year of the Agreement being signed and a follow-up review of the Framework one year before the completion of the Agreement. The New Brunswick framework identifies the issues: rationale; objectives achievement; implementation; impacts and effects; future action; and coordination (federal linkage). No information is currently available on evaluation work done in Quebec.

In the prairie provinces, there were no specific provisions for an evaluation in the Agreements but in each province the Management Committee arranged for the completion of an evaluation plan and an interim evaluation. Results of their evaluations follow.

Alberta

An Evaluation Plan was developed in October 1990 and an Interim Evaluation was completed in February 1991. The purpose of the interim evaluation was to "fine tune" the existing program and to guide the development of a replacement program.

The focus was on design and delivery, including the following evaluation questions:

  • Should CASCI emphasize specific objectives (e.g., develop awareness or improve technology transfer)?

  • What is the most appropriate distribution of funds across CASCI programs (currently and prospectively)?

Given the long term nature of the anticipated effects of CASCI and the timing of the interim evaluation, the evaluation emphasized CASCI participants' qualitative perceptions and judgement regarding the potential impacts and effects of CASCI programs.

Each sub-program was reviewed, areas of particular interest highlighted, and recommendations made. In total more that 40 recommendations were made, many suggesting that a subsequent agreement follow the NSCP.

Concluding comments indicate little evidence of major problems affecting the efficiency and effectiveness of CASCI. It was suggested that recommendations would enhance the leadership role of Alberta's program in land resource and production management and will advantageously position Alberta within the sustainable agriculture sector over the long-term. Some of the recommendations include:

  • provide assurance for funding based on 3-year or 5-year plans rather than funding based on government fiscal years;

  • encourage participants to develop systems approach to soil conservation that incorporate the interrelationships of soil conservation, moisture management, production management, agricultural sustainability and environmental sustainability;

  • develop and disseminate clear, simple explanations of the different CASCI programs and available CASCI funding;

  • develop clear, straight-forward, factual information for producers that will encourage adoption of soil conservation and moisture management practices/technology;

  • encourage demonstration projects which build upon the results of other demonstrations and ensure CASCI funded projects include a commitment to communicate the results of such to producers and extension personnel;

  • less support for equipment purchases in a subsequent agreement;

  • encourage local personnel to assume responsibility for technology transfer activities and monitoring the quality of CASCI supported projects; and

  • educate participants regarding the effects of changing annually cultivated land to permanent cover may have on farm operations as a whole.

Saskatchewan

An Evaluation Plan was prepared and an Interim Evaluation was completed in April 1992. The two basic objectives of the interim evaluation were to determine:

  • how the Agreement had operated thus far and what could be changed to ensure maximum effectiveness over its remaining mandate; and

  • what lessons had been learned from the Agreement which could provide input to the next round of soil conservation initiatives (Green Plan).

The evaluation used four data collection methodologies:

  • management and administrative file review;

  • focus group interviews with local organizations;

  • short mail-back questionnaire of local organizations; and

  • in-person interviews with Regional Specialists, Implementation Committee Executive, and Agreement Management.

Findings were categorized under the issues of rationale; awareness; adoption; land use conversion; and delivery.

The conclusion was that the Canada-Saskatchewan Soil Conservation agreement (CSSCA) has not been a cause for changed farmer attitudes toward conservation; neither has it been a catalyst for these shifts. The CSSCA has played the valuable role of facilitating movement toward greater conservation during a very difficult period in Saskatchewan agriculture. There was strong support for maintaining the Agreement in the immediate term. The following six recommendations are a sample of a total of sixteen recommendations made to the Management Committee:

  • Local organizations which are lagging in expenditures and activity should be reviewed and either strongly encouraged to increase activity or funds should be allocated to areas where effective conservation spending will occur.

  • Local organizations and other conservation groups need to meet as a group, with other agricultural professionals to share experience. This will encourage groups which are lagging to generate ideas for activity.

  • Immediately direct SOS Committees and Technicians to operate out of Rural Service Centres and arrange for services to be purchased a direct cost.

  • Clarify the role of Extension Agrologists with respect to appropriate support for SOS Committees.

  • Reduce the reliance on general promotional brochures as this is probably the least effective form of communications.

  • Special projects should be encouraged and funds should be moved between activity areas to maximize effectiveness. Extension Agrologists and other professionals will need to take the lead on this point to encourage local organizations to break out of the "menu" mentality.

Manitoba

An Evaluation Plan was prepared and an Interim Evaluation was completed in May, 1992. The two basic objectives of the interim evaluation were to determine:

  • how the Agreement had operated thus far and what could be changed to ensure maximum effectiveness over its remaining mandate; and

  • what lessons had been learned from the Agreement which could provide input to the next round of soil conservation initiatives (Green Plan).

The evaluation used four data collection methodologies:

  • management and administrative file review;

  • focus group interviews with local organizations;

  • short mail-back questionnaire of local organizations; and

  • in-person interviews with Regional Specialists, Implementation Committee Executive, and Agreement Management.

Findings were categorized under the issues of rationale; awareness; adoption; land use conversion; and delivery.

Like Saskatchewan, the conclusion was that the Canada-Manitoba Soil Conservation Agreement (CMSCA) has not been a cause for changed farmer attitudes toward conservation; neither has it been a catalyst for these shifts. CMSCA has played the valuable role of facilitating movement toward greater conservation during a very difficult period in Manitoba agriculture. There was strong support for maintaining the Agreement in the immediate term. The interim evaluation made a total of seventeen recommendations to the Management Committee, many similar to those made for the Canada-Saskatchewan Agreement. The following are some additional recommendations:

  • Devolve additional decision making to the Executive Committee to ensure Agreement funds are effectively disbursed in the last year.

  • Review the entire information management process to reduce the data capture from On-Farm Programs and the Conservation cover Program. Much of the information currently being collected is not used for management or evaluation.

  • Prepare workshops for technicians and other professions on the preparation of effective newsletters.

  • Land purchases must proceed under the (Sustainable Land Use) Program by the end of August 1992. The commitment must be reviewed to examine potential reallocations to other programs where funds appear to be in danger of lapsing.

  • Agreement Management (Committee) should regularly review activities of external agencies which receive funding and which work on behalf of Agreement objectives.

 

EVALUATION ISSUES AND QUESTIONS

Following a review of the background of the program and discussion with the Advisory Committee, four evaluation issues and thirty-one questions were identified. The four evaluation issues were: rationale; design and delivery; program results; and linkages with other programs. In order to select the questions most appropriate for the evaluation, the proposed questions were rated on the basis of their importance for the evaluation and their measurability3. The issues and selected questions are:

____________________________________

3

Thirty-one questions were reviewed for level of importance and measurability. Although all questions were considered to be important, their relative importance at the present time was rated on a scale of 1 to 3 with 3 being of greatest current importance. The degree of measurability was rated on a similar basis with the questions most easily measured receiving the highest rating. Taking the two factors into account, there were eleven questions which received a score of 5 or higher. (A question would receive a score of 5 with a high level of importance and medium measurability or high measurability and medium current importance.) The twelve questions were later broken into components to result in twenty-one evaluation questions.

Rationale

  • What was the extent and severity of soil degradation? How has that changed?
  • What are the costs/benefits of soil conservation practices:
    - input costs;
    - difference in yield;
    - vulnerability to drought; and
    - others.
  • Are the various sub-programs undertaken in each province consistent with NSCP objectives?

Design and Delivery

  • Was the process of allocating funds by province appropriate?
  • How effective was the concept of having provinces provide matching funds?
  • Have federal-provincial management committees, by province, been effective?
  • Has the NSCP used existing provincial infrastructures?
  • In what ways have producer organizations been used?
  • Has this been effective?
  • What is the optimal size of producer organizations?
  • What have been the effects of local delivery (i.e. producer involvement/delivery)?

Results

  • To what extent is technical expertise available to farmers to assist in planning production in a soil conserving manner?
  • Has there been a change?
  • To what extent has awareness increased on the part of producers, the general public, and students of soil degradation causes, effects and remedial measures?
  • To what extent has highly erodible marginal land been converted to permanent cover in Western Canada?
  • What is the number of acres under conservation farming?
  • What is the percentage of acres under conservation farming?
  • Has % acreage under conservation farming changed during the program periods?
  • To what extent has the program induced private sector investments, as a result of federal and/or provincial subsidies for capital investments for such things as shelterbelts, terracing, various drainage structures?
  • Has our knowledge and understanding of soil degradation and associated costs changed since NSCP was initiated?

Linkages with Other Programs

  • What other programs are in place which promote soil conservation?
  • Did these programs exist before the NSCP?
  • How do they differ from NSCP?

 

EVALUATION METHODOLOGY

It is proposed that the following methodologies be used to address the four evaluation issues:

Issues

Approach

Rationale File Review
Literature Review
Interviews
Case Studies
Design & Delivery File Review
Focus Groups
Interviews
Program Results File Review
Review of existing surveys
Program Data
Interviews
Case Studies
Focus Groups
Linkages File Review
Interviews
Workshop

File Review

In addition to the file review carried out for the Evaluation Assessment, minutes and reasons for decisions taken by Management and Implementation Committees will be reviewed to study the criteria for acceptance or rejection of producer proposals. As part of the file review, program data banks, where available, would be reviewed and analysed.

 

Literature Review

In an attempt to establish baseline criteria, a bibliography of documents, articles, etc. on soil conservation, written since 1987, will be developed and reviewed. It is expected that the Advisory Committee and program officials will be able to provide suggestions on documents to be included.

Interviews with Officials

Between 50 and 75 in-depth interviews will be conducted with federal and provincial officials and with representatives of producer organizations and farm groups.

The Management and Implementation Committee members will be consulted about the strengths and weaknesses of the program and have them explain more fully how they decided to accept or deny applications. Officials will be a good source of information on where to find additional files and data sources. Program managers will also be consulted on program results and linkages with other programs.

In each province, local organizations play a role in the implementation of the program. Interviews with farm group leaders responsible for on-farm implementation will focus primarily on program delivery and results. Names of people to be interviewed will be solicited from members of the Management and Implementation Committees. The proportion of interviews per province will be allocated based on the level and areas of producer group involvement by province. It is expected that the largest number of producers interviewed will be from western Canada.

Case Studies of On-Farm Components

Approximately 10 cases studies will be carried out concerning individual projects sponsored under the NSCP. Given the nature of the program and the complexity of the on-farm element, case studies will allow to close examination of several of the projects undertaken. The study would include a discussion of the project and its benefits and problems. A cost benefit analysis of the project may also be undertaken if feasible.

Focus Groups

Focus groups allow qualitative data collection by obtaining information from groups of individuals. Using a focus group guide built from evaluation issues contained in the assessment, opinion based information can be obtained from the stakeholders (farmers or farm group representatives) being addressed. The focus group allows interaction among the participants to enhance idea development.

Findings Workshop

At the end of the research, a one-day findings workshop, consisting of approximately 10 specialists in the field of soil conservation will be organized. The purpose of this panel will be to help put the results of the research into perspective for a findings report, given the long-term nature of soil conservation and the relatively short elapsed time.

Surveys

A review of the results of previously conducted surveys will be undertaken to provide baseline data on soil conservation awareness. In the western provinces, surveys have been carried out, in recent years, following Soil Conservation Week and for activities undertaken under ERDA. In Ontario at least one survey has been carried out concerning soil conservation. Advisory Committee members and federal and provincial officials will be consulted to determine if other surveys are available.

Surveys provide the evaluator with access to the greatest number of potential respondents. Surveys are also useful for addressing both participants and non-participants, an important consideration when addressing activities dealing with program awareness. Given the broad scope of NSCP, five elements and different programs in each province with multiple subprograms, a survey is one way of collecting a wide range of opinions. A major difficulty with a survey for this type of project is the number of participants necessary to get a representative sample and the resulting cost.

If the existing survey results indicate that a further survey would be necessary the feasibility of adding some questions to the existing national Farm Financial Survey, undertaken by the Farm Credit Corporation and Agriculture Canada, will be investigated.

EVALUATION MANAGEMENT

The evaluation will be the responsibility of the Program Evaluation Division with advice and assistance provided by an Advisory Committee. The list of Advisory Committee members is attached as Annex A. The evaluation will be directed by PED staff, although much of the data collection will be conducted by an outside contractor.

Data collection is expected to take place starting late August and continue through November. The evaluation is scheduled to be completed by January, 1993.

The estimated cost of contract work is $85,000; the cost of the workshop $12,000; and PED travel costs $8,000. In addition an estimated $45,000 may be required to include soil conservation questions in existing surveys to provide ongoing information in attitudes and soil conservation activities.

NSCP EVALUATION ADVISORY COMMITTEE

Chair Carol Motuz
Program Evaluation Division
Agriculture Canada 9th Floor,
Fontaine Building
c/o Sir John Carling Building 930 Carling Avenue
Ottawa, Ontario K1A 0C5
British Columbia Ronald A. Bertrand
Soils and Engineering Branch
B.C. Ministry of Agriculture and Food
1-33832 South Fraserway Abbotsford,
British Columbia V2S 2C5
PFRA Larry Lenton
PFRA
603-CIBC Tower
1800 Hamilton Street Regina,
Saskatchewan S4P 4L2
Western Diversification Ed Wiens
Manager, Monitoring and Co-ordination
Agriculture Sector
Western Diversification - Saskatchewan
P.O. Box 2025 Saskatoon, Saskatchewan S7K 3S7
Alberta Jack Hazelwood
Head, Program Coordination Section
Conservation and Development Branch
Alberta Agriculture
J. G. O'Donoghue Bldg. 7000-113 Street
Edmonton, Alberta T6H ST6

Gordon Bussey
RR # 2 Airdrie,
Alberta T4B 2A4
(Chairperson - Provincial Advisory Committee for CASCI)
Saskatchewan Mike McAvoy
Program Co-ordinator
Soils and Crops Branch
Saskatchewan Agriculture and Food
Room 133 30835 Albert Street
Regina, Saskatchewan S4S OB1
Manitoba Rask Klagenberg
RR# 1 Selkirk, Manitoba R1A 0W6

Peter Entz
Acting Chief, Soil and Water Management Section
Soils and Crops Branch
Manitoba Agriculture
Box 2000 Carman, Manitoba ROG 0J0
Ontario Galen Driver
Soil and Water Management Branch
Ontario Ministry of Agriculture and Food
52 Royal Road
Guelph Agricultural Centre
P.O. Box 1030 Guelph, Ontario N1H 6N1

Harold Rudy
Project Manager Ontario Soil and Crop Improvement Association
Box 1030 Guelph Agricultural Centre
Guelph, Ontario N1H 6N1
Quebec Jean Lamoureux
Implementation and Operational Co-ordination
Agri-Food Development Branch Gare Maritime Champlain
901 Cap Diamant,
Pièce 350-4 Québec (Québec) G1K 4K1
New Brunswick Ibrahim Ghanem
Director, Land And Soils
New Brunswick Department of Agriculture
P.O. Box 600 Fredericton,
New Brunswick E3B SH1
Nova Scotia Roland Hayman
Agricultural Services Division
Nova Scotia Department of Agriculture and Marketing
P.O. Box 550 Truro,
Nova Scotia B2N 5E3
P.E.I. Ron Dehaan
Planning and Operations Branch
PEI Department of Agriculture
P.O. Box 1600 Charlottetown, PEI C1A 7N3
National Steve Henderson
Chief, Economic Assessment
Grains & Oilseeds Branch
Agriculture Canada
Room 1039 Sir John Carling Building
930 Carling Avenue Ottawa, Ontario K1A 0C5

Christian De Kimpe, Research Coordinator,
Natural Resources
Research Branch, Agriculture Canada
Sir John Carling Building
930 Carling Avenue Ottawa, Ontario K1A 0C5

 

NSCP Home

 

Created: 09-21-1996
Last revised: Sunday, May 08, 2011 02:43:00 PM