FY 2005 Performance Budget Justification
Appendix C: Program Evaluations of ARC Investments
I. Appalachian Development Highways Economic Impact Studies by Wilbur Smith Associates, July 1998. This study evaluated both the transportation efficiency benefits and economic development impacts of the completed portions of the Appalachian Development Highway System (ADHS). The study focused on 12 of the 26 ADHS highway corridors that were largely completed, totaling 1,400 miles of the system. A primary objective of the study was to measure, in retrospect, the extent to which the completed portions of the ADHS have contributed to the economic well-being of Appalachia.
Two major effects are estimated using transportation data and a state-of-the-art, regional economic model: (1) travel efficiency benefits due to reduced travel time, lower vehicle operating costs, and the reduced number of accidents; and (2) economic development impacts on the Region due to the enhanced competitive position of industry in the Region, increased roadside business, and increased tourism. The economic model used for the study was developed by Regional Economic Models Inc., REMI, of Amherst, Massachusetts, which estimated the employment, wages, value-added and population increases attributable to the ADHS in the 165-county study region.
The following impact estimates are only for the completed portions of the system and do not include benefits from completing the system.
In fiscal year 1999, the Appalachian Regional Commission issued its first evaluation report on nonhighway ARC projects, Evaluation of the Appalachian Regional Commission's Infrastructure and Public Works Program, by the Brandow Company and Economic Development Research Group. Ninety-nine projects were examined in this study and all were initiated and completed between 1990 and 1997. Total ARC funding for these projects was $32.4 million. These infrastructure projects are representative of the range of different types of projects typically funded by the Commission, including industrial parks and sites, water and sewer systems, access roads, and business incubators. The universe of these types of projects (both open and closed) during this period was 1,376, with 663 projects closed. Thus the sample represents fifteen percent of closed projects.
Job Creation. The estimated total number of jobs created by the ARC-funded sample projects was 44,331. This includes direct new jobs and indirect/induced new jobs. It does not include the retained jobs. Overall, the economic development projects examined cost $2,412 per new job created, including indirect and induced job creation. If retained jobs are also counted, then the average cost drops to $1,761 per job. The benefit cost ratio of income generated by these jobs was 5.4 to 1 when counting only income resulting directly from job creation, but increased to 8.9 to 1 when indirect job creation was included.
Personal Income. The new jobs led to increased personal income for residents of the affected counties. For a one-time public investment in these economic development projects, there was approximately $9 of annual recurring personal income per public dollar invested.
Projections and Results. All project types performed well:
Non-residential water and sewer projects: The number of new jobs created was about 62 percent over projections, and the number of new businesses served was almost four times the projected total.
Industrial parks: Projections for new businesses and existing households served were exceeded; projections for existing businesses were met, while the percentage of actual jobs retained was slightly lower than projected (91 percent).
Incubator projects: Actual results for new businesses ran more than three times the projections. The number of retained businesses served was double the projection. The number of new jobs created was almost five times the projections, and the number of retained jobs was 50 percent above projections.
Industrial access road projects: The number of new businesses served was greater than four times the projections, while the percentage of retained businesses served was below projections (77 percent). However, the numbers of new and especially retained jobs assisted by the projects came in above projections.
Recommendations: Key recommendations in the study were that:
In fiscal year 2001, ARC issued a report, Evaluation of the Appalachian Regional Commission's Educational Projects, by the Westat Corporation, that assessed the implementation and impact of 84 education projects funded by the Appalachian Regional Commission during the 1990s. The study examined the type of activities that projects used to enhance learning opportunities, the extent to which these activities were implemented, the accomplishments associated with these activities, and whether the projects were able to sustain themselves beyond the ARC grant period. Of particular interest was the extent to which projects achieved the outcomes set forth in their original proposals to ARC. In addition, site visits were conducted at eight projects that had successfully provided community residents with a new or enhanced educational service.
Types of Performance Measured:
This study, issued in March 2001, provided an early evaluation of the Appalachian Regional Commission's entrepreneurship program. The report evaluated 24 entrepreneurship projects that were complete or nearly completed during the 1997-2000 period. The sample was generally representative of the project mix and participation rates by state as drawn from the 48 projects that were complete or nearly complete. Total ARC funding for these 24 projects was $2,124,700 which leveraged another $1,412,000 in funding from other sources. When the evaluation began there had been 133 projects funded. Through November 2000, the total universe of projects was 169, representing a total ARC investment of $17.6 million, which leveraged $13.9 million from other sources.
The study found that three-quarters of the projects had assisted firms to develop new products or upgrade new technologies. In addition, half of the projects reported starting new businesses which led to the creation of 304 new firms—46 new firms with employees and 258 firms that were sole proprietors. There were 377 new jobs created according to the survey, with 69 jobs in new firms, 50 in existing firms, and 258 through self-employment. Furthermore, there were 74 jobs reported to be saved by project interventions. Of client businesses or entrepreneurs served by the project, 70 percent reported being highly satisfied by the services they received, 17 percent were satisfied, 8 percent were dissatisfied, and the balance did not respond.
Thirty-six percent of the businesses reported that their business strategies were oriented to expand or grow the business and its market share, while 55 percent viewed their business as stable "lifestyle businesses," and the balance saw their businesses as a response to local economic decline.
The study recommended that the program increase the amount of technical assistance provided to grantees, support the development of more internal evaluation and self-monitoring systems within the projects and businesses served, make fewer small grants, recognize the risk inherent in the program and seek to replicate the successful projects. Finally, the report recommends that a broader regional economic impact from the program will require greater scale and more focus on growth-oriented businesses.
V. Evaluation of ARC's Vocational Education and Workforce Training Projects
In fiscal year 2002, the Appalachian Regional Commission issued its report Evaluation of the Appalachian Regional Commission's Vocational Education and Workforce Training Projects by the Westat Corporation. The study examined 92 representative projects started and completed during the 1995-2000 period. The sample constituted about one-third of the universe during period, after adjusting for continuation projects. A mail survey collected data on implementation, monitoring and impact. In addition, five case study site visits were conducted. A two-tier sample of projects was developed to assess the impact before and after full implementation of ARC's performance measurement system: Cohort 1 selected 67 projects from the 1995-1999 period; and Cohort 2 selected 25 projects funded in 2000.
Types of Performance Measured
Assessment of ARC's Performance Measurement
VI. External Evaluation of ARC's Telecommunications Projects
ARC issued its report a Program Evaluation of the Appalachian Regional Commission's Telecommunications Projects by the Westat Corporation. This program evaluation examined 70 projects that were started and completed between 1995 and 2001, including project investments in various types of information technology-based training, e-learning and distance learning, e-commerce, telemedicine, network and infrastructure projects, and community access center projects.
Project Accomplishments and Impacts: Most projects reported fulfilling their goals to the same or greater extent as projected. For example:
Performance Measurement: The report examined 16 projects to evaluate how the projects anticipated results were ultimately measured at the end of the project. While this sample does not reflect the overall outcome reports for all 70 projects, it did provide a more detailed picture of how ARC's performance measurement system was implemented for telecommunications projects in the late 1990s. An analysis of the 16 final reports revealed that two-thirds of the projects expected outputs and outcomes were addressed in the final reports, but 20 percent of project outputs and 55 percent of project outcomes failed to provide adequate information or data supporting their claims that these results had been met.
VII. Health Studies on Health Disparities in Appalachia and the Financial Conditions of the Health Care Institutions in Appalachia.
A. An Analysis of the Financial Conditions of Health Care Institution in the Appalachia and their Local Economic Impacts by the Center for Health Affairs of Project Hope was published the ARC in January, 2003.
This report describes the availability of health care services in Appalachia, the financial stability of Appalachian health care institutions, and the effect of hospital closures on Appalachian counties.
The analysis of Appalachian hospitals based on American Hospital Association (AHA) data revealed weakness in access to certain services that fall outside the core functions of primary-care physicians, rural hospitals, and skilled nursing care facilities. Most Appalachian counties have not been successful at improving access to dentistry, outpatient alcohol treatment, outpatient drug treatment, and outpatient mental health services.
B. Health Disparities in Appalachia
The Commission issued a contract in FY 2001 to the Office of Social Environment and Health Research at West Virginia University to analyze of the health status and disparities in Appalachia so that policy makers and regional public health practitioners can identify their health-related priorities for health surveillance and research, health education, and investments to improve the delivery of health care, and health outcomes.
A draft report has just been submitted for 3 of the 4 chapters. The research covers the following topics:
VIII. An Evaluation of ARC's Civic Capacity and Leadership Projects.
A forthcoming evaluation report examines ARC's leadership and civic capacity-building projects including adult and youth leadership programs, civic leadership, development of community-capacity projects and organizations, community revitalization, education projects, and technical assistance. From a functional perspective these projects are more usefully classified in terms of the nature of the interventions they support including: conveners that provide technical assistance; community initiative projects; leadership training; strategic planning, and the provision of a variety of community "tools." Each of these functional investments is targeted to meet the leadership or capacity-building needs of groups of individuals, organizations, and/or the community at large. Since 1995 nearly $12.5 million has been invested in 168 projects, excluding continuations. ARC expects a draft final report by April of 2004.