FY 2005 Performance Budget Justification

Appendix C: Program Evaluations of ARC Investments

I. Appalachian Development Highways Economic Impact Studies by Wilbur Smith Associates, July 1998. This study evaluated both the transportation efficiency benefits and economic development impacts of the completed portions of the Appalachian Development Highway System (ADHS). The study focused on 12 of the 26 ADHS highway corridors that were largely completed, totaling 1,400 miles of the system. A primary objective of the study was to measure, in retrospect, the extent to which the completed portions of the ADHS have contributed to the economic well-being of Appalachia.

Two major effects are estimated using transportation data and a state-of-the-art, regional economic model: (1) travel efficiency benefits due to reduced travel time, lower vehicle operating costs, and the reduced number of accidents; and (2) economic development impacts on the Region due to the enhanced competitive position of industry in the Region, increased roadside business, and increased tourism. The economic model used for the study was developed by Regional Economic Models Inc., REMI, of Amherst, Massachusetts, which estimated the employment, wages, value-added and population increases attributable to the ADHS in the 165-county study region.

The following impact estimates are only for the completed portions of the system and do not include benefits from completing the system.

• Economic Development Job Creation. By 1995 a net increase of 16,000 jobs are estimated to have been created that would not have existed without the completed portions of the ADHS; it is estimated that these twelve corridors will, by the year 2015, have created a net increase of 42,000 Appalachian jobs, and will rise to 52,000 by 2025.
• ADHS Has Led to Increased Production. The net increase in value added was $1 billion in 1995, and will increase to $2.9 billion by the year 2015.
• ADHS Has Created Efficiency. The ADHS highway corridors have created travel efficiencies valued at $4.89 billion over the 1965–2025 period. Improved road conditions and access resulted in greater efficiency.
• Life Cycle Return on Federal Investment. Over the life cycle of these portions of ADHS, the federal investment will realize a solid return:
  • In terms of efficiency benefits, for each $1 invested, the return is expected to be $1.18;
  • In terms of economic development benefits, each $1 invested is expected to yield a return of $1.32.

Job Creation. The estimated total number of jobs created by the ARC-funded sample projects was 44,331. This includes direct new jobs and indirect/induced new jobs. It does not include the retained jobs. Overall, the economic development projects examined cost $2,412 per new job created, including indirect and induced job creation. If retained jobs are also counted, then the average cost drops to $1,761 per job. The benefit cost ratio of income generated by these jobs was 5.4 to 1 when counting only income resulting directly from job creation, but increased to 8.9 to 1 when indirect job creation was included.

Personal Income. The new jobs led to increased personal income for residents of the affected counties. For a one-time public investment in these economic development projects, there was approximately $9 of annual recurring personal income per public dollar invested.

Projections and Results. All project types performed well:

Non-residential water and sewer projects: The number of new jobs created was about 62 percent over projections, and the number of new businesses served was almost four times the projected total.

Industrial parks: Projections for new businesses and existing households served were exceeded; projections for existing businesses were met, while the percentage of actual jobs retained was slightly lower than projected (91 percent).

Incubator projects: Actual results for new businesses ran more than three times the projections. The number of retained businesses served was double the projection. The number of new jobs created was almost five times the projections, and the number of retained jobs was 50 percent above projections.

Industrial access road projects: The number of new businesses served was greater than four times the projections, while the percentage of retained businesses served was below projections (77 percent). However, the numbers of new and especially retained jobs assisted by the projects came in above projections.

Recommendations: Key recommendations in the study were that:

• ARC work with other federal economic development agencies to address the generic issue of how to develop methods for assessing the quality of jobs created by public sector economic development investments, and;
• Project close-out reports should re-estimate the final amount of private investment made in each project.

• Study findings indicate that most of the projects in the study reached those segments of Appalachia that are most economically disadvantaged or geographically isolated.
• Projects were successful in achieving the outcomes they set forth in their original requests for ARC support.
• Almost three-quarters of projects reported that results met or exceeded original expectations.
• Just under half met expectations;
• Nearly one-third achieved more than planned;
• Thirteen percent achieved less than planned.

Types of Performance Measured:

• Educational attainment, e.g. high school completion rates, and college-going rates.
• Economic well-being, e.g. job skills; wages.
• Family/individual well-being, e.g. family stability.
• Barriers, e.g. student behavior problems; access to educational support.
• Case Studies provided convincing evidence that the sample projects resulted in a broad range of educational, economic, and social gains. However, only a few of the typical outcomes had a quantifiable benchmark against which project success could be measured quantitatively.
• ARC funding was critical for 67 percent of the projects which indicated that they would never have been implemented without their ARC award.
• Recommendations: A key recommendation of the report was the need to clarify and expand upon the performance goals for assessing improvements through education and non-workforce training projects that target individual or families, and develop methods to gauge wider economic benefits of educational programs.

The study found that three-quarters of the projects had assisted firms to develop new products or upgrade new technologies. In addition, half of the projects reported starting new businesses which led to the creation of 304 new firms—46 new firms with employees and 258 firms that were sole proprietors. There were 377 new jobs created according to the survey, with 69 jobs in new firms, 50 in existing firms, and 258 through self-employment. Furthermore, there were 74 jobs reported to be saved by project interventions. Of client businesses or entrepreneurs served by the project, 70 percent reported being highly satisfied by the services they received, 17 percent were satisfied, 8 percent were dissatisfied, and the balance did not respond.

Thirty-six percent of the businesses reported that their business strategies were oriented to expand or grow the business and its market share, while 55 percent viewed their business as stable "lifestyle businesses," and the balance saw their businesses as a response to local economic decline.

The study recommended that the program increase the amount of technical assistance provided to grantees, support the development of more internal evaluation and self-monitoring systems within the projects and businesses served, make fewer small grants, recognize the risk inherent in the program and seek to replicate the successful projects. Finally, the report recommends that a broader regional economic impact from the program will require greater scale and more focus on growth-oriented businesses.

V. Evaluation of ARC's Vocational Education and Workforce Training Projects

In fiscal year 2002, the Appalachian Regional Commission issued its report Evaluation of the Appalachian Regional Commission's Vocational Education and Workforce Training Projects by the Westat Corporation. The study examined 92 representative projects started and completed during the 1995-2000 period. The sample constituted about one-third of the universe during period, after adjusting for continuation projects. A mail survey collected data on implementation, monitoring and impact. In addition, five case study site visits were conducted. A two-tier sample of projects was developed to assess the impact before and after full implementation of ARC's performance measurement system: Cohort 1 selected 67 projects from the 1995-1999 period; and Cohort 2 selected 25 projects funded in 2000.

Types of Performance Measured

• Obtained skills, e.g.: basic skills, academic skills, vocational skills, or employability habits;
• Individual employment gains, e.g.:
  • helped laid-off workers or underemployed obtain new work;
  • helped those without full-time job experience gain initial full-time jobs;
  • helped employed individuals increase skills, responsibilities, wages and position.

Project Outcomes

• A clear majority of Cohort 1 projects reported achieving all of their objectives (45 percent) or all but one objective (27 percent).
• Only 9 percent reported achieving less than half of their objectives (six projects).
• The vast majority of projects had quantifiable output measurements, but a higher proportion of Cohort 2 projects had clear and quantifiable outputs.

Assessment of ARC's Performance Measurement

• Results from comparison of Cohorts 1 and 2 indicate that the quality of project application and implementation has increased.
• Many of Cohort 2 projects are planning to collect some new data through mail or telephone surveys.
• The key issue is how to collect data on long-term employment impacts without placing undue burdens and high overhead costs on project grantees.

Recommendations

• Realign designations used to classify vocational education and workforce training projects.
• Disseminate information on best practices.
• Enhance quality of final reports.
• Require each project to have at least one numeric outcome and output.
• Develop application materials on each project type.
• Meet with other federal agencies to better understand their funding and reporting requirements.
• Assess common reporting practices among states
• Use project approval process to reinforce performance measurement criteria.
• Provide additional evaluation training and technical assistance to project grantees and ARC staff.

VI. External Evaluation of ARC's Telecommunications Projects

ARC issued its report a Program Evaluation of the Appalachian Regional Commission's Telecommunications Projects by the Westat Corporation. This program evaluation examined 70 projects that were started and completed between 1995 and 2001, including project investments in various types of information technology-based training, e-learning and distance learning, e-commerce, telemedicine, network and infrastructure projects, and community access center projects.

Methodology:

• Reviewed project documentation and examined the accomplishments and outcomes of 70 representative telecommunications projects;
• Measured the extent to which the projects enhanced access to telecommunications services and improved the use of these services to meet communities' needs;
• Assessed the degree to which projects involved and served multiple community stakeholders;
• Conducted a telephone survey of the 70 projects;
• Conducted 16 in-depth case studies, and;
• Provided a summary literature review of best practices in the telecommunications and community development field.

Project Types:

• Projects were most frequently designed to improve skills training and educational opportunities (87 percent), and to enhance economic development (74 percent).
• Nearly two-thirds of the 70 projects aimed to enhance community development, long-term telecommunications capabilities, and/or the coordination of community-wide information (66 percent for each).
• Forty-two of the 70 projects (60 percent) aimed to improve delivery of and access to government services, and 35 projects (50 percent) aimed to enhance employment opportunities.

Project Accomplishments and Impacts: Most projects reported fulfilling their goals to the same or greater extent as projected. For example:

• Skills training and educational applications, 69 percent indicated that their success was the same as expected, 23 percent indicated that it was more than expected, and 8 percent reported that it was less than expected.
• Economic development applications, 71 percent reported their success to be the same as expected, 14 percent indicated that it was more than expected, and 15 percent indicated that it was less than expected.

Performance Measurement: The report examined 16 projects to evaluate how the projects anticipated results were ultimately measured at the end of the project. While this sample does not reflect the overall outcome reports for all 70 projects, it did provide a more detailed picture of how ARC's performance measurement system was implemented for telecommunications projects in the late 1990s. An analysis of the 16 final reports revealed that two-thirds of the projects expected outputs and outcomes were addressed in the final reports, but 20 percent of project outputs and 55 percent of project outcomes failed to provide adequate information or data supporting their claims that these results had been met.

Recommendations:

• Provide more feedback to projects during the application review process.
• Provide prospective applicants with examples of telecommunications outputs and outcomes.
• Encourage applicants to describe how technology will affect residents and organizations in participating communities.
• Reinforce ARC's reporting structure.
• Continue to disseminate information about innovative telecommunications practices to prospective grantees.
• Provide projects with written materials on high-quality evaluation practices.

VII. Health Studies on Health Disparities in Appalachia and the Financial Conditions of the Health Care Institutions in Appalachia.

A. An Analysis of the Financial Conditions of Health Care Institution in the Appalachia and their Local Economic Impacts by the Center for Health Affairs of Project Hope was published the ARC in January, 2003.

This report describes the availability of health care services in Appalachia, the financial stability of Appalachian health care institutions, and the effect of hospital closures on Appalachian counties.

• An important overall finding of the report is that core of the Appalachian health care infrastructure has been getting stronger. There has been an expansion in the number of primary-care physicians per capita in Appalachia. Even distressed counties are attracting more primary care physicians.
  • Physician supply increased from 1990-1999
    
  • Distressed counties attracted increasing numbers of primary-care physicians
    
  • The number of skilled nursing facilities increased through 1999
    
  • Profits at Appalachian skilled nursing facilities were above national averages
  • Most county economies were resilient to the closure of hospitals
• In counties that lost a hospital, income per capita grew at rates similar to the average for Appalachia.
• Counties that lost their only hospital experienced a rate of population growth that was similar to average for rural Appalachia.
• Counties that lost their only hospital usually experienced employment growth, though the long-term rate of job growth tended to be slightly lower than Appalachian averages.

The analysis of Appalachian hospitals based on American Hospital Association (AHA) data revealed weakness in access to certain services that fall outside the core functions of primary-care physicians, rural hospitals, and skilled nursing care facilities. Most Appalachian counties have not been successful at improving access to dentistry, outpatient alcohol treatment, outpatient drug treatment, and outpatient mental health services.

• Low levels of dentists per capita, particularly in distressed counties. The supply did not improve from 1987 through 1998
• A lack of hospital-affiliated substance abuse treatment services, particularly in distressed counties
• A lack of hospital-affiliated psychiatric services, particularly in distressed counties
• Lack of obstetric care in economically distressed counties

B. Health Disparities in Appalachia

The Commission issued a contract in FY 2001 to the Office of Social Environment and Health Research at West Virginia University to analyze of the health status and disparities in Appalachia so that policy makers and regional public health practitioners can identify their health-related priorities for health surveillance and research, health education, and investments to improve the delivery of health care, and health outcomes.

A draft report has just been submitted for 3 of the 4 chapters. The research covers the following topics:

• A regional, national and county-level analysis of morbidity and mortality rates, and risk factors adjusted for age, sex and race.
• Identification of specific disparities in morbidity or mortality rates (particularly excess rates) within the region, and any apparent clusters of incidence within the region.
• Analysis of risk factors to assess any apparent excess behavioral health risks.
• Measure the accessibility to health care within the region, such as insurance coverage by households, location of health care, limitations on available transportation, or health services received.
• Analysis of the influence of socioeconomic factors on health status.
• Finally, the report analyzes the public health policy implications of the findings.

VIII. An Evaluation of ARC's Civic Capacity and Leadership Projects.

A forthcoming evaluation report examines ARC's leadership and civic capacity-building projects including adult and youth leadership programs, civic leadership, development of community-capacity projects and organizations, community revitalization, education projects, and technical assistance. From a functional perspective these projects are more usefully classified in terms of the nature of the interventions they support including: conveners that provide technical assistance; community initiative projects; leadership training; strategic planning, and the provision of a variety of community "tools." Each of these functional investments is targeted to meet the leadership or capacity-building needs of groups of individuals, organizations, and/or the community at large. Since 1995 nearly $12.5 million has been invested in 168 projects, excluding continuations. ARC expects a draft final report by April of 2004.