Workforce Developments

AEA final post: workforce safety on American railroads

The title might have been dry, but the session was fascinating (not just because I'm a big fan of Amtrak): The Federal Railroad Administration's (FRA) Research and Development Agenda to Improve Safety and Safety Culture in the Railroad Industry. Joyce Ranney of the Volpe National Transportation Systems Center and Jonathan Morrell of New Vectors reported at AEA on a series of ongoing projects in the industry.

Derailments are a daily occurance on American railroads. Between 1980-85 there was a 50% reduction in human factor accidents (HFAs) on the railroads. Since 1985 the number of HFAs has remained constant. Because most derailments and accidents happen in train yards - and 98% of our railroads carry freight instead of passengers - few injuries occur. (In Europe, by contrast, 98% of trains carry passengers.) But all HFAs cost money, and the Federal Railroad Administration is working diligently to find ways to reduce accident and improve safety.

Part of the problem is the density of traffic moving on railroads. The stats are stunning. Trains move throughout the country pulling as many as 150 cars each. One rail yard might shift 800 cars in a single shift. New tracks are being laid to meet demand on major routes, such as between Los Angeles and Denver. Production of coal in the Powder River Basin of Montana and Wyoming alone generates 200 trains with 150 cars each of coal every single day. Rail volume is only expected to increase. The trucking industry is maxed out and expecting railroads to pick up the slack as transportation of intermodal containers continues to grow.   

Volpe estimates the railroad industry will need to hire 80,000 new employees over the next five years at all levels, to cover new positions, retirements and other turnover. 100 percent of the railroad labor force is union-represented, although there are thirteen different unions, and they don't all get along. There are five class-one railroads in the U.S. (or check out this article), plus more than 300 small local lines.

Evaluators from Volpe, New Vectors and a few other orgs are conducting a series of quantitative and qualitative evaluations of several new initiatives the FRA has implemented in order to increase safety and create a "safety culture" on the job. They've developed logic models for each of three key initiatives. One initiative is designed to increase the confidential reporting of incidents that didn't lead to accidents but could have (precursers), because getting data on actual accidents is almost impossible. A behavioral-based safety initiative has been developed that trains peers to evaluate and give feedback to each other on safety and workplace behavior.

Each initiative is being implemented at a different level. Most are still at the pilot phase. All of them require the participation of the railroad carriers and unions. If you're interested in more info on railroad safety, the FRA makes a fair amount of data available here, on its website.

This concludes my posts on the American Evaluation Association conference. To read the full coverage from Workforce Developments, click here.

AEA: Cultural competency in economic self-sufficiency programs

Of the roughly 150,000 African immigrants living in Minnesota, 130,000 are Somali. The African Development Center (ADC) is a three-year old project spearheaded by Hussein Samatar to help Africans and others in the Minneapolis area overcome barriers to financial success. They recently hired Rainbow Research to evaluate their Business Development (BD) and Home Ownership (HO) programs.

Theartrice Williams and Mia Robillos presented the findings of their research at the AEA conference last week. They used an empowerment evaluation approach, designing a process that would both evaluate ADC programs and build capacity within the organization to evaluate themselves in the future. Key questions that they asked include

• How are ADC's clients benefitting from the BD and HO services?
• To what extent are clients satisfied?
• How can ADC improve its services?

Rainbow Research designed the survey and randomly selected clients to be interviewed, then trained ADC staff to carry out the interviews. One of the key issues was language, as many clients speak only their native language (primarily, but not exclusively, Somali). Having ADC staff conduct the interviews dealt with that issue, while also building staff skills.

Over the past two years, 40 clients of the BD program have started businesses, and 46 clients of the HO program have bought homes. Among homeowners, none have fallen victim to predatory lenders after the program. Only one has been delinquent with a mortgage payment, and only one has defaulted on a loan. The evaluation found that clients generally have a greater understanding of the financial system, have greater confidence in nagivating it, and have a greater ability to assess their own capacity for undertaking these major financial endeavors. They also found that through this program ADC clients widened their social networks. Finally, many ADC clients of both programs report they can now support family members in the U.S. and their home countries.

Williams also briefly presented some information on a new effort by ADC to develop sharia-compliant lending. Islamic sharia law bans interest payments. ADC has found that because of this, some Somalis and other Muslims opt out of common American financial opportunities. ADC has leveraged bank, CDC and other funds to implement their own sharia-compliant financing. Long term, however, their goal is to mainstream this model of lending so that it will be more widely available.

Faith-based workforce development

Another very interesting session, this one on Evaluating the California Community and Faith-Based Initiative. In 2002 Gov. Gray Davis set aside some of his 15% WIA discretionary funds (plus a bit of general fund money) for a Community and Faith-Based Initiative, funding workforce development projects at 40 faith-based organizations (FBOs) over a three year period. Interestingly, this project was not continued under his successor in the governor's office.

Evaluators Eric Glunt of Sonoma State and Dave Campbell of the UC Davis Cooperative Extension are finishing up work on a detailed evaluation of the project. Their work includes a mix of qualitative and quantitative methods, including standard analysis of unemployment records for the 13,730 people who were served in total, combined with interviews with stakeholders and community network analysis. They asked two key research questions: 

• Did these FBOs reach a population that isn't ordinary reached by traditional WIA-funded programs?
• How did the FBOs' outcomes compare to traditional WIA-funded programs?

Their findings aren't yet complete, so the details are still a little hush-hush. To hit a few general highlights, though. They found that FBOs can't really be treated as a monolith, and they created a typology of the organizations based on the types of services they provide and the networks they are part of. In general, they found that these organizations did reach a different, harder-to-serve population. How their outcomes compare to standard WIA programs depends on the type of FBO in question.

The final report, when it's complete, will be posted to the website for the California Communities Program, where you can find some preliminary data, plus other reports and analyses of the state workforce development system.

Evaluating adult basic ed

I'm more familiar with WIA's Title I programs, so this morning's session on a new tool for evaluating Title II Adult Basic Education (ABE) programs was fairly interesting. Apparently all those federally mandated outcomes (employment rate, credential rate, retention rate, etc.) that I'm familiar with just don't exist in the ABE universe. It creates an opening for states to develop their own unique measurements, but it also means that many ABE programs are less outcomes-oriented.

Judith Alamprese of Abt Associates has developed a process for both evaluating ABE programs and for introducing evaluative thinking among staff and administrators:

• Analyze data
• Identify program areas requiring change
• Develop a plan
• Document new practices
• Evaluate outcomes

The acronym spells AIDDE. After piloting the AIDDE process in Pennsylvania, it was rolled out on a larger scale in the Pacific Northwest, with the participation of state ABE administrators and ABE programs in Alaska, Idaho, Montana, Oregon and Washington. She found that in this process administrators and staff often want to go straight to analyzing instruction, but they usually need to evaluating their own internal processes  and practices before they can get to that.

Kristen Kulongoski of the Oregon Department of Community Colleges and Workforce Development, which oversees ABE, talked about the experience in Oregon with AIDDE. They committed to the AIDDE process for five years, involving 18 service providers (primarily community colleges) serving 35,000 individuals.

One of the major changes they've seen is that state oversight of ABE programs is now cooperative, not monitoring visits full of surprises. ABE programs are now required to review their own performance data and report to the state their five strongest and weakest areas. State monitors then come in for intensive site visits to review and validate (or otherwise) those reports. Reports are actually read, both strengths and potential areas for improvement are agreed upon by both parties. In the process, data collection systems have been improved, while staff skills and attitudes about data collection have been strengthened.

Posts on other AEA sessions to follow.

Measuring advocacy and policy work

This afternoon’s session for me was on Measuring Advocacy and Policy Work. Often, folks working for policy change are working toward a very clear win or loss on a specific piece of legislation or regulation. These two projects looked at advocacy and policy change more broadly, in an effort to identify ways to identify and measure short-term successes (or otherwise) on the way to that one big win or loss.

Carlyn Orians and Shyanika Rose of Battelle Centers for Public Health Research and Evaluation worked on the Allies Against Asthma project funded by the Robert Wood Johnson Foundation. Six local coalitions of community-based groups across the U.S. were funded to do work addressing pediatric asthma. Lots of other evaluative work was done as part of the project. Orians and Rose specifically worked on a qualitative research project designed to identify intermediate outcomes and development of community capacity.

Orians and Rose used a series of interviews with key members of the six coalitions at two points in time, asking people to reflect on the impacts they had. Often individuals thought their impact hadn’t been significant. When probed further about details, however, they discovered they’d actually achieved quite a lot. Some of those probes included legislative/government involvement in pediatric asthma; increase in community involvement; nonmembers of the coalition expressing interest in the coalition activities; dissemination of results within the community; new policy changes in clinical care; etc. This helped the community advocates document what had occurred in a way that was systematic and grounded in their own experience.

The second example came to us courtesy of the Annie E. Casey Foundation. Tom Kelly of the Foundation joined Jane Reisman, Anne Gienapp and Sarah Stachowiak of Organizational Research Services to present highlights of a not-yet-ready-for-publication report, A Practical Guide to Measuring Advocacy and Policy. Funded by Casey, ORS set out to document what others have done in trying to measure the impact of their advocacy and policy work. Like Orians and Rose, the ORT associates are interested in intermediate successes. It looks like rather than best practices, they're trying to provide a comprehensive yet summary compendium of what's being done in the field.

The pages they shared with us has DO NOT CITE printed prominently on every page, so I won't go into details. They've identified types of outcomes and indicators different organizations have identified, methodologies for measuring them, and tools that can be used for different outcomes and methods. This is definitely something to look for, forthcoming on the Casey website in the not-too-distant future.

-

-

                  And with this post, Workforce Developments turns 100!

Building evaluation capacity in nonprofits

Three sets of speakers talked about the work they've done to improve the capacity of nonprofits to evaluate their own work.

Evaluator Abe Wandersman of the U of South Carolina and Jan Yost of the Health Foundation of Central Massachusetts have created a results-oriented grantmaking (ROG) model where evaluators are brought in as a partner in developing the program even before the foundation makes the grants. This way, evaluators can help nonprofits both develop their own evaluative mindset and also begin collecting data from the very start. Yost noted that nonprofit staff tend to be deeply devoted to their clients and passionate about what they're trying to do, but they often take offense when a funder or evaluator starts asking questions like, Are you really doing what you set out to do? Are you doing it in the best way possible? Is this really what the community needs?

Independent consultant Michael Hendricks described the mentoring and coaching model he uses with small nonprofits. He works with a foundation in the northeast that requires anyone who's serious about applying for their funds to attend a letter of intent workshop to understand, among other things, the evaluative activities they'll have to undertake if they receive a grant. If they're invited to apply, they'll have to go through another grant application workshop, and they must submit a logic model as part of their application.

Hendricks helps each grantee develop a set of outcomes and indicators they will be required to report on bi-monthly. What's unique about this model is that the foundation is more interested in getting those reports than in getting reports meeting the funder's need. The funder provides a format for reporting, but the nonprofits are reporting on the outcomes and indicators they've set for themselves.

Ann Bessell of the U of Miami talked about how she uses spider diagrams. She has worked with a wide range of nonprofits, from those who don't do any evaluation or data tracking, to those that generate 60-page reports full of bar charts on their indicators, but who are losing sight of the forest for the trees.

Bessell gave a great example of a metaphor she uses to get nonprofits interested in and understanding evaluation. She asks them to think about the process of car buying. As a buyer, you develope all sorts of factors you're looking for in a car - appearance, gas mileage, purpose, reliability, cost, etc. To collect that data, you visit car lots, do online research, read Consumer Reports, take test drives, and so on. In other words, you evaluate each car accordingly to determine which one to buy.

Lots more was said, but I'll end on one important note from Jan Yost: Most foundations don't fund enough evaluation activities. How can we ask nonprofits to take on this additional work without adequately funding it?

Posters: enviros and bioterrorism

News flash: environmentalists might actually create jobs. Had a great conversation with Lisa Skumatz of SERA consulting firm. She's been calculating the "non-energy benefits" (NEBs) of energy-related programs. She covers a lot of benefits; I was specifically interested in the impact on jobs. For example, investments in retrofitting homes to be less environmentally wasteful creates construction jobs. Programs to replace old household appliances with more efficient ones creates jobs selling and delivering those appliances, although it creates fewer jobs than retrofitting because many of those appliances are manufactured overseas. Skumatz also turned me on to the National Recycling Coalition, which estimates that the industry generates 1.4 million jobs in the U.S. that pay better than average national wages.

Perhaps a new sector for workforce development professionals to explore?

Quality of training for bioterrorism pros The Arizona Center for Public Health Preparedness also presented their research on "Evaluating the Competence in Bioterrorism Professionals." Since 9-11 many programs have emerged to train professionals for potential bioterrorism events, but there is no consensus on how to evaluate their effectiveness. The AZ Center is working to develop some basic guiding principles to evaluate these courses. Their study is finding that the intensity and duration of training are two key components.

Workforce training of two very different kinds

I'm scouring the AEA schedule for workforce-related papers and discussions. The two fields that seem to dominate here are education and health care. There's a smattering of various types of social services. The session I attended this afternoon combined two completely different workforce training issues in two very different sectors: pharmaceutical manufacturing and health care.

First was a talk by Willis Thomas of Pfizer Global Manufacturing and doctoral student at Western Michigan U. He works with Pfizer employees to get them the training they need to meet their own career goals and to keep the company in compliance with FDA, EPA, OSHA, etc. regulations. Pharmaceutical manufacturing is heavily regulated, so they have complex sets of rules and standard operating procedures (SOP) that change regularly, and it's a big workforce of 3,500, so they're constantly training their workers. SOP documents have been translated into video or computer-based training that allow workers to physically see what they are being asked to do in the manufacturing environment.

Pharmaceutical manufacturing pays fairly well. A person with a high school degree can get a job on a production line and, with overtime, eventually earn $60-70,000 a year. However, they can't advance beyond that. When Pfizer made layoffs recently, those without a college degree were the first to be let go. During another round of cuts, the company made a decision that anyone without a college degree cannot be promoted into management. Thomas has been working with the Instrumentation Society of America to help workers with less education get credentialed, and then to link those credentialed workers to to college courses at Western Michigan.

The second session, led by Niels Agger-Gupta of Fielding Graduate U and Saumitra SenGupta of the Behavioral Health Research Center of the Southwest, was about a project to provide cultural competency training to health workers in California. They both worked on a California Endowment-funded project to train workers in L.A., Contra Costa and San Francisco Counties. Each county designed its own training. In L.A., they trained workers at all levels from the management on down. In San Fran, they trained health care organizations on contract to the city to develop cultural competency plans.

Evaluation of this project hasn't been completed, so there's no final data for analyzing their experience. We did have a lively discussion about the meaning of cultural competency and how it is practiced by health care professionals. Does having good cultural competency mean knowing how to deal with an individual patient, or does it require changing systems so that they work better for all patients?

A couple of people in the room raised the issue of resistence or even defensiveness among health workers to any kind of cultural competency training. I wondered if there might be room for a "vocational cultural competency" training modeled on vocational English. Something very focused on the specific skills and "vocabulary" needed to help you serve patients (or customers) from other cultures more effectively. Granted, it would miss a lot of important nuances, but perhaps it would help overcome some of the knee-jerk reactions and fears that some people have of what "cultural competency" means to them.

AEA plenary session: social justice, cultural competency, and public dialogue

Just a few highlights from the plenary session of the American Evaluation Association annual conference:

AEA President Mel Mark from Penn State said this year's AEA conference is "probably the largest gathering of evaluators in the history of the known world."

Jennifer Greene from the University of Illinois at Urbana-Champaign gave a fascinating overview of "A Social Justice Mandate for Evaluation." What does evaluation have to do with democracy? Everything, she says. Evaluation is an intervention in the lives of people. The work we do and the choices we make as evaluators are inherently value-laden, and serves some interest. As a result, we should be conscious and intentional in the values we use and the interests we serve. Therefore, we should choose democratic values in our work, and serving the interests of those who are disadvantaged in society. In other words, a social justice orientation.

Stafford Hood from Arizona State U began by asking, "What are the consequences that I hope will result from my efforts as an evaluator?" A more racially diverse universe of evaluators, and a next generation of evaluators who are more culturally responsive and are better trained to work with underrepresented groups. Too few people of color are entering graduate programs for training in evaluation, and too few of the professors who teach them are people of color. He was encouraged by the addition of a statement on cultural competency in AEA's guiding principles, but encouraged us to do more.

Robin Miller from Michigan State U discussed scholarly publishing in the evaluation field, raising concerns about who reads it, for what purpose, and whether the divide between those who write and publish scholarly work and practitioners in the field is too wide. She pointed out that even among academic scholars, many read from a fairly narrow field of people and subjects they are already familiar with, describing this as being like "subsistence foraging." Scholarly publication is a public dialogue - are enough people engaged in the process? She asked whether there is space in the academic evaluation journals for practioner-oriented work. If not, perhaps those spaces should be opened elsewhere.

TriMet is hiring

Took the Max red line from the airport to downtown Portland today on my way to the American Evaluation Association conference. On the way I picked up a flyer about jobs at TriMet, the transit district for the three county region, that says "Start at a great hourly wage, with double the pay in three years."

Double the pay in three years? Sounds pretty good.

Right now, TriMet has openings for Mini-Run Operators (a.k.a. bus drivers), engineers and management types. Driver pay starts at $12.18 per hour, and tops out at $22.14. TriMet drivers are representated by Amalgamated Transit Union Local 757. Bennies don't look bad - medical, pension, credit union, free transit passes for the whole family, on-the-job training, and a 24-hour fitness center open to employees and spouses.

Blogging the Evaluation conference

Just a quick note to let folks know Workforce Developments will be attending the annual conference of the American Evaluation Association this week. Not just because it's being held in beautiful Portland, OR, home of the best bookstore in the world, Powells City of Books, and not just because PDX airport provides free wireless. I'll be attending as many sessions on workforce development, job training and labor issues as I can find, and will blog about it right here.