Communities of Opportunity lacks a single central agreed-upon set of goals. A review of the initiative's 2008 Business Plan revealed three separate, overlapping but inconsistent sets of goals. This lack of consistency undercuts Communities of Opportunity's ability to meaningfully understand its progress, achievements, or shortcomings. Furthermore, the multiple goal sets contribute to the difficulty many have in understanding what Communities of Opportunity is and what it does. In order to be a coherent, focused, data-driven initiative, Communities of Opportunity needs to establish one set of goals and align its efforts­­ ­– including establishing quantitative objectives, setting deadlines, and recording data ­­– to those goals   The Communities of Opportunity End Goal Matrix is a useful, well-designed performance measurement tool, but it requires refinement. Communities of Opportunity uses the Matrix to report on its progress toward 10 goals, aligning achievement "strategies" to each goal, quantitative and qualitative "targets" or objectives to each strategy, and deadlines for those targets. Communities of Opportunity updates the Matrix in its semiannual financial reports. This Matrix puts Communities of Opportunity's efforts into an understandable perspective and should therefore be emphasized, particularly on www.coosf.org and in reports to elected officials and the public. To be more meaningful, the Matrix should relate to the Dashboard, if the Dashboard continues to be a major reporting tool; it should list specific projects alongside its strategies; and it should refine some targets to better align them to the strategies or make them to be more meaningful. Furthermore, the Matrix needs to be aligned to the single set of goals (see bullet above).   Quantitative targets and objectives often seem arbitrary, and generally lack a relationship to similar efforts. A number of Communities of Opportunity's quantitative targets lack a real-world justification. The definitions and quantitative targets for "crisis," "fragile," and "stable" families were developed in a retreat and are not standard measures. Furthermore, the targets are not benchmarked against other cities' efforts. Although Communities of Opportunity may be a unique effort, many of Communities of Opportunity's components and strategies are borrowed from established programs elsewhere. Therefore, Communities of Opportunity and City Departments can and should measure the City's progress against that of other cities.

Managing for Results

This section of the audit examines Communities of Opportunity's performance management and evaluation efforts. Communities of Opportunity committed to performance management from its onset, as noted in the Communities of Opportunity Pilot Phase Business Plan, 2006-2010 (2006 Plan). In the Executive Summary of the 2006 Plan, two of Communities of Opportunity's five "bedrock principles" highlight tenets of performance management:

1. Focus on outcomes for people and the place, not delivering particular services and programs

and

5. Manage change dynamically by quantifiable outcomes; expand successful approaches, stop failed ones, and introduce new evidence-based approaches

In the 2006 Plan's Executive Summary, Communities of Opportunity commits to producing "a dashboard that reports progress against key metrics, thereby ensuring accountability and enabling course corrections."

The Communities of Opportunity 2.0 Business Plan Update, dated May 28, 2008 (2008 Plan) continued the 2006 Plan's commitment to performance management. The 2008 Plan's "Theory of Change" highlighted the importance of services having "clear accountability for results" and for City government to "track outcomes across systems." The 2008 Plan specifies that as part of its effort to develop strong communities, resident teams will "hold everyone – the city, the nonprofit providers and the residents themselves – accountable for measuring and achieving real results." The 2008 Plan's commitment to performance management is most evident in the sections on "Accountability" and "Measuring Success." The Accountability section reads:

(I)t is critical that the city is accountable for results. In the past, different programs tracked the number of people they served but few tracked the long-term outcomes for their clients. As a result it was hard to tell which programs were providing a short-term fix versus those that were providing long term solutions. By creating a division to track community investment and outcomes the city will be able to measure the long-term effects of its programs. To be a part of COO (the Communities of Opportunity), programs will need to commit to track and reach long-term outcomes for the communities they serve. Each department and CBO involved will be responsible for following the progress of their clients and working to improve services where goals are not being met.

These themes are continued in the Measuring Success section:

Accountability is a central tenant of COO. Accountability to our residents, to our funders, to our partners and to the City as a whole. While we recognize that this work is often complex, slow, and hard to track, we believe it is critical to use data and evaluation to follow the progress of our work and ultimately that of the neighborhood so that we can continue to refine our approach and document our trials and successes for others. (emphasis added)

The 2008 Plan highlights the need to track program engagement in the short term while evaluating progress toward long term goals.

(W)e can use these metrics to test what hypotheses seemed to have worked and where our efforts didn't produce the outcomes we hoped. & Unless we track these activities it will be difficult to tell if they are having the effect we hope on the overall picture.

The 2008 Plan proposes the use of outside assistance to monitor Communities of Opportunity's progress. Specifically, the 2008 Plan indicates that Communities of Opportunity will work with the Gardner Center at Stanford University to conduct academic evaluations of Communities of Opportunity's outcomes.

Methodology and Other Evaluations

Performance management is a results-driven management approach that aligns strategies to short-term objectives to long-term goals and evaluates performance in terms of progress against measurable outcomes. As demonstrated above, Communities of Opportunity aspires to be a data-driven effort, employing practices that have been and must prove to be effective.

One common performance management tool is known as the creation of "SMART goals," where the acronym SMART stands for: Specific, Measurable, Achievable, Relevant, and Time Based. The Manager's Guide to Rewards (Doug Jensen, et. al, 2007, p. 169) defines SMART goals as follows:

• Specific: Be sure to be clear on what you are trying to achieve
   
• Measurable: Ensure that you have good data on how you're doing
   
• Achievable: Don't try to achieve too much; ensure you have resources to achieve your goal
   
• Relevant: Align individual goals with broader organizational goals.
   
• Time Based: Set deadlines for when you intend to achieve the goals.

For the purpose of this management audit, the Budget Analyst focused on four of the above areas in examining Communities of Opportunity's plans and practices. We looked at whether their goals are Specific, Measurable, Relevant, and Time Based. With regard to Specific, we examined whether their goals were well defined, both individually and as a unit. With regard to Measurable, we examined what data they are collecting and reporting, and what their plans are for collecting and sharing data going forward. With regard to Relevant, we looked at the extent to which the data provides insight into objectives, and the extent to which short-time goals relate to primary goals. And with regard to Time Based, we looked for the matching of specific deadlines to short term and long term hard quantifiable objectives. An assessment of whether Communities of Opportunity's goals are Achievable is beyond the scope of this audit.

The Foundations' Preliminary Evaluation of Communities of Opportunity

The foundations funding Communities of Opportunity commissioned a preliminary evaluation of the initiative by the Aspen Institute, with a draft report issued in July 2007. The Aspen Institute's Roundtable on Community Change performed the first phase of an evaluation of Communities of Opportunity, reviewing Communities of Opportunity's written materials, and interviewing Mayor Gavin Newsom, the Communities of Opportunity Director and Deputy Director, steering committee members, and other key stakeholders. The following are a selection of performance management related observations made by the Institute in its July 23, 2007 draft report:

• COO is both a people-based and place-based strategy & (and) thus requires two levels of data collection and outcome tracking and some strategy for linking the two.
   
• The vision for COO was conceived in broad terms & . Translating broad vision into operations on the ground involves a very ambitious multi-year agenda. But shorter-term implementation goals and activities need to be articulated broadly in order to identify and track useful benchmarks of progress, as well as to focus the efforts of multiple and diverse stakeholders. This very labor-intensive work is particularly challenging given the scale and reach of COO's ambitions and the range of activities and players/organizations involved.
   
• COO was also conceived in quite specific terms & . Despite some expectations to the contrary, COO was not launched with a field-tested information system its managers could use to generate reliable individual and family outcome data about its target group. Instead COO has had to build such a system at the same time that it works to implement the strategies intended to produce these outcomes. Currently underway, the effort to get this system–or a variant–in place is a prerequisite to developing and implementing the coordinated case management approach that is one of COO's central elements.

The Aspen Institute's survey respondents identified performance management-related priorities and concerns:

• A high priority was placed on evaluation as a vehicle for both continuous improvement and for accountability. Although these two goals for evaluation were cited most frequently, a second-level goal included building the capacity of the NBOs (Neighborhood Benefit Organizations) to self-evaluate as part of a larger organizational capacity-building strategy.
   
• In contrast, little support appeared to exist for traditional impact research that would aim to draw clear cause-and-effect connections between COO and individual and community outcomes. Respondents want the evaluation to track performance–who gets what services, how they feel about those services, and what outcomes result–but they are generally skeptical about & and willing to forgo an impact design & .
   
• Respondents want evaluation integrated into the work rather than treated as a stand-alone activity, viewing it as an ongoing, collective responsibility of COO and its many partners. But they also caution against creating overly burdensome demands for data collection, especially for NBOs with limited experience and capacity in this area. In a similar vein, department heads flagged the need to build on rather than duplicate evaluations they were planning or already had underway.
   
• Many respondents talked about an evaluator as an independent but active player in COO rather than as an outside auditor. [& ]
   
• Several respondents highlighted the need for the evaluator to establish multiple vehicles for feeding data and lessons back to different stakeholder groups. Different audiences need different kinds of data presented in different ways, but everyone needs to be part of the feedback loop.

The Aspen Institute developed recommended tasks, with an initial recommended completion deadline of 15-18 months. Some of the recommended tasks are reflected in Communities of Opportunity's 2008 Plan. Two in particular are related to performance management:

• Assist COO managers (as they) continue to implement a strategy for data sharing and tracking regarding service receipt and individual/family/neighborhood outcomes. Provide an independent review and analysis of the data as they become available. Develop one or more periodic reporting mechanisms that convey these data, even if imperfect or incomplete, to stakeholders in a meaningful way.
   
• Inventory existing data collection and evaluation activities that are either underway or would be relatively easy to get in place sooner than later. Coordinate and improve these activities as feasible and appropriate. Assess NBO capacity to collect data and evaluate their own programs and, if necessary, develop a plan for enhancing this capacity. In addition to the services-level data, what neighborhood level indicators are already being collected (how, by whom, with what frequency and quality), and is any additional investment in data infrastructure needed?

Other related recommended tasks include:

• Conduct a Random Household Survey in the neighborhood with particular attention to resident knowledge and attitudes about and level of engagement with services, perceptions of safety, feelings of empowerment and belief that their lives are moving in the right direction, and other responses that COO would hope to change over time. It's important to track perceptions because part of COO's ambition is to build hope and confidence that lead to new behaviors and a new willingness to invest, both within and outside of the neighborhood. The goal would be to establish a baseline against which change could be assessed.
   
• Manage an Evaluation and Learning Fund. This would be a pot of funds to be used flexibly as needs and opportunities arise to address key questions. Examples include: comparative mini-studies that look across the 4 neighborhoods on some dimension; an ethnographic case study on changing neighborhood culture; a systems change expert's analysis of some particularly interesting or successful change spurred by COO; an analysis of COO's strategy to establish productive connections with organizing groups working in the target neighborhoods; a study of people who move out of the neighborhoods, etc. The evaluator or evaluation team might carry out the research or they might hire a consultant with particular expertise related to the topic.
   
• Develop an Evaluation Design and Learning Plan for Phase II of the evaluation. By the end of Phase I, the evaluator will be well grounded in COO's operation, cognizant of the metrics most appropriate for tracking COO's progress, and able to assess the feasibility and value of various design elements for Phase II. This plan should be informed by the evaluation and learning priorities of different stakeholders in light of the realities of COO implementation on the ground. It should also include explicit structures and supports for maximum learning.

According to the Director of Communities of Opportunity, the Aspen report is still in draft largely awaiting the foundations' response to the changes in approach reflected in the 2008 Plan. The Director of Communities of Opportunity anticipates resuming the evaluation process with the second round of foundation funding beginning in November, 2008.

Communities of Opportunity's Various Sets of Goals

By its nature, Communities of Opportunity is difficult to define–it is not a program, a department, a nonprofit, or an initiative. Complicating Communities of Opportunity's effort to define itself is the fact that Communities of Opportunity's goals are inconsistently defined, making it difficult to know what Communities of Opportunity aims to achieve.

Communities of Opportunity's vision is overt and consistent, "to create safe and healthy neighborhoods that provide opportunities for individuals and families to achieve self-sufficiency and, for children to realize their dreams." In order to realize this vision, the 2008 Plan has three separate sets of goals. The Budget Analyst refers to these goal sets as "2011 Goals," "Smart Government Goals," and "End Goals." These goals are both redundant and inconsistent, and lack an expressed relationship to one another. Communities of Opportunity's Director and Deputy Director did not identify any of the three sets as primary.

2011 Goals

The first set of goals listed in the 2008 Plan (p. 4) is billed as Communities of Opportunity's five-year goals for 2011 (known herein as the "2011 Goals").

• A majority of families will be stable or self-sufficient (able to make ends meet)
• A majority of children will be flourishing (able to pursue their goals)
• Less than 10% of families and children will be in crisis
• Communities will provide a safe environment, sound physical infrastructure, connected social networks, and sustainable economic vitality

As noted in the 2008 Plan, Communities of Opportunity intends to achieve these goals by fall 2011. These 2011 Goals are aligned with Communities of Opportunity's vision. Two of the goals are supported by Communities of Opportunity's "Dashboard" (see below). Although it is tempting to see these as the overarching goals for Communities of Opportunity, as they are featured in the "Vision and goals" section of the 2008 Plan, the remainder of the 2008 Plan, conversation with Communities of Opportunity's Director and Deputy Director, and Communities of Opportunity's progress reporting do not suggest any systematic alignment to the 2011 Goals.

Smart Government Goals

Delving further into the 2008 Plan, we see different goals that don't quite line up with the 2011 Goals. Under the banner of "Smart Government" (itself part of Communities of Opportunity's three-point "Theory of Change" focus on "Smart Government," "Strong Communities," and "Serious Collaboration"), we get three new goals (p. 7, herein known as the "Smart Government Goals"), each tied to a policy area:

• Housing: Healthy, high quality homes for all San Franciscans
• Employment: Living-wage jobs with opportunities for career advancement
• Youth/Education: All students graduate high school and have the ability to go to college

Each of the Smart Government Goals has City and community-based organization programs that correspond to the goal. These goals support the Vision. The Smart Government Goals overlap the 2011 Goals somewhat; however they invite different measures for evaluation.

As indicated above, Smart Government is one of three components of Communities of Opportunity's Theory of Change. The other two, "Strong Communities," and "Serious Collaboration" do not have expressed goals like Smart Government; rather, the 2008 Plan lists series of strategies aligned to those efforts.

End Goals

In explaining how Communities of Opportunity will measure success, the 2008 Plan also lists three "change elements": policy, systems, and community. Despite a similarity, these categorizations only somewhat match up with the Smart Government, Serious Collaboration, and Strong Communities components of the Theory of Change. Ten "End Goals" (p. 22) correspond to these change elements, each with between one and four corresponding strategies and 12-18 month targets:

Policy Change Elements

• Eliminate barriers to getting on and climbing the job ladder
• Help people living in the Bayview stay in the Bayview (HopeSF and Redevelopment Plan)
• Ensure kids go to school, stay in school, and have meaningful afterschool activities

Systems Change Elements

• Provide coordinated case management
• Share relevant data across city agencies
• Create multi services centers that address different populations' needs

Community Change Elements

• Housing Stability
• Family Stability
• Financial Stability
• Community Stability

Once again, these goals support the Vision. However, they are clearly expressed and organized differently than the 2011 and Smart Government Goal sets, and therefore again invite different measures and standards.

No Agreed-Upon Goals

In total, the 2008 Report presents three sets of similar but apparently unrelated goals, 2011 Goals (4 goals), Smart Government Goals (3 goals), and End Goals (10 goals). The 2008 Plan does not explicitly state the alignment or relationship of these goal sets. Therefore, the reader of the 2008 Plan is justifiably confused as to Communities of Opportunity's central aims.

When the Budget Analyst asked the Communities of Opportunity Director and Deputy Director as to the main goals of the initiative, the Director and Deputy Director provided a list of first- and second-tier goals that were, again, similar but inconsistent with any of the three sets in the 2008 Plan. The Budget Analyst therefore concludes that Communities of Opportunity does not have a central, agreed-upon set of goals.

***

Goals and objectives can be subject to change, but Communities of Opportunity's difficulty in expressing what Communities of Opportunity is and what it does suggests an underlying tension as to what it aspires to achieve. Without a clear understanding of Communities of Opportunity's goals, knowing how Communities of Opportunity is progressing against its goals is impossible. Communities of Opportunity's Theory of Change, with its "clear accountability for results," is in tension with its business plan's lack of consistently expressed goals.

Table 4.1

Summary of the Communities of Opportunity Goal Sets

Goal Set	2011 Goals (4)	Smart Government Goals (3)	End Goals (10)
Goals	" A majority of families will be stable or self-sufficient (able to make ends meet) " A majority of children will be flourishing (able to pursue their goals) " Less than 10% of families and children will be in crisis " Communities will provide a safe environment, sound physical infrastructure, connected social networks, and sustainable economic vitality	" Housing: Healthy, high quality homes for all San Franciscans " Employment: Living-wage jobs with opportunities for career advancement " Youth/ Education: All students graduate high school and have the ability to go to college	" Eliminate barriers to getting on and climbing the job ladder " Help people living in the Bayview stay in the Bayview (HopeSF and Redevelopment Plan) " Ensure kids go to school, stay in school, and have meaningful afterschool activities " Provide coordinated case management " Share relevant data across city agencies " Create multi services centers that address different populations' needs " Housing Stability " Family Stability " Financial Stability " Community Stability

Data Collection, Quantified Objectives, and Reporting

Communities of Opportunity has two primary reporting tools for its overall programs, the Dashboard and the End Goal Matrix. Both are published in Communities of Opportunity's semiannual report to their funders. Communities of Opportunity also publishes the Dashboard on its website.

The Dashboard

Communities of Opportunity's 2008 Plan describes the Dashboard as "a simple snapshot of where our families are and where they are headed." Communities of Opportunity's Dashboard is a measure of the 1,203 families living in the public housing developments located in the Communities of Opportunity nodes and registered as Communities of Opportunity participants. The Dashboard reports aggregated data on those families, counting the percentage of families that are "in crisis," "economically fragile," and "stable or self-sufficient." These categorizations are defined as:

1. In Crisis: Families are earning less than 50% of the Federal Poverty Level and/or are in multiple systems.

2. Fragile: Families are on CALWorks or earn between 50% and 185% of the Federal Poverty Level.

3. Stable: Families earn greater than 185% of the Federal Poverty Level and are not in systems of care.

The designations "in crisis," "fragile," and "stable" are not standard definitions, but were established in a two-day retreat involving the Mayor, the Communities of Opportunity Directors, and the original Steering Committee members. The definitions were a consensus of what the participants believed the tipping points for these communities. Communities of Opportunity illustrates the Dashboard as a series of buckets, shown in the figure below. The dark portion of the illustration and the corresponding percentages represent Communities of Opportunity's 2011 aspirations.

Figure 4.1
Communities of Opportunity's Visual Representation of the Dashboard

Source: 2008 Plan

Communities of Opportunity acquires the data for the Dashboard from the Housing Authority, based on the 1,203 public housing addresses in Communities of Opportunity's four nodes. Each family self-reports its information on an annual basis, with 25 percent of the families reporting each quarter. Therefore, the Housing Authority updates one quarter of the information every three months. Although the information reported in the Dashboard reflects aggregate data, Communities of Opportunity is working to develop a method of following individual families' progress along the crisis-fragile-stable continuum.

The Dashboard is most closely associated with the 2011 Goals, with alignment to two of the four 2011 Goals:

• A majority of families will be stable or self-sufficient (able to make ends meet)
• Less than 10% of families and children will be in crisis

Communities of Opportunity's annual goal is to move 100 families from crisis to fragile and 110 families from fragile to stable, with an ultimate goal of reducing the number of its families categorized as in crisis to 10 percent, reducing the number of families categorized as fragile to 40 percent, and increasing the number of families categorized as stable to 50 percent. These ultimate goals were established in the aforementioned retreat, based on what the participants agreed was a realistic measure of success. The first Dashboard containing data was included in the 2008 Plan, reporting May 2008 data.

End Goal Strategies and Targets

In addition to the Dashboard, Communities of Opportunity has established 21 strategies and 23 qualitative and quantitative targets that, according to the 2008 Plan, attempt to track engagement in Communities of Opportunity's programs, engagement strategies, and systems change elements. These strategies and targets are aligned to Communities of Opportunity's 10 End Goals and the targets have a deadline of January 1, 2010. The strategies and targets are listed in the End Goal Matrix below.

Table 4.2
The Communities of Opportunity End Goal Matrix

End Goal	Strategy	12-18 Month Target
Eliminate barriers to getting on and climbing the job ladder	· Individuals have the skills they need to gain entry-level positions · The incentive system is aligned to help them move up the ladder · Youth are placed in quality summer and ongoing jobs · Workers obtain living wage jobs	· 300 people remove barriers · New incentive system is designed & launched · 200 people enroll in system · 300 youth placed in jobs · 50 people are "matched" with priority, living-wage jobs for redevelopment and begin the appropriate job training activities
Help people living in the Bayview stay in the Bayview (HopeSF and Redevelopment Plan)	· Families living in public housing are in good standing · Families living in and around our nodes accumulate assets to participate in homeownership	· 200 families who currently owe back rent get resources and payment plans to get into good standing · 85 families are enrolled in asset building programs to save for homeownership
Ensure kids go to school, stay in school, and have meaningful afterschool activities	· Kids attend safe, enriching afterschool programs · Students re-engage with school and obtain GEDs	· 600 kids from our nodes are enrolled in afterschool programming · Gateway to College partnership is established and program launched
Provide coordinated case management	· Develop unified case plans for families in multiple systems · Streamline points of contact · Incorporate family and community supports	· Train 40 case workers and community members in new model · Enroll 130 families in new program

End Goal	Strategy	12-18 Month Target
Share relevant data across city agencies	· Data required for coordinated case management is available · Outcomes for a family can be tracked across city agencies & CBOs · Services standards are improved and enforces	· Informed consent documents are complete · Data system for coordinated case management in place · Task force created to work on broader data integration
Create multi services centers that address different populations' needs	· Community hubs offering a range of services on site and connecting residents to broader services elsewhere are available for different cross-sections of the community	· Cross-agency team created to evaluate roles and effectiveness of current multi service centers · Proposal for re-composition, budget alignment, and service map created
Housing Stability	· Residents understand the redevelopment and HopeSF processes · Families have a medical home and access to health care	· Increased attendance at redevelopment meetings · 750 families enroll in Healthy SF
Family Stability	· Kids attend school regularly · Families have a medical home and access to health care	· 100 chronic truants re-engage in school · 750 families enroll in Healthy SF
Financial Stability	· Household create and understand their budgets	· 300 (75%) of employed families take advantage of the Working Families Credit · 100 (25%) get new bank accounts with Bank on SF
Community Stability	· Social networks are rebuilt in our communities · Increased civic participation	· 1 community event planned by residents each quarter · 400 families in covenant clubs

 

Like the Dashboard, Communities of Opportunity intends to report the status of these targets semiannually in its funder reports. The first status report on the targets can be found on pages 13-14 of the July 1, 2008 Fiscal-Year End Report (Appendix).

 

Although the structure of the End Goal Matrix is sound, the Matrix would benefit from refinement. Some targets are not well-aligned to the corresponding strategies, and some strategies are not well-aligned to their corresponding end goals. In other words, even if all the targets are realized, the strategies may not be achieved, and even if all the strategies are achieved, they may not fulfill their respective end goals. For example:

· Goal: Eliminate barriers to getting on and climbing the job ladder
Problems: Much of the emphasis of the strategies and the targets is on job placement, not eliminating barriers. "Workers obtain living wage jobs" is less a strategy for eliminating barriers than it is the benefit of eliminating barriers.

· Goal: Ensure kids go to school, stay in school, and have meaningful afterschool activities
Problem: The 12-18 Month Targets do not measure nor establish targets for attendance or graduation rates.

· Goal: Housing Stability
Problems: The target is not quantified and lacks a baseline. Also, attendance (target) is not a measure of resident understanding (strategy).

· Goal: Family Stability
Problems: Numerous City staff we spoke with cited the desire to keep children in families and out of foster care, or to reunite families sooner; this strategy is not reflected in the Matrix. Furthermore, the truancy strategy and target support the "Ensure kids go to school, stay in school & " end goal above.

· Goal: Financial Stability
Problem: While the targets (taking advantage of tax credits and opening bank accounts) may contribute to a family's achieving financial stability, neither target reflects the designated strategy (households create and understand their budgets).

The problems that the Budget Analyst observes above should not detract from the value End Goal Matrix. These perceived problems are simply meant to serve as examples of areas where the Matrix would benefit from refinement.

The Goal and Outcome Measuring Gap

While the Dashboard and End Goal Matrix report progress toward some goals, they fail to capture others. Therefore, with regard to some expressed goals, Communities of Opportunity does not currently appear to have a method of tracking its progress, nor do officials, funders, or the public seem to have a way of evaluating Communities of Opportunity's achievements or shortcomings. Furthermore, numerous quantitative objectives lack real-world benchmarks against which Communities of Opportunity can objectively evaluate its progress.

2011 Goals

Two of the 2011 Goals appear to be underrepresented by Communities of Opportunity's existing reporting mechanisms:

• A majority of children will be flourishing (able to pursue their goals)
• Communities will provide a safe environment, sound physical infrastructure, connected social networks, and sustainable economic vitality

Part of the problem with the "A majority of children will be flourishing (able to pursue their goals)" is the vagueness or subjectivity of "flourishing," a vagueness or subjectivity that is not cleared up by the almost equally vague and subjective "able to pursue their goals." That Communities of Opportunity strives for a majority suggests that this is a countable population of children, but nowhere in Communities of Opportunity's materials is "flourishing" defined, nor has Communities of Opportunity said how it will determine that 50 percent plus one of the children living in their nodes are flourishing.

The End Goal Matrix does address some components of the 2011 Goal, "Communities will provide a safe environment, sound physical infrastructure, connected social networks, and sustainable economic vitality." The most glaring gap is the provision of a safe environment. Although "safe environment" can be interpreted and measured in a number of ways, there does not appear to be any such measurement occurring in the Matrix. Communities of Opportunity does receive crime statistics for its nodes from the San Francisco Police Department, but Communities of Opportunity is not currently reporting these statistics in any meaningful or regular way. Furthermore, Communities of Opportunity has not established any strategies or targets for the direct provision of a safe environment.

Similarly, the provision of sound physical infrastructure and sustainable economic vitality are lacking from reported measures.

If Communities of Opportunity aims to achieve the "genuine transformation" of their nodes by Fall 2011, as they express in the 2008 Plan, Communities of Opportunity needs to define what it means by "flourishing," "safe environment," "sound physical infrastructure," and "sustainable economic vitality." It then needs to align efforts to work toward achieving those goals, and develop quantifiable measures, objectives, and timelines that relate to those ideas.

Smart Government Goals

Progress toward achieving Communities of Opportunity's Smart Government Goals is unevenly reported. Based on its current data collection and reporting efforts, Communities of Opportunity does not appear to be collecting or reporting data, nor setting quantifiable objectives, for the Smart Government Goals:

• Housing: Healthy, high quality homes for all San Franciscans
• Employment: Living-wage jobs with opportunities for career advancement
• Youth/Education: All students graduate high school and have the ability to go to college

The information reported in the Dashboard does not reflect progress toward these goals. The strategies and targets in the End Goal Matrix address related issues, but do not evaluate the number of healthy, high quality homes or identify strategies for realizing such homes; they focus more on removing barriers to employment more than opportunities for career advancement (although the job ladder may address this goal somewhat); and they do not measure or set goals for graduation rates or college preparedness (though they do hope to establish Gateway to College and are working on strategies for lowering truancy).

End Goals

The End Goals are the one set of goals where Communities of Opportunity has done the most work toward setting goals, establishing deadlines, collecting and reporting data, and otherwise making it possible for evaluating its performance. Although the Budget Analyst has addressed some shortcomings above, it strongly recommends building on the strength of the End Goal Matrix as a central performance management tool for Communities of Opportunity (see Recommendations).

Benchmarking Against Similar Efforts

In addition to tracking changes over time, evaluating achievement against similar efforts is an important performance management tool. To date, Communities of Opportunity has not evaluated its performance against other cities. Although Communities of Opportunity is unique as a single initiative, many of its efforts and programs are borrowed from other municipalities and agencies. Therefore, Communities of Opportunity should be able to measure the achievements of its various efforts against those efforts elsewhere.

Conclusions

Without changes, Communities of Opportunity will continue to lack sufficient direction and its achievements will be impossible to evaluate.

Communities of Opportunity's Success or Failure Is Impossible to Determine

Communities of Opportunity does not have a central, agreed-upon set of goals. The 2008 Plan has three separate but similar goal sets, none of which drives programmatic alignment or reflects current thinking. Furthermore, some of these goals are vague or otherwise poorly defined. Because Communities of Opportunity's goals are not understood and agreed upon, it is impossible for the City, the public, Communities of Opportunity's partners, or Communities of Opportunity's clients to know how it is progressing against its goals. From a resource perspective, it is impossible to determine whether fiscal and staff resources are being used efficiently.

Outcome Data Is Essential

Part of the City's investment in Communities of Opportunity was based on developing successful anti-poverty strategies that can be applied elsewhere in the City. Communities of Opportunity has repeatedly expressed its interest in outcomes. However, much of the data that it currently records and reports is on service provision. This limitation of Communities of Opportunity's data collection and reporting will make the determination of programmatic causality difficult to determine. Failing or flailing programs will not be able to be corrected, and successful programs may not be emulated or expanded from one node to the next, or from the study area to other parts of the City.

With coordinated case management and other efforts, Communities of Opportunity is attempting to collect and analyze additional outcome data. The Budget Analyst understands that the technical and legal barriers to sharing data are challenging to overcome, but from conversations with City staff we know that these hurdles can and have been surmounted or worked around. We encourage Communities of Opportunity to continue to work with City staff in developing creative ways to ascertain and share outcome data.

Several of Communities of Opportunity programs have been borrowed from elsewhere. The initiative should borrow evaluation metrics from similar efforts where available. Furthermore, the initiative should evaluate performance against other cities.

The End Goal Matrix Is a Valuable Tool Worthy of Expansion

The End Goal Matrix, which aligns goals, strategies, targets, status, and deadlines, can be an excellent performance management tool. It provides a framework in which various audiences can be made to understand Communities of Opportunity's intent, methods, direction, and progress. With an effort like Communities of Opportunity that is so hard to define and grasp, such a framework makes Communities of Opportunity's vision much more tangible. Although the Matrix would benefit from refinement, it is an excellent starting point. However, currently the Matrix does not relate to the Dashboard­–Communities of Opportunity needs to bridge these reporting tools to show how incremental achievements relate to overall goals.

Recommendations

In order to refine Communities of Opportunity's focus and improve its accountability, the Director of Communities of Opportunity should:

4.1 Establish a single set of goals and align all work to those goals.

4.2 Refine and expand the End Goal Matrix; align the Matrix to the Dashboard.

4.3 Work with departments throughout performance measurement processes and measure progress against other cities.

Costs and Benefits

The intent of the Budget Analyst's recommendations is for Communities of Opportunity to refine its goal setting, performance measurement, and evaluation practices. The development, implementation, and evaluation of Communities of Opportunity have and continue to be iterative processes; the Budget Analyst believes that these recommendations can and should be fitted into those processes. Implementing these recommendations should improve (a) external understanding of Communities of Opportunity and its objectives, (b) internal understanding of Communities of Opportunity's strengths, weaknesses, and progress, and (c) improve Communities of Opportunity's accountability by creating a more informed and contextualized understanding of its achievements.