Home   Publications   Papers you can read on line   Workshops   Links to other sites   New Stuff   Mark's Page  Feedback
Trying Hard Is Not Good Enough  Results Accountability 101 DVD

RBA/OBA Ideas
Postings on Various Topics
by Mark Friedman
(Replaces the RBAOBA Google blog site)

This space is devoted to the presentation of  insights, advice, and general rambling on topics related to RBA/OBA. This replaces the Google blog site which never really worked very well. If you have questions about RBA/OBA that you would like to see addressed here, please send an email to. xfpsi@aol.com.

Thanks.
Mark
January 22, 2010

Organic vs. architectural change
Posting #9 April 14, 2010 

I believe the case can be made that much if not most, philanthropy in this country and around the world is highly inefficient at creating the change we want. This inefficiency derives in part from the difference between “architectural change” and “organic change” processes. Architectural change is the “design and implement” model at the heart of most government programs and most grantmaking initiatives. Organic change is the “plant and grow” model we see in grassroots movements and self organizing systems. Architectural change is less efficient because it requires a constant infusion of new energy (money) to keep it going.  Organic change is often self-generating and self-renewing.

We have plenty of examples of organic change processes, incentives perhaps being the most obvious. Incentivize certain behaviors (with money, recognition, etc.) and people naturally respond with the desired changed behavior. RBA is, at its heart, an organic process. Show people an indicator or performance measurement curve that’s not OK and watch them organize the partners and resources to turn the curve. Embedded in any good RBA practice is the idea of what works or theory of change. And embedded within theory of change is the idea of “theory of grantmaking,” that part of the theory of change discussion unique to those who have money to give away.  

When I was at CSSP, I proposed some grantmaking ideas under the rubric of “Most Results for the Least Money.” These were inexpensive organic change ideas that you can link to here, as examples of the type of thing I am talking about. Foundations, United Ways and other funders should consider convening a Theory of Grantmaking advisory group to look at how you can get the most change (or best results) from the investment of funding and other resources. Such a group could examine the wide range of organic and other grantmaking strategies, examine the evidence for these approaches, and make recommendations on how to balance the grantmaking agenda to greatest effect.

The performance of teachers
Posting #8 April 14, 2010

Pretend, for a moment, that I can split you into two people and give you two separate teaching assignments for one year. You, Version A, are given a 3^rd grade class in an upper middle class neighborhood. You, Version B, are given a 3^rd grade class in a poor urban neighborhood. In both cases, your job is to teach the children in your class to read. At the end of the year, you will be evaluated on the basis of the average reading scores in your classrooms. The one with the higher score will be retained. The one with the lower score will be fired. Which of you will be fired and why?

The answer is obvious. The children in the poorer classroom started out with lower scores and finished with lower scores. You did a lousy job teaching this class. You are a bad teacher and deserve to be fired.

OK. That’s unfair. Let’s change the rules. Let’s look at the amount of improvement in reading from the beginning of the year to the end of the year. Well the one of you who taught in the poor school still gets fired because the children not only started off behind, but more of them have learning difficulties, live in homes where there are no books and have parents who can’t read.

OK. That’s not fair either. You can’t compare classes in two different schools like that. Let’s back up and give you two different classes in the same school. But you will quickly see that the same things will also happen between any two classes in the same school. Some students will do better or worse for reasons that have nothing to do with who you are or how good a teacher you are.

Fast forward to the 2012 Olympic Diving Competitions. Diver A does a triple back flip off the high dive but misses the water entry. Diver B does a swan dive off the low dive but nails the entry. How can we compare Diver A and Diver B. The Olympic Committee has an answer: “degree of difficulty.” One dive is harder than the other and is judged differently.

The same principle could be applied to teaching, but imagine how much harder it would be to assign a “degree of difficulty” to a child. Now try creating a degree of difficulty for a whole classroom. Now think about how to adjust that degree of difficulty over the year as the class changes when children move from one school to another.

There is a lot of discussion today about getting rid of “bad teachers.” You would think from hearing this that the problems in our education system are caused principally by a large number of bad teachers that the system refuses to fire. One could, of course, make a similar case that the problem with our legal system is a large number of bad lawyers that the system refuses to debar. The problem with the medical system is a large number of bad doctors that the system refuses to decertify. The point is that there is no simple way to use achievement test data to evaluate teachers, any more than there are simple ways to use patient data to evaluate doctors or law suit data to evaluate lawyers. Anyone who believes otherwise doesn’t know very much about teaching (medicine or law).

This does not mean that we can not use data to improve our education system. Using data to drive school improvement is different than using data to fire teachers. When schools make use of data to drive improvement it is usually site based (or school level) data, not individual teacher data, (e.g. total school 3^rd grade reading scores). Schools have a place where trend data on achievement test scores, attendance rates, graduation rates and rates of teacher retention are prominently displayed. There is an inclusive process for interpreting this data and taking action that involves a wide range of partners, including teachers, principals, superintendents, parents, business leaders, the faith community, elected officials, the media and the students themselves. 

When school improvement is managed this way it has a much greater chance for success and is far more constructive than blaming teachers or anyone else. The vast majority of teachers are dedicated hard-working professionals who often work under extremely difficult conditions. People who suggest that fixing schools equates with firing teachers do so because it is easier to name scapegoats than do the hard work of making our schools better. We will never get anywhere if we misconstrue the very nature of the problems we seek to solve.

Note: I am aware that I have touched only lightly on school improvement planning. Education is addressed in many places in "Trying Hard Is Not Good Enough," but see in particular the section "Unified Planning for Education" on pages 118-119. I have also not endorsed any particular method for teacher evaluation (e.g. peer review). I believe that individual performance evaluation in education or any other service system is a by-product of good supervision. When I was a first year high school math teacher I received essentially no supervision or support which is one of the reasons I only lasted one year. We have generally done a poor job of training people to be supervisors in the public and non-profit sectors. Supervision and individual performance evaluation must be addressed together and are topics for future writing. See pages 81, 84 and 105 for some more thoughts on these subjects.

How to organize the work of "turn the curve" tables
Posting #7 February 16, 2010

A "turn the curve table" is a group of people who come together with the intention of turning one or more population indicator (or performance measurement) curves. It is a good idea for people working together in this way to have experience with the "Turn the Curve Exercise" and some grounding in RBA (either from an RBA 101 workshop, reading the RBA book or watching the workshop DVD.) Following is some supplementary advice recently given to support the work of several  newly formed state level turn the curve tables. References that might identify the specific state have been deleted. 

At the start, it is always important to remind people that turning any curve is a hard thing to do. Population level indicator curves require the efforts of many partners, usually take years, not months, and can not be turned through the single cycle development and implementation of a static plan. Sustained efforts and living plans are the only way.

In general, the works groups can benefit by following the steps laid out in the "Turn the Curve" exercise. In fact, this would be a good way to structure some or even most of the first meeting. The turn the curve process is designed to be iterative and so subsequent meetings could constitute successive passes at this process. The best way to think about successive meetings is to address every step in the process at every meeting. At each of these passes the work gets better. The use of time limits on each part of the meeting can help groups develop this discipline. 

Following are a few notes on some of these steps that I hope will be helpful.

1.  Forecasting: Most people have little experience with forecasting. Forecasting is discussed in some detail on pages 56 to 59 of "Trying Hard Is Not Good Enough." And there is still more guidance on www.raguide.org. The two most important things to remember about forecasting is that (1) it must be politically credible, and (2) it must represent our best thinking about where the line is headed if we don't do something more or different (i.e. not where we want to go, but where we are going if we don't do something to change.) This might involve more than one forecast using different assumptions. However, when multiple forecasts are used, remember that forecasting quickly loses its utility if there is too wide a variance between alternative scenarios.

2. The story behind the baseline: The story is the place where we address causes. There is a natural tendency to give this part of the process short shrift. But the "story" is the diagnostic step in the process. Failure to adequately address the story would be like a doctor prescribing medicine without first diagnosing what is wrong with you. The story need not be about just what is wrong. It should address both what is working and what is not working. The story can and should reflect a rich array of perspectives and there should be an opportunity for a wide range of people to weigh in. This will almost always create a complex picture with competing or even conflicting interpretations. But this is OK. The story does not have to be resolved into a single monolithic consensus before other parts of the process can continue. A rich picture of causes will lead to a much richer discussion about choices for action. 

3. What works to turn the curve. This is the part I most wanted to write about, because there is less written guidance on how to have this discussion than for some other parts of the RBA process. It may be helpful to segment the "what works" discussion as follows:

3a. In the Turn the Curve exercise we ask people to wear different hats so that a broader range of partners can be represented. In the real work, the discussion of existing and potential partners must be deeper than this. The groups may find it useful to begin to create an inventory of government and non-governmental partners in (at least) two broad classes: partners who are now actively engaged; partners who need to be actively engaged. All good action plans include components to recruit and engage new partners.

3b. A well developed partners list allows the discussion of what works to proceed partner by partner, starting with the various partners in state government.

3b1. Think of what the state can do that is no-cost /low-cost and what it should do with it's next dollar to turn the curve. The "next dollar" idea can help focus the discussion quickly on the most highly cost effective measures. Note that the next dollar does not have to be a "new" dollar, but the next available dollar whether new or reallocated from another use.

3b2. What can other partners do, nc/lc and next $ and how can you recruit them to be part of the process. This is the same as the state discussion but for each partner. It's very helpful if these partners are part of this discussion, since "telling people what they should do to help" is usually counterproductive. Partners should have the opportunity to tell what they are already doing and to offer up new contributions without being pressured to do so. In the end, Everyone should get a share of the credit for the success of this work.

3b3. Of all these ideas, which ones do we most highly recommend in the short (this year), medium (next year) and longer term (3 to 5 years)? There are many ways to have this discussion and display these recommendations. In every case the groups will need to exercise judgments based on some criteria. One approach to doing this is offered in the book on pages 43 to 45, but there are certainly other possibilities. Most importantly, the groups should pay a lot of attention to no-cost and low-cost things that can be done quickly that will allow for early actions and some quick success.

3c. One line of action that is sometimes left on the cutting room floor is the idea of helping create new, or support existing, local parallel efforts. Many states have a well developed set of local councils, commissions and trusts who are natural partners in any effort  to turn a statewide curve. The relationship with these bodies should be about finding common interests and partnering as equals on curves of mutual interest. 

4. What would it take? Finally, there is another way to frame the "turn the curve" challenge. In the "normal" turn the curve process we ask "what works" to turn the curve as a way to generate a shopping list of ideas from which we later choose what to do. The more powerful question, however, is "what would it take?" "What would it take to significantly reduce child abuse or teen pregnancy or raise the rates of school readiness or school success?" The best analogies for this type of question come from the successful effort to invade Normandy in 1944 or land on the moon in 1969. This question is much tougher than taking on incremental progress, but it may be a perspective that top leadership should consider. Incremental progress, however it is ultimately constructed, may be more readily achieved in the context of a powerful vision for the future that can come from answering this kind of question. 

The population role of education
Posting #6 February 1, 2010

It is often true that conversations within the Education community focus so intently on school performance improvement that it is difficult to have a conversation about the Population Accountability dimension of education. This problem is compounded by the fact that many important education measures (such as graduation rate or percent reading at grade level) double for population indicators. (See Chapter 5 of "Trying Hard Is Not Good Enough.")

So what is the population conversation we should have when it comes to education? I think it is about why we need education in the first place, about the role of education in our society. So why do we need education? The answer has not always been obvious. (It would be interesting to revisit here the work of Noah Webster.) Public education has been such a long accepted part of our society that the basic need for education is rarely questioned. But when we think about first purposes, it seems that education's role is arguably about (at least) three things: (1) creating good citizens and a civil society, (2) helping young people reach their potential and become happy, productive, contributing adults, and (3) creating our next generation work force and securing our economic future. Each of these three roles lends itself to measurement. (For example, the adult literacy rate or the percentage of young adults in work or education. And in each case, the education system could be clearly seen as one of many partners with a role to play.

Part of the reason I think this conversation is worth having, particularly for those immersed in the day to day management of K-12 (primary/secondary) system, is that it shows that the education system is not responsible for curing the ills of the world. There is such a long history of expecting just this from educators that we forget how unfair that expectation is. By stepping back and thinking about the role of education in society we have a chance to get back to our shared values, to remember why our large monetary investments in education are worth every cent and more, and maybe to begin to re-conceptualize education as more than a set of services but rather a community responsibility for which everyone has something to contribute.

Shortcut method for choosing population indicators
Posting #5 January 24, 2010

The "regular" method presented in on pages 54 to 56 of "Trying Hard Is Not Good Enough" involves rating each potential measure "high, medium or low" on three criteria: Communication Power, Proxy Power and Data Power. A simpler method for doing this mirrors the selection steps in choosing performance measures in steps 4 and 5 of Appendix G.

For each potential indicator, identify those for which you currently have good data. Put a circle next to these measures. Then ask "If you had to stand up in a public place and explain to your neighbors what you mean by (the result in question, e.g. Healthy Children), and could only use one of the measures with a circle next to it, which one would you use?" Then ask, "What if you could have a second.... a third?" Through this method you identify the top three indicators for any given result. This method is exactly equivalent to the Communication/Proxy/Data method for this reason. By first identifying the measures for which you have good data, you address Data Power. In asking which one people would use in a public place, it forces them to consider Communication and Proxy power together, namely, which measures are most powerful that people will understand? This method is faster and easier to use than the process of rating each measure.

This method also sets up the creation of a Data Development Agenda. Again, paralleling the performance process, ask "If we could buy one of the measures for which we don't have data, which would be the first we would buy?" "What if we could buy a second?" Through this process you can create a prioritized list of needed new and better data.

Measures are categorized on the basis of what question they answer.
Posting #4 September 10, 2009
(See prompting email)

The upper right quadrant "How well did we do it" category can contain measures about customers. Consider the measures % of classes delivered on time and % of customers who say classes were delivered on time. They are almost identical measures. Customer data can often tell us how well services were delivered (How well did we do it?). The fundamental question to ask when thinking about whether a customer measure is upper or lower right quadrant is this: Could we do well on this measure and customers might still not be better off? For example, the % of customers who think the class is delivered on time is 95%. But the class itself is terrible and no one learns anything.

This same principle is illustrated by how we categorize the completion rate for a service or training course. The mere completion of a service or training course usually tells us little or nothing about whether people got anything positive out of it. So % of clients completing service or training is usually an upper right quadrant measure. But there are exceptions to this rule. If the training has an established reputation for giving its students new skills, then completion can be a lower right quadrant measure. Life saving courses, where earning a certificate really means something, is a good example.

This principle can be carried further to the general question of how to classify different measures of customer satisfaction. Customer satisfaction measures fall into two broad categories: Did we treat you well? and Did we help you with your problems? Customers can say they were treated well, that they liked the counselor, thought the building was comfortable, accessible etc. but still not be helped with their problems. So "Did we treat you well" is an upper right measure and "Did we help you with your problems" is lower right.

The deepest underlying principle here is that measures are classified not on the basis of some intrinsic characteristic of the measure, but rather on the basis of what question the measure helps to answer. If measures help answer How well did we do it? they go in the upper right quadrant. If measures help answer Is anyone better off? they in the lower right. If they could conceivably answer both questions, then pick the stronger answer, or as a last resort put it in both places. This same principle applies to the difference between population indicators and performance measures. Sometimes a given measure can play both roles, so that sometimes it serves as a performance measure, and sometimes as an indicator.

I would encourage you to read Chapter 4 (especially pages 65 to 79 and 99) of "Trying Hard Is Not Good Enough" where these matters are discussed. There can sometimes be legitimate differences of opinion about how to categorize a measure. When this happens, just put is somewhere and move on. Because all the measures in both upper right and lower right quadrants will be considered in Step 4 of the 5 step process for identifying the most important performance measures (See Appendix G).

Finally, the reference to "circumstance" is specifically about the client or customer's circumstance. Are they in stable housing?. Do they have a living wage job? It is not about a more general definition of circumstances. For this reason it always goes in the lower right quadrant?

Creating Results Culture Change
Posting #3 September 9, 2009

People talk about culture change the way they talk about magic. We know this wonderful transformation is needed and will somehow happen, but we're not sure how.

Let's take a shot at a definition of organizational culture. Organizational culture is the range of accepted norms of for what people in an organization think and how they act. An organization might have a dominant culture and numerous subcultures. Subcultures might vary by program, profession, or organizational role. Organizational culture also includes a story line or mythology about why the norms exist the way they do.

Culture change then is any significant variation in or deviation from organizational culture or subculture. Culture change can be positive or negative. Sustained culture change is any culture change that survives some defined degree of turnover in key positions. It is worth noting that culture change is not a smooth transition. There are holdouts. Some parts of the organization go ahead of others. Some people never get it.

It is arguable that there are three elements needed for culture change to take place. This is a kind of hypothesis that might be tested with regard to creating a results culture.

1. Leadership: No culture change happens without leadership. It may be possible to talk about the evolution of organizational culture without leadership direction. But any form of deliberate change requires leadership.

2. Vision and small steps: Culture change requires the odd combination of vision and small steps. First, a vision of what the organization culture should be, that is a picture of destination. Deliberate or directed change also requires a well defined series of small steps that can begin the process and move it forward. Massive overhauls are often not practical and usually don't work. So culture change requires a pathway of small steps that lead to bigger change. One guide to these small steps is the RBA Self Assessment Questionnaire new version (available on resultsaccountability.com). Start with one supervisor identifying and using performance measures. Then two supervisors, and so forth. After the accumulation of enough small steps, it is possible to reach a tipping point where change happens much more rapidly. It is worth noting here that among the seemingly small steps to take is the change in forms and formats for strategic plans, budgets, RFP's and contracts. Forms are the skeleton of any organizations culture. They can live for decades. And if you can change them the change can last for decades.

3. Finally, feedback. Any change process requires a feedback loop to see if you are making progress. Two types of feedback are needed. Information is needed on the extent of implementation. One can use the RBA Self Assessment Questionnaire to calculate a score on the extent of implementation. The second type of feedback is on whether curves are turning on key measures.

Measuring the success of population level strategies
Posting #2 August 21, 2009

The general population level progression from results to indicators to baselines, story, partners, what works and strategies, often leads people to ask next, "What performance measures will tell us if our strategies are working?" This, of course, is the wrong question. To tell whether our population level strategies are working, we must look to see if the indicator curves are turning. Indicators measure the extent to which strategies are working. Performance measures tell us if the individual components of our strategies are working.

For example, a partnership might come together to promote community safety as measured by indicators such as the crime rate or the percentage of people who feel safe. After working through the RBA/OBA process, they might settle on an initial three part strategy which includes community policing, improved lighting and a neighborhood watch program. To see if the overall strategy is working, we would look to see if the curves are turning on the selected indicators (i.e. crime rate and percentage who feel safe). As for performance, we would take each component of the strategy in turn and identify performance measures for that component. So for the neighborhood watch program, we might look at the percentage of neighbors signed up, or the crime rate for neighbors in the program compared to neighbors not in the program.

There is one other kind of performance measure relevant here, which may be the source of some of the confusion on this subject. For those managing the overall strategic planning process, there is a need to know the extent to which the strategy (or strategies) have been implemented, and how well they have been implemented. So a performance measure for the partnership managing the strategy might be the percentage of agreed action steps that are on track. Notice how this measure tells us how well the partnership is working, not how well the overall strategy is working. One could easily have a strategy that is implemented beautifully, but has no effect on the indicator baselines.

September 15, 2009: And more....Having re-read your initial note: Does it follow that successful performance of strategy components is a predictor of strategy success? And if not does this mean that they are the wrong components? 

Another way of phrasing this question is, "Could all components of a strategy succeed and the overall strategy fail?" I think the answer is "yes."

(1) We might be doing the right things but not enough of them. 

(2) We might be doing the right things and yet the strategy is incomplete. I believe we have learned that strategies sufficient to turn population level curves require the contribution of many partners, and include no-cost and low-cost elements, some of which may be unusually difficult to assess. In addition, successful strategies must be sustained over time and must include a process (an ethos?) for continuous rethinking and improvement. 

In both of these cases, successful performance measures of strategy components would not be a good predictor of the success of the overall strategy to turn a curve. 

This does not mean that partnerships responsible for developing such strategies do not have a responsibility to track component performance, to spur performance improvement and where necessary to redirect investments.

But we have such a long history of settling for the appearance and not the reality of change in quality of life for children and families. In looking for intermediate measures, we must be careful not to repeat this mistake in another guise. The improved performance of individual services and even whole service systems would indeed bode well for the quality of life of the customers of those services or systems. But these improvements could well take place in the context of overall deterioration in the well-being of the whole population of children. This is a simple summary of the whole history of child protection reform. We fix problems with services for abused children, but more children are abused.

Mark

Next Generation Contracting - Key Provisions
Posting #1 August 15, 2009

People have been asking me for more detail on what I mean by "next generation" contacting (or commissioning as it is sometimes called in the UK). Here are some additional thoughts. I welcome your comments and suggestions.

Next generation contracts will have three essential provisions, and an optional fourth. Provision 1 (Most important performance measures) and 2 (Continuous improvement process) are sometimes used in present generation contracts, but these two with the addition of provision 3 (Performance partnership between funder and contractor) create the "next generation" funder / contractor relationship. Provision 4, having to do with simplifying and standardizing contractor reporting, is a highly desirable but optional component. Clearly there are many other components of a complete contract genome (see Dawkins) that are necessary to have a fully functioning contract, so these are necessarily parts of a larger whole.

1. Most important performance measures (Provision 1) specifies the most important 3 to 5 How well did we do it? (Upper right quadrant) and Is anyone better off? (Lower right quadrant) measures by which the contractor's performance including its effectiveness on behalf of customers can be judged. If these measures are not identified in the contract itself, the contract should specify a process for quickly agreeing on the measures, with possible reference to the RBA 5 step process in Appendix G of the RBA book.

2. Continuous improvement process (Provision 2) specifies that the contractor will use a continuous improvement process (the RBA 7 Questions are naturally recommended) to monitor and improve performance over time. The contract must specify that this process takes place (usually monthly) at the highest level of overall contractor performance (usually agency or major program level) but should also, if possible, specify that the RBA 7 questions will be used (or phased in) to every supervisory relationship.

3. Performance partnership between funder and contractor (Provision 3) specifies that the funder and contractor will meet periodically (most likely quarterly) to jointly assess performance on the 3 to 5 measures, the story behind the baselines for these measures, and what each party can do to improve performance in the coming period. These meetings should be preceded by the submission of a performance report in RBA format (See Wyoming Part II and San Mateo budget formats) from the contractor to the funder. In these meetings, the funder and contractor act as co-equal partners in helping facilitate improvement. This means that the funder will also agree to tasks that can help the contractor improve, such as providing technical assistance, working to break barriers the contractor may be experiencing in getting access to necessary resources or cooperation from other agencies, essentially anything within the funder's means that can materially support the contractor's performance improvement. Nothing in this provision lessens the funder's usual responsibility for oversight of the contractor's performance and use of funds. Nothing in this provision substitutes for the grantee's ongoing improvement process in Provision 2 nor for the contractor's responsibility to pro-actively solve its own problems without the funder's help.

4. Coordination with other funders (Provision 4) - optional - The funder agrees to actively pursue agreement with other funders who also purchase services from the contractor for the purpose of simplifying and standardizing contractor reporting requirements.

These are conceptual descriptions of the contract provisions and must, of course, be translated into actual contract language. Suggested language or examples of such language are welcome.

Beginning of time