How an Enterprise Architect Measures Value Realization in the Public Sector (Trends and Insights)

  • Article

How do we measure value and value realization in the public sector? Public projects are typically conducted for the benefit of social welfare, quality of life, citizen safety, or other dimensions that can be difficult to quantify in the traditional sense of ROI.

This post, drawn from a paper by Stephen Kell, presents ideas about considering social value while comparing the impacts of various initiatives.

Applying Social Value Metrics

Social value metrics are often useful for representing the performance of a government or the social impact of an organization. Social impact may also be an important consideration when an Enterprise Architect compares the value and risk of an array of initiatives. Some examples of the practical situations that social value modeling can apply to include:

Evaluating Impact for Upgrading Metropolitan Digital Capabilities

Applying social value metrics to work done to date for the digital on such initiatives can be used to provide a basis for prioritizing and calculating the return on investment for future digital initiatives.

Choosing Between Initiatives of Change

In one engagement, business leaders compared the relative merits of spending money to upgrade hospital computer systems vs. spending money on a new hospital wing. Though expanding the hospital at first seemed the best option for expanding care, the quality of care was more impacted by improving physician access to patient information. This increase in efficiency reduced the length of patient stays and increased hospital capacity. Improving quality of care was a more important dimension than the capacity of the hospital. A detailed social analysis gave a different picture than an initial impression based on traditional ROI.

Representing Performance of Social Enterprises and Charities

Social Enterprises in this context are enterprises set up for reasons other than purely profit. They tend to have objectives such as reducing unemployment or employing people with certain disabilities or a marginalized section of the community, therefore their value needs to be expressed in term of the social impact as well as the P&L.

Charities are set up with specific social aims in mind so that the value add has to be in terms of social impact and not the P&L. Indeed, for a well-run charity the P&L will be close to zero.

Generating Corporate Citizenship Reports

Many companies such as Microsoft, now publish citizenship reports that are intended to represent the positive social impact of the company. Social value modeling can be used to assess such impact and the citizenship agenda of these companies.

Planning for Extreme Events

Extreme value theory, a type of mathematical analysis, can be used when considering the social value of implementing various initiatives to see how each of a set of proposed initiatives affects the risks associated with extreme events. The benefit will be the difference between the current Value at Risk and the Value at Risk after the initiative has been implemented.

Characterizing Full Value of Consulting Services

The value of consulting services has largely been assessed on the basis cost savings or risk reduction. Using a social value model, value can additionally be expressed in terms of social impact.

Determining Value Dimensions

Value dimensions are, like beauty, very much in the eye of the beholder. Dimensions that are well-defined for the commercial sector are often more difficult to define or less relevant when applied to the public sector.

Often the main driver in the public sector is not money, but a desired social impact. Many government departments and charities have to balance budgets, while still achieving the social outcomes they pursue. To identify the most important value dimensions for a project, we need to understand how an organization is measured, how it is judged, and what it wants to achieve.

Increasingly, commercial companies are looking at their corporate citizenship and therefore are starting to have value dimensions other than financial. The techniques and methods outlined here are applicable to those situations as well.

Some value dimensions typically applied in the commercial sector include:

  • Financial –the financial measures that need to be met
  • Business strategy – alignment with current business strategy
  • Market and industry trends – level of response to trends
  • Regulatory changes – regulatory changes to which an organization needs to adhere
  • People – values of the organization that will attract, retain and engage staff
  • Social responsibility – alignment to the social responsibility and ethical business agenda of the organization
  • Business Decision Maker’s personal drivers – Revenue, margin, market penetration, expanding to new markets, acquiring new companies, changing the demographics of the customer base, and so on.
  • Ability to execute – ability of organization to make necessary changes to realize value
  • Time to value – time required before the value starts being realized

For the public sector, we need to consider additional value dimensions, such as:

  • Alignment to government policy
  • Alignment to social aims of a charity
  • Economic Impact
  • Social Impact
  • Environmental Impact

These value dimensions are often obtained by consolidating multiple KPIs according to a social value model.

Using Consolidated KPIs

Many models exist defining KPIs for government regarding social impact investing and performance. Though some models attempt to be general, no model is definitive or correct for all governments and all initiatives, as social values vary. Corporations have also defined KPIs to reflect corporate citizenship, demonstrating how social modeling can be applied to corporations as well as government departments and charities.

Consolidating KPIs into Core Value Dimensions

To aid in comparison, the KPIs for public initiatives can be consolidated into the following core value dimensions:

  • Economic impact
  • Social Impact
  • Environmental impact
  • Alignment to government policy
  • Alignment to strategy (IT/Business)

Other dimensions to consider for core value include:

  • Cost of implementation and running, including any additional costs to users
  • Time to value
  • Ability to Execute
  • Number of citizens affected (indirect)
  • Number of clients (direct users)

One of the issues when creating consolidated dimensions is that KPIs are measuring different units using different scales, so some way of normalizing them to a common unit and range needs to be used.

One method of doing this is to use an intermediate value (T-Value) as proposed by Charles Taylor (“Composite Indicators: Reporting KRIs to Senior Management.” The RMAJournal, 16-20. April, 2006). T-values are calculated by identifying thresholds for strongly positive, positive, neutral, negative, and strongly negative ranges within a KPI, normalizing the values among KPIs, and then averaging them.

Using this method, the T-Value can take values from -2 to +2, which can then be visualized using Spider diagrams.

Visualizing Value Dimensions Using Spider Diagrams

The traditional Spider diagram, or more accurately the Kiviat chart, normally has a scale from 0 to 5. In the case of the value dimensions we are talking about, they can be both positive and negative. Therefore we use a scale from -2 to +2. 

clip_image001_thumb

Anything outside the circle is positive; anything within the circle is detrimental to the value dimension. For example, a large construction project might produce the following chart, showing that the economic impact is very positive but the environmental impact is very negative.

clip_image002_thumb

In order to compare different investment initiatives, we can produce Spider diagrams for each initiative and then place them on a grid of costs against risks, as shown in the following figure. This helps us compare and analyze the relative benefits of the initiatives in order to make investment decisions.

clip_image003_thumb

Mathematical Modeling of Social Value

A social value model has two parts: the Financial Benefits and the Social Impact. We mathematically model social value to enable us to evaluate, in a pseudo financial way, different initiatives or investment opportunities. We can apply analytical techniques to investments in social value in a similar way as commercial companies assess investments in increased revenue or productivity.

The basic social value model is a way of including non-financial benefits into the ROI and IRR calculations in order to do a comparison of investment opportunities where the main objective is not financial.

The benefits to consider are:

  • Financial benefits in terms of cost savings to the social department/organization and to the client of the service.
  • Risk reduction, which can provide a benefit in terms of reducing the value at risk, or in terms of reducing risks which have a social impact. It is also possible that an initiative will increase some risks.
  • Social Impact benefits in terms of the economic, social, and environmental impacts.

Financial Benefits

The purely financial benefits for an initiative form one side of the picture. The financial benefits need to be identified, weighted and then added up. Such benefits are generally in terms of cost savings or more efficient collection of revenue (taxes).

The steps to quantify the financial benefits are:

  1. Identify a benefit.
  2. Weight the benefit in terms of priority to the government or organization.
  3. Add the weighted benefit to the other benefits identified.

These benefits can be spread out over time and be represented by a series of time-related value measurements. We can work out the value per time interval (monthly, quarterly or yearly) and then apply discounted cash flow analysis.

Note also that some “benefits” may be negative because the effect of implementing the initiative may be to derive benefits in one area at the cost of benefits in another area.

Social Impact

The social impact is much more difficult to put a figure on, as well as being more emotive. The approach taken is as follows:

  1. Identify and quantify the economic benefits using either financial figures or an index of values relating to impact.
  2. Identify the social benefits and quantify.
  3. Identify the environmental benefits and quantify.
  4. Put a political weighting on each of the benefits.
  5. Convert any financial benefits into an index.
  6. Sum the weighted benefits to form the Social Impact figure.

In the same way as the Financial Impact can be spread out over time, the Social Impact benefits can also be spread out over time. We can calculate the benefits per period, monthly, quarterly or yearly, and then apply techniques such as discounted cash flow.

Note also that some “benefits” may be negative because the effect of implementing the initiative may be to derive benefits in one area at the cost of another area. For instance there may be initiatives that have large economic benefits at the cost of the environment.

Calculating ROI and IRR

The methodology for calculating both ROI and IRR or indeed other financial measures uses a combination of the above Financial Benefits and Social Impacts in order to take into account the social impact of the investment or initiative. The process follows for the index values we use in our calculations, based on US dollars (other currencies would produce different scaling).

  1. Convert the Financial Impact value to an index at the rate of US $1 = 1 unit and then add it to the Social Impact. This gives the benefits side of the equation as an index, spread out over periods as appropriate.
  2. Convert the cost side of the equation to an index at the rate of US $1 = 1 unit so that all the figures are dimensionless units.
  3. Calculate ROI and IRR. Calculate NPV if costs and benefits are spread out over time.

Resources

Alvarado, E.; Sandberg, D. V.; Pickford, S. G. “Modeling Large Forest Fires as Extreme Events.” Northwest Science, Vol 72: 66-75. 1998.

Franklin, J. “Operational Risk under Basel II: A model for Extreme Risk Evaluation. Banking and Financial Services Policy Report.” Volume 27, Number 10, 10-16. 2008.

Franklin, J., Sisson, S. A., Bergman, M. A., & Martin, J. K. “Evaluating Extreme Risks in Invasion Ecology: Learning from Banking Compliance.” Diversity and Distributions, Blackwell Publishing, 14, 581-591. 2008.

Abarbanel, H.; Koonin, S.; Levine, H.; MacDonald, G.; Rothaus, O. “Statistics of Extreme Events with Application to Climate.” McLean, Virginia: The Mitre Corporation. 1992.

Taylor, C. “Composite Indicators: Reporting KRIs to Senior Management.” The RMAJournal, 16-20. April, 2006.

You May Also Like