Recommendation 14

Recommendation 14: Apply a systematic approach to assessing and monitoring the benefits and performance of location-based services

Implementation guidance	Related information
Why How Challenges Best Practices	LIFO Monitoring Related Frameworks: EIF Related Frameworks: UN-GGIM IGIF ELISE Resources Further Reading

Why

Public sector data is a valuable asset on which added value products and services can be built.
Understanding the extent, use and value of location enabled digital public services enables the value of the investment to be determined and helps target further investments and promote wider reuse.
Comparisons with other MS can help in identifying opportunities for re-use and collaboration.
Core location datasets such as addresses and geographical names are used in multiple cases. Interoperability of these and other datasets makes re-use easier in different sectors and geographies. However, justification of ‘infrastructure’ or ‘enabling’ policies and implementation projects involving core location datasets or interoperability measures is complex. However, learning is possible from successful cases where this has been done.
Initial cost benefit assumptions used to justify a policy or implementation project are subject to change over time, with new factors often emerging which require consideration. Rigorous downstream evaluation of cost benefit assumptions is not always done. This is vital where the policy and technology landscape is changing rapidly.
Infrastructure investments support both intended and unintended uses. Estimates on the frequency and extent of use may prove to be inaccurate. Datasets and delivery mechanisms are built to meet planned needs and estimates of usage. Continuous monitoring is therefore needed to help in planning relevant operational measures.
Directives such as INSPIRE and the Open Data Directive determine the need for specific actions relating to location data that may or may not otherwise have been taken or may have been taken at a later time. They present important policy drivers. However, they do not remove the need for national impact assessments and implementation business cases. Instead, because of the mandatory nature of these policies, they amplify the need to ensure there is a strong focus on implementation efficiency and cost minimisation and a parallel focus on ensuring widest possible use of the data to drive through benefits.

[Top]

How

Key to any proposed policy or implementation project is the identification of target benefits, Identification of target benefits effectively defines the purpose of a policy or project. Quantification of the target benefits supports the rationale for making the investment. Demonstration of proven benefits can also be useful in securing investment for further development or investment by other parties in similar initiatives of their own.
Return on investment (ROI) assessments are needed both to justify an initiative and to verify that the initiative has delivered the expected outcome in terms of costs and benefits. Such initiatives may include either ‘policies’ or ‘implementation projects’, both of which typically involve ROI assessments to determine the comparative merits of different options in terms of scope and approach. As well as looking at ROI, these assessments also consider the feasibility of different options, including the availability of funding related to the initiative. Governments or organisations also have to consider the relative priorities of different policies or projects as part of a portfolio approach.

What types of benefit to target?

Productivity benefits: There are many types of direct benefit relating to the sharing and use of location data. One of the simplest to communicate and understand is improved productivity, when tasks are made more efficient by removing manual processes or simplifying a digital process., for example simplifying the data collection and analytical processes in understanding Environmental Impact Assessments or Strategic Environmental Assessments (a staple of the INSPIRE investment case) or transport modelling using more accurate location data to reduce journey times for commuters, delivery drivers, shoppers, health care visits etc.
Another type of direct productivity benefit is reducing the cost of citizens or businesses interacting with the government (often called ‘reducing the administrative burden’). Integrating the national registers that hold land and property ownership, company registrations and addresses enable changes to be reported once and then synchronised across the different ‘basic’ registers. If these basic registers are used throughout all downstream processes and systems, benefits are thereby maximised. The principle behind this approach is known as the ‘Once Only Principle’.
Productivity benefits may or may not result in economic benefit (e.g. cost saving, increased time available to generate more income). Reducing journey times for parcel deliveries may have a direct economic benefit in terms of lower staff and vehicles costs. Reducing journey times in healthcare situations may do the same for home healthcare visits. There may also be indirect benefits related to health outcomes and costs to the health system. Reducing the administrative burden for citizens will not usually have an economic benefit but may have a social benefit (e.g. increased time available to perform tasks that are more enjoyable) or a democratic benefit (e.g. increasing trust in government).
Economic or financial benefits: These include cost savings, increased revenues, increased profitability (a combination of the two) and impact on GDP. Benefits may either be direct (measurable and attributable) or indirect (e.g. having a measurable impact on data accessibility or a calculable impact on GDP). The role of location data is an important factor when communicating benefits. Where location data is the product or service (e.g. a mapping product) any economic benefit can be more readily traced to the availability of the product or service. In many situations, the use of location data is only part of the policy, project or service that is being communicated. The contribution of location data, therefore, needs to be explained if the focus of the communication is on the impact of location data (or the impact of interoperability measures associated with the location data).
Social or environmental benefits: The challenges of climate change place demands on data and information, not least spatial data and location information, as everything happens somewhere. Utilisation of new technologies, efficient data collection and easy access to value-creating data is a foundation for the work of climate adaptation. The INSPIRE spatial data infrastructure for Europe is concerned with supporting better environmental outcomes. INSPIRE came about because there is so much location-related data about the environment and there were significant barriers in availability and use of that data. Environmental benefits are therefore important in identifying and communicating location data benefits.
By providing society with easily accessible environmental data that can be combined cross sector and cross countries, data-driven decisions on and investments in climate adaptation can be made. This includes the ability to carry out the analysis as efficiently and effectively as possible, through interoperability of data and availability of relevant technologies (e.g. sensors) and tools (e.g. data management, analytical tools). Arguments of this type can be important when central funding is needed from finance ministries to cover the cost of a national or other wide-ranging initiative.
Social value is often related to environmental factors, such as living in proximity to green spaces and availability of tree cover in urban locations. Extending tree coverage can have social impacts in terms of improved wellbeing but environmental impacts in terms of helping to deal with air pollution. Social value can also have measures of a cultural or community nature, including access to community services, safety and security etc. Safety and security is an area where location-related crime statistics are often compared against causal factors. Impacts may come not only from the investment in policing but also other factors such as street lighting, surveillance cameras etc. There are some useful ideas on social value investment that may be important in communications (see Social Value International Standards and Guidance). Social value may also have close connections with democratic value.
Innovation and effectiveness: Digital public service investment may result in new innovative services in areas not previously possible without the assistance of technology and associated location-related data. Such services may range from more personalised services in areas such as transportation (e.g. bike sharing), volunteering (selection, organisation and outcomes) and environmental services (e.g. waste) to services involving public participation (such as ‘fix-my-street’ type reporting of problems). The latter services also fit in the democratic benefits category. Effectiveness is not always associated with innovation. An effective service meets user expectations, gives them what they need, and has a user interface that handles information well for both transactional and information related activity. Digital public services do not always arrive ‘right for time’. Communication of new and improved capabilities promotes take-up and can also invite feedback to help plan further improvements. There are also links between effectiveness and other measures of benefit. For example, increasing the level of investment in local policing is not necessarily effective in reducing crime – the investment has to be targeting at the right measures.
Democratic benefits: Democratic benefits may relate to increased participation, transparency and trust. Often different elements may be related. Local administrations may share details of planning applications and invite feedback electronically. These are examples of transparency and participation. Trust is also a factor when actions are taken (or not taken) reflecting the feedback of the local community. Reporting problems gives citizens an ‘agent’ role as well as their ‘user’ role in the community and improves trust, when actions are taken swiftly to resolve problems. Participation can take broader forms to help prioritise local policy, which is often related to targeting of available investment. Listening to the views of citizens can go a long way to maintaining trust. Fairness is also a factor to consider in that government should not be seen to favour one sector of society or location above another in undertaking its public service task. Location-related statistics often feature in the ‘evidence’ for policy affecting different locations and their communities, including the outcomes of particular policy measures.
Conclusion: There are many different types of benefit resulting from effective sharing and re-use of location data, as many as there are generally for investments in digital public services. Many of the types of benefit have overlaps, e.g. productivity improvements may deliver economic benefits, increased participation may help in delivering social and environmental benefits, and measures aimed at improving the environment may also have economic benefits. Because all policy changes or implementation projects will have cost implications, it is essential that any economic or financial benefits are identified in addition to the on-economic or financial benefits. This enables both an economic or financial justification and a non-economic justification (where applicable) to be carried out. The economic or financial justification may be in terms of net benefits (justifying the extra investment), cost savings (delivering more for less) or more efficient government (getting more out of a given investment). Where targeted stakeholders may be required to make commitments, they will also want to know the funding model related to those commitments.

Using a business case or impact assessment for decision making

Business cases or impact assessments are usually needed to help decision makers understand the rationale for taking action, the most appropriate action to take, and the impacts that can be anticipated in terms of costs and benefits. A business case or impact assessment will typically include the elements outlined below.

Definition and scope

A statement of the problem; the stakeholders involved, reasons for taking action (e.g. policy or strategic drivers), potential solution or solutions to be assessed.

Assessment framework

The assessment framework contains the cost elements and types of benefit to be assessed. Cost elements include people, technology, data, and physical resources. Types of benefit include economic and financial benefits (cost savings, increased revenues, GDP enhancement), efficiency gains (increased productivity, burden reduction), social and environmental benefits (e.g. improved health, improved air quality, reduced carbon emissions), and democratic value (e.g. increased trust and participation). Benefits may be further subdivided into ’intermediate benefits’ and ’outcome benefits’ or ’end benefits’.
As an example, public administrations may publish their datasets on a national (open data) portal (output), create standard metadata to describe the datasets (output) and enable easier discovery of relevant datasets (intermediate benefit) and through use of these datasets enable private companies to create value added products (intermediate benefits) and increase profitability (outcome benefits) or enable more informative or easier to use digital public services (outcome benefits).
The assessment framework will often be constructed in the form of a time-related model, with multiple data points and underlying assumptions. Such a model can be used to assess different option scenarios varying, for example, scope, time, resource assumptions, and take-up assumptions. Simpler models are used for project cost benefit assessments. More complex models are required for infrastructure investments or crosscutting policies or projects with multiple stakeholders and various network effects.

Quantitative assessment

This involves quantification of costs and benefits with appropriate risk-based adjustments (sensitivity analysis).
Financial cases will typically use discounted cash flow techniques to determine the net present value of an investment based on future projections. This technique is important in assessing different investment options with varying payback periods. Sensitivity analysis will usually make adjustments to increase cost estimates and reduce benefits estimates, often making assumptions about the differences across stakeholders and implementations.
Macro-level socio-economic models may be constructed for complex policy or project scenarios (i.e. those involving multiple stakeholders, locations, baseline scenarios and possible outcomes). Socio economic analysis or ’value chain analysis’ typically involves ’input-output’ models to construct a representation of the stakeholders and processes involved in the scope of the policy or project. Such models provide a simplified representation of often complex network effects. Analysis may focus on part of the overall picture (e.g. a single dataset, core processes, processes with proven issues, sector use that is relatively easy to analyse, an individual geography). Assumptions are then made to extrapolate the analysis to the wider landscape, a technique sometimes referred to as ’benefits transfer’. Related benefits should also be flagged, for example the link between improvements in air quality, health impacts and cost of healthcare.

Strategic or qualitative assessment

Quantitative assessments may also be complemented by strategic or qualitative assessments that have a variable role between mandating the need for an investment (e.g. to fulfil a legislative commitment, respond to a disaster event) or provide additional backing to the overall case (including the likelihood of support for or resistance towards a particular option).

Interoperability and location data considerations

While the business case and decision-making approach is common regardless of the nature of the policy or project, there are certain factors that may be relevant to highlight where interoperability and location data elements are important parts of the picture.
Interoperability elements: It may be important in the business case to highlight the interoperability elements relevant to the problem (interoperability barriers) or the proposed solution (interoperability measures). This analysis can be used to demonstrate the ‘interoperability benefits’ of the proposed project. This can be done by indicating the relevant aspects of the EIF applying to the project, such as the interoperability layers or principles, or in the case of a project with a significant location data component, the relevant aspects of the EULF Blueprint, such as the focus areas or recommendations.
Policy drivers: A project investment may be linked to policies that are either direct or indirect drivers for the project. A ‘direct’ policy driver may be the policy requiring or even mandating the project. Without the direct policy link the project would not be needed or feasible. For example, a project to implement INSPIRE-compliant datasets and services might be necessary because of the legislative requirement under INSPIRE. Similarly, a project to fund and make available high value open location datasets via standard APIs may be needed as a result of the Open Data Directive. In these cases, the INSPIRE and Open Data Directives are direct policy drivers for the respective projects. On the other hand, an indirect policy driver is one which is supportive to the achievement of the project aims but is not a sufficient reason for undertaking the project. If the indirect policy driver were not there, the project could still go ahead and would have to be justified by other means. For example, a project to assess the collective energy efficiency of building stock in a particular district may be required as a result of the Energy Efficiency Directive or Energy Performance of Buildings Directive (direct policy drivers) but would benefit from the availability of INSPIRE datasets and standards to locate the position and extent of the buildings within the defined district (INSPIRE is an indirect policy driver).
It should be noted that, although interoperability elements may be highly significant in digital public service problem analysis and implementation solutions, it is unlikely that the European Interoperability Framework in its current form or equivalent National Interoperability Policies will be direct policy drivers for many digital public service investments. Interoperability is more often seen as enabler rather than a reason on its own for a project. More likely direct policy drivers will come from the thematic policies that involve interoperability elements to enhance exchange of data and connectedness of systems to implement the policies (for example, transportation, health, environment, energy policies}.
Synergies regarding interoperability measures between European interoperability policy and European thematic policies and between National interoperability policies and National thematic policies are important to promote in order to lower barriers and maximise the integration opportunities and benefits coming from interoperability. In a basic sense, interoperability policy aims to simplify integration. There are many mutually-reinforcing policy options through which this can be achieved (see table below).

*Policy options to lower barriers and realise benefits associated with interoperability*
Policy options	Lower economic barriers	Realise benefits
Lead on developing interoperability vision and foster alignment on roadmap Examples: EIF, EULF Blueprint, INSPIRE	An interoperability policy aims to simplify integration. Impact: Reduce costs of integration, operation, installation, maintenance and upgrade (for systems using the data; the cost of implementing and maintaining the policy also have to be taken into account) Cross-policy alignment to minimise burdens	An interoperability policy can indirectly stimulate reuse, lower exit / shifting costs / lock-in Impact: More innovation opportunities, more price points, more features, more choices in products – more competition. Encourage and facilitate stakeholder participation in the development of a roadmap of activities that lead toward the realisation of the interoperability vision.
Support consensus building Examples: Minimal Interoperability Mechanisms (MIMs), INSPIRE	Common interfaces support reuse and reduce costs of data access	Capture interoperability requirements of interfaces, encourage and facilitate stakeholder participation (in defining interface requirements, a roadmap of activities, etc) Establish effective governance when moving from project to maintenance – who’s doing what, who decides, and who’s paying?
Promote open standards, open data and use of open source software Examples: DG CNCT open data policies, ODD high value datasets, INSPIRE cross-border harmonisation, MIM data models for smart cities, Open APIs	Open high value public datasets accessible through open APIs reduces costs and provides access to high quality interoperable data	Open standards for data and use of open-source software are extremely important and can be enforced through policy and public procurement actions. Open public data supports innovation and competition and provides transparency.
Provide funding Example: cost of infrastructure and maintenance	Address funding gaps and any uncertainties related to hidden costs	Implement services not otherwise possible
Encourage public / private partnerships Example: DIAS	Address funding gaps and any uncertainties related to hidden costs	Implement digital public services not otherwise possible Private sector possibilities in other administrations

Economic modelling of the impact of location data: Location data may be a major focus or even the entire focus of an assessment. In this context, Alan Smart of ACIL Allen Consulting and Andrew Coote of ConsultingWhere have carried out several studies estimating the economic impact of location data, from national and regional studies, to studies of individual organisations (e.g. mapping agencies), particular high value datasets (e.g. addresses) and important use cases (e.g. land administration).. A useful description of techniques from these and other studies can be found in their presentation to the World Bank, Land and Poverty Conference 2017 entitled “Economic and Financial Modelling of the Impact of Geospatial Information – Techniques and Results for Land Administrations in Developing Nations”. A summary of the different techniques and uses is given below.

*Techniques used in economic and financial modelling of the impact of geospatial information*
Technique	Description	Usage
Welfare analysis	A model that captures the economic value of a good or service, applying concepts of consumer surplus (willingness to pay) and producer surplus (difference between revenues and costs).	Generally appropriate for valuation of specific geospatial data products or the economic effects of different pricing policies. Pollock, UK (2008) estimated a net benefit of £156m in moving from average cost pricing to marginal cost pricing for geospatial data. This involved using multipliers, which can be challenging to apply coherently. Welfare analysis is best suited to evaluating a single product or service rather than a whole package of datasets.
Gross revenue estimates	Estimate gross revenue for the geospatial sector.	Oxera (2013) in a report for Google estimated that global revenues from geo services were between USD150bn and USD279bn. This technique is not a true indicator of economic value as it ignores input costs associated with generating the revenue.
Value added analysis	Total revenues of an organisation less cost of inputs. Gross value added makes up the bulk of GDP.	Gross Value Added (GVA) is a more realistic indicator of the economic contribution than measures of total revenue. The Oxera report estimated that GVA of geo services was USD113bn compared with GVA of all sectors in the global economy of USD70tn, suggesting geo services account for 0.2% of global GDP.
Value added along supply chains	Extends the estimate of value added to examine the contribution of geospatial systems to the supply chain.	Used by Oxera in 1999 to estimate the economic contribution of Ordnance Survey in the UK. The result was a total Gross Value Added (GVA) of between £79bn and £136bn with breakdowns by sector. Allen Consulting did a similar analysis in Australia in 2010, estimating GVA at around AUSD12.5bn.
Value chain analysis	This approach analyses business processes for different parties in the supply chain to identify opportunities for adding value and increasing competitiveness.	Value chain analysis can be useful to illustrate linkages between the geospatial data supply chain and related industries and reveal productivity and employment impacts not captured in more static analysis. These network effects can lead to further value creation as outlined in Longhorn and Blackmore (2008).
Economic impact assessment	Estimates the additional value created by an investment or policy.	Geospatial information economic impact assessments attempt to estimate part or all of the total extra value. To do this, the analysis must establish two scenarios: 1) a reference case for the services to be assessed 2) a counterfactual for the situation without the services (typically representing the next best option)
Cost benefit analysis	Benefits represent the additional value produced when compared to the counterfactual. A similar approach is taken to costs. Justification is when the change in benefits exceeds the change in costs over time (with discounting back to present value).	Used for assessing investment decisions or policy change. Benefits can be either tangible (direct or indirect) or intangible (unpriced) benefits. In some cases, the stated preference of consumers (willingness to pay) or their revealed preference (estimated trade-offs in terms of concessions in exchange for a benefit) are taken into account. Cost benefit analysis is a partial analysis that does not take into account changes elsewhere in the economy. Multipliers have been used but are difficult to estimate.
Computable general equilibrium modelling	A CGE model is a representation of all markets in an economy. The models recognise that changes in one entity or sector can have repercussions elsewhere. The technique draws on different economic outcomes for a reference case compared with a counterfactual.	CGE modelling is a more rigorous method of estimating the economy wide benefits when significant resource shifts are likely. The technique can be used to show the national and if necessary regional economic impacts of an investment or policy change. It has been applied in studies in Australia, New Zealand, Great Britain, England, Wales and Canada. The strength of CGE analysis is its ability to incorporate technological change and to manage the consequent resource shifts in the economy, while overcoming the lack of resource constraints in multiplier analysis. CGE modelling is nevertheless heavily dependent on data, requiring extensive surveys and case studies to provide credible and verifiable results.

Source: Smart, A and Coote, A, 2017 (Adaptation)

UN-GGIM IGIF Strategic Pathway 3: Financial Implementation Guide. Appendix 3.7 gives an example of a socio-economic impact assessment approach. The guidance outlines a process with the following steps:

Agree on scope and priorities;
Develop the engagement plan;
Gather the socio-economic evidence;
Analyse the information gathered; and
Justify the benefits.

A series of case studies is also presented to illustrate the approach. Appendix 3.8, meanwhile, describes the components of a business case and includes a case study on the Danish Basic Data Programme.

Portfolio based decisions

Policy or investment decisions are rarely taken in isolation. Any national government has a range of policies to consider in its portfolio, taking into existing policy commitments, the manifesto of the party in power and the legislative programme agreed in government. Maintenance and implementation of policies has funding considerations that require budgetary support from the Treasury and ultimately the taxpayer. Public sector and private sector organisations have a mixture of ongoing costs and revenues as well as discretionary investments and any associated benefits, including revenue impacts, to consider.
Decisions on individual policies, projects or purchases are taken in the context of the overall portfolio as part of a budget cycle (typically one year but may be longer). The Executive Board of an organisation will assess the commitments for the budget cycle on a collective (portfolio) basis. The budget portfolio for a period will involve existing commitments and discretionary investments or funding reductions. Discretionary projects or purchases will be evaluated typically on the basis of strategic and financial considerations. In this respect, proposed projects with a positive business case may not be approved because other projects have a higher strategic priority, a higher financial return or are simply more affordable.
As ICT related projects typically have multi-year budget implications, it is often worthwhile to break them down into multiple stages to smooth out the discretionary investment need and allow for further investment decisions based on positive outcomes (see benefit realisation).

Benefit realisation

The ‘baseline’ estimates of costs and benefits used in making an investment decision should be evaluated periodically to confirm the validity of the investment, inform any changes needed in the implementation programme, update the funding requirements, and aid communications. The approach to monitoring costs and benefits should be included in the original business case. Funding commitments and future decision points should also be highlighted. This is particularly important with a complex policy or project having a multi-year investment programme.
Cost benefit monitoring is easiest where the costs and benefits are confined to an individual organisation. However digital public services often involve multiple delivery agencies and usually have very large external stakeholder communities, in terms of citizens and businesses. Infrastructure investments such as those associated with data sharing, which rely on multiple uses and in some cases network effects, are also more challenging to justify and monitor than policies or projects with a direct end-purpose.
For infrastructure investments, surveying the views of key stakeholders before and at different points in the implementation cycle can provide useful metrics. An example taken from the INSPIRE evaluation is shown in the table below. As mentioned earlier, it is important to distinguish between ‘intermediate benefits’ (e.g. number of dataset downloads resulting from improved discoverability and accessibility) and ‘outcome benefits’ (e.g. use of data to create or improve products and service). Most of the benefits in the table are ‘intermediate benefits’. The traceability of outcome benefits is also important. For example, did the use of a location dataset which delivers outcome benefits come from discovering and downloading the dataset on a data portal or finding out about the dataset from personal contacts or news items? Traceability of end user benefits can be problematic when providing and distributing free and open data. New business models can address this issue, for example digital platforms or other community-oriented SDI approaches where the relationship between the provider and the user can become closer.

*Benefits deriving from INSPIRE identified by Member States*
Type of benefit	Benefit identified by stakeholders	No. (%) of Member States
Direct benefits
Benefits from production / processing of geospatial data
	Improved quality and reliability of data	4 (13%)
	Harmonisation and interoperability	11 (35%)
	Improved cooperation among stakeholders	7 (23%)
Benefits from products (public / private) based on geospatial data
	Reduction of time / costs (efficiency)	10 (32%)
	Sharing and reuse of data	9 (29%
	Economic profit and new business opportunities	4 (13%)
	Innovation technologies and technical knowledge	11 (35%)
	Better overview, discoverability, availability, accessibility of data	18 (58%)
	Harmonisation and interoperability	11 (35%)
	Improved quality and reliability of data	7 (23%)
Indirect benefits
Transparency and improved policy making
	Contribution to policy making in various areas	4 (13%)
	Increased openness to share data by data providers	3 (10%)
Benefits at national and EU level	Socio-economic benefits	5 (16%)
	National infrastructure and data strategy development	6 (19%)
	EU-wide collaboration	4 (13%)

Source: INSPIRE evaluation country forms, 2021

‘Sampling’ will be important for both estimates and monitoring. The cost model will need to cover all supply organisations and components and take into consideration the cost drivers and any variations in assumptions (e.g. cross programme costs, organisational costs, size of ‘customer’ base). Benefits ‘sampling’ will need to consider ‘public value’ both in terms of supply-side impacts (e.g. savings in costs and effort in delivery agencies) and external impacts (e.g. burden reduction, growth, environmental outcomes). The benefits sample may focus on the most significant benefits and a sample community that is statistically representative to give an appropriately accurate picture of outcomes. As in the original business case, risk-based adjustments may still be needed in monitoring outturns but at a lower level.

Stakeholder surveys and workshops

Stakeholder surveys may be undertaken at various stages in the assessment of the policy or project to determine their views on such things as:
- barriers (the nature of the problem, the effects on different stakeholders, the underlying causes of the problem) and enablers or measures (potential solutions to solve the problem;
- development and validation of the ‘value chain model’ where different actors are involved and determination of processes where improved digitalisation and use of data may have the greatest social or economic impact;
- costs and benefits associated with the problem and potential solutions, quantified as far as is possible;
- views on the potential impact and ranking of different solution options and associated benefits:
- risk adjustments on both the level of costs and benefits to address for example, difficulties in identifying current costs across all elements and potential variances in future estimates of costs and benefits, taking into account the feasibility of delivery and extent of take-up;
- ex-post validation and updating of cost and benefit assumptions used in ex-ante assessment models, possibly done on a periodic basis as part of a long term implementation programme.
Where possible, different types of stakeholders should be consulted (e.g. providers of data, users of data) to obtain a consolidated picture. Different techniques may be used to make risk adjustments. These include removal of outliers and risk adjustment of costs (typically upwards) and benefits (typically downwards). Evidence should also be sought from stakeholders to back their views.
Workshops can be run to evaluate options and create consensus. The Delphi method [1] can be useful in orchestrating the evaluation process (the Delphi method is a process for arriving at a group opinion or decision by a panel of experts, based on a series of question rounds and sharing of results after each successive round of questions).This approach originated in assessing warfare scenarios but can also be applied in an investment context.

Performance benchmarking and improvement

In the same way that individual investment decisions need to be considered in the context of the overall portfolio, investment in continuation and improvement of a digital public service needs to be considered through monitoring at a portfolio level. In this context, a possible approach to performance benchmarking and improvement is defined below.

Agreed list of services for benchmarking

Define a list of ‘basic services’ to identify what can be expected to be implemented and measure / benchmark location-enabled digital government development against this list. Use a ‘basic services’ list which addresses all basic digital public services, with a balanced contribution of those involving location information.
Align the list of digital public services with those used by other countries to support both national and international performance benchmarking.

Regular performance monitoring

Apply a regular monitoring approach that looks at both “upstream” and “downstream” aspects of location-enabled digital public services, considering:
- The available components (technological and non-technological) for enabling the availability and access to location data and services;
- The e-services and processes that have integrated location data and web services;
- The use (take-up) of these location-enabled e-services by public administrations, businesses and citizens;
- The financial and non-financial benefits of using location data and services.
Use the indicators that are included in the INSPIRE monitoring and reporting obligations, e.g.:
- Existence, accessibility and conformance of data, metadata and network services;
- Use and benefits of data and network services.

Impact-based improvement

In identifying and monitoring the benefits of location information, it is important to focus on the benefits of the use and especially the integration of location data and services in (digital government) processes of public administrations, as this is where the benefits are most visible and tangible. The identification of the benefits of integrating location information in processes can be done at different levels. Benefits can be measured: 1) of one single location-enabled service that is provided in the process (in comparison with a traditional service) to support a G2C, G2B and/or G2G interactions, 2) of the entire location-enabled processes (in comparison with the traditional processes), or 3) of several processes within a policy action or policy domain. Moreover, it is important to look, not only at the benefits for government, but also to take into account the benefits for citizens, businesses and other parties and even broader socio-economic benefits.
Use the regular monitoring of “upstream” (i.e. production and dissemination) and “downstream” (i.e. use) aspects of location data and services to obtain a good understanding of return on investment across the public sector.
Use the information to fund improvements in particular location data or services and to prioritise investment across the governmental portfolio.
For projects applying agile methods or divided into specific implementation stages, use an ‘agile financing’ approach to release funds progressively taking into account the needs of each project iteration or planned implementation stage, and the success in delivery and take-up as well as the learning from completed iterations or project stages.
Use a common maturity assessment method across EU Member States and benchmark the performance measurement with other MS to understand the relative degree of maturity and identify where good models may be found for future service improvements.

[Top]

Challenges

Different types of benefits are not always spelt out clearly and dealt with appropriately in business cases and impact assessments. It is important to distinguish between ‘outputs’ and ‘outcomes’, between benefits associated with intermediate stages in the value chain (‘intermediate benefits’) and benefits where value is actually realised (‘outcome benefits’ or ‘end benefits’). The assessment of productivity gains or burden reductions also needs to consider whether the time savings have a material impact in terms actual cost savings or substantial service improvement that frees up valuable time for individual users.
It can be difficult to attribute benefits when multiple drivers are involved. For data sharing initiatives, ‘benefits’ may be attributed in many ways, for example to the appointment of a digital champion that promoted the initiative, to the government digital strategy (where this initiative may have been one of 10 key elements of the strategy), to the governments open data policy which has the aim of opening up public sector data for the widest possible use, or to the government’s interoperability policy, which aims to promote good practices in interoperability of digital public services. The justifications for each of these initiatives could all use the same ‘evidence’. Furthermore, EU policies and initiatives operate in the same field of ‘good practice’. Attribution of ‘benefits’ could therefore also be applied to a European level digital strategy, data strategy, open data strategy, interoperability strategy and any associated legislative or policy interventions (including funding). Finally, the role and impact of any policy, local, national or EU, sits in a landscape where practitioners are applying good practices out of common sense, experience and demands from stakeholders (either other public employees, partners of government or the citizens and businesses they serve). The value of any policy has to be considered therefore in the context of these external and related factors. To attribute benefits to any one ‘policy driver’ is ‘very challenging’, if not impossible.
The narrower the policy or action, the easier its potential benefits can be predicted and measured and the broader the policy or action, the more difficult it is to predict or measure the benefits or impact. The easiest to construct justifications and business cases are those at a project level in single administrations. In the circumstances, where broad-ranging policies sit within a highly complex landscape of external factors and varying degrees of history and maturity in dealing with a topic in different administrations (nationally) or different Member States (at an EU level), these are more difficult to justify or construct a business case and correspondingly, to define and assess the quantifiable benefits of different policy options.
The EIF is an example of a very broad ranging framework. The scope of ‘interoperability’ in the framework includes legal, organisational, semantic, technical and governance elements, with 47 recommendations across the whole framework. These are most of the aspects of an overall digital strategy, expressed from the reference point of interoperability. In fact, the national interoperability initiatives (policies and strategies) across Europe are a mixture of digital strategies, dedicated broad-ranging interoperability initiatives and more-specific interoperability initiatives that focus, typically, on more technical or data-related matters. Both the breadth of the EIF and its overlap with digital strategies makes the identification and attribution of benefits difficult to assess and provides a corresponding challenge in determining the appropriate type of policy intervention.
Interoperability policy and infrastructure projects suffer similar challenges in terms of defining and attributing benefits. Both are enablers in the delivery of benefits and may have intermediate results that can be attributed, for example more organisations adopting relevant standards, more cases of external organisations using public datasets. However, outcome benefits such as cost savings, environmental improvements, increasing revenues come from the use of standards or data for specific purposes. In simple terms, interoperability policies or infrastructure projects do not deliver benefits but are enablers to the delivery of benefits in different circumstances.
Use of cost benefit analysis for investments of policies has its own set of challenges, particularly when the range of affected stakeholders and processes is significant and the investment or policy applies across a range of sectors and geographies. Optimism bias should be avoided. This can be done with appropriate sensitivity analysis. An important challenge is in defining the counterfactual to compare the proposed change against. Policy impact assessments typically require a baseline scenario which looks at what happens if we “do nothing”. However, a common error is to assume that without the change, nothing happens but this is rarely the case. To be credible, a cost benefit analysis must have a credible counterfactual.
A further challenge is in analysing all the costs and benefits when the range of stakeholders in terms of types and maturity, the number of cost elements, and the types of benefit are substantial. Sensitivity analysis will be important to address the different levels of uncertainty. It may also be useful to consider focusing only on major benefits (this may be sufficient to justify the investment), restricting the analysis of benefits to productivity effects (which are inherently understandable) rather than modelling broader economic effects, or creating estimates for different classes of implementation in the assessment model (e.g. by size of administration).
Each of the assessment techniques has benefits and limitations. It is important to choose the appropriate technique for the required purpose. The summary of the different techniques in Figure 14 provides some pointers in this respect. It is also important to recognise the limitations of any technique and to factor in those limitations to the risk analysis for the investment or policy change. Below are some potential limitations to note:
- A socio-economic model is always a model and any repeat assessment after implementation is refining the assumptions in the model rather than providing ‘evidence’ of results;
- Views from surveys are also ‘views’ and not hard evidence;
- Monitoring number of downloads or website visits does not demonstrate that what has been accessed has been used to create value (i.e. saved time, money or contributed to a product or service);
- Very small time savings on frequent tasks or moderately large savings on infrequently carried out tasks may appear to be significant where very large numbers of people are involved. However, the time savings for each person may not be sufficient to spend on productivity activity which will contribute to GDP.
Monitoring and benchmarking in the context of digital public services tends to focus on the main upstream activities of the value chain (readiness and availability), while the downstream elements (use and impact) are neglected because of the difficulty of finding this information.
Indicators can sometimes be difficult to measure, with information provided too vague, general or abstract. Involve professional investment analysts to validate indicators.
Impacts of new services or service improvements can be difficult to predict. This is why ongoing monitoring and targeting of improvements is needed. An iterative approach to service delivery and improvement (see recommendation 8) can also be beneficial.
Financing a reliable policy assessment, investment case or monitoring approach can be challenging. The more complex the assessment and the more stakeholders involved the more costly is the process. Similarly, it can be costly to maintain frequent automated monitoring or carry out surveys to provide extensive business analysis / intelligence.

[Top]

Best Practices

[Top]

LIFO Monitoring

The Location Information Framework Observatory (LIFO) monitors the implementation of EULF Blueprint recommendations in European countries. Read about the implementation of Recommendation 14 in the LIFO Country Factsheets or the LIFO European State of Play Report. Explore the results for selected countries at LIFO Interactive Dashboards - Recommendations.

[Top]

Related Frameworks: European Interoperability Framework (EIF)

EIF Pillars	Recommendations
Underlying Principle 12: Assessment of effectiveness and efficiency	Recommendation 19: Evaluate the effectiveness and efficiency of different interoperability solutions and technological options considering user needs, proportionality and balance between costs and benefits.

[Top]

Related Frameworks: UN-GGIM Integrated Geospatial Information Framework (IGIF)

Strategic Pathway 1: Governance and Institutions

Documentation	Elements
Implementation Guide Appendices	Leadership Value Proposition

Documentation

Elements

Implementation Guide

Appendices

Leadership

Value Proposition

Actions	Tools
3. Defining Value
Strategic Alignment Study	APP1.2: Strategic Alignment Template
Value Proposition Statement	FIG1.6: Value Proposition Canvas
4. Setting Direction
Geospatial Information Management Strategy	APP1.3: Guidance for Mission, Vision and Goals Statements Future trends in geospatial information management: the five to ten year vision (third edition) Global Statistical Geospatial Framework Framework for Effective Land Administration Strategic Framework on Geospatial Information and Services for Disasters COVID-19: Ready to Respond - The role of the Geospatial Community in Responding to COVID-19
Change Strategy
5. Creating a Plan of Action
Country-level Action Plan	APP1.4: Country-level Action Plan Template
5. Tracking Success
Monitoring and Evaluation	APP1.5: Monitoring and Evaluation Template
Success Indicators	APP1.6: Success Indicators

Strategic Pathway 2: Policy and Legal

Documentation	Elements
Implementation Guide Appendices	Governance and Accountability

Documentation

Elements

Implementation Guide

Appendices

Governance and Accountability

Actions	Tools
4. Future Proofing
Future Proofing
6. Delivering Compliance
Impact Assessment

Strategic Pathway 3: Financial

Documentation	Elements
Implementation Guide Appendices	Business Model Opportunities Investment Benefits Realisation

Documentation

Elements

Implementation Guide

Appendices

Business Model

Opportunities

Investment

Benefits Realisation

Actions	Tools
2. Situational Assessment
Current Business Model	APP3.4: Example of a Business Model Canvas
3. Financial Plan
Desired Business Model	APP3.5: Developing a Business Model - Some Considerations
4. Case for Investment
Socio-Economic Impact Assessment	APP3.7: Socio-Economic Impact Assessment Approach
Business Case	APP3.8: Components of a Business Case
Investment Appraisal
6. Deriving Value
Benefits Realisation
Communicate Benefits

Strategic Pathway 9: Communication and Engagement

Documentation	Elements
Implementation Guide Appendices	Stakeholder and User Engagement Strategic Messaging Strategy, Plans and Methods Monitoring and Evaluation

Documentation

Elements

Implementation Guide

Appendices

Stakeholder and User Engagement

Strategic Messaging

Strategy, Plans and Methods

Monitoring and Evaluation

Actions	Tools
1. Providing Leadership
Communication and Engagement Strategy
2. Understanding Opportunities
Stakeholder Identification	APP9.1: Categories of Stakeholders APP9.2: Identifying and Classifying Stakeholders
Stakeholder Analysis	APP9.3: Stakeholder Analysis Matrix
3. Setting Direction
Policy Platform
5. Monitoring Progress
Review and Evaluation	APP9.8: Review and Evaluation - Methods for Benchmarking
Stakeholder Surveys
6. Communicating Value
Benefits Communications
Lessons Learned Resource

[Top]

ELISE Resources

Type	Resource	Date
Study	Digital Platform for Public Services	2018
Study	The Role of spatial data infrastructures in the digital government transformation of public administrations: See impact section which gathers indicators concerning the breadth of usage of the SDI and the benefits derived, as well as the cross-border perspective.	2019
Study	Leveraging the Power of Location Information and Technologies to Improve Public Services at the Local Level	2021
Study	Evolution of the access to spatial data for environmental purposes	2022
Study	Quantifying the benefits of location interoperability in the European Union	2022
Webinar	The Role of Geospatial for Digital Government Transformation	2019
Webinar	The Role of Spatial Data Infrastructures for Digital Government Transformation	2019
Webinar	Evolution of the access to spatial data for environmental purposes – Study presentation	2021
Webinar	Location Interoperability State of Play – Results of a Europewide Maturity Assessment	2022

[Top]

Recommendation 14