Reflections on NUAR Part 1
Why bother with a data model?
I was proud to be involved with the design and development of the National Underground Asset Register over several years. I was Product Owner on behalf of the Geospatial Commission for the “build phase” of the programme, when the main effort took place to build the foundations for the platform that Ordnance Survey operate today1 and, just as important, if not more so, to gather and load the data from hundreds of asset owners having transformed it into a standardised form.
I’m no longer directly involved with NUAR but decided to take a look back at some key developments during the building of the service in a short series of articles, starting with this one looking at the standardised data model.
Designing what that standardised form looks like, in the form of a harmonised data model, was a central pillar of the programme, and the publication of that model as an open resource is, in my opinion, one of its main legacies2.
The NUAR harmonised data model was the first implementation worldwide of an international standard developed by the Open Geospatial Consortium, called MUDDI – the Model for Underground Data Definition and Integration3.
This international standard provides a conceptual framework for those seeking to model buried assets and other subsurface features for different use cases and jurisdictions, and that conceptual model provides a common thread between all those implementations.
But defining a data model, making it conform to an international standard, and ensuring that ingested data conforms to it is a complicated business and takes a lot of time and effort. Why would you bother going to all that trouble?
Why bother with a data model?
Given the complexity of transforming, combining and standardising data from hundreds of disparate organisations, sectors and geographies, it is perfectly valid to ask the question: “Why bother with a data model?”
After all, it would be possible to retain source data in its original form, render it on a map and express the attributes in the same form and using the same terminology as supplied.
If we follow this line of reasoning through to its logical conclusion though, we can start to see issues with the approach. If rendering source features on a combined map, how do we define and implement consistent cartographic rules and conventions without a means of reliably and consistently identifying the sector, type and key characteristics of a given feature which dictate elements like colours and linestyles?
Then when we wish to dig into the detailed characteristics of a feature on the map, we may expect to encounter further problems. By simply expressing the attributes as represented at source we will find that we are describing the same thing in several different ways depending on who supplied the data. Worse, we will likely be propagating the spelling mistakes and inconsistencies that tend to arise from legacy data capture methods.
Finally, every time we add a new data provider or dataset to the mix, we will need to try and make it fit nicely alongside what is already there. Whatever manner we use to define cartographic rules will need to be extended and changed to accommodate the new data. We might find things in the new data that we haven’t encountered before, and for which, at best, we will need to define new rules. At worst, what we find may break the logic that we have developed for previous datasets and require a full reworking of that logic. This may happen every time we onboard a new data provider from the hundreds that may be targeted.
A standardised data model does not solve every problem outlined above in one stroke. It does however move the problems of standardisation and harmonisation, and the impacts of non-compliant data, to much earlier in the workflow, so we get early sight and have an opportunity to intervene early when an intervention is relatively cheaper and easier.
A standardised data model with a comprehensive and flexible data specification provides us with a blueprint, a set of joining instructions to which every new data provider and their data will need to conform. While the challenges of guaranteeing that conformance remain, the systems downstream of that process can be developed in parallel, safe in the knowledge that if the rules and processes implemented conform to the agreed data model, then any data coming through the pipeline will be presented in a consistent and coherent manner, and not require redevelopment of the downstream system.
And this is where the real value arises in any system which places user needs at its heart. By standardising and harmonising the data that is ultimately presented to users we are able to present that data in a cohesive and consistent manner. When users view two instances of the same type of feature on a map, they will be styled in the same way, because they can be consistently and reliably identified as being of the same type. When users query the attributes of features, they will see standard terminology to describe the same concepts, rather than seeing several different ways of saying the same thing.
Beyond a cohesive, positive and consistent user experience, we can also add opportunities to contribute to the improvement of data quality.
How do we tackle the mammoth task of improving data quality in the especially challenging domain of buried infrastructure where out of sight out of mind rules the day? We can be sure that a single isolated intervention won’t do the whole job, but by standardising and harmonising disparate source datasets into a consistent and well-defined target model, we can start to gain objective insights about data quality, by framing data quality in terms of conformance to that standard.
We can also start talking about disparate datasets in a standard and consistent way and make observations and measurements about data quality in terms of conformance, and those observations and measurements apply across all data because it is all being represented in the same way. Even where there are gaps in data or different levels of detail that prevent us from comparing apples with apples, those gaps and those differences in themselves give us insight about what needs to be improved in order to gain that full picture.
I will talk in more detail about the opportunities for data quality improvement in my next article.
Lastly, if we were to represent data exactly as it was supplied, then by definition we would only be representing data as it is collected and stored today. We would be bounded by the constraints and limitations applied to that data at the present time and at the time of collection.
By defining a standardised data model, particularly one that is based on international standards, we have an opportunity to model the world as we would wish it to be. There is an opportunity to try and build an aspirational model that gives us a roadmap and some futureproofing for eventualities that may arise due to, for example, improved detection and location technology.
There is an opportunity to model data that is not routinely captured currently, but which we judge could add real value to end users, and we can learn from good practice from around the world. Providing a home for this data in a standardised form presents an opportunity to encourage and incentivise its capture in future.
So to summarise, implementing a standardised data model allows us to:
Provide a coherent, positive and consistent view of buried assets and their characteristics to end users.
Provide a blueprint and a set of joining instructions to each new data provider coming on board, rather than having to reinvent the wheel several hundred times.
Analyse and measure data in a consistent, objective manner so that we can gain a foothold in the mammoth task of implementing incremental data quality improvement.
Model the world not just as it is, but how we would like it to be, futureproofing for evolving technologies and providing a home for useful information which may not be routinely captured at present but whose capture may be enabled in future by advances in technology, regulation, good practice and culture.
If we accept that a standardised data model is a good idea, why base it on international standards?
If we acknowledge the benefits of developing and implementing a standardised data model as outlined above, then what do we gain from basing that data model on international standards?
Surely, we know our own data landscape and audience better than anyone else? We can define our own standard that precisely meets those needs.
One benefit of basing work on established standards, is that you are effectively gaining access to expertise from around the world. Standards developed under the auspices of established standards bodies go through a rigorous process of development, challenge, review and approval, often involving experts in the field who typically contribute their time and effort to develop standards that crystallise the collective knowledge and experience of the contributors.
That’s not to say that international standards are always “correct” or perfectly crafted. Another benefit of aligning your work with such standards however is that they are under a constant process of review and improvement. The update cycle for an international standard is generally five years, so there is every chance that imperfections in a standard will be identified and improved in the next revision. There may even be an opportunity to get involved in the development of that next revision to ensure that your views are represented. So, adopting international standards gives some level of assurance that developments in the field will be incorporated through the revision cycle that is part of standards development.
Most standards are not overly prescriptive and allow specialisations to be incorporated while still conforming to the core requirements. This allows specialised aspects of your use case and jurisdiction to be incorporated while directly adopting the elements of the international standard that are universally applicable.
Finally, and possibly most important in the context of developing a data model for a particular use case but in the broader context of a digital ecosystem, adopting established standards is the best way to enable integration and interoperability between different systems in that ecosystem, provided that all systems are mandated to respect the same set of common standards.
So, was it worth it?
Developing the NUAR harmonised data model, and co-editing the international MUDDI standard, has been amongst the most challenging but rewarding experiences of my career. Publishing it as an open resource and seeing other organisations and initiatives around the world picking it up and reshaping if for their own purposes is even more satisfying.
In many ways though it would have been easier to just take the data as it came and apply light touch validation to it. I hope I have convincingly argued why this approach would have been unsatisfactory and indeed, in my view, would have made delivery of the NUAR service impossible.
The benefits of adopting a harmonised data model conforming to an international standard are clear, both when considering the immediate practical challenges of ingesting and combining data from hundreds of providers, and also when considering the future opportunities for improving the information we collect and share about the subsurface.
I will talk about some of these opportunities in my next article.