The lifecycle of reference set work spans from the point where there is an initial idea related to a specific clinical information requirement based on SNOMED CT through reference set development to long term maintenance. At each stage it is essential that a defined process is followed which ensures the quality and validity of the product is maintained. The following figure provides an overview of the steps that are relevant to any project that either builds its own reference set, or adapts reference sets authored from elsewhere. The general process is similar to the process used when developing other information artefacts, and as for many development processes it is essential to be clear and specific about the requirements to be met, and to ensure that these requirements drive the design and implementation of the artefact. Related to reference set development this means that the process is not just about creating a reference set, but more about how to address the requirements, and this will often mean a combination of development of different reference sets.

The overview provided in following diagram is not intended to imply that a waterfall methodology should be used to develop a reference set. It may be delivered using agile or iterative project methodologies, with the reference set and supporting project documentation evolving as the project develops.

The initial focus of the reference set development work is to establish a clear set of initial requirements, which may require further elaboration at a later date. The requirements should include an assessment of the release requirements for the reference set from a user perspective.

Defining the scope of the reference set involves answering a range of different questions related to each step of the development process. Following questions are relevant to consider when defining the scope of a reference set. Additional questions will probably be relevant for a specific development process.

• What is the main purpose of the reference set?

• Who are the users of the reference set?

• What is the scope of content of the reference set?

• Who is responsible for developing and maintaining the reference set?

• Should we develop a new reference set or does any existing reference set meet our requirements?

Once the scope of the reference set and the requirements has been identified, it is essential that an owner of the reference set is clearly identified (this may be an individual or a specific group). Furthermore, the requirements must be clearly documented, particularly the scope, purpose and use case(-s).

The owners of the reference set must then assess what content to be included in the reference set. This process may involve evaluating existing reference sets, or assessing what parts within the International Release of SNOMED CT (and potentially National and Affiliate Editions) will meet the stated requirements. At this stage the requirements for ensuring the content of the subset is quality assured should also be documented.

Careful planning in the initial stage is crucial for successful completion of the reference set development and onwards use of the reference set. Following considerations should be included in any reference set development process.

Table: Considerations important when designing the reference set and planning the development process

As part of the design and planning of a new reference set may it may be enumerative to look at reference sets developed by other organisations. Existing reference sets may be useful for inspiration or they may be sufficient for either adoption or adaption. In either case exploring existing reference sets and evaluate whether these reference sets meet the requirements identified for this particular reference set should be considered.

In order to be able to adopt or adapt an existing reference set, you need to know what reference sets already exist. As there is no common library of SNOMED CT reference sets an authoring organization should apply alternative search strategies to explore what reference sets has already been developed.

Table: Overview of potential sources of reference sets that can be approached when seeking for existing reference sets

To determine whether a reference set is useful either to adopt or to build from, there is a range of parameters which can be assessed to determine whether a reference set is fit for the purpose or requirements that the developing organization have. The following diagram illustrates some of the central aspects that must be assessed when evaluating an existing reference set.

Firstly, the basic parameters of the existing reference set should be compared with the actual needs related to use of the new reference set. The following table provides an overview of some of the basic parameters, which should be assessed when starting to evaluate an existing reference set. Having assessed these basic parameters, it will be possible to determine whether an existing reference set is sufficient for further inspection, or whether it should be rejected as candidate source for a new reference set.

Table: Overview of basic reference set parameters relevant for evaluating reference sets

Having compared the more basic parameters it is also important to Inspect the specific content of the existing reference set to determine the appropriateness of this to fulfill the requirements that an organization have. The following table provides an overview of typical parameters to be inspected when evaluating the content of an existing reference set.

Table: Overview of typical aspects, which should be inspected when evaluating an existing reference set

In the stage of planning and designing a reference set, the use of the reference set should be carefully analyzed. Specifically, it should be analyzed how the reference set is going to function with surrounding information models or software artefacts.

Bounded reference sets are designed to be used together with a specific information model. For example, if a reference set is developed as a value set for recording the smoking status in a specific software system, like shown in the example illustrated below.

When a reference set is bound to a specific information model, it is important to carefully consider how the binding affect the reference set members. So, for bounded reference sets it is important to clearly specify the relationship between the reference set and the associated information model to guide users, and to ensure correct interpretation of data, when data is subsequently retrieved for purposes such as display, analytics and communication.

Table: Questions to consider for bounded reference sets

Unbounded reference sets mean that the reference set is designed to be applicable to multiple use cases, organisations or systems. An example is the SNOMED CT to ICD-10 map, which is released with the International Edition of SNOMED CT, and hence, available for use by any of the Members and affiliates. It may also be reference sets developed in a Member country to be used by all cardiovascular surgery departments in that country for reporting the procedures done and the procedure outcome. Such reference sets is distributed together with a National Edition of SNOMED CT.

For unbounded reference sets it is important for consistent and proper use to clearly specify the purpose and use of the reference set. Therefore, written specification and guidelines should be developed and distributed together with the reference set, to ensure that users of the reference set know how to implement and use the reference set.

Reference Sets can be distributed as part of the International Release, as part of a National Edition or as part of an Affiliate Edition, and the way the reference set is distributed will typically depend on how the Edition is distributed. However, in some situations, reference sets may be distributed independently.

Distributing reference sets usually involves performing any additional quality checks required to ensure that the data can be exported correctly, and then exporting the reference set from the terminology management tool. Finally, the reference set is distributed to the users, for example as release files accessed from an online library using a terminology server API.

Distribution Format

The standard format for distributing SNOMED CT reference sets are the reference set format, as described in Common Reference Set Format. For example, an extensional definition of a subsets of components is represented as a simple reference set, and the standard format for distributing intensional definitions of concept subsets is as a query specification reference set. As part of the SNOMED CT release format, reference sets support unique identification, versioning and recognition of dependencies.

Using the reference set format for distributing SNOMED CT derivatives is beneficial from the view of maintenance, because the versioning attributes is identical to the versioning attributes used for SNOMED CT components. This means that the checks that need to be done as part of a regular release cycle is easier to manage compared to having a range of different versioning mechanisms to adjust to.

Other distribution formats may be used where necessary. For example to comply with requirements for representation of value sets including codes from other code systems. However, care should be taken to ensure that distributed subsets are uniquely identified and versioned. Subsets need to be accessed, selected and where appropriate bound to information models using standard approaches.

Some subsets will be distributed biannually as part of the International Release. However, some others may not require this release schedule. Release processes must include a period of time dedicated to subset manipulation, following the 'freezing' of SNOMED CT itself. This is the period where all the changes has been made to the core components of SNOMED CT. Timing of release and distribution of specific subsets will be dictated by individual use case-specific requirements. These should be identified and documented initially as part of the subset development process. These may change over time based on user feedback.

With time, the set of concepts, or other components, referenced in a reference set will need to change, either because of changes in SNOMED CT, changes in clinical knowledge, or changes dictated by errors or omissions in the original reference set. The original reference set will have been given an identifier that is used to reference the reference set in models or other entities that use that particular reference set. When a change is required that maintains the original intent but updates the content, the change is considered a new version of the reference set. Versioning maintains the reference set identifier and additional reference set attributes.

Following topics will be presented in relation to reference set maintenance:

• Reference Set Management Considerations

• Managing Reference Set Changes

When developing or evaluating a change management process it is important to take different factors into account. Some of these factors are introduced in the table below.

Table: Change management considerations

Maintaining reference sets starts as soon as the reference set is specified. The maintenance process involves ensuring that the content of the reference set refers to active concepts in the release upon which the reference set depends. It also involves adding new content to the reference set, for example when additions to the dependee Release fall within the scope of the reference set specification. The alignment of a reference set with the International Release will be undertaken during the international release process, at which time a change report will be created for circulation, providing details of the changes made, in a format that can be circulated virtually to subject matter experts.

The tasks related to reference set changes will typically involve substituting a prior version of a reference set with the more recent version. For example, if a reference set represents a value set which is used in a data entry interface, the reference set may simply be replaced by the updated version of that reference set. On the other hand, if the reference set is of a more complex nature, for example, if it represents a map between SNOMED CT and a classification, and is used for quality monitoring, research, reimbursement etc., it might be necessary to go into more detail about the implications of the change. Regardless of the type of reference set change (addition, inactivation or change) it will always have implications.

If a reference set is added data entry protocols should be updated to link to this reference set where this is found appropriate. For example, if a reference set has been developed to function as a value set for cardiovascular diseases, data entry protocols should link to the reference set in the data entry protocols used to capture cardiovascular diseases. The nature of the binding depends on the underlying information model of the protocol. Additionally, it is important to update the storage models appropriately. See, to learn about this topic. It may also be necessary to develop queries to generate reports or views based on the new reference set data. Alternatively, an added reference set may include rules to enable decision support, which must be integrated. It may also be necessary to create or update links between reference set members and communication protocols.

A reference set may be inactivated in situations where an organization is no longer able or prepared to maintain it.

Inactivation of an entire reference set is done by inactivating the concept that identifies the reference set, and inactivating the relationship used to place the reference set in the subtype hierarchy.

Reference sets that are being actively used should not be inactivated without giving prior notice to the users of the reference set, to allow remedial steps to be taken.

Any function using a reference set that is no longer maintained will need updating. For example, if a reference set is inactivated, which is used for data entry, it is important to determine how the inactivation is managed in the system. Thus, it should be clarified whether a new reference set should be defined or whether the inactivated reference set should be added in a local extension. It may also be that a replacement reference set has already been defined.

The types of changes that can occur in a reference set include:

• Changes to reference set members

• Addition of reference set members

• Inactivation of reference set members

The effect of changes to reference set members depends on the type of reference set and the way the reference set is used. Generally, it is important to assess the extent to which the reference set changes affect the situation in which the reference set is used, such as:

• Data entry. Changes to reference set members may require changing existing data entry protocols.

• Data storage. Changes to reference set members may require migrating or processing pre-existing data in a particular way to ensure consistency.

• Data retrieval. Changes to reference set members may require revising existing queries to take account of the changes.

• Knowledge linkage. Changes to reference set members may require updating bindings to knowledge resources or decision support rules.

One way of determining the changes to a reference set is by creating and comparing the following two views of the reference set:

• SNAPSHOT view of the previous reference set release. This view contains one version of every member released up to the time of the snapshot. The version of each member contained in a snapshot is the most recent version of that member at the time of the snapshot.

• DELTA view of the new reference set release. This view contains only reference set member versions created since the previous reference set release. Each reference set member version in the DELTA view represents either a new reference set member or a change to an existing member.

These two views make it possible to automatically identify whether a reference set member has been created, changed, reactivated or inactivated. The table below illustrates how to interpret the 'active' attribute, when comparing the delta view of the new release of the reference set with the snapshot view of the previous release of the reference set.

As clinical practice evolves and adapts to meet the needs of healthcare advances, SNOMED CT content is updated to address these changes. Reference sets may also need to be updated to reflect content changes and to enable clinicians to adequately represent new and emerging techniques and practices. It is therefore important to establish processes for end users and other stakeholders to submit requests for changes. Related to reference sets the types of changes that may be required include:

• Addition of new reference set members

• Removal of reference set members

• Changes to existing reference set members

When new members are requested for inclusion in a reference set, it should be determined whether any existing components in SNOMED CT are sufficient, or whether it requires a new component to be added - either in the International Edition or as part of an Extension.

To remove a referenced component from a reference set, the relevant reference set member is inactivated. This is done by adding a new row to the reference set in which the value of the active column is set to false. The SNOMED CT versioning mechanism tracks the full history of additions, changes and inactivations because each row in the reference set contains a effective time column indicating when the change became effective. This is the same versioning mechanism with is used to track addition, inactivation and modification of SNOMED CT components.

Changes to reference set members can vary, dependent on the refset type and the situation. Examples of changes include:

• Changing the order assigned to a member of an ordered reference set

• Changing the annotation associated to a member of an annotation reference set

• Changing a description from being acceptable to preferred in a language reference set

• Updating the query for an intensional subset definition represented by a member of a query specification reference set

The type of service implemented to support change requests will depend on the amount of reference sets, the number of users, the organisation etc.. Smaller organisations with relatively a few end-users may apply a simple email service for proposing any changes to the refset. Other institutions may implement dedicated systems that track each change request and support proper management of those.

Authoring and managing reference sets require some level of tooling support. Broadly speaking, two types of tooling support exists:

1. Combination of browsing functionality and simple spreadsheets or database system

2. Dedicated reference set management tools with extended features

The extend of sophistication of the tool required depends on various factors, and the combination of those factors:

• Type and size of reference set

  • Simple reference sets with few members

  • Large reference sets of various types

• Use of reference set

Relevant functionalities for reference set creation and management include:

• Creation of both extensional and intensional reference sets

  • Services for browsing and displaying SNOMED CT content for manually selecting reference set members

  • Services for manual and intensional expansion of reference set (inclusion of reference set members)

  • Services for manual and intensional exclusion of reference set members

Generally, it is not recommended to use a spreadsheet approach for managing reference sets, because there is a dependency between the reference set and the core content of SNOMED CT. The SNOMED CT release files are important for interpreting the components referenced by the reference set, for example:

• Description file: Access to the descriptions are required to display the terms of the components referenced by the reference set.

• Concept file: Access to the concept file is required to assess whether the components referenced by the reference set are active or inactive

However, in some situations spreadsheets (supported by a browsing facility) may be feasible, for example,

• If the reference set only consists of a few members

• If the reference set is only used locally

• If only one person is responsible for managing the reference set

If a spreadsheet approach is chosen it is recommended to create a of that reference set to support interpretation of the identifiers used in the reference set.

When a reference set has been developed and tested it should be prepared for use by implementing it in the environment where it is going to function. I.e. the implementation process and the tasks involved will vary dependent on the setting.

Often, implementation of a reference set will mean integration with one or more software artefacts, to enable data capture, represented by appropriate reference set members.

An important part of this process is proper binding between the information model and the reference set to ensure effective integration with data entry and storage functions. Many situations will also involve implementation of two or more interdependent reference sets, which require their dependencies to be considered to reach the most optimal implementation.

In this guide we highlight two aspects that are important when implementing SNOMED CT reference sets, i.e. supporting implementation with proper implementation guidance and considerations related to optimizing the reference set for a particular implementation.

Implementation Guidance

The tasks involved with implementing a reference set will depend on the situation and the technical setting where the reference set is going to function. To support successful implementation, and to ensure that the reference set is used as intended it is important with precise and clear instructions for the people responsible for implementing the reference set, i.e. the people preparing the reference set for routine use within a particular setting.

Implementation guidance should, as a minimum, include instructions on:

• The overall purpose and intended use of the reference set

• The reference set design, i.e. attributes and data types

• Dependencies to terminology artifacts, such as

  • other reference sets

Additionally, it is valuable to include instructions on how to test the implementation, i.e. exemplar test cases with information on expected outcome given a specific action.

As part of an implementation process it is relevant to consider any optimization that could be done to meet requirements for implementation or result in a more effective and/or efficient use at runtime. Because, even though reference sets are distributed in separate files and represented according to the reference set file format, it may be useful to transform these reference sets files into another structure that is more appropriate for the specific use case.

Optimizations that could be considered include:

• Controlled redundancy. For some matters it may be useful to introduce some level of redundancy, e.g. by joining reference sets or release files together so that the data is more efficient to access at runtime

  • e.g. combining a simple reference set of concepts with the description file, so that the term preferred for display can be retrieved directly from the subset file.

  • e.g. combining the description file with a language refset, so that the term is in the same row as the acceptability of the description, and the concept id that it describes.

In this part topics related to the actual reference set development process are presented.

Prerequisites for Creating a Reference Set

As part of developing a new reference set it should be created and identified as part of the Edition that it belongs to.

When creating a new reference set, the organization developing the reference set will need access to a namespace in order to generate SCTIds. Within their namespace, a moduleId concept (with an FSN and Preferred Term) should be added, placed in the 900000000000443000 | Module (core metadata concept)| subhierarchy within the metadata, for each authoring organization.

To specify any reference set it is necessary to identify the reference set by creating a concept and name the reference set by associated descriptions. The concepts should be a subtype of the relevant reference set type concept in the foundation metadata hierarchy. This could for example be as a descendant of the 446609009 | Simple type reference set (foundation metadata concept)| , or the 900000000000512005 | Query specification type reference set (foundation metadata concept)| . The latter is used for intensional reference set definitions. Following table shows the steps to follow when creating a new reference set.

Authoring Reference Sets

Authoring reference sets in an extension may involve creating a new reference set, inactivating a reference set or modifying a reference set. Additionally, reference set members may also be added, modified or inactivated in an extension.

The sections that follow will examine the purpose, principles and process for each of these authoring tasks.

Authoring Reference Sets

Authoring Reference Set Members

When creating a new reference set it is important that the reference set metadata concept (which identifies and names the reference set) and the associated reference set members belongs to a module of the reference set producer. I.e. if a reference set is created as part of an extension, the reference set metadata concept should belong to a module within the namespace of the extension producer.

All reference sets follow a logical model. Each reference set type is represented by a specific data structure which enables a specific functionality. SNOMED International specifies a set of reference set types, which can be used to support a set of terminology management purposes. SNOMED CT Members (National Release Centers) and Affiliates may also create reference sets to assist with localization and effective implementation of SNOMED CT. In general, an extension producer can create a reference set within their extension, but the reference set type should be defined in either

• • The producers extension

  • The International Edition

  • Other extensions (which the producers extension modules are dependent on)

For more information about the different reference set types and their data structures, please refer to the .

Please note that for all reference sets, there should be guidance on what is possible and what is not possible. For some International reference sets, it is acceptable and even required to add extension rows to the International reference set. For example, the Module Dependency Refset, the MRCM Module Scope Refset, and Language Refset. This is not necessarily applicable to all reference sets. Over time, further guidance in this area will be developed.

The table below provides a summary of the process to follow when creating a new reference set.

Define the reference set in the metadata hierarchy

The following steps should be taken for creating the reference set concept:

Define the reference set Attributes within the metadata hierarchy

Add new concepts for each of the reference set member attributes, if necessary. If the reference set attributes describing the pattern are adequate to describe the reference set's attributes, then these can be used instead. You may add new concepts for some of the attributes, and reuse existing concepts for other attributes, if you wish.

Following steps should be taken for each attribute that you wish to create:

Create the Descriptor for the reference set

If the reference set does not follow an existing reference set pattern, the additional attributes specific for this customized reference set should be included in the .

Add members to the reference set

Reference set members are added to the reference set, which includes specifying the attribute values for each reference set member, see .

Principles for modifying reference sets include:

• Reference sets can be modified by changing the value of the active attribute or the value of the moduleId attribute, i.e. for activating or inactivating the reference set, or moving the reference set to another module (a module on which the current module depends).

  • The reference set metadata concept is always primitive and the concept can therefore not be changed from primitive to fully defined, or from fully defined to primitive, i.e. the value of the definitionStatusId attribute should be primitive for all versions of the reference set

• The reference set type should not be changed, i.e. the |is a| relationship, linking the reference set to the appropriate reference set type concept should not be changed

Modifying a reference set created within the extension module

The process for modifying the reference set concept, is similar to the general process for modifying concepts in SNOMED CT. Therefore, please refer to for further guidance.

When the owner of a reference set wants to end support for the reference set, this should be indicated by making the changes described in the process section below.

In the situation where the reference set belongs to an extension, the owner of the extension may only make these changes to a reference set that is currently in an extension module for which it is responsible. The exception to that rule is that, in the case where a responsibility for maintenance of a reference set is transferred to another organization, the organization to which responsibility is transferred is required to take some of these steps.

Prior to ending support for a reference set, it is important that the reference set producer has an overview of the extent to which the reference set is used. If a producer and owner of a reference set continues to distribute an unsupported reference set with active members, there is an inherent risk that it will continue to be used. However, deprecation formally inactivates the references set members to mim this possibility.

The table below provides details and considerations on the process of inactivating a reference set.

If the extension producer wants to avoid users from needing to import a deprecated or transferred reference set in future releases, the inactivated reference set may be separated from the main extension release (e.g. it could be in a separate release package, or accessible via a separate service or from a static location). Changes in packaging must be formally notified to users of the extension in advance of the change.

Specifying members of a reference set can be done in different ways. It will depend on the requirements for the reference set what approach is feasible and possible. Please refer to the and sections for detailed instructions on the approaches and methods for identifying the SNOMED CT components that will be referenced by the members of the reference set.

However, each reference set member will be represented in the reference set in accordance with the following principles and process.

Reference set members may be added to reference sets belonging to modules in

• The producers extension

• The International Edition

• Other extensions (which the producers extension modules are dependent on)

Reference set members created in the extension should be created within the module of the extension producer, so that it is possible to distinguish the reference set members created within the producers extension from reference set members created by other organisations (i.e. belonging to other modules).

Even though the individual reference set members belong to the module of the extension producer, the actual components that are referenced by the reference set member, may belong to modules in

• The International Edition

• The extension in which the reference set and its members are produced

• Other extensions (which the producers extension modules are dependent on)

The table below provides a summary of the process to follow when adding members to a reference set.

Principles for modifying reference sets include:

• Reference sets can be modified by

  • Adding or inactivating reference set members. Please refer to the guidance on or .

  • Modifying mutable attribute values of reference set members. To see what attributes are mutable for each reference set type, please refer to the specification of the specific reference set type in the . If the reference set type is a locally defined reference set, please consider any mutability constraints on the individual attributes.

The table below provides a summary of the process to follow when modifying an existing member in a Reference Set.

Table: Modify reference set member

Any inactivation of reference set members should be done in the module of the reference set producer.

The table below provides a summary of the process to follow when removing members from a reference set.

Developing reference sets can be done in different ways, depending on requirements, resources and skills of the organization who wish to develop a reference set. In some situation it may be the best approach to build the reference set without making use of or relying on any previous work for assistance. In other situations, existing reference sets may meet the requirements for another organization that wants to develop a reference set, and in these cases there are different approaches to utilize the existing work. In this section we introduce different general approaches to the creation of SNOMED CT reference sets.

One way to create a reference set is to specify the content from SNOMED CT to be included in the reference set without looking at any existing SNOMED CT reference sets. This approach may be chosen, if no reference set is available that meets the requirements of the developing organization. It may also be because the requirements for the new reference set is so clearly defined or limited in scope that it is easier to create the reference set from scratch.

This could for example be, if a group of orthopedic surgeons are developing a subset of SNOMED CT procedure concepts to be included in a certain pick list of their local electronic health record. The group already know what options should be available in the pick list, so they choose to create their own reference set and add the required component references to this reference set, see figure below.

Depending on the scope of content for the reference set there are different that can be applied for selecting, or defining, reference set members.

Developing a new reference set also requires the developing organization to make changes to the reference set as necessary to meet evolving requirements.

As the experience of the SNOMED CT community grows, it becomes increasingly likely that there is some existing work that will provide a useful starting point, or that can be used to validate the work that you are doing. In this case, adopting an existing reference set may be a useful approach to take for acquiring a reference set.

Adopting an existing reference set is when the adopting organization/project use the existing reference set and future updates of that reference set as provided. This means, that the adopting organization commits to adopting any changes that are made to the source subset over time.

To be able to adopt and existing reference set it is a prerequisite that the existing reference set is part of a module that is included in the SNOMED CT Edition used by the adopting organization.

It is also a prerequisite that those maintaining the source subset publish changes in a predictable fashion, and the adopting project has processes in place to manage this adoption.

Adopting an existing reference set is an attractive solution in situations where the source subset is already being used within the communicating community, and so avoids the need for mappings to be maintained, or other strategies developed to deal with differences.

The drawback is that if the adopting project requires changes to the contents of the subset it does not have any mechanism for requesting such changes or making them happen.

In the situation where an existing reference set meets the requirements of an organization who wish to use a SNOMED CT reference set, it may be a solution to copy this reference set instead of adopting the reference set. Copying a reference set is a useful approach in the situation where the existing reference set is part of a Module which is not included in the SNOMED CT Edition that the copying organization use.

Copying an existing reference set means create a new reference set with members referencing the same components as the existing reference set.

It is important to have a clear strategy for maintenance of a copied reference set. When an organization chooses to copy an existing reference set they need to make changes to the reference set as necessary to meet their evolving requirements. Whether it is the copying organization or the authoring organization who is responsible for adding or inactivating content to the existing reference set depends on the agreement between the involved parties. It also depends on whether the authors of the existing reference set have established a process, which deals with requests for changes. In either situation the copying organization will need to apply the changes made to any new version of the original reference set to the copied reference set.

Another approach to develop a reference set, is to adapt an existing reference set. This is where an existing reference set is used as a source of inspiration for the developing organization. It may also be, that the existing reference set almost meet the requirements of the adapting organization, but some modifications are needed. The adapting organization may wish to add more content to the existing reference set, or there may be content in the existing reference set which should be excluded from the new reference set.

Like copying reference sets, adapting a reference set will include the creation of a new reference set which is authored and maintained by the adapting organization. The members of the new reference set may then reference components that are also referenced in the existing reference set, they may exclude references that are in the existing reference set, or they may add other component references to meet the requirements of the adapting organization.

When developing simple reference sets, there are different methods which can be applied to determine what concepts to be included in the subset. The two overall approaches to defining the reference set members are manual and definition.

Identifying reference set members by manual enumeration is useful when the number of members required is limited or when migrating from a well-specified domain. The source information may come from a variety of sources, such as:

• Subsets of concepts from other code systems

• Legacy codes (when migrating from a system that do not use SNOMED CT)

• Clinical guidelines and other knowledge resources

Manual enumeration is also useful when the reference set should be stable over time, i.e when addition of concepts, in subsequent releases of SNOMED CT, is not required to be reflected in the reference set.

Identifying reference set members intensionally is useful for creating subset of concepts who share a set of common characteristics, such as:

• Being a descendant of the same focus concept

• Sharing one or more attribute relationships

Intensional definition of reference set members is also useful when changes to concepts, in subsequent releases of SNOMED CT, is required to be reflected in the reference set.

Ensuring the quality of a reference set is important for the successful use of SNOMED CT. It is crucial that the content of the reference set references the concepts which represent the meaning of the data it is to be linked to. In addition, it is important that the human interpretation of the concepts referred to aligns with the logical definition provided by SNOMED CT. Even when using reference sets for less patient safety critical purposes, the quality of the reference set needs to be sufficient to be trusted to serve its purpose.

Reviewing a reference set is important throughout the reference set development process, however at least three types of validation should be emphasized, see illustration below.

• Design review: The overall objective with this review is to verify whether the reference set design meets the requirements.

• Content review: The overall objective with this review is to verify whether the selected reference set members are sufficient for the context where the reference set is to be used.

• Test: The overall objective with testing the reference set is to validate the reference set in the context where it is to be used. Testing is done to assure that the reference set meets the needs of the involved stakeholders.

Reviewing a reference set before testing it in the context of its usage is important in order to correct the most distinctive errors, such as content gaps or disagreements about selection of concepts. Additionally, reviewing a reference set is important, because it is much easier to make major adjustments as early as possible. In addition, the adaptability of the reference set increases the higher the quality of the reference set, according to the perspective of the subset users. The level of quality assurance needed will of course be dependent on the practical use case of the reference set.

The checks that should be done as part of the review process include:

1. Check that all concept ids are current (that there is no references to inactive concepts)

2. Check that all concepts included in the reference set are appropriate to the use case

3. Check that the terms used to describe these concept ids are appropriate

4. Check that there are no gaps in content. I.e. concepts that should be included in the reference set that aren't

The people involved with reviewing the reference set should be by a combination of domain experts and SNOMED CT experts.

• Domain experts: There may be different types of domain experts, but at least two types of domain experts can be specified :

  • Clinical domain experts: Persons who have knowledge about the use of the reference set and knowledge about the context where the subset is going to function. Often, clinical domain experts will be clinical users who have the sufficient level of clinical knowledge, to evaluate whether the scope and the content of the reference set is sufficient.

  • Technical domain experts: Persons who have knowledge about the technical environment where the reference set is going to be implemented. technical experts should have knowledge about the various models to which this reference set should be bound. Often, these experts will have a technical background, for example software architects, programmers, IT-managers. These experts are especially important at an early stage of the reference set development process, to review and validate the design of the reference set in relation to the requirements set out by the technical infrastructure.

SNOMED International does not recommend ONE specific approach for review of reference sets, but blinded approaches are typically a feasible approach to achieve high quality reference set. Alternatively, or additionally, a combination of review methods can be recommendable to ensure a reliable and usable review process. Ideally, reference sets should be reviewed iteratively until the reviewers are satisfied. Examples of the approaches to take are illustrated in the following table.

Reference set review approaches

In addition to the abstract assurance of the subset in isolation, there may be a requirement for a level of testing to be undertaken in healthcare systems, and tested under the exact circumstances of intended use. This may be undertaken by releasing a technology preview targeted at specific bodies for feedback. An impact assessment and, most importantly, safety testing will need to take place upon deployment of the subset into systems, particularly where significant changes have taken place.