Govern · Topic 4 of 6
Duplicate CIs and De-duplication
CIS-DF duplicate CIs: why duplicates are created, how the IRE raises de-duplication tasks, and how the remediation wizard and templates resolve them.
- 5 min
- Includes a diagram
- Duplicates come from data that was not matched to an existing CI, usually by bypassing the IRE or weak identifiers.
- The IRE groups duplicate sets into de-duplication tasks in reconcile_duplicate_task.
- The wizard order: main CI, approach, attributes, relationships and related items, duplicate action, review.
What it is
A duplicate CI is two or more CI records that describe the same real thing, such as one physical server recorded twice. Duplicates split incident and change history across records, break relationships and lower the correctness score.
ServiceNow handles them in two stages. The Identification and Reconciliation Engine (IRE) detects duplicates and groups them into de-duplication tasks. You then remediate each task with the Duplicate CI Remediator wizard or, in bulk, with de-duplication templates in CMDB Workspace.
How it works
Why duplicates are created
| Cause | Example |
|---|---|
| Data bypasses the IRE | A transform map or script inserts CIs directly instead of through the IRE |
| Weak identification rules | The identifier uses a name that differs between sources |
| Identifier values differ | One source sends a serial number with a prefix, another without |
| Manual creation | A user creates a CI that a source later reports again |
| Data loaded before rules were tuned | An early bulk load created records the current rules would have matched |
The pattern is the same each time: the incoming data was not matched to the existing CI, so a new one was created. Prevention is the Ingest chapter's job: send all sources through the IRE with sound identification rules. Remediation is what this page covers.
How de-duplication tasks appear
When the IRE finds that more than one existing CI matches the same identifier, it groups those CIs into a de-duplication task, stored in the Remediate Duplicate Task [reconcile_duplicate_task] table. One task covers one set of duplicates.
The tasks are listed under Configuration Identification/Reconciliation, De-duplication Tasks, and on the de-duplication dashboard in CMDB Workspace. The Duplicate metric in CMDB Health and the "Handled Duplicate CIs" indicator on the Data Foundations Dashboard both show the scale of the problem.
The Duplicate CI Remediator wizard
Open the task and select Remediate, then choose Duplicate CI Remediator. The wizard walks through these steps:
- Select main CI: pick the record that survives. The system recommends candidates using factors such as most relationships, most related items, and most recently discovered or updated.
- Choose the approach: remediate manually, or choose Use Main CI, which keeps its attribute values and merges relationships and the default related items.
- Merge attribute values: for each conflicting attribute, keep the main CI's value or take a duplicate's value. System, discovery and read only fields are excluded.
- Merge relationships and related items: choose whether to move all relationships to the main CI, and which related items to merge; the default items are preselected.
- Decide what happens to duplicates: set attributes to custom values (recommended), or delete them.
- Review and confirm: nothing changes until you confirm; the task then closes and processing runs in the background.
The recommended duplicate action keeps the duplicate records but marks them invalid, for example by setting the operational status to retired. Their duplicate of field points to the main CI, and they are left out of future de-duplication tasks. Add identification inclusion rules that exclude them, or new tasks may still appear.
Remediation needs sn_cmdb_admin, cmdb_dedup_admin or itil_admin; sn_cmdb_editor or itil gives read access.
Templates for bulk remediation
The wizard handles one task at a time. When hundreds of tasks share the same pattern, use de-duplication templates in CMDB Workspace. A template stores the remediation settings for a class and can be kept in a template library and applied to many tasks at once.
| Situation | Best fit |
|---|---|
| A few tasks, each needing judgement | Duplicate CI Remediator wizard |
| Many tasks with the same rules for a class | De-duplication template |
| Your own remediation logic | Custom remediation workflow (CMDB remediation rule) |
Templates that run on a schedule need their scheduler job running; otherwise the tasks stay in the ready state.
How the exam asks about this
- "After a new import, many servers appear twice." The import bypassed the IRE or the identifiers do not match; fix the source, then remediate the tasks.
- "An analyst must decide which record keeps its relationships and incident history." Use the Duplicate CI Remediator wizard and select the main CI.
- "Five hundred de-duplication tasks for the same class follow one pattern." Use a de-duplication template from CMDB Workspace.
What to remember for the exam
- Duplicates come from data that was not matched to an existing CI, usually by bypassing the IRE or weak identifiers.
- The IRE groups duplicate sets into de-duplication tasks in reconcile_duplicate_task.
- The wizard order: main CI, approach, attributes, relationships and related items, duplicate action, review.
- The recommended duplicate action marks duplicates with custom values rather than deleting them.
- Templates in CMDB Workspace handle bulk remediation for a class.
- sn_cmdb_admin, cmdb_dedup_admin or itil_admin can remediate.
Remediating duplicates without fixing their cause brings them back. Correct the source or identification rule as well as the records.
Common mistakes
Deleting duplicates straight away. Deleting loses history the main CI may need. The wizard's recommended option marks duplicates and links them to the main CI instead.
Using the wizard for a large backlog. One task at a time is slow when the pattern repeats. A template applies the same settings across many tasks.
Blaming CMDB Health for duplicates. Health only reports them. The IRE detects them and creates the tasks; the source data created them.
Check yourself
What creates de-duplication tasks?
The IRE, when it finds more than one existing CI matching the same identifier, groups those CIs into a task.
What is the first step of the Duplicate CI Remediator wizard?
Selecting the main CI, the record that survives and receives merged data.
When should you use a de-duplication template instead of the wizard?
When many tasks for a class can be remediated with the same settings, so they can be handled in bulk.
Topic 4 of 6 in Govern
Test yourself on Govern
The full ServiceNow CIS-DF question bank has 769 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by ServiceNow. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.