This service definition and ASN.1 specification has been approved by the ZIG. An amendment (Z39.50-1995 Amendment 2), has been approved, adopting this definition into the Z39.50 Service Definition and Protocol.

The Z39.50 Duplicate Detection Service

Parameters

The following notation is used in the "Origin Request" and "Target Response" column:

• [0,1] means parameter is optional, not repeatable; i.e. zero or one.
• 0+ means parameter is optional, repeatable; i.e. zero or more.
• 1 means parameter is mandatory, not repeatable; i.e exactly one.
• 1+ means parameter is mandatory, repeatable; i.e. one or more.

Input Result Set Id and Output Result Set Name
The origin identifies one or more transient result sets belonging to the current Z-association. The target is to logically merge the sets (removing duplicates and ordering equivalence classes according to the parameters below) into a single result set, specified by the parameter Output Result Set Name.

Applicable Portion of Record
The origin may specify what portion of the record is subject to matching (for example, one or more fields) for purposes of duplicate-detection. If this parameter is omitted, the target decides what portion of the record is subject to matching.

Duplicate-detection Criterion
For modeling purposes, a temporary, intermediate result set (not the output result set) is assumed to be created, which includes all of the result set items from all of the input result sets (including duplicate result set items). The target applies duplicate-detection criteria supplied in this parameter (or if the origin omits this parameter, the target applies whatever duplicate detection criteria it chooses) to partition the intermediate result set into one or more equivalence classes where two result set items are considered equivalent if they are duplicate. That is, the partitioning has the following properties:

• Every result set item from one of the input result sets is in exactly one class.
• Any two result set items are in the same class if and only of they are duplicates.

The origin may specify one or more criteria for detecting duplicates. These include the following (the list is subject to extension):

1. level of match
   If this criterion is included, the origin specifies a level of match in terms of a percentage. For example, fingerprints might be duplicates based on a 60% match; 100% might mean that records are duplicates only if they are identical.
2. Case sensitive
3. Punctuation sensitive
4. Regular expression
   If this criterion is included the origin supplies a regular expression to govern matching.
5. result-set duplicates
   (Two result set items are result-set duplicates if they point to the same database record.)

1. Clusters
   The output result set is to contain one item for each equivalence class. For each equivalence class, create a result set item for the representative record only and maintain duplicates as metadata. (Records may subsequently be presented either as (a) representative record with duplicates attached as metadata, using, for example, GRS; or (b) as a cluster record, using an appropriate cluster syntax.)
2. Individual Entries
   Create individual result set items for representative records as well as duplicates that are to be retained (according to Retention Criterion). Order the output result set such that records within an equivalence are grouped together. The parameter Sort Criterion may be supplied, to indicate how the records within a class are to be ordered.

Retention Criterion
The origin specifies one or more criteria for how records are to be selected for inclusion in or exclusion from each equivalence class. These include the following (the list is subject to extension):

1. Number of entries
   If this criterion is selected, the origin supplies a number, N>0, meaning retain (up to) N entries in each equivalence class. N=1 means retain the representative record only. This value may be used in combination with (3) and/or (4), but not (2).
2. Percent of entries
   If this criterion is selected, the origin supplies a percentage, xx, meaning retain xx percent of the entries in each equivalence class. xx=100 means retain all entries. This value may be used in combination with (3) and/or (4), but not (1).
3. Duplicates only
   Discard representative record. This value should not be specified unless the value of parameter Clustering is 'Individual Entries'. This value may be used in combination with (1) or (2), and/or (4).
4. Discard result-set duplicates
   This value may be used in combination with (1) or (2), and/or (3). If used with (1) or (2) the result-set duplicates should be discarded first (before entries are selected).

This parameter will affect the ordering of result set items only within an equivalence class (it does not affect the ordering of equivalence classes). If the value of parameter Clustering is 'Clusters' then this parameter will have no effect whatever on the result set order (though it may be supplied anyway, to govern the selection of representative records as well as the order in which duplicates are presented within a single cluster record).

More than a single sort criterion may be supplied; if so, the order in which they are supplied is from major to minor, and only the first criterion supplied is used to govern selection of a representative record. The sort criteria include the following (the list is subject to extension):

1. Most Comprehensive
   Select the longest (more comprehensive) record as the representative record; order duplicates within an equivalence class by descending comprehensiveness.
2. Least Comprehensive
   Select the shortest (least comprehensive) record as the representative record; order duplicates within an equivalence class by ascending comprehensiveness.
3. Most Recent
   Select the most recent record as the representative record; order duplicates within an equivalence class by ascending age.
4. Oldest
   Select the oldest record as the representative record; order duplicates within an equivalence class by descending age.
5. Least Cost
   Select the least expensive record as the representative record; order duplicates within an equivalence class by ascending cost.
6. Preferred Database
   Select a record from the most preferred database as the representative record; order duplicates within an equivalence class corresponding to order of preference of databases. When this criterion is supplied the origin includes a list of databases in order of preference.

Result Count
If the value of parameter Status is 'success' then the value of this parameter is the size of the output result set.

Diagnostic
The target may always include one or more diagnostics in the response. If the value of parameter Status is 'failure', at least one diagnostic must be included.

ASN.1 Changes and Additions

Change to Definition of PDU

PDU ::= CHOICE{
 initRequest		[20] IMPLICIT InitializeRequest,
 initResponse		[21] IMPLICIT InitializeResponse,
......
 close			[48] IMPLICIT Close}

Change it to:

PDU ::= CHOICE{
 initRequest  			[20] IMPLICIT InitializeRequest,
 initResponse			[21] IMPLICIT InitializeResponse,
......
 close				[48] IMPLICIT Close,
 duplicateDetectionRequest	[49] IMPLICIT DuplicateDetectionRequest,
 duplicateDetectionResponse	[50] IMPLICIT DuplicateDetectionResponse}



ASN.1 Definition for Duplicate Detection APDUs

DuplicateDetectionRequest ::= SEQUENCE {
   referenceId			    ReferenceId  OPTIONAL,
   inputResultSetIds 		[3] IMPLICIT SEQUENCE OF InternationalString,
   outputResultSetName		[4] IMPLICIT InternationalString,
   applicablePortionOfRecord	[5] IMPLICIT EXTERNAL OPTIONAL,
   duplicateDetectionCriteria	[6] IMPLICIT SEQUENCE OF
                                            DuplicateDetectionCriterion OPTIONAL,
   clustering			[7] IMPLICIT BOOLEAN OPTIONAL,
				  	   -- 'true' means "clustered".
					   -- This parameter may be omitted
					   -- only if retentionCriteria CHOICE is
					   -- 'numberOfEntries' and its value is 1.
   retentionCriteria		[8] IMPLICIT SEQUENCE OF
                                            RetentionCriterion,
   sortCriteria			[9] IMPLICIT SEQUENCE OF
                                            SortCriterion OPTIONAL,
   otherInfo 			    OtherInformation OPTIONAL}

		DuplicateDetectionCriterion ::= CHOICE{
		  	   levelOfMatch		[1] IMPLICIT INTEGER,
				-- a percentage; 1-100.
			   caseSensitive 	[2] IMPLICIT NULL,
			   punctuationSensitive	[3] IMPLICIT NULL,
			   regularExpression	[4] IMPLICIT EXTERNAL,
			   rsDuplicates		[5] IMPLICIT NULL
                 -- values 6-100 reserved for future assignment.
                                                       }

		RetentionCriterion ::= CHOICE{
			  numberOfEntries	[1] IMPLICIT INTEGER,
							--  greater than 0
			  percentOfEntries	[2] IMPLICIT INTEGER,
							-- 1-100,
			  duplicatesOnly 	[3] IMPLICIT NULL,
						 -- should not be chosen
						 -- if clustering is 'true'
			  discardRsDuplicates	[4] IMPLICIT NULL
                 -- values 5-100 reserved for future assignment.
                                                       }


		SortCriterion ::= CHOICE{
			 mostComprehensive	[1] IMPLICIT NULL,
			 leastComprehensive	[2] IMPLICIT NULL,
			 mostRecent		[3] IMPLICIT NULL,
			 oldest			[4] IMPLICIT NULL,
			 leastCost		[5] IMPLICIT NULL,
			 preferredDatabases	[6] IMPLICIT
                                     SEQUENCE OF InternationalString
                 -- values 7-100 reserved for future assignment.
                                                       }


DuplicateDetectionResponse ::= SEQUENCE {
   referenceId				ReferenceId  OPTIONAL,
   status	 		[3]	IMPLICIT INTEGER{
					  success 		(0),
					  failure 		(1)},
   resultSetCount		[4]	IMPLICIT INTEGER OPTIONAL,
   diagnostics			[5]	IMPLICIT SEQUENCE OF DiagRec OPTIONAL,
    otherInfo				OtherInformation OPTIONAL}




Option Bit Assigned to the Duplicate Detection Service

Option bit 18 is assigned to this service. See Z39.50 Option Bits.


 Library
of Congress