Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 1 and 11 are objected to because of the following informalities:
In claims 1 and 11, first paragraph, “each pair of classes of N classes of classes of N classes” is a typographical error in which “classes of N classes” is repeated twice. This phrase should be corrected to “each pair of classes of N classes.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-15 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
In claims 1 and 11, the limitation of “the value of each cell” at the bottom paragraph of the claims lacks antecedent basis, and is therefore indefinite. For purposes of examination, this phrase has been interpreted to be “a value of each cell.”
In claims 1 and 11, the phrase “match counts between the class along the X-axis class at which the cell is located and the class along the Y-axis at which the cell is located” (which appears om the bottom paragraph of the claims) is indefinite for the following reasons:
(1) The limitations of “the class along the X-axis class at which the cell is located” and “the class along the Y-axis at which the cell is located” lack antecedent basis.
(2) Because “match counts” is plural, it is unclear whether “the class” in this context refers to a concept of a plurality of pairs of classes (as generally used in paragraph 77 of the specification) or instead refers to a single pair of class. Alternatively, since the context is “each cell,” the term “match counts” can also be interpreted as being respective of each cell. However, such an interpretation would not be formally expressed correctly because the claim uses the singular term “the value.” Since “the value” is a singular term, “match counts,” if the intended concept is respective for each cell, should also be singular (e.g., a “respective match count”). In other words, given that the claim uses both singular terms (e.g., “the value” and “the class”) and plural terms (“match counts”) that appear to correspond to each other, it is unclear whether there is a one-to-one relationship among all these terms, or instead a one-to-many (i.e., multiple match counts for each “value” and “the class”) relationship.
(3) The term “the X-axis class” lacks antecedent basis, because the claim previously recites “an X-axis” but not “an X-axis class.” The Examiner believes that “class” may be a typographical error.
For purposes of Examination, the above phrase has been interpreted as having the meaning of:
“a respective match count[[s]] between a[[the]] class along the X-axis at which the cell is located and a[[the]] class along the Y-axis at which the cell is located”
This part of the rejection can be overcome by amending the claim in the above manner.
In claims 4-5, the limitations of “wherein q=1” (in claim 4) and “wherein q>1” (in claim 5) have an unclear antecedent under the currently recited claim dependency on claim 1, because “wherein q” refers to an existing “q” variable, but no such variable appears in claim 1. Note that q is instead introduced in claim 2. Based on the context of this claim and claims 14-15, the Examiner believes that the claim dependency of claims 4-5 may have been intended to be claim 2 rather than claim 1. Therefore, for purposes of Examination, these claims have been interpreted to depend from claim 2 rather than claim 1. This part of the rejections can be overcome by amending the claim dependency in this manner.
In claim 6, the limitation of “the N! orderings” lacks antecedent basis, and is therefore indefinite. For purposes of examination, this limitation has been interpreted as “N! orderings.”
In claim 8, “the output of the processing” lacks antecedent basis, and is therefore indefinite. For purposes of examination, this limitation has been interpreted as “an output of the processing.”
The grounds for rejection given for claims 1 and 11 are also applied to their dependent claims 2-10 and 12-15, since claims 2-10 and 12-15 incorporate the deficiencies of claims 1 and 11, respectively.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
1. Claims 1-2, 4-6, and 9-10 are rejected under 35 U.S.C. 103 as being unpatentable over Alsallakh et al. (US 2019/0034557 A1) in view of Hahsler et al., “Getting Things in Order: An Introduction to the R Package seriation,” Journal of Statistical Software, 25(3), 1–34 (2008) (“Hahsler”), Wu et al., “On rearrangement inequalities for multiple sequences,” arXiv:2002.10514v10 [math.CO] 14 Apr 2022 (“Wu”), and University of New Mexico, "Math 401 - Homework #4", 2018, archived 2021 (see Form-892 for full citation) (“UNM.edu”).
As to claim 1, Alsallakh teaches a non-transitory computer readable medium storing instructions readable and executable by an electronic processor to perform a classifier visualization method, [[0028]: “programmed instructions (e.g. the visual analytics program 28 and/or the image classification model 30) stored in non-transitory computer readable storage media (e.g., the memory 16) operatively connected to the controller or processor to manipulate data or to operate one or more components in the visual analytics system 10 to perform the task or function.”] the classifier visualization method comprising: […]
[…] N classes wherein N is equal to or greater than four [As shown in FIG. 3 generally, the number of classes can be numerous, including 4 or more classes. The example of 1000 classes is disclosed in [0025]: “each having a respective class labels from a defined number (e.g., 1000) of possible classes”; [0031]: “The output of the last fully connected layer is fed to a 1000-way soft-max output layer 114, which produces a probability distribution over 1000 different possible class labels.”]
determining an ordering of the N classes that […] [[0054]: “In at least one embodiment, the processor 14 is configured to apply at least one seriation algorithm to reveal block patterns in the confusion matrix and to order the class hierarchy viewer 210 and the confusion matrix 230 accordingly.”]; and
displaying a confusion matrix for a classifier to be visualized, [[0036]: “FIG. 3 shows an overview of an exemplary main user interface 200 of the visual analytics program 28.” [0048]: “FIG. 6 shows a detailed view of the confusion matrix 230 of the main user interface 200.”] the displayed confusion matrix having the N classes ordered in the determined ordering along an X-axis of the confusion matrix and having the N classes ordered in the determined ordering along a Y-axis of the confusion matrix, [[0050]: “The actual classes on the vertical dimension correspond the predefined set of possible classes (e.g., 1000 possible classes) and are ordered along the vertical dimension according the sequential ordering, which was determined based on the similarity hierarchy, as discussed above. Similarly, the predicted classes also correspond to the predefined set of possible classes (e.g., 1000 possible classes) and are ordered along the horizontal dimension also according the determined sequential ordering. In the illustrated embodiment, the actual classes are ordered 1 through 1000 from top to bottom and the predicted classes are ordered 1 through 1000 from left to right.”] and the value of each cell of the displayed confusion matrix corresponding to match counts between the class along the X-axis class at which the cell is located and the class along the Y-axis at which the cell is located. [[0049]: “Each cell 408 includes a graphical element (e.g., a coloring or shading of the cell and/or a point or dot at the cell location) which encodes an output of the model with respect to the particular combination of an actual class label and a predicted class label. In at least one embodiment, the coloring or shading of the cells 408 correspond to a number images that have been classified by the image classification model 30. In the embodiment shown, the vertical dimension and/or rows of the two-dimensional grid indicates an actual class of the sample image(s) and the horizontal dimension and/or rows of the two-dimensional grid indicates a predicted class of the sample image(s).” [0052]: “Particularly, in at least one embodiment, the processor 14 is configured to, for each cell on the confusion matrix 230, determine a value of each cell as a number of times the classification or misclassification occurred. The processor 14 is configured to display the cells of the confusion matrix 230 using a sequential color scale which maps a value 1 to a light shade and the largest value to a dark shade.” That is, the value of the cell is disclosed in the form of a color scale (having a color value) that corresponds to the number of images that have been classified, which is a match count between the predicted and actual classes.]
Alsallakh does not explicitly teach “receiving or determining an accepted correlation coefficient between each pair of classes” of the N classes, and the order being one that “maximizes a gravity metric for the ordering, wherein the gravity metric is computed as a sum of pairwise terms in which each pairwise term comprises a fraction with the accepted correlation coefficient of a corresponding pair of classes of the N classes in the numerator and a distance metric in the denominator that is indicative of a distance in the ordering between the classes of the corresponding pair of the classes.”
Hahsler teaches “receiving or determining an accepted correlation coefficient between each pair of classes” [§ 1, paragraph 1: “A basic problem in data analysis, called seriation or sometimes sequencing, is to arrange all objects in a set in a linear order given available data and some loss or merit function in order to reveal structural information.” Specifically, the seriation can be performed based on a similarity (correlation coefficient), as described in § 2.7: “The 2-Sum loss criterion (Barnard, Pothen, and Simon 1993) multiplies the similarity between objects with the squared rank differences
L
(
D
)
∑
i
,
j
=
1
p
1
1
+
d
i
j
i
-
j
2
where
s
i
j
=
1
1
+
d
i
j
represents the similarity between objects i and j.” That is, sij constitutes a correlation coefficient, noting that paragraph 26 of the instant application’s specification defines correlation coefficient as referring to similarity (“correlation coefficient between a pair of classes can be viewed as representing the actual correlation or actual similarity between those classes”) and paragraph 27 of a specification states that the coefficient can be arbitrarily defined. Furthermore, given the context of seriating classes in a confusion “objects i and j” in a seriation problem are analogous to two classes.] and the use of a “[…] metric for the ordering, wherein the […] metric is computed as a sum of pairwise terms in which each pairwise term comprises a fraction with the accepted correlation coefficient of a corresponding pair of classes of the N classes in the numerator and a distance metric […] that is indicative of a distance in the ordering between the classes of the corresponding pair of the classes.” [In the above expression,
s
i
j
=
1
1
+
d
i
j
is the correlation coefficient, and
i
-
j
2
is a distance metric squared, known as the squared rank difference. Additionally, it is noted that the objective is maximization of a merit function or the minimization of a loss function, as disclosed in § 2, paragraph 1: “The seriation problem is to find a permutation function Ψ∗ which optimizes the value of a given loss function L or merit function M. This results in the optimization problems…” Note that the objective is to group similar objects closely (see § 2.1, text after equation 3: “the closer objects are together in the order of the matrix, the higher their similarity. This provides a natural objective for seriation.”). For example, if there are three items (for three total pairs), the possible values for distance-squared are 1, 1, and 4. If the similarities are 0.1, 0.2, and 0.9, loss would be high if similarity of 0.9 is assigned to positions corresponding to the distance-squared value of 4. Furthermore, as stated in § 2, a loss function is to be minimized as given by the notation of “argmin.”].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of Alsallakh with the teachings of Hahsler by implementing the seriation technique taught in Hahsler as described above, so as to arrive at the features of “receiving or determining an accepted correlation coefficient between each pair of classes” of the N classes, and the use of a “metric for the ordering, wherein the gravity metric is computed as a sum of pairwise terms in which each pairwise term comprises a fraction with the accepted correlation coefficient of a corresponding pair of classes of the N classes in the numerator and a distance metric” “that is indicative of a distance in the ordering between the classes of the corresponding pair of the classes.” The motivation would have been to arrange classes based on similarity to reveal structural information, as suggested by Hahsler (see § 1, paragraph 1: “…to arrange all objects in a set in a linear order given available data and some loss or merit function in order to reveal structural information…”).
The combination of references thus far does not teach the limitation that the ordering “maximizes” such metric, and the distance metric being “in the denominator” (such that the metric being maximized is a “gravity metric”). It is noted that Hahsler, § (2), equation (1) teaches that maximization of a merit function is equivalent to minimizing a loss function, but does not specifically formulate the loss function noted above as a maximization problem.
Wu et al. teaches “maximizes” [§ 1, text above and below equations (1)-(2): “The rearrangement inequality [1] states that given two finite sequences of real numbers the sum of the product of pairs of terms is maximal when the sequences are similarly ordered and minimal when oppositely ordered… Eq. (2) says that similarly ordered terms minimize the product of sums of pairs, while opposite ordered terms maximize the product of sums.” That is, this reference teaches that the sum of the product of pairs of terms (i.e.,
1
1
+
d
i
j
and
i
-
j
2
in Hahsler is a pair of terms), the sum is maximized when similarly ordered and minimized when oppositely ordered. Therefore, the minimization problem noted in Hahsler is equivalently formulated as a maximization problem by using an opposite order of the terms.]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of the references combined thus far with the teachings of Wu by reformulating the problem as a maximization problem, so as to arrive at the limitation of “maximizes.” Doing so would have been a simple substitution of one known element (minimization) for another (maximization) to obtain predictable results. Specifically, the substituted components and their functions were known in the art and one of ordinary skill in the art could have substituted one known element for another, and the results of the substitution would have been predictable (see MPEP § 2143(I)(B)) because the results are mathematically equivalent based on the teachings of Wu and its discussion of the rearrangement inequality, which is a mathematical principle.
UNM.edu teaches the distance metric being “in the denominator” such that the metric being maximized is a “gravity metric” [Item 2(f) teaches the mathematical principle: “If 0 < x < y then 0 < 1/y < 1/x.” That is, for a given sequence of non-negative numbers x < y, the opposite order is obtained by their reciprocals, i.e., 1/x > 1/y. Therefore, when applied to the distance term
i
-
j
2
, use of the reciprocal
1
i
-
j
2
would correspond to
s
i
j
i
-
j
2
, where the denominator term is the distance squared. Note that a term that includes a distance in the denominator is denoted as a gravity metric.]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of the references combined thus far with the teachings of UNM.edu by reformulating the problem such that the order of the distance terms are in opposite order in reciprocal form in a maximization problem, so as to arrive at the
s
i
j
i
-
j
2
formulation noted above. Doing so would have been a simple combination of prior art elements according to known methods to yield predictable results, namely the combination of the reciprocal formulation with the prior art as combined thus far. Here, one of ordinary skill in the art could have combined the elements as claimed by known methods, in combination, each element merely performs the same function as it does separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable (see MPEP § 2143(I)(A)). This is because Wu already teaches that “opposite ordered terms maximize the product of sums” (as quoted above), and the use of the reciprocal as taught in UNM.edu is a well-known mathematical transformation that reverses the order of a sequence of non-negative numbers (note that
i
-
j
2
in Hahsler is a non-negative integer). Thus, the results are predictable mathematically, and the reciprocals maintain the function of reversal of order satisfying the criteria in Wu when implemented for the term
i
-
j
2
.
As to claim 2, the combination of Alsallakh, Hahsler, Wu and UNM.edu teaches the non-transitory computer readable medium of claim 1, as set forth above, wherein the gravity metric comprises:
G
=
∑
[
…
]
[
…
]
∑
[
…
]
[
…
]
c
i
j
r
i
j
q
where
G
is the gravity metric, and
c
i
j
is the accepted correlation coefficient for the pair of classes
i
and
j
, and
r
i
j
is a distance metric indicative of a distance in the ordering between the classes
i
and
j
, and
q
is a positive real value. [The transformation to
s
i
j
i
-
j
2
that is rendered obvious by the combination of references thus far, as discussed in the rejection of the parent independent claim, teaches cij in the form of
s
i
j
=
1
1
+
d
i
j
as described in Hahsler, the distance rij in the form (i-j) or (i-j)2. Note that since rij is not specifically defined, either (i-j) or (i-j)2 can correspond to rij, in which case q is taught as either 2 or 1.]
The combination of reference thus far does not explicitly teach the parameters of the summation terms (i.e., i = 0 to N-1 and j=i+1 to N-1 in the expression). Instead, Hahsler, § 2.7 teaches the more generalized notation of summation from i,j=1 to P. However, since this more generalized formulation in Hahsler teaches the general condition that the summation is for i and j, the formulation of the summation limits
∑
i
=
0
N
-
1
∑
j
=
i
+
1
N
-
1
(
…
)
would have been obvious as discovery of optimal or workable ranges of the summation limitations by routine optimization under the principle of MPEP § 2144.05(II)(A) (“[W]here the general conditions of a claim are disclosed in the prior art, it is not inventive to discover the optimum or workable ranges by routine experimentation. In re Aller, 220 F.2d 454, 456, 105 USPQ 233, 235 (CCPA 1955).”). Note that the summation limits are result-effective variables (see MPEP § 2144.05(II)(B)) because they define the summation operation.
As to claim 4, the combination of Alsallakh, Hahsler, Wu and UNM.edu teaches the non-transitory computer readable medium of claim 1, wherein
q
=1. [As noted in the rejection of claim 2, the case of q=1 is taught by (i-j)2 since rij broadly encompasses the whole expression of (i-j)2 including the exponent.]
As to claim 5, the combination of Alsallakh, Hahsler, Wu and UNM.edu teaches the non-transitory computer readable medium of claim 1, wherein
q
>1. [As noted in the rejection of claim 2, the case of q=2 is taught by (i-j)2.]
As to claim 6, the combination of Alsallakh, Hahsler, Wu and UNM.edu teaches the non-transitory computer readable medium of claim 1, as set forth above.
wherein the determining of the ordering of the N classes that maximizes a gravity metric for the ordering includes: […]
selecting the determined ordering as the ordering of the N classes for which the computed gravity metric is largest. [Optimization as a maximization or minimization problem in general is taught by equation (2) of Hahlser, which refer to finding the argmax or argmin. In view of the teachings of Wu, the concept of “the largest” corresponds to the maximization equivalents problem described in Wu. Therefore, this limitation is taught by the minimization problem of Hahsler transformed into a maximization problem in the thus-far combination of references set forth in the rejection of the parent independent claim.]
Hahsler further teaches computing the gravity metric for each of the N! orderings of the N classes; [This limitation is met because the optimization in § 2.7 of Hahsler is a loss function which is optimized in accordance with § 2, equation (2), which defines Ψ as a “permutation function.” The concept of “permutation” in conjunction with the notation of argmin (or argmax) is further discussed in Hahlser, § 3, which teaches “N! orderings”: Solving the discrete optimization problem for seriation with most loss/merit functions is clearly very hard. The number of possible permutations for n objects is n! which makes an exhaustive search for sets with a medium to large number of objects infeasible.” The case of complete permutation is disclosed in § 3.1, paragraph 3: “At each step, it is checked if the permutation is valid and several fathoming tests are performed to check if the algorithm should continue with the partial permutation. The most important fathoming test is the boundary test, which checks if the partial permutation can possibly lead to a complete permutation with a better solution than the currently best one. In this way large parts of the search space can be omitted.” Given the context, “complete permutation” here is understood as referring to each of the permutations.] and
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of the references combined thus far, including the above further teachings of Hahsler, so as to have also arrived at the claimed invention of the instant dependent claim. The motivation for doing so is covered by the motivations given for the teachings of Hahsler in the rejection of the parent independent claim.
As to claim 9, the combination of Alsallakh, Hahsler, Wu and UNM.edu teaches the non-transitory computer readable medium of claim 1, wherein the displayed confusion matrix comprises a heat map in which the value of each cell of the displayed confusion matrix corresponding to the match counts between the class along the X-axis class at which the cell is located and the class along the Y-axis at which the cell is located is represented as a color. [Alsallakh, [0049]: “Each cell 408 includes a graphical element (e.g., a coloring or shading of the cell and/or a point or dot at the cell location) which encodes an output of the model with respect to the particular combination of an actual class label and a predicted class label. In at least one embodiment, the coloring or shading of the cells 408 correspond to a number images that have been classified by the image classification model 30. In the embodiment shown, the vertical dimension and/or rows of the two-dimensional grid indicates an actual class of the sample image(s) and the horizontal dimension and/or rows of the two-dimensional grid indicates a predicted class of the sample image(s).” Alsallakh, [0052]: “Particularly, in at least one embodiment, the processor 14 is configured to, for each cell on the confusion matrix 230, determine a value of each cell as a number of times the classification or misclassification occurred. The processor 14 is configured to display the cells of the confusion matrix 230 using a sequential color scale which maps a value 1 to a light shade and the largest value to a dark shade.” That is, shading color constitutes a heat map that reflects the determined value of each cell.]
As to claim 10, the combination of Alsallakh, Hahsler, Wu and UNM.edu teaches the non-transitory computer readable medium of claim 1, wherein the classifier to be visualized comprises a single N-class classifier or N single-class classifiers. [Alsallakh, [0031]: “FIG. 2 shows an exemplary image classification model 30 having at least one convolutional neural network (CNN)… The output of the last fully connected layer is fed to a 1000-way soft-max output layer 114, which produces a probability distribution over 1000 different possible class labels.” That is, the classification model is a multi-class classifier.]
2. Claims 7-8 are rejected under 35 U.S.C. 103 as being unpatentable over Alsallakh in view of Hahsler, Wu and UNM.edu, and further in view of Li et al., “Modeling global and local label correlation with graph convolutional networks for multi‑label chest X‑ray image classification,” Medical & Biological Engineering & Computing (2022) 60:2567–2588 (“Li”).
As to claim 7, the combination of Alsallakh, Hahsler, Wu and UNM.edu teaches the non-transitory computer readable medium of claim 1, as set forth above, but does not teach the further limitations of the instant dependent claim.
Li teaches “wherein the N classes are N clinical findings, and the classifier to be visualized is a classifier configured to classify whether each of the N clinical findings is present in an input medical image.” [§ 3.1: “For any unseen instance xi, the multi-label classifier h(·) predicts Yi = h(xi) as the set of proper labels for xi.” This paper teaches the specific application of medical image classification. § 1, paragraph 1: “multiple lesions in the chest including lung opacity, lung lesion, and pneumonia are usually present in a single scan at the same time as shown in Fig. 1, which indicates that chest disease diagnosis is a multi-label classification (MLC) problem.” § 5.1: “CheXpert dataset is a large-scale public benchmark for chest radiograph interpretation, which consists of 224,316 chest radiographs (frontal and lateral views) of 65,240 patients with their labels to show whether the patient is normal, abnormal, or uncertain (having different types of chest diseases) [19]. It is worth noting that the uncertain labels in CheXpert could be regarded as positives or negatives. The dataset consists of 14 labels, 12 different diseases in addition to no finding and support devices. The CheXpert dataset is divided into training, validation and test sets.” That is, there are 14 classes of clinical findings in the set of images.]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of the references combined thus far with the teachings of Li by implementing the confusion technique of Alsallakh as modified thus far with the medical images application of Li. The motivation would have been to perform classification of x-ray images for diagnosis of disease to assist radiologists, as suggested by Li (§ 1, paragraph 1: “Lung diseases are among the leading causes of death and disability [1, 2]. According to the Forum of International Respiratory Societies, about 334 million people are threatened with asthma, and more than 1 million adults are hospitalized with pneumonia each year [1]. Moreover, millions of people are killed by tuberculosis and lung cancer. Clinically, chest X-ray is the most commonly used imaging modality to assist doctors to evaluate these diseases due to its low cost and easy access… Therefore, computer-aided detection (CAD) [4, 5] is needed to assist radiologists to ease the laborious work and benefit the patients towards precision medicine and individualized treatment.”).
As to claim 8, the combination of Alsallakh, Hahsler, Wu, UNM.edu, and Li teaches the non-transitory computer readable medium of claim 7, as set forth above.
Li further teaches wherein the computing of the match counts between each pair of classes of the N classes using the classifier to be visualized includes performing operations including:
processing each medical image of a plurality of medical images using the classifier to be visualized to determine, for each image, which of the N clinical findings is present in the medical image, [§ 3.1: “For any unseen instance xi, the multi-label classifier h(·) predicts Yi = h(xi) as the set of proper labels for xi.” This paper teaches the specific application of medical image classification. § 1, paragraph 1: “multiple lesions in the chest including lung opacity, lung lesion, and pneumonia are usually present in a single scan at the same time as shown in Fig. 1, which indicates that chest disease diagnosis is a multi-label classification (MLC) problem.” § 5.1: “CheXpert dataset is a large-scale public benchmark for chest radiograph interpretation, which consists of 224,316 chest radiographs (frontal and lateral views) of 65,240 patients with their labels to show whether the patient is normal, abnormal, or uncertain (having different types of chest diseases) [19]. It is worth noting that the uncertain labels in CheXpert could be regarded as positives or negatives. The dataset consists of 14 labels, 12 different diseases in addition to no finding and support devices. The CheXpert dataset is divided into training, validation and test sets.”] and
determining classifier-computed match counts based on rates of co-occurrences of pairs of clinical findings in the output of the processing. [§ 4.3.1: “The estimated correlation matrix represents the statistical label relationships. We first count the co-occurrence frequency of each label pair (i, j) in the training set and compose the label co-occurrence matrix: …” See equations (6) and (11). Furthermore, the global co-occurrence matrix is supplemented with an “adaptively learned local correlation matrix” (§ 5.3.3, part (6)), and the final label prediction confidence is based on the global and local co-occurrence, as described in § 4.4, equation (15). Therefore, the classifier outputs, which would form the match counts when applied to a confusion matrix, are based on both the global and local co-occurrences, the combination of which is part of the output of the GLLC (global and local label correlation) module and is part of the classifier as a whole.]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of the references combined thus far, including the above further teachings of Li, so as to have also arrived at the claimed invention of the instant dependent claim. The motivation for doing so is covered by the motivation given for Li in the rejection of the parent dependent claim.
3. Claims 11-12 and 14-15 are rejected under 35 U.S.C. 103 as being unpatentable over Alsallakh in view of Hahsler, Wu, UNM.edu, and Li.
As to claim 11, Alsallakh teaches a classifier visualization method, comprising:
[…] N classes, wherein N is equal to or greater than four […] [As shown in FIG. 3 generally, the number of classes can be numerous, including 4 or more classes. The example of 1000 classes is disclosed in [0025]: “each having a respective class labels from a defined number (e.g., 1000) of possible classes”; [0031]: “The output of the last fully connected layer is fed to a 1000-way soft-max output layer 114, which produces a probability distribution over 1000 different possible class labels.”]
determining an ordering of the N classes that […]; [[0054]: “In at least one embodiment, the processor 14 is configured to apply at least one seriation algorithm to reveal block patterns in the confusion matrix and to order the class hierarchy viewer 210 and the confusion matrix 230 accordingly.”] […] and
displaying a confusion matrix for a classifier to be visualized, [[0036]: “FIG. 3 shows an overview of an exemplary main user interface 200 of the visual analytics program 28.” [0048]: “FIG. 6 shows a detailed view of the confusion matrix 230 of the main user interface 200.”] the displayed confusion matrix having the N classes ordered in the determined ordering along an X-axis of the confusion matrix and having the N classes ordered in the determined ordering along a Y-axis of the confusion matrix, [[0050]: “The actual classes on the vertical dimension correspond the predefined set of possible classes (e.g., 1000 possible classes) and are ordered along the vertical dimension according the sequential ordering, which was determined based on the similarity hierarchy, as discussed above. Similarly, the predicted classes also correspond to the predefined set of possible classes (e.g., 1000 possible classes) and are ordered along the horizontal dimension also according the determined sequential ordering. In the illustrated embodiment, the actual classes are ordered 1 through 1000 from top to bottom and the predicted classes are ordered 1 through 1000 from left to right.”] and the value of each cell of the displayed confusion matrix corresponding to the match counts between the class along the X-axis class at which the cell is located and the class along the Y-axis at which the cell is located. [[0049]: “Each cell 408 includes a graphical element (e.g., a coloring or shading of the cell and/or a point or dot at the cell location) which encodes an output of the model with respect to the particular combination of an actual class label and a predicted class label. In at least one embodiment, the coloring or shading of the cells 408 correspond to a number images that have been classified by the image classification model 30. In the embodiment shown, the vertical dimension and/or rows of the two-dimensional grid indicates an actual class of the sample image(s) and the horizontal dimension and/or rows of the two-dimensional grid indicates a predicted class of the sample image(s).” [0052]: “Particularly, in at least one embodiment, the processor 14 is configured to, for each cell on the confusion matrix 230, determine a value of each cell as a number of times the classification or misclassification occurred. The processor 14 is configured to display the cells of the confusion matrix 230 using a sequential color scale which maps a value 1 to a light shade and the largest value to a dark shade.” That is, the value of the cell is disclosed in the form of a color scale (having a color value) that corresponds to the number of images that have been classified, which is a match count between the predicted and actual classes.]
Alsallakh does not explicitly teach:
(1) “receiving or determining an accepted correlation coefficient between each pair of classes” of the N classes; and
(2) “maximizes a gravity metric for the ordering, wherein the gravity metric is computed as a sum of pairwise terms in which each pairwise term comprises a fraction with the accepted correlation coefficient of a corresponding pair of classes of the N classes in the numerator and a distance metric in the denominator that is indicative of a distance in the ordering between the classes of the corresponding pair of the classes”; and
(3) “wherein the N classes are N clinical findings, and a classifier to be visualized is a classifier configured to classify whether each of the N clinical findings is present in an input medical image” and “processing each medical image of a plurality of medical images using the classifier to be visualized to determine, for each image, which of the N clinical findings is present in the medical image; determining classifier-computed match counts based on rates of co-occurrences of pairs of clinical findings in the output of the processing.”
Hahsler teaches “receiving or determining an accepted correlation coefficient between each pair of classes” [§ 1, paragraph 1: “A basic problem in data analysis, called seriation or sometimes sequencing, is to arrange all objects in a set in a linear order given available data and some loss or merit function in order to reveal structural information.” Specifically, the seriation can be performed based on a similarity (correlation coefficient), as described in § 2.7: “The 2-Sum loss criterion (Barnard, Pothen, and Simon 1993) multiplies the similarity between objects with the squared rank differences
L
(
D
)
∑
i
,
j
=
1
p
1
1
+
d
i
j
i
-
j
2
where
s
i
j
=
1
1
+
d
i
j
represents the similarity between objects i and j.” That is, sij constitutes a correlation coefficient, noting that paragraph 26 of the instant application’s specification defines correlation coefficient as referring to similarity (“correlation coefficient between a pair of classes can be viewed as representing the actual correlation or actual similarity between those classes”) and paragraph 27 of a specification states that the coefficient can be arbitrarily defined. Furthermore, given the context of seriating classes in a confusion “objects i and j” in a seriation problem are analogous to two classes.] and the use of a “[…] metric for the ordering, wherein the […] metric is computed as a sum of pairwise terms in which each pairwise term comprises a fraction with the accepted correlation coefficient of a corresponding pair of classes of the N classes in the numerator and a distance metric […] that is indicative of a distance in the ordering between the classes of the corresponding pair of the classes.” [In the above expression,
s
i
j
=
1
1
+
d
i
j
is the correlation coefficient, and
i
-
j
2
is a distance metric squared, known as the squared rank difference. Additionally, it is noted that the objective is maximization of a merit function or the minimization of a loss function, as disclosed in § 2, paragraph 1: “The seriation problem is to find a permutation function Ψ∗ which optimizes the value of a given loss function L or merit function M. This results in the optimization problems…” Note that the objective is to group similar objects closely (see § 2.1, text after equation 3: “the closer objects are together in the order of the matrix, the higher their similarity. This provides a natural objective for seriation.”). For example, if there are three items (for three total pairs), the possible values for distance-squared are 1, 1, and 4. If the similarities are 0.1, 0.2, and 0.9, loss would be high if similarity of 0.9 is assigned to positions corresponding to the distance-squared value of 4. Furthermore, as stated in § 2, a loss function is to be minimized as given by the notation of “argmin.”].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of Alsallakh with the teachings of Hahsler by implementing the seriation technique taught in Hahsler as described above, so as to arrive at the features of “receiving or determining an accepted correlation coefficient between each pair of classes” of the N classes, and the use of a “metric for the ordering, wherein the gravity metric is computed as a sum of pairwise terms in which each pairwise term comprises a fraction with the accepted correlation coefficient of a corresponding pair of classes of the N classes in the numerator and a distance metric” “that is indicative of a distance in the ordering between the classes of the corresponding pair of the classes.” The motivation would have been to arrange classes based on similarity to reveal structural information, as suggested by Hahsler (see § 1, paragraph 1: “…to arrange all objects in a set in a linear order given available data and some loss or merit function in order to reveal structural information…”).
The combination of references thus far does not teach the limitation that the ordering “maximizes” such metric, and the distance metric being “in the denominator” (such that the metric being maximized is a “gravity metric”), as well as limitations (3) listed above. It is noted that Hahsler, § (2), equation (1) teaches that maximization of a merit function is equivalent to minimizing a loss function, but does not specifically formulate the loss function noted above as a maximization problem.
Wu et al. teaches “maximizes” [§ 1, text above and below equations (1)-(2): “The rearrangement inequality [1] states that given two finite sequences of real numbers the sum of the product of pairs of terms is maximal when the sequences are similarly ordered and minimal when oppositely ordered… Eq. (2) says that similarly ordered terms minimize the product of sums of pairs, while opposite ordered terms maximize the product of sums.” That is, this reference teaches that the sum of the product of pairs of terms (i.e.,
1
1
+
d
i
j
and
i
-
j
2
in Hahsler is a pair of terms), the sum is maximized when similarly ordered and minimized when oppositely ordered. Therefore, the minimization problem noted in Hahsler is equivalently formulated as a maximization problem by using an opposite order of the terms.]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of the references combined thus far with the teachings of Wu by reformulating the problem as a maximization problem, so as to arrive at the limitation of “maximizes.” Doing so would have been a simple substitution of one known element (minimization) for another (maximization) to obtain predictable results. Specifically, the substituted components and their functions were known in the art and one of ordinary skill in the art could have substituted one known element for another, and the results of the substitution would have been predictable (see MPEP § 2143(I)(B)) because the results are mathematically equivalent based on the teachings of Wu and its discussion of the rearrangement inequality, which is a mathematical principle.
UNM.edu teaches the distance metric being “in the denominator” such that the metric being maximized is a “gravity metric” [Item 2(f) teaches the mathematical principle: “If 0 < x < y then 0 < 1/y < 1/x.” That is, for a given sequence of non-negative numbers x < y, the opposite order is obtained by their reciprocals, i.e., 1/x > 1/y. Therefore, when applied to the distance term
i
-
j
2
, use of the reciprocal
1
i
-
j
2
would correspond to
s
i
j
i
-
j
2
, where the denominator term is the distance squared. Note that a term that includes a distance in the denominator is denoted as a gravity metric.]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of the references combined thus far with the teachings of UNM.edu by reformulating the problem such that the order of the distance terms are in opposite order in reciprocal form in a maximization problem, so as to arrive at the
s
i
j
i
-
j
2
formulation noted above. Doing so would have been a simple combination of prior art elements according to known methods to yield predictable results, namely the combination of the reciprocal formulation with the prior art as combined thus far. Here, one of ordinary skill in the art could have combined the elements as claimed by known methods, in combination, each element merely performs the same function as it does separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable (see MPEP § 2143(I)(A)). This is because Wu already teaches that “opposite ordered terms maximize the product of sums” (as quoted above), and the use of the reciprocal as taught in UNM.edu is a well-known mathematical transformation that reverses the order of a sequence of non-negative numbers (note that
i
-
j
2
in Hahsler is a non-negative integer). Thus, the results are predictable mathematically, and the reciprocals maintain the function of reversal of order satisfying the criteria in Wu when implemented for the term
i
-
j
2
.
The combination of references thus far does not teach limitations (3) listed above.
Li teaches “wherein the N classes are N clinical findings, and a classifier to be visualized is a classifier configured to classify whether each of the N clinical findings is present in an input medical image” and “processing each medical image of a plurality of medical images using the classifier to be visualized to determine, for each image, which of the N clinical findings is present in the medical image” [§ 3.1: “For any unseen instance xi, the multi-label classifier h(·) predicts Yi = h(xi) as the set of proper labels for xi.” This paper teaches the specific application of medical image classification. § 1, paragraph 1: “multiple lesions in the chest including lung opacity, lung lesion, and pneumonia are usually present in a single scan at the same time as shown in Fig. 1, which indicates that chest disease diagnosis is a multi-label classification (MLC) problem.” § 5.1: “CheXpert dataset is a large-scale public benchmark for chest radiograph interpretation, which consists of 224,316 chest radiographs (frontal and lateral views) of 65,240 patients with their labels to show whether the patient is normal, abnormal, or uncertain (having different types of chest diseases) [19]. It is worth noting that the uncertain labels in CheXpert could be regarded as positives or negatives. The dataset consists of 14 labels, 12 different diseases in addition to no finding and support devices. The CheXpert dataset is divided into training, validation and test sets.” That is, there are 14 classes of clinical findings in the set of images.] and “determining classifier-computed match counts based on rates of co-occurrences of pairs of clinical findings in the output of the processing.” [§ 4.3.1: “The estimated correlation matrix represents the statistical label relationships. We first count the co-occurrence frequency of each label pair (i, j) in the training set and compose the label co-occurrence matrix: …” See equations (6) and (11). Furthermore, the global co-occurrence matrix is supplemented with an “adaptively learned local correlation matrix” (§ 5.3.3, part (6)), and the final label prediction confidence is based on the global and local co-occurrence, as described in § 4.4, equation (15). Therefore, the classifier outputs, which would form the match counts when applied to a confusion matrix, are based on both the global and local co-occurrences, the combination of which is part of the output of the GLLC (global and local label correlation) module and is part of the classifier as a whole.]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of the references combined thus far with the teachings of Li by implementing the confusion technique of Alsallakh as modified thus far with the medical images application of Li. The motivation would have been to perform classification of x-ray images for diagnosis of disease to assist radiologists, as suggested by Li (§ 1, paragraph 1: “Lung diseases are among the leading causes of death and disability [1, 2]. According to the Forum of International Respiratory Societies, about 334 million people are threatened with asthma, and more than 1 million adults are hospitalized with pneumonia each year [1]. Moreover, millions of people are killed by tuberculosis and lung cancer. Clinically, chest X-ray is the most commonly used imaging modality to assist doctors to evaluate these diseases due to its low cost and easy access… Therefore, computer-aided detection (CAD) [4, 5] is needed to assist radiologists to ease the laborious work and benefit the patients towards precision medicine and individualized treatment.”).
As to claim 12, the combination of Alsallakh, Hahsler, Wu, UNM.edu, and Li teaches the method of claim 11, wherein the gravity metric comprises:
G
=
∑
[
…
]
[
…
]
∑
[
…
]
[
…
]
c
i
j
r
i
j
q
where
G
is the gravity metric, and
c
i
j
is the accepted correlation coefficient for the pair of classes
i
and
j
, and
r
i
j
is a distance metric indicative of a distance in the ordering between the classes
i
and
j
, and
q
is a positive real value. [The transformation to
s
i
j
i
-
j
2
that is rendered obvious by the combination of references thus far, as discussed in the rejection of the parent independent claim, teaches cij in the form of
s
i
j
=
1
1
+
d
i
j
as described in Hahsler, the distance rij in the form (i-j) or (i-j)2. Note that since rij is not specifically defined, either (i-j) or (i-j)2 can correspond to rij, in which case q is taught as either 2 or 1.]
The combination of reference thus far does not explicitly teach the parameters of the summation terms (i.e., i = 0 to N-1 and j=i+1 to N-1 in the expression). Instead, Hahsler, § 2.7 teaches the more generalized notation of summation from i,j=1 to P. However, since this more generalized formulation in Hahsler teaches the general condition that the summation is for i and j, the formulation of the summation limits
∑
i
=
0
N
-
1
∑
j
=
i
+
1
N
-
1
(
…
)
would have been obvious as discovery of optimal or workable ranges of the summation limitations by routine optimization under the principle of MPEP § 2144.05(II)(A) (“[W]here the general conditions of a claim are disclosed in the prior art, it is not inventive to discover the optimum or workable ranges by routine experimentation. In re Aller, 220 F.2d 454, 456, 105 USPQ 233, 235 (CCPA 1955).”). Note that the summation limits are result-effective variables (see MPEP § 2144.05(II)(B)) because they define the summation operation.
As to claim 14, the combination of Alsallakh, Hahsler, Wu, UNM.edu, and Li teaches the method of claim 12, wherein
q
=1. [As noted in the rejection of the parent dependent claim, the case of q=1 is taught by (i-j)2 since rij broadly encompasses the whole expression of (i-j)2 including the exponent.]
As to claim 15, the combination of Alsallakh, Hahsler, Wu, UNM.edu, and Li teaches the method of claim 12, wherein
q
>1. [As noted in the rejection of the parent dependent claim, the case of q=2 is taught by (i-j)2.]
Allowable Subject Matter
Claims 3 and 13 contain allowable subject matter but are currently rejected for indefiniteness under § 112(b). If these claims are amended to resolve the § 112(b) rejection (in a manner consistent with the current interpretations of any indefinite limitations), these claims would be objected to as being dependent upon a rejected base claim, and allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
The prior art of record does not teach or fairly suggest the following limitations of claims 3 and 13 in combination with the remaining limitations of their parent claims:
wherein the gravity metric comprises:
G
=
∑
i
=
0
N
-
1
∑
j
=
i
+
1
m
i
n
(
i
+
D
v
-
1
,
N
-
1
)
c
i
j
r
i
j
q
where
G
is the gravity metric, and
c
i
j
is the accepted correlation coefficient for the pair of classes
i
and
j
, and
r
i
j
is a distance metric indicative of a distance in the ordering between the classes
i
and
j
, and
q
is a positive real value, and
D
v
is a vanishing distance having a value between 2 and N.
The closest prior art is discussed as follows.
Although Hahsler, for example, teaches various optimization functions or ordering items, the addition of an additional variable Dv and is use in a minimization operation in the summation limit is not disclosed by Hahlser, or the other prior art made of record.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. The following documents depict the state of the art.
Katsis et al., US 11288115 B1, teaches configuring “an order of labels on the axes associated with the confusion matrix”
Alsallakh et al., “Do Convolutional Neural Networks Learn Class Hierarchy?,” arXiv:1710.06501v1 [cs.CV] 17 Oct 2017 teaches the concepts in the Alsallakh patent literature document cited in this action.
UCSD.edu, “Linear programming Notes V: Problem Transformations", archived 2016 (see form PTO-892 for full citation) teaches transforming between maximization and minimization problems.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YAO DAVID HUANG whose telephone number is (571)270-1764. The examiner can normally be reached Monday - Friday 9:00 am - 5:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Miranda Huang can be reached at (571) 270-7092. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Y.D.H./Examiner, Art Unit 2124
/MIRANDA M HUANG/Supervisory Patent Examiner, Art Unit 2124
