DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Amendments
This Office Action is in response to the amendment filed on 07/27/2026.
Claims 1 and 17 have been amended.
No claim has been cancelled.
No new claims have been added.
The objections and rejections from the prior correspondence that are not restated herein are withdrawn.
No prior art was found that anticipates or renders Claim 13 obvious.
Response to Arguments
Applicant's arguments filed on 07/27/2026 have been fully considered.
Applicant’s arguments regarding the Claim Objections of the previous office action have been fully considered but are not persuasive. The amendments of the independent claims only added clarity regarding what is being trained and the order in which training occurs but did not clearly distinguish “calibration training data” from “first input training data”. For this reason, the claim objections to claims 1 and 17 are maintained.
Applicant’s arguments regarding the 35 U.S.C. 103 rejections of the previous office action have been fully considered but are not persuasive. Applicant argues:
“Applicant respectfully submits that the Zhang reference fails to disclose processing circuitry configured to train a first calibration model that receives, as input, first processing data and a first processing label assigned by a first user to the first processing data, and outputs calibration data relating to calibration of individual characteristics in label assignment by the first user, based on (1) first input training data, (2) a first training label assigned by the first user to the first input training data, and (3) calibration training data; and train, by using the trained first calibration model, a target model based on at least the first processing data and the calibration data or a calibration label which is a label calibrated in the individual characteristics included in the first processing label using the calibration data, as recited in amended Claim 1.”
(See page 9 of the remarks)
Examiner respectfully disagrees. Due to claim amendments, VEIT is now the primary reference in view of ZHANG. VEIT teaches first training a label cleaning network
g
with human verified labels
v
and, after training the network
g
, using the outputs of
g
as ground truth labels to train a second classifier
h
to annotate images by imitating the first classifier
g
. ZHANG teaches labels assigned by annotators to input training data and the confusion matrix with the annotator behaviors for doing corrections. Specifically, VEIT teaches:
processing circuitry configured to […] (VEIT [page 844, 4.4. Training Details] teaches: “We trained the baseline model on 50 NVIDIA K40 GPUs using the noisy labels from the Open Images training set. […] The four different fine-tuning variants are trained for additional 4 million batches each.”)
[…] train a first calibration model that receives, as input, first processing data and a first processing label assigned […] to the first processing data, and outputs calibration data relating to calibration of […] label assignment […], based on (1) first input training data, (2) a first training label assigned […] to the first input training data, and (3) calibration training data; (VEIT [page 841, section 3. Our Approach] teaches: “Formally, we have a very large training dataset
T
comprising tuples of noisy labels
y
(i.e., a first processing label assigned […] to the first processing data) and images
I
,
T
=
y
i
,
I
i
,
…
, and a small dataset
V
(i.e., and (3) calibration training data) of triplets of verified labels
v
, noisy labels
y
(i.e., (2) a first training label assigned […] to the first input training data) and images
I
(i.e., (1) first input training data)
,
V
=
v
i
,
y
i
,
I
i
,
…
.” VEIT [page 842, section 3.1. Multi-Task Label Cleaning Architecture] teaches: “The first classifier is a label cleaning network denoted as
g
(i.e., a first calibration model) that models the structure in the label space and learns a mapping from the noisy labels
y
to the human verified labels
v
, conditional on the input image. We denote the cleaned labels output by
g
as
c
^
so that
c
^
=
g
y
,
I
(i.e., and outputs calibration data relating to calibration of […] label assignment […]). […] The label cleaning network
g
is shown in the top row of Figure 3. In order to model the label structure and noise conditional on the image, the network has two separate inputs (i.e., that receives, as input), the noisy labels
y
(i.e., a first processing label assigned […] to the first processing data) as well as the visual features
f
I
(i.e., first processing data). […] Denoting the residual cleaning module as
g
'
, the label cleaning network
g
computes cleaned labels
c
^
=
c
l
i
p
y
+
g
'
y
,
f
I
,
0
,
1
1
VEIT [page 842, section 3.2. Model Training] teaches: “The label cleaning network is supervised by (i.e., based on) the verified labels of all samples
i
(i.e., the first input training data) in the human rated set
V
(i.e., (2) a first training label assigned to the first input training data, and (3) calibration training data).” VEIT [page 844, section 4.3. Baselines and Model Variants] teaches: “For first variant, we initially train just the label cleaning network on the human rated data. Then, subsequently we train the cleaning network and the classification layer jointly.” Examiner’s note: VEIT trains the first classifier
g
before training the second classifier
h
.)
train, by using the trained first calibration model, a target model based on at least the first processing data and the calibration data or a calibration label which is a label calibrated in the […] first processing label using the calibration data, […] (VEIT [page 842, section 3.1. Multi-Task Label Cleaning Architecture] teaches: “We denote the cleaned labels output by
g
as
c
^
so that
c
^
=
g
y
,
I
(i.e., the calibration data or a calibration label which is a label calibrated in the […] first processing label using the calibration data, […]). […] The second classifier is an image classifier denoted as
h
that learns to annotate images by imitating the first classifier
g
(i.e., train, by using the trained first calibration model, a target model) by using
g
’s predictions as ground truth targets. […] The image classifier
h
is shown in the bottom row of Figure 3. First, a sample image (i.e., based on at least the first processing data) is processed by the convolutional network to compute high level image features. Then, these features are passed through a fully-connected layer
w
followed by a sigmoid
σ
,
h
=
σ
w
f
I
. […] Denoting the residual cleaning module as
g
'
, the label cleaning network
g
computes cleaned labels
c
^
=
c
l
i
p
y
+
g
'
y
,
f
I
,
0
,
1
1
"
)
VEIT is not relied upon for teaching, but ZHANG teaches: […] a […] calibration model […] a first processing label assigned by a first user to the first processing data, and outputs calibration data relating to calibration of individual characteristics in label assignment by the first user, based on […] (2) a first training label assigned by the first user to the first input training data, […] (ZHANG [page 1, Abstract] teaches: "In this work, we present a method for jointly learning, from purely noisy observations alone, the reliability of individual annotators and the true segmentation label distributions, using two coupled CNNs." ZHANG [page 4, Figure 1] teaches: "An architecture schematic in the presence of 3 annotators of varying characteristics (over-segmentation, under-segmentation and confusing between two classes, red and blue). The model consists of two parts: (1) segmentation network parametrised by
θ
that generates an estimate of the unobserved true segmentation probabilities,
p
θ
x
; (2) annotator network, parametrised by
ϕ
, that estimates the pixelwise confusion matrices (CMs),
A
ϕ
r
x
r
=
1
3
of the annotators for the given input image x.” ZHANG [page 5, Section 3.3] teaches: "Given training
X
=
x
n
n
=
1
N
and noisy labels
Y
~
(
r
)
=
y
~
n
r
:
r
∈
S
x
n
n
=
1
N
for
r
=
1
,
…
,
R
, we optimize the parameters
{
θ
,
ϕ
}
by minimizing the negative log-likelihood (NLL),
-
l
o
g
p
Y
~
1
,
…
,
Y
~
R
X
. From eqs. (1) and (2), this optimization objective equates to the sum of cross-entropy losses between the observed noisy segmentations and the estimated annotator label distributions:
PNG
media_image1.png
113
1114
media_image1.png
Greyscale
ZHANG [page 2, Section 1 Introduction] teaches: "In this work, we introduce the first instance of an end-to-end supervised segmentation method that jointly estimates, from noisy labels alone, the reliability of multiple human annotators and true segmentation labels. The proposed architecture (Fig. 1) consists of two coupled CNNs where one estimates the true segmentation probabilities and the other models the characteristics of individual annotators (e.g., tendency to over-segmentation, mix-up between different classes, etc.) by estimating the pixel-wise confusion matrices (CMs) on a per image basis." Examiner's note: A calibration model can be reasonably interpreted as the "annotator network, parametrised by
ϕ
" which outputs the confusion matrices. The training process also uses noisy segmentation labels (i.e., first processing label) provided by the annotator for an image
x
n
in the loss function (ZHANG [Eq. 4 and Figure 1]).)
[…] a label calibrated in the individual characteristics included in the first processing label using the calibration data, […] (ZHANG [page 5, section 3.3] teaches the loss function used to minimize the loss of the annotator network and segmentation network. Under broadest reasonable interpretation, a label calibrated in the individual characteristics included in the first processing label using the calibration data, […] can be interpreted as ZHANG’s confusion matrices (CMs)
A
^
ϕ
r
x
n
, which model the individual characteristics of each annotator (e.g., tendency to over-segmentation, mix-up between different classes, etc.) for an input image
x
n
.)
Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of VEIT and ZHANG before them, to include ZHANG’s user annotations and confusion matrices in VEIT’s noisy label learning method. One would have been motivated to make such a combination in order to disentangle annotators’ mistakes and the true labels based on correlations between similar image samples (ZHANG [page 2, 1 Introduction]), to estimate individual annotators and improve robustness against label noise and for educating inexperienced annotators by using the estimated spatial characteristics of segmentation mistakes for insights into the annotation behaviors (ZHANG [page 9, 5. Discussion and Conclusion]).
Therefore, VEIT in view of ZHANG teaches every argued limitation of claim 1.
Applicant further argues:
“[…] the Zhang reference clearly fails to disclose training the target model after the calibration model has been trained, as required by amended Claim 1. Moreover, Zhang discloses training both models simultaneously so as to minimize the loss function, and therefore the segmentation CNN of Zhang is not trained by supervised learning using calibrated labels as ground truth, as is the target model recited in Claim 1.”
(See page 10 of the remarks)
Examiner respectfully disagrees. VEIT is used to teach the training of the calibration model as training the label cleaning network
g
, and once this network is trained, VEIT trains a second classifier
h
using the outputs of
g
as ground truth, as shown in the rejections above.
Applicant further argues:
“Applicant respectfully submits that the VEIT reference fails to disclose processing circuitry configured to train a first calibration model that receives, as input, first processing data and a first processing label assigned by a first user to the first processing data, and outputs calibration data relating to calibration of individual characteristics in label assignment by the first user, based on (1) first input training data, (2) a first training label assigned by the first user to the first input training data, and (3) calibration training data; and train, by using the trained first calibration model, a target model based on at least the first processing data and the calibration data or a calibration label which is a label calibrated in the individual characteristics included in the first processing label using the calibration data, as recited in amended Claim 1.”
(See pages 10-11 of the remarks)
Examiner respectfully disagrees. In response to applicant's arguments against references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references (See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986)). The combination of VEIT and ZHANG teaches all the limitations of claim 1. The processing circuitry can be interpreted as VEIT’s NVIDIA k40 GPUs that were used to conduct the training of the classifiers. Further, VEIT first trains the first classifier
g
and then trains the second classifier
h
(VEIT [page 842, section 3.1. Multi-Task Label Cleaning Architecture], [page 842, section 3.2. Model Training], and [page 844, section 4.3. Baselines and Model Variants]). Accordingly,
g
is trained before
h
is trained. Additionally, nothing in the claim restricts the training of the target model from occurring in a joint setting once the first calibration model has been trained to clean the noisy labels. ZHANG’s estimation of individual annotators improves robustness against label noise, and the estimated characteristics of the annotators’ mistakes give insight for educating inexperienced annotators (ZHANG [page 9, section 5. Discussion and Conclusion]). As such, the noisy labels
y
that VEIT’s first classifier
g
receives can be the user annotations of ZHANG. The training of the first classifier
g
in VEIT prior to training the second classifier
h
can be done using ZHANG’s confusion matrices, which capture the individual tendencies and mistakes of annotators to further improve robustness against label noise. Accordingly, the combination teaches all the limitations of claim 1 as shown above on pages 3-7 of this Office Action.
Applicant further argues:
“Claim 17 recites limitations analogous to the limitations recited in Claim 1, and is amended in a manner analogous to the amendment to Claim 1. Accordingly, for the reasons stated above, Applicant respectfully submits that the rejection of Claim 17 is rendered moot by the present amendment to that claim.”
(See page 12 of the remarks)
Examiner respectfully disagrees. Claim 17 is rejected for similar reasons as claim 1 and using similar rationale.
Applicant further argues:
“Applicant respectfully submits that the '207 application, the Houlsby reference, and the Guan reference fail to remedy the deficiencies of the Zhang reference, and the VEIT reference, as discussed above. Accordingly, Applicant respectfully submits that the rejections of the above-noted dependent claims are rendered moot by the present amendment to Claim 1.
Thus, it is respectfully submitted that independent Claims 1 and 17 (and all associated dependent claims) patentably define over any proper combination of the cited references.”
(See page 12 of the remarks)
Examiner respectfully disagrees. The 207’ application (SHAMIR), HOULSBY, and GUAN are not required to remedy any alleged deficiency because the combination of VEIT and ZHANG teaches all the limitations of claims 1 and 17. Accordingly, all associated dependent claims 2-4, 6-12, and 14-16 remain rejected as shown in the 103 rejections below.
Claim Objections
Claim 1 is objected to because, according to page 11, lines 18-20, the specification suggests that "first input training data" appears to be "data for use in a label assignment is an MR image in which a measurement voxel of a magnetic resonance imaging apparatus is set"; claim 8 also supports this interpretation. Additionally, page 15, lines 10-19 state that "The trial data is the calibration training data assigned a label for use in machine learning of the calibration model. The following description assumes that the trial data is an MR image which may be generated by actually imaging a patient or phantom with a magnetic resonance imaging apparatus, or may be a pseudo-MR image artificially generated through image processing or prediction calculation." Therefore, under broadest reasonable interpretation, there is no substantial difference between "calibration training data" and "first input training data" because the specification states that they both can be an MR image to which a label is assigned. For purposes of examination, "first input training data" and "calibration training data" will be treated as equivalent terms.
Claim 17 is objected to for the same reasons described in the objection to claim 1.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 6-7, 9-12, 14, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over VEIT ("Learning From Noisy Large-Scale Datasets With Minimal Supervision") in view of ZHANG (“Disentangling Human Error from the Ground Truth in Segmentation of Medical Images”), hereinafter VEIT and ZHANG, respectively.
Regarding Claim 1:
VEIT teaches:
An apparatus of machine learning, comprising: processing circuitry configured to […] (VEIT [page 844, 4.4. Training Details] teaches: “We trained the baseline model on 50 NVIDIA K40 GPUs using the noisy labels from the Open Images training set. […] The four different fine-tuning variants are trained for additional 4 million batches each.”)
[…] train a first calibration model that receives, as input, first processing data and a first processing label assigned […] to the first processing data, and outputs calibration data relating to calibration of […] label assignment […], based on (1) first input training data, (2) a first training label assigned […] to the first input training data, and (3) calibration training data; (VEIT [page 841, section 3. Our Approach] teaches: “Formally, we have a very large training dataset
T
comprising tuples of noisy labels
y
(i.e., a first processing label assigned […] to the first processing data) and images
I
,
T
=
y
i
,
I
i
,
…
, and a small dataset
V
(i.e., and (3) calibration training data) of triplets of verified labels
v
, noisy labels
y
(i.e., (2) a first training label assigned […] to the first input training data) and images
I
(i.e., (1) first input training data)
,
V
=
v
i
,
y
i
,
I
i
,
…
.” VEIT [page 842, section 3.1. Multi-Task Label Cleaning Architecture] teaches: “The first classifier is a label cleaning network denoted as
g
(i.e., a first calibration model) that models the structure in the label space and learns a mapping from the noisy labels
y
to the human verified labels
v
, conditional on the input image. We denote the cleaned labels output by
g
as
c
^
so that
c
^
=
g
y
,
I
(i.e., and outputs calibration data relating to calibration of […] label assignment […]). […] The label cleaning network
g
is shown in the top row of Figure 3. In order to model the label structure and noise conditional on the image, the network has two separate inputs (i.e., that receives, as input), the noisy labels
y
(i.e., a first processing label assigned […] to the first processing data) as well as the visual features
f
I
(i.e., first processing data). […] Denoting the residual cleaning module as
g
'
, the label cleaning network
g
computes cleaned labels
c
^
=
c
l
i
p
y
+
g
'
y
,
f
I
,
0
,
1
1
VEIT [page 842, section 3.2. Model Training] teaches: “The label cleaning network is supervised by (i.e., based on) the verified labels of all samples
i
(i.e., the first input training data) in the human rated set
V
(i.e., (2) a first training label assigned to the first input training data, and (3) calibration training data).” VEIT [page 844, section 4.3. Baselines and Model Variants] teaches: “For first variant, we initially train just the label cleaning network on the human rated data. Then, subsequently we train the cleaning network and the classification layer jointly.” Examiner’s note: VEIT trains the first classifier
g
before training the second classifier
h
.)
train, by using the trained first calibration model, a target model based on at least the first processing data and the calibration data or a calibration label which is a label calibrated in the […] first processing label using the calibration data, […] (VEIT [page 842, section 3.1. Multi-Task Label Cleaning Architecture] teaches: “We denote the cleaned labels output by
g
as
c
^
so that
c
^
=
g
y
,
I
(i.e., the calibration data or a calibration label which is a label calibrated in the […] first processing label using the calibration data, […]). […] The second classifier is an image classifier denoted as
h
that learns to annotate images by imitating the first classifier
g
(i.e., train, by using the trained first calibration model, a target model) by using
g
’s predictions as ground truth targets. […] The image classifier
h
is shown in the bottom row of Figure 3. First, a sample image (i.e., based on at least the first processing data) is processed by the convolutional network to compute high level image features. Then, these features are passed through a fully-connected layer
w
followed by a sigmoid
σ
,
h
=
σ
w
f
I
. […] Denoting the residual cleaning module as
g
'
, the label cleaning network
g
computes cleaned labels
c
^
=
c
l
i
p
y
+
g
'
y
,
f
I
,
0
,
1
1
"
)
wherein the processing circuitry is further configured to: generate, as the calibration data, a calibration parameter with respect to the calibration label or the first processing label by inputting the first processing data and the first processing label to the first calibration model; (VEIT [page 844, section 4.4. Training Details] teaches: “We trained the baseline model on 50 NVIDIA K40 GPUs using the noisy labels from the Open Images training set. We stopped training after 49 million mini-batches (with 32 images each). This network is the starting point for all model variants.” VEIT [page 840, section 1. Introduction] teaches: "We propose an alternative approach: instead of using the small clean dataset to learn visual representations directly, we use it to learn a mapping between noisy and clean annotations. We argue that this mapping not only learns the patterns of noise, but it also captures the structure in the label space. The learned mapping between noisy and clean annotations allows to clean the noisy dataset and fine-tune the network using both the clean and the full dataset with reduced noise." VEIT [page 841, section 3. Our Approach] teaches: "Formally, we have a very large training dataset
T
comprising tuples of noisy labels
y
and images
I
,
T
=
y
i
,
I
i
,
…
, and a small dataset
V
of triplets of verified labels
v
, noisy labels
y
and images
I
,
V
=
v
i
,
y
i
,
I
i
,
…
.” VEIT [page 842, section 3.1. Multi-Task Label Cleaning Architecture] teaches: "The first classifier is a label cleaning network denoted as
g
that models the structure in the label space and learns a mapping from the noisy labels
y
to the human verified labels
v
, conditional on the input image. We denote the cleaned labels output by
g
as
c
^
so that
c
^
=
g
(
y
,
I
)
. [...] In order to model the label structure and noise conditional on the image, the network has two separate inputs, the noisy labels
y
as well as the visual features
f
I
. […] Denoting the residual cleaning module as
g
'
, the label cleaning network
g
computes cleaned labels
c
^
=
c
l
i
p
(
y
+
g
'
y
,
f
I
,
0,1
Examiner's note: Under broadest reasonable interpretation, the first calibration model can be interpreted as the label cleaning network g, which learns the mapping between the noisy annotations (i.e., with respect to the […] first processing label) and clean annotations. The label cleaning network takes as input the noisy labels and images (i.e., by inputting the first processing data and the first processing label to the calibration model) to learn the mapping. Additionally, generate […] a calibration parameter can be interpreted as the output of the residual cleaning module
g
'
, which computes the residual (i.e., difference) between the noisy annotations and clean annotations, for generating the cleaned label
c
^
.)
train the target model based on the first processing data and the calibration data based on the generated calibration label or the calibration parameter and the first processing label. (VEIT [page 842, section 3.1. Multi-Task Label Cleaning Architecture] teaches: “The second classifier is an image classifier denoted as
h
that learns to annotate images by imitating the first classifier
g
by using
g
’s predictions as ground truth targets. We denote the predicted labels output by
h
as
p
^
so that
p
^
=
h
I
(i.e., train the target model based on the first processing data).” VEIT [page 842, section 3.2. Model Training] teaches: “For the image classifier (i.e., the target model), the supervision (i.e., train) depends on the source of the training sample. For all samples
j
from the noisy dataset
T
, the classifier is supervised by the cleaned labels
c
^
j
produced by the label cleaning network (i.e., train the target model based on […] the calibration data based on the generated calibration label).”)
VEIT is not relied upon for teaching, but ZHANG teaches: […] a […] calibration model […] a first processing label assigned by a first user to the first processing data, and outputs calibration data relating to calibration of individual characteristics in label assignment by the first user, based on […] (2) a first training label assigned by the first user to the first input training data, […] (ZHANG [page 1, Abstract] teaches: "In this work, we present a method for jointly learning, from purely noisy observations alone, the reliability of individual annotators and the true segmentation label distributions, using two coupled CNNs." ZHANG [page 4, Figure 1] teaches: "An architecture schematic in the presence of 3 annotators of varying characteristics (over-segmentation, under-segmentation and confusing between two classes, red and blue). The model consists of two parts: (1) segmentation network parametrised by
θ
that generates an estimate of the unobserved true segmentation probabilities,
p
θ
x
; (2) annotator network, parametrised by
ϕ
, that estimates the pixelwise confusion matrices (CMs),
A
ϕ
r
x
r
=
1
3
of the annotators for the given input image x.” ZHANG [page 5, Section 3.3] teaches: "Given training
X
=
x
n
n
=
1
N
and noisy labels
Y
~
(
r
)
=
y
~
n
r
:
r
∈
S
x
n
n
=
1
N
for
r
=
1
,
…
,
R
, we optimize the parameters
{
θ
,
ϕ
}
by minimizing the negative log-likelihood (NLL),
-
l
o
g
p
Y
~
1
,
…
,
Y
~
R
X
. From eqs. (1) and (2), this optimization objective equates to the sum of cross-entropy losses between the observed noisy segmentations and the estimated annotator label distributions:
PNG
media_image1.png
113
1114
media_image1.png
Greyscale
ZHANG [page 2, Section 1 Introduction] teaches: "In this work, we introduce the first instance of an end-to-end supervised segmentation method that jointly estimates, from noisy labels alone, the reliability of multiple human annotators and true segmentation labels. The proposed architecture (Fig. 1) consists of two coupled CNNs where one estimates the true segmentation probabilities and the other models the characteristics of individual annotators (e.g., tendency to over-segmentation, mix-up between different classes, etc.) by estimating the pixel-wise confusion matrices (CMs) on a per image basis." Examiner's note: A calibration model can be reasonably interpreted as the "annotator network, parametrised by
ϕ
" which outputs the confusion matrices. The training process also uses noisy segmentation labels (i.e., first processing label) provided by the annotator for an image
x
n
in the loss function (ZHANG [Eq. 4 and Figure 1]).)
[…] a label calibrated in the individual characteristics included in the first processing label using the calibration data, […] (ZHANG [page 5, section 3.3] teaches the loss function used to minimize the loss of the annotator network and segmentation network. Under broadest reasonable interpretation, a label calibrated in the individual characteristics included in the first processing label using the calibration data, […] can be interpreted as ZHANG’s confusion matrices (CMs)
A
^
ϕ
r
x
n
, which model the individual characteristics of each annotator (e.g., tendency to over-segmentation, mix-up between different classes, etc.) for an input image
x
n
.)
Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of VEIT and ZHANG before them, to include ZHANG’s user annotations and confusion matrices in VEIT’s noisy label learning method. One would have been motivated to make such a combination in order to disentangle annotators’ mistakes and the true labels based on correlations between similar image samples (ZHANG [page 2, 1 Introduction]), to estimate individual annotators and improve robustness against label noise and for educating inexperienced annotators by using the estimated spatial characteristics of segmentation mistakes for insights into the annotation behaviors (ZHANG [page 9, 5. Discussion and Conclusion]).
Regarding Claim 2:
VEIT in view of ZHANG teaches the elements of claim 1 as outlined above. VEIT further teaches:
wherein the processing circuitry is further configured to: generate, as the calibration data, the calibration parameter with respect to the first processing label by inputting the first processing data and the first processing label to the first calibration model; (VEIT [page 844, section 4.4. Training Details] teaches: “We trained the baseline model on 50 NVIDIA K40 GPUs using the noisy labels from the Open Images training set. We stopped training after 49 million mini-batches (with 32 images each). This network is the starting point for all model variants.” VEIT [page 840, section 1. Introduction] teaches: "We propose an alternative approach: instead of using the small clean dataset to learn visual representations directly, we use it to learn a mapping between noisy and clean annotations. We argue that this mapping not only learns the patterns of noise, but it also captures the structure in the label space. The learned mapping between noisy and clean annotations allows to clean the noisy dataset and fine-tune the network using both the clean and the full dataset with reduced noise." VEIT [page 841, section 3. Our Approach] teaches: "Formally, we have a very large training dataset
T
comprising tuples of noisy labels
y
and images
I
,
T
=
y
i
,
I
i
,
…
, and a small dataset
V
of triplets of verified labels
v
, noisy labels
y
and images
I
,
V
=
v
i
,
y
i
,
I
i
,
…
.” VEIT [page 842, section 3.1. Multi-Task Label Cleaning Architecture] teaches: "The first classifier is a label cleaning network denoted as
g
that models the structure in the label space and learns a mapping from the noisy labels
y
to the human verified labels
v
, conditional on the input image. We denote the cleaned labels output by
g
as
c
^
so that
c
^
=
g
(
y
,
I
)
. [...] In order to model the label structure and noise conditional on the image, the network has two separate inputs, the noisy labels
y
as well as the visual features
f
I
. […] Denoting the residual cleaning module as
g
'
, the label cleaning network
g
computes cleaned labels
c
^
=
c
l
i
p
(
y
+
g
'
y
,
f
I
,
0,1
"
)
Examiner's note: Under broadest reasonable interpretation, the first calibration model can be interpreted as the label cleaning network g, which learns the mapping between the noisy annotations (i.e., with respect to the […] first processing label) and clean annotations. The label cleaning network takes as input the noisy labels and images (i.e., by inputting the first processing data and the first processing label to the calibration model) to learn the mapping. Additionally, generate […] a calibration parameter can be interpreted as the output of the residual cleaning module
g
'
, which computes the residual (i.e., difference) between the noisy annotations and clean annotations, for generating the cleaned label
c
^
.)
VEIT further teaches: generate the calibration label by applying the calibration parameter to the first processing label; (VEIT [page 842, section 3.1. Multi-Task Label Cleaning Architecture] teaches: "Denoting the residual cleaning module as
g
'
, the label cleaning network
g
computes cleaned labels
c
^
=
c
l
i
p
(
y
+
g
'
y
,
f
I
,
0,1
Examiner’s note: Under broadest reasonable interpretation, generate […] the calibration label can be interpreted as
c
^
, which results from applying the output of the residual cleaning module
g
'
to the noisy label
y
(i.e., first processing label). The residual cleaning module computes the residual (i.e., difference) between the noisy annotations and clean annotations, for generating the cleaned label
c
^
.)
train the target model based on the first processing data and the first calibration label. (VEIT [page 842, section 3.1. Multi-Task Label Cleaning Architecture] teaches: “The second classifier is an image classifier denoted as
h
that learns to annotate images by imitating the first classifier
g
by using
g
’s predictions as ground truth targets. We denote the predicted labels output by
h
as
p
^
so that
p
^
=
h
I
(i.e., train the target model based on the first processing data).” VEIT [page 842, section 3.2. Model Training] teaches: “For the image classifier (i.e., the target model), the supervision (i.e., train) depends on the source of the training sample. For all samples
j
from the noisy dataset
T
, the classifier is supervised by the cleaned labels
c
^
j
produced by the label cleaning network (i.e., based on […] the first calibration label).” Examiner’s note: Under broadest reasonable interpretation, first processing data can be interpreted as the image
I
in each tuple sample
j
from dataset
T
. The label cleaning network takes as input the noisy labels and images (i.e., by inputting the first processing data and the first processing label to the calibration model) to learn the mapping. Additionally, generate […] a calibration parameter can be interpreted as the output of the residual cleaning module
g
'
, which computes the residual (i.e., difference) between the noisy annotations and clean annotations, for generating the cleaned label
c
^
.)
Regarding Claim 3:
VEIT in view of ZHANG teaches the elements of claim 1 as outlined above. VEIT further teaches:
wherein the processing circuitry is further configured to: generate, as the calibration data, the calibration label by inputting the first processing data and the first processing label to the first calibration model; (VEIT [page 844, section 4.4. Training Details] teaches: “We trained the baseline model on 50 NVIDIA K40 GPUs (i.e., processing circuitry) using the noisy labels from the Open Images training set. We stopped training after 49 million mini-batches (with 32 images each). This network is the starting point for all model variants.” VEIT [page 840, section 1. Introduction] teaches: "We propose an alternative approach: instead of using the small clean dataset to learn visual representations directly, we use it to learn a mapping between noisy and clean annotations. We argue that this mapping not only learns the patterns of noise, but it also captures the structure in the label space. The learned mapping between noisy and clean annotations allows to clean the noisy dataset and fine-tune the network using both the clean and the full dataset with reduced noise." VEIT [page 841, section 3. Our Approach] teaches: "Formally, we have a very large training dataset
T
comprising tuples of noisy labels
y
and images
I
,
T
=
y
i
,
I
i
,
…
, and a small dataset
V
of triplets of verified labels
v
, noisy labels
y
and images
I
,
V
=
v
i
,
y
i
,
I
i
,
…
.” VEIT [page 842, section 3.1. Multi-Task Label Cleaning Architecture] teaches: "The first classifier is a label cleaning network denoted as
g
that models the structure in the label space and learns a mapping from the noisy labels
y
to the human verified labels
v
, conditional on the input image. We denote the cleaned labels output by
g
as
c
^
so that
c
^
=
g
(
y
,
I
)
. [...] In order to model the label structure and noise conditional on the image, the network has two separate inputs, the noisy labels
y
as well as the visual features
f
I
. […] Denoting the residual cleaning module as
g
'
, the label cleaning network
g
computes cleaned labels
c
^
=
c
l
i
p
(
y
+
g
'
y
,
f
I
,
0,1
Examiner's note: Under broadest reasonable interpretation, the first calibration model can be interpreted as the label cleaning network g, which learns the mapping between the noisy annotations (i.e., with respect to the […] first processing label) and clean annotations. The label cleaning network takes as input the noisy labels and images (i.e., by inputting the first processing data and the first processing label to the calibration model) to learn the mapping. Additionally, generate […] the calibration label can be interpreted as
c
^
, which results from applying the output of the residual cleaning module
g
'
to the noisy label
y
(i.e., first processing label). The residual cleaning module computes the residual (i.e., difference) between the noisy annotations and clean annotations, for generating the cleaned label
c
^
.)
train the target model based on the first processing data and the generated calibration label. (VEIT [page 842, section 3.1. Multi-Task Label Cleaning Architecture] teaches: “The second classifier is an image classifier denoted as
h
that learns to annotate images by imitating the first classifier
g
by using
g
’s predictions as ground truth targets. We denote the predicted labels output by
h
as
p
^
so that
p
^
=
h
I
(i.e., train the target model based on the first processing data).” VEIT [page 842, section 3.2. Model Training] teaches: “For the image classifier (i.e., the target model), the supervision (i.e., train) depends on the source of the training sample. For all samples
j
from the noisy dataset
T
, the classifier is supervised by the cleaned labels
c
^
j
produced by the label cleaning network (i.e., based on […] the generated calibration label).” Examiner’s note: Under broadest reasonable interpretation, first processing data can be interpreted as the image
I
in each tuple sample
j
from dataset
T
. The label cleaning network takes as input the noisy labels and images to learn the mapping for generating the cleaned label
c
^
.)
Regarding Claim 4:
VEIT in view of ZHANG teaches the elements of claim 1 as outlined above. VEIT further teaches:
wherein the processing circuitry is further configured to: […] train the target model based on the first processing data and the first processing label […] (VEIT [page 844, 4.4. Training Details] teaches: “We trained the baseline model on 50 NVIDIA K40 GPUs using the noisy labels from the Open Images training set. […] The four different fine-tuning variants are trained for additional 4 million batches each.” VEIT [page 842, section 3.1. Multi-Task Label Cleaning Architecture] teaches: “We denote the cleaned labels output by
g
as
c
^
so that
c
^
=
g
y
,
I
(i.e., y is the first processing label). […] The second classifier is an image classifier denoted as
h
that learns to annotate images by imitating the first classifier
g
(i.e., train the target model) by using
g
’s predictions as ground truth targets. […] The image classifier
h
is shown in the bottom row of Figure 3. First, a sample image (i.e., based on the first processing data) is processed by the convolutional network to compute high level image features. Then, these features are passed through a fully-connected layer
w
followed by a sigmoid
σ
,
h
=
σ
w
f
I
. […] Denoting the residual cleaning module as
g
'
, the label cleaning network
g
computes cleaned labels
c
^
=
c
l
i
p
y
+
g
'
y
,
f
I
,
0
,
1
1
ZHANG further teaches: output, as the calibration data, a reliability with respect to the first processing label by inputting the first processing data and the first processing label to the first calibration model; (ZHANG [page 5, section 3.3] teaches: "[…] and
t
r
(
A
)
denotes the trace of matrix A. The mean trace represents the average probability that a randomly selected annotator provides an accurate label (i.e., reliability)." Additionally, ZHANG [page 4, Figure 1] teaches a term of the loss (i.e., parameter of a loss function)
[
…
]
+
∑
r
=
1
3
t
r
(
A
^
θ
r
x
)
=
L
t
o
t
a
l
(
θ
,
ϕ
)
where the mean trace
t
r
(
A
^
θ
r
x
) (i.e., reliability) denotes the average probability that a randomly selected annotator provides an accurate label (i.e., with respect to the first processing label) to an image
x
n
. Furthermore, under BRI, the annotator network (i.e., first calibration model) receives both the input image
x
n
(i.e., first processing data) and, through the loss, the noisy segmentation label (i.e., first processing label) when computing the mean trace.)
train the […] model based on the first processing data and the first processing label by using the reliability as a parameter of a loss function. (ZHANG [page 5, section 3.3] teaches the loss function used to minimize the loss of (i.e., train) the annotator network and segmentation network (i.e., the […] model). The first processing data can be interpreted as the input image
x
n
. Furthermore, ZHANG [page 5, Eq (4)] teaches noisy labels
Y
~
(
r
)
=
y
~
n
r
:
r
∈
S
x
n
n
=
1
N
, which are the labels assigned (i.e., first processing label) by the annotators. Additionally, ZHANG [page 4, Figure 1] teaches the loss
∑
r
=
1
3
t
r
(
A
^
θ
r
x
)
=
L
t
o
t
a
l
(
θ
,
ϕ
)
where the mean trace
t
r
(
A
^
θ
r
x
) (i.e., reliability), a term in the loss function (i.e., parameter of a loss function), denotes the average probability that a randomly selected annotator provides an accurate label.)
Regarding Claim 6:
VEIT in view of ZHANG teaches the elements of claim 1 as outlined above. VEIT further teaches:
wherein the processing circuitry is further configured to […] (VEIT [page 844, 4.4. Training Details] teaches: “We trained the baseline model on 50 NVIDIA K40 GPUs using the noisy labels from the Open Images training set. […] The four different fine-tuning variants are trained for additional 4 million batches each.”)
ZHANG further teaches: […] assign the first training label to the first input training data in accordance with an instruction by the first user. (ZHANG [page 3, section 3 Method] teaches: “Specifically, we consider a scenario where set of images (i.e., first input training data) […] are assigned with noisy segmentation labels (i.e., assign the first training label) […] from multiple annotators where
y
~
n
r
denotes the label from annotator
r
∈
{
1
,
…
,
R
}
[…]”.)
Regarding Claim 7:
VEIT in view of ZHANG teaches the elements of claim 1 as outlined above. ZHANG further teaches:
wherein the first input training data is medical data generated by a medical device. (ZHANG [page 1, Abstract] teaches: “We then demonstrate the utility of the method on three public medical imaging segmentation datasets with simulated (when necessary) and real diverse annotations: 1) MSLSC (multiple-sclerosis lesions); 2) BraTS (brain tumours); 3) LIDC-IDRI (lung abnormalities).” Examiner’s note: MSLSC, BraTS, and LIDC-IDRI are all examples of MRI scan datasets (i.e., MR or magnetic resonance).)
Regarding Claim 9:
VEIT in view of ZHANG teaches the elements of claim 1 as outlined above. VEIT further teaches:
wherein the processing circuitry is further configured to: […] the first training label, a first calibration parameter for calibrating […] (VEIT [page 844, 4.4. Training Details] teaches: “We trained the baseline model on 50 NVIDIA K40 GPUs using the noisy labels from the Open Images training set. […] The four different fine-tuning variants are trained for additional 4 million batches each.” VEIT [page 842, section 3.1. Multi-Task Label Cleaning Architecture] teaches: “Due to the residual connection, the network only needs to learn the difference between the noisy (i.e., the first training label) and clean labels (i.e., a first calibration parameter for calibrating) instead of regressing the entire label vector. […] Denoting the residual cleaning module as
g
'
, the label cleaning network
g
computes cleaned labels
c
^
=
c
l
i
p
(
y
+
g
'
y
,
f
I
,
0,1
"
)
Examiner’s note: Under BRI, a first calibration parameter can be interpreted as
g
'
y
,
f
I
which is calculated from the label
y
in order to compute a cleaned label
c
^
.)
train, based on the first input training data, the first training label, […], the first calibration model configured to receive input data as input, and to output as the calibration data the calibration parameter corresponding to the input data. (VEIT [page 841, section 3. Our Approach] teaches: “Formally, we have a very large training dataset
T
comprising tuples of noisy labels
y
and images
I
,
T
=
y
i
,
I
i
,
…
, and a small dataset
V
of triplets of verified labels
v
, noisy labels
y
(i.e., the first training label) and images
I
(i.e., the first input training data)
,
V
=
v
i
,
y
i
,
I
i
,
…
.” VEIT [page 842, section 3.1. Multi-Task Label Cleaning Architecture] teaches: “The first classifier is a label cleaning network denoted as
g
(i.e., the first calibration model) that models the structure in the label space and learns a mapping from the noisy labels
y
to the human verified labels
v
, conditional on the input image. We denote the cleaned labels output by
g
as
c
^
so that
c
^
=
g
y
,
I
(i.e., and outputs calibration data relating to calibration of […] label assignment […]). […] The label cleaning network
g
is shown in the top row of Figure 3. In order to model the label structure and noise conditional on the image, the network has two separate inputs (i.e., that receives, as input), the noisy labels
y
(i.e., a first processing label assigned […] to the first processing data) as well as the visual features
f
I
(i.e., first processing data). […] Denoting the residual cleaning module as
g
'
, the label cleaning network
g
computes cleaned labels
c
^
=
c
l
i
p
y
+
g
'
y
,
f
I
,
0
,
1
1
VEIT [page 842, section 3.2. Model Training] teaches: “The label cleaning network is supervised by (i.e., train, based on) the verified labels (i.e., first training label) of all samples
i
(i.e., the first input training data) in the human rated set
V
.” VEIT [page 844, section 4.3. Baselines and Model Variants] teaches: “For first variant, we initially train just the label cleaning network on the human rated data. Then, subsequently we train the cleaning network and the classification layer jointly.” Examiner’s note: The label cleaning network trains on images
I
to learn a mapping from noisy labels
y
to the human verified labels
v
, thus outputting a label for each sample
i
in images
I
(i.e., configured to receive input data as input, and to output as the calibration data the calibration parameter corresponding to the input data).)
ZHANG further teaches: train, based on the […] input training data, the […] training label, and the first calibration parameter, the […] calibration model […]. (ZHANG [page 5, Eq (4)] teaches the training using a loss function to be minimized, which includes the annotator network (i.e., calibration model) given (i.e., based on) training input
X
=
x
n
n
=
1
N
(i.e., input training data), noisy labels (i.e., training label)
Y
~
(
r
)
=
y
~
n
r
:
r
∈
S
x
n
n
=
1
N
, which are the labels assigned by the annotators, and confusion matrices (i.e., the first calibration parameter) based on the annotator’s individual tendencies when performing segmentation.)
[…] a […] calibration parameter for calibrating individual characteristics in label assignment by the first user; (ZHANG [page 4, Figure 1] teaches the annotator network that outputs confusion matrices (i.e., a calibration parameter) based on the input image
x
n
. Additionally, the confusion matrices, as described in Figure 1, contain estimations of individual characteristics of each annotator in order to estimate the annotator's individual tendencies when performing segmentation (i.e., first training label).)
Regarding Claim 10:
VEIT in view of ZHANG teaches the elements of claim 1 as outlined above. VEIT further teaches:
wherein the processing circuitry is further configured to train, based on the first input training data, the first training label, and a correct label with respect to the first input training data, the first calibration model that receives, as input, input data and a label […] to the input data and outputs, as the calibration data, a correct label for the input data. (VEIT [page 844, 4.4. Training Details] teaches: “We trained the baseline model on 50 NVIDIA K40 GPUs using the noisy labels from the Open Images training set. […] The four different fine-tuning variants are trained for additional 4 million batches each.” VEIT [page 842, section 3.1. Multi-Task Label Cleaning Architecture] teaches: "The first classifier is a label cleaning network denoted as
g
that models the structure in the label space and learns a mapping from the noisy labels
y
to the human verified labels
v
, conditional on the input image. We denote the cleaned labels output by
g
as
c
^
so that
c
^
=
g
(
y
,
I
)
. [...] In order to model the label structure and noise conditional on the image, the network has two separate inputs (i.e., the first calibration model that receives, as input, input data and a label […] to the input data), the noisy labels
y
(i.e., the first training label) as well as the visual features
f
I
(i.e., based on the first input training data). […] Denoting the residual cleaning module as
g
'
, the label cleaning network
g
computes cleaned labels
c
^
=
c
l
i
p
(
y
+
g
'
y
,
f
I
,
0,1
VEIT [page 842, section 3.2. Model Training] teaches: “The label cleaning network is supervised (i.e., train […] the first calibration model) by the verified labels of all samples i in the human rated set
V
(i.e., based on a correct label with respect to the first input training data). The cleaning loss is based on the difference between the cleaned labels
c
^
i
and the corresponding ground truth verified labels
v
i
,
L
c
l
e
a
n
=
∑
i
∈
V
c
^
i
-
v
i
We choose the absolute distance as error measure, since the label vectors are very sparse.” VEIT [page 842, section 3.1. Multi-Task Label Cleaning Architecture] teaches: "The second classifier is an image classifier denoted as
h
that learns to annotate images by imitating the first classifier
g
by using
g
’s predictions as ground truth targets.” Examiner’s note: Under broadest reasonable interpretation, and outputs, as the calibration data, a correct label for the input data can be interpreted as
c
^
, which is the corrected label from the difference between the noisy annotations and clean annotations of the image
I
.)
ZHANG further teaches: […] a label assigned by the first user to the input data (ZHANG [page 5, Eq (4)] teaches the training using a loss function to be minimized, which includes the annotator network given training input
X
=
x
n
n
=
1
N
, noisy labels
Y
~
(
r
)
=
y
~
n
r
:
r
∈
S
x
n
n
=
1
N
, which are the labels assigned to inputs
x
n
(i.e., a label assigned […] to the input data) by the annotators (i.e., by the first user), and confusion matrices based on the annotator’s individual tendencies when performing segmentation.)
Regarding Claim 11:
VEIT in view of ZHANG teaches the elements of claim 1 as outlined above. VEIT further teaches:
wherein the processing circuitry is further configured to: […] train the first calibration model that receives, as input, input data and a label […] the first input training data, the first training label […] (VEIT [page 844, 4.4. Training Details] teaches: “We trained the baseline model on 50 NVIDIA K40 GPUs using the noisy labels from the Open Images training set. […] The four different fine-tuning variants are trained for additional 4 million batches each.” VEIT [page 842, section 3.1. Multi-Task Label Cleaning Architecture] teaches: "The first classifier is a label cleaning network denoted as
g
that models the structure in the label space and learns a mapping from the noisy labels
y
to the human verified labels
v
, conditional on the input image. We denote the cleaned labels output by
g
as
c
^
so that
c
^
=
g
(
y
,
I
)
. [...] In order to model the label structure and noise conditional on the image, the network has two separate inputs (i.e., the first calibration model that receives, as input, input data and a label), the noisy labels
y
(i.e., the first training label) as well as the visual features
f
I
(i.e., the first input training data). […] Denoting the residual cleaning module as
g
'
, the label cleaning network
g
computes cleaned labels
c
^
=
c
l
i
p
(
y
+
g
'
y
,
f
I
,
0,1
VEIT [page 842, section 3.2. Model Training] teaches: “The label cleaning network is supervised (i.e., train […] the first calibration model) by the verified labels of all samples i in the human rated set
V
. The cleaning loss is based on the difference between the cleaned labels
c
^
i
and the corresponding ground truth verified labels
v
i
,
L
c
l
e
a
n
=
∑
i
∈
V
c
^
i
-
v
i
We choose the absolute distance as error measure, since the label vectors are very sparse.” VEIT [page 842, section 3.1. Multi-Task Label Cleaning Architecture] teaches: "The second classifier is an image classifier denoted as
h
that learns to annotate images by imitating the first classifier
g
by using
g
’s predictions as ground truth targets.”)
ZHANG further teaches: wherein the processing circuitry is further configured to: determine a reliability of the first training label with respect to the […] input training data; (ZHANG [page 5, section 3.3] teaches: "[…] and
t
r
(
A
)
denotes the trace of matrix A. The mean trace represents the average probability that a randomly selected annotator provides an accurate label (i.e., reliability)." Additionally, ZHANG [page 4, Figure 1] teaches a term of the loss (i.e., parameter of a loss function)
[
…
]
+
∑
r
=
1
3
t
r
(
A
^
θ
r
x
)
=
L
t
o
t
a
l
(
θ
,
ϕ
)
where the mean trace
t
r
(
A
^
θ
r
x
) (i.e., reliability) denotes the average probability that a randomly selected annotator provides an accurate label to an image
x
n
(i.e., with respect to the […] input training data).)
train the […] calibration model that receives, as input, input data and a label assigned by the first user to the input data based on the […] input training data, the […] training label, and the reliability of the […] training label, and outputs a reliability of the label with respect to the input data. (ZHANG [Figure 1; page 5, Eq (4)] teaches the training of the annotator network (i.e., train the […] calibration model) via a loss function. The loss includes an input training image
x
n
(i.e., input training data), a noisy label assigned by the annotator (i.e., training label), and the mean trace, which represents the average probability that an annotator provides an accurate label (i.e., reliability).)
Regarding Claim 12:
VEIT in view of ZHANG teaches the elements of claim 1 as outlined above. VEIT further teaches:
[…] the target model […] (VEIT [page 842, section 3.1. Multi-Task Label Cleaning Architecture] teaches: “The second classifier is an image classifier denoted as
h
(i.e., the target model) that learns to annotate images by imitating the first classifier
g
by using
g
’s predictions as ground truth targets.”)
[…] the first calibration model […] (VEIT [page 842, section 3.1. Multi-Task Label Cleaning Architecture] teaches: “The first classifier is a label cleaning network denoted as
g
(i.e., the first calibration model) that models the structure in the label space and learns a mapping from the noisy labels
y
to the human verified labels
v
, conditional on the input image.”)
ZHANG further teaches: wherein the processing circuitry is further configured to train the […] model based on a combination of the first processing data, the first processing label, and the […] calibration model, and a combination of second processing data, a second processing label assigned by a second user to the second processing data, and a second calibration model for calibrating individual characteristics in label assignment by the second user. (ZHANG [page 4, Figure 1] teaches both the annotator network (i.e., first/second calibration model) and the segmentation network (i.e., the […] model) being trained via a loss function by using noisy labels from multiple annotators (i.e., combination), with each annotator providing a noisy label (i.e., first/second processing label) for an input image (i.e., first/second processing data). Under BRI, the “second calibration model” can be interpreted as the model configured with different parameters than the first annotator when receiving input data (i.e., labels) and learning the varying characteristics of another annotator’s labeling style, as described in ZHANG’s Figure 1. Furthermore, ZHANG [page 2, Section 1 Introduction] teaches: "In this work, we introduce the first instance of an end-to-end supervised segmentation method that jointly estimates, from noisy labels alone, the reliability of multiple human annotators (i.e., a second processing label assigned by a second user) and true segmentation labels. The proposed architecture (Fig. 1) consists of two coupled CNNs where one estimates the true segmentation probabilities and the other models the characteristics of individual annotators (e.g., tendency to over-segmentation, mix-up between different classes, etc.) by estimating the pixel-wise confusion matrices (CMs) on a per image basis.")
Regarding Claim 14:
VEIT in view of ZHANG teaches the elements of claim 1 as outlined above. VEIT further teaches:
wherein the target model is a machine learning model trained in such a manner as to receive input processing data as input and to output prediction data corresponding to the input processing data. (VEIT [page 842, section 3.1. Multi-Task Label Cleaning Architecture] teaches: “The second classifier (i.e., the target model) is an image classifier (i.e., is a machine learning model) denoted as
h
that learns to annotate images (i.e., to output prediction data corresponding to the input processing data) by imitating the first classifier
g
by using
g
’s predictions as ground truth targets. […] The image classifier
h
is shown in the bottom row of Figure 3. First, a sample image is processed (i.e., trained in such a manner as to receive input processing data as input) by the convolutional network to compute high level image features.”)
Regarding Claim 17:
The claim recites similar limitations as corresponding claim 1 and is rejected for similar reasons as claim 1 using similar teachings and rationale.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over VEIT in view of ZHANG, as applied to claim 1 above, and further in view of SHAMIR (US 20210196207 A1), hereinafter SHAMIR.
Regarding Claim 8:
VEIT in view of ZHANG teaches the elements of claim 1 as outlined above. ZHANG further teaches:
wherein the first input training data is an MR image (ZHANG [page 1, Abstract] teaches: “We then demonstrate the utility of the method on three public medical imaging segmentation datasets with simulated (when necessary) and real diverse annotations: 1) MSLSC (multiple-sclerosis lesions); 2) BraTS (brain tumours); 3) LIDC-IDRI (lung abnormalities).” Examiner’s note: MSLSC, BraTS, and LIDC-IDRI are all examples of MRI scan datasets (i.e., MR or magnetic resonance).)
However, VEIT in view of ZHANG is not relied upon for teaching, but SHAMIR teaches: in which a measurement voxel by MR spectroscopy with a magnetic resonance imaging apparatus is set, (SHAMIR [0068] teaches: “The medical images may include images derived (i.e., set) from magnetic resonance imaging (MRI), radiography, ultrasound, elastography, photoacoustic imaging, positron emission tomography, echocardiography, magnetic particle imaging, functional near-infrared spectroscopy, and/or any other imaging technique.”)
and the first training label is a mark indicative of a position of the measurement voxel. (SHAMIR [0068] teaches: “In some instances, features extracted from each image may indicate a specific area of a patient's body, such as a head or torso, where voxels are labeled to indicate segmentation of a tumor and different tissue types, such as: combined skin and muscle tissue, skull/bone, cerebrospinal fluid, gray matter, and white matter. […] For example, a marker in an image, such as a voxel, pixel, and/or the like may be correlated to a point in a patient's anatomy from which a tissue sample is excised.”)
Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of VEIT, ZHANG and SHAMIR before them, to include SHAMIR’s MR spectroscopy training data in VEIT and ZHANG’s segmentation method. One would have been motivated to make such a combination in order to improve the accuracy of the model during training (SHAMIR [0086-0087]).
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over VEIT in view of ZHANG, as applied to claim 1 above, and further in view of HOULSBY (“Parameter-Efficient Transfer Learning for NLP”), hereinafter HOULSBY.
Regarding Claim 15:
VEIT in view of ZHANG teaches the elements of claim 1 as outlined above. ZHANG further teaches:
[…] model corresponding to a user other than the first user […] (Under BRI, the “model corresponding to a user other than the first user” can be interpreted as the annotator network configured with different parameters
ϕ
than the parameters of a first user based on the received input data (i.e., labels) and learned characteristics of the annotator’s labeling style, as described in ZHANG’s Figure 1, which shows different confusion matrices for each individual annotator.)
based on the first input training data, the first training label, and the calibration training data. (ZHANG [page 5, Section 3.3] teaches input training image
x
n
(i.e., first input training data), noisy labels assigned by annotators (i.e., first training label), and confusion matrix, or mean trace (i.e., calibration training data).)
However, VEIT in view of ZHANG is not relied upon for teaching, but HOULSBY teaches: train an untrained first calibration model by copying a training parameter of a trained calibration […] to the untrained first calibration model, and training a training parameter of the untrained first calibration model (HOULSBY [page 1, section 1 Introduction] teaches: “The embeddings are then fed to custom downstream models (i.e., untrained first calibration models). Fine-tuning involves copying the weights (i.e., training parameters) from a pre-trained network and tuning (i.e., train an untrained first calibration model) them (i.e., training parameter) on the downstream task.”)
Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of VEIT, ZHANG, and HOULSBY before them, to include HOULSBY’s transfer learning in VEIT and ZHANG’s segmentation method. One would have been motivated to make such a combination in order to build a system that performs well, without having to build the system from scratch (HOULSBY [page 1, section 1 Introduction]).
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over VEIT in view of ZHANG, as applied to claim 1 above, and further in view of GUAN (“Who Said What: Modeling Individual Labelers Improves Classification”), hereinafter GUAN.
Regarding Claim 16:
VEIT in view of ZHANG teaches the elements of claim 1 as outlined above. However, VEIT in view of ZHANG is not relied upon for teaching, but GUAN teaches:
wherein the first calibration model includes a plurality of calibration models respectively corresponding to a plurality of users other than the first user, (GUAN [page 4, Summary of Procedure] teaches: “We learn a doctor model for each doctor.”)
and an addition layer that weight-adds a plurality of calibration data pieces from the calibration models (GUAN [page 4, Model Architecture] teaches: “Weighted Doctor Net (WDN): Fixed DN with averaging weights for combining the predictions of the doctor models learned on top, one weight per doctor model.”)
in accordance with a weight trained for calibration of label assignment by the first user, and outputs the calibration data pieces. (GUAN [page 5, Summary of Procedure] teaches: “During evaluation of WDN, the prediction made for our model is a linear combination of the doctor models predictions (i.e., label assignment) where the coefficients are the averaging weights learned (i.e., trained for calibration) in Phase 2 of training.” Examiner’s note: under BRI, each weight in the linear combination can be interpreted as the calibration data pieces.)
Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of VEIT, ZHANG, and GUAN before them, to include GUAN’s models for each doctor in VEIT and ZHANG’s segmentation method. One would have been motivated to make such a combination in order to “make more effective use of noisy labels when every example is labeled by a subset of a larger pool of experts (GUAN [page 8, Conclusion])”.
Allowable Subject Matter
Claim 13 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Alvaro S Laham Bauzo whose telephone number is (571) 272-5650. The examiner can normally be reached Mon-Fri 7:30 AM - 11:00 AM | 1:00 PM - 5:30 PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Usmaan Saeed can be reached on (571) 272-4046. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/A.S.L./Examiner, Art Unit 2146
/USMAAN SAEED/Supervisory Patent Examiner, Art Unit 2146