Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 04/12/2024 was filed and is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1 and 8-10 are rejected under 35 U.S.C. 102 (a)(1) as being anticipated by Upretee et al. (FixMatchSeg: Fixing FixMatch for Semi-Supervised Semantic Segmentation, hereinafter Upretee).
Regarding claims 1, 9, and 10, Upretee discloses
Claim 1: A medical image processing method comprising:
Claim 9: A medical image processing apparatus comprising processing circuitry configured:
Claim 10: A non-transitory computer-readable storage medium storing therein a program that causes a computer to perform:
a labeled image data training step of training a deep neural network used for performing medical image processing, by using labeled image data being input (Section 3 Para. 1: “Using the notations of FixMatch [1] for consistency, let
X
=
{
(
x
b
,
p
b
)
:
b
|
∈
1
,
…
,
B
}
be a labeled data set of size
B
, where
x
b
is an image and
p
b
is the corresponding ground truth mask. Let unlabeled batch be represented by a set
U
=
{
u
b
:
b
∈
(
1
,
…
,
μ
B
)
}
where
μ
is a hyperparameter of the model determining the ratio of the unlabeled data to the labeled data. Thus, we have
B
labeled examples and
μ
B
unlabeled examples in the training data.”, Section 3 Para. 4: “
PNG
media_image1.png
89
536
media_image1.png
Greyscale
p
b
is the ground truth label and
p
m
is the predicted mask”);
a first augmenting step of obtaining a first augmented image by carrying out a weak data augmentation on unlabeled image data being input (P. 5 Para. 1: “For each unlabeled input image
u
b
, we obtained an artificial label from its weakly augmented version
α
(
u
b
)
.”);
an attention setting step of performing a predicting process on the first augmented image by using the deep neural network and determining whether or not each of pixels in the first augmented image is able to serve as a pseudo-label on a basis of prediction information of the pixel (Section 3 Para. 3: “FixMatchSeg uses two types of image augmentations: strong denoted by and weak denoted by
α
…For weak augmentation
α
,we chose random rotation and elastic distortions.”, P. 5 Para. 1: “We compute pixel-wise max of this image to obtain
q
b
, and compute pixel-wise argmax to obtain pseudo-label,
q
b
^
=
a
r
g
m
a
x
[
p
m
(
y
|
α
(
u
b
)
)
]
. Thus,
q
b
^
is the segmentation output predicted by the model for the weakly augmented unlabeled image. In order to decide whether to use
q
b
^
as a pseudo-label or not, we first compute the confidence score as the mean of the pixel values of
q
b
, denoted by
q
b
-
which gives us the average maximum confidence of the model in predicting different classes over the whole image. Note that while
q
b
^
is an integer image,
q
b
-
is a scalar number. If the average
q
b
-
is higher than the confidence threshold τ , it is considered as a pseudo-label, i.e. ground truth for the strongly augmented unlabeled image A(
α
(
u
b
)
)
.”);
a second augmenting step of obtaining a second augmented image by carrying out a strong data augmentation on the first augmented image (Section 3 Para. 3: “FixMatchSeg uses two types of image augmentations: strong denoted by and weak denoted by
α
… For strong augmentation A, we modified the sharpness and contrast of the weakly augmented images and added Gaussian blur. In order to have the same output target from the weakly augmented and strongly augmented images, instead of applying strong augmentation directly to the unlabeled image, we applied it to the weakly augmented version of that image. For the strong augmentation, we did not apply any kind of geometric or shape-changing transformation, which thus maintains the same geometrical shape of the objects in both weakly and strongly augmented images.”);
an unlabeled image data training step of training the deep neural network, by using the second augmented image and the pseudo-labels determined at the attention setting step (P. 5 Para. 1: “If the average
q
b
-
is higher than the confidence threshold τ , it is considered as a pseudo-label, i.e. ground truth for the strongly augmented unlabeled image A(
α
(
u
b
)
)
… Thus, we have unsupervised loss defined as follows:
PNG
media_image2.png
86
800
media_image2.png
Greyscale
Then, the total loss for FixMatchSeg,
l
is given by
l
=
l
s
+
λ
u
l
u
, where
λ
u
is the weight for the unsupervised loss.”) ; and
an image processing step of processing a medical image being input, by using the deep neural network updated on a basis of a training result of the labeled image data and a training result of the unlabeled image data (Table 1-4, Fig. 2).
Regarding claim 8, dependent upon claim 1, Upretee discloses everything regarding claim 1.
Upretee further discloses
at the image processing step, at least one selected from between segmentation of a medical anatomical structure and segmentation in units of organ functions is performed on the medical image being input (Table 1-4, Fig. 2).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 2-4 and 6 are rejected under 35 U.S.C. 103 as being unpatentable over Upretee et al. (FixMatchSeg: Fixing FixMatch for Semi-Supervised Semantic Segmentation, hereinafter Upretee) in view of Qi et al. (PICS FOR SEMISUPERVISED REMOTE SENSING IMAGES SEMANTIC SEGMENTATION, hereinafter Qi).
Regarding claim 2, dependent upon claim 1, Upretee discloses everything regarding claim 1.
Upretee further discloses
a pseudo-label determining step of judging whether or not the probability map average value corresponding to each of the pixels in the first augmented image is larger than a prescribed threshold value and determining the probability map average values of certain pixels larger than the prescribed threshold value as the pseudo-labels (P. 5 Para. 1: “If the average
q
b
-
is higher than the confidence threshold τ , it is considered as a pseudo-label, i.e. ground truth for the strongly augmented unlabeled image A(
α
(
u
b
)
)
”).
However Upretee does not explicitly disclose
a probability map average value obtaining step of obtaining probability maps by performing the predicting process on the first augmented image while using the deep neural network and calculating probability map average values of the first augmented image; and
generating probability map average value corresponding to each of the pixels.
Qi teaches
a probability map average value obtaining step of obtaining probability maps by performing the predicting process on the first augmented image while using the deep neural network and calculating probability map average values of the first augmented image; and generating probability map average value corresponding to each of the pixels (Fig. 3, P. 5 Section 1 Pseudolabels’ Generation: “As shown in Fig. 3, the weak disturbance Disturb (∙) contains common augmentation operations, such as flipping and rotating. For each batch, the labeled image
X
is first stochastically augmented once, and
|
{
u
i
;
i
∈
|
(
1
,
…
,
N
)
}
. unlabeled images are stochastically augmented
K
times… Then, the consistency prediction result
P
i
-
for the original unlabeled image
u
i
-
is obtained through the average operation of the transformed probability maps, which is formulated as follows:
PNG
media_image3.png
121
229
media_image3.png
Greyscale
”).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Upretee with generating multiple probability maps of the image with different weak augmentation and average the probability maps to form a high quality pseudolabel of Qi as both Upretee and Qi utilizes pseudolabel for segmentation task and it is obvious for a person with ordinary skill in the art to try improving the probability map of Qi.
Regarding claim 3, dependent upon claim 2, Upretee in view of Qi teaches everything regarding claim 2.
Upretee further discloses
at the unlabeled image data training step, the deep neural network is trained by using the second augmented image, the pseudo-labels, and the reliability weights (P. 5 Para. 1: “If the average
q
b
-
is higher than the confidence threshold τ , it is considered as a pseudo-label, i.e. ground truth for the strongly augmented unlabeled image A(
α
(
u
b
)
)
… Thus, we have unsupervised loss defined as follows:
PNG
media_image2.png
86
800
media_image2.png
Greyscale
Then, the total loss for FixMatchSeg,
l
is given by
l
=
l
s
+
λ
u
l
u
, where
λ
u
is the weight for the unsupervised loss.”).
Qi further teaches
the attention setting step further includes: a reliability weight determining step of setting a reliability weight with respect to each of the pixels in the first augmented image, in correspondence with a magnitude of the probability map average value of the pixel (Fig. 3, P. 5 Section 1 Pseudolabels’ Generation: “As shown in Fig. 3, the weak disturbance Disturb (∙) contains common augmentation operations, such as flipping and rotating. For each batch, the labeled image
X
is first stochastically augmented once, and
|
{
u
i
;
i
∈
|
(
1
,
…
,
N
)
}
. unlabeled images are stochastically augmented
K
times… Then, the consistency prediction result
P
i
-
for the original unlabeled image
u
i
-
is obtained through the average operation of the transformed probability maps, which is formulated as follows:
PNG
media_image3.png
121
229
media_image3.png
Greyscale
”
Based on the specification of the current application, Para [0079] of the application publication, the reliability weight and the probability map average value can be the same).
Regarding claim 4, dependent upon claim 3, Upretee in view of Qi teaches everything regarding claim 3.
Upretee further discloses
at the unlabeled image data training step, the second augmented image is input to the deep neural network; a probability map of the second augmented image is predicted on a basis of the deep neural network; and a training result taking the reliability weights into consideration is obtained on a basis of the probability map of the second augmented image, the pseudo-labels, and the reliability weights of the pixels (Fig.1, P. 5 Para. 1: “If the average
q
b
-
is higher than the confidence threshold τ , it is considered as a pseudo-label, i.e. ground truth for the strongly augmented unlabeled image A(
α
(
u
b
)
)
… Thus, we have unsupervised loss defined as follows:
PNG
media_image2.png
86
800
media_image2.png
Greyscale
Then, the total loss for FixMatchSeg,
l
is given by
l
=
l
s
+
λ
u
l
u
, where
λ
u
is the weight for the unsupervised loss.”; Loss function implies training of machine learning model. The loss function is based on the pseudo-labels, which is based on the probability map. Since the probability map value of each pixel can be the same as reliability weight, the loss function is based on pseudo-labels and reliability weight, which means the training of the model with the second augmented image as the input is based on pseudo-labels and reliability weight ).
Regarding claim 6, dependent upon claim 2, Upretee in view of Qi teaches everything regarding claim 2.
Qi further teaches
the probability map average value obtaining step includes: a probability map obtaining step of obtaining one or more probability maps by performing a predicting process on the first augmented image obtained through a positional transformation performed one or more times by using the deep neural network (Fig. 3, P. 5 Section 1 Pseudolabels’ Generation: “As shown in Fig. 3, the weak disturbance Disturb (∙) contains common augmentation operations, such as flipping and rotating. For each batch, the labeled image
X
is first stochastically augmented once, and
|
{
u
i
;
i
∈
|
(
1
,
…
,
N
)
}
. unlabeled images are stochastically augmented
K
times”); and
a probability map average value calculating step of performing a reverse positional transformation which is a reversal of the positional transformation, on each of the one or more probability maps and further calculating a probability map average value of one or more probability maps resulting from the reverse positional transformation (Fig. 3, P. 5 Section 1 Pseudolabels’ Generation: “the weak disturbance of an image includes random crop, rotation, and flip.”).
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Upretee et al. (FixMatchSeg: Fixing FixMatch for Semi-Supervised Semantic Segmentation, hereinafter Upretee) in view of Arora et al. (US 2024/0331872 A1, hereinafter Arora).
Regarding claim 5, dependent upon claim 1, Upretee discloses everything regarding claim 1.
Upretee further discloses
at the first augmenting step, the first augmented image is obtained by carrying out a weak data augmentation on the region of interest data (P. 5 Para. 1: “For each unlabeled input image
u
b
, we obtained an artificial label from its weakly augmented version
α
(
u
b
)
.”).
However Upretee does not explicitly disclose
a region of interest extracting step at which, prior to the first augmenting step, partial data including a region of interest in the unlabeled image data is extracted as region of interest data, with respect to the input unlabeled image data, on a basis of a prediction result obtained by the deep neural network, wherein
at the first augmenting step, the first augmented image is obtained by carrying out a weak data augmentation on the region of interest data.
Arora teaches
a region of interest extracting step at which, prior to the first augmenting step, partial data including a region of interest in the unlabeled image data is extracted as region of interest data, with respect to the input unlabeled image data, on a basis of a prediction result obtained by the deep neural network (Para [0056]: “The processor (201) may be configured to execute instructions stored in the Region of Interests (RoI) identification module (206) for analyzing the one or more target chest X-ray images for identifying one or more Region of Interests (RoI's).”, Para [0059]: “The processor (201) may be configured to execute instructions stored in the anatomical segmentation module (207) for performing an anatomical segmentation on the one or more Roi's to detect one or more medical abnormalities using the trained artificial intelligence model.”), wherein
at the first augmenting step, the first augmented image is obtained by carrying out a weak data augmentation on the region of interest data (Para [0059]: “The processor (201) may be configured to execute instructions stored in the anatomical segmentation module (207) for performing an anatomical segmentation on the one or more Roi's to detect one or more medical abnormalities using the trained artificial intelligence model.”, Para [0069]: “In anatomical segmentation module (207), a training set of chest X-rays may be annotated at the pixel level with anatomical labels. A U-net based neural network was trained to output anatomical segmentation masks corresponding to the lungs, diaphragm, mediastinum and ribs.”).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Upretee with determining region of interests and perform segmentation only on the region of interests of Arora to effectively reduce the time needed for generating diagnosis of medical images.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Upretee et al. (FixMatchSeg: Fixing FixMatch for Semi-Supervised Semantic Segmentation, hereinafter Upretee) in view of Qi et al. (PICS FOR SEMISUPERVISED REMOTE SENSING IMAGES SEMANTIC SEGMENTATION, hereinafter Qi) and De Fauw et al. (US 2019/0005684 A1, hereinafter De Fauw).
Regarding claim 7, dependent upon claim 2, Upretee in view of Qi teaches everything regarding claim 2.
Qi further teaches
a probability map obtaining step of obtaining a plurality of probability maps…a probability map average value calculating step of calculating an average value of the plurality of probability maps as a probability map average value (Fig. 3, P. 5 Section 1 Pseudolabels’ Generation: “Then, the consistency prediction result
P
i
-
for the original unlabeled image
u
i
-
is obtained through the average operation of the transformed probability maps, which is formulated as follows:
PNG
media_image3.png
121
229
media_image3.png
Greyscale
”).
However Upretee in view of Qi does not explicitly teach
a probability map obtaining step of obtaining a plurality of probability maps, by performing the predicting process on the first augmented image while using each of two or more of the deep neural networks corresponding to the training performed multiple times.
De Fauw teaches
a probability map obtaining step of obtaining a plurality of probability maps, by performing the predicting process on the first augmented image while using each of two or more of the deep neural networks corresponding to the training performed multiple times (Para [0013]: “The system may have a plurality of different first, image segmentation neural networks, each coupled to the image data input, and each providing different said tissue segmentation map data to one or more of the second, classification neural networks…Thus the system may include a set of different, more particularly differently trained, image seg mentation neural networks to allow the system to produce a corresponding set of classifications. The different image segmentation neural networks will typically produce broadly similar tissue segmentation maps differing in details of the mapping, particularly with "difficult" or ambiguous images. This allows the system to produce a set of different classifications, each corresponding to a slightly different hypothesis regarding the underlying tissue map.”).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Upretee in view of Qi with generating multiple segmentation with multiple models of De Fauw to effectively increase the reliability of the segmentation results.
Relevant Prior Art Directed to State of Art
Sohn et al. (FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence, hereinafter Sohn) is prior art not applied in the rejection(s) above. Sohn discloses FixMatch, an algorithm that is a significant simplification of existing SSL methods. FixMatch first generates pseudo-labels using the model’s predictions on weakly augmented unlabeled images. For a given image, the pseudo-label is only retained if the model produces a high-confidence prediction. The model is then trained to predict the pseudo-label when fed a strongly-augmented version of the same image.
Jiao et al. (Learning with Limited Annotations: A Survey on Deep Semi-Supervised Learning for Medical Image Segmentation, hereinafter Jiao) is prior art not applied in the rejection(s) above. Jiao discloses a comprehensive review of recently proposed semi-supervised learning methods for medical image segmentation and summarized both the technical novelties and empirical results.
Seibold et al. (Reference-guided Pseudo-Label Generation for Medical Semantic Segmentation, hereinafter Seibold) is prior art not applied in the rejection(s) above. Seibold discloses a novel approach to generate supervision for semi-supervised semantic segmentation using a small number of labeled images as reference material and match pixels in an unlabeled image to the semantics of the best fitting pixel in a reference set to avoid pitfalls such as confirmation bias, which is common in purely prediction-based pseudo-labeling.
Bengtsson et al. (US 12,115,015 B2, hereinafter Bengtsson) is prior art not applied in the rejection(s) above. Bengtsson discloses techniques for segmenting tumors with positron emission tomography (PET) using deep convolutional neural networks for image and lesion metabolism analysis.
LISOUSKI et al. (WO 2022/129628 A1, hereinafter Lisouski) is prior art not applied in the rejection(s) above. Lisouski discloses computer-implemented method for determining a region-of-interest in medical images having a first and high resolution by downscaling the image for a first segmentation and region-of-interest identification and subsequently, the region of interest upscaled for final neural network supported image segmentation to produce a segmentation output image using a single neural network for both segmentations.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOSHUA CHEN whose telephone number is (703)756-5394. The examiner can normally be reached M-Th: 9:30 am - 4:30pm ET F: 9:30 am - 2:30pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, STEPHEN R KOZIOL can be reached at (408)918-7630. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J. C./Examiner, Art Unit 2665
/Stephen R Koziol/Supervisory Patent Examiner, Art Unit 2665
Prosecution Insights