DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on March 05, 2025, is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the Examiner.
Response to Amendment
Applicant’s Preliminary Amendment filed on September 10, 2025, has been entered and made of record. Amendments to the claims, specification, and abstract are included.
Currently pending Claim(s) 1-15
Independent Claim(s) 1, 6, and 11
Amended Claim(s) 1-4, 6, and 11
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1 and 6 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Niu et al. (CN 112559781 A), hereafter Niu.
Regarding claim 1, Niu teaches an image retrieval method, applied to an image retrieval apparatus ([0001] “This invention relates to the field of image retrieval technology, and in particular to an image retrieval system and method.” [0107] “In addition, the present invention provides a computer-readable storage medium in which a computer program is stored. The computer program is configured to execute the image retrieval method described above during runtime.”), the method comprising:
inputting a to-be-retrieved image into a target model (Fig. 1 shows the overview of Niu’s model. The first block shows a to-be-retrieved image.),
wherein the target model comprises a first neural network and a second neural network (Niu’s model in Fig. 1 includes the semantic segmentation model (first neural network) and the hash encoding model (second neural network). [0022] “The image retrieval system provided by this invention comprises three modules: a semantic segmentation module, a hash encoding module, and a hash retrieval module.”);
segmenting a target region in the to-be-retrieved image through the first neural network, to obtain at least one first feature, wherein the at least one first feature is an intermediate feature extracted in a process of segmenting the target region (The semantic segmentation model (first neural network) obtains a segmented feature image to be subsequently input into the hash encoding model (second neural network). [0060-0062] “This semantic segmentation module can output an image that incorporates attention mechanisms and includes key details and features of the image, which can then be used as input for the next module… The hash encoding module 2 takes the relevant features generated by the semantic segmentation module 1 as input and performs supervised hash training using a deep hashing algorithm based on a convolutional neural network.”),
processing the to-be-retrieved image and the at least one first feature through the second neural network to obtain a feature corresponding to the target region (The hash encoding model (second neural network) receives the segmented feature image, which is the original image with more complete feature information, clearer details, and greater weight given to key features. [0062] “The hash encoding module 2 takes the relevant features generated by the semantic segmentation module 1 as input and performs supervised hash training using a deep hashing algorithm based on a convolutional neural network. The trained model is used to generate the hash code corresponding to the image, thereby realizing the mapping from high-dimensional features to binary hash codes.”);
generating, based on the feature corresponding to the target region, target code corresponding to the to-be-retrieved image ([0022] “A hashing algorithm based on a convolutional neural network is used to perform hash encoding, thereby transforming high-dimensional features into low-dimensional hash codes. The output of the last layer is constrained to be a binary code, which can learn the similarity of labels and perform self-correction. The final output is a code that better reflects the image feature information.”); and
retrieving a target image from a preset image set based on the target code, wherein the preset image set comprises a plurality of images and code corresponding to each of the plurality of images, and a similarity between code corresponding to the target image and the target code meets a preset condition ([0022] “The hash retrieval module takes the code generated by the hash encoding module as input, compares and sorts the generated hash codes according to the Hamming distance, and outputs the hash code with the closest distance and its corresponding image, thereby retrieving the image most similar to the candidate image. In addition, other similar images can also be retrieved by sorting them according to the length of the Hamming distance.”).
Regarding claim 6, Niu teaches an image retrieval apparatus, comprising a processor, a memory, wherein the memory is configured to store an instruction, and the processor is configured to invoke the instruction in the memory ([0001] “This invention relates to the field of image retrieval technology, and in particular to an image retrieval system and method.” [0107] “In addition, the present invention provides a computer-readable storage medium in which a computer program is stored. The computer program is configured to execute the image retrieval method described above during runtime.”) to:
input a to-be-retrieved image into a target model, wherein the target model comprises a first neural network and a second neural network (Fig. 1 shows the overview of Niu’s model. The first block shows a to-be-retrieved image.);
segment a target region in the to-be-retrieved image through the first neural network, to obtain at least one first feature, wherein the at least one first feature is an intermediate feature extracted in a process of segmenting the target region (The semantic segmentation model (first neural network) obtains a segmented feature image to be subsequently input into the hash encoding model (second neural network). [0060-0062] “This semantic segmentation module can output an image that incorporates attention mechanisms and includes key details and features of the image, which can then be used as input for the next module… The hash encoding module 2 takes the relevant features generated by the semantic segmentation module 1 as input and performs supervised hash training using a deep hashing algorithm based on a convolutional neural network.”);
process the to-be-retrieved image and the at least one first feature through the second neural network to obtain a feature corresponding to the target region (The hash encoding model (second neural network) receives the segmented feature image, which is the original image with more complete feature information, clearer details, and greater weight given to key features. [0062] “The hash encoding module 2 takes the relevant features generated by the semantic segmentation module 1 as input and performs supervised hash training using a deep hashing algorithm based on a convolutional neural network. The trained model is used to generate the hash code corresponding to the image, thereby realizing the mapping from high-dimensional features to binary hash codes.”).
generate, based on the feature corresponding to the target region, target code corresponding to the to-be-retrieved image ([0022] “A hashing algorithm based on a convolutional neural network is used to perform hash encoding, thereby transforming high-dimensional features into low-dimensional hash codes. The output of the last layer is constrained to be a binary code, which can learn the similarity of labels and perform self-correction. The final output is a code that better reflects the image feature information.”); and
retrieve a target image from a preset image set based on the target code, wherein the preset image set comprises a plurality of images and code corresponding to each of the plurality of images, and a similarity between code corresponding to the target image and the target code meets a preset condition ([0022] “The hash retrieval module takes the code generated by the hash encoding module as input, compares and sorts the generated hash codes according to the Hamming distance, and outputs the hash code with the closest distance and its corresponding image, thereby retrieving the image most similar to the candidate image. In addition, other similar images can also be retrieved by sorting them according to the length of the Hamming distance.”).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2 and 7 are rejected under 35 U.S.C. 103 as being unpatentable over (CN 112559781 A) in view of Gao et al. (Dual-branch combination network (DCN): Towards accurate diagnosis and lesion segmentation of COVID-19 using CT images, Medical Image Analysis, vol. 67, ISSN 1361-8415.), hereafter Gao.
Regarding claim 2, Niu teaches the method according to claim 1, but Niu fails to teach wherein the processing the to-be-retrieved image and the at least one first feature through the second neural network, to obtain the feature corresponding to the target region comprises: obtaining a first feature that is in the at least one first feature and that corresponds to at least one convolutional layer in the second neural network, wherein a size of the first feature corresponding to the at least one convolutional layer is the same as a size of a feature extracted by the at least one convolutional layer; fusing the feature extracted by the at least one convolutional layer and the first feature corresponding to the at least one convolutional layer, to obtain a feature corresponding to the at least one convolutional layer; and processing the feature corresponding to the at least one convolutional layer, to obtain the feature corresponding to the target region.
However, Gao teaches wherein the processing the to-be-retrieved image and the at least one first feature through the second neural network, to obtain the feature corresponding to the target region (Fig. 1(A-B) shows a system for classifying an image, and the system comprises a first and second neural network. The first neural network extracts and outputs segmentation features and classification features for the lungs (target region) from the image, and at least one segmentation image and the classification features are input into the second neural network. The second neural network includes a lesion attention module, which combines classification features and segmentation features using an attention mechanism for obtaining fine features (such as lesion segmentation and lesion-specific classifications) within the lungs (target region) used for the subsequent disease-classification process.) comprises:
obtaining a first feature that is in the at least one first feature and that corresponds to at least one convolutional layer in the second neural network, wherein a size of the first feature corresponding to the at least one convolutional layer is the same as a size of a feature extracted by the at least one convolutional layer (In [Section 2.3], Gao teaches the implementation of the DCN (second neural network). The DCN comprises a ResNet-50 for the classification branch and a U-Net for the segmentation branch, and the intermediate results of the branches are combined with lesion attention modules. The lesion attention modules receive classification features and segmentation features, and in Fig. 1(B-C), the segmentation features correspond in scale to the convolution layers.);
fusing the feature extracted by the at least one convolutional layer and the first feature corresponding to the at least one convolutional layer, to obtain a feature corresponding to the at least one convolutional layer (In [Section 2.3], Gao teaches utilizing the lesion attention modules by fusing intermediate segmentation features with classification features in the DCN (second neural network) to produce a combined feature representation used by the DCN. See Eq.1.); and
processing the feature corresponding to the at least one convolutional layer, to obtain the feature corresponding to the target region (Fig. 1(B) shows that the lesion attention modules are placed before the ResNet-50 blocks. Thus, a feature is obtained through fusion in a lesion attention model first, then the feature is processed by the ResNet-50.).
Niu and Gao are analogous in the art to the claimed invention, because both teach utilizing a first neural network to perform segmentation for a target region and a second neural network to determine further features for a subsequent process. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify Niu’s hash coding neural network by receiving and fusing intermediate segmentation features. This modification would allow the hash coding neural network to preserve the spatially corresponding semantic information from the segmentation neural network while emphasizing features within the target region and suppressing background noise ([Gao Abstract] “To focus the classification branch more intensively on the lesion areas, a novel lesion attention module was developed to integrate the intermediate segmentation results.”).
Regarding claim 7, Niu and Gao teach the image retrieval apparatus according to claim 6. Gao further teaches wherein the processor is configured to invoke the instruction in the memory to:
obtain a first feature that is in the at least one first feature and that corresponds to at least one convolutional layer in the second neural network, wherein a size of the first feature corresponding to the at least one convolutional layer is the same as a size of a feature extracted by the at least one convolutional layer (In [Section 2.3], Gao teaches the implementation of the DCN (second neural network). The DCN comprises a ResNet-50 for the classification branch and a U-Net for the segmentation branch, and the intermediate results of the branches are combined with lesion attention modules. The lesion attention modules receive classification features and segmentation features, and in Fig. 1(B-C), the segmentation features correspond in scale to the convolution layers.);
fuse the feature extracted by the at least one convolutional layer and the first feature corresponding to the at least one convolutional layer, to obtain a feature corresponding to the at least one convolutional layer (In [Section 2.3], Gao teaches utilizing the lesion attention modules by fusing intermediate segmentation features with classification features in the DCN (second neural network) to produce a combined feature representation used by the DCN. See Eq.1.); and
process the feature corresponding to the at least one convolutional layer, to obtain the feature corresponding to the target region (Fig. 1(B) shows that the lesion attention modules are placed before the ResNet-50 blocks. Thus, a feature is obtained through fusion in a lesion attention model first, then the feature is processed by the ResNet-50.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify Niu’s hash coding neural network by receiving and fusing intermediate segmentation features. This modification would allow the hash coding neural network to preserve the spatially corresponding semantic information from the segmentation neural network while emphasizing features within the target region and suppressing background noise ([Gao Abstract] “To focus the classification branch more intensively on the lesion areas, a novel lesion attention module was developed to integrate the intermediate segmentation results.”).
Claims 3 and 8 are rejected under 35 U.S.C. 103 as being unpatentable over Niu (CN 112559781 A) in view of Wu et al. (Coarse-to-Fine Lung Nodule Segmentation in CT Images With Image Enhancement and Dual-Branch Network. IEEE Access, vol. 9, pp. 7255-7262.), hereafter Wu.
Regarding claim 3, Niu teaches the method according to claim 1. Niu teaches segmenting the to-be-retrieved image by a first neural network. However, Niu fails to teach displaying the to-be-retrieved image; determining, in response to a mark operation performed by a user on the to-be-retrieved image, a target seed point corresponding to the to-be-retrieved image; and inputting the target seed point into the target model; and wherein the segmenting the target region in the to-be-retrieved image through the first neural network, to obtain the at least one first feature comprises: performing region growing based on the target seed point, to obtain an initial region; and segmenting the initial region and the target region in the to-be-retrieved image through the first neural network, to obtain the at least one first feature.
However, Wu teaches further comprising: displaying the to-be-retrieved image; determining, in response to a mark operation performed by a user on the to-be-retrieved image, a target seed point corresponding to the to-be-retrieved image; and inputting the target seed point into the target model (In [Section 2(A)(4)], Wu teaches allowing a user to select a target seed point to locate a target area. Fig. 4(a) shows the process of selecting a seed point and performing region growing.); and
wherein the segmenting the target region in the to-be-retrieved image through the first neural network (This initial region resulting from region growing can be the focus of segmentation subsequently performed by a neural network. [Section 2(B)] “After roughly locating the lung nodules, we propose a dual-branch network based on U-Net (DB U-Net) for fine segmentation.”), to obtain the at least one first feature comprises:
performing region growing based on the target seed point, to obtain an initial region (Fig. 4(b) shows an example of region growing based on a target seed point. See [Section 2(A)(4)] explaining the details of the region growing process.); and
segmenting the initial region and the target region in the to-be-retrieved image through the first neural network, to obtain the at least one first feature (In [Section 2(B)], Wu teaches inputting the initial region into a neural network for extracting features and determining a segmentation result of a target region.).
Niu and Wu are analogous in the art to the claimed invention, because both teach methods of obtaining at least one first feature using a first neural network for performing segmentation. Therefore, it would have been obvious to one of ordinary skill in the art to modify Niu’s invention by additionally inputting a user-selected initial region into the first neural network. This modification would predictably improve localization of the target region and reduce background interference, because the first neural network would be focused on a user-guided region ([Wu Abstract] “First, we preprocess the image to enhance the discrimination of the nodules and roughly locate the lesion area so that we can eliminate the noises from background and focus on learning the features around the boundaries.”).
Regarding claim 8, Niu teaches the image retrieval apparatus according to claim 6. Niu teaches segmenting the to-be-retrieved image by a first neural network. However, Niu fails to teach wherein the processor is configured to invoke the instruction in the memory to: display the to-be-retrieved image; and determine, in response to a mark operation performed by a user on the to-be-retrieved image, a target seed point corresponding to the to- be-retrieved image, input the target seed point into the target model; perform region growing based on the target seed point, to obtain an initial region; and process the initial region and the to-be-retrieved image through the first neural network, to obtain the at least one first feature.
However, Wu teaches wherein the processor is configured to invoke the instruction in the memory to: display the to-be-retrieved image; and determine, in response to a mark operation performed by a user on the to-be-retrieved image, a target seed point corresponding to the to- be-retrieved image, input the target seed point into the target model (In [Section 2(A)(4)], Wu teaches allowing a user to select a target seed point to locate a target area. Fig. 4(a) shows the process of selecting a seed point and performing region growing.);
perform region growing based on the target seed point, to obtain an initial region (Fig. 4(b) shows an example of region growing based on a target seed point. See [Section 2(A)(4)] explaining the details of the region growing process.); and
process the initial region and the to-be-retrieved image through the first neural network, to obtain the at least one first feature (In [Section 2(B)], Wu teaches inputting the initial region into a neural network for extracting features and determining a segmentation result of a target region.).
Therefore, it would have been obvious to one of ordinary skill in the art to modify Niu’s invention by additionally inputting a user-selected initial region into the first neural network. This modification would predictably improve localization of the target region and reduce background interference, because the first neural network would be focused on a user-guided region ([Wu Abstract] “First, we preprocess the image to enhance the discrimination of the nodules and roughly locate the lesion area so that we can eliminate the noises from background and focus on learning the features around the boundaries.”).
Claims 4 and 9 are rejected under 35 U.S.C. 103 as being unpatentable over Niu (CN 112559781 A) in view of Laaksonen et al. (US 2021/0183070 A1), hereafter Laaksonen.
Regarding claim 4, Niu teaches the method according to claim 1, but Nui fails to teach displaying at least one candidate category; determining a target category in response to a selection operation performed by the user in the at least one candidate category, wherein the target category indicates a category corresponding to the target region; and determining the target model from at least one preset model based on the target category.
However, Laaksonen teaches further comprising: displaying at least one candidate category (Laakonsen teaches a GUI (see Fig. 3 label 311) for allowing a user to select a target category to segment from a medical image; a model to perform the segmentation is selected based on the category. [0013] "Embodiments of the disclosed subject matter provide a platform and supported graphical user interface decision-making tools to allow a user to automatically segment a plurality of anatomical structures using multiple segmentation models.”);
determining a target category in response to a selection operation performed by the user in the at least one candidate category, wherein the target category indicates a category corresponding to the target region; and determining the target model from at least one preset model based on the target category ([0011] “Embodiments of the disclosed subject matter provide a platform and supported graphical user interface (GUI) decision-making tools for use by a user (e.g., medical personnel, for example) to aid in the process of automatic multi-structure segmentation by allowing the user to access a database containing different segmentation models of different segmentation protocols (i.e., models of DNN, machine learning, shape or appearance and atlas-based protocols, for example), to select all or a subset of segmentation models that are applicable and/or are better for contouring a particular anatomical structure, and to selectively combine the results of the different segmentation model outcomes.”).
Niu and Laaksonen are analogous in the art to the claimed invention, because both teach methods of utilizing a first neural network to perform segmentation of a medical image to extract at least one first feature. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Niu’s invention by including different models and selecting one based on a target category. Laaksonen acknowledges that deep learning segmentation methods are dependent upon the data set they are trained on; therefore, it would improve the performance and accuracy of segmentation to utilize a model trained specifically on one target category ([Laaksonen 0008] “Since deep learning based segmentation methods are strongly dependent on the data set they train on, these segmentation models have their own list of structures/organs that they create when applied. As such, a single deep learning (DNN) segmentation model may not contain all the desired structures to contour, or the model might fail to contour properly a particular structure, even though it properly contours others.”).
Regarding claim 9, Niu teaches the image retrieval apparatus according to claim 6, but Niu fails to teach wherein the processor is configured to invoke the instruction in the memory to: display at least one candidate category; determine a target category in response to a selection operation performed by the user in the at least one candidate category, wherein the target category indicates a category corresponding to the target region; and determine the target model from at least one preset model based on the target category.
However, Laaksonen teaches wherein the processor is configured to invoke the instruction in the memory to: display at least one candidate category (Laakonsen teaches a GUI (see Fig. 3 label 311) for allowing a user to select a target category to segment from a medical image; a model to perform the segmentation is selected based on the category. [0013] "Embodiments of the disclosed subject matter provide a platform and supported graphical user interface decision-making tools to allow a user to automatically segment a plurality of anatomical structures using multiple segmentation models.”);
determine a target category in response to a selection operation performed by the user in the at least one candidate category, wherein the target category indicates a category corresponding to the target region; and determine the target model from at least one preset model based on the target category ([0011] “Embodiments of the disclosed subject matter provide a platform and supported graphical user interface (GUI) decision-making tools for use by a user (e.g., medical personnel, for example) to aid in the process of automatic multi-structure segmentation by allowing the user to access a database containing different segmentation models of different segmentation protocols (i.e., models of DNN, machine learning, shape or appearance and atlas-based protocols, for example), to select all or a subset of segmentation models that are applicable and/or are better for contouring a particular anatomical structure, and to selectively combine the results of the different segmentation model outcomes.”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Niu’s invention by including different models and selecting one based on a target category. Laaksonen acknowledges that deep learning segmentation methods are dependent upon the data set they are trained on; therefore, it would improve the performance and accuracy of segmentation to utilize a model trained specifically on one target category ([Laaksonen 0008] “Since deep learning based segmentation methods are strongly dependent on the data set they train on, these segmentation models have their own list of structures/organs that they create when applied. As such, a single deep learning (DNN) segmentation model may not contain all the desired structures to contour, or the model might fail to contour properly a particular structure, even though it properly contours others.”).
Claims 5 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Niu (CN 112559781 A) in view of Golden et al. (US 2020/0380675 A1), hereafter Golden.
Regarding claim 5, Niu teaches the method according to claim 1. Niu teaches processing a medical image through a first neural network to obtain a region segmentation result, and Niu teaches using the segmentation result for subsequent processing with a second neural network. However, Niu does not specifically mention saving the segmentation result in the image set or database and later retrieving and displaying the segmentation result with its corresponding target image. Thus, Niu fails to teach wherein the preset image set further comprises: a region segmentation result corresponding to each of the plurality of images; and the method further comprises: displaying a segmentation result obtained by segmenting the target region and a region segmentation result corresponding to the target image.
However, Golden teaches wherein the preset image set further comprises: a region segmentation result corresponding to each of the plurality of images; and the method further comprises: displaying a segmentation result obtained by segmenting the target region and a region segmentation result corresponding to the target image (Although not utilizing codes, Golden teaches a system similar to Niu which involves segmenting an image, extracting features, and utilizing those features for image retrieval from a database (preset image set); see Fig. 29 as an overview of Golden’s method. The segmentation model is a first neural network, [0141-0142] “In this disclosure, a fully convolutional network (FCN) is utilized for segmentation to locate as many lesion candidates as possible… possible segmentation architectures include but are not limited to ENet, U-Net, and their variants.” In [0131-0139] and Fig. 8, Golden teaches that new images are loaded, and the segmentation model is used to determine segmentation masks for each image. Furthermore, Golden’s invention includes a system for retrieving images and their corresponding information, including segmentation masks. In [0137], Golden specifically mentions that segmentation results are saved in the database. [0137] “Once the label mask has been created, to ease viewing, user interaction, and database storage, the mask may be converted to a spline contour for each axial slice.”).
Niu and Golden are analogous in the art to the claimed invention, because both teach method of performing image segmentation on a medical image using a first neural network. Furthermore, both Niu and Golden teach an image retrieval system. Niu already teaches saving the images and their corresponding codes in a database (preset image set), and Niu further teaches obtaining a segmentation result during the process of determining the corresponding code for a to-be-retrieved image. Therefore, it would have been obvious to one of ordinary skill in the art to modify Niu’s invention by additionally saving segmentation results in the database and facilitating display of the segmentation results along with the target image. This modification would allow for users to evaluate and view segmentation results, and it would permit previously generated segmentation results to be displayed without repeating the segmentation operations ([Golden 0115-0119] “Image/segmentation mask pairs are stored in the format required for training so they require no further preprocessing at training time. Reading image/segmentation mask pairs is a computationally cheap transaction The training data could have been stored in a variety of other formats, including named files on disk and real-time generation of masks from the ground truth database for each image. These methods would have achieved the same result, though they would likely have slowed down training.”).
Regarding claim 10, Niu teaches the image retrieval apparatus according to claim 6. Niu teaches processing a medical image through a first neural network to obtain a region segmentation result, and Niu teaches using the segmentation result for subsequent processing with a second neural network. However, Niu does not specifically mention saving the segmentation result in the image set or database and later retrieving and displaying the segmentation result with its corresponding target image. Thus, Niu fails to teach wherein the preset image set further comprises: a region segmentation result corresponding to each of the plurality of images; and the processor is configured to invoke the instruction in the memory to: display a segmentation result obtained by segmenting the target region and a region segmentation result corresponding to the target image.
However, Golden teaches the image retrieval apparatus according to claim 6, wherein the preset image set further comprises: a region segmentation result corresponding to each of the plurality of images; and the processor is configured to invoke the instruction in the memory to: display a segmentation result obtained by segmenting the target region and a region segmentation result corresponding to the target image (Although not utilizing codes, Golden teaches a system similar to Niu which involves segmenting an image, extracting features, and utilizing those features for image retrieval from a database (preset image set); see Fig. 29 as an overview of Golden’s method. The segmentation model is a first neural network, [0141-0142] “In this disclosure, a fully convolutional network (FCN) is utilized for segmentation to locate as many lesion candidates as possible… possible segmentation architectures include but are not limited to ENet, U-Net, and their variants.” In [0131-0139] and Fig. 8, Golden teaches that new images are loaded, and the segmentation model is used to determine segmentation masks for each image. Furthermore, Golden’s invention includes a system for retrieving images and their corresponding information, including segmentation masks. In [0137], Golden specifically mentions that segmentation results are saved in the database. [0137] “Once the label mask has been created, to ease viewing, user interaction, and database storage, the mask may be converted to a spline contour for each axial slice.”).
Therefore, it would have been obvious to one of ordinary skill in the art to modify Niu’s invention by additionally saving segmentation results in the database and facilitating display of the segmentation results along with the target image. This modification would allow for users to evaluate and view segmentation results, and it would permit previously generated segmentation results to be displayed without repeating the segmentation operations ([Golden 0115-0119] “Image/segmentation mask pairs are stored in the format required for training so they require no further preprocessing at training time. Reading image/segmentation mask pairs is a computationally cheap transaction The training data could have been stored in a variety of other formats, including named files on disk and real-time generation of masks from the ground truth database for each image. These methods would have achieved the same result, though they would likely have slowed down training.”).
Claims 11 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Niu (CN 112559781 A) in view of Jiang et al. (Dual Attention Triplet Hashing Network for Image Retrieval. Front. Neurorobot. 15:728161.), hereafter Jiang.
Regarding claim 11, Niu teaches a model training apparatus, comprising a processor, a memory, wherein the memory is configured to store an instruction, and the processor is configured to invoke the instruction in the memory (At [0089], Niu discusses the training set and teaches that supervised training of both the first and second neural networks is performed.) to:
input a sample image group into a target preset model, and the target preset model comprises a first preset neural network and a second preset neural network (In [0089], Niu teaches supervised training processes using batches of images. [0089] “The test batch of data was taken from each of the 10 categories, with 50 images randomly selected from each category. The remaining samples were randomly arranged to form the training batch.”),
perform region segmentation on each sample image in the sample image group through the first preset neural network, to obtain a first segmentation result corresponding to each sample image (The semantic segmentation model (first neural network) obtains a segmented feature image for a sample image. [0060-0063] “This semantic segmentation module can output an image that incorporates attention mechanisms and includes key details and features of the image, which can then be used as input for the next module… The hash encoding module 2 takes the relevant features generated by the semantic segmentation module 1 as input and performs supervised hash training using a deep hashing algorithm based on a convolutional neural network… Specifically, the hash encoding module 2 takes the feature image extracted after semantic segmentation as input.”);
process, through the second preset neural network, each sample image and at least one first feature corresponding to each sample image, to obtain a feature corresponding to a segmented region in each sample image (The hash encoding model (second neural network) receives the segmented feature image, which is the original image with more complete feature information, clearer details, and greater weight given to key features. [0062] “The hash encoding module 2 takes the relevant features generated by the semantic segmentation module 1 as input and performs supervised hash training using a deep hashing algorithm based on a convolutional neural network. The trained model is used to generate the hash code corresponding to the image, thereby realizing the mapping from high-dimensional features to binary hash codes.”);
generate, based on the feature corresponding to the segmented region in each sample image, code corresponding to each sample image, wherein the first feature is an intermediate feature extracted in a process of performing the region segmentation (Using the features extracted by the segmentation neural network, the hash coding neural network transforms features and determines corresponding codes. [0022] “A hashing algorithm based on a convolutional neural network is used to perform hash encoding, thereby transforming high-dimensional features into low-dimensional hash codes. The output of the last layer is constrained to be a binary code, which can learn the similarity of labels and perform self-correction. The final output is a code that better reflects the image feature information.”); and
train the target preset model based on the code corresponding to each sample image and the first segmentation result corresponding to each sample image, to obtain a target model (Niu teaches supervised training of both the segmentation neural network (first neural network) and the hash coding neural network (second neural network), which comprise the target model. [0052] “The input to semantic segmentation module 1 is the original image. The original image is input into a U-Net network with an attention mechanism. The network is trained based on the information annotation of the image. The trained model is used for image semantic segmentation.” [0062] “The hash encoding module 2 takes the relevant features generated by the semantic segmentation module 1 as input and performs supervised hash training using a deep hashing algorithm based on a convolutional neural network. The trained model is used to generate the hash code corresponding to the image, thereby realizing the mapping from high-dimensional features to binary hash codes.”).
Niu teaches training using batches of images, but Niu does not disclose using sample image groups of three sample images. Specifically, Niu fails to teach wherein the sample image group comprises three sample images, categories of two sample images are the same, a category of a 3rd sample image is different from the categories of the two sample images.
However, Jiang teaches wherein the sample image group comprises three sample images, categories of two sample images are the same, a category of a 3rd sample image is different from the categories of the two sample images. (See Fig. 1 showing an overview of Jiang’s target model. Similar to Niu, Jiang teaches a target model which extracts feature information from an image and determines a hash code corresponding to the image for later database retrieval. Fig. 5 and the section titled “Triplet Input” show the training process for Jiang’s target model. The process involves an image triplet input, including an anchor image, an image of the same class as the anchor image, and a negative image of a different class.).
Niu and Jiang are analogous in the art to the claimed invention, because both teach a target model for extracting features from an image and determining a hash code representative of the image to use for database retrieval. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Niu’s training processes by utilizing triplet image inputs. Jiang teaches that triple labels have an advantage over pairwise labels, because triplet training simultaneously encourages the hash code of an anchor image to be similar to another code from the same class and discourages the hash code of an anchor image from being similar to another code from a different class. Pairwise constraints can only enforce one similarity relationship at a time ([Jiang “Introduction”] “Each triplet label can be naturally decomposed into two pairwise labels. A triplet label ensures that in the learned hash code space, the query image is close to the positive image and far from the negative image simultaneously. However, a pairwise label can only ensure that one constraint is observed. Triplet labels explicitly provide a notion of relative similarities between images while pairwise labels can only encode that implicitly.”).
Regarding claim 13, Niu and Jiang teach the model training apparatus according to claim 11. Jiang further teaches wherein the processor is configured to invoke the instruction in the memory to: determine a first loss function value based on the first segmentation result corresponding to each sample image and a label corresponding to each sample image (In the section titled “Network Architecture,” Jiang explains that channel attention and spatial attention are combined for determining segmented features and classifications for pixels in an image. Eq. 6-7 shows loss function values for the classifications and predicted labels.);
determine a second loss function value based on the code corresponding to each sample image (Eq. 5 shows a loss function value J2, which represents the relationship between the learned hash code and label information.).
determine a target loss function value based on the first loss function value and the second loss function value; and update the target preset model based on the target loss function value until a preset training end condition is met, to obtain the target model (Eq. 9 shows the overall loss function, which includes all loss function values discussed above. The target model was trained utilizing that loss function, and in the section titled “Convergence Degree of Triplet Loss,” training continued until the loss reached a “stable value.” See Fig. 9.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Niu’s invention by considering segmentation results and the resulting code in the target model loss. Jiang teaches that additionally considering the classification result in the loss function improves retrieval accuracy ([Jiang “Hyper-Parameter Analysis”] “In the case without the classification loss, that is β = 0, comparing the best MAP of the two data sets, we can find that not combining the classification loss will cause a significant decrease in retrieval accuracy, which proves the rationality of the joint classification loss. In the cases of weighted quantization loss, we can find that combining different proportions of quantization loss will have different effects on retrieval accuracy. When γ = 0.01, the retrieval accuracy will be slightly improved. In general, combining classification loss to the network improves retrieval accuracy more obviously.”).
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Niu (CN 112559781 A) in view of Jiang (Dual Attention Triplet Hashing Network for Image Retrieval. Front. Neurorobot. 15:728161.) and further in view of Gao (Dual-branch combination network (DCN): Towards accurate diagnosis and lesion segmentation of COVID-19 using CT images, Medical Image Analysis, vol. 67, ISSN 1361-8415.).
Regarding claim 12, Niu and Jiang teach the model training apparatus according to claim 11, but both fail to teach wherein the processor is configured to invoke the instruction in the memory to: obtain a first feature that is in at least one first feature corresponding to a first sample image and that corresponds to at least one convolutional layer in the second preset neural network, wherein the first sample image is any image in the sample image group, and a size of the first feature corresponding to the at least one convolutional layer is the same as a size of a feature extracted by the at least one convolutional layer; fuse the feature extracted by the at least one convolutional layer and the first feature corresponding to the at least one convolutional layer, to obtain a feature corresponding to the at least one convolutional layer; and process the feature corresponding to the at least one convolutional layer, to obtain a feature corresponding to a segmented region in the first sample image.
However, Gao teaches wherein the processor is configured to invoke the instruction in the memory to: obtain a first feature that is in at least one first feature corresponding to a first sample image and that corresponds to at least one convolutional layer in the second preset neural network, wherein the first sample image is any image in the sample image group, and a size of the first feature corresponding to the at least one convolutional layer is the same as a size of a feature extracted by the at least one convolutional layer (In [Section 2.3], Gao teaches the implementation of the DCN (second neural network). The DCN comprises a ResNet-50 for the classification branch and a U-Net for the segmentation branch, and the intermediate results of the branches are combined with lesion attention modules. The lesion attention modules receive classification features and segmentation features, and in Fig. 1(B-C), the segmentation features correspond in scale to the convolution layers.);
fuse the feature extracted by the at least one convolutional layer and the first feature corresponding to the at least one convolutional layer, to obtain a feature corresponding to the at least one convolutional layer (In [Section 2.3], Gao teaches utilizing the lesion attention modules by fusing intermediate segmentation features with classification features in the DCN (second neural network) to produce a combined feature representation used by the DCN. See Eq.1.); and
process the feature corresponding to the at least one convolutional layer, to obtain a feature corresponding to a segmented region in the first sample image (Fig. 1(B) shows that the lesion attention modules are placed before the ResNet-50 blocks. Thus, a feature is obtained through fusion in a lesion attention model first, then the feature is processed by the ResNet-50.).
Niu, Jiang, and Gao are all analogous in the art to the claimed invention, because all teach utilizing at least one neural network to extract features from an input image for use in a subsequent process. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify Niu’s hash coding neural network by receiving and fusing intermediate segmentation features. This modification would allow the hash coding neural network to preserve the spatially corresponding semantic information from the segmentation neural network while emphasizing features within the target region and suppressing background noise ([Gao “Abstract”] “To focus the classification branch more intensively on the lesion areas, a novel lesion attention module was developed to integrate the intermediate segmentation results.”).
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Niu (CN 112559781 A) in view of Jiang (Dual Attention Triplet Hashing Network for Image Retrieval. Front. Neurorobot. 15:728161.) and further in view of Wu (Coarse-to-Fine Lung Nodule Segmentation in CT Images With Image Enhancement and Dual-Branch Network. IEEE Access, vol. 9, pp. 7255-7262.).
Regarding claim 14, Niu and Jiang teach the model training apparatus according to claim 11, but both fail to teach wherein the processor is configured to invoke the instruction in the memory to: display each sample image; and determine, in response to a mark operation performed by a user on each sample image, a seed point corresponding to each sample image, input the seed point corresponding to each sample image into the target preset model; perform region growing based on a seed point corresponding to a first sample image, to obtain a second segmentation result corresponding to the first sample image; and process, through the first preset neural network, the first sample image and the second segmentation result corresponding to the first sample image, to obtain a first segmentation result corresponding to the first sample image.
However, Wu teaches wherein the processor is configured to invoke the instruction in the memory to: display each sample image (In [Section 2(A)(4)], Wu teaches allowing a user to select a target seed point to locate a target area. Fig. 4(a) shows the process of selecting a seed point and performing region growing.); and
determine, in response to a mark operation performed by a user on each sample image, a seed point corresponding to each sample image, input the seed point corresponding to each sample image into the target preset model (This initial region resulting from region growing can be the focus of segmentation subsequently performed by a neural network. [Section 2(B)] “After roughly locating the lung nodules, we propose a dual-branch network based on U-Net (DB U-Net) for fine segmentation.”);
perform region growing based on a seed point corresponding to a first sample image, to obtain a second segmentation result corresponding to the first sample image (Fig. 4(b) shows an example of region growing based on a target seed point. See [Section 2(A)(4)] explaining the details of the region growing process.); and
process, through the first preset neural network, the first sample image and the second segmentation result corresponding to the first sample image, to obtain a first segmentation result corresponding to the first sample image (In [Section 2(B)], Wu teaches inputting the initial region into a neural network for extracting features and determining a segmentation result of a target region).
Niu, Jiang, and Wu are analogous in the art to the claimed invention, because all teach methods of obtaining image features using a first neural network. Therefore, it would have been obvious to one of ordinary skill in the art to modify Niu’s invention by additionally inputting a user-selected initial region into the first neural network. This modification would predictably improve localization of the target region and reduce background interference, because the first neural network would be focused on a user-guided region ([Wu “Abstract”] “First, we preprocess the image to enhance the discrimination of the nodules and roughly locate the lesion area so that we can eliminate the noises from background and focus on learning the features around the boundaries.”).
Allowable Subject Matter
Claim 15 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding claim 15, the combination of Niu, Jiang, and Wu teaches the model training apparatus according to claim 14. However, neither Niu, Jiang, nor Wu teach wherein the processor is configured to invoke the instruction in the memory to: determine a difference between the first segmentation result corresponding to the first sample image and the second segmentation result corresponding to the first sample image; determine, based on the difference, a label corresponding to the first sample image; and when the difference is greater than a preset value, use the first segmentation result corresponding to the first sample image as the label corresponding to the first sample image, or when the difference is not greater than a preset value, use the second segmentation result corresponding to the first sample image as the label corresponding to the first sample image.
Wu was relied upon for teaching the limitations regarding obtaining a first and second segmentation results; however, Wu does not teach or motivate selecting one of the segmentation results—either from the region growing operation or the subsequent result from the first neural network—to use as a label based on a preset condition.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Yuan et al. (US 2022/0004808 A1) teaches systems and methods for performing semantic segmentation of images.
Berger (US 2008/0118118 A1) teaches systems and methods for selecting a target seed point in an image and segmenting a surrounding region.
Cai et al. (Medical Image Retrieval Based on Convolutional Neural Network and Supervised Hashing. IEEE Access, vol. 7, pp. 51877-51885.) teaches systems and methods for extracting features from an input image and determining a hash code representative of features for image retrieval.
Ahmad et al. (SiNC: Saliency-injected neural codes for representation and efficient retrieval of medical radiographs. PLoS ONE 12(8): e0181707.) teaches systems and methods for extracting features from an input image and determining a saliency injected neural code for image retrieval.
Moradi et al. (US 2021/0224995 A1) teaches machine learning methods for disease detection in medical images. The process involves a first neural network for segmenting features from the input image, and the resulting feature maps along with the original image are concatenated and input into a second neural network for disease classification.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ERIC JAMES SHOEMAKER whose telephone number is (571)272-6605. The examiner can normally be reached Monday through Friday from 8am to 5pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner' s supervisor, JENNIFER MEHMOOD, can be reached at (571)272-2976. The fax phone number for the organization where this application or proceeding is assigned is (571)273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Eric Shoemaker/
Patent Examiner
/JENNIFER MEHMOOD/Supervisory Patent Examiner, Art Unit 2664
