DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. CN202210067538.9, filed on 01/20/2022.
Response to Amendment
Applicant's Amendments filed on 08/05/2026 have been entered and made of record.
Status of Claims
Currently pending Claim(s):
Amended claim(s):
Withdrawn claim(s):
1-13, 15-16, and 18-22
1, 15, and 16
14 and 17
Response to Arguments
This office action is responsive to Applicant's Arguments/Remarks made in an Amendment received on 08/05/2026.
In view of the new claim amendments and applicant arguments, Remarks filed on 08/05/2026, with respect to the 35 U.S.C. 101 claim rejections have been carefully considered and the claims rejections to claim 16 under 35 U.S.C. 101 is withdrawn.
In view of applicant’s arguments, Remarks filed on 08/19/2026, with respect to independent claims 1, 6, and 15 under 35 U.S.C. 102, the arguments have been fully considered and are found to be not persuasive. The Applicant argues on page 11:
PNG
media_image1.png
112
702
media_image1.png
Greyscale
The Examiner respectfully disagrees. Liu discloses “wherein the sample initial edge mask image is a binary image” at Paragraph [0046] “The sample mask mentioned in this application is a binary image. The method for obtaining the sample edge information corresponding to a certain sample mask mentioned in step S101…The sample edge information obtained in this way is the same as the sample mask, which is a binary image.”. Further, Applicant’s argument that Liu fails to teach enhancing an already-existing edge mask image is moot as explained below.
Applicant’s Reply (08/05/2026) includes substantive amendments to the claims. This Office action has been updated with new grounds of rejection addressing those amendments. Further Applicant’s Arguments/Remarks with respect to independent claims 1, 15, and 16 have been considered but are moot because the arguments do not apply to any of the references being used in the current rejection and the arguments are now rejected by newly cited art ‘Huang, Xiaoyang, et al. "Two-stage segmentation framework based on distance transformation." Sensors 22.1 (2021): 250.’ as explained in the body of rejection below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 7, 8, 13, 15, and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Liu (CN 110,660,066 A) in view of Huang et al. (Huang, Xiaoyang, et al. "Two-stage segmentation framework based on distance transformation." Sensors 22.1 (2021): 250.) (hereinafter, “Huang”).
Regarding claim 1, Liu discloses an edge extraction method, comprising: acquiring a target image to be extracted (Paragraph [0126] “an image to be processed is obtained”); and
inputting the target image to be extracted to a target edge extraction model to obtain a target edge mask image corresponding to the target image to be extracted (Paragraph [0090] “the generated mask output by the image segmentation network approximates the sample mask, and further ensures that the contour edge of the target object represented in the generated mask output by the image segmentation network is closer to the real contour edge.”; Paragraph [0126] “the image to be processed is input into a trained image segmentation network to obtain a mask corresponding to the image to be processed.”),
wherein the target edge extraction model (image segmentation model in Paragraph [0005] equates to edge extraction model) is trained by (Paragraph [0005] “first aspect of embodiments of this application provides a training method for an image segmentation network, including steps S101 S105”):
acquiring a sample initial image (sample image in Paragraph [0006] equates to sample initial image) to be extracted and a sample initial edge mask image (sample mask in Paragraph [0006] equates to sample initial edge mask) corresponding to the sample initial image to be extracted (Paragraph [0006] “S101, acquire each sample image containing the target object, the sample mask corresponding to each sample image, and the sample edge information corresponding to each sample mask”), wherein the sample initial edge mask image is a binary image (Paragraph [0046] “The sample mask mentioned in this application is a binary image. The method for obtaining the sample edge information corresponding to a certain sample mask mentioned in step S101…The sample edge information obtained in this way is the same as the sample mask, which is a binary image.”; Paragraph [0053] “the "generated mask" mentioned in this step is the same as the sample mask mentioned in step S101, and can be a binary image.”);
performing image enhancement processing on the sample initial image to be extracted to obtain a sample target image (sample image after being expanded by using one or more of the mentioned methos In Paragraph [0045] equates to sample target image) to be extracted with a target size (Paragraph [0045] “a portion of sample images can be obtained from the dataset first, and then the number of sample images used to train the image segmentation network can be expanded in the following way: mirror reversal, scale scaling and/or Gamma change are performed on the pre-obtained sample images to increase the number of sample images, thereby obtaining each sample image mentioned in step S101.”), and performing image enhancement processing on the sample initial edge mask image to obtain a sample target edge mask image (sample edge information in Paragraph [0046] equates to target edge mask image) with the target size (Paragraph [0046] “The method for obtaining the sample edge information corresponding to a certain sample mask mentioned in step S101 can be: performing a dilation operation on the sample mask to obtain the dilated mask image, and subtracting the dilated mask image from the sample mask to obtain the sample edge information corresponding to the sample mask.”), [wherein the image enhancement processing is performed on the sample initial edge mask image to add additional information or transformation data to the sample initial edge mask image to highlight the features of interest in the sample initial image]; and
training an initial deep learning model (image segmentation network prior to training in Paragraph [0010] equates to initial deep learning model) according to the sample target image to be extracted and the sample target edge mask image corresponding to the sample initial image to be extracted to obtain the target edge extraction model (Paragraph [0010] “S105, adjust the parameters of the image segmentation network, and then return to execute S102 until the loss function of the image segmentation network is less than a first preset threshold, thereby obtaining the trained image segmentation network.”).
However, Liu fails to teach wherein the image enhancement processing is performed on the sample initial edge mask image to add additional information or transformation data to the sample initial edge mask image to highlight the features of interest in the sample initial image.
Huang teaches wherein the image enhancement processing (Euclidean distance transformation on Page 5 Subsection 3.2 second paragraph equates to the image enhancement process) is performed on the sample initial edge mask image (edge image on Page 5 Subsection 3.2 second paragraph equates to the sample initial edge mask image) to add additional information or transformation data to the sample initial edge mask image to highlight the features of interest (pixels close to the edge that have a more enormous value on Page 5 Subsection 3.2 second paragraph equates to the features of interest) in the sample initial image (Page 5 Subsection 3.2 first paragraph “When the pixel value of the target area in the label is 1 and the pixel value of the background area is 0, the distance map generated according to the label should satisfy that the closer the target edge area is, the larger the pixel value, and vice versa. Corresponding to the pixels in the original image, the pixel values in the distance map represent the strength that the network needs to learn.”; Page 5 Subsection 3.2 second paragraph “The processing step of the first method, named Method A, is to obtain the edge image of the left atrium and reverse it. Then, generate a distance map according to the Euclidean distance transformation. Next, the distance map is normalized to [0, 1] and the result of subtracting the distance map from 1 is used as the final distance map for supervision. At this time, the supervision distance map satisfies that the pixels close to the edge have a more enormous value.”; Page 7 Subsection 4.2 first paragraph “The method designed in this paper is based on the idea that the distance map generated with the edge of the left atrium as the target has a more significant weight in the area close to the edge, which can guide the network to pay more attention to the edge and strengthen the learning of edge features. Therefore, finding a distance map that can reasonably represent the edge learning weight becomes a key point.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Liu’s reference to include wherein the image enhancement processing is performed on the sample initial edge mask image to add additional information or transformation data to the sample initial edge mask image to highlight the features of interest in the sample initial image taught by Huang’s reference. The motivation for doing so would have been to guide the network to pay more attention to the edge and strengthen the learning of edge features as suggested by Huang (see Huang, Page 7 Subsection 4.2 first paragraph).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Huang with Liu to obtain the invention specified in claim 1.
Regarding claim 7, which claim 1 is incorporated, Liu discloses wherein the initial deep learning model comprises at least two edge extraction layers (Paragraph [0078] “the edge neural network can be composed of 3 convolutional blocks, and each convolutional block is a convolutional layer”; Paragraph [0121] “After obtaining the trained edge neural network, the image segmentation network can be trained”); and
training the initial deep learning model according to the sample target image to be extracted and the sample target edge mask image corresponding to the sample initial image to be extracted to obtain the target edge extraction model comprises (Paragraph [0010] “S105, adjust the parameters of the image segmentation network, and then return to execute S102 until the loss function of the image segmentation network is less than a first preset threshold, thereby obtaining the trained image segmentation network.”):
inputting the sample target image to be extracted to the initial deep learning model to obtain a layer output edge mask image (hc(mask1) and hc(mask2) in Paragraph [0076] equate to the output edge mask image) which is output by each edge extraction layer in the initial deep learning model and corresponds to the sample target image to be extracted (Paragraph [0076] “mask1 is the generation mask corresponding to the j-th sample image, mask2 is the sample mask corresponding to the j-th sample image, hc(mask1) is the output of the c-th convolutional block when the input of the trained edge neural network is mask1, hc(mask2) is the output of the c-th convolutional block when the input of the trained edge neural network is mask2, and ÿc is a constant.”);
determining a target loss of the initial deep learning model according to the layer output edge mask image (generated edge information in Paragraph [0058] equates to output edge mask image) which is output by each edge extraction layer, the sample target edge mask image (sample edge information in Paragraph [0058] equates to sample target edge mask image) corresponding to the sample initial image to be extracted, and a loss function of the initial deep learning model (Paragraph [0010] “S105, adjust the parameters of the image segmentation network, and then return to execute S102 until the loss function of the image segmentation network is less than a first preset threshold, thereby obtaining the trained image segmentation network.”; Paragraph [0058] “The loss function is used to measure the difference between the sample mask and the generation mask corresponding to each sample image. The loss function is also used to measure the difference between the generated edge information and the sample edge information corresponding to each sample image.”); and
adjusting model parameters of the initial deep learning model based on the target loss to obtain the target edge extraction model (Paragraph [0079-0080] “In step S105, it is determined whether the above loss function is less than the first preset threshold. If yes, step S107 is executed; otherwise, step S106 is executed. In step S106, the parameters of the above image segmentation network are adjusted…”).
Regarding claim 8, which claim 7 is incorporated, Liu discloses wherein determining the target loss of the initial deep learning model according to the layer output edge mask image which is output by each edge extraction layer, the sample target edge mask image corresponding to the sample initial image to be extracted, and the loss function of the initial deep learning model comprises (Paragraph [0058] “The loss function is used to measure the difference between the sample mask and the generation mask corresponding to each sample image. The loss function is also used to measure the difference between the generated edge information and the sample edge information corresponding to each sample image.”; (Paragraph [0010] “S105, adjust the parameters of the image segmentation network, and then return to execute S102 until the loss function of the image segmentation network is less than a first preset threshold, thereby obtaining the trained image segmentation network.”):
calculating, for the layer output edge mask image (generated mask in Paragraph [0074-0077] equates to output edge mask image) which is output by each edge extraction layer, a layer output loss (F2j loss on Paragraph [0074-0077] equates to layer loss) between the layer output edge mask image and the sample target edge mask image (sample mask in Paragraph [0074-0077] equates to sample target edge mask image) corresponding to the sample target image to be extracted according to the loss function of the initial deep learning model (Paragraph [0074-0077] “if the above-trained edge neural network is composed of A concatenated convolutional blocks, and each convolutional block consists of B convolutional layers, then the formula for calculating F2j can be as follows:
PNG
media_image2.png
76
788
media_image2.png
Greyscale
mask1 is the generation mask corresponding to the j-th sample image, mask2 is the sample mask corresponding to the j-th sample image, hc(mask1) is the output of the c-th convolutional block when the input of the trained edge neural network is mask1, hc(mask2) is the output of the c-th convolutional block when the input of the trained edge neural network is mask2, and ÿc is a constant. formula for calculating F2j, when the input of the trained edge neural network is mask2, the output of the last convolutional block can be considered as equivalent to the sample edge information. Therefore, the difference between the sample edge information and the generated edge information can be measured by the above formula (3).”); and
determining an initial loss of the initial deep learning model according to the layer output losses corresponding to a plurality of edge extraction layers, and determining the target loss (LOSS1 in paragraph [0067] equates to target loss) according to the initial loss (Paragraph [0067] “LOSS1 is the loss function of the above image segmentation network, N is the total number of sample images, F1j is used to measure the difference between the sample mask corresponding to the j-th sample image and the generated mask, and F2j is used to measure the difference between the sample mask corresponding to the j-th sample image and the generated mask.”).
Regarding claim 13, which claim 1 is incorporated, Liu discloses wherein the initial deep learning model comprises a convolutional neural network model (Paragraph [0078] “the edge neural network can be composed of 3 convolutional blocks, and each convolutional block is a convolutional layer”), [and the convolutional neural network model comprises at least one of a u2net model, a unet model, a deeplab model, a transformer model, and a pidinet model].
However, Liu fails to teach the convolutional neural network model comprises at least one of a u2net model, a unet model, a deeplab model, a transformer model, and a pidinet model.
Huang teaches the convolutional neural network model comprises at least one of a u2net model, a unet model, a deeplab model, a transformer model, and a pidinet model (Page 4 second paragraph “The network training is divided into two stages. In the first stage, a variant of the UNet [21] structure is used as the training network (U-Net1). It adds a branch parallel to the original U-Net up-sampling path. For clarity, the down-sampling path in the original U-Net is named image encoder and the up-sampling path is named image decoder. The newly added up-sampling branch is named distance decoder.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Liu’s reference to include the convolutional neural network model comprises at least one of a u2net model, a unet model, a deeplab model, a transformer model, and a pidinet model taught by Huang’s reference. The motivation for doing so would have been to guide the network to pay more attention to the edge and strengthen the learning of edge features as suggested by Huang (see Huang, Page 7 Subsection 4.2 first paragraph).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Huang with Liu to obtain the invention specified in claim 13.
Regarding claim 15, Liu discloses an electronic device (Paragraph [0181] “The elements and algorithm steps can be implemented using electronic hardware, or a combination of computer software and electronic hardware.”, comprising:
at least one processor (Paragraph [0185] “When the computer program is executed by a processor, it may implement the steps of the various method embodiments described above.”; and
a storage apparatus, configured for storing at least one program (Paragraph [0185] “The aforementioned computer program includes computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The aforementioned computer-readable medium may include: any entity or device capable of carrying the aforementioned computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM)”),
wherein the at least one program, when executed by the at least one processor, causes the at least one processor to (Paragraph [0185] “computer program instructing related hardware. The computer program described above may be stored in a computer-readable storage medium.”):
acquire a target image to be extracted (Paragraph [0126] “an image to be processed is obtained”); and
input the target image to be extracted to a target edge extraction model to obtain a target edge mask image corresponding to the target image to be extracted (Paragraph [0090] “the generated mask output by the image segmentation network approximates the sample mask, and further ensures that the contour edge of the target object represented in the generated mask output by the image segmentation network is closer to the real contour edge.”; Paragraph [0126] “the image to be processed is input into a trained image segmentation network to obtain a mask corresponding to the image to be processed.”),
wherein the target edge extraction model (image segmentation model in Paragraph [0005] equates to edge extraction model) is trained by (Paragraph [0005] “first aspect of embodiments of this application provides a training method for an image segmentation network, including steps S101 S105”):
acquiring a sample initial image (sample image in Paragraph [0006] equates to sample initial image) to be extracted and a sample initial edge mask image (sample mask in Paragraph [0006] equates to sample initial edge mask) corresponding to the sample initial image to be extracted (Paragraph [0006] “S101, acquire each sample image containing the target object, the sample mask corresponding to each sample image, and the sample edge information corresponding to each sample mask”), wherein the sample initial edge mask image is a binary image (Paragraph [0046] “The sample mask mentioned in this application is a binary image. The method for obtaining the sample edge information corresponding to a certain sample mask mentioned in step S101…The sample edge information obtained in this way is the same as the sample mask, which is a binary image.”; Paragraph [0053] “the "generated mask" mentioned in this step is the same as the sample mask mentioned in step S101, and can be a binary image.”);
performing image enhancement processing on the sample initial image to be extracted to obtain a sample target image (sample image after being expanded by using one or more of the mentioned methos In Paragraph [0045] equates to sample target image) to be extracted with a target size (Paragraph [0045] “a portion of sample images can be obtained from the dataset first, and then the number of sample images used to train the image segmentation network can be expanded in the following way: mirror reversal, scale scaling and/or Gamma change are performed on the pre-obtained sample images to increase the number of sample images, thereby obtaining each sample image mentioned in step S101.”), and performing image enhancement processing on the sample initial edge mask image to obtain a sample target edge mask image (sample edge information in Paragraph [0046] equates to target edge mask image) with the target size (Paragraph [0046] “The method for obtaining the sample edge information corresponding to a certain sample mask mentioned in step S101 can be: performing a dilation operation on the sample mask to obtain the dilated mask image, and subtracting the dilated mask image from the sample mask to obtain the sample edge information corresponding to the sample mask.”), [wherein the image enhancement processing is performed on the sample initial edge mask image to add additional information or transformation data to the sample initial edge mask image to highlight the features of interest in the sample initial image]; and
training an initial deep learning model (image segmentation network prior to training in Paragraph [0010] equates to initial deep learning model) according to the sample target image to be extracted and the sample target edge mask image corresponding to the sample initial image to be extracted to obtain the target edge extraction model (Paragraph [0010] “S105, adjust the parameters of the image segmentation network, and then return to execute S102 until the loss function of the image segmentation network is less than a first preset threshold, thereby obtaining the trained image segmentation network.”).
However, Liu fails to teach wherein the image enhancement processing is performed on the sample initial edge mask image to add additional information or transformation data to the sample initial edge mask image to highlight the features of interest in the sample initial image.
Huang teaches wherein the image enhancement processing (Euclidean distance transformation on Page 5 Subsection 3.2 second paragraph equates to the image enhancement process) is performed on the sample initial edge mask image (edge image on Page 5 Subsection 3.2 second paragraph equates to the sample initial edge mask image) to add additional information or transformation data to the sample initial edge mask image to highlight the features of interest (pixels close to the edge that have a more enormous value on Page 5 Subsection 3.2 second paragraph equates to the features of interest) in the sample initial image (Page 5 Subsection 3.2 first paragraph “When the pixel value of the target area in the label is 1 and the pixel value of the background area is 0, the distance map generated according to the label should satisfy that the closer the target edge area is, the larger the pixel value, and vice versa. Corresponding to the pixels in the original image, the pixel values in the distance map represent the strength that the network needs to learn.”; Page 5 Subsection 3.2 second paragraph “The processing step of the first method, named Method A, is to obtain the edge image of the left atrium and reverse it. Then, generate a distance map according to the Euclidean distance transformation. Next, the distance map is normalized to [0, 1] and the result of subtracting the distance map from 1 is used as the final distance map for supervision. At this time, the supervision distance map satisfies that the pixels close to the edge have a more enormous value.”; Page 7 Subsection 4.2 first paragraph “The method designed in this paper is based on the idea that the distance map generated with the edge of the left atrium as the target has a more significant weight in the area close to the edge, which can guide the network to pay more attention to the edge and strengthen the learning of edge features. Therefore, finding a distance map that can reasonably represent the edge learning weight becomes a key point.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Liu’s reference to include wherein the image enhancement processing is performed on the sample initial edge mask image to add additional information or transformation data to the sample initial edge mask image to highlight the features of interest in the sample initial image taught by Huang’s reference. The motivation for doing so would have been to guide the network to pay more attention to the edge and strengthen the learning of edge features as suggested by Huang (see Huang, Page 7 Subsection 4.2 first paragraph).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Huang with Liu to obtain the invention specified in claim 15.
Regarding claim 16, Liu discloses a non-transitory computer-readable storage medium, which stores a computer program, wherein the computer program, when executed by a processor, causes the processor to (Paragraph [0185] “computer program instructing related hardware. The computer program described above may be stored in a computer-readable storage medium. When the computer program is executed by a processor, it may implement the steps of the various method embodiments described above.”):
acquire a target image to be extracted (Paragraph [0126] “an image to be processed is obtained”); and
input the target image to be extracted to a target edge extraction model to obtain a target edge mask image corresponding to the target image to be extracted (Paragraph [0090] “the generated mask output by the image segmentation network approximates the sample mask, and further ensures that the contour edge of the target object represented in the generated mask output by the image segmentation network is closer to the real contour edge.”; Paragraph [0126] “the image to be processed is input into a trained image segmentation network to obtain a mask corresponding to the image to be processed.”),
wherein the target edge extraction model (image segmentation model in Paragraph [0005] equates to edge extraction model) is trained by (Paragraph [0005] “first aspect of embodiments of this application provides a training method for an image segmentation network, including steps S101 S105”):
acquiring a sample initial image (sample image in Paragraph [0006] equates to sample initial image) to be extracted and a sample initial edge mask image (sample mask in Paragraph [0006] equates to sample initial edge mask) corresponding to the sample initial image to be extracted (Paragraph [0006] “S101, acquire each sample image containing the target object, the sample mask corresponding to each sample image, and the sample edge information corresponding to each sample mask”), wherein the sample initial edge mask image is a binary image (Paragraph [0046] “The sample mask mentioned in this application is a binary image. The method for obtaining the sample edge information corresponding to a certain sample mask mentioned in step S101…The sample edge information obtained in this way is the same as the sample mask, which is a binary image.”; Paragraph [0053] “the "generated mask" mentioned in this step is the same as the sample mask mentioned in step S101, and can be a binary image.”);
performing image enhancement processing on the sample initial image to be extracted to obtain a sample target image (sample image after being expanded by using one or more of the mentioned methos In Paragraph [0045] equates to sample target image) to be extracted with a target size (Paragraph [0045] “a portion of sample images can be obtained from the dataset first, and then the number of sample images used to train the image segmentation network can be expanded in the following way: mirror reversal, scale scaling and/or Gamma change are performed on the pre-obtained sample images to increase the number of sample images, thereby obtaining each sample image mentioned in step S101.”), and performing image enhancement processing on the sample initial edge mask image to obtain a sample target edge mask image (sample edge information in Paragraph [0046] equates to target edge mask image) with the target size (Paragraph [0046] “The method for obtaining the sample edge information corresponding to a certain sample mask mentioned in step S101 can be: performing a dilation operation on the sample mask to obtain the dilated mask image, and subtracting the dilated mask image from the sample mask to obtain the sample edge information corresponding to the sample mask.”), [wherein the image enhancement processing is performed on the sample initial edge mask image to add additional information or transformation data to the sample initial edge mask image to highlight the features of interest in the sample initial image]; and
training an initial deep learning model according to the sample target image to be extracted (image segmentation network prior to training in Paragraph [0010] equates to initial deep learning model) and the sample target edge mask image corresponding to the sample initial image to be extracted to obtain the target edge extraction model (Paragraph [0010] “S105, adjust the parameters of the image segmentation network, and then return to execute S102 until the loss function of the image segmentation network is less than a first preset threshold, thereby obtaining the trained image segmentation network.”).
However, Liu fails to teach wherein the image enhancement processing is performed on the sample initial edge mask image to add additional information or transformation data to the sample initial edge mask image to highlight the features of interest in the sample initial image.
Huang teaches wherein the image enhancement processing (Euclidean distance transformation on Page 5 Subsection 3.2 second paragraph equates to the image enhancement process) is performed on the sample initial edge mask image (edge image on Page 5 Subsection 3.2 second paragraph equates to the sample initial edge mask image) to add additional information or transformation data to the sample initial edge mask image to highlight the features of interest (pixels close to the edge that have a more enormous value on Page 5 Subsection 3.2 second paragraph equates to the features of interest) in the sample initial image (Page 5 Subsection 3.2 first paragraph “When the pixel value of the target area in the label is 1 and the pixel value of the background area is 0, the distance map generated according to the label should satisfy that the closer the target edge area is, the larger the pixel value, and vice versa. Corresponding to the pixels in the original image, the pixel values in the distance map represent the strength that the network needs to learn.”; Page 5 Subsection 3.2 second paragraph “The processing step of the first method, named Method A, is to obtain the edge image of the left atrium and reverse it. Then, generate a distance map according to the Euclidean distance transformation. Next, the distance map is normalized to [0, 1] and the result of subtracting the distance map from 1 is used as the final distance map for supervision. At this time, the supervision distance map satisfies that the pixels close to the edge have a more enormous value.”; Page 7 Subsection 4.2 first paragraph “The method designed in this paper is based on the idea that the distance map generated with the edge of the left atrium as the target has a more significant weight in the area close to the edge, which can guide the network to pay more attention to the edge and strengthen the learning of edge features. Therefore, finding a distance map that can reasonably represent the edge learning weight becomes a key point.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Liu’s reference to include wherein the image enhancement processing is performed on the sample initial edge mask image to add additional information or transformation data to the sample initial edge mask image to highlight the features of interest in the sample initial image taught by Huang’s reference. The motivation for doing so would have been to guide the network to pay more attention to the edge and strengthen the learning of edge features as suggested by Huang (see Huang, Page 7 Subsection 4.2 first paragraph).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Huang with Liu to obtain the invention specified in claim 16.
Claim 2, 3, 5, 18, 19, and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Liu (CN 110,660,066 A) in view of Huang et al. (Huang, Xiaoyang, et al. "Two-stage segmentation framework based on distance transformation." Sensors 22.1 (2021): 250.) (hereinafter, “Huang”), as applied to claims 1, 15, and 17 above; and further in view of Fujimoto et al. (US 2021/0390704 A1) (hereinafter, “Fujimoto”).
Regarding claim 2, which claim 1 is incorporated, Liu teaches wherein performing the image enhancement processing on the sample initial image to be extracted to obtain the sample target image to be extracted with the target size comprises (Paragraph [0045] “a portion of sample images can be obtained from the dataset first, and then the number of sample images used to train the image segmentation network can be expanded in the following way: mirror reversal, scale scaling and/or Gamma change are performed on the pre-obtained sample images to increase the number of sample images, thereby obtaining each sample image mentioned in step S101.”):
scaling the sample initial image to be extracted to obtain a sample initial image to be extracted with a first size (Paragraph [0045] “a portion of sample images can be obtained from the dataset first, and then the number of sample images used to train the image segmentation network can be expanded in the following way: mirror reversal, scale scaling and/or Gamma change are performed on the pre-obtained sample images to increase the number of sample images, thereby obtaining each sample image mentioned in step S101.”).
However, Liu and Huang both fail to teach interpolating the sample initial image to be extracted with the first size according to a nearest neighbor interpolation approach to obtain the sample target image to be extracted with the target size.
Fujimoto teaches interpolating the sample initial image to be extracted with the first size according to a nearest neighbor interpolation approach to obtain the sample target image to be extracted with the target size (Paragraph [0019] “The image preprocessing module comprises and/or is configured to cooperate with an image-resizing algorithm, such as a conventional image resizing algorithm, and/or a custom resizing algorithm to receive an image, such as a red-green-blue (RGB) image, and to reduce a size of the image. The conventional image-resizing algorithm is a conventional downsampling interpolation algorithm. For example, the image-resizing algorithm may be a Nearest Neighbor resampling algorithm”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Liu in view of Huang to include interpolating the sample initial image to be extracted with the first size according to a nearest neighbor interpolation approach to obtain the sample target image to be extracted with the target size taught by Fujimoto’s reference. The motivation for doing so would have been to yield a resized image suitable for providing to a segmentation model as suggested by Fujimoto (see Fujimoto, Paragraph [0070]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Fujimoto with Liu and Huang to obtain the invention specified in claim 2.
Regarding claim 3, which claim 2 is incorporated, Liu discloses wherein scaling the sample initial image to be extracted (Paragraph [0045] “a portion of sample images can be obtained from the dataset first, and then the number of sample images used to train the image segmentation network can be expanded in the following way: mirror reversal, scale scaling and/or Gamma change are performed on the pre-obtained sample images to increase the number of sample images, thereby obtaining each sample image mentioned in step S101.”) [comprises:
scaling a length and a width of the sample initial image to be extracted separately according to a preset size transformation range].
However, Liu and Huang both fail to teach scaling a length and a width of the sample initial image to be extracted separately according to a preset size transformation range.
Fujimoto teaches scaling a length and a width of the sample initial image to be extracted separately according to a preset size transformation range (Paragraph [0063] "A raw image 901 having a size of 3524×2520 pixels is fed through various algorithms, including existing and conventional Area, Cubic, Lanczos4, Linear, and Nearest Neighbor downsampling algorithms, corresponding to images 902, 903, 904, 905, 906, respectively, to a final size of 1409×1008 pixels", Examiner interprets this is indicating that the horizonal and the vertical dimension are each reduced from their original values).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Liu in view of Huang to include scaling a length and a width of the sample initial image to be extracted separately according to a preset size transformation range taught by Fujimoto’s reference. The motivation for doing so would have been to yield a resized image suitable for providing to a segmentation model as suggested by Fujimoto (see Fujimoto, Paragraph [0070]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Fujimoto with Liu and Huang to obtain the invention specified in claim 3.
Regarding claim 5, which claim 1 is incorporated, Liu discloses wherein performing the image enhancement processing on the sample initial edge mask image to obtain the sample target edge mask image with the target size comprises (Paragraph [0046] “The method for obtaining the sample edge information corresponding to a certain sample mask mentioned in step S101 can be: performing a dilation operation on the sample mask to obtain the dilated mask image, and subtracting the dilated mask image from the sample mask to obtain the sample edge information corresponding to the sample mask.”):
scaling the sample initial edge mask image to obtain a sample initial edge mask image with a second size (Paragraph [0046] “The method for obtaining the sample edge information corresponding to a certain sample mask mentioned in step S101 can be: performing a dilation operation on the sample mask to obtain the dilated mask image, and subtracting the dilated mask image from the sample mask to obtain the sample edge information corresponding to the sample mask.”); and
However, Liu and Huang both fail to teach interpolating the sample [initial edge mask image] with the [second] size according to a nearest neighbor interpolation approach to obtain the sample target [edge mask] image with the target size.
Fujimoto teaches interpolating the sample [initial edge mask image] with the [second] size according to a nearest neighbor interpolation approach to obtain the sample target [edge mask] image with the target size (Paragraph [0019] “The image preprocessing module comprises and/or is configured to cooperate with an image-resizing algorithm, such as a conventional image resizing algorithm, and/or a custom resizing algorithm to receive an image, such as a red-green-blue (RGB) image, and to reduce a size of the image. The conventional image-resizing algorithm is a conventional downsampling interpolation algorithm. For example, the image-resizing algorithm may be a Nearest Neighbor resampling algorithm”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Liu in view of Huang to include interpolating the sample [initial edge mask image] with the [second] size according to a nearest neighbor interpolation approach to obtain the sample target [edge mask] image with the target size taught by Fujimoto’s reference. The motivation for doing so would have been to yield a resized image suitable for providing to a segmentation model as suggested by Fujimoto (see Fujimoto, Paragraph [0070]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Fujimoto with Liu and Huang to obtain the invention specified in claim 5.
Regarding claim 18 (drawn to an electronic device), claim 18 is rejected the same as claim 2 and the arguments similar to that presented above for claim 2 are equally applicable to the claim 18, and all the other limitations similar to claim 2 are not repeated herein, but incorporated by reference.
Regarding claim 19 (drawn to an electronic device), claim 19 is rejected the same as claim 3 and the arguments similar to that presented above for claim 3 are equally applicable to the claim 19, and all the other limitations similar to claim 3 are not repeated herein, but incorporated by reference.
Regarding claim 21 (drawn to an electronic device), claim 21 is rejected the same as claim 5 and the arguments similar to that presented above for claim 5 are equally applicable to the claim 21, and all the other limitations similar to claim 5 are not repeated herein, but incorporated by reference.
Claim 4 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Liu (CN 110,660,066 A) in view of Huang et al. (Huang, Xiaoyang, et al. "Two-stage segmentation framework based on distance transformation." Sensors 22.1 (2021): 250.) (hereinafter, “Huang”) and Fujimoto et al. (US 2021/0390704 A1), as applied to claims 2 and 18 above; and further in view of McNally (US 2017/0053380 A1).
Regarding claim 4, which claim 2 is incorporated, Liu discloses wherein the method further comprises [prior to] scaling the sample initial image to be extracted (Paragraph [0045] “a portion of sample images can be obtained from the dataset first, and then the number of sample images used to train the image segmentation network can be expanded in the following way: mirror reversal, scale scaling and/or Gamma change are performed on the pre-obtained sample images to increase the number of sample images, thereby obtaining each sample image mentioned in step S101.”).
However, Liu, Huang and Fujimoto fail to teach sharpening the sample initial image to be extracted.
McNally teaches sharpening the sample initial image to be extracted (Paragraph [0082] “perform sharpening instead of interpolation. In such an embodiment, instead of processing an upsampled pixel, the technique may process a current pixel (a pixel that is already a part of the image rather than a pixel that may be a part of an upsampled version of the image). Accordingly, instead of applying an edge weight kernel and/or spatial kernel to an upsampled pixel and shifting the kernels responsive to the location of the upsampled pixel to that of a current pixel or nearest current pixel, the edge weight kernel and/or spatial kernel may be applied to a current pixel.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Liu in view of Haung; and further in view of Fujimoto to include sharpening the sample initial image to be extracted taught by McNally’s reference. The motivation for doing so would have been to apply the weighting factors to the current pixel to refine the image as suggested by McNally (see McNally, Paragraph [0083] and Paragraph [0084]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine McNally with Liu, Huang, and Fujimoto to obtain the invention specified in claim 4.
Regarding claim 20 (drawn to an electronic device), claim 20 is rejected the same as claim 4 and the arguments similar to that presented above for claim 4 are equally applicable to the claim 20, and all the other limitations similar to claim 4 are not repeated herein, but incorporated by reference.
Claims 6 and 22 are rejected under 35 U.S.C. 103 as being unpatentable over Liu (CN 110,660,066 A) in view of Huang et al. (Huang, Xiaoyang, et al. "Two-stage segmentation framework based on distance transformation." Sensors 22.1 (2021): 250.) (hereinafter, “Huang”) and Fujimoto et al. (US 2021/0390704 A1), and further in view of Panichev et al. ("U-net based convolutional neural network for skeleton extraction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2019.) (hereinafter, “Panichev”).
Regarding claim 6, which claim 5 is incorporated, Liu discloses wherein the method further comprises [prior to scaling] the sample initial edge mask image: dilating the sample initial edge mask image (Paragraph [0046] “The method for obtaining the sample edge information corresponding to a certain sample mask mentioned in step S101 can be: performing a dilation operation on the sample mask to obtain the dilated mask image, and subtracting the dilated mask image from the sample mask to obtain the sample edge information corresponding to the sample mask.”); and
[the method further comprises after interpolating the sample initial edge mask image with the second size according to the nearest neighbor interpolation approach, and prior to] obtaining the sample target edge mask image with the target size (Paragraph [0046] “The method for obtaining the sample edge information corresponding to a certain sample mask mentioned in step S101 can be: performing a dilation operation on the sample mask to obtain the dilated mask image, and subtracting the dilated mask image from the sample mask to obtain the sample edge information corresponding to the sample mask.”):
[thinning the sample initial edge mask image].
However, Liu and Huang both fail to teach [dilating] prior to scaling; the method further comprises after interpolating the sample initial edge mask image with the second size according to the nearest neighbor interpolation approach: thinning the sample initial edge mask image.
Fujimoto teaches [dilating] prior to scaling (Paragraph [0063] “A raw image 901 having a size of 3524×2520 pixels is fed through various algorithms”); the method further comprises after interpolating the sample initial edge mask image with the second size according to the nearest neighbor interpolation approach (Paragraph [0019] “The image preprocessing module comprises and/or is configured to cooperate with an image-resizing algorithm, such as a conventional image resizing algorithm, and/or a custom resizing algorithm to receive an image, such as a red-green-blue (RGB) image, and to reduce a size of the image. The conventional image-resizing algorithm is a conventional downsampling interpolation algorithm. For example, the image-resizing algorithm may be a Nearest Neighbor resampling algorithm”
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Liu in view of Huang to include [dilating] prior to scaling; the method further comprises after interpolating the sample initial edge mask image with the second size according to the nearest neighbor interpolation approach The motivation for doing so would have been to ensure the raw image is appropriately sized for later processing and to yield a resized image suitable for providing to a segmentation model as suggested by Fujimoto (see Fujimoto, Paragraph [0070]).
Further, Liu, Huang and Fujimoto fail to teach thinning the sample initial edge mask image.
Panichev teaches thinning the sample initial [edge mask] image (Abstract “Skeletonization is a process aimed to extract a line-like object shape representation, skeleton”; Page 2 left column last paragraph “we describe our CNN-based approach for object shape skeletonization within SkelNetOn 2019 competition [3]. We believe the modification of initial U-Net architecture [15], which is widely used basic architecture for semantic segmentation, could be used for skeletonization”; Figure 1; Examiner interprets skeletonizations as performing thinning of an image to a line-like representation.).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Liu in view of Huang; and further in view of Fujimoto to include thinning the sample initial [edge mask] image taught by Panichev’s reference. The motivation for doing so would have been for optical character recognition as suggested by Panichev (see Panichev, Page 1 left column Introduction first paragraph).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Panichev with Liu, Huang, and Fujimoto to obtain the invention specified in claim 6.
Regarding claim 22 (drawn to an electronic device), claim 22 is rejected the same as claim 6 and the arguments similar to that presented above for claim 6 are equally applicable to the claim 22, and all the other limitations similar to claim 6 are not repeated herein, but incorporated by reference.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Liu (CN 110660066 A) in view of Huang et al. (Huang, Xiaoyang, et al. "Two-stage segmentation framework based on distance transformation." Sensors 22.1 (2021): 250.) (hereinafter, “Huang”), as applied to claim 8 above, and Freeman et al. (US 2022/0245955 A1) (hereinafter, “Freeman”), further in view of Panichev et al. ("U-net based convolutional neural network for skeleton extraction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2019.) (hereinafter, “Panichev”).
Regarding claim 9, which claim 8 is incorporated, Liu discloses determining the target loss according to the initial loss (Paragraph [0067] “LOSS1 is the loss function of the above image segmentation network, N is the total number of sample images, F1j is used to measure the difference between the sample mask corresponding to the j-th sample image and the generated mask, and F2j is used to measure the difference between the sample mask corresponding to the j-th sample image and the generated mask.”)
However, Liu and Huang both fail to teach determining edge pixel points in the sample target edge mask image as positive sample pixel points, and determining pixel points in the sample target edge mask image other than the edge pixel points as negative sample pixel points; determining a number of the positive sample pixel points in the sample target edge mask image, and determining a number of the negative sample pixel points in the sample target edge mask image, and determining a total number of pixel points in the sample target edge mask image; calculating a pixel point loss weight corresponding to each pixel point in the sample target image to be extracted according to the number of the positive sample pixel points, the number of the negative sample pixel points, and the total number of the pixel points; and weighting the initial loss according to the pixel point loss weight corresponding to each pixel point to obtain a target loss corresponding to each pixel point.
Freeman teaches determining edge pixel points in the sample target edge mask image as positive sample pixel points (Paragraph [0022] “If a pixel belongs to a boundary between objects, the element of the mask for this pixel may be assigned a value of one or close to one”), and determining pixel points in the sample target edge mask image other than the edge pixel points as [negative] sample pixel points (Paragraph [0022] “the elements of the mask whose pixels are outside the detected boundaries may be assigned a value close to zero.”);
[determining a number of the positive sample pixel points in the sample target edge mask image, and determining a number of the negative sample pixel points in the sample target edge mask image], and determining a total number of pixel points in the sample target edge mask image (Paragraph [0022] “Such a small value close to zero may ensure that all pixels are accounted for when generating the loss function”);
calculating a pixel point loss weight corresponding to each pixel point in the sample target image to be extracted according to [the number of the positive sample pixel points, the number of the negative sample pixel points], and the total number of the pixel points (Paragraph [0022] “The mask may include a respective element for each pixel, a loss matrix including an element for each pixel may be calculated by relating the predicted segmentation map to the ground truth labels, and each element of the mask may be multiplied by the corresponding element of the loss matrix for each pixel when generating the loss function.”); and
weighting the initial loss according to the pixel point loss weight corresponding to each pixel point to obtain a target loss corresponding to each pixel point (Paragraph [0059] “for each pixel the contribution to the loss is calculated according to the cross-entropy function as defined in (1) such that a “preliminary loss” or loss matrix is generated for which each element includes the contribution of the respective pixel. Thereafter, the final mask 47 is applied to the preliminary loss or loss matrix. In detail, for each pixel the preliminary loss is multiplied by the corresponding element of the final mask 47. Due to this, object boundaries and small objects have a greater contribution to the loss function 37 than areas which do not belong to the detected boundaries between objects.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Liu in view of Huang to include determining edge pixel points in the sample target edge mask image as positive sample pixel points, and determining pixel points in the sample target edge mask image other than the edge pixel points as [negative] sample pixel points; [determining a number of the positive sample pixel points in the sample target edge mask image, and determining a number of the negative sample pixel points in the sample target edge mask image], and determining a total number of pixel points in the sample target edge mask image; calculating a pixel point loss weight corresponding to each pixel point in the sample target image to be extracted according to [the number of the positive sample pixel points, the number of the negative sample pixel points], and the total number of the pixel points; and weighting the initial loss according to the pixel point loss weight corresponding to each pixel point to obtain a target loss corresponding to each pixel point taught by Freeman’s reference. The motivation for doing so would have been to ensure all pixels are accounted for when generating the loss function and using the loss unction to train the neural network as suggested by Freeman (see Freeman, Paragraph [0022] and Paragraph [0025]).
Further, Liu, Huang, and Freeman fail to teach negative sample pixel points and determining a number of the positive sample pixel points in the sample target edge mask image, and determining a number of the negative sample pixel points in the sample target edge mask image.
Panichev teaches negative sample pixel points (Page 2 right column Subsection 3.2 first paragraph “where wpos and wneg- class weights for positive and negative class correspondingly”; Page 3 left column Section 4 second paragraph “A ratio between the number of background pixels and pixels that represent skeleton is near 127:1 in the training dataset. We compared different values of weights in the weighted focal loss and found that wpos = 50 and wneg = 0.75”) and determining a number of the positive sample pixel points in the sample target edge mask image, and determining a number of the negative sample pixel points in the sample target edge mask image (Page 3 left column Section 4 second paragraph “A ratio between the number of background pixels and pixels that represent skeleton is near 127:1 in the training dataset.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Liu in view of Huang; and further in view of Fujimoto to include negative sample pixel points and determining a number of the positive sample pixel points in the sample target edge mask image, and determining a number of the negative sample pixel points in the sample target edge mask image taught by Panichev’s reference. The motivation for doing so would have been to determine a loss and introduce a high penalty for incorrect classification of positive pixels as suggested by Panichev (see Panichev, Page 2 right column Subsection 3.2 first paragraph).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Panichev with Liu, Huang, and Fujimoto to obtain the invention specified in claim 9.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Liu (CN 110,660,066 A) in view of Huang et al. (Huang, Xiaoyang, et al. "Two-stage segmentation framework based on distance transformation." Sensors 22.1 (2021): 250.) (hereinafter, “Huang”), as applied to claim 7 above, and further in view of McNally (US 2017/0053380 A1).
Regarding claim 10, which claim 7 is incorporated, Liu discloses wherein an edge extraction layer comprises a convolution module (Paragraph [0078] “the edge neural network can be composed of 3 convolutional blocks, and each convolutional block is a convolutional layer.” [and an upsampling module]; and
wherein obtaining the layer output edge mask image which is output by each edge extraction layer in the initial deep learning model and corresponds to the sample target image to be extracted comprises (Paragraph [0076] “Wherein, mask1 is the generation mask corresponding to the j-th sample image, mask2 is the sample mask corresponding to the j-th sample image, hc(mask1) is the output of the c-th convolutional block when the input of the trained edge neural network is mask1, hc(mask2) is the output of the c-th convolutional block when the input of the trained edge neural network is mask2, and ÿc is a constant.”):
[convolving, for each edge extraction layer in the initial deep learning model, a layer input image of the edge extraction layer using the convolution module of the edge extraction layer, and upsampling, using the upsampling module of the edge extraction layer, the convolved layer input image to obtain the layer output edge mask image corresponding to the sample target image to be extracted, wherein the layer output edge mask image and the sample target edge mask image have the same sizes.]
However, Liu and Huang both fail to teach an upsampling module and convolving, for each edge extraction layer in the initial deep learning model, a layer input image of the edge extraction layer using the convolution module of the edge extraction layer, and upsampling, using the upsampling module of the edge extraction layer, the convolved layer input image to obtain the layer output edge mask image corresponding to the sample target image to be extracted, wherein the layer output edge mask image and the sample target edge mask image have the same sizes.
McNally teaches an upsampling module (Paragraph [0024] “Processing component 110 may include one or more mode modules”; Paragraph [0070] “as part of the upsampled pixel selection process, the closest low resolution pixels (i.e., the pixels of the non-upsampled image that are closest to the upsampled pixel) may be identified.) and convolving, for each edge extraction layer in the initial deep learning model, a layer input image of the edge extraction layer using the convolution module of the edge extraction layer, and upsampling, using the upsampling module of the edge extraction layer, the convolved layer input image to obtain the layer output edge mask image corresponding to the sample target image to be extracted, wherein the layer output edge mask image and the sample target edge mask image have the same sizes (Paragraph [0070] “as part of the upsampled pixel selection process, the closest low resolution pixels (i.e., the pixels of the non-upsampled image that are closest to the upsampled pixel) may be identified. In certain such embodiments, the closest 2×2, 3×3, 4×4, 5×5, 3×2, 4×2, 4×3, or other sized group of low resolution pixels to the upsampled pixel may be identified. The size of the group of closest low resolution pixels identified may be any size. The group of closest low resolution pixels may be used to determine the edge weight and the spatial weight and may be referred to as a neighborhood set. The neighborhood set may be a set of low resolution pixels that may be processed by a kernel. Certain embodiments may use the same neighborhood set for calculating the edge weight and the spatial weight, but other embodiments may use different neighborhood sets (e.g., different size groups of closest low resolution pixels) for calculating the edge weight and the spatial weight.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Liu in view of Huang to include an upsampling module and convolving, for each edge extraction layer in the initial deep learning model, a layer input image of the edge extraction layer using the convolution module of the edge extraction layer, and upsampling, using the upsampling module of the edge extraction layer, the convolved layer input image to obtain the layer output edge mask image corresponding to the sample target image to be extracted, wherein the layer output edge mask image and the sample target edge mask image have the same sizes taught by McNally’s reference. The motivation for doing so would have been to refine the image as suggested by McNally (see McNally, Paragraph [0084]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine McNally with Liu and Huang to obtain the invention specified in claim 10.
Claims 11 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Liu (CN 110660066 A) in view of Huang et al. (Huang, Xiaoyang, et al. "Two-stage segmentation framework based on distance transformation." Sensors 22.1 (2021): 250.) (hereinafter, “Huang”), as applied to claim 1 above, and further in view of Yamada et al. (US 7,656,437 B2) (hereinafter, “Yamada”).
Regarding claim 11, which claim 1 is incorporated, Liu discloses wherein the method further comprises [after] obtaining the target edge mask image corresponding to the target image to be extracted (Paragraph [0126] “the image to be processed is input into a trained image segmentation network to obtain a mask corresponding to the image to be processed.”).
However, Liu and Huang both fail to teach adjusting image brightness of the [target edge mask] image based on a preset color lookup table.
Yamada teaches after obtaining the [target edge mask] image… adjusting image brightness of the [target edge mask] image based on a preset color lookup table (Column 19 [lines 39-46] “This processing corresponds to step S102…a pixel having a luminance that is greater than or equal to a predetermined luminance threshold is detected and processing for generating a white saturation mask S (x, y) for identifying the white saturated pixel is executed.”; Column 19 [lines 55-64] “This processing corresponds to step S103…using a lookup table (LUT) having false-color-pixel detection-area information associated with the distance from the optical center on the image to the white-saturated pixel (x, y) and the diaphragm and focal distance information during the photography of image data, the false-color-pixel detection-area setting unit 403 sets a false-color-pixel detection area around each white-saturated pixel (x, y).”; Column 20 [lines 28-31] “The false color (purple fringe) compensation interpolation unit 411 performs processing for correcting a pixel determined to have false color by the false color (purple fringe) detection unit 404. The processing corresponds to step S105” ).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Liu in view of Huang to include after obtaining the [target edge mask] image… adjusting image brightness of the [target edge mask] image based on a preset color lookup table taught by Yamada’s reference. The motivation for doing so would have been to output high quality image data without affecting the entire image by efficiently detecting the area of false color and to determine the false-color pixels using the lookup table and correcting the false colors as suggested by Yamada (see Yamada, Abstract and Column 30 [lines 29-33]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Yamada with Liu and Huang to obtain the invention specified in claim 11.
Regarding claim 12, which claim 1 is incorporated, Liu discloses wherein the method further comprises [after] obtaining the target edge mask image corresponding to the target image to be extracted (Paragraph [0126] “the image to be processed is input into a trained image segmentation network to obtain a mask corresponding to the image to be processed.”):
recognizing edge pixel points in the target edge mask image based on a preset contour recognition algorithm (Paragraph [0019] “the trained edge neural network 001 outputs edge information 003 according to the image region (pure white region) where the target object is indicated by the mask 002 input to the edge neural network 001. The edge information is used to indicate the position of the contour edge of the image region. The edge information 003 in Figure 1 is presented in the form of an image.”; Paragraph [0054] “for each generated mask, the generated mask is input to the trained edge neural network to obtain the generated edge information output by the edge neural network. The generated edge information is used to indicate the contour edge of the region where the target object indicated by the generated mask is located”)
However, Liu and Huang both fail to teach after obtaining the [target edge mask] image…storing the recognized [edge] pixel points in the form of point vectors.
Yamada teaches teach after obtaining the [target edge mask] image…storing the recognized [edge] pixel points in the form of point vectors (Column 10 [lines 22-36] “In step S203, a determination is made as to whether or not the luminance of a pixel (x, y) having the luminance component image L.in is higher than or equal to a predetermined threshold. When the luminance is higher than or equal to the threshold, the process proceeds to step S204, in which the white-saturation mask S (x, y) is set to true.”; Column 10 [lines 40-50] “the pixel position x is updated. In step S207, a determination is made as to whether or not x reaches a maximum value (x=w). When x does not reach the maximum value, step S203 and the subsequent steps are repeatedly executed. When x reaches the maximum value (x=w) in step S207, the process proceeds to step S208, in which the pixel position y is updated. In step S209, a determination is made as to whether or not y reaches a maximum value (y=h). When y does not reach the maximum value, step S202 and the subsequent steps are repeatedly executed. When processing on all pixels (x=0 to w, y=0 to h) is finished”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Liu in view of Huang to include storing the recognized [edge] pixel points in the form of point vectors taught by Yamada’s reference. The motivation for doing so would have been to output high quality image data without affecting the entire image by efficiently detecting the area of false color and to obtain the distance from the optical center of the image to the white saturated pixel using the pixel position information set as information to be inspected in this flow as suggested by Yamada (see Yamada, Column 11 [lines 47-53]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Yamada with Liu and Huang to obtain the invention specified in claim 12.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Kim et al. (US 7,876,954 B2) discloses a method for generating a disparity map expressed by 3D coordinates, which is generated from pixel depth information using the disparity between the reference image and the search image with respect to all pixels.
Lee et al. (US 2020/0372608 A1) discloses a method for applying a first weight to the first image and a second weight to the second image based on information on the edge areas and the texture areas of the image to acquire an output image.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to UROOJ FATIMA whose telephone number is (571)272-2096. The examiner can normally be reached M-F 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Henok Shiferaw can be reached at (571) 272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/UROOJ FATIMA/Examiner, Art Unit 2676
/Henok Shiferaw/Supervisory Patent Examiner, Art Unit 2676
Prosecution Insights