DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 1, 15, and 18 has been entered.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3-6, 8-11, 14-15, 17-18, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tandon (PGPUB: 20180211380 A1) in view of ISOLA (CN 104135931 A), and further in view of GALEOTTI (WO 2020252271 A1).
Regarding claims 1, 15, and 18, Tandon teaches a method of image segmentation, the method comprising:
accessing an input image that depicts a section of a tissue and includes a plurality of artifact regions (see Fig. 25, item 2501); and
generating a segmentation image by processing the input image (see paragraph 4, segment the one or more images of the biological sample to obtain a plurality of cellular artifacts) using a generator network, the generator network having been trained using a training data set that includes a plurality of pairs of images (see paragraph 355, a neural network of this disclosure, e.g., a convolutional neural network, takes as input the pixel data of cellular artifacts extracted through segmentation. The pixels making up the cellular artifact are divided into slices of predetermined sizes, with each slice being fed to a different node at an input layer of the neural network),
wherein the segmentation image indicates, for each of the plurality of artifact regions of the input image, a boundary of the artifact region (see paragraph 39 and 147, segment the one or more images to identify groups of pixels containing images of sample features from the images, wherein each group of pixels includes a cellular artifact; the collection of contiguous pixels is within or proximate to a boundary defined through segmentation. Often, a cellular artifact includes pixels of an identified boundary, all pixels within that boundary, and optionally some relatively small number of pixels surrounding the boundary (e.g., a penumbra around the periphery of the sample feature)), and
wherein at least one of the plurality of artifact regions depicts an anomaly that is not a structure of the tissue (see paragraph 150, takes cellular artifacts extracted from an image and classifies them as, for example, particular cell types, parasites, bacteria), and
wherein, for each pair of images of the plurality of pairs of images, the pair includes:
a first image of a section of a tissue, the first image including at least one artifact region (see Fig. 24, paragraph 329, the one or more processors are further configured to segment the one or more images of the biological sample to obtain a plurality of images of cellular artifacts. See block 2404; see paragraph 275, the segmentation process employs a gradient technique to identify cellular artifact edges. Such techniques identify regions of an image where over a relatively short distance, pixel magnitudes transition abruptly from darker to lighter), and
a second image that indicates, for each of the at least one artifact region of the first image, a boundary of the artifact region (see paragraph 146, segmentation may define boundaries in an image of the cellular artifacts. The boundaries may be defined by collections of Cartesian coordinates, polar coordinates, pixel IDs, etc.).
However, Tandon does not expressly teach to indicate a boundary of a respective artifact region.
Tandon teaches that a cellular artifact is any item in an image of a biological sample that is identified—typically by segmentation—that might qualify as a cell, parasite or other sample feature of interest. An image of a sample feature may be converted to a cellular artifact. From an image processing perspective, a cellular artifact represents a collection of contiguous pixels (with associated position and magnitude values) that are identified as likely belonging to a cell, parasite, or other sample feature of interest in a biological sample. Typically, the collection of contiguous pixels is within or proximate to a boundary defined through segmentation. Often, a cellular artifact includes pixels of an identified boundary, all pixels within that boundary, and optionally some relatively small number of pixels surrounding the boundary (e.g., a penumbra around the periphery of the sample feature) (see paragraph 147).
ISOLA teaches that method of one step 200 of defining 220 each moving area 216 comprises region 222 around the artifact of interest. assumed motion artifact derived from the moving area 216. Therefore, the estimated motion artifact generally increases along with the distance of 216 to the suspicious moving region and the amplitude is reduced. Therefore, steps 220 to 214 of the part of image data reconstructed through the partitioning definition affected by motion artifacts. one or more inner boundary each artifact comprises region 222 corresponding to the suspicious moving area divided in step 212 identification of 216. each artifact comprising a region 222 of the outer boundary is determined by step 220 (see Fig.2-5, paragraph 32).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Tandon by ISOLA for providing the estimated motion artifact generally increases along with the distance of 216 to the suspicious moving region and the amplitude is reduced. Therefore, steps 220 to 214 of the part of image data reconstructed through the partitioning definition affected by motion artifacts. one or more inner boundary each artifact comprises region 222 corresponding to the suspicious moving area divided in step 212 identification of 216. each artifact comprising a region 222 of the outer boundary is determined by step 220, as to indicate a boundary of a respective artifact region. Therefore, combining the elements from prior arts according to known methods and technique, such as each artifact comprising a region 222 of the outer boundary is determined by step 220, as to indicate a boundary of a respective artifact region, would yield predictable results.
However, the combination does not expressly teach artifact region to be excluded from a subsequent analysis.
GALEOTTI teaches that at step 404, a modified OCT image is generated by processing the input OCT image with the GAN. As explained herein, the modified OCT image may be based on changing pixel values in the original input OCT image. For example, a plurality of background pixels corresponding to a speckle pattern (e.g., noise) and/or a specular artifact may be identified. In some non-limiting embodiments, weights may be applied to background pixels and foreground pixels. In some examples, the background pixels (e.g., pixels representing undesired elements that are positioned above the shallowest tissue interface boundary from the perspective of the image) may be weighed more than the foreground pixels (e.g., pixels representing part of the eye including the tissue interface and elements of the eye below the interface). In some non-limiting embodiments, the background pixels (e.g., pixels just prior to the tissue interface) may be set to a specified vaiue (e.g.,“0” for biack) and the foreground pixels (e.g., pixels Including the tissue interface and other elements of the eye) may be set to another specified value (e.g.,“1”). It will be appreciated that other methods may be used to generate a modified OCT image, such as inserting flags or other indicators into the image data to mark pixels or regions of an image (see Fig. 4, paragraph 66); the result of step 404 is a modified OCT image that allows for more efficient and accurate subsequent processing steps. For example, the modified OCT image may be input to a TISN as described herein to segment the corneal tissue. In FIG. 4, subsequent step 406 may therefore include segmenting tissue structures in the OCT image by processing the modified OCT image with a CNN, such as a TISN Although step 406 is shown in FIG. 4, it will be appreciated that the method may end with the generation of the modified OCT image. For example, the modified OCT image having noise and/or artifacts removed may be stored in a database for later retrieval, review, and/or the like (see Fig. 4, paragraph 67).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination by GALEOTTI to obtain the modified OCT image may be input to a TISN as described herein to segment the corneal tissue. In FIG. 4, subsequent step 406 may therefore include segmenting tissue structures in the OCT image by processing the modified OCT image with a CNN, such as a TISN Although step 406 is shown in FIG. 4, it will be appreciated that the method may end with the generation of the modified OCT image. For example, the modified OCT image having noise and/or artifacts removed may be stored in a database for later retrieval, review, and/or the like, in order to provide artifact region to be excluded from a subsequent analysis. Therefore, combining the elements from prior arts according to known methods and technique would yield predictable results.
Regarding claim 3. Tandon does not expressly teach the method of claim 1, wherein the anomaly is a fold in the section of the tissue.
The examiner is taking "Official Notice" that the limitation about wherein the anomaly is a fold in the section of the tissue is well known in the art.
Therefore, it would have been obvious to a person having ordinary skill in the art at the time the invention was made to have modified the combination so that wherein the anomaly is a fold in the section of the tissue would be available.
Regarding claim 4. Tandon does not expressly teach method of claim 1, wherein the anomaly is a deposit of pigment in the section of the tissue.
The examiner is taking "Official Notice" that the limitation about wherein the anomaly is a deposit of pigment in the section of the tissue is well known in the art.
Therefore, it would have been obvious to a person having ordinary skill in the art at the time the invention was made to have modified the combination so that wherein the anomaly is a deposit of pigment in the section of the tissue would be available.
Regarding claim 5. Tandon does not expressly teach method of claim 1, wherein the segmentation image comprises a binary segmentation mask.
The examiner is taking "Official Notice" that the limitation about wherein the segmentation image comprises a binary segmentation mask is well known in the art.
Therefore, it would have been obvious to a person having ordinary skill in the art at the time the invention was made to have modified the combination so that wherein the segmentation image comprises a binary segmentation mask would be available.
Regarding claim 6. Tandon teaches the method of claim 1, wherein the method further comprises producing an annotated image that includes the segmentation image overlaid on the input image (see Tandon, paragraph 357, convolutional neural networks include multiple layers of receptive fields. As known to those of skill in the art, these layers mimic small neuron collections that process portions of the input image. Individual nodes of these layers receive a limited portion of the cellular artifact. The receptive fields of the nodes partially overlap such that they tile the visual field. The response of a node to its portion of the cellular artifact is treated mathematically by a convolutional operation. The outputs of the nodes in a layer of a convolution network are then arranged so that their input regions overlap, to obtain a better representation of the original image. This may be repeated for every such layer).
Regarding claim 8. Tandon teaches the method of claim 1, wherein the input image includes a second plurality of artifact regions, and wherein the method further comprises:
generating a second segmentation image by processing the input image using a second generator network, the second generator network having been trained using a second training data set that includes a second plurality of pairs of images, wherein the second segmentation image indicates, for each of the second plurality of artifact regions of the input image, a boundary of the artifact region, and wherein at least one of the second plurality of artifact regions depicts a biological structure of the tissue (see Tandon, Fig. 28 B and C, paragraph 38, 39, and 355, segment, by the one or more processors, the one or more images of the biological sample to obtain a plurality of images of cellular artifacts; apply, by the one or more processors, a machine-learning classification model to the plurality of images of cellular artifacts to classify the cellular artifacts; and determine, by the one or more processors, that at least one of the classified cellular artifacts belongs to a class to which the sample feature of interest belongs; segment the one or more images to identify groups of pixels containing images of sample features from the images, wherein each group of pixels includes a cellular artifact; and classify some or all of the cellular artifacts using the deep learning classification model, wherein the classification model discriminates between cellular artifacts created from images of at least one cell type of the host and images of at least one non-host feature; a neural network of this disclosure, e.g., a convolutional neural network, takes as input the pixel data of cellular artifacts extracted through segmentation. The pixels making up the cellular artifact are divided into slices of predetermined sizes, with each slice being fed to a different node at an input layer of the neural network. The input nodes operate on their respective slices of pixels and feed the resulting computed outputs to nodes on a next layer of the neural network, which layer is deemed a hidden layer of the neural network. Values calculated at the nodes of this second layer of the network are then fed forward to a third layer of the neural network where the nodes of the third layer act on the inputs they receive from the second layer and generate new values which are fed to a fourth layer).
Regarding claim 9, 17, and 20. The method of claim 1, wherein the generator network is implemented as a fully convolutional network (see Tandon, paragraph 357, the machine-learning classification model includes a convolutional neural network classifier. In some implementations, applying the machine-learning classification model to the plurality of images of cellular artifacts to classify the cellular artifacts includes: applying a principal component analysis (PCA) to the plurality of images of cellular artifacts to obtain a plurality of feature vectors for the plurality of cellular artifacts; and applying a random forest classifier to the plurality of feature vectors for the plurality of cellular artifacts to classify the cellular artifacts).
Regarding claim 10. Tandon does not expressly teach the method of claim 1, wherein the generator network is implemented as a U-Net.
The examiner is taking "Official Notice" that the limitation about wherein the generator network is implemented as a U-Net is well known in the art.
Therefore, it would have been obvious to a person having ordinary skill in the art at the time the invention was made to have modified the combination so that wherein the generator network is implemented as a U-Net would be available.
Regarding claim 11. Tandon does not expressly teach the method of claim 1, wherein the generator network is implemented as an encoder-decoder network.
The examiner is taking "Official Notice" that the limitation about wherein the generator network is implemented as an encoder-decoder network.
Therefore, it would have been obvious to a person having ordinary skill in the art at the time the invention was made to have modified the combination so that wherein the generator network is implemented as an encoder-decoder network would be available.
Regarding claim 14. Tandon teaches the method of claim 13, further comprising administering, by the user, a treatment with a compound based on (i) the segmentation image, and/or (ii) the diagnosis of the subject (see Tandon, paragraph 365, the classifier provides at least one of two main types of diagnosis: positive identification of a specific organism or cell type, and quantitative analysis of cells or organisms classified as a particular type or of multiple types, whether host cells or non-host cells. One class of host cell quantitation counts leukocytes. Cell count information may be absolute or differential (e.g., ratios of two different cell types). As an example, an absolute red blood cell count lower than a reference range is considered anemic).
Claim(s) 2, 13, 16, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tandon (PGPUB: 20180211380 A1) in view of ISOLA (CN 104135931 A), and further in view of ZHENG (CN 112489062 A).
Regarding claim 2 , 16 and 19. Tandon does not expressly teach the method of claim 1, wherein the anomaly is a focus blur.
Zheng teaches that a segmentation result of an image blurred in the focus boundary (see Fig. 5).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Tandon by Zheng for providing an image blurred in the focus boundary, as wherein the anomaly is a focus blur. Therefore, combining the elements from prior arts according to known methods and technique, such as image blurred in the focus boundary, would yield predictable results.
Regarding claim 13. Tandon does not expressly teach the method of claim 1, further comprising: determining, by a user, a diagnosis of a subject based on the segmentation image.
Zheng teaches that the automatic segmentation of medical images is the basis of medical image analysis, and also a key step in computer-assisted diagnosis. such as polyp segmentation of digestive tract endoscope image, and lesion area segmentation of the skin mirror image. the shape, appearance and position of the divided area has important meaning for early diagnosis of gastrointestinal tract and skin disease. Although the method based on full convolutional neural network obtains excellent performance in many medical image segmentation task (see page 3 and 4, lines 30-31 and 1-4).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Tandon by Zheng for providing segmentation of digestive tract endoscope image, and lesion area segmentation of the skin mirror image. the shape, appearance and position of the divided area has important meaning for early diagnosis of gastrointestinal tract and skin disease. Therefore, combining the elements from prior arts according to known methods and technique would yield predictable results.
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tandon (PGPUB: 20180211380 A1) in view of ISOLA (CN 104135931 A), and further in view of Chen (CN 111862121 A).
Regarding claim 7. Tandon does not expressly teaches the method of claim 1, wherein the method further comprises estimating a quality of the input image, based on a total area of the plurality of artifact regions.
Chen teaches that for the original FAZ image obtained by the DRI OCT Triton machine, the obtained part of the image quality fraction range is the original FAZ image of 60-80, it has the characteristic of easy to generate local high signal-to-noise ratio region in FAZ, the obtained original FAZ image is converted into 8-bit grey FAZ image, the region of local high signal-to-noise ratio in FAZ still exists, so the optimized grey value parameter mainly aims at the problem of high signal-to-noise ratio. As shown in FIG. 15, the present embodiment selects the image with local high signal-to-noise ratio in the FAZ as the target test image (see page 16, lines 11-17).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Tandon by Chen for providing for the original FAZ image obtained by the DRI OCT Triton machine, the obtained part of the image quality fraction range is the original FAZ image of 60-80, it has the characteristic of easy to generate local high signal-to-noise ratio region in FAZ, the obtained original FAZ image is converted into 8-bit grey FAZ image, the region of local high signal-to-noise ratio in FAZ still exists, so the optimized grey value parameter mainly aims at the problem of high signal-to-noise ratio. As shown in FIG. 15, the present embodiment selects the image with local high signal-to-noise ratio in the FAZ as the target test image, as estimating a quality of the input image, based on a total area of the plurality of artifact regions. Therefore, combining the elements from prior arts according to known methods and technique, such as the obtained part of the image quality fraction range is the original FAZ image of 60-80, it has the characteristic of easy to generate local high signal-to-noise ratio region in FAZ, would yield predictable results.
Claim(s) 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tandon (PGPUB: 20180211380 A1) in view of ISOLA (CN 104135931 A), and further in view of Zhang (KR 20200100558 A).
Regarding claim 12. Tandon does not expressly teach the method of claim 1, wherein the generator network is updated via a cross-entropy loss measured between an image by the generator network and an expected output image.
Zhang teaches that in the GAN, the creator is responsible for transferring the image style and confuses the discriminator. The content image X .sub.C is input to the constructor, the constructor creates the style transfer image X .sub.G , and the discriminator is responsible for determining whether the generated image is generated by the current model or the style transfer image ground truth, In other words, it is responsible for determining whether the image X .sub.G generated by the creator is the style transfer image ground truth of the content image X .sub.C. The generator loss includes GAN loss, feature loss and pixel loss. GAN loss refers to a cross-entropy loss output by the discriminator after the style transfer image X .sub.G and the style transfer image ground truth X output by the current creator are input to the discriminator. Characterized in loss refers to the loss of L2 X .sub.G and VGG network with the parameters X is fixed after the input (which may be a neural network of a different type) X .sub.G and X in a specific layer feature map. Pixel loss refers to pixel-by-pixel loss of X .sub.G and X, such as total variation (TV) loss and L2 loss. When training the GAN network, the parameters of the discriminator can be fixed when the generator is trained, and the parameters of the generator can be updated by weighting and summing the above three types of losses. When training the discriminator, the parameters of the constructor are fixed, and the discriminator is updated by classifying the cross entropy loss output by the discriminator after X .sub.G and X are input to the discriminator. Both the generator and the discriminator are trained alternately until close to convergence, so the training of the GAN is complete. At this time, the style transfer image generated by the creator should be very close to the style transfer image ground truth with high quality (see Fig. 16, page 35 and 36, lines 22-37 and 1-5).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Tandon by Zhang for providing The content image X .sub.C is input to the constructor, the constructor creates the style transfer image X .sub.G , and the discriminator is responsible for determining whether the generated image is generated by the current model or the style transfer image ground truth, In other words, it is responsible for determining whether the image X .sub.G generated by the creator is the style transfer image ground truth of the content image X .sub.C. The generator loss includes GAN loss, feature loss and pixel loss. GAN loss refers to a cross-entropy loss output by the discriminator after the style transfer image X .sub.G and the style transfer image ground truth X output by the current creator are input to the discriminator. Characterized in loss refers to the loss of L2 X .sub.G and VGG network with the parameters X is fixed after the input (which may be a neural network of a different type) X .sub.G and X in a specific layer feature map. Pixel loss refers to pixel-by-pixel loss of X .sub.G and X, such as total variation (TV) loss and L2 loss. When training the GAN network, the parameters of the discriminator can be fixed when the generator is trained, and the parameters of the generator can be updated by weighting and summing the above three types of losses, as wherein the generator network is updated via a cross-entropy loss measured between an image by the generator network and an expected output image. Therefore, combining the elements from prior arts according to known methods and technique, such as GAN loss refers to a cross-entropy loss output by the discriminator after the style transfer image X .sub.G and the style transfer image ground truth X output by the current creator are input to the discriminator, would yield predictable results.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1, 15, and 18 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to XIN JIA whose telephone number is (571)270-5536. The examiner can normally be reached 9:00 am-7:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached at (571)272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/XIN JIA/Primary Examiner, Art Unit 2663
Prosecution Insights