Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 07/23/2024 is being considered by the examiner.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1 and 4 are rejected under 35 U.S.C. 103 as being unpatentable over Sathish et al. (Lung Segmentation and Nodule Detection in Computed Tomography Scan using a Convolutional Neural Network Trained Adversarially using Turing Test Loss, published 2020), in view of Kiraly et al. (US 2016/0110632 A1).
Regarding Claim 1, Sathish teaches “A method for automated detection of a predetermined biological element in a tissue sample of a human or animal, said method comprising the steps of:
obtaining a plurality of machine-learning images of said biological element, from a plurality of tissue samples of at least one subject” (Sathish, Section IV, paragraph 1 discloses “The dataset consists of CT volumes from 880 subjects, provided as ten subsets for 10-fold cross validation. In each fold of the experiment, eight subsets from the dataset was used for training and one each for validation and testing”; where dataset used for training is a plurality of machine-learning images; where subjects are at least one subject; where CT volumes are tissue samples. Sathish, Section III, Stage 1 discloses “The overall workflow is divided into two stages. In Stage 1, lung area is segmented followed by detection of nodules in Stage 2”; where nodules are biological elements);
“feeding at least one artificial neural network with said plurality of machine-learning images” (Sathish, Fig. 1 and III, Stage 1 discloses ”In each iteration of training, the discriminator is first trained to performs a Turing test to identify the ground truth (GT) and the segmentation map (Pred.) by presenting them together as the input to the network [12]”; where convolutional neural network including a segmentation network and classifier network is at least one artificial neural network);
“obtaining a stack of images, in a plurality of parallel planes, of said tissue sample of said human or animal” (Sathish, Section II discloses “Consider a CT volume V consisting of N number of axial 2D slice S”; where axial 2D slices are a stack of images);
“concatenating the images of said stack into a single image” (Sathish, Section III, Stage 1 discloses “The concatenated input tensor of size 2 × 512 × 512 is shuffled along the depth to randomize the order of the two channels as shown in Fig. 3(a). This ensures that the discriminator doesn’t leverage the order of concatenation to differentiate between GT and Pred.”; where concatenated input tensor of GT and Pred. is concatenating the images into a single image);
“performing automatic edge detection on said single image, and applying the detected edges to each image of said stack for each of said channels” (Sathish, Section III, Stage 2 discloses “Therefore, patches of size 64 × 64 are extracted from the region segmented as lungs in each slice in Stage 1 for further processing in Stage 2”; where segmenting lungs in each slice in Stage 1 is performing automatic edge detection; see in Fig. 1, green arrow from Pred (prediction) segmentation image to Depth Concatenation with G.T. (ground truth); thus, Sathish also teaches applying detected edges (Pred) to each image of the stack (G.T.));
“extracting from each image of said stack a region of interest, where said biological element is to be detected” (Sathish, Section III, Stage 2 discloses “Therefore, patches of size 64 × 64 are extracted from the region segmented as lungs in each slice in Stage 1 for further processing in Stage 2”; where a region segmented as lungs is a region of interest);
“subdividing each region of interest into a plurality of patches” (Sathish, Section III, Stage 2 discloses “Therefore, patches of size 64 × 64 are extracted from the region segmented as lungs in each slice in Stage 1 for further processing in Stage 2”; where patches are a plurality of patches);
“in each patch, using said at least one artificial neural network, automatically obtaining a detection prediction for said biological element” (Sathish, Section III, Section 2 discloses “A LeNet [13] based classifier is used to detect the presence of nodules in the patches as shown in Fig. 4”; where nodules are biological elements);
(Sathish, Section III, Stage 2 discloses “Thus, by evaluating patches within the lung region, the presence of nodules in each slice of the CT volume is determined”; where slices of a CT volume is an image stack).
PNG
media_image1.png
366
549
media_image1.png
Greyscale
Fig. 1 of Sathish
Sathish does not explicitly teach “combining each plurality of patches comprising said detection predictions into an image of each region of interest, then stacking the images of each region of interest, so as to obtain a reconstructed image of said tissue sample of said human or animal comprising said detection predictions of said biological element” (emphasis added).
However, in an analogous field of endeavor, Kiraly teaches “combining each plurality of patches comprising said detection predictions into an image of each region of interest, then stacking the images of each region of interest, so as to obtain a reconstructed image of said tissue sample of said human or animal comprising said detection predictions of said biological element” (Kiraly, [0059] discloses “In act 22, the locations indicated as including the anatomical structure are merged together. The processor combines the locations of the anatomical structure determined by classifying the patches with the locations of the anatomical structure determined in act 16”; where combining locations of an anatomical structure determined by classifying patches with an encapsulating region of anatomy determined in a prior step (see Fig. 2, element 16) is combining patches comprising detection predictions into an image of each region of interest; where an output image is a reconstructed image of said tissue sample. Kiraly, [0032] also discloses “The medical image represents a one, two, or three-dimensional region of the patient. For example, the medical image represents an area or slice of the patient. Values are provided for each of multiple locations distributed in two or three dimensions. The medical image is acquired as a frame of data.” Kiraly, [0020] discloses “High-resolution CT images of the chest contain detailed information of the lungs and airways. These images are used for lung nodule detection, navigation guidance, and/or diagnosis of a wide range of specific airway diseases”).
PNG
media_image2.png
537
626
media_image2.png
Greyscale
Fig. 2 of Kiraly
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to have modified Sathish to incorporate the teachings of Kiraly by merging locations of an anatomical structure determined in patches with an encapsulating region of anatomy. The prior art contained a ‘base’ method upon which the claimed invention can be seen as an ‘improvement.’ Sathish teaches a method of segmenting and classifying nodules (identified in patches) in lung CT image slices; the claimed invention also teaches a step of combining patches into an image of the region of interest. That is, the claimed invention can be seen as an improvement because the invention also claims recombining identified patches into an image of the region. The prior art contained a ‘comparable’ method that has been improved in the same way as the claimed invention. Kiraly teaches lung nodule detection in CT images, and teaches merging patch locations with a structure encapsulating region of anatomy. One of ordinary skill in the art could have applied the known ‘improvement’ technique in the same way to the ‘base’ method and the results would have been predictable to one of ordinary skill in the art. That is, it would have been obvious to one of ordinary skill in the art that applying the merging step of Kiraly to the patch and CT image slices of Sathish would produce a combined image, or an output image as in Kiraly. Accordingly, the combination of Sathish and Kiraly discloses the invention of Claim 1.
Regarding Claim 4, the combination of Sathish and Kiraly teaches “The method as claimed in claim 1, wherein the step of extracting said region of interest implements a semantic- segmentation algorithm” (Sathish, Section II, paragraph 1 discloses “In Stage 1, each pixel of the slice Sn is classified into one of the two classes {lung, background}, to obtain a segmentation map.” Sathish, Section III, Stage 1 discloses “In each iteration of training, the discriminator is first trained to performs a Turing test to identify the ground truth (GT) and the segmentation map (Pred.) by presenting them together as the input to the network [12].” Sathish, reference [12] is: “Adversarially trained convolutional neural networks for semantic segmentation of ischaemic stroke lesion using multisequence magnetic resonance imaging”; where segmenting into classes is a semantic segmentation algorithm).
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Sathish et al. (Lung Segmentation and Nodule Detection in Computed Tomography Scan using a Convolutional Neural Network Trained Adversarially using Turing Test Loss, published 2020), in view of Kiraly et al. (US 2016/0110632 A1), further in view of Ryu (US 2017/0371138 A1).
Regarding Claim 2, the combination of Sathish and Kiraly does not explicitly teach the method of Claim 2.
However, in an analogous field of endeavor, Ryu teaches “The method as claimed in claim 1, wherein the step of obtaining a stack of images, in a plurality of parallel planes, of said tissue sample of said human or animal comprises using an immunofluorescence scanner” (Ryu, [0034] discloses “for each sheet of planar excitation light incident on the specimen S from mutually different directions along a plurality of incident planes that are parallel to each other with a prescribed spacing therebetween in the direction along the detection optical axis Q of the detection optical system 17.” Ryu, [0048] also discloses “Then, the microscope control unit 41 controls the image acquisition device 27 to acquire fluorescence images at certain time intervals (the image acquiring step S8). Accordingly, through radiation of the excitation light from the right-side light path R, a plurality of fluorescence images, i.e., Z stack images, are acquired at regular intervals within the range from the Z0-A position to the Z0-A+H position of the stage 13”; where acquiring fluorescence Z stack images is obtaining a stack of images in parallel planes using an immunofluorescences scanner).
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to have modified the combination of Sathish and Kiraly to incorporate the teachings of Ryu by acquiring a Z stack of fluorescence images. One of ordinary skill in the art would be motivated to combine the Sathish, Kiraly, and Ryu references in order to acquire clear images by combining fluorescence images: Ryu, [0053] discloses “Accordingly, without adjusting the illumination device 5 so as to align the incident planes of the excitation light in the respective incident directions with each other, it is possible to generate clear combined images simply by combining fluorescence images acquired when respective sheets of excitation light are incident from the left-side light path L and the right-side light path R along the same incident plane at different times. Therefore, with the sheet-illumination type, clear combined images can be easily acquired with a simple configuration, without using a complicated and expensive adjustment mechanism for adjusting the illumination device 5 and without requiring complicated adjustment.”) It would be obvious to one of ordinary skill in the art that the segmentation and detection methods of Sathish may be applied to other types of images; use of a known technique (segmentation and detection method in CT scans containing multiple slices, taught by Sathish) to improve similar methods (fluorescence imaging comprising Z stack images, taught by Ryu) would be obvious to one of ordinary skill in the art. Accordingly, the combination of Sathish, Kiraly, and Ryu discloses the invention of Claim 2.
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Sathish et al. (Lung Segmentation and Nodule Detection in Computed Tomography Scan using a Convolutional Neural Network Trained Adversarially using Turing Test Loss, published 2020), in view of Kiraly et al. (US 2016/0110632 A1), further in view of Lalitha et al. (AU 2020100868 A4).
Regarding Claim 3, the combination of Sathish and Kiraly does not explicitly teach the method of Claim 3.
However, in an analogous field of endeavor, Lalitha teaches “The method as claimed in claim 1, wherein the step of automatic edge detection comprises steps of blurring, and of image dilation and erosion” (Lalitha, page 5, paragraph 1 discloses “Filtering is the most important concept to improve the image quality and to clear the noise present in the images. Image blurring can be done to remove some background variations” and “The Edge detection algorithm is used for the detection of blood vessels and optical disc after pre-processing. Erosion and Dilation operation guidance to eliminate the high level of complete blood vessels and contrasts vessels of blood from the image”; where image blurring prior to edge detection, and performing erosion and dilation after edge detection is an automatic edge detection step comprising blurring, dilation, and erosion).
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to have modified the combination of Sathish and Kiraly to incorporate the teachings of Lalitha by performing image blurring, erosion, and dilation as part of an edge detection method. One of ordinary skill in the art would be motivated to combine the Sathish, Kiraly, and Lalitha references in order to remove noise: (Lalitha, page 5, paragraph 1 discloses “Filtering is the most important concept to improve the image quality and to clear the noise present in the images” and “Erosion and Dilation operation guidance to eliminate the high level of complete blood vessels and contrasts vessels of blood from the image.” Accordingly, the combination of Sathish, Kiraly, and Lalitha discloses the invention of Claim 3.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Sathish et al. (Lung Segmentation and Nodule Detection in Computed Tomography Scan using a Convolutional Neural Network Trained Adversarially using Turing Test Loss, published 2020), in view of Kiraly et al. (US 2016/0110632 A1), further in view of Chan et al. (US 2005/0207630 A1).
Regarding Claim 5, the combination of Sathish and Kiraly does not explicitly teach the method of Claim 5.
However, in an analogous field of endeavor, Chan teaches “The method as claimed in claim 1, wherein each detection prediction is represented by a bounding box that is the smallest rectangle containing the image of said biological element detection of which is predicted” (Chan, [0102] discloses “The object 134 of FIG. 9 is an example of such a long, thin structure. According to this rule, and as illustrated in FIG. 10A, each segmented object is enclosed by the smallest rectangular bounding box and the ratio R of the long (b) to the short (a) side length of the rectangle, is calculated. When the ratio R exceeds a chosen threshold and the object is therefore long and thin, the segmented object is considered to be a blood vessel and is eliminated from further processing as a nodule candidate.”)
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to have modified the combination of Sathish and Kiraly to incorporate the teachings of Chan by determining a smallest rectangular bounding box of a segmented object to determine a potential lung nodule. One of ordinary skill in the art would be motivated to combine the Sathish, Kiraly, and Chan references in order to eliminate some segmented objects from further processing: (Chan, [0102] discloses “When the ratio R exceeds a chosen threshold and the object is therefore long and thin, the segmented object is considered to be a blood vessel and is eliminated from further processing as a nodule candidate.”) Sathish and Chan are both directed to lung nodule detection, and thus it would be obvious to one of ordinary skill in the art to apply a known technique of bounding box definition, as taught by Chan, to a known method of lung segmentation and nodule detection, as taught by Sathish. Accordingly, the combination of Sathish, Kiraly, and Chan discloses the invention of Claim 5.
Claims 6 and 7 are rejected under 35 U.S.C. 103 as being unpatentable over Sathish et al. (Lung Segmentation and Nodule Detection in Computed Tomography Scan using a Convolutional Neural Network Trained Adversarially using Turing Test Loss, published 2020), in view of Kiraly et al. (US 2016/0110632 A1), further in view of Huang et al. (US 2011/0142283 A1).
Regarding Claim 6, the combination of Sathish and Kiraly does not explicitly teach the method of Claim 6.
However, in an analogous field of endeavor, Huang teaches “The method as claimed in claim 1, wherein the step of automatically obtaining said detection prediction using said at least one artificial neural network comprises a step of computing posterior probability for some images” (Huang, [0062] discloses “Moving object tracking may be translated into an inference problem for solving, for example, based on Bayesian theory, given the prior probability of the state of tracked object, to find the posterior probability of the state after obtaining new measurement.”)
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to have modified the combination of Sathish and Kiraly to incorporate the teachings of Huang by finding the posterior probability of a state of a tracked object. One of ordinary skill in the art would be motivated to combine the Sathish, Kiraly, and Huang references in order to track a detected object and predict and update the state of the object: (Huang, [0062] discloses “This theory is to define a motion model and an observation model of the moving object and, with these two models, to treat the moving object tracking as (1) based on measurement prior to time t to predict the state at time t+1, and (2) based on measurement at time t+1 to update the prediction.”) That is, it would have been obvious to one of ordinary skill in the art to use the known technique of finding posterior probability to track a moving object as taught by Huang to improve a similar method of object detection and segmentation as taught by Sathish in the same way, thus tracking an object such as the nodule of Sathish. Accordingly, the combination of Sathish, Kiraly, and Huang discloses the invention of Claim 6.
Regarding Claim 7, the combination of Sathish, Kiraly, and Huang teaches “The method as claimed in claim 6, characterized in that wherein said step of computing posterior probability for some images implements a Kalman filter” (Huang, [0065] discloses “When system state transition function A and measurement transition function H have a linear relation, and state transmission noise V and measurement noise W both satisfy Gaussian model, the filter, such as, Kalman filter, may be used to solve the posterior probability. If A and H do not have a linear relation, an extended Kalman filter may still be used to solve and the posterior probability still satisfies Gaussian distribution.”) The proposed combination as well as the motivation for combining the Sathish, Kiraly, and Huang references presented in the rejection of Claim 6, apply to Claim 7 and are incorporated herein by reference. Thus, the apparatus recited in Claim 7 is met by Sathish, Kiraly, and Huang.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Sathish et al. (Lung Segmentation and Nodule Detection in Computed Tomography Scan using a Convolutional Neural Network Trained Adversarially using Turing Test Loss, published 2020), in view of Kiraly et al. (US 2016/0110632 A1), further in view of Corra et al. (Advantages of an Automated Method Compared With
Manual Methods for the Quantification of Intraepidermal Nerve Fiber in Skin Biopsy, published 2021).
Regarding Claim 8, the combination of Sathish and Kiraly does not explicitly teach the method of Claim 8.
However, in an analogous field of endeavor, Corra discloses “The method as claimed in claim 1, wherein said tissue is the skin and said biological element is an intra-epidermal nerve fiber” (Corra, page 3, discloses
PNG
media_image3.png
157
425
media_image3.png
Greyscale
PNG
media_image4.png
209
425
media_image4.png
Greyscale
Excerpt of Corra
; where IENFD is “intraepidermal nerve fiber density” (see Corra, Abstract) ).
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to have modified the combination of Sathish and Kiraly to incorporate the teachings of Corra by automatically detecting and counting intraepidermal nerve fiber density. One of ordinary skill in the art would be motivated to combine the Sathish, Kiraly, and Corra references in order to diagnostic accuracy: Corra, Abstract discloses “To improve diagnostic accuracy and applicability in clinical practice, we developed an automated method for fast IENFD determination with low operator-dependency.”) Additionally, Corra, Discussion, page 9 discloses “Future work will be needed to focus on the reproducibility of counting within and between different neuropathological institutions and on implementing the automated counting algorithm with deep learning techniques”; thus, there is a motivation to combine the CNN based method of Sathish and IENFD counting method of Corra. Accordingly, the combination of Sathish, Kiraly, and Corra discloses the invention of Claim 8.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure: Burgess et al. (Automated immunohistochemistry of intra-epidermal nerve fibres in skin biopsies: A proof-of-concept study, published 2024) discloses a method of intra-epidermal nerve fiber quantification and automatically detecting a particular protein gene product.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CAROLINE TABANCAY DUFFY whose telephone number is (703)756-1859. The examiner can normally be reached Monday - Friday 8:00 am - 5:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached at 5712723382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CAROLINE TABANCAY DUFFY/Examiner, Art Unit 2662
/Siamak Harandi/Primary Examiner, Art Unit 2662
Prosecution Insights