DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Regarding Applicant’s arguments with respect to Prior Art allegedly not teaching “real-time” characterization and display: Ding does teach real-time image acquisition in a first embodiment, which is used to render it obvious to one skilled in the art to also perform real-time processing for the characterization and display steps. See 35 USC 103 rejections below, where this element is incorporated with motivation and rationale
Regarding Applicant’s remaining arguments regarding the Prior Art: these arguments have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Applicant's arguments regarding 35 USC 101 have been fully considered but they are not persuasive.
On Pages 12-13, Applicant argues that the amended independent claims cannot be performed mentally. The described parallel-executed trained neural networks amounts to a generically recited additional element, which is routine in the technical field. The object-central and temporal information association is a clear mental process, as a person can look at size and location of an object of interest in several images and aggregate data mentally. The evaluating of data is also a clear mental process, and the displaying amounts to a generic additional element which is routine in the field (outputting data), based on a mental determination relating to body region and size.
On Pages 14-15, Applicant argues that the claims describe a series of distinct steps that are specific to the technological environment of real-time medical imaging systems, and that the claims are additionally integrated into a practical application “that improves the functioning of a medical imaging system.” While the claims are specific to the technological environment of real-time medical imaging systems, the improvement to this technological environment is unclear. In view of the Applicant’s arguments, there does not appear to be a clear limitation in the Prior Art related to the technical field that the invention specifically addresses and overcomes. Arguing a generic improvement to the functioning of a medical imaging system is not sufficient to describe a meaningful improvement to the technical field.
On Page 15, Applicant argues that the claims recite additional elements that amount to significantly more than any alleged judicial exception, because the combination of elements represents a nonconventional and nonroutine technological arrangement that goes beyond generic data analysis or information presentation. The Applicant further argues that the invention “constitutes a specific technological solution to the problem of assisting clinicians during live medical imaging procedures”. Assisting clinicians is not a “problem”, and thus it is unclear what particular problem the invention is actually overcoming. Moreover, the additional elements are highly generic. Parallel neural networks are routine and ubiquitous across many image analysis platforms. Analysis of size and location of regions of interest using machine learning is also extremely routine across automated image analysis of medical images. Accordingly, there is a wide breadth of literature that uses neural networks for tumor characterization, and the use of parallel processing is commonplace in machine learning. The conditional displaying amounts to a mental process, as will be described in the 35 USC 101 rejections below.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 32, 40, and 48 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 32, 40, and 48 recite “the plurality of features”, which lacks antecedence and is being interpreted as “a plurality of features”.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 30-53 are rejected under 35 U.S.C. 101.
Claim 30 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea of analyzing medical images, aggregating and evaluating data, and making determinations, without significantly more.
The claim recites: “A computer-implemented system for processing real-time video, the system comprising at least one processor configured to:
process real-time video captured from a medical image device, the real-time video comprising a plurality of frames;
detect an object of interest in the plurality of frames;
characterize, in real-time for each of the plurality of frames, the object of interest, including executing a plurality of neural networks in parallel, the plurality of neural networks including:
a trained location network configured to determine a location of the object of interest; and
a trained size network configured to determine a size of the object of interest;
aggregate, when the object of interest persists over more than one of the plurality of frames, information associated with the location of the object of interest and the size of the object of interest; evaluate, based on the aggregated information, the location of the object of interest and the size of the object of interest
display, in real-time on a display device, when the determined location is in a first body region and the determined size is within a first range, the aggregated information for the object of interest; and
Display, in real-time, on the display device, when the determined location is in a second body region and the determined size is within a second range, information indicating a status of the characterization of the object of interest.”
The limitations, as drafted, are processes that, under their broadest reasonable interpretation, cover performance of the limitation in the mind. A person can, in real time, identify objects in images, characterize the location and size of the object, aggregate information obtained across multiple images, evaluate location and size based on the aggregate information, and present information and characterization status (verbally or written) based on body region and size ranges. The obtaining of real-time medical images amounts to data collection (insignificant extra-solution activity).
This judicial exception is not integrated into a practical application. In particular, the claim recites the additional elements of a computer-implemented system, a processor, a medical device, and a display device. These are recited at a level of generality such that they amount to generic elements for implementing the abstract idea (computer-implemented system, a processor), obtaining medical images (medical device), and outputting determinations (display). The claim also recites the additional element of parallel neural networks for location and size determination. This amounts to a routine and conventional neural network for characterizing an object implemented using a routine and conventional architecture (parallel processing). Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements are recited at a high-level of generality. It is therefore a judicial exception that is not integrated into a practical application, and does not include additional elements that are sufficient to amount to significantly more than the judicial exception. This claim is not patent eligible.
Claim 31 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea of determining a medical guideline based on location and size, then presenting the medical guideline. This amounts to identifying an image as a result, without significantly more. This claim is not patent eligible.
Claim 32 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea of classifying an object of interest by several means, each of which can be done mentally. This claim is not patent eligible.
Claim 33 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea specifying the location of the object as a physiological region such as rectum, sigmoid colon, etc. This modification of claim 1 can be performed with the human mind, as the human mind can make a determination based on a body region. This claim is not patent eligible.
Claim 34 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea of specifying the determined size of the object of interest as a number or size classification, which can be done by the human mind. This claim is not patent eligible.
Claim 35 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea of detecting multiple objects in multiple frames, characterizing the objects, and presenting information, each of which may be done mentally/verbally. This claim is not patent eligible.
Claims 36-37 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea of identifying the object as diminutive or non-diminutive and generating information about the object of interest based on the identification, which can be done mentally. The claims are not patent eligible.
Claims 38-45 and 46-53 are rejected under 35 U.S.C. 101 because they are directed to a computer-implemented method and non-transitory computer readable medium with instructions for a processor, which are analogous to the abstract idea recited in claims 30-37. The computer-implementation, non-transitory computer readable medium, and processor are recited at a high level of generality such that they amount to generic computer components for the performance of the abstract idea. The claims are not patent eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 30, 32, 34-35, 38, 40, 42-43, 46, 48, and 50-51 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ding (CN 107230198 A) in view of Majdoubi (Estimation of tumor parameters using neural networks for inverse bioheat problem) further in view of Zur (US 20200387706 A1).
Regarding claim 30, Ding teaches “A computer-implemented system for processing real-time video, (Ding, Page 4, Paragraphs 13-14, “In step S101, a first gastroscope image is dynamically acquired. In this step, the first gastroscope image may be an image to be processed obtained from a gastroscope, and obtained through optical processing to obtain a white light or NBI (NarrowBand Imaging endoscopic narrow-band imaging) gastroscope image. The gastroscope image is acquired dynamically in real time by the above-mentioned gastroscope device, specifically, dynamic images at preset time intervals can be acquired through a preset setting. The first gastroscope image can be pre-processed through operations including rotation and color difference and color temperature adjustment to complete local image processing with low performance requirements. In addition, the first gastroscope image can also receive information input from an interactive display device.”)
“the system comprising at least one processor” (Ding, Figure 3, translated below)
[AltContent: rect]
PNG
media_image1.png
830
512
media_image1.png
Greyscale
“configured to: process real-time video captured from a medical image device, the real-time video comprising a plurality of frames; (Ding, Page 4, Paragraphs 13-14, “In step S101, a first gastroscope image is dynamically acquired. In this step, the first gastroscope image may be an image to be processed obtained from a gastroscope, and obtained through optical processing to obtain a white light or NBI (NarrowBand Imaging endoscopic narrow-band imaging) gastroscope image. The gastroscope image is acquired dynamically in real time by the above-mentioned gastroscope device, specifically, dynamic images at preset time intervals can be acquired through a preset setting. The first gastroscope image can be pre-processed through operations including rotation and color difference and color temperature adjustment to complete local image processing with low performance requirements. In addition, the first gastroscope image can also receive information input from an interactive display device.”)
“detect an object of interest in the plurality of frames; (Ding, Page 4, Paragraphs 15-16, “In step S102, segment the segmented area with lesion features on the first gastroscope image, mark the first-level feature information and position information corresponding to the segmented area on the first gastroscope image as the second gastroscope image, and output The second gastroscopy image. In this step, the first gastroscope image is received, and a target detection algorithm is used to perform region division on the first gastroscope image, including lesion separation and feature extraction. For example, by scanning the region on the first gastroscope image that may have a feature of a lesion, the region on the first gastroscope image that may have a feature of a lesion is segmented, and the segmented area is marked as a suspicious area. Then, the image information data obtained by the frame-selected segmented area is used as input, the segmented area is selected from the first gastroscope image, the positional relationship between the segmented areas is recorded, and the primary feature information contained in the image of the segmented area is extracted , including the extraction of features such as boundary definition, color, surface smoothness, and shape. Finally, a preliminary annotation is performed on the first gastroscope image, so as to obtain a second gastroscope image. In this step, the second gastroscope image can be interactively displayed on the display device.)
While Ding discloses characterizing the location and size of the object of interest (Ding, Page 4, Paragraphs 15-16, “In step S102, segment the segmented area with lesion features on the first gastroscope image, mark the first-level feature information and position information corresponding to the segmented area on the first gastroscope image as the second gastroscope image, and output The second gastroscopy image. In this step, the first gastroscope image is received, and a target detection algorithm is used to perform region division on the first gastroscope image, including lesion separation and feature extraction. For example, by scanning the region on the first gastroscope image that may have a feature of a lesion, the region on the first gastroscope image that may have a feature of a lesion is segmented, and the segmented area is marked as a suspicious area. Then, the image information data obtained by the frame-selected segmented area is used as input, the segmented area is selected from the first gastroscope image, the positional relationship between the segmented areas is recorded, and the primary feature information contained in the image of the segmented area is extracted , including the extraction of features such as boundary definition, color, surface smoothness, and shape. Finally, a preliminary annotation is performed on the first gastroscope image, so as to obtain a second gastroscope image. In this step, the second gastroscope image can be interactively displayed on the display device.” Note that boundary definition and shape features amount to location and size), Ding does not expressly disclose performing the characterization in real-time using parallel trained neural networks respectively performing the location determination and the size determination.
However, in a first embodiment, Ding does disclose performing the image acquisition in real time (Ding, Page 4, Paragraphs 13-14, “In step S101, a first gastroscope image is dynamically acquired. In this step, the first gastroscope image may be an image to be processed obtained from a gastroscope, and obtained through optical processing to obtain a white light or NBI (NarrowBand Imaging endoscopic narrow-band imaging) gastroscope image. The gastroscope image is acquired dynamically in real time by the above-mentioned gastroscope device, specifically, dynamic images at preset time intervals can be acquired through a preset setting. The first gastroscope image can be pre-processed through operations including rotation and color difference and color temperature adjustment to complete local image processing with low performance requirements. In addition, the first gastroscope image can also receive information input from an interactive display device.”)
Majdoubi discloses the use of a trained neural network for determination of size and location of an object of interest in medical images (Majdoubi, Section 3.5 describes neural network training; Section 4.2.1 and Figure 10 describes object of interest radius (size) determination; Sections 4.2.2-4.2.4 and Figures 11-13 describe object of interest coordinate (location) determination.)
Zur discloses performing region of interest characterizations in parallel (Zur, Paragraphs 101 and 103, “The automated polyp detection process implemented by the detection neural network may be executed in parallel to, and/or independently of features 106-114, for example, on the same computing device and/or processor(s) and/or on another real-time connected computing device and/or platform that is connected to the computing device executing the features described with reference to 106-114.”; “Optionally, one or more endoscopic images of a sequential sub-set of the endoscopic images including the respective endoscopic image and one or more images sequentially located earlier than the respective endoscopic images (e.g., captured prior to the respective endoscopic image) that depict the tracked ROI (as described with reference to 106) are fed into the detection neural network. The sequential sub-set of images may be fed into the detection neural network in parallel to the tracking processing (as described with reference to 106). Alternatively, the sub-set of images are first processed by the tracking process as described with reference to 106. One or more of the post-processed images may be translated and/or rotated one to create a sub-set of endoscopic images where the region depicting the polyp(s) (e.g., ROI) detected by the tracking process is at a same approximate position in all of the images, for example, at the same pixel locations on the display for all images. The processed sequential-sub set of images are fed into the detection neural network for outputting the current region delineating the polyp(s). The image may be augmented with the region detected by the detection neural network.”)
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to: firstly, perform the full medical image processing, including characterization, in real time, based on the concept of real-time image acquisition taught by the first embodiment of Ding; secondly, replace the size and location determination of Ding with the neural network of Majdoubi for size and location determination; thirdly, separate said size and location determination neural networks into two, parallel-run trained neural networks, based on the concept of parallel region of interest characterization taught by Zur.
The motivation for performing the full medical image processing in real time would have been to facilitate immediate results for both the clinician and the patient, enabling faster treatment planning for better outcomes. The motivation for using the size and location neural network would have been to actually perform the location and size determination used in Ding. In particular, Ding describes region of interest feature extraction of location and size generically, and does not appear to provide a particular means to do so. One skilled in the art would therefore need an actual implementation strategy, as is provided by Majdoubi, for the extraction of the features including size and location. The motivation for the parallel processing would have been to further facilitate faster results. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Ding with the above additional teachings of Ding, Majdoubi, and Zur to fully disclose, “characterize, in real-time for each of the plurality of frames, the object of interest, including executing a plurality of neural networks in parallel, the plurality of neural networks including: a trained location network configured to determine a location of the object of interest; and a trained size network configured to determine a size of the object of interest;”.
Ding in view of Majdoubi further in view of Zur further disclose, “aggregate, when the object of interest persists over more than one of the plurality of frames, information associated with the location of the object of interest and the size of the object of interest; evaluate, based on the aggregated information, the location of the object of interest and the size of the object of interest;” (Ding, Page 4, Paragraph 17, “In step S103, the second gastroscope image is received, and the primary feature information and position information in a plurality of the second gastroscope images are combined and analyzed to form secondary feature information and an area position corresponding to the secondary feature information, and output a third gastroscope image labeled with the secondary feature information and the region location information.” Note that the combining and analyzing of primary feature information and position information amounts to the aggregating and evaluating of the determined location and size.)
“display, in real-time on a display device, when the determined location is in a first body region and the determined size is within a first range, the aggregated information for the object of interest; and
display, in real-time, on the display device, when the determined location is in a second body region and the determined size is within a second range, information indicating a status of the characterization of the object of interest.” (Ding, Page 4, Paragraph 17, “In step S103, the second gastroscope image is received, and the primary feature information and position information in a plurality of the second gastroscope images are combined and analyzed to form secondary feature information and an area position corresponding to the secondary feature information, and output a third gastroscope image labeled with the secondary feature information and the region location information.” Note that the display of the secondary feature information and region location information are mapped to the presentation of the aggregated information and a status of the characterization, respectively. Additionally, the first and second human body regions are not expressly defined as different, and under broadest reasonable interpretation, can be interpreted as any region inside the human body. Similarly, the first and second size are not expressly defined as different, and could therefore be interpreted as any size >0. Still, the “when” statement merely describes when the step is applicable thus it does not expressly provide a conditional requirement. Additionally, note that as the references are combined above with motivation and rationale, the full medical image processing is performed in real-time (i.e. including the display).)
Regarding claim 32, Ding in view of Majdoubi further in view of Zur teach “The system of claim 30,”
“wherein the plurality of features further includes a classification of the object of interest, the classification being based on at least one of a histological classification, a morphological classification, a structural classification, or a malignancy classification.” (Ding, Page 4, Paragraphs 15-16, “In step S102, segment the segmented area with lesion features on the first gastroscope image, mark the first-level feature information and position information corresponding to the segmented area on the first gastroscope image as the second gastroscope image, and output the second gastroscopy image. In this step, the first gastroscope image is received, and a target detection algorithm is used to perform region division on the first gastroscope image, including lesion separation and feature extraction. For example, by scanning the region on the first gastroscope image that may have a feature of a lesion, the region on the first gastroscope image that may have a feature of a lesion is segmented, and the segmented area is marked as a suspicious area. Then, the image information data obtained by the frame-selected segmented area is used as input, the segmented area is selected from the first gastroscope image, the positional relationship between the segmented areas is recorded, and the primary feature information contained in the image of the segmented area is extracted , including the extraction of features such as boundary definition, color, surface smoothness, and shape. Finally, a preliminary annotation is performed on the first gastroscope image, so as to obtain a second gastroscope image. In this step, the second gastroscope image can be interactively displayed on the display device.” Note that the features described such as boundary definition, color, smoothness, and shape, determined by Ding, amount to structural and morphological feature classifications.).
Regarding claim 33, Ding in view of Majdoubi further in view of Zur teaches “The system of claim 30,”
While Ding in view of Majdoubi further in view of Zur discloses endoscopic imaging using a gastroscope for lesion detection (Ding, Page 4, Paragraphs 13-14, “In step S101, a first gastroscope image is dynamically acquired. In this step, the first gastroscope image may be an image to be processed obtained from a gastroscope, and obtained through optical processing to obtain a white light or NBI (NarrowBand Imaging endoscopic narrow-band imaging) gastroscope image. The gastroscope image is acquired dynamically in real time by the above-mentioned gastroscope device, specifically, dynamic images at preset time intervals can be acquired through a preset setting. The first gastroscope image can be pre-processed through operations including rotation and color difference and color temperature adjustment to complete local image processing with low performance requirements. In addition, the first gastroscope image can also receive information input from an interactive display device.”), Ding in view of Majdoubi further in view of Zur does not expressly disclose endoscopic imaging of the colon.
Zur further discloses the endoscopic imaging of the colon for lesion detection. (Zur, Paragraph 3, “According to a first aspect, a method of generating instructions for presenting a graphical user interface (GUI) for dynamically tracking at least one polyp in a plurality of endoscopic images of a colon of a patient, comprises: iterating for the plurality of endoscopic images: tracking a location of a region depicting at least one polyp within the respective endoscopic image relative to at least one previous endoscopic image, when the location of the region is external to the respective endoscopic image: computing a vector from within the respective endoscopic image to the location of the region external to the respective endoscopic image, creating an augmented endoscopic image by augmenting the respective endoscopic image with an indication of the vector, and generating instructions for presenting the augmented endoscopic image within the GUI.”)
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to incorporate the colon region imaging further taught by Zur into the endoscopic imaging of Ding in view of Majdoubi further in view of Zur
The motivation for doing so would have been to expand clinical utility of the lesion detection to more physiological systems. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Ding in view of Majdoubi further in view of Zur with the additional teach of Zur to fully disclose, “wherein the determined location associated with the object of interest is a location in at least one of a rectum, sigmoid colon, descending colon, transverse colon, ascending colon, or cecum.”
Regarding claim 34, Ding in view of Majdoubi further in view of Zur teaches “The system of claim 30,”
“wherein the determined size associated with the object of interest is a numeric value or a size classification.” (Majdoubi, Figure 10, shows predicted radius (numeric value) as the determined size. This was incorporated above in the rejection of claim 30 with motivation and rationale.)
Regarding claim 35, Ding in view of Majdoubi further in view of Zur teaches “The system of claim 30,”
“wherein the at least one processor is further configured to: detect a plurality of objects of interest in the plurality of frames; characterize the plurality of objects of interest, the characterization including determining a plurality of sets of features associated with the plurality of objects of interest, wherein a set of features in the plurality of sets of features includes characterization and size information associated with a detected object of interest in the plurality of objects of interest; and present, on the display device, information associated with one or more sets of features in the plurality of sets of features.” (Note that this claim recites overlapping limitations with claim 30, therefore the rejection of claim 30 is applied here. This claim is distinct in that a plurality of objects are detected, characterized, and presented, whereas claim 30 specifies one object of interest. Ding teaches the detection and subsequent analysis and characterization steps on a plurality of objects of interest: Ding, Page 5, Paragraph 8, “For example, after receiving the NBI gastroscope image delivered by the primary screening diagnosis module, the target detection algorithm is used to divide the NBI gastroscope image into regions and extract the first-level index (first-level feature information). The extraction process includes lesion segmentation and feature extraction. The lesion segmentation is to scan the areas where lesions may exist on the NBI gastroscope image, frame possible areas on the graph, and mark suspicious areas; feature extraction mainly completes suspicious areas In the process of feature extraction, the image information data obtained by segmentation is used as input, and the segmented area is selected from the segmented image, and the features such as boundary definition, color, surface smoothness, and shape are extracted, and a preliminary analysis is performed on the image. Annotate to form the second gastroscopy image. In this step, the second gastroscope image can also be interactively displayed on the display device for review by the doctor, which is convenient for the doctor to understand the image.”)
Claims 38 and 40-43 recite a computer implemented method with steps corresponding to the elements of the system recited in Claims 30 and 32-35. Therefore, the recited steps of these claims are mapped to the analogous elements in the corresponding system claims. The rationale and motivation to combine the references is applied here as well.
Regarding claims 46, and 48-51, these claims recite a non-transitory computer readable medium including instructions that when executed by at least one processor, cause the at least one processor to perform operations corresponding to the steps recited in Claims 30 and 32-35. Therefore, the recited programming instructions of these claims are mapped to the analogous steps in the corresponding method claims. Ding discloses a processor (see Figure 3 in the above rejection of claim 30), and the recited non-transitory computer readable medium is a well known feature in the art and is not considered novel. The rationale and motivation to combine the references is applied here as well.
Claim(s) 31, 36, 37, 39, 44, 45, 47, 52 and 53 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ding in view of Majdoubi further in view of Zur further in view of Bryne (Real-time differentiation of adenomatous and hyperplastic diminutive colorectal polyps during analysis of unaltered videos of standard colonoscopy using a deep learning model).
Regarding claim 31, Ding in view of Majdoubi further in view of Zur teaches “The system of claim 30,”
Ding in view of Majdoubi further in view of Zur does not expressly disclose “wherein the at least one processor is further configured to: identify, based on the determined location and size of the object of interest, a medical guideline; and present, on the display device, information associated with the identified medical guideline.”
Byrne discloses “wherein the at least one processor is further configured to: identify, based on the determined location and size of the object of interest, a medical guideline; and present, on the display device, information associated with the identified medical guideline.” (Byrne, Figure 4 shows diagnostic determination (medical guideline) being displayed.)
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to incorporate the medical guideline identification and presentation taught by Byrne into diagnostic strategy of Ding in view of Majdoubi further in view of Zur.
The motivation for doing so would have been to output more details on the lesion to better inform doctors. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Ding in view of Majdoubi further in view of Zur with the above teaching of Byrne to fully disclose the invention of claim 31.
Regarding claim 36, Ding in view of Majdoubi further in view of Zur teaches “The system of claim 30,”
While Ding in view of Majdoubi further in view of Zur discloses identifying a lesion as suspicious (Ding, Page 5, Paragraph 8, “For example, after receiving the NBI gastroscope image delivered by the primary screening diagnosis module, the target detection algorithm is used to divide the NBI gastroscope image into regions and extract the first-level index (first-level feature information). The extraction process includes lesion segmentation and feature extraction. The lesion segmentation is to scan the areas where lesions may exist on the NBI gastroscope image, frame possible areas on the graph, and mark suspicious areas; feature extraction mainly completes suspicious areas In the process of feature extraction, the image information data obtained by segmentation is used as input, and the segmented area is selected from the segmented image, and the features such as boundary definition, color, surface smoothness, and shape are extracted, and a preliminary analysis is performed on the image. Annotate to form the second gastroscopy image. In this step, the second gastroscope image can also be interactively displayed on the display device for review by the doctor, which is convenient for the doctor to understand the image.”), and presenting lesion information (Ding, Page 4, Paragraph 17, “In step S103, the second gastroscope image is received, and the primary feature information and position information in a plurality of the second gastroscope images are combined and analyzed to form secondary feature information and an area position corresponding to the secondary feature information, and output a third gastroscope image labeled with the secondary feature information and the region location information.”), Ding in view of Majdoubi further in view of Zur does not expressly disclose identifying a lesion as non-diminutive and presenting that information.
Byrne discloses identification and presentation of a lesion as non-diminutive (Byrne, Figures 1 and 4(B) shows lesion identification as Type 2 (non-diminutive).)
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to incorporate the non-diminutive identification and presentation taught by Byrne into the lesion characterization and presentation of Ding in view of Majdoubi further in view of Zur.
The motivation for doing so would have been to provide further information to the provider and patient. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Ding in view of Majdoubi further in view of Zur with the above teaching of Byrne to fully disclose, “wherein presenting information indicating the status of the characterization of the object of interest further comprises: identify the object of interest as non-diminutive;”
Ding in view of Majdoubi further in view of Zur further in view of Byrne further disclose, “and generate aggregate information of the object of interest, wherein the aggregate information includes a location and a size of the object of interest in the plurality of frames.” (Ding, Page 4, Paragraphs 15-16, “In step S102, segment the segmented area with lesion features on the first gastroscope image, mark the first-level feature information and position information corresponding to the segmented area on the first gastroscope image as the second gastroscope image, and output The second gastroscopy image. In this step, the first gastroscope image is received, and a target detection algorithm is used to perform region division on the first gastroscope image, including lesion separation and feature extraction. For example, by scanning the region on the first gastroscope image that may have a feature of a lesion, the region on the first gastroscope image that may have a feature of a lesion is segmented, and the segmented area is marked as a suspicious area. Then, the image information data obtained by the frame-selected segmented area is used as input, the segmented area is selected from the first gastroscope image, the positional relationship between the segmented areas is recorded, and the primary feature information contained in the image of the segmented area is extracted , including the extraction of features such as boundary definition, color, surface smoothness, and shape. Finally, a preliminary annotation is performed on the first gastroscope image, so as to obtain a second gastroscope image. In this step, the second gastroscope image can be interactively displayed on the display device.” Note that the location and size determination is mapped to the boundary and size identification.)
Regarding claim 37, Ding in view of Majdoubi further in view of Zur teaches “The system of claim 30,”
While Ding in view of Majdoubi further in view of Zur discloses identifying suspicious vs. non-suspicious lesions (Ding, Page 5, Paragraph 8, “For example, after receiving the NBI gastroscope image delivered by the primary screening diagnosis module, the target detection algorithm is used to divide the NBI gastroscope image into regions and extract the first-level index (first-level feature information). The extraction process includes lesion segmentation and feature extraction. The lesion segmentation is to scan the areas where lesions may exist on the NBI gastroscope image, frame possible areas on the graph, and mark suspicious areas; feature extraction mainly completes suspicious areas In the process of feature extraction, the image information data obtained by segmentation is used as input, and the segmented area is selected from the segmented image, and the features such as boundary definition, color, surface smoothness, and shape are extracted, and a preliminary analysis is performed on the image. Annotate to form the second gastroscopy image. In this step, the second gastroscope image can also be interactively displayed on the display device for review by the doctor, which is convenient for the doctor to understand the image.” Note that the identification of a suspicious lesion requires that in the absence of this identification, a lesion is identified as non-suspicious.), and presenting lesion information (Ding, Page 4, Paragraph 17, “In step S103, the second gastroscope image is received, and the primary feature information and position information in a plurality of the second gastroscope images are combined and analyzed to form secondary feature information and an area position corresponding to the secondary feature information, and output a third gastroscope image labeled with the secondary feature information and the region location information.”), Ding in view of Majdoubi further in view of Zur does not expressly disclose identifying a lesion as diminutive and presenting that information.
Byrne discloses identifying a lesion as diminutive and presenting that information (Byrne, Figures 1 and 4(A) shows lesion identification and presentation as Type 1 (diminutive).)
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to incorporate the diminutive identification and presentation taught by Byrne into the lesion characterization and presentation of Ding in view of Majdoubi further in view of Zur.
The motivation for doing so would have been to provide further information to the provider and patient. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Ding in view of Majdoubi further in view of Zur with the above teaching of Byrne to fully disclose, “wherein presenting information indicating the status of the characterization of the object of interest further comprises: identify the object of interest as diminutive;”.
Ding in view of Majdoubi further in view of Zur further in view of Byrne further disclose “and generate information indicating the status of characterization of the object of interest, wherein the status of characterization includes non-aggregate information of classification of the object of interest.” (Byrne, Figure 4(A), as incorporated above, displays a Type 1 (diminutive) lesion. This is mapped to the status of characterization of non-aggregate information of classification. Note that this was previously incorporated above with motivation and rationale.)
Claims 39, 44, and 45 recite a computer implemented method with steps corresponding to the elements of the system recited in Claims 31, 36, and 37. Therefore, the recited steps of these claims are mapped to the analogous elements in the corresponding system claims. The rationale and motivation to combine the references apply here.
Regarding claims 47, 52, and 53, these claims recite a non-transitory computer readable medium including instructions that when executed by at least one processor, cause the at least one processor to perform operations corresponding to the steps recited in Claims 39, 44, and 45. Therefore, the recited programming instructions of these claims are mapped to the analogous steps in the corresponding method claims. The rationale and motivation to combine the references apply here. Ding discloses a processor (see Figure 3 in the above rejection of claim 30), and the recited non-transitory computer readable medium is a well-known feature in the art and is not considered novel.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AARON JOSEPH SORRIN whose telephone number is (703)756-1565. The examiner can normally be reached Monday - Friday 9am - 5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AARON JOSEPH SORRIN/Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672
Prosecution Insights