DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they do not include the following reference sign(s) mentioned in the description: “memory 240” (see specification [090]), “850” (see specification [191]), and “930” (see specification [197]). Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they include the following reference character(s) not mentioned in the description: “518” (see Fig. 5A and 5B); “641”, “642”, “643”, “644”, “645”, “646”, “647”, “648”, “651”, “652”, “654”, “662”, and “663” (see Fig. 6B and 6D); “680”, “681”, and “682” (see Fig. 6D); “701” (see Fig. 7); “801” and “950” (see Fig. 8); “901” and “940” (see Fig. 9). Corrected drawing sheets in compliance with 37 CFR 1.121(d), or amendment to the specification to add the reference character(s) in the description in compliance with 37 CFR 1.121(b) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because reference character “317” has been used to designate both “detector network 317” (see Fig. 3A) and “other components 312 and 315-317” (see specification [147]), “using networks 315-317” (see specification [149]), and “additional components such as networks 315-317” (see specification [156]); Examiner further notes that “detector network 317” is not described in the specification.
The drawings are objected to because “1090” of Fig. 10 appears to be erroneously separated into “109” and “0” on separate lines.
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
Applicant is reminded of the proper language and format for an abstract of the disclosure.
The abstract should be in narrative form and generally limited to a single paragraph on a separate sheet within the range of 50 to 150 words in length. The abstract should describe the disclosure sufficiently to assist readers in deciding whether there is a need for consulting the full patent text for details.
The language should be clear and concise and should not repeat information given in the title. It should avoid using phrases which can be implied, such as, “The disclosure concerns,” “The disclosure defined by this invention,” “The disclosure describes,” etc. In addition, the form and legal phraseology often used in patent claims, such as “means” and “said,” should be avoided.
The abstract of the disclosure is objected to because the abstract is greater than 150 words in length and not on a separate sheet. A corrected abstract of the disclosure is required and must be presented on a separate sheet, apart from any other text. See MPEP § 608.01(b).
The disclosure is objected to because of the following informalities:
Specification [158] recites, “In some embodiments, pipelines 610 and 630”, where the recitation of “630” appears to be a typographical error and “620” is assumed to be intended.
Appropriate correction is required.
The specification is objected to as failing to provide proper and clear support for the claim term “non-transitory computer-readable medium” recited in claims 10-16. See 37 CFR 1.75(d)(1) and MPEP § 608.01(o). Correction of the following is required:
The specification [042] recites, “[N]on-transitory computer readable mediums may be any medium capable of storing data in any memory in a way that may be read by any computing device with a processor to carry out methods or any other instructions stored in the memory. The non-transitory computer readable medium may be implemented as software, firmware, hardware, or any combination thereof. Software may preferably be implemented as an application program tangibly embodied on a program storage unit or computer readable medium consisting of parts, or of certain devices and/or a combination of devices.”
As the specification disclosure appears to presently include an interpretation where “non-transitory computer readable medium” can be implemented as purely software, the specification description for the term “non-transitory computer readable medium” is ambiguous as to whether “non-transitory computer readable medium” is intended to include purely software embodiments, which would be directed to “software per se”; or is intended to be directed towards non-transitory, physically tangible computer readable medium which embodies or stores corresponding software or application program instructions.
For the purposes of further treating the Application on the merits, the Examiner assumes that the claim term “non-transitory computer readable medium” is not intended to include purely software embodiments, and is intended to be directed towards non-transitory physically tangible computer readable medium.
Appropriate amendments to the specification to correct the ambiguity is required.
As the ambiguity for the claim term stems from the specification description, claims 10-16 recitation for “non-transitory computer readable medium” are presently not rejected under 35 U.S.C. 101 for not falling within at least one of the four categories of patent eligible subject matter.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 13 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 13 recites the limitation "quality score" in the body of the claim. There is insufficient antecedent basis for this limitation in the claim. Claim 13 is noted to be dependent upon claim 10, however, claim 10 does not provide sufficient antecedent basis for the term “quality score”.
For the purposes of further treating the application on the merits, the Examiner assumes that claim 13 is intended to be dependent upon claim 12, which recites “a vector of quality scores”.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-11, 13, and 17-25 are provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-11, 13, and 17-25 of co-pending Application No. 18/348,785 (reference application). Although the claims at issue are not identical, they are not patentably distinct from each other because they recite common subject matter.
The following table illustrates exemplary conflicting claim pairs and the corresponding mapping of limitations.
Instant Application (U.S. App. No. 18/992,155)
Co-Pending Application (U.S. App. No. 18/348,785)
Claim
Limitations
Claim
Limitations
1
A computer-implemented system for processing images, comprising:
1
A system for processing images, comprising:
one or more memory devices storing processor-executable instructions; and
one or more memory devices storing processor-executable instructions; and
one or more processors configured to execute instructions to cause the system to perform operations to perform a plurality of tasks on a set of images, the operations comprising:
one or more processors configured to execute instructions to cause the system to perform operations to perform plurality of tasks on a set of images, the operations comprising:
receive the plurality of tasks, wherein at least one task of the plurality of tasks is associated with a request to identify at least one feature of interest in the set of images;
receiving the plurality of tasks, wherein at least one task of the plurality of tasks is associated with a request to identify at least one feature of interest in the set of images;
analyze, using a local spatio-temporal processing module, a subset of images of the set of images to identify a presence of characteristics associated with the at least one feature of interest; and
analyzing, using a local spatio-temporal processing module, a subset of images of the set of images to identify presence of characteristics associated with the at least one feature of interest; and
iterate execution of a timeseries analysis module for each task of the plurality of tasks to associate a numerical score for each task with each image of the subset of images.
iterating execution of a timeseries analysis module for each task of the plurality of tasks to associate a numerical score for each task with each image of the subset of images, wherein performing at least one of the plurality of tasks generates a metric that improves an assessment of a medical procedure.
Independent claims 10 and 17 are similarly rejected for reciting similar subject matter as that of conflicting claims 10 and 17, respectively, of the conflicting reference application.
Dependent claims 2-9 are rejected for reciting similar subject matter as that of corresponding conflicting claims 2-9, respectively, of the conflicting reference application.
Dependent claims 11 and 13 are rejected for reciting similar subject matter as that of corresponding conflicting claims 11 and 13, respectively, of the conflicting reference application.
Dependent claims 18-25 are rejected for reciting similar subject matter as that of corresponding conflicting claims 18-25, respectively, of the conflicting reference application.
This is a provisional nonstatutory double patenting rejection because the patentably indistinct claims have not in fact been patented.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-25 are rejected under 35 U.S.C. 101 because the claimed invention is directed to the judicial exception of mental process type abstract idea (concepts performable in the human mind, including an observation, evaluation, judgement, and opinion) without significantly more.
Independent claim 1 recites the subject matter, “analyze, using a local spatio-temporal processing module, a subset of images of the set of images to identify a presence of characteristics associated with the at least one feature of interest; and iterate execution of a timeseries analysis module for each task of the plurality of tasks to associate a numerical score for each task with each image of the subset of images”.
Similarly, independent claims 10 and 17 recites,
“analyzing, using a local spatio-temporal processing module, a subset of images of the set of images to identify a presence of characteristics associated with the at least one feature of interest; and iterating execution of a timeseries analysis module for each task of the plurality of tasks to associate a numerical score for each task with each image of the subset of images” (claim 10);
“analyzing, using a local spatio-temporal processing module, a subset of images of the set of images to identify a presence of characteristics associated with at least one feature of interest; and iterating execution of a timeseries analysis module for each task of the plurality of tasks to associate a numerical score for each task with each image of the subset of images” (claim 17).
The noted subject matter of claims 1, 10, and 17 refers to analyzing a subset of images of the set of images to identify a presence of characteristics associated with at least one feature of interest, and, for each task of the plurality of tasks, iteratively associating a numerical score for each task with each image of the subset of images, which are described with a high level of generality which a person may practically perform in the human mind by viewing and evaluating the images of the subset of images to mentally determine the presence of characteristics of a feature of interest associated with a task for identifying the feature of interest and mentally assigning a score to the viewed image for the presence of the feature of interest. Thus, the broadest reasonable interpretation, in light of the specification, of the claimed subject matter directs to performing mental observations and evaluations, falling within the “mental processes” grouping of abstract ideas.
The judicial exceptions of claims 1 and 8 are not integrated into a practical application because the additional claim limitations of, “receive the plurality of tasks, wherein at least one task of the plurality of tasks is associated with a request to identify at least one feature of interest in the set of images” (claim 1) and “receiving the plurality of tasks, wherein at least one task of the plurality of tasks is associated with a request to identify at least one feature of interest in the set of images” (claims 10 and 17); describe the data gathering steps and selecting particular data source or type of data to be manipulated, such that the claims merely add insignificant extra-solution activity to the judicial exceptions. See MPEP 2106.04(d) and MPEP 2106.05(g).
The judicial exception of claims 1, 10, and 17 are further not integrated into a practical application because the additional claim limitations of, “one or more memory devices” (claim 1), “one or more processors” or “at least one processor” (claims 1, 10, and 17), “local spatio-temporal processing module” (claims 1, 10, and 17), and “timeseries analysis module” (claims 1, 10, and 17) describe generic computing elements and generic units performing functions, which given its broadest reasonable interpretation, includes the use of generic computing elements to implement the noted abstract idea with a high level of generality, such that the claims amounts to merely implementing the abstract idea on generic computing elements. See MPEP 2106.04(d), MPEP 2106.05(b), and MPEP 2106.05(f).
Claims 1, 10, and 17 do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the above noted additional claim limitations, similarly as discussed above, continue to merely encompass performing additional insignificant extra solution activity to the noted abstract idea activity, and are insufficient to provide significantly more than the noted judicial exceptions. See MPEP 2106.05(g). Claims 1, 10, and 17 above noted additional claim limitations which describe generic computing elements and generic units performing functions, which given their broadest reasonable interpretation, continue to merely encompass the use of generic computing elements to implement the noted abstract idea with a high level of generality, such that the claims amounts to merely implementing the abstract idea on generic computing elements, and are insufficient to provide significantly more than the noted judicial exceptions. See MPEP 2106.05(b) and MPEP 2106.05(f). Furthermore, in consideration of the claims 1, 10, and 17 additional elements as a combination, the additional elements continue to merely perform additional insignificant extra solution activity to the noted abstract idea activity and implementing the abstract idea on generic computing elements, and thus do not provide significantly more than the noted judicial exception.
The claims 2-7, 9, 11-16, 18-23, and 25 do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the claims recite additional subject matter of,
“outputs subsets of analyzed images of the set of images, wherein each subset is associated with a task of the plurality of tasks” (claims 2, 11, and 18);
“determines the presence of characteristics by determining a vector of quality scores, wherein each quality score in the vector of quality scores corresponds to each image of the subset of images” (claims 3, 12, and 19);
“wherein each quality score is an ordinal number between 0 and R, wherein a score 0 represents minimum quality and a score R represents maximum quality” (claims 4, 13, and 20);
“generates a set of feature vectors for features of interest related to the plurality of tasks” (claims 5, 14, and 21)
“analyze sets of feature vectors for the subset of images analyzed by the local spatio-temporal processing module” (claims 6, 15, and 22)
“aggregate output of the local spatio-temporal processing module for each task of the plurality of tasks” (claims 7, 16, and 23)
“a presence of at least one feature of interest is determined from a portion of the set of images” (claims 9 and 25) which refer to additional steps of the noted mental process type abstract idea that are described with a high level of generality in which a person may continue to practically perform the broadest reasonable interpretation of the claimed steps within the human mind. See MPEP 2106.04 II. A. 2. and MPEP 2106.04(a)(2) III.
In regard to claims 2, 11, and 18 subject matter, a person can indicate the subset of images which have been analyzed for detecting the presence of the feature of interest. See MPEP 2106.04(a)(2) III.
In regard to claims 3, 12, and 19 subject matter, a person can mentally assign a quality score to each of the analyzed images, where the corresponding quality scores of the subset of analyzed images would correspond to a vector of quality scores, where the claims are noted to not recite limitations which require complex calculations or analysis for determining the quality scores beyond the capability of a human mind. See MPEP 2106.04(a)(2) III. and MPEP 2106.04(a)(2) III. B
In regard to claims 4, 13, and 20 subject matter, a person can mentally assign a quality score within an ordinal number range between minimum score 0 and a maximum score R for the analyzed images, where the claims are noted that to not recite limitations which require complex calculations or analysis for determining the quality scores beyond the capability of a human mind. See MPEP 2106.04(a)(2) III. and MPEP 2106.04(a)(2) III. B
In regard to claims 5, 14, and 21 subject matter, a person can mentally construct feature vectors for the features of interest to be detected in the images, where the claims are noted to not recite limitations which require complex calculations or analysis for generated the feature vectors beyond the capability of a human mind. See MPEP 2106.04(a)(2) III. and MPEP 2106.04(a)(2) III. B
In regard to claims 6, 15, and 22 subject matter, a person can mentally analyze sets of feature vectors for the analyzed subsets of images where the claims are noted to not recite limitations which require complex calculations or analysis beyond the capability of a human mind. See MPEP 2106.04(a)(2) III. and MPEP 2106.04(a)(2) III. B
In regard to claims 7, 16, and 23 subject matter, a person can mentally collect and consider the results of the analyzed subsets of images for detecting the presence of the feature of interest, where the claims are noted to not recite limitations which require complex calculations or analysis beyond the capability of a human mind. See MPEP 2106.04(a)(2) III. and MPEP 2106.04(a)(2) III. B
In regard to claims 9 and 25 subject matter, a person can view and analyze a portion of the set of images to determine the presence of the feature of interest. See MPEP 2106.04(a)(2) III.
If the additional claim elements merely recite another judicial exception, that is insufficient to integrate the judicial exception into a practical application. See, e.g., RecogniCorp, LLC v. Nintendo Co., 855 F.3d 1322, 1327, 122 USPQ2d 1377 (Fed. Cir. 2017). See MPEP 2106.04 II. A. 2.
Claims 2-7, 9, 11-16, 18-23, and 25 do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the above noted additional claim limitations, similarly as discussed above, continue to merely encompass performing respective additional steps of the mental processes type abstract idea, and are insufficient to provide significantly more than the noted judicial exceptions. See MPEP 2106.04 II. A. 2. and MPEP 2106.04(a)(2) III. Furthermore, in consideration of the claims 2-7, 9, 11-16, 18-23, and 25 additional elements as a combination, the additional elements continue to merely perform respective additional steps of the mental process activity, and do not provide significantly more than the noted judicial exception.
Claims 2-7, 9, 11-16, 18-23, and 25 do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the above noted additional claim limitations, similarly as discussed above, continue to merely encompass performing respective additional steps of the mental processes type abstract idea, and are insufficient to provide significantly more than the noted judicial exceptions. See MPEP 2106.04 II. A. 2. and MPEP 2106.04(a)(2) III. Furthermore, in consideration of the claims 2-7, 9, 11-16, 18-23, and 25 additional elements as a combination, the additional elements continue to merely perform respective additional steps of the mental process activity, and do not provide significantly more than the noted judicial exception.
Claims 6, 15, and 22 are not integrated into a practical application because the additional claimed subject matter of “global spatio-temporal processing module” describe a generic unit for performing functions, which given its broadest reasonable interpretation, includes the use of generic computing elements to implement the noted abstract idea with a high level of generality, such that the claims amounts to merely implementing the abstract idea on generic computing elements. See MPEP 2106.04(d), MPEP 2106.05(b), and MPEP 2106.05(f).
Claims 6, 15, and 22 do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the above noted additional claim limitations, similarly as discussed above, continue to merely encompass the use of generic computing elements to implement the noted abstract idea with a high level of generality, such that the claims amounts to merely implementing the abstract idea on generic computing elements, and are insufficient to provide significantly more than the noted judicial exceptions. See MPEP 2106.05(b) and MPEP 2106.05(f). Furthermore, in consideration of the claims 6, 15, and 22 additional element as a combination, the additional element continue to merely implement the abstract idea on generic computing elements, and thus do not provide significantly more than the noted judicial exception.
Claims 8 and 24 do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional claimed subject matter of “received directly from an imaging device during a medical procedure” describe data gathering steps and selecting particular data source or type of data to be manipulated, such that the claims merely add insignificant extra-solution activity to the judicial exceptions. See MPEP 2106.04(d) and MPEP 2106.05(g).
Claims 8 and 24 do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the above noted additional claim limitations, similarly as discussed above, continue to merely encompass performing additional insignificant extra solution activity to the noted abstract idea activity, and are insufficient to provide significantly more than the noted judicial exceptions. See MPEP 2106.05(g). Furthermore, in consideration of the claims 8 and 24 additional elements as a combination, the additional elements continue to merely perform additional insignificant extra solution activity to the noted abstract idea activity, and do not provide significantly more than the noted judicial exception.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 5-11, 14-18, and 21-25 are rejected under 35 U.S.C. 103 as being unpatentable over Kumar et al. (US 2012/0316421), herein Kumar, in view of Adewole et al. (“Graph Convolutional Neural Network For Weakly Supervised Abnormality Localization In Long Capsule Endoscopy Videos”), herein Adewole.
Regarding claim 1, Kumar discloses a computer-implemented system for processing images, comprising:
one or more memory devices storing processor-executable instructions (see Kumar [0186]-[0190], where a computer system includes one or more memory devices storing computer software for implementing the disclosed teachings); and
one or more processors configured to execute instructions (see Kumar [0186]-[0190], where the computer system includes one or more processors for implementing the disclosed teachings according to stored software) to cause the system to perform operations to perform a plurality of tasks on a set of images (see Kumar [0199]-[0200], where received endoscopic images are processed to detect an attribute of interest, determine whether an attribute of interest is present in the images based on predetermined criterion, classifying a set of images that contains at least one attribute of interest, and classifying another set of images that does not contain at least one attribute of interest; see also Kumar Fig. 14 and [0061]-[0070]), the operations comprising:
receive the plurality of tasks, wherein at least one task of the plurality of tasks is associated with a request to identify at least one feature of interest in the set of images (see Kumar Fig. 14 and [0061]-[0070] and [0199]-[0200], where received image data from an imager are processed to perform feature extraction, classification, and severity assessment to determine the presence and severity of an attribute of interest in the images; see Kumar [0186]-[0200], where instructions are provided to the computer system for implementing the disclosed teachings, thus suggesting that the instructions to perform the disclosed image processes are received);
analyze, using a local spatio-temporal processing module, a subset of images of the set of images to identify a presence of characteristics associated with the at least one feature of interest (see Kumar [0063]-[0069], where feature extraction, classification, and severity assessment is performed on the images to determine the presence of a lesion or other attribute of interest in the images; and see Kumar [0206]-[0207], where the images are taken within a gastrointestinal track and the attribute of interest includes anatomic abnormalities in the gastrointestinal track)
While Kumar teaches that a severity assessment score is calculated for a located lesion or attribute of interest (see Kumar [0068]-[0069]), and that machine learning can be applied to the classification and ranking of severity (see Kumar [0067], [0091]-[0093], and [0100]);
Kumar does not explicitly disclose iterate execution of a timeseries analysis module for each task of the plurality of tasks to associate a numerical score for each task with each image of the subset of images.
Adewole teaches in a related and pertinent temporal abnormality localization for long endoscopy videos based on graph convolutional neural network models (see Adewole Abstract), where features are extracted from endoscopy videos with a pretrained network model and segmented into video segments, the videos are represented as graphs and processed by a graph convolutional neural network and trained to identify abnormal frames, and the trained parameters are used with a temporal pool layer to generate a class activation maps over the sequence of frames in the video and the nodes with high activation scores for abnormal frames are identified (see Adewole sect. III. A. Overview of Framework and sect. III. F. Graph Localization Network), where the training datasets include different disease categories are used to predict an abnormal label (see Adewole Fig. 6, sect. III. E. Graph Convolutional Network, and Table I).
At the time of filing, one of ordinary skill in the art would have found it obvious to apply the teachings of Adewole to the teachings of Kumar, such that lesions or other attributes of interest are localized by repeatedly generating class activation maps for the sequence of endoscopic image frames in the endoscopic video.
This modification is rationalized as an application of a known technique to a known system ready for improvement to yield predictable results.
In this instance, Kumar discloses a base system to process endoscopic images to detect an attribute of interest, where feature extraction, classification, and severity assessment is performed on the images to determine the presence of a lesion or other attribute of interest in the images.
Adewole teaches a known technique for localizing abnormal video frames of an endoscopy video, where a pretrained network model is used to extract feature vectors from endoscopy videos, the video is segmented into video segments and represented as graphs to be processed by a graph convolutional neural network trained to identify abnormal frames, and a temporal pool layer is used to generate class activation maps over the sequence of frames in the video, where nodes, representing video frames, with high activation scores are identified for abnormal frames.
One of ordinary skill in the art would have recognized that by applying Adewole’s technique would allow for the system of Kumar to localize lesions or other attributes of interest are localized by repeatedly generating class activation maps for the sequence of endoscopic image frames in the endoscopic video, predictably leading to an improved system for processing endoscopic video images to detect and localize an attribute of interest, such as lesions, in the video images.
Regarding claim 2, please see the above rejection of claim 1. Kumar and Adewole disclose the system of claim 1, wherein the local spatio-temporal processing module outputs subsets of analyzed images of the set of images, wherein each subset is associated with a task of the plurality of tasks (see Kumar [0063]-[0069], where feature extraction, classification, and severity assessment is performed on the images to identify images sequences with the presence of a lesion or other attribute of interest in the images, and feature extraction can be used to filter ay normal or unusable data from image data and include removing unusable images from further consideration).
Regarding claim 5, please see the above rejection of claim 1. Kumar and Adewole disclose the system of claim 1, wherein the local spatio-temporal processing module generates a set of feature vectors for features of interest related to the plurality of tasks (see Kumar [0063]-[0068], where feature vectors and localized descriptors are generated from the image data for identifying the lesion or attribute of interest).
Regarding claim 6, please see the above rejection of claim 1. Kumar and Adewole disclose the system of claim 1, wherein the operations further comprise:
analyze, using a global spatio-temporal processing module, sets of feature vectors for the subset of images analyzed by the local spatio-temporal processing module (see Kumar [0067]-[0068], where temporal consistency in the classification of images may be used and a second classification procedure can be performed on neighboring images; see Adewole sect. III. A. Overview of Framework and sect. III. E. Graph Convolutional Network, where the graph convolutional network processes video segments represented as graphs where each node (frame) feature is transformed to a weighted average of the neighboring features).
Regarding claim 7, please see the above rejection of claim 1. Kumar and Adewole disclose the system of claim 1, wherein the operations further comprise:
aggregate output of the local spatio-temporal processing module for each task of the plurality of tasks using the timeseries analysis module (see Adewole sect. III. A. Overview of Framework and sect. III. E. Graph Convolutional Network, where the graph convolutional network processes video segments represented as graphs where each node (frame) feature is transformed to a weighted average of the neighboring features, providing a weighted aggregation of all neighboring node features).
Regarding claim 8, please see the above rejection of claim 1. Kumar and Adewole disclose the system of claim 1, wherein the set of images are received directly from an imaging device during a medical procedure (see Kumar Fig. 16, [0199] and [0206]-[0207], where the endoscopic images are captured from an endoscopic imaging device, where the images are taken within a gastrointestinal track to identify abnormalities).
Regarding claim 9, please see the above rejection of claim 1. Kumar and Adewole disclose the system of claim 1, wherein a presence of at least one feature of interest is determined from a portion of the set of images (see Kumar Fig. 12 and [0063]-[0064], where a lesion is detected on a region of an image).
Regarding claim 10, it recites a non-transitory computer readable medium performing the system functions of claim 1. Kumar and Adewole teach a non-transitory computer readable medium performing the system functions of claim 1 (see Kumar [0186]-[0190], where a computer system includes one or more memory devices storing computer software for implementing the disclosed teachings). Please see above for detailed claim analysis.
Please see the above rejection for claim 1, as the rationale to combine the teachings of Kumar and Adewole are similar, mutatis mutandis.
Regarding claim 11, see above rejection for claim 10. It is a non-transitory computer readable medium claim reciting similar subject matter as claim 2. Please see above claim 2 for detailed claim analysis as the limitations of claim 11 are similarly rejected.
Regarding claim 14, see above rejection for claim 10. It is a non-transitory computer readable medium claim reciting similar subject matter as claim 5. Please see above claim 5 for detailed claim analysis as the limitations of claim 14 are similarly rejected.
Regarding claim 15, see above rejection for claim 10. It is a non-transitory computer readable medium claim reciting similar subject matter as claim 6. Please see above claim 6 for detailed claim analysis as the limitations of claim 15 are similarly rejected.
Regarding claim 16, see above rejection for claim 10. It is a non-transitory computer readable medium claim reciting similar subject matter as claim 7. Please see above claim 7 for detailed claim analysis as the limitations of claim 16 are similarly rejected.
Regarding claim 17, it recites a computer-implemented method performing the system functions of claim 1. Kumar and Adewole teach the method by performing the system functions of claim 1. Please see above for detailed claim analysis.
Please see the above rejection for claim 1, as the rationale to combine the teachings of Kumar and Adewole are similar, mutatis mutandis.
Regarding claim 18, see above rejection for claim 17. It is a method claim reciting similar subject matter as claim 2. Please see above claim 2 for detailed claim analysis as the limitations of claim 18 are similarly rejected.
Regarding claim 21, see above rejection for claim 17. It is a method claim reciting similar subject matter as claim 5. Please see above claim 5 for detailed claim analysis as the limitations of claim 21 are similarly rejected.
Regarding claim 22, see above rejection for claim 17. It is a method claim reciting similar subject matter as claim 6. Please see above claim 6 for detailed claim analysis as the limitations of claim 22 are similarly rejected.
Regarding claim 23, see above rejection for claim 17. It is a method claim reciting similar subject matter as claim 7. Please see above claim 7 for detailed claim analysis as the limitations of claim 23 are similarly rejected.
Regarding claim 24, see above rejection for claim 17. It is a method claim reciting similar subject matter as claim 8. Please see above claim 8 for detailed claim analysis as the limitations of claim 24 are similarly rejected.
Regarding claim 25, see above rejection for claim 17. It is a method claim reciting similar subject matter as claim 9. Please see above claim 9 for detailed claim analysis as the limitations of claim 25 are similarly rejected.
Claims 3-4, 12-13, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Kumar and Adewole as applied to claims 1, 10, and 17 above, and further in view of Gottlieb et al. (“Central Reading of Ulcerative Colitis Clinical Trial Videos Using Neural Networks”), herein Gottlieb.
Regarding claim 3, please see the above rejection of claim 1. Kumar and Adewole do not explicitly disclose the system of claim 1, wherein the local spatio- temporal processing module determines the presence of characteristics by determining a vector of quality scores, wherein each quality score in the vector of quality scores corresponds to each image of the subset of images.
Gottlieb teaches in a related and pertinent method for analyzing full length endoscopy videos with a trained neural network model to predict endoscopic Mayo score (eMS) and Ulcerative Colitis Endoscopic Index of Severity (UCEIS) score (see Gottlieb Abstract), where data preprocessing is performed for each frame of the video to assign a prediction for visual clarity to determine clear frames that meet a minimum clarity threshold to be included, and each clear frame is further assigned a bowel prep score that can be graded on a scale from 0-3 (see Gottlieb p. 712, left column).
At the time of filing, one of ordinary skill in the art would have found it obvious to apply the teachings of Gottlieb to the teachings of Kumar and Adewole, such that data preprocessing is performed for the endoscopic video image frames to determine clear frames that meet a minimum clarity threshold and grading the image frames with a prep quality score.
This modification is rationalized as an application of a known technique to a known system ready for improvement to yield predictable results.
In this instance, Kumar and Adewole discloses a base system to process endoscopic video images to detect an attribute of interest, where feature extraction, classification, and severity assessment is performed on the images to determine and localize the presence of a lesion or other attribute of interest in the endoscopic video images.
Gottlieb teaches a known technique for performing data preprocessing for each frame of endoscopic video to assign a prediction for visual clarity to determine clear frames that meet a minimum clarity threshold to be included, and each clear frame is further assigned a bowel prep score that can be graded on a scale from 0-3.
One of ordinary skill in the art would have recognized that by applying Gottlieb’s technique would allow for the system of Kumar and Adewole to perform data preprocessing for the endoscopic video image frames and determine clear frames that meet a minimum clarity threshold and grading the image frames with a prep quality score, predictably leading to an improved system for processing endoscopic video images to detect and localize an attribute of interest, such as lesions, in the video by identifying video image frames with adequate clarity and prep quality.
Regarding claim 4, please see the above rejection of claim 3. Kumar, Adewole, and Gottlieb disclose the system of claim 3, wherein each quality score is an ordinal number between 0 and R, wherein a score 0 represents minimum quality and a score R represents maximum quality (see Gottlieb p. 712, left column, where each clear frame that meets a minimum clarity threshold is further assigned a bowel prep score that can be graded on a scale from 0-3).
Regarding claim 12, see above rejection for claim 10. It is a non-transitory computer readable medium claim reciting similar subject matter as claim 3. Please see above claim 3 for detailed claim analysis as the limitations of claim 12 are similarly rejected.
Please see the above rejection for claim 3, as the rationale to combine the teachings of Kumar, Adewole, and Gottlieb are similar, mutatis mutandis.
Regarding claim 13, see above rejection for claim 10. It is a non-transitory computer readable medium claim reciting similar subject matter as claim 3. Please see above claim 3 for detailed claim analysis as the limitations of claim 13 are similarly rejected.
Please see the above rejection for claim 3, as the rationale to combine the teachings of Kumar, Adewole, and Gottlieb are similar, mutatis mutandis.
Regarding claim 19, see above rejection for claim 17. It is a method claim reciting similar subject matter as claim 3. Please see above claim 3 for detailed claim analysis as the limitations of claim 19 are similarly rejected.
Please see the above rejection for claim 3, as the rationale to combine the teachings of Kumar, Adewole, and Gottlieb are similar, mutatis mutandis.
Regarding claim 20, see above rejection for claim 19. It is a method claim reciting similar subject matter as claim 4. Please see above claim 4 for detailed claim analysis as the limitations of claim 20 are similarly rejected.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Biffi et al. (“A novel AI device for real-time optical characterization of colorectal polyps”, listed in the IDS, filed 7 January 2025) is pertinent in teaching a computer aided diagnosis device for performing real-time optical characterization of colorectal polyps in endoscopy video.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TIMOTHY WING HO CHOI whose telephone number is (571)270-3814. The examiner can normally be reached 9:00 AM to 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, VINCENT RUDOLPH can be reached at (571) 272-8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TIMOTHY CHOI/Examiner, Art Unit 2671
/VINCENT RUDOLPH/Supervisory Patent Examiner, Art Unit 2671
