DETAILED ACTION
Remarks
This office action is issued in response to communication filed on 4/16/2026 . Claims 1-8 , 10-21 and 23-34 and 36-40 are pending in this Office Action.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed on 4/16/2026 with respect to 35 USC 103 rejection have been considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Vasilakakis et al. “Weakly supervised multilabel classification for semantic interpretation of endoscopy video frames”, Evolving Systems (publication 2020), hereinafter “ Vasilakakis”.
Allowable Subject Matter
Claims 4-5,17-18 and 30-31 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1,10-12,14,23-25,27 and 36-38 are rejected under 35 U.S.C. 103 as being unpatentable over Holmstrom.(US Patent Application Publication 2023/0346392A1, hereinafter “Holmstrom”), and further in view of Vasilakakis
As to claim 1, Holmstrom teaches a method for controlling a fluid pump for use in surgical procedures, the method comprising: receiving video data captured from an imaging tool configured to image an internal portion of a patient (Holmstrom par [0042] teaches receiving an image of the view of the area .Holmstrom par [0042] teaches the image analysis engine receives an image of the view of the area within the body of the of the patient that is within a field of view of vision of the endoscope) ;
[applying one or more machine learning classifiers to the received video data to generate one or more classification metrics based on the received video data, wherein the one or more machine learning classifiers are created using a supervised training process that comprises using one or more annotated images to train the one or more machine learning classifier , wherein the one or more machine learning classifiers comprise an image clarity classifier configured to generate one or more classification metrics associated with a presence of at least one of blood, turbidity, bubles, smoke, or debris in the received video data ];
determining the presence of one or more conditions in the received video data based on the generated one or more classification metrics; and adjusting a setting for the flow through or head pressure from the fluid pump based on the determined presence of the one or more conditions in the received video data. (Holmstrom par [0043]-[0045] teaches the image analysis engine can determine a characteristic of the image received using neural network. Holmstrom par [0053] teaches the control engine is to control the medium management based on the characteristics of the image )
Holmstrom fails to expressly teach applying one or more machine learning classifiers to the received video data to generate one or more classification metrics based on the received video data,
wherein the one or more machine learning classifiers are created using a supervised training process that comprises using one or more annotated images to train the one or more machine learning classifier ,
wherein the one or more machine learning classifiers comprise an image clarity classifier configured to generate one or more classification metrics associated with a presence of at least one of blood, turbidity, bubbles, smoke, or debris in the received video data.
However, Vasilakakis teaches applying one or more machine learning classifiers to the received video data to generate one or more classification metrics based on the received video data, (Vasilakakis section 3.2 teaches weakly-supervised classification. Section 3.3 teaches multi-label classification)
wherein the one or more machine learning classifiers are created using a supervised training process that comprises using one or more annotated images to train the one or more machine learning classifier , (Vasilakakis section 3.2 , “network was pre-trained using non-medical video frames and an SVM was used for classification)
wherein the one or more machine learning classifiers comprise an image clarity classifier configured to generate one or more classification metrics associated with a presence of at least one of blood, turbidity, bubbles, smoke, or debris in the received video data.( Vasilakakis section 3.2, page 411, “Jia and Meng (2018) replaced the second fully connected layer of a CNN with an SVM to detect blood. Vasilakakis section 3.3, page 412 right column, teaches “in the context of the endoscopic video frame classification investigated by this paper, five binary classifiers are used to determine the existence of each of the five categories of content considered, e.g., the existence of abnormalities or not, the existence of debris or not, etc. )
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teaching of Holmstrom and Vasilakakis to achieve the claimed invention. One would have been motivated to make such combination to provide enhanced discrimination of the gastrointestinal abnormalities.( Vasilakakis’s abstract)
As to claim 10, Holmstrom and Vasilakakis teach the method of claim 1, wherein the image clarity machine classifier is configured to generate one or more classification metrics associated with an amount of blood visible in the received video data.( Vasilakakis section 3.2, page 412, teaches color histograms have been used for bleeding dectection)
As to claim 11, Holmstrom and Vasilakakis teach the method of claim 1, wherein the image clarity machine classifier is configured to generate one or more classification metrics associated with an amount of bubbles visible in the received video data. ( Vasilakakis section 1, introduction, teaches “considering that the video frame features extracted from these contents are usually different(e.g., bubbles include white reflections, debris has green/yellow hues) the proposed approach identifies them as members of separate classes , aiming to simplify the detection of abnormalities)
As to claim 12, Holmstrom and Vasilakakis teach the method of claim 1 wherein the image clarity machine classifier is configured to generate one or more classification metrics associated with an amount of debris visible in the received video data. (Vasilakakis’s abstract teaches in the context of gastrointestinal video-endoscopy, addressed in this study, the semantics of the normal contents of the endoscopic video frames include normal mucosal tissues, bubbles, debris and the hole of the lumen, whereas the abnormal video frames may include additional semantics corresponding to lesions or blood)
Claims 14 and 23-25 merely recites a system to perform the method of claims 1 and 10-12 respectively. Accordingly, Holmstrom and Vasilakakis teach every limitation of Claims 14 and 23-25 as indicates in the above rejection of claims 1 and 10-12 respectively.
Claims 27 and 36-38 merely recites a non-transitory computer readable storage medium storing one or more program when executed by a processor , performs the method of claims 1 and 10-12 respectively. Accordingly, Holmstrom and Vasilakakis teach every limitation of Claims 27 and 36-38 as indicates in the above rejection of 1 and 10-12 respectively.
Claims 2-3 ,15-16 and 28-29 are rejected under 35 U.S.C. 103 as being unpatentable over Holmstrom , Vasilakakis and further in view of Sreenivasan et al.(US Patent Application Publication 2019/0362835 A1, hereinafter “Sreenivasan”)
As to claim 2, Holmstrom and Vasilakakis teach the method of claim 1 but fail to teach wherein the one or more machine learning classifiers comprises a joint type machine learning classifier configured to generate one or more classification metrics associated with identifying a type of joint pictured in the received video data.
However, Sreenivasan teaches wherein the one or more machine learning classifiers comprises a joint type machine learning classifier configured to generate one or more classification metrics associated with identifying a type of joint pictured in the received video data.( Sreenivasan [0032] teaches the system comprises an image modality classifier trained to utilize one or more parameters, features or other aspects of an image to determine the imaging modality utilized to obtain the image. Sreenivasan [0054] teaches the image comprises the elbow region of a right arm)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teaching of Holmstrom, Vasilakakis and Sreenivasan to achieve the claimed invention. One would have been motivated to make such combination to improve patient care.( Sreenivasan par [0004])
As to claim 3, Holmstrom, Vasilakakis and Sreenivasan teach the method of claim 2, wherein the joint type machine learning classifier is configured to identify one or more joints selected from the group consisting of a hip, a shoulder, a knee, an ankle, a wrist, and an elbow. (Sreenivasan [0054] teaches the image comprises the elbow region of a right arm)
As to claims 15-16 and 28-29 , see the above rejection of claims 2-3 respectively.
Claims 6-8,19-21 and 32-34 are rejected under 35 U.S.C. 103 as being unpatentable over Holmstrom , Vasilakakis and further in view of Kumar et al.(US Patent Application Publication 2021/0236227 A1, hereinafter “Kumar”
As to claim 6, Holmstrom and Vasilakakis teach the method of claim 1 but fail to teach wherein the one or more machine learning classifiers comprises an instrument identification machine classifier configured to generate one or more classification metrics associated with identifying one or more instruments in the received video data.
However, Kumar teaches wherein the one or more machine learning classifiers comprises an instrument identification machine classifier configured to generate one or more classification metrics associated with identifying one or more instruments in the received video data. (Kumar par [0015] teaches In example situations involving a large collection of surgical instruments, the machine (e.g., functioning as an instrument classifier) may act as an identification tool to quickly find the corresponding types of several instruments by scanning them in real time)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teaching of Holmstrom , Vasilakakis and Kumar to achieve the claimed invention. One would have been motivated to make such combination to quickly find the corresponding types of several instruments by scanning them in real time.(Kumar par [0015])
As to claim 7, Holmstrom , Vasilakakis and Kumar teach the method of claim 6, wherein the instrument identification machine classifier is configured to identify instruments selected from the group consisting of a shaver tool, a radio frequency (RF) probe, and a dedicated suction device. (Kumar par [0014] for surgical instruments, examples of instrument types include graspers (e.g., forceps), clamps (e.g., occluders), needle drivers (e.g., needle holders), retractors, distractors, cutters, specula, suction tips, sealing devices, scopes, probes, and calipers)
As to claim 8, Holmstrom, Vasilakakis and Kumar teach the method of claim 6, wherein the fluid pump is configured to activate a suction functionality of the one or more instruments based on the one or more classification metrics generated by the instrument identification machine classifier.( Holmstrom par [0053] teaches the control engine is to control the medium management based on the characteristics of the image. Kumar par [0014] for surgical instruments, examples of instrument types include graspers (e.g., forceps), clamps (e.g., occluders), needle drivers (e.g., needle holders), retractors, distractors, cutters, specula, suction tips, sealing devices, scopes, probes, and calipers)
As to claims 19-21 and 32-34, see the above rejection of claims 6-8.
Claims 13, 26 and 39 are rejected under 35 U.S.C. 103 as being unpatentable over Holmstrom , Vasilakakis and further in view of Campanella et al.(US Patent Application Publication 2019/0197362 A1, hereinafter “Campanella”)
As to claim 13, Holmstrom and Vasilakakis teach the method of claim 1 but fail to teach wherein determining the presence of one or more conditions in the received video data based on the generated one or more classification metrics comprises determining if a clarity of the video is above a pre-determined threshold, and wherein the determination is based on the one more classification metrics generated by the image clarity machine classifier.
However, Campanella determining if a clarity of the video is above a pre-determined threshold and wherein the determination is based on the one more classification metrics generated by the image clarity machine classifier.( Campanella par [0090] teaches the entire image 212 can be marked as blurry if more than 20%, more than 30%, more than 40%, more than 50%, more than 60%, or more than 70% of the patches 214 have a blur score above the predetermined threshold)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teaching of Holmstrom, Vasilakakis and Campanella to achieve the claimed invention. One would have been motivated to make such combination to improve the analysis of the video image.
As to claim 26 and 39, see the above rejection of claim 13.
Claim 40 is rejected under 35 U.S.C. 103 as being unpatentable over Holmstrom , Vasilakakis and further in view of Pinhasov et al.,(US Patent Application Publication 2022/0060619 A1, hereinafter “Pinhasov”)
As to claim 40, Holmstrom and Vasilakakis teach the method of claim 1 but fail to teach comprising: prior to applying the one or more machine learning classifiers to the received video data, converting the received video data from a first color space to a second color space to accentuate clarity-affecting features of the received video data.
However, Pinhasov teaches prior to applying the one or more machine learning classifiers to the received video data, converting the received video data from a first color space to a second color space to accentuate clarity-affecting features of the received video data.(Pinhasov par [0060] teaches the second copy of the raw image data 215 may be converted from one color space (e.g., the RGB color space) into another color space (e.g., the YUV color space) before the second copy of the raw image data 215 is received by the classification engine 220)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teaching of Holmstrom, Vasilakakis and Pinhasov to achieve the claimed invention. One would have been motivated to make such combination to enhance texture clarity in the processed image (Pinhasov par [0041])
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HIEN DUONG whose telephone number is (571)270-7335. The examiner can normally be reached Monday-Friday 8:00AM-5:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached at 571-270-5871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HIEN L DUONG/Primary Examiner, Art Unit 2147
Prosecution Insights