DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Joint Inventors
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Preliminary Amendment
Receipt is acknowledged of a preliminary amendment prior to the first office action on the merits, pertaining to the claim set subject to restriction as detailed further below. The examiner has reviewed the amendment and determined that no new matter was submitted. As such, the preliminary amendment is being considered, but claims are subject to the restriction requirement 18 March 2026, outlined further below. The most recent revision of the claim set is dated 30 April 2026, wherein claims 15, 17-19, 21-32 are pending, and the aforementioned claims are the subject of examination on the merits.
Information Disclosure Statement
The information disclosure statements (IDSs) filed on 18 March 2025 and 04 December 2025 comply with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Priority
The examiner acknowledges that the instant application is a Continuation of application 17/566,116, filed 30 December 2021.
Response to Amendment/Election/Restriction and Status of Claims
The examiner issued a requirement for election/restriction dated 18 March 2026. Applicant’s election of invention II in the reply filed on 30 April 2026 is acknowledged. Because applicant did not distinctly and specifically point out the supposed errors in the restriction requirement, the election has been treated as an election without traverse (MPEP § 818.01(a)).
Applicant has added new claims 21-32, drawn to the elected invention. No new matter was entered. The most recent revision of the claim set is dated 30 April 2026.
The following claims are cancelled: 1-14, 16, 20
The following claims are pending: 15, 17-19, 21-32
As the election was made without traverse, the previously made restriction requirement is hereby made FINAL. All pending claims are rejected for the reasons provided below.
Claim Interpretation
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art.
The examiner notes that "unsafe event", such as that recited in claim 1 for example, is defined within the corresponding claims and is not purely a term/phrase of relative degree in light of the explanation given. Thus, a 35 U.S.C. 112(b) rejection for this language is withheld.
Regarding "taking a proper action" (such as that recited in claim 15) or “action”, the examiner notes that a basis for understanding is provided in specification paragraphs [0011, 0017] and claims 23, 29 (as examples), in addition to a person having ordinary skill in the art being able to identify one of a limited number of corrective actions for an energized tool located outside of the field of view of an endoscope tracking its movement. Thus, while the terminology utilized is broad, it is not considered to be indefinite.
Claim Objections
Claims 22-26 and 28-32 are objected to because of the following informalities:
Claims 22-26 and 28-32 lack a comma following the preamble prior to recitation of "wherein".
Claim 32 states "The article of manufacture of claim 27 the instructions…" and is missing the word "wherein" prior to "the instructions".
Appropriate correction is required.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claim 15, 17-18, 21-24, 27-30 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-2, 5-7, 10-12 of U.S. Patent No. 12,207,861 B2.
Instant Application
19/001,323
Reference US Patent
US 12,207,861 B2
15. An ultrasonic or bipolar tissue cutting/cauterizing surgical system, the surgical system comprising:
a signal and power generator;
an energy tool;
one or more processors; and
a memory coupled to the one or more processors, the memory storing instructions that, when executed by the one or more processors, cause the surgical system to:
11. An ultrasonic or bipolar tissue cutting/cauterizing surgical system, the surgical system comprising:
a signal and power generator;
an energy tool;
one or more processors; and
a memory coupled to the one or more processors, the memory storing instructions that, when executed by the one or more processors, cause the surgical system to:
receive a real-time control signal from the generator, wherein the real-time control signal indicates an operating state of the energy tool during a surgery by including a plurality of activation pulses, each activation pulse represents a time duration when the energy tool is activated;
receive a real-time tool control signal from the generator, wherein the real-time tool control signal indicates an operating state of the energy tool during a surgery, the real-time tool control signal includes a plurality of activation pulses, each activation pulse represents a time duration when the energy tool is activated;
simultaneously receive real-time endoscope video images of the surgery;
simultaneously receive real-time endoscope video images of the surgery;
simultaneously apply a machine-learning model to the real-time endoscope video images to generate real-time decisions on a location of the energy tool in the real-time endoscope video images wherein for each video frame in the real-time endoscope video images a tool presence/absence decision is generated indicating whether the energy tool is present or absent in the video frame;
simultaneously apply a machine-learning model to the real-time endoscope video images to generate real-time decisions on a location of the energy tool in the real-time endoscope video images wherein for each video frame in the real-time endoscope video images a tool presence/absence decision is generated indicating whether the energy tool is present or absent in the video frame and a confidence level associated with the tool presence/absence decision;
check the real-time control signal against the real-time decisions to identify an unsafe event; and
identify an unsafe event by determining when the tool presence/absence decision is a tool absence decision that coincides with an activation pulse of the plurality of activation pulses; and
take a proper action when an unsafe event is identified.
take a proper action when the unsafe event is identified.
Instant Application
19/001,323
Reference US Patent
US 12,207,861 B2
17. The surgical system of claim 15, wherein the memory further stores instructions that, when executed by the one or more processors, cause the surgical system to generate the real-time decisions by: for each video frame in the real-time endoscope video images, processing the video frame to generate a confidence level associated with the tool presence/absence decision.
11. simultaneously apply a machine-learning model to the real-time endoscope video images to generate real-time decisions on a location of the energy tool in the real-time endoscope video images wherein for each video frame in the real-time endoscope video images a tool presence/absence decision is generated indicating whether the energy tool is present or absent in the video frame and a confidence level associated with the tool presence/absence decision;
Instant Application
19/001,323
Reference US Patent
US 12,207,861 B2
18. The surgical system of claim 15, wherein the energy tool is an ultrasonic energy tool for cutting and sealing tissues at the same time using two jaws, and wherein a tool presence decision for the video frame is generated only when both of the two jaws are detected in the video frame.
12. The surgical system of claim 11, wherein the energy tool is an ultrasonic energy tool for cutting and sealing tissues at the same time using two jaws, and wherein the machine-learning model is trained to generate a tool presence decision for a video frame only when both of the two jaws are detected in the video frame.
Instant Application
19/001,323
Reference US Patent
US 12,207,861 B2
21. A method for enhancing an ultrasonic or bipolar tissue cutting/cauterizing process, the method comprising the following operations performed by a computer system:
1. A computer-implemented method for ensuring patient safety during a laparoscopic or robotic surgery involving an energy tool, the method comprising:
receiving a real-time control signal from a signal and power generator, wherein the real- time control signal indicates an operating state of an energy tool being used during a surgery by including a plurality of activation pulses, each activation pulse represents a time duration when the energy tool is activated;
receiving a real-time tool control signal indicating an operating state of an energy tool during the surgery, the real-time tool control signal includes a plurality of activation pulses and wherein each activation pulse in the plurality of activation pulses corresponds to a time duration when the energy tool is activated;
simultaneously with receiving the real-time control signal, receiving real-time endoscope video images of the surgery;
simultaneously receiving real-time endoscope video images of the surgery;
applying a machine-learning model to the real-time endoscope video images to generate real-time tool presence/absence decisions about the energy tool in the real-time endoscope video images so that for each video frame in the real-time endoscope video images a respective tool presence/absence decision is generated by the machine-learning model indicating whether the energy tool is present or absent in the video frame;
simultaneously applying a machine-learning model to the real-time endoscope video images to generate real-time decisions on a location of the energy tool in the real-time endoscope video images wherein for each video frame in the real-time endoscope video images a tool presence/absence decision is generated indicating whether the energy tool is present or absent in the video frame and a confidence level associated with the tool presence/absence decision;
checking the real-time control signal against the real-time tool presence/absence decisions being made by the machine-learning model, to identify an unsafe event; and
identifying an unsafe event in real-time by determining whenever the tool presence/absence decision is a tool absence decision that coincides with an activation pulse of the plurality of activation pulses; and
take an action when the unsafe event is identified.
taking a proper action when the unsafe event is identified.
Instant Application
19/001,323
Reference US Patent
US 12,207,861 B2
22. The method of claim 21 wherein generating the real-time tool presence/absence decisions comprises, for each video frame in the real-time endoscope video images, processing the video frame to generate a confidence level associated with the respective tool presence/absence decision.
1. simultaneously applying a machine-learning model to the real-time endoscope video images to generate real-time decisions on a location of the energy tool in the real-time endoscope video images wherein for each video frame in the real-time endoscope video images a tool presence/absence decision is generated indicating whether the energy tool is present or absent in the video frame and a confidence level associated with the tool presence/absence decision;
Instant Application
19/001,323
Reference US Patent
US 12,207,861 B2
23. The method of claim 22 wherein taking the action when the unsafe event is identified comprises:
if the confidence level is above a high confidence level threshold, then disabling the energy tool; and
if the confidence level is below the high confidence level threshold, then taking an action selected from the following without disabling the energy tool:
displaying a visual alert on an endoscope monitor;
generating an audio alert;
generating a mechanical vibration through the energy tool; or
delaying the firing of the energy tool until a user takes a further action on the energy tool.
5. The computer-implemented method of claim 1, wherein taking the proper action when the unsafe event is identified includes:
if the confidence level is above a high confidence level threshold, immediately disabling the energy tool; and
if the confidence level is below the high confidence level threshold, taking one or more actions selected from the following options without disabling the energy tool:
displaying a visual alert on an endoscope monitor;
generating an audio alert;
generating a mechanical vibration through the energy tool; and
delaying the firing of the energy tool until a user takes a further action on the energy tool.
Instant Application
19/001,323
Reference US Patent
US 12,207,861 B2
24. The method of claim 21 wherein the energy tool is an ultrasonic energy tool for cutting and sealing tissues at the same time using two jaws, and wherein a real-time tool presence decision for the video frame is generated only when both of the two jaws are detected in the video frame.
2. The computer-implemented method of claim 1, wherein the energy tool is an ultrasonic energy tool for cutting and sealing tissues at the same time using two jaws, and wherein the machine-learning model is trained to generate a tool presence decision for a video frame only when both of the two jaws are detected in the video frame.
Instant Application
19/001,323
Reference US Patent
US 12,207,861 B2
27. An article of manufacture comprising non-transitory computer-readable media having stored therein instructions or data structures that when accessed by a computer enhance an ultrasonic or bipolar tissue cutting/cauterizing process by:
6. A system for ensuring patient safety during a laparoscopic or robotic surgery involving an energy tool, the system comprising:
one or more processors; and
a memory coupled to the one or more processors, the memory storing instructions that, when executed by the one or more processors, cause the system to:
receiving a real-time control signal from a signal and power generator, wherein the real- time control signal indicates an operating state of an energy tool being used during a surgery by including a plurality of activation pulses, each activation pulse represents a time duration when the energy tool is activated;
receive a real-time tool control signal indicating an operating state of an energy tool during the surgery, the real-time tool control signal includes a plurality of activation pulses, each activation pulse represents a time duration when the energy tool is activated;
simultaneously with receiving the real-time control signal, receiving real-time endoscope video images of the surgery;
simultaneously receive real-time endoscope video images of the surgery;
applying a machine-learning model to the real-time endoscope video images to generate real-time tool presence/absence decisions about the energy tool in the real-time endoscope video images so that for each video frame in the real-time endoscope video images a respective tool presence/absence decision is generated by the machine-learning model indicating whether the energy tool is present or absent in the video frame;
simultaneously apply a machine-learning model to the real-time endoscope video images to generate real-time decisions on a location of the energy tool in the real-time endoscope video images wherein for each video frame in the real-time endoscope video images a tool presence/absence decision is generated indicating whether the energy tool is present or absent in the video frame and a confidence level associated with the tool presence/absence decision;
checking the real-time control signal against the real-time tool presence/absence decisions being made by the machine-learning model, to identify an unsafe event; and
identifying an unsafe event by determining when the tool presence/absence decision is a tool absence decision that coincides with an activation pulse of the plurality of activation pulses; and
take an action when the unsafe event is identified.
take a proper action when the unsafe event is identified.
Instant Application
19/001,323
Reference US Patent
US 12,207,861 B2
28. The article of manufacture of claim 27 wherein the instructions or data structures configure the computer to generate the real-time tool presence/absence decisions, for each video frame in the real-time endoscope video images, by processing the video frame to generate a confidence level associated with the respective tool presence/absence decision.
6. simultaneously apply a machine-learning model to the real-time endoscope video images to generate real-time decisions on a location of the energy tool in the real-time endoscope video images wherein for each video frame in the real-time endoscope video images a tool presence/absence decision is generated indicating whether the energy tool is present or absent in the video frame and a confidence level associated with the tool presence/absence decision;
Instant Application
19/001,323
Reference US Patent
US 12,207,861 B2
29. The article of manufacture of claim 28 wherein the instructions or data structure configure the computer to take the action when the unsafe event is identified as follows:
if the confidence level is above a high confidence level threshold, then disabling the energy tool; and
if the confidence level is below the high confidence level threshold, then taking an action selected from the following without disabling the energy tool:
displaying a visual alert on an endoscope monitor;
generating an audio alert;
generating a mechanical vibration through the energy tool; or
delaying the firing of the energy tool until a user takes a further action on the energy tool.
10. The system of claim 6, wherein the memory further stores instructions that, when executed by the one or more processors, cause the system to take the proper action when the unsafe event is identified by:
immediately disabling the energy tool if the confidence level is above a high confidence level threshold; and
taking one or more actions selected from the following options without disabling the energy tool if the confidence level is below the high confidence level threshold:
displaying a visual alert on an endoscope monitor;
generating an audio alert;
generating a mechanical vibration through the energy tool; and
delaying the firing of the energy tool until a user takes a further action on the energy tool.
Instant Application
19/001,323
Reference US Patent
US 12,207,861 B2
30. The article of manufacture of claim 27 wherein the energy tool is an ultrasonic energy tool for cutting and sealing tissues at the same time using two jaws, and wherein the instructions or data structures configure the computer to generate a real-time tool presence decision for the video frame only when both of the two jaws are detected in the video frame.
7. The system of claim 6, wherein the energy tool is an ultrasonic energy tool for cutting and sealing tissues at the same time using two jaws, and wherein the machine-learning model is trained to generate a tool presence decision for a video frame only when both of the two jaws are detected in the video frame.
Although the claims at issue are not identical, they are not patentably distinct from each other because a person having ordinary skill in the art at the time of effective filing would consider the instant application claims to merely be in broader form compared to the reference US Patent.
Thus, claims 15, 17-18, 21-24, 27-30 are rejected for Non-Statutory Double Patenting.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 15, 18, 21, 24, 26-27, 30, 32 are rejected under 35 U.S.C. 103 as being unpatentable over Chow et al. (US 2021/0015554 A1; published 21 Jan 2021, hereinafter Chow) in view of Hancock et al. (US 2022/0202490 A1; filed 28 Apr 2020, hereinafter Hancock).
Regarding independent claims 15, 21, and 27: Chow discloses An ultrasonic or bipolar tissue cutting/cauterizing surgical system, the surgical system comprising: (per claim 15) (Paragraph [0001, 0019, 0029, 0032-0033, 0063, 0072-0079] and Claim [30], Chow discloses a system for control of energized surgical tools such as an ultrasonic cutting/sealing tool during surgery) / A method for enhancing an ultrasonic or bipolar tissue cutting/cauterizing process, the method comprising the following operations performed by a computer system: (per claim 21) (The examiner notes that “for enhancing…” is an intended use and not afforded significant patentable weight. Paragraph [0001, 0019, 0029, 0032-0033, 0063, 0074-0075] and Claim [21], Chow discloses a computer-implemented method for control of energized surgical tools during surgery (laparoscopic and robotic are given as possible types of surgery) using tools such as an ultrasonic cutting/sealing tool) / An article of manufacture comprising non-transitory computer-readable media having stored therein instructions or data structures that when accessed by a computer enhance an ultrasonic or bipolar tissue cutting/cauterizing process by: (per claim 27) (Paragraph [0001, 0019, 0029, 0032-0033, 0063, 0074-0075, 0097-0099] and Claim [21], Chow discloses a computer-implemented method for control of energized surgical tools during surgery (laparoscopic and robotic are given as possible types of surgery) using tools such as an ultrasonic cutting/sealing tool, the method being stored on a machine readable storage medium)
a signal and power generator; (per claim 15) (Paragraph [0050-0055] and Figure [1], Chow teaches an output generator that controls energization/de-energization of the tool)
an energy tool; (per claim 15) (Paragraph [0072-0075], Chow discloses an energy tool, in an exemplary embodiment this is an ultrasonic cutting device)
one or more processors; and (per claim 15) (Paragraph [0017-0018, 0095, 0098] and Claim [30], Chow discloses one or more processors)
a memory coupled to the one or more processors, the memory storing instructions that, when executed by the one or more processors, cause the surgical system to: (per claim 15) (Paragraph [0017-0018, 0095, 0097-0099] and Claim [30], Chow discloses a non-transitory machine-readable storage medium (memory) communicatively coupled to the processor(s) to store instructions for the method)
receive a real-time control signal from the generator, [wherein the real-time control signal indicates an operating state of the energy tool during a surgery by including a plurality of activation pulses, each activation pulse represents a time duration when the energy tool is activated]; (per claim 15) / receiving a real-time control signal from a signal and power generator, [wherein the real-time control signal indicates an operating state of an energy tool being used during a surgery by including a plurality of activation pulses, each activation pulse represents a time duration when the energy tool is activated]; (per claim 21) / receiving a real-time control signal from a signal and power generator, [wherein the real- time control signal indicates an operating state of an energy tool being used during a surgery by including a plurality of activation pulses, each activation pulse represents a time duration when the energy tool is activated]; (per claim 27) (Paragraph [0008, 0029, 0036, 0039, 0046-0048, 0060-0063, 0072-0079] and Figure [2-4], Chow discloses that a position and state of an object (a tool can be an object) are collected/received by the system. Both the current position and the state may be considered “operating states” of the tool. Further, by identifying that the tool is within the field of view of the imaging device, the tool may be “enabled”, and as such, all sequential frames of the tool remaining in this position contain an operational state characteristic of being “enabled” (an operating state). Thus, a plurality of interpretations of Chow’s disclosure read upon the claim limitation. Regarding the receiving of a plurality of activation pulses, Chow states that the processing can include receiving and processing one or more data streams, the data streams including data streams from a surgical tool (such as haptic feedback, the stream of haptic feedback reasonably alluding to a series of activation pulses for the tool))
simultaneously receive real-time endoscope video images of the surgery; (per claim 15) / simultaneously with receiving the real-time control signal, receiving real-time endoscope video images of the surgery; (per claim 21) / simultaneously with receiving the real-time control signal, receiving real-time endoscope video images of the surgery; (per claim 27) (Paragraph [0036, 0049, 0063, 0069, 0074, 0077, 0080], Chow discloses that any imaging device to capture a live feed may be used, and discusses a laparoscope as an exemplary embodiment, however a video stream from an endoscope is also disclosed in paragraph [0049]. Thus, while Chow primarily discusses laparoscopes throughout the disclosure, the examiner is considering all instances of imaging devices (including that of a laparoscope) to be analogous to that of the endoscope of the instant application as simply a generic means to provide video/image data to a machine learning model. Further, Chow states “The trained machine-learning model can then be used in real-time to process one or more data streams”, listing haptic feedback of the tool as an example; thus, receiving multiple data streams including the real time video and the real time tool control data)
simultaneously apply a machine-learning model to the real-time endoscope video images to generate real-time decisions on a location of the energy tool in the real-time endoscope video images wherein for each video frame in the real-time endoscope video images a tool presence/absence decision is generated indicating whether the energy tool is present or absent in the video frame; (per claim 1) / applying a machine-learning model to the real-time endoscope video images to generate real-time tool presence/absence decisions about the energy tool in the real-time endoscope video images so that for each video frame in the real-time endoscope video images a respective tool presence/absence decision is generated by the machine-learning model indicating whether the energy tool is present or absent in the video frame; (per claim 21) / applying a machine-learning model to the real-time endoscope video images to generate real-time tool presence/absence decisions about the energy tool in the real-time endoscope video images so that for each video frame in the real-time endoscope video images a respective tool presence/absence decision is generated by the machine-learning model indicating whether the energy tool is present or absent in the video frame; (per claim 27) (Paragraph [0045-0046, 0058-0060, 0074-0080], The examiner notes that the word “simultaneous”, used in this way, is a relative term. While not necessarily indefinite, the examiner notes that “simultaneously” can mean anything from “in the same instant” (which would not necessarily be possible as the disclosed invention relies on a data transfer of images to a machine learning model, which may be comparatively fast, but still involves a level of latency, and therefore less likely) to “in the same process flow”. The examiner will utilize the latter interpretation as it more realistically captures the spirit and scope of the claimed invention. Chow discloses that an image feed may provide a machine learning model with image data in real time by providing each frame of the image data to the machine learning model. The machine learning model may then determine the type and location of the tool(s) identified. Chow discloses iterative frame by frame analysis)
check the real-time control signal against the real-time decisions to identify an unsafe event; and (per claim 15) / checking the real-time control signal against the real-time tool presence/absence decisions being made by the machine-learning model, to identify an unsafe event; and (per claim 21) / checking the real-time control signal against the real-time tool presence/absence decisions being made by the machine-learning model, to identify an unsafe event; and (per claim 27) (Paragraph [0002, 0008, 0029, 0062-0063, 0068-0072, 0075] and Figure [2-4], Chow discloses that the state (including an allowable energization state and a location, as discussed earlier) of the tool is determined in conjunction with the field of view (FOV) of the imaging device in order to improve patient safety by ensuring that certain functionality is only enabled when the surgical tool is detected within the field of view and not being handled erroneously, which could cause injury to the patient)
take a proper action when an unsafe event is identified. (per claim 15) / take an action when the unsafe event is identified. (per claim 21) / take an action when the unsafe event is identified. (per claim 27) (Paragraph [0002, 0008, 0029, 0032, 0055, 0062-0063, 0068-0072, 0075] and Figure [2-4], Chow discloses that erroneous handling of equipment/tools may lead to injury of a patient (an unsafe condition). In order to reduce erroneous handling, Chow discloses that when the tool leaves the FOV of the imaging device, certain functionality may be disabled (considered “taking a proper action”))
Chow does not explicitly disclose wherein the real-time control signal indicates an operating state of the energy tool during a surgery by including a plurality of activation pulses, each activation pulse represents a time duration when the energy tool is activated (per claim 15) / wherein the real-time control signal indicates an operating state of an energy tool being used during a surgery by including a plurality of activation pulses, each activation pulse represents a time duration when the energy tool is activated (per claim 21) / wherein the real- time control signal indicates an operating state of an energy tool being used during a surgery by including a plurality of activation pulses, each activation pulse represents a time duration when the energy tool is activated (per claim 27).
However, Hancock, in a similar field of endeavor of surgical system control, teaches wherein the real-time control signal indicates an operating state of the energy tool during a surgery by including a plurality of activation pulses, each activation pulse represents a time duration when the energy tool is activated (per claim 15) / wherein the real-time control signal indicates an operating state of an energy tool being used during a surgery by including a plurality of activation pulses, each activation pulse represents a time duration when the energy tool is activated (per claim 21) / wherein the real- time control signal indicates an operating state of an energy tool being used during a surgery by including a plurality of activation pulses, each activation pulse represents a time duration when the energy tool is activated (per claim 27) (Paragraph [0013, 0022, 0063], Hancock teaches an electrosurgical system that utilizes a plurality of pulses made up of “on” and “off” periods wherein the pulse duration corresponds to a time duration of a pulse of energy)
Chow and Hancock are in a similar field of endeavor of surgical system control. It would have been obvious to one having ordinary skill in the art at the time of effective filing, with a reasonable expectation of success, to have modified the surgical tool control system/method/apparatus taught by Chow by further describing the types of and controls regarding the tools that may be utilized alongside the disclosure of Chow with the control method disclosed by Hancock, as one of ordinary skill in the art would have been motivated to understand the timing in which enabling or disabling signals as disclosed by Chow (Paragraph [0075], as an example) may exactly impact the tool in use. Pulse Width Modulation (PWM) is a common control method in which the activation timing and output magnitude of a device may be adjusted through fine tuning of an overall cycle duration and a duty cycle in order to improve efficiency of control (Hancock, Paragraph [0021]). Hancock describes the use of a Pulse Width Modulation control scheme on a surgical tool, thereby directly teaching that it is known in the art to utilize Pulse Width Modulation (as described by this claim) to control a device. As stated above, Chow discloses receiving data streams from the tool (Paragraph [0036], for instance, describes haptic data from the tool, reasonably considered tool activation), but merely falls short in explicitly describing the data stream as a series of pulses with time duration of activation, though the examiner submits that this is implicit, as the haptic data stream at least implies a series of data points indicating when the haptic data is present or absent/active or inactive. Merely quantifying this data in a more particular type of well-known data (tracking of pulses, per Hancock) that does not have a patentable effect on the claimed invention constitutes merely combining known elements according to known methods to produce predictable results. One would have been motivated to make this combination in order to provide fine control over the instrument as disclosed by Chow with a reasonable expectation of success.
Regarding claims 18, 24, and 30: Parent claims 15, 21, and 27 are unpatentable over Chow in view of Hancock. Chow further discloses wherein the energy tool is an ultrasonic energy tool for cutting and sealing tissues at the same time using two jaws, and (per claim 18) / wherein the energy tool is an ultrasonic energy tool for cutting and sealing tissues at the same time using two jaws, and (per claim 24) / wherein the energy tool is an ultrasonic energy tool for cutting and sealing tissues at the same time using two jaws, and (per claim 30) (Paragraph [0074-0075], Chow teaches that the tool may be ultrasonic shears (an ultrasonic energy tool) powered by an energy device, capable of performing a cutting function and a sealing function using jaws)
wherein a tool presence decision for the video frame is generated only when both of the two jaws are detected in the video frame. (per claim 18) / wherein a real-time tool presence decision for the video frame is generated only when both of the two jaws are detected in the video frame. (per claim 24) / wherein the instructions or data structures configure the computer to generate a real-time tool presence decision for the video frame only when both of the two jaws are detected in the video frame. (per claim 30) (Paragraph [0070, 0074-0075] and Figure [4], Chow teaches that that the model is trained to recognize features of a surgical tool and that the “machine learning model may recognize the pixels showing the jaws and the shears shaft as being a part of ultrasonic shears”, thus indicating that the presence of both jaws and the shaft provide the information to the model that the ultrasonic shears are present)
Regarding claims 26 and 32: Parent claims 21 and 27, respectively, are unpatentable over Chow in view of Hancock. Chow further discloses wherein checking the real-time control signal against the real-time tool presence/absence decisions being made by the machine-learning model comprises: determining that a real-time absence decision does not coincide with any of the plurality of activation pulses, in response to which no unsafe event is identified. (per claim 26) / the instructions or data structures configure the computer to check the real-time control signal against the real-time tool presence/absence decisions being made by the machine-learning model by: determining that a real-time absence decision does not coincide with any of the plurality of activation pulses, in response to which no unsafe event is identified. (per claim 32) (Paragraph [0055, 0072-0073], Chow discloses that a plurality of surgical tools may be controlled, even if only some (but not all) of the plurality of surgical tools are detected within the camera’s field of view (thus, one of the tools is absent and coincidence with any pulse, permitting continued use of the tools). Further, merely adjusting the magnitude may be considered permission of continued use when absent from the FOV of the camera when an “unsafe event” indicates an event necessary to disable the tool)
Claims 17, 22-23, and 28-29 are rejected under 35 U.S.C. 103 as being unpatentable over Chow in view of Hancock in further view of Tamarozzi et al. (US 2022/0313388 A1; filed 25 Aug 2020, hereinafter Tamarozzi).
Regarding claims 17, 22, and 28: Parent claims 15, 21, and 27, respectively, are unpatentable over Chow in view of Hancock. Chow further discloses wherein the memory further stores instructions that, when executed by the one or more processors, cause the surgical system to generate the real-time decisions by: for each video frame in the real-time endoscope video images, processing the video frame to generate a [confidence level] associated with the tool presence/absence decision. (per claim 17) / wherein generating the real-time tool presence/absence decisions comprises, for each video frame in the real-time endoscope video images, processing the video frame to generate a [confidence level] associated with the respective tool presence/absence decision. (per claim 22) / wherein the instructions or data structures configure the computer to generate the real-time tool presence/absence decisions, for each video frame in the real-time endoscope video images, by processing the video frame to generate a [confidence level] associated with the respective tool presence/absence decision. (per claim 28) (Paragraph [0063, 0075-0080], Chow discloses that each image frame may be provided to the machine learning model to recognize surgical tools in the field of view of the imaging device and recognize when the tools are not in the field of view of the imaging device)
Regarding the limitation of a confidence level, Chow discloses that surgical tool features may be patterns of pixels that indicate a likelihood that an image contains a surgical tool (Chow, Paragraph [0057], thereby teaching that a likelihood/probability is associated with a detected object in a selected image) and that detection of objects (in this case, an “avoidance zone”) may have a confidence of a certain amount (Chow, Paragraph [0040], again implicitly noting that a confidence is associated with the determination).
Tamarozzi, in a similar field of endeavor of surgical instrument detection through image analysis, teaches a confidence level associated with the presence/absence decision. (per claims 17, 22, and 28) (Tamarozzi, Paragraph [0037, 0092, 0131, 0144], Tamarozzi teaches that the recognition of surgical instruments based on the images provided from an imaging device contain a level of confidence with the determinations made)
Chow and Tamarozzi are in a similar field of endeavor of the instant application of surgical instrument detection through image analysis. It would have been obvious to a person having ordinary skill in the art at the time of effective filing, with a reasonable expectation of success, to have modified the surgical procedure monitoring system and method of Chow to more explicitly describe the confidence of the machine learning model (as noted above in Chow Paragraph [0040, 0057]), as taught by Tamarozzi, as one would be motivated to not disable a surgical tool in use at a critical step due to an erroneous incorrect judgement made by the machine learning model. This would have a net negative effect on the safety of the patient, and would be counter to the disclosure of Chow (Paragraph [0019, 0029, 0032]). Therefore, this would constitute a combination of known elements according to known methods in order to produce predictable results with a reasonable expectation of success.
Regarding claims 23 and 29: Parent claims 22 and 28, respectively, are unpatentable over Chow in view of Hancock in further view of Tamarozzi. Chow further discloses wherein taking the action when the unsafe event is identified comprises: if the confidence level is above a high confidence level threshold, then disabling the energy tool; and (per claim 23) / wherein the instructions or data structure configure the computer to take the action when the unsafe event is identified as follows: if the confidence level is above a high confidence level threshold, then disabling the energy tool; and (per claim 29) (Paragraph [0072-0079], Chow teaches that a decision by the system is made in which the tool is detected or not detected, which under broadest reasonable interpretation includes a level of confidence. Therefore, Chow teaches that when the system determines/is confident that the tool is not present within the FOV of the imaging device, the tool is disabled)
if the confidence level is below the high confidence level threshold, then taking an action selected from the following without disabling the energy tool: (per claim 23) / if the confidence level is below the high confidence level threshold, then taking an action selected from the following without disabling the energy tool: (per claim 29) (Paragraph [0050-0052, 0072-0079] and Figure [1], As noted above, by making a determination regarding the presence or absence of the tool, a confidence determination is also made. Thus, Chow teaches that a plurality of actions may be taken based on the result of the tool absence/presence determination, including those that do not disable the tool. The examiner notes that while only one of the following list is needed (hence “or”), the examiner will provide reference to disclosure of multiple items)
displaying a visual alert on an endoscope monitor; (per claims 23 and 29) (Paragraph [0051-0052, 0072], Chow teaches that the alert may be displayed in a visual format)
generating an audio alert; (per claims 23 and 29) (Paragraph [0051], Chow teaches that the alert may be in an audio format)
generating a mechanical vibration through the energy tool; or (per claims 23 and 29) (Paragraph [0040, 0051], Chow teaches that the alert may be in a haptic (mechanical vibration/movement) format)
delaying the firing of the energy tool until a user takes a further action on the energy tool. (per claims 23 and 29) (Paragraph [0072-0079], Chow teaches that the tool is disabled until it returns to the FOV of the camera, at which point it is enabled (thus delaying the firing of the energy tool until the user places the tool back into the FOV, or “takes a further action on the energy tool”))
ALLOWABLE SUBJECT MATTER
Claims 19, 25, and 31 currently depend upon rejected base claims, however they are not being rejected by prior art. Thus, claims 19, 25, and 31 are merely objected to for depending upon a rejected base claim.
The following is an examiner’s reasoning for where there are no prior art rejections:
The prior art of record does not disclose each and every limitation in such a way that would have been obvious to one of ordinary skill in the art at the time of effective filing to arrive at the claimed invention. A hypothetical prior art rejection would require impermissible hindsight reasoning. Specifically, the following underlined limitations are not disclosed, taught, suggested, or rendered obvious by the prior art of record:
CLAIM 19: The surgical system of claim 15, wherein the memory further stores instructions that, when executed by the one or more processors, cause the surgical system to when determining that the a tool absence decision is at the end of the activation pulse, determining that the energy tool is safe to use.
CLAIM 25: The method of claim 21 wherein checking the real-time control signal against the real-time tool presence/absence decisions being made by the machine-learning model comprises: determining that a real-time tool absence decision is not at the beginning of any of the plurality of activation pulses, in response to which no unsafe event is identified.
CLAIM 31: The article of manufacture of claim 27 wherein the instructions or data structures configure the computer to check the real-time control signal against the real-time tool presence/absence decisions being made by the machine-learning model by: determining that a real-time tool absence decision is not at the beginning of any of the plurality of activation pulses, in response to which no unsafe event is identified.
The claimed invention by way of the underlined limitations provides the benefit of a safety interlock that prevents tool energization when outside of a field of view of an imaging device, but does not over-restrict use, permitting continued energization when the tool is likely still in a safe state (as described, for example, in paragraph [0073] of the specification). While the examiner has located art which provides for a safety interlock of preventing energization of a surgical tool when located outside of a field of view of an endoscope, the alternative case of permitting continued use of a tool was not found. Closest prior art of note (other than those identified on the attached IDSs, with exception of closest art of note below) from a search are as follows:
US 2021/0015554 A1 - Chow et al.: Discusses use of computer vision to detect a tool in a live video stream using machine learning, detect the tool in the video stream, and control the tool based upon the detection criteria. In the event that the tool is moved outside of the detection area, it is deactivated.
US 2024/0008931 A1 - Takahashi et al.: Discusses a surgical system that prevents energization of a medical instrument when it is located outside of a field of view of an endoscope.
US 2022/0218426 A1 and US 2024/0090962 A1 - Itkowitz et al.: Discusses a surgical system that determines when a surgical tool has left the field of view of an operator and provides a notification based upon the determination.
US 2022/0022982 A1 - Hares et al.: Discusses a surgical robot system with a module that determines if an energized instrument is outside of the FOV of an endoscope of the system.
US 2022/0233255 A1 - Kawabata et al.: Discusses a surgical system that identifies if a surgical tool is located outside of the field of view of the image/endoscope, then indicate a direction/position of the tool to a user.
The prior art of record does not disclose, teach, suggest, or render obvious the claimed invention of the aforementioned claims. Therefore, the dependent claims of the instant application noted above recite subject matter that is novel and non-obvious over prior art.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BENJAMIN J BROSH whose telephone number is (571)270-0105. The examiner can normally be reached M-F 0730-1700.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, THOMAS WORDEN can be reached at (571)272-4876. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/B.J.B./Examiner, Art Unit 3658
Prosecution Insights