Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to filed Amendments
Applicant’s Amendments/Remarks filed on 03/27/2026 have been received and made of record.
Claims 1, 6, and 9 have been amended.
Claims 2, and 10 have been cancelled.
New claims 21-22 have been added.
Claims 1, 3-9, and 11-22 are currently pending.
The outstanding USC 101 rejection to claims 1 and 9 have been withdrawn based at least the amendments/remarks filed on 03/27/2026. However, the outstanding USC 101 rejection to claim 18 has been maintained.
Please refer to the action below.
Examiner Notes
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. However, the claimed subject matter, not the specification, is the measure of the invention.
Response to Remarks/Arguments
Applicants’ arguments of 03/27/2026, corresponding to page 7 of 15 pertaining to the USC 101 rejection of claim 8 citing “Regarding claim 18, it's respectfully noted that no rationale for a rejection under 35 U.S.C. § 101 has been provided. Applicant believes that the inclusion of claims 18-20 in the rejection on page 3, item 5 was inadvertent. Furthermore, claim 18 recites, inter alia, "augmenting the video of the surgical procedure in response to a key point of the surgical instrument being out of view from the video", and thus is not directed to an abstract idea. Claims 3-7, 11, 14-15, and 19-20 depend respectively from claims 1, 9, and 18, which comply with 35 U.S.C. § 101, and thus, claims 3-7, 11, 14-15, and 19-20 also comply with 35 U.S.C. § 101. Withdrawal of the rejection is respectfully requested”.
A. The Examiner respectfully disagrees with the above assertions. The 101 rejection of claim 18 remained for the following reasons: 101 Analysis – Step 2A, Prong I
Regarding Prong I of the Step 2A analysis in the 2019 Revised Patent Subject Matter Eligibility Guidance (PEG), the claims are to be analyzed to determine whether they recite subject matter that falls within one of the following groups of abstract ideas: a) certain methods of organizing human activity, b) mental processes, and/or c) mathematical concepts.
Independent claim 18 includes limitations that recite an abstract idea (emphasized below) wherein nothing in the claim element precludes the steps from practically being performed in the mind and/or purely by software.
Claim 18 recites:
A computer program product comprising a memory device with computer-readable instructions stored thereon,
wherein executing the computer-readable instructions by one or more processing units causes the one or more processing units to perform a method comprising:
accessing, a video of a surgical procedure comprising use of a plurality of surgical instruments concurrently;
identifying, autonomously, one or more key points associated with the surgical instruments;
concurrently performing, using one or more machine learning models:
grouping a set of key points from the one or more key points, the set of key points associated with a surgical instrument;
identifying a type of the surgical instrument based on the set of key points; and estimating a pose of the surgical instrument based on the set of key points; and augmenting the video of the surgical procedure in response to a key point of the surgical instrument being out of view from the video.
The examiner submits that the foregoing bolded limitation(s) constitute a “certain methods of organizing and grouping human activity,” or a “mental process”. Specifically, the “accessing”, and “identifying” steps encompass the methods of organizing and grouping human activity and/or the mental steps found in the obtaining, accessing and identifying surgical tools tips ends or endpoints at a surgical site. Furthermore, the steps “concurrently performing” “grouping”, “identifying”, “estimating”, “augmenting”, further encompasses mathematical calculation and mathematical steps found in geometry of defining a bounding box, through the corners, edges and planes, determining endpoints and keypoint of the surgical, travel paths and the like. Accordingly, claim 1 recites an abstract idea.
Regarding the analysis of 101 Analysis – Step 2A, Prong II.
Regarding Prong II of the Step 2A analysis in the 2019 PEG, the claims are to be analyzed to determine whether the claim, as a whole, integrates the abstract into a practical application. As noted in the 2019 PEG, it must be determined whether any additional elements in the claim beyond the abstract idea integrate the exception into a practical application in a manner that imposes a meaningful limit on the judicial exception. The courts have indicated that additional elements merely using a computer to implement an abstract idea, adding insignificant extra solution activity, or generally linking use of a judicial exception to a particular technological environment or field of use do not integrate a judicial exception into a “practical application.”
In the present case, the additional limitations beyond the above-noted abstract idea are as follows (where the underlined portions are the “additional limitations” while the bolded portions continue to represent the “abstract idea”):
A computer program product comprising a memory device with computer-readable instructions stored thereon,
wherein executing the computer-readable instructions by one or more processing units causes the one or more processing units to perform a method comprising:
accessing, a video of a surgical procedure comprising use of a plurality of surgical instruments concurrently;
identifying, autonomously, one or more key points associated with the surgical instruments;
concurrently performing, using one or more machine learning models:
grouping a set of key points from the one or more key points, the set of key points associated with a surgical instrument;
identifying a type of the surgical instrument based on the set of key points; and estimating a pose of the surgical instrument based on the set of key points; and augmenting the video of the surgical procedure in response to a key point of the surgical instrument being out of view from the video.
For the following reason(s), the examiner submits that the above identified additional limitations do not integrate the above-noted abstract idea into a practical application.
Regarding the additional limitations of “computer program product comprising a memory device with computer-readable instructions stored thereon, wherein executing the computer-readable instructions by one or more processing units causes the one or more processing units to perform a method …” and “concurrently performing, using one or more machine learning models:
grouping a set of key points from the one or more key points, the set of key points associated with a surgical instrument; identifying a type of the surgical instrument based on the set of key points; and estimating a pose of the surgical instrument based on the set of key points; and augmenting the video of the surgical procedure in response to a key point of the surgical instrument being out of view from the video…” the examiner submits that these limitations are an attempt to generally link additional elements to a technological environment. In particular, the “computer program product comprising a memory device with computer-readable instructions stored thereon, wherein executing the computer-readable instructions by one or more processing units causes the one or more processing units to perform a method …” and “concurrently performing, using one or more machine learning models: grouping a set of key points from the one or more key points, the set of key points associated with a surgical instrument; identifying a type of the surgical instrument based on the set of key points; and estimating a pose of the surgical instrument based on the set of key points; and augmenting the video of the surgical procedure in response to a key point of the surgical instrument being out of view from the video” limitations are recited at a high level of generality and merely automates the concurrently performed steps of “grouping”, “identifying”, “estimating”, and “augmenting” steps, respectively, therefore acting as a generic computer to perform the abstract idea. The computer program product and the one or more processing units are claimed generically and do not use the judicial exception in a manner that imposes a meaningful limit on the judicial exception, such that the claim is more than a drafting effort designed to monopolize the exception. These additional limitation(s) is/are no more than mere instructions to apply the exception using a computer (the data provider and the central node). Furthermore, regarding the additional limitation of “computer program product comprising a memory device with computer-readable instructions stored thereon, wherein executing the computer-readable instructions by one or more processing units causes the one or more processing units to perform a method …” and “concurrently performing, using one or more machine learning models: grouping a set of key points from the one or more key points, the set of key points associated with a surgical instrument; identifying a type of the surgical instrument based on the set of key points; and estimating a pose of the surgical instrument based on the set of key points; and augmenting the video of the surgical procedure in response to a key point of the surgical instrument being out of view from the video”, the examiner submits that these limitations are insignificant extra-solution activities that merely use a computer to perform the process. Specifically, they are insignificant extra-solution activities, which are mere data gathering, monitoring, and identifying presence and/or absence of overlaid surgical tools at a surgical site based on a determined tools poses.
Thus, taken alone, the additional elements do not integrate the abstract idea into a practical application. Further, looking at the additional limitation(s) as an order combination or as a whole, the limitation(s) add nothing that is not already present when looking at the elements taken individually. Accordingly, the additional limitation(s) do/does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
101 Analysis – Step 2B
Regarding Step 2B of the PEG, representative independent claim 1 does not include additional elements (considered both individually and as an ordered combination) that are sufficient to amount to significantly more than the judicial exception for the same reasons to those discussed above with respect to determining that the claim does not integrate the abstract idea into a practical application. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements of the concurrently performed steps of “grouping”, “identifying”, “estimating”, and “augmenting” steps limitations amount to nothing more than mere instructions to apply the exception using a generic computer component (the one or more processing units). Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept.
Further, a conclusion that an additional element is insignificant extra-solution activity in Step 2A should be re-evaluated in Step 2B to determine if they are more than what is well-understood, routine, conventional activity in the field. The additional limitations of “computer program product comprising a memory device with computer-readable instructions stored thereon, wherein executing the computer-readable instructions by one or more processing units causes the one or more processing units to perform a method …” and “concurrently performing, using one or more machine learning models: grouping a set of key points from the one or more key points, the set of key points associated with a surgical instrument; identifying a type of the surgical instrument based on the set of key points; and estimating a pose of the surgical instrument based on the set of key points; and augmenting the video of the surgical procedure in response to a key point of the surgical instrument being out of view from the video”, are well-understood, routine of obtaining, accessing, identifying and classifying objects types and/or endpoints that can be performed mentally by a person or a visual language model software. MPEP 2106.05(d)(II), and the cases cited therein, including Intellectual Ventures I, LLC v. Symantec Corp., 838 F.3d 1307, 1321 (Fed. Cir. 2016), TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610 (Fed. Cir. 2016), and OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363 (Fed. Cir. 2015), indicate that mere collection or receipt of data over a network is a well‐understood, routine, and conventional function when it is claimed in a merely generic manner. The additional limitation of “concurrently performing, using one or more machine learning models …” is a well-understood, routine, and conventional activity because the Federal Circuit in Trading Techs. Int’l v. IBG LLC, 921 F.3d 1084, 1093 (Fed. Cir. 2019), and Intellectual Ventures I LLC v. Erie Indemnity Co., 850 F.3d 1315, 1331 (Fed. Cir. 2017), for example, indicated that the mere outputting of data is a well understood, routine, and conventional function. Hence, claim 18 is not subject matter eligible.
B. Applicant’s arguments of pages 7-11 regarding the prior arts of record of Chen in view of Jarvis, have been considered, however they are moot in light of the new ground of rejection and the newly added claim limitations of independent claims 1, and 9.
C. Applicant’s arguments of pages 10-15 regarding the prior arts of record of Chen in view of Jarvis, as pertaining to claim 18, further citing “It appears that the Office Action intended to cite US20220080060A1 to Jarvis et al. Jarvis et al. is directed to a disinfection monitoring system that includes one or more image capture devices configured to capture a set of images or videos of an object to be disinfected, such as a user's hands, fingers, and body during an disinfection procedure. (Abstract). At paragraph [0033], Jarvis et al. states, "The detection-based tracking of a user's fingers, arms, or other body parts may be performed using keypoint or key area- based tracking and/or object tracking". Jarvis et al. seeks "to identify a user's arms, wrists, hands, and/or fingers or any other suitable type, number, and combination of keypoints or key areas, and to determine a pose, such as one of the steps of the recommended handwashing procedure, in a particular frame" Id. Paragraph [0101] of Jarvis et al. describes a training procedure, where a "presenter or viewer may apply markings onto areas of the surface of the object before placing the object in the field of view of the camera so as to train the system to identify the markings as keypoints or key areas". Annotating a physical object or manually annotating frames to support training is not reasonably equivalent to grouping of one or more key points according to a plurality of surgical instruments, and determining, based on the one or more key points that are grouped, poses of surgical instruments and types of surgical instruments using a second machine learning model, wherein the poses and the types are determined concurrently. Further, observing a pose of a person performing handwashing is not reasonably equivalent to poses of surgical instruments. Moreover, the determination of poses for handwashing would not reasonably be combined with the determinations of Chen performed during a surgical procedure, such as cutting, clipping, clamping, etc” have been considered, however they are not persuasive.
The Examiner would like to note on the records that the Examiner, after the Attorney had reached out to the Examiner on 03/16/2026 of the grammatical error made in the citing of the PGPUB prior art of Jarvis, had verbally disclosed to Applicant on 03/19/2026, the correct PGPUB No. US20220080060, US 20200080060 was erroneously listed instead of the correct US 20220080060.
Secondly, the Examiner respectfully disagrees, as the combination of Chen and the namely, US20220080060A1, do teach the claimed invention. As, Chen clearly teaches in at least Figs. 1-4 obtaining of a plurality of realtime surgical site images including a plurality of image frames consisting of overlaid surgical instrument tools on the images of the surgical site, the overlaid images include detected tools tips and endpoints/keypoints of the utilized tools in a video of a surgical procedure where based on the determined tools tips and endpoints/keypoint depicted on the surgical site images the system is configured to determine at least an orientation, position and/or poses of the said instruments to provide guidance and surgical assist to the surgeon. Although, Chen is silent regarding specifically said group of said one or more key points according to the plurality of surgical instruments, and said determine, based on the one or more key points that are grouped, specifically poses of said surgical instruments and types of the surgical instruments respectively using a second machine learning model, wherein said specific poses and the types are determined concurrently.
Jarvis clearly teaches depicted overlaid images of objects posture and detected key-points which may be grouped to identify, track, estimate and augment poses and/or orientation of the said objects on the said obtained images. It would have been obvious to one of ordinary skill in the art at the time the invention was made to combine the teachings of Chen in view of Jarvis to include wherein said set of key points from the one or more key points, the set of key points associated with a surgical instrument, as discussed above, where the Jarvis’s combination of poses determination and detected endpoints of tracked objects types complements the autonomously identified one or more key points associated with the plurality of surgical instruments illustrated in the video of the surgical procedure of Chen in the sense that the combination determination of the poses and detected endpoints of tracked objects when combined with the autonomously tracked surgical instruments key points of Chen help facilitate tracking of one or more objects or currently used surgical tools based on their tracked endpoints or keypoints, wherein the realtime tracking further suppresses and/or prevent operation of erroneous tools at the surgical site and erroneously operation of said tools outside the field view of the doctor which ultimately prevent damage and injuries to the patient according to further known methods to yield predictable results since known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if the variations are predictable to one of ordinary skill in the art as said combination is thus the adaptation of an old idea or invention using newer technology that is either commonly available and understood in the art thereby a variation on already known art (See MPEP 2143, KSR Exemplary Rationale F).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-9, and 12-17 is/are rejected under 35 U.S.C. 103 as obvious over Chen et al. (US 20200397509, cited in IDS), in view of Liu et al. (US 12551298, A1).
Regarding claim 1, Chen teaches a system (the system of at least para. 0040-0051 and Figs. 2-6 illustrates a case of identifying captured surgical tools types, and current tools position, orientation collectively indicating in the art a detected tool pose based at least on identified plurality of key points), comprising:
a memory device (para. 0002); and
one or more processors (para. 0002) coupled with the memory device, the one or more processors configured to:
identify, autonomously, one or more key points associated with a plurality of surgical instruments in a video of a surgical procedure (autonomously identified said one or more key points of further para. 0044-0045, and 0048-0051 associated with a plurality of surgical instruments in a video of a surgical procedure);
cluster the one or more key points according to each surgical instrument of the plurality of surgical instruments using one or more machine learning models (clustering and connecting of at least para. 0045 the one or more key points according to the plurality of surgical instruments of further para. 0044-0045, and 0048-0051 using a first machine learning model of para. 0048-0049); and
determine, based on the one or more key points that are clustered, position, location, and orientation of the surgical instruments and types of the surgical instruments respectively using a second machine learning model,
wherein the position, location, and orientation and the types are determined concurrently (the system further may determine further in at least para. 0047-0051 types of tools detected in the surgical based on their specific keypoints information that are clustered and connected, their detected position, location, and orientation of the surgical instruments essentially in the art further illustrates the poses of said instruments respectively using obviously one or more second machine learning model of further para. 0047-0049, wherein the position, location, and orientation collectively indicating their poses and the types according to said keypoints are determined understoodly concurrently),
wherein a posture or orientation of a surgical instrument from the plurality of surgical instruments is used to provide user feedback (a posture depicting a pose of at least Fig. 1 and 3-4 further overlays on the surgical image area further in the art used to provide user feedback).
However, Chen is silent regarding specifically said group the one or more key points according to each surgical instrument of the plurality of surgical instruments using one or more machine learning models; and determine, based on the one or more key points that are grouped, poses of the surgical instruments and types of the surgical instruments respectively using the one or more machine learning models, wherein the poses and the types are determined concurrently, wherein a pose of a surgical instrument from the plurality of surgical instruments is used to provide user feedback.
Liu teaches at least in Fig. 3 a “system 300 may use machine learning to perform any of the operations described herein. For example, system 300 may use machine learning to determine whether a target object is located in a surgical space”, further in Figs. 8 and 17-19 detecting and determining a plurality of surgical instrument tools types in a surgical space, further identifying and impliedly grouping each surgical instrument of the plurality of surgical instruments based on implied one or more key points, further in Figs. 8 and 17-19 detecting and determining the position, poses, and types of the surgical instrument tools and further depicting an image overlay of the surgical instrument tools and depiction overlay of their the travel routes/paths of the surgical instrument tools in the surgical space further implying in the art grouping of said one or more key points according to the plurality of surgical instruments, and specifically illustrate further Figs. 8 and 17-19 poses of said surgical instruments and types of the surgical instruments poses further determined obviously concurrently, wherein a pose of a surgical instrument of further Figs. 8 and 17-19 from the plurality of surgical instruments is used to provide user feedback. It would have been obvious to one of ordinary skill in the art at the time the invention was made to combine the teachings of Chen in view of Liu to include wherein group said one or more key points according to each surgical instrument of the plurality of surgical instruments, and determine, based on the one or more key points that are grouped, said poses and types of the surgical instruments respectively and wherein said pose of a surgical instrument from the plurality of surgical instruments is used to provide user feedback, as discussed above, as Chen in view of Liu are in the same field of endeavor of capturing in realtime a performed surgical procedure, tracking surgical instruments location, position and poses to determine at least an orientation and types of said surgical tools, Liu’s combination of determining poses and types of the surgical instruments respectively using the one or more machine learning models, and overlay pose depiction of the plurality of surgical instruments further complements the identified plurality of surgical instrument key points depiction of Chen in the sense when combined with the combination architecture of Liu causes the overlay depiction of Chen as combined with Liu’s architecture further helps facilitate optimized poses detection of the surgical tools and their types at the surgical space or site which further helps indicate a case said tools may go outside the field of view of the camera, said realtime tracking further suppresses and/or prevent operation of erroneous tools at the surgical site and erroneously operation of said tools outside the field view of the doctor which ultimately prevent damage and injuries to the patient according to further known methods to yield predictable results since known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if the variations are predictable to one of ordinary skill in the art as said combination is thus the adaptation of an old idea or invention using newer technology that is either commonly available and understood in the art thereby a variation on already known art (See MPEP 2143, KSR Exemplary Rationale F).
Regarding claim 3 (according to claim 1), Chen further teaches wherein the one or more processors are further configured to generate a bounding box of a surgical instrument based on the one or more key points grouped according to the surgical instrument (at least Figs. 3-4 and para. 0011-0013 further teaches said generate bounding box of a surgical instrument based on the one or more key points grouped according to the surgical instrument).
Regarding claim 4 (according to claim 1), Chen further teaches wherein the video of the surgical procedure is captured by an endoscopic camera from inside a patient's body (at least para. 0053 further illustrates video of the surgical procedure captured by an endoscopic camera from inside a patient's body).
Regarding claim 5 (according to claim 1), Chen further teaches wherein the video of the surgical procedure is captured by a camera from outside a patient's body (at least para. 0053 further illustrates video of the surgical procedure captured in a case captured procedure by obviously a camera from outside a patient's body).
Regarding claim 6 (according to claim 1), Chen further teaches wherein the one or more machine learning model outputs an annotation for each of the one or more key points identified (a machine learning model of at least para. 0047-0049 is understoodly adapted for further marking in at least para. 0022-0024 visual indicators or outputs annotation for each of the one or more key points identified).
Regarding claim 7 (according to claim 1), Chen further teaches wherein the poses and types of the surgical instruments are identified with temporal continuity (surgical tools types determination of further para. 0051-0052 and orientation indicative of the poses of said surgical instruments are identified continuously with implied temporal continuity).
Regarding claim 8 (according to claim 1), Chen further teaches wherein the one or more processors are configured to test a surgical robotic arm (said robotic arm of at least para. 0053 may be obviously configured to be tested via at least known issuing commands to said the surgical robotic arm),
the test comprising:
issuing a command to the surgical robotic arm that results in a surgical instrument associated with the surgical robotic arm to be in a predetermined pose (manipulating the robotic arm of at least para. 0001 by at least one or more known
issuing command to the surgical robotic arm that results in a surgical instrument of at least Figs. 2-6 associated with the surgical robotic arm to be in a predetermined orientation or pose);
determining a first pose of the surgical instrument based on the one or more key points that are grouped (determining of Figs. 2-6 further comprises determining at least first orientation or first pose of the surgical instrument based on the one or more key points that are grouped);
and comparing the first pose and the predetermined pose (ascertaining of the surgical tools position, and orientation further indicative in the art of at least a first pose of the tools compared to an implied predetermined pose or predetermined orientation to arrive at or detect an instrument tool that needs in para. 0051 and 0044 to be re-centered based on at least implied comparison of the current surgical tools position, and orientation further indicative in the art of at least a first pose of the tools compared to an implied predetermined pose or predetermined orientation).
Regarding claim 9, Chen teaches a computer-implemented method (the system and methods of at least para. 0040-0051 and Figs. 2-6 illustrates a case of identifying captured surgical tools types, and position and orientation of the tools, said position and orientation are indicative in the art of a pose of the detected tool tip based at least on identified plurality of key points), comprising:
identifying, autonomously, one or more key points associated with a plurality of surgical instruments in a video of a surgical procedure (autonomously identified said one or more key points of further para. 0044-0045, and 0048-0051 associated with a plurality of surgical instruments in a video of a surgical procedure);
clustering a set of key points from the one or more key points associated with a surgical instrument using one or more machine learning model (clustering and connecting of at least para. 0045 the one or more key points according to the plurality of surgical instruments of further para. 0044-0045, and 0048-0051 using a first machine learning model of para. 0048-0049);
determining, based on the set of key points that are grouped, position/location, and orientation of the surgical instrument (ascertaining of the surgical tools position, and orientation of at least para. 0026, and 0044-0051 at the surgical site, said position and orientation are indicative in the art of at least a first pose of the tools based on the set of key points);
wherein a posture or orientation of a surgical instrument from the plurality of surgical instruments is used to provide user feedback (a posture depicting a pose of at least Fig. 1 and 3-4 further overlays on the surgical image area further in the art used to provide user feedback).
However, Chen is silent regarding specifically said grouping set of one or more key points according to each surgical instrument of the plurality of surgical instruments using one or more machine learning models; and determining, based on the one or more key points that are grouped, a pose of the surgical instrument; and depicting a graphical overlay on the video to indicate the identified pose of the surgical instrument.
Liu teaches at least in Fig. 3 a “system 300 may use machine learning to perform any of the operations described herein. For example, system 300 may use machine learning to determine whether a target object is located in a surgical space”, further teaches at least in Figs. 8 and 17-19 detecting and determining a plurality of surgical instrument tools types in a surgical space, further identifying and impliedly grouping each surgical instrument of the plurality of surgical instruments based on implied one or more key points, further in Figs. 8 and 17-19 determining the position, poses, and types of the surgical instrument tools and further depicting an image overlay of the surgical instrument tools further depicting a graphical overlay on the video to indicate the identified pose of the surgical instrument. It would have been obvious to one of ordinary skill in the art at the time the invention was made to combine the teachings of Chen in view of Liu to include wherein group said one or more key points, said determining and said depicting, as discussed above, as Chen in view of Liu are in the same field of endeavor of capturing in realtime a performed surgical procedure, tracking surgical instruments location, position and poses to determine at least an orientation and types of said surgical tools, Liu’s combination of determining poses and types of the surgical instruments respectively using the one or more machine learning models, and overlay pose depiction of the plurality of surgical instruments further complements the identified plurality of surgical instrument key points depiction of Chen in the sense when combined with the combination architecture of Liu causes the overlay depiction of Chen as combined with Liu’s architecture further helps facilitate optimized poses detection of the surgical tools and their types at the surgical space or site which further helps indicate a case said tools may go outside the field of view of the camera, said realtime tracking further suppresses and/or prevent operation of erroneous tools at the surgical site and erroneously operation of said tools outside the field view of the doctor which ultimately prevent damage and injuries to the patient according to further known methods to yield predictable results since known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if the variations are predictable to one of ordinary skill in the art as said combination is thus the adaptation of an old idea or invention using newer technology that is either commonly available and understood in the art thereby a variation on already known art (See MPEP 2143, KSR Exemplary Rationale F).
Regarding claim 12 (according to claim 9), Chen is silent regarding wherein further comprising in response to the pose of the surgical instrument matching a threshold pose, generating a user notification.
Liu further teaches in at least Figs. 8 and 16-19 determined poses of a surgical tool at a surgical site and further in response to the pose of the surgical instrument matching a threshold pose, generating a prompt and an haptic/audible user notification and guidance to a user. It would have been obvious to one of ordinary skill in the art at the time the invention was made to combine the teachings of Chen in view of Liu to include wherein said comprising in response to the pose of the surgical instrument matching a threshold pose, generating a user notification, as discussed above, as Chen in view of Liu are in the same field of endeavor of capturing in realtime a performed surgical procedure, tracking surgical instruments location, position and poses to determine at least an orientation and types of said surgical tools, Liu’s combination of determining poses and types of the surgical instruments, and provided audible feedback and guidance to a user further complements the identified plurality of surgical instrument key points depiction of Chen in the sense when combined with the combination architecture of Liu causes based in a case on the determined poses of the surgical instrument tools at the surgical space not meeting a predetermined threshold, causes the outputting of an audible notification and/or user feedback to the user where the realtime tracking and monitoring poses of the said surgical instrument further suppresses and/or prevent operation of erroneous tools at the surgical site and erroneously operation of said tools which ultimately prevent damage and injuries to the patient according to further known methods to yield predictable results since known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if the variations are predictable to one of ordinary skill in the art as said combination is thus the adaptation of an old idea or invention using newer technology that is either commonly available and understood in the art thereby a variation on already known art (See MPEP 2143, KSR Exemplary Rationale F).
Regarding claim 13 (according to claim 12), Chen is silent regarding wherein the user notification is a first user notification, and in response to the pose of the surgical instrument not matching the threshold pose, generating a second user notification, different from the first user notification.
Liu further teaches in at least Figs. 8 and 16-19 and the disclosure determined poses of a surgical tool at a surgical site and further in response to the pose of the surgical instrument matching a threshold pose, generating a prompt and an haptic/audible user notification and guidance to a user, obviously in a case of not matching the threshold pose, generating an audible/haptic warning notification, which may be obviously different from the first user notification. It would have been obvious to one of ordinary skill in the art at the time the invention was made to combine the teachings of Chen in view of Liu to include wherein said user notification is a first user notification, and in response to the pose of the surgical instrument not matching the threshold pose, generating a second user notification, different from the first user notification, as discussed above, as Chen in view of Liu are in the same field of endeavor of capturing in realtime a performed surgical procedure, tracking surgical instruments location, position and poses to determine at least an orientation and types of said surgical tools, Liu’s combination of determining poses and types of the surgical instruments, and provided audible feedback and user notification further complements the identified plurality of surgical instrument key points depiction of Chen in the sense when combined with the combination architecture of Liu causes based in a case on the determined poses of the surgical instrument tools at the surgical space not meeting a predetermined threshold, causes the outputting of an audible notification and/or user feedback to the user as said user notification as understood in the art may obviously comprises a first user notification to confirm a pose within a predetermined or obviously one of not matching the threshold pose, to further suppress and/or prevent operation of erroneous tools at the surgical site and erroneously operation of said tools which ultimately prevent damage and injuries to the patient according to further known methods to yield predictable results since known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if the variations are predictable to one of ordinary skill in the art as said combination is thus the adaptation of an old idea or invention using newer technology that is either commonly available and understood in the art thereby a variation on already known art (See MPEP 2143, KSR Exemplary Rationale F).
Regarding claim 14 (according to claim 12), Chen is silent regarding wherein the threshold pose is indicative of a desired pose of the surgical instrument based on a surgical action to be performed.
Liu further teaches in at least Figs. 8 and 16-19 and the disclosure determined poses of a surgical tool at a surgical site and further in response to the pose of the surgical instrument matching a threshold pose, generating a prompt and an haptic/audible user notification and guidance to a user, as said notification may obviously comprise a pose notification indicative of a desired pose of the surgical instrument based on a surgical action to be performed. It would have been obvious to one of ordinary skill in the art at the time the invention was made to combine the teachings of Chen in view of Liu to include wherein said threshold pose is indicative of a desired pose of the surgical instrument based on a surgical action to be performed, as discussed above, as Chen in view of Liu are in the same field of endeavor of capturing in realtime a performed surgical procedure, tracking surgical instruments location, position and poses to determine at least an orientation and types of said surgical tools, Liu’s combination of determining poses and types of the surgical instruments, and provided audible feedback and user notification further complements the identified plurality of surgical instrument key points depiction of Chen in the sense when combined with the combination architecture of Liu causes based in a case on the determined poses of the surgical instrument tools at the surgical space not meeting a predetermined threshold, causes the outputting of an audible notification and/or user feedback to the user as said user notification as understood in the art may obviously comprises a first user notification to confirm a pose within a predetermined or obviously one of not matching the threshold pose, to further suppress and/or prevent operation of erroneous tools at the surgical site and erroneously operation of said tools which ultimately prevent damage and injuries to the patient according to further known methods to yield predictable results since known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if the variations are predictable to one of ordinary skill in the art as said combination is thus the adaptation of an old idea or invention using newer technology that is either commonly available and understood in the art thereby a variation on already known art (See MPEP 2143, KSR Exemplary Rationale F).
Regarding claim 15 (according to claim 12), Chen is silent regarding wherein
the threshold pose is indicative of an undesired pose of the surgical instrument.
Liu further teaches in at least Figs. 8 and 16-19 and the disclosure determined poses of a surgical tool at a surgical site and further in response to the pose of the surgical instrument matching a threshold pose, generating a prompt and an haptic/audible user notification and guidance to a user, as said pose notification may obviously be indicative of an undesired pose of the surgical instrument. It would have been obvious to one of ordinary skill in the art at the time the invention was made to combine the teachings of Chen in view of Liu to include wherein said threshold pose is indicative of an undesired pose of the surgical instrument, as discussed above, as Chen in view of Liu are in the same field of endeavor of capturing in realtime a performed surgical procedure, tracking surgical instruments location, position and poses to determine at least an orientation and types of said surgical tools, Liu’s combination of determining poses and types of the surgical instruments, and provided audible feedback and user notification further complements the identified plurality of surgical instrument key points depiction of Chen in the sense when combined with the combination architecture of Liu causes based in a case on the determined poses of the surgical instrument tools at the surgical space not meeting a predetermined threshold, causes the outputting of an audible notification and/or user feedback to the user as said user notification as understood in the art may obviously comprises a first user notification to confirm a pose within a predetermined or obviously one of not matching the threshold pose, to further suppress and/or prevent operation of erroneous tools at the surgical site and erroneously operation of said tools which ultimately prevent damage and injuries to the patient according to further known methods to yield predictable results since known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if the variations are predictable to one of ordinary skill in the art as said combination is thus the adaptation of an old idea or invention using newer technology that is either commonly available and understood in the art thereby a variation on already known art (See MPEP 2143, KSR Exemplary Rationale F).
Regarding claim 16 (according to claim 12), Chen further teaches wherein the user notification includes an audible notification (transmitted surgical warnings of further para. 0053 entails in a case user notification includes obviously audible notification).
Regarding claim 17 (according to claim 12), Chen further teaches wherein the user notification is provided on a separate display, distinct from the video (the generated surgical warnings of further para. 0053 output by the warning systems of further para. 0029 may be provided obviously on a separate display of further para. 0057, distinct in a case from the video).
Claim 11 is/are rejected under 35 U.S.C. 103 as obvious over Chen in view of Liu, and further in view of Wang et al. (WO 2018/195221, A1).
Regarding claim 11 (according to claim 10), Chen in view of Liu are silent regarding wherein the graphical overlay includes a depiction of the one or more key points to identify specifically an exit path to move the surgical instrument.
Wang teaches identify in at least para. 0156 endpoints of surgical instrument tool, endpoints of a path and the like and generating graphical path data overlay includes a depiction of the one or more key points to identify an exit path to move the surgical instrument. It would have been obvious to one of ordinary skill in the art at the time the invention was made to combine the teachings of Chen in view of Liu, and further in view of Wang to include wherein said graphical overlay, as discussed above, as Chen in view of Liu, and further in view of Wang are in the same field of endeavor of capturing in realtime a video of a performed surgical procedure, tracking surgical instruments location, position to determine at least an orientation and types of said surgical tools, Wang’s identified surgical tool exit path complements the identified one or more key points of the surgical instrument tools of Chen in view of Liu in the sense that the identified surgical tools exit path architecture of Wang when combined with the detected surgical instrument tools key points of Chen in view of Liu, causes the system to facilitate a redirection or optimal travel path of the surgical instrument tool in a case the tool becomes out of the field of view, to ultimately facilitate safe travel movements of the surgical instrument tool at the surgical site and whereby prevent damage and injuries to the patient according to further known methods to yield predictable results since known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if the variations are predictable to one of ordinary skill in the art as said combination is thus the adaptation of an old idea or invention using newer technology that is either commonly available and understood in the art thereby a variation on already known art (See MPEP 2143, KSR Exemplary Rationale F).
Claims 18, and 20 is/are rejected under 35 U.S.C. 103 as obvious over Chen et al. (US 20200397509, cited in IDS), in view of Jarvis et al. (US 2022/0080060, A1).
Regarding claim 18, Chen teaches in at least para. 0005 a computer program product comprising a memory device with computer-readable instructions stored thereon, wherein executing the computer-readable instructions by one or more processing units causes the one or more processing units to perform a method comprising:
accessing, a video of a surgical procedure comprising use of a plurality of surgical instruments concurrently (at least para. 0005 and 0053 further teaches utilizing and accessing, a camera attached to at least a robotic arm, a video of a surgical procedure comprising use of a plurality of surgical instruments concurrently);
identifying, autonomously, one or more key points associated with the surgical instruments (identifying further in at least para. 0005, 0044-0045, and 0048-0051 autonomously, one or more key points associated with the surgical instruments);
concurrently performing, using one or more machine learning models of further para. 0047-0049:
clustering a set of key points from the one or more key points, the set of key points associated with a surgical instrument (concurrently performing as implied clustering and connecting of at least para. 0045 the one or more key points associated with the plurality of surgical instruments of further para. 0044-0045, and 0048-0051); and
identifying a type of the surgical instrument based on the set of key points (identifying further in at least para. 0047-0051 the detected tools type based on the set of key points); and
estimating a pose of the surgical instrument based on the set of key points (estimating further in at least para. 0047-0051 the detected tools poses based on the set of key points); and
augmenting the video of the surgical procedure in response to a key point of the surgical instrument being out of view from the video (the system further in at least para. 0051 and 0053 may normalized the captured video by at least by placing the target surgical tool in the center of the video, or perform re-centering of the detected tools, said placing and/or re-centering is understood in the art to comprise a video augmenting process in response to obviously a key point of the surgical instrument of further para. 0045-0047 being out of view from the video).
However, Chen is silent regarding specifically grouping said set of key points from the one or more key points, the set of key points associated with a surgical instrument.
Jarvis teaches at least in para. 0033-0034 detected poses of identified objects which objects identification as further illustrated in at least para. 0101-0102 may represent as noted by Jarvis medical instrument tools detected in a case at the surgical site corresponding to obviously grouped or overlaid key points of further para. 0094 which are further used to determine specific poses and types of tools determined which may further obviously determined concurrently. It would have been obvious to one of ordinary skill in the art at the time the invention was made to combine the teachings of Chen in view of Jarvis to include wherein said set of key points from the one or more key points, the set of key points associated with a surgical instrument, as discussed above, Jarvis’s determination of the poses and types of the surgical instruments at the surgical site based on the detected keypoints information complements the autonomously identified one or more key points of the plurality of surgical instruments of Chen in the sense that the determination of the poses and types of the surgical instruments architecture of Jarvis when combined with the architecture of the autonomously identified tools key points of Chen to advantageously facilitates an optimized surgical tools tip endpoints detection in the surgical space, further allowing the system to more accurately determine position, orientation and trajectory of the tools in realtime to reduce in a case the possibility of the tools ending outside the field of view of the camera, thereby suppresses and/or prevent operation of erroneous tools at the surgical site and ultimately prevent damage and injuries to the patient according to further known methods to yield predictable results since known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if the variations are predictable to one of ordinary skill in the art as said combination is thus the adaptation of an old idea or invention using newer technology that is either commonly available and understood in the art thereby a variation on already known art (See MPEP 2143, KSR Exemplary Rationale F).
Regarding claim 20 (according to claim 18), Chen further teaches wherein the method further comprises augmenting the video of the surgical procedure in response to the key point of the surgical instrument being within a predetermined proximity of an anatomical structure (aligning of further para. 0029 further comprises in the art augmenting of the video of the surgical procedure in response to the localization or a key point of the surgical instrument being within a predetermined proximity of an anatomical structure, eye retina and the like).
Claim 19 is/are rejected under 35 U.S.C. 103 as obvious over Chen in view of Jarvis, and further in view of Tang et al. (CN 111144321, A1).
Regarding claim 19 (according to claim 18), Chen in view of Jarvis are silent regarding wherein the one or more machine learning models comprise multi-tasking convolutional neural network layers that aggregate spatio-temporal features in one or more frames of the video.
Tang teaches at least in the disclosure one or more machine learning models comprise multi-tasking convolutional neural network layers that use and aggregate spatio-temporal features in one or more frames of the video such as “in order to ensure that model in the video frame sequence of temporal and spatial information learning, can be deep-space multi-task 3 D convolution neural network structure to sample the video frame sequence, sample type, sample attention level result and the sample object position information for learning, training to obtain the attention level detection model”. It would have been obvious to one of ordinary skill in the art at the time the invention was made to combine the teachings of Chen in view of Jarvis, and further in view of Tang to include wherein said one or more machine learning models comprise multi-tasking convolutional neural network layers that aggregate spatio-temporal features in one or more frames of the video, as discussed above, Tang’s multi-tasking convolutional neural network layers architecture complements the autonomously identified one or more key points associated with the plurality of surgical instruments illustrated in the video of the surgical procedure of Chen in view of Jarvis in the sense that the multi-tasking convolutional neural network layers architecture of Tang when combined with the architecture of the autonomously identified one or more key points associated with the plurality of surgical instruments of Chen in view of Jarvis facilitates temporal and spatial detection of the objects of interest in the captured surgical procedure video frames to accurately detect and predict the surgical tools poses and exit path spatially and temporally which ultimately suppresses and/or prevent operation of erroneous tools at the surgical site and erroneously operation of said tools outside the field view of the doctor which thereby prevent damage and injuries to the patient according to further known methods to yield predictable results since known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if the variations are predictable to one of ordinary skill in the art as said combination is thus the adaptation of an old idea or invention using newer technology that is either commonly available and understood in the art thereby a variation on already known art (See MPEP 2143, KSR Exemplary Rationale F).
Claims Standings
Claims 21-22 are not rejected over the prior arts of record, but would be allowable if properly rewritten in independent form including all of the limitations of the base claim and any intervening claims, or properly incorporated in the respective indepddent claims and if all outstanding rejection are overcome. The prior arts do not appear to specifically teach: claim 21. (New) The computer program product of claim 18, wherein augmenting the video of the surgical procedure in response to the key point of the surgical instrument being out of view from the video comprises outputting a user notification warning that one or more tips of the surgical instrument are out of view.
22. (New) The computer program product of claim 18, wherein augmenting the video of the surgical procedure in response to the key point of the surgical instrument being out of view from the video comprises a guidance overlay identifying a path to move a portion of the surgical instrument away from an anatomical structure.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MARCELLUS AUGUSTIN whose telephone number is (571)270-3384. The examiner can normally be reached 9 AM- 5 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, BENNY TIEU can be reached on 571-272-7490. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MARCELLUS J AUGUSTIN/Primary Examiner, Art Unit 2682 06/03/2026
Prosecution Insights