Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Filed IDS of 11/15/2023 has been entered and considered.
Claims 1-20 are currently pending.
Please refer to the action below.
Examiner Notes
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. However, the claimed subject matter, not the specification, is the measure of the invention. Furthermore, the computer program product (Emphasis Added) cited in independent claim 18 is interpreted as non-transitory or tangible program product based further on at least para. 0139 of the specification citing “The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to carry out aspects of the present invention. The computer-readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device…… A computer-readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire”.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, 3-7, 9-11, 14-15, and 18-20 is/are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim(s) 1, recite(s) mental processes and software processes directed to a system, a memory device, and one or more processors.
Independent claim 1 includes limitations that recite an abstract idea. Claim 1 recites: A system comprising: a memory device; and one or more processors coupled with the memory device, the one or more processors configured to: identify, autonomously, one or more key points associated with a plurality of surgical instruments in a video of a surgical procedure; group the one or more key points according to the plurality of surgical instruments using a first machine learning model; and determine, based on the one or more key points that are grouped, poses of the surgical instruments and types of the surgical instruments respectively using a second machine learning model, wherein the poses and the types are determined concurrently.
Claim 1 (being the representative claim) further recites similar limitations to those discussed with regards to independent claim 9, and therefore discussion is omitted for brevity.
This judicial exception is not integrated into a practical application because the claims merely recite mental steps that can be performed by a person and/or software steps that can be performed by component or units of a software. That is, other than reciting “determine, based on the one or more key points that are grouped, poses of the surgical instruments and types of the surgical instruments respectively using a second machine learning model, wherein the poses and the types are determined concurrently” nothing in the claim element precludes the steps from practically being performed in the mind and/or purely by software. The additional elements of “determine, based on the one or more key points that are grouped, poses of the surgical instruments and types of the surgical instruments respectively using a second machine learning model, wherein the poses and the types are determined concurrently” does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception. Hence, claim 1 is not subject matter eligible.
The dependent claims 3-7, and 10-11, and 14-15, do not recite any further limitations that cause the claim(s) to be subject matter eligible. Rather, the limitations of dependent claims are directed toward additional aspects of the judicial exception and/or well-understood, routine and conventional additional elements that do not integrate the judicial exception into a practical application. Based on broadest reasonable interpretation of the claims, all of the steps recited in the independent claims 1, and 9, and corresponding dependent claims 3-7, and 10-11, and 14-15 further correspond to concepts performed by at least software components which may be further performed in the human mind. Additionally, a person can mentally performed in the human mind and/or software the assessing of surgical tools endpoints and/or keypoints to determine poses indicative of the position and trajectory of the target tools in a targeted surgical site illustrated in the received video of the surgical site, said endpoints and/or keypoints being identified based on known methods and techniques such as at least bounding boxes and the like. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claims are directed to an abstract idea.
However, dependent claims 2, 8, 12-13, and 16-17 do recite further limitations that cause the claim(s) to be subject matter eligible. These claims are not directed to an abstract idea.
Concepts performed in the human mind have been identified in the 2019 PEG as an exemplar in the “Mental Process” grouping of abstract ideas. For the reasons above, the claims do not amount to significantly more than an abstract idea. Even when considered in combination, these additional elements represent mere instructions to apply an exception and insignificant extra-solution activity, which do not provide an inventive concept and therefore, the claims are not patent-eligible.
Furthermore, these additional generic hardware elements perform no more than their basic computer function. Generic computer‐implementation of a method is not a meaningful limitation that alone can amount to significantly more than an abstract idea. Moreover, when viewed as a whole with such additional element considered as an ordered combination, claims modified by adding generic hardware elements are nothing more than a purely conventional computerized implementation of an idea in the general field of computer processing and do not provide significantly more than an abstract idea.
Consequently, the identified additional generic hardware elements taken into consideration individually and in combination fail to amount to significantly more than the abstract idea above.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-10, 18, and 20 is/are rejected under 35 U.S.C. 103 as obvious over Chen et al. (US 20200397509, cited in IDS), in view of Jarvis et al. (US 2020/0080060, A1).
Regarding claim 1, Chen teaches a system (the system of at least para. 0040-0051 and Figs. 2-6 illustrates a case of identifying captured surgical tools types, and current tools position, orientation collectively indicating in the art a detected tool pose based at least on identified plurality of key points), comprising:
a memory device (para. 0002); and
one or more processors (para. 0002) coupled with the memory device, the one or more processors configured to:
identify, autonomously, one or more key points associated with a plurality of surgical instruments in a video of a surgical procedure (autonomously identified said one or more key points of further para. 0044-0045, and 0048-0051 associated with a plurality of surgical instruments in a video of a surgical procedure);
cluster the one or more key points according to the plurality of surgical instruments using a first machine learning model (clustering and connecting of at least para. 0045 the one or more key points according to the plurality of surgical instruments of further para. 0044-0045, and 0048-0051 using a first machine learning model of para. 0048-0049); and
determine, based on the one or more key points that are clustered, position, location, and orientation of the surgical instruments and types of the surgical instruments respectively using a second machine learning model, wherein the position, location, and orientation and the types are determined concurrently (the system further may determine further in at least para. 0047-0051 types of tools detected in the surgical based on their specific keypoints information that are clustered and connected, their detected position, location, and orientation of the surgical instruments essentially in the art further illustrates the poses of said instruments respectively using obviously one or more second machine learning model of further para. 0047-0049, wherein the position, location, and orientation collectively indicating their poses and the types according to said keypoints are determined understoodly concurrently).
However, Chen is silent regarding specifically said group of said one or more key points according to the plurality of surgical instruments, and said determine, based on the one or more key points that are grouped, specifically poses of said surgical instruments and types of the surgical instruments respectively using a second machine learning model, wherein said specific poses and the types are determined concurrently.
Jarvis teaches at least in para. 0033-0034 detected poses of identified objects which are further illustrated in at least para. 0101 to indicate or represent identified medical tools/objects at the surgical site corresponding to grouped or overlaid one or more grouped key points of further para. 0094 which are further used to determine specific poses and types of tools determined understanbly in the art concurrently to determine in a case if the identified surgical tools are in a field of view. It would have been obvious to one of ordinary skill in the art at the time the invention was made to combine the teachings of Chen in view of Jarvis to include wherein said group of said one or more key points according to the plurality of surgical instruments, and said determine, based on the one or more key points that are grouped, specifically poses of said surgical instruments and types of the surgical instruments respectively using a second machine learning model, wherein said specific poses and the types are determined concurrently, as discussed above, as Chen in view of Jarvis are in the same field of endeavor of capturing in realtime a performed surgical procedure, tracking surgical instruments location, position and poses to determine at least an orientation and types of said surgical tools, Jarvis’s determination of the poses and types of the surgical instruments at the surgical site according to detected keypoints information complements the autonomously identified one or more key points associated with the plurality of surgical instruments illustrated in the video of the surgical procedure of Chen in the sense that the determination of the poses and types of the surgical instruments architecture of Jarvis when combined with the architecture of the autonomously identified one or more key points associated with the plurality of surgical instruments of Chen facilitates detection of the poses of the surgical tools at the surgical site which poses in the art indicate further their position, orientation or trajectory to further track said tools in realtime continuously which in a case may go outside the field of view of the camera, the realtime tracking further suppresses and/or prevent operation of erroneous tools at the surgical site and erroneously operation of said tools outside the field view of the doctor which ultimately prevent damage and injuries to the patient according to further known methods to yield predictable results since known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if the variations are predictable to one of ordinary skill in the art as said combination is thus the adaptation of an old idea or invention using newer technology that is either commonly available and understood in the art thereby a variation on already known art (See MPEP 2143, KSR Exemplary Rationale F).
Regarding claim 2 (according to claim 1), Chen further teaches wherein a pose of a surgical instrument from the plurality of surgical instruments is used to provide user feedback (orientation of tool tips of further para. 0045-0053 further indicative in the art of the poses of the surgical tools from the plurality of surgical instruments is used to provide further in at least para. 0053 user feedback).
Regarding claim 3 (according to claim 1), Chen further teaches wherein the one or more processors are further configured to generate a bounding box of a surgical instrument based on the one or more key points grouped according to the surgical instrument (at least Figs. 3-4 and para. 0011-0013 further teaches said generate bounding box of a surgical instrument based on the one or more key points grouped according to the surgical instrument).
Regarding claim 4 (according to claim 1), Chen further teaches wherein the video of the surgical procedure is captured by an endoscopic camera from inside a patient's body (at least para. 0053 further illustrates video of the surgical procedure captured by an endoscopic camera from inside a patient's body).
Regarding claim 5 (according to claim 1), Chen further teaches wherein the video of the surgical procedure is captured by a camera from outside a patient's body (at least para. 0053 further illustrates video of the surgical procedure captured in a case captured procedure by obviously a camera from outside a patient's body).
Regarding claim 6 (according to claim 1), Chen further teaches wherein the first machine learning model outputs an annotation for each of the one or more key points identified (a machine learning model of at least para. 0047-0049 is understoodly adapted for further marking in at least para. 0022-0024 visual indicators or outputs annotation for each of the one or more key points identified).
Regarding claim 7 (according to claim 1), Chen further teaches wherein the poses and types of the surgical instruments are identified with temporal continuity (surgical tools types determination of further para. 0051-0052 and orientation indicative of the poses of said surgical instruments are identified continuously with implied temporal continuity).
Regarding claim 8 (according to claim 1), Chen further teaches wherein the one or more processors are configured to test a surgical robotic arm (said robotic arm of at least para. 0053 may be obviously configured to be tested via at least known issuing commands to said the surgical robotic arm),
the test comprising:
issuing a command to the surgical robotic arm that results in a surgical instrument associated with the surgical robotic arm to be in a predetermined pose (manipulating the robotic arm of at least para. 0001 by at least one or more known
issuing command to the surgical robotic arm that results in a surgical instrument of at least Figs. 2-6 associated with the surgical robotic arm to be in a predetermined orientation or pose);
determining a first pose of the surgical instrument based on the one or more key points that are grouped (determining of Figs. 2-6 further comprises determining at least first orientation or first pose of the surgical instrument based on the one or more key points that are grouped);
and comparing the first pose and the predetermined pose (ascertaining of the surgical tools position, and orientation further indicative in the art of at least a first pose of the tools compared to an implied predetermined pose or predetermined orientation to arrive at or detect an instrument tool that needs in para. 0051 and 0044 to be re-centered based on at least implied comparison of the current surgical tools position, and orientation further indicative in the art of at least a first pose of the tools compared to an implied predetermined pose or predetermined orientation).
Regarding claim 9, Chen teaches a computer-implemented method (the system and methods of at least para. 0040-0051 and Figs. 2-6 illustrates a case of identifying captured surgical tools types, and position and orientation of the tools, said position and orientation are indicative in the art of a pose of the detected tool tip based at least on identified plurality of key points), comprising:
identifying, autonomously, one or more key points associated with a plurality of surgical instruments in a video of a surgical procedure (autonomously identified said one or more key points of further para. 0044-0045, and 0048-0051 associated with a plurality of surgical instruments in a video of a surgical procedure);
clustering a set of key points from the one or more key points associated with a surgical instrument using a first machine learning model (clustering and connecting of at least para. 0045 the one or more key points according to the plurality of surgical instruments of further para. 0044-0045, and 0048-0051 using a first machine learning model of para. 0048-0049); and
determining, based on the set of key points that are grouped, position/location, and orientation of the surgical instrument (ascertaining of the surgical tools position, and orientation of at least para. 0026, and 0044-0051 at the surgical site, said position and orientation are indicative in the art of at least a first pose of the tools based on the set of key points).
However, Chen is silent regarding specifically said grouping set of key points from the one or more key points and determine, based on the one or more key points that are grouped, specifically a pose of the surgical instrument.
Jarvis teaches at least in para. 0033-0034 detected poses of identified objects which are further illustrated in at least para. 0101 to indicate or represent identified medical tools/objects at the surgical site corresponding to grouped or overlaid one or more grouped key points of further para. 0094 which are further used to determine specific poses and types of tools determined understanbly in the art concurrently to determine in a case if the identified surgical tools are in a field of view. It would have been obvious to one of ordinary skill in the art at the time the invention was made to combine the teachings of Chen in view of Jarvis to include wherein said grouping set of key points from the one or more key points and determine, based on the one or more key points that are grouped, specifically a pose of the surgical instrument, as discussed above, as Chen in view of Jarvis are in the same field of endeavor of capturing in realtime a performed surgical procedure, tracking surgical instruments location, position and poses to determine at least an orientation and types of said surgical tools, Jarvis’s determination of the poses and types of the surgical instruments at the surgical site according to detected keypoints information complements the autonomously identified one or more key points associated with the plurality of surgical instruments illustrated in the video of the surgical procedure of Chen in the sense that the determination of the poses and types of the surgical instruments architecture of Jarvis when combined with the architecture of the autonomously identified one or more key points associated with the plurality of surgical instruments of Chen facilitates detection of the poses of the surgical tools at the surgical site which poses in the art indicate further their position, orientation or trajectory to further track said tools in realtime continuously which in a case may go outside the field of view of the camera, the realtime tracking further suppresses and/or prevent operation of erroneous tools at the surgical site and erroneously operation of said tools outside the field view of the doctor which ultimately prevent damage and injuries to the patient according to further known methods to yield predictable results since known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if the variations are predictable to one of ordinary skill in the art as said combination is thus the adaptation of an old idea or invention using newer technology that is either commonly available and understood in the art thereby a variation on already known art (See MPEP 2143, KSR Exemplary Rationale F).
Regarding claim 10 (according to claim 1), Chen is silent regarding wherein further comprising depicting a graphical overlay on the video to indicate the identified pose of the surgical instrument.
Jarvis further teaches in at least para. 0032-0033 graphical overlaid of one or more keypoint on the video to indicate identified pose of identified objects being indicative further in at least para. 0101 of medical/surgical instrument. It would have been obvious to one of ordinary skill in the art at the time the invention was made to combine the teachings of Chen in view of Jarvis to include wherein said depicting graphical overlay on the video to indicate the identified pose of the surgical instrument, as discussed above, as Chen in view of Jarvis are in the same field of endeavor of capturing in realtime a performed surgical procedure, tracking surgical instruments location, position and poses to determine at least an orientation and types of said surgical tools, Jarvis’s determination of the poses and types of the surgical instruments at the surgical site according to detected keypoints information complements the autonomously identified one or more key points associated with the plurality of surgical instruments illustrated in the video of the surgical procedure of Chen in the sense that the determination of the poses and types of the surgical instruments architecture of Jarvis when combined with the architecture of the autonomously identified one or more key points associated with the plurality of surgical instruments of Chen facilitates detection of the poses of the surgical tools at the surgical site which poses in the art indicate further their position, orientation or trajectory to further track said tools in realtime continuously which in a case may go outside the field of view of the camera, the realtime tracking further suppresses and/or prevent operation of erroneous tools at the surgical site and erroneously operation of said tools outside the field view of the doctor which ultimately prevent damage and injuries to the patient according to further known methods to yield predictable results since known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if the variations are predictable to one of ordinary skill in the art as said combination is thus the adaptation of an old idea or invention using newer technology that is either commonly available and understood in the art thereby a variation on already known art (See MPEP 2143, KSR Exemplary Rationale F).
Regarding claim 18, Chen teaches in at least para. 0005 a computer program product comprising a memory device with computer-readable instructions stored thereon, wherein executing the computer-readable instructions by one or more processing units causes the one or more processing units to perform a method comprising:
accessing, a video of a surgical procedure comprising use of a plurality of surgical instruments concurrently (at least para. 0005 and 0053 further teaches utilizing and accessing, a camera attached to at least a robotic arm, a video of a surgical procedure comprising use of a plurality of surgical instruments concurrently);
identifying, autonomously, one or more key points associated with the surgical instruments (identifying further in at least para. 0005, 0044-0045, and 0048-0051 autonomously, one or more key points associated with the surgical instruments);
concurrently performing, using one or more machine learning models of further para. 0047-0049:
clustering a set of key points from the one or more key points, the set of key points associated with a surgical instrument (concurrently performing as implied clustering and connecting of at least para. 0045 the one or more key points associated with the plurality of surgical instruments of further para. 0044-0045, and 0048-0051); and
identifying a type of the surgical instrument based on the set of key points (identifying further in at least para. 0047-0051 the detected tools type based on the set of key points); and
estimating a pose of the surgical instrument based on the set of key points (estimating further in at least para. 0047-0051 the detected tools poses based on the set of key points); and
augmenting the video of the surgical procedure in response to a key point of the surgical instrument being out of view from the video (the system further in at least para. 0051 and 0053 may normalized the captured video by at least by placing the target surgical tool in the center of the video, or perform re-centering of the detected tools, said placing and/or re-centering is understood in the art to comprise a video augmenting process in response to obviously a key point of the surgical instrument of further para. 0045-0047 being out of view from the video).
However, Chen is silent regarding specifically grouping said set of key points from the one or more key points, the set of key points associated with a surgical instrument.
Jarvis teaches at least in para. 0033-0034 detected poses of identified objects which are further illustrated in at least para. 0101 to indicate or represent identified medical tools/objects at the surgical site corresponding to grouped or overlaid one or more grouped key points of further para. 0094 which are further used to determine specific poses and types of tools determined understanbly in the art concurrently to determine in a case if the identified surgical tools are in a field of view. It would have been obvious to one of ordinary skill in the art at the time the invention was made to combine the teachings of Chen in view of Jarvis to include wherein said set of key points from the one or more key points, the set of key points associated with a surgical instrument, as discussed above, as Chen in view of Jarvis are in the same field of endeavor of capturing in realtime a performed surgical procedure, tracking surgical instruments location, position and poses to determine at least an orientation and types of said surgical tools, Jarvis’s determination of the poses and types of the surgical instruments at the surgical site based on the grouping of the one or more according to detected keypoints information complements the autonomously identified one or more key points associated with the plurality of surgical instruments illustrated in the video of the surgical procedure of Chen in the sense that the determination of the poses and types of the surgical instruments architecture of Jarvis when combined with the architecture of the autonomously identified one or more key points associated with the plurality of surgical instruments of Chen facilitates detection of the poses of the surgical tools at the surgical site which poses in the art indicate further their position, orientation or trajectory to further track said tools in realtime continuously which in a case may go outside the field of view of the camera, the realtime tracking further suppresses and/or prevent operation of erroneous tools at the surgical site and erroneously operation of said tools outside the field view of the doctor which ultimately prevent damage and injuries to the patient according to further known methods to yield predictable results since known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if the variations are predictable to one of ordinary skill in the art as said combination is thus the adaptation of an old idea or invention using newer technology that is either commonly available and understood in the art thereby a variation on already known art (See MPEP 2143, KSR Exemplary Rationale F).
Regarding claim 20 (according to claim 18), Chen further teaches wherein the method further comprises augmenting the video of the surgical procedure in response to the key point of the surgical instrument being within a predetermined proximity of an anatomical structure (aligning of further para. 0029 further comprises in the art augmenting of the video of the surgical procedure in response to the localization or a key point of the surgical instrument being within a predetermined proximity of an anatomical structure, eye retina and the like).
Claim 11 is/are rejected under 35 U.S.C. 103 as obvious over Chen in view of Jarvis, and further in view of Wang et al. (WO 2018/195221, A1).
Regarding claim 11 (according to claim 10), Chen in view of Jarvis are silent regarding wherein the graphical overlay includes a depiction of the one or more key points to identify an exit path to move the surgical instrument.
Wang teaches identify in at least para. 0156 endpoints of surgical instrument tool, endpoints of a path and the like and generating graphical path data overlay includes a depiction of the one or more key points to identify an exit path to move the surgical instrument. It would have been obvious to one of ordinary skill in the art at the time the invention was made to combine the teachings of Chen in view of Jarvis, and further in view of Wang to include wherein said graphical overlay includes a depiction of the one or more key points to identify an exit path to move the surgical instrument, as discussed above, as Chen in view of Jarvis, and further in view of Wang are in the same field of endeavor of capturing in realtime a video of a performed surgical procedure, tracking surgical instruments location, position to determine at least an orientation and types of said surgical tools, Wang’s identified exit path to move the surgical instrument according to detected endpoints information complements the autonomously identified one or more key points associated with the plurality of surgical instruments illustrated in the video of the surgical procedure of Chen in view of Jarvis in the sense that the identified exit path architecture of Wang when combined with the architecture of the autonomously identified one or more key points associated with the plurality of surgical instruments of Chen in view of Jarvis facilitates safe travel movements of the surgical instrument tool at the surgical site corresponding o further detected position, orientation or trajectory of the tools which safe travel path or exiting ultimately prevent damage and injuries to the patient according to further known methods to yield predictable results since known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if the variations are predictable to one of ordinary skill in the art as said combination is thus the adaptation of an old idea or invention using newer technology that is either commonly available and understood in the art thereby a variation on already known art (See MPEP 2143, KSR Exemplary Rationale F).
Claims 12-17 is/are rejected under 35 U.S.C. 103 as obvious over Chen in view of Jarvis, and further in view of Donhowe et al. (2020/0184248, cited in IDS).
Regarding claim 12 (according to claim 9), Chen in view of Jarvis are silent regarding wherein further comprising in response to the pose of the surgical instrument matching a threshold pose, generating a user notification.
Donhowe further teaches in at least para. 0027, and 0035-0039 a determining pose estimation of a surgical tool at a surgical site and further in para. 0035-0039, and 0043-0048 estimating whether the tool is outside or inside a desired implied threshold range amount of a field of view and/or a threshold pose force where the detected field of view either being outside or inside the field of view of further para. 0044-0045 may be in a case a desired or predetermined tool pose in said field of view further indicative of said desired pose of the surgical instrument based on a surgical action to be performed, a case further exists in at least para. 0050 whereby in response to the pose of the surgical instrument not matching the desired field of view range to generating a user notification. It would have been obvious to one of ordinary skill in the art at the time the invention was made to combine the teachings of Chen in view of Jarvis, and further in view of Donhowe to include wherein said comprising in response to the pose of the surgical instrument matching a threshold pose, generating a user notification, as discussed above, as Chen in view of Jarvis, and further in view of Donhowe are in the same field of endeavor of capturing in realtime a performed surgical procedure, tracking surgical instruments location, position and poses to determine at least an orientation and types of said surgical tools, Donhowe’s surgical tools threshold pose estimation at the surgical site complements the autonomously identified one or more key points associated with the plurality of surgical instruments illustrated in the video of the surgical procedure of Chen in view of Jarvis in the sense that the surgical tools threshold pose estimation architecture of Jarvis when combined with the architecture of the autonomously identified one or more key points associated with the plurality of surgical instruments of Chen in view of Jarvis facilitates further detection of the poses of the surgical tools in a field of view and/or outside a field of view according to a predefined surgical route site where in a case the tools deviate from the desired or predefined surgical route or path to quickly disable the tools and/or sending warning notification to the medical personnel of the operating room thereby ultimately prevent damage and injuries to the patient according to further known methods to yield predictable results since known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if the variations are predictable to one of ordinary skill in the art as said combination is thus the adaptation of an old idea or invention using newer technology that is either commonly available and understood in the art thereby a variation on already known art (See MPEP 2143, KSR Exemplary Rationale F).
Regarding claim 13 (according to claim 12), Chen in view of Jarvis are silent regarding wherein the user notification is a first user notification, and in response to the pose of the surgical instrument not matching the threshold pose, generating a second user notification, different from the first user notification.
Donhowe further teaches in at least para. 0027, and 0035-0039 a determining pose estimation of a surgical tool at a surgical site and further in para. 0035-0039, and 0043-0048 estimating whether the tool is outside or inside a desired implied threshold range amount of a field of view and/or a threshold pose force in such case to send an enabled notification command, and in response to the pose of the surgical instrument further in at least para. 0050 not matching the desired field of view range to generating a second notification, different from the first user notification. It would have been obvious to one of ordinary skill in the art at the time the invention was made to combine the teachings of Chen in view of Jarvis, and further in view of Donhowe to include wherein said user notification is a first user notification, and in response to the pose of the surgical instrument not matching the threshold pose, generating a second user notification, different from the first user notification, as discussed above, as Chen in view of Jarvis, and further in view of Donhowe are in the same field of endeavor of capturing in realtime a performed surgical procedure, tracking surgical instruments location, position and poses to determine at least an orientation and types of said surgical tools, Donhowe’s surgical tools threshold pose estimation at the surgical site complements the autonomously identified one or more key points associated with the plurality of surgical instruments illustrated in the video of the surgical procedure of Chen in view of Jarvis in the sense that the surgical tools threshold pose estimation architecture of Jarvis when combined with the architecture of the autonomously identified one or more key points associated with the plurality of surgical instruments of Chen in view of Jarvis facilitates further detection of the poses of the surgical tools in a field of view and/or outside a field of view according to a predefined surgical route site where in a case the tools deviate from the desired or predefined surgical route or path to quickly disable the tools and/or sending warning notification to the medical personnel of the operating room thereby ultimately prevent damage and injuries to the patient according to further known methods to yield predictable results since known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if the variations are predictable to one of ordinary skill in the art as said combination is thus the adaptation of an old idea or invention using newer technology that is either commonly available and understood in the art thereby a variation on already known art (See MPEP 2143, KSR Exemplary Rationale F).
Regarding claim 14 (according to claim 12), Chen in view of Jarvis are silent regarding wherein the threshold pose is indicative of a desired pose of the surgical instrument based on a surgical action to be performed.
Donhowe further teaches in at least para. 0027, and 0035-0039 a determining pose estimation of a surgical tool at a surgical site and further in para. 0035-0039, and 0043-0048 estimating whether the tool is outside or inside a desired implied threshold range amount of a field of view and/or a threshold pose force where the detected field of view either being outside or inside the field of view of further para. 0044-0045 may be in a case a desired or predetermined tool pose in said field of view further indicative of said desired pose of the surgical instrument based on a surgical action to be performed. It would have been obvious to one of ordinary skill in the art at the time the invention was made to combine the teachings of Chen in view of Jarvis, and further in view of Donhowe to include wherein said threshold pose is indicative of a desired pose of the surgical instrument based on a surgical action to be performed, as discussed above, as Chen in view of Jarvis, and further in view of Donhowe are in the same field of endeavor of capturing in realtime a performed surgical procedure, tracking surgical instruments location, position and poses to determine at least an orientation and types of said surgical tools, Donhowe’s surgical tools threshold pose estimation at the surgical site complements the autonomously identified one or more key points associated with the plurality of surgical instruments illustrated in the video of the surgical procedure of Chen in view of Jarvis in the sense that the surgical tools threshold pose estimation architecture of Jarvis when combined with the architecture of the autonomously identified one or more key points associated with the plurality of surgical instruments of Chen in view of Jarvis facilitates further detection of the poses of the surgical tools in a field of view and/or outside a field of view according to a predefined surgical route site where in a case the tools deviate from the desired or predefined surgical route or path to quickly disable the tools and/or sending warning notification to the medical personnel of the operating room thereby ultimately prevent damage and injuries to the patient according to further known methods to yield predictable results since known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if the variations are predictable to one of ordinary skill in the art as said combination is thus the adaptation of an old idea or invention using newer technology that is either commonly available and understood in the art thereby a variation on already known art (See MPEP 2143, KSR Exemplary Rationale F).
Regarding claim 15 (according to claim 12), Chen in view of Jarvis are silent regarding wherein the threshold pose is indicative of an undesired pose of the surgical instrument.
Donhowe further teaches in at least para. 0027, and 0035-0039 a determining pose estimation of a surgical tool at a surgical site and further in para. 0035-0039, and 0043-0048 estimating whether the tool is outside or inside a desired implied threshold range amount of a field of view and/or a threshold pose force where the detected field of view either being outside or inside the field of view of further para. 0044-0045 may be in a case an undesired or erroneous tool pose further indicative of said undesired pose of the surgical instrument. It would have been obvious to one of ordinary skill in the art at the time the invention was made to combine the teachings of Chen in view of Jarvis, and further in view of Donhowe to include wherein said threshold pose is indicative of an undesired pose of the surgical instrument, as discussed above, as Chen in view of Jarvis, and further in view of Donhowe are in the same field of endeavor of capturing in realtime a performed surgical procedure, tracking surgical instruments location, position and poses to determine at least an orientation and types of said surgical tools, Donhowe’s surgical tools threshold pose estimation at the surgical site complements the autonomously identified one or more key points associated with the plurality of surgical instruments illustrated in the video of the surgical procedure of Chen in view of Jarvis in the sense that the surgical tools threshold pose estimation architecture of Jarvis when combined with the architecture of the autonomously identified one or more key points associated with the plurality of surgical instruments of Chen in view of Jarvis facilitates further detection of the poses of the surgical tools in a field of view and/or outside a field of view according to a predefined surgical route site where in a case the tools deviate from the desired or predefined surgical route or path to quickly disable the tools and/or sending warning notification to the medical personnel of the operating room thereby ultimately prevent damage and injuries to the patient according to further known methods to yield predictable results since known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if the variations are predictable to one of ordinary skill in the art as said combination is thus the adaptation of an old idea or invention using newer technology that is either commonly available and understood in the art thereby a variation on already known art (See MPEP 2143, KSR Exemplary Rationale F).
Regarding claim 16 (according to claim 12), Chen further teaches wherein the user notification includes an audible notification (transmitted surgical warnings of further para. 0053 entails in a case user notification includes obviously audible notification).
Regarding claim 17 (according to claim 12), Chen further teaches wherein the user notification is provided on a separate display, distinct from the video (the generated surgical warnings of further para. 0053 output by the warning systems of further para. 0029 may be provided obviously on a separate display of further para. 0057, distinct in a case from the video).
Claim 19 is/are rejected under 35 U.S.C. 103 as obvious over Chen in view of Jarvis, and further in view of Tang et al. (CN 111144321, A1).
Regarding claim 19 (according to claim 18), Chen in view of Jarvis are silent regarding wherein the one or more machine learning models comprise multi-tasking convolutional neural network layers that aggregate spatio-temporal features in one or more frames of the video.
Tang teaches at least in the disclosure one or more machine learning models comprise multi-tasking convolutional neural network layers that use and aggregate spatio-temporal features in one or more frames of the video such as “in order to ensure that model in the video frame sequence of temporal and spatial information learning, can be deep-space multi-task 3 D convolution neural network structure to sample the video frame sequence, sample type, sample attention level result and the sample object position information for learning, training to obtain the attention level detection model”. It would have been obvious to one of ordinary skill in the art at the time the invention was made to combine the teachings of Chen in view of Jarvis, and further in view of Tang to include wherein said one or more machine learning models comprise multi-tasking convolutional neural network layers that aggregate spatio-temporal features in one or more frames of the video, as discussed above, Tang’s multi-tasking convolutional neural network layers architecture complements the autonomously identified one or more key points associated with the plurality of surgical instruments illustrated in the video of the surgical procedure of Chen in view of Jarvis in the sense that the multi-tasking convolutional neural network layers architecture of Tang when combined with the architecture of the autonomously identified one or more key points associated with the plurality of surgical instruments of Chen in view of Jarvis facilitates temporal and spatial detection of the objects of interest in the captured surgical procedure video frames to accurately detect and predict the surgical tools poses and exit path spatially and temporally which ultimately suppresses and/or prevent operation of erroneous tools at the surgical site and erroneously operation of said tools outside the field view of the doctor which thereby prevent damage and injuries to the patient according to further known methods to yield predictable results since known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if the variations are predictable to one of ordinary skill in the art as said combination is thus the adaptation of an old idea or invention using newer technology that is either commonly available and understood in the art thereby a variation on already known art (See MPEP 2143, KSR Exemplary Rationale F).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MARCELLUS AUGUSTIN whose telephone number is (571)270-3384. The examiner can normally be reached 9 AM- 5 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, BENNY TIEU can be reached at 571-272-7490. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MARCELLUS J AUGUSTIN/Primary Examiner, Art Unit 2682 12/31/2025