DETAIL OFFICE ACTIONS
The United States Patent & Trademark Office appreciates the response filed for the current application that is submitted on 02/23/2026. The United States Patent & Trademark Office reviewed the following documents submitted and has made the following comments below.
Amendment
Applicant submitted amendments on 02/23/2026. The Examiner acknowledges the amendment and has reviewed the claims accordingly.
Applicant Arguments:
Applicant/s state/s that the cited prior arts do not teach the amended claims, specially, the limitation “a digital surgical microscope (DSM) that is coupled to a robotic arm” and “the DSM camera to track the distal end of the tool of interest by at least transmitting one or more robotic arm controls to the robotic arm to which the DSM is connected.”; therefore, the rejection under 35 U.S.C. 103 should be withdrawn.
Examiner’s Responses:
Applicant’s arguments and amendments, see Remarks, filed 02/23/2026, with respect to the rejection(s) of claim(s) 1-20 under 35 U.S.C. 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration of amendments, a new ground(s) of rejection is made in view of Chen (US-20200397509-A1) in view of Zucker (US-20220241032-A1), and further in view of Meglan (US-20220304555-A1).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 9 and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over
Chen et al. (US-20200397509-A1, hereinafter Chen) in view of
Zucker et al. (US-20220241032-A1, provisional benefit claimed 02/01/2021, hereinafter Zucker), and further in view of
Meglan et al. (US-20220304555-A1, filed 2020, hereinafter Meglan).
CLAIM 1
In regards to Claim 1, Chen teaches a method of tracking a surgical tool in real-time (Chen, ¶ [0003]: “With further regard to the system…determining the position of the tip of the surgical tool and determining the orientation of the surgical tool in real-time”).
Chen does not explicitly disclose a digital surgical microscope (DSM) that is coupled to a robotic arm.
Zucker is in the same field of art of tracking surgical instruments. Further, Zucker teaches a digital surgical microscope (DSM) (Zucker, ¶ [0072]: “The imaging device 132 may be operable to image anatomical feature(s) (e.g., a bone, veins, tissue, etc.) and/or other aspects of patient anatomy, a target, a tracking marker, and/or any surgical instruments or tools within the field of view … The imaging device 132 may be or comprise … a microscope”) that is coupled to a robotic arm. (Zucker, ¶ [0072-0073]: “two imaging devices 132 may be supported on a single robotic arm 148 to capture images from different angles at the same time”; ¶ [0014, 0019 and 0029]: “a first imaging device secured to the first robotic arm”)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Chen by incorporating a robotic arm with attached microscope camera that is taught by Zucker, to make a surgical tool tracking system that has camera supported by robotic arm; thus, one of ordinary skilled in the art would be motivated to combine the references since among its several aspects, the present invention recognizes there is a need to improve target identification accuracy using robotic arm (Zucker, ¶ [0059-0060]: “a robotic arm may automatically change its position “around” a target to obtain more angles and improve an accuracy of an identified target position.”). In addition, Chen mentions his invention can be used with robotic arm to support camera and surgical tools. (Chen, ¶ [0053]: “Implementations may be used with robotic arms during surgery (e.g., camera holder, tool holder, etc.).”)
The combination of Chen and Zucker then teaches the method comprising:
receiving, in real-time, by a computing device having a processor (Chen, ¶ [0002]: “a system includes one or more processors”), image data of a surgical video stream captured by a camera of the DSM (Zucker, ¶ [0073]: “The imaging device 132 may be operable to generate a stream of image data”), wherein the surgical video stream shows a tool of interest (Chen, ¶ [0017]: “a system receives an image frame from a camera…, the image frame may be one of a series of image frames of a video stream. The system detects a surgical instrument or tool in the at least one of the image frames”);
applying the image data to a first trained neural network model (Chen, ¶ [0047-0049]: “the system uses deep learning for tool detection and key points estimation on the surgical tool”) to determine a location (Chen, ¶ [0032]. Chen teaches determining location of surgical tool by key-point matching) for a bounding box around the tool of interest (Chen, ¶ [0021-0022]: “the system initially detects surgical tool … in image frame within boundary box”, see FIG. 3)
The combination of Chen and Zucker does not explicitly disclose generating augmented image data comprising a bounding box around the tool of interest;
Meglan is in the same field of art of tracking surgical instrument. Further, Meglan teaches generating augmented image data comprising a bounding box around the tool of interest. (Meglan, ¶ [0096]: “an image is input into a neural network … a region proposal network is slid over the convolutional feature map and generates proposals for the region of interest … Finally, the augmented image is output with bounding boxes …”; ¶ [0077-0079], Meglan teaches various ways to generate an augmented/enhanced image; see FIG. 5 for the overall process, see FIG. 9-10 for augmented image with bounding box)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Chen and Zucker by incorporating the region proposal network that is taught by Meglan, to make a surgical tool tracking system that can generate augmented image of surgical tool within bounding boxes; thus, one of ordinary skilled in the art would be motivated to combine the references since among its several aspects, the present invention recognizes there is a need to improve object detection by generating augmented/enhanced image with discernable features (Meglan, ¶ [0001]: “to enhancing aspects of discernable features of objects during surgical procedures”, ¶ [0077]: “the enhanced image may be fed as an input into the neural network of FIG. 7 for additional object detection.”).
The combination of Chen, Zucker and Meglan then teaches applying the augmented image data (Meglan, ¶ [0077]: “the enhanced image may be fed as an input into the neural network of FIG. 7 for additional object detection) to a second trained neural network model to determine a distal end point of the tool of interest (Chen, ¶ [0033]: “At block 508, the system determines the position of the tip of surgical tool based on key points…”, ¶ [0047-0049]: “the system uses deep learning for tool detection and key points estimation on the surgical tool”); and
causing, in real-time by the computing device, the DSM camera to track the distal end of the tool of interest (Chen, ¶ [0028]: “… the system tracks the movement of the surgical tool tip.”) by at least transmitting one or more robotic arm controls to the robotic arm to which the DSM is connected. (Zucker, ¶ [0059-0064]: “a robotic system that contains two or more robotic arms. Each robotic arm may be mounted with a camera or otherwise support a camera … a robotic arm may automatically change its position “around” a target to obtain more angles and improve an accuracy of an identified target position … a robotic arm may support a tool and perform an action with the tool and one or more different robotic arms may track the action with one or more cameras … (1) determining a pose, a position, or an orientation of a target; (2) tracking movement of a target; (3) adjusting a path of a tool, an instrument, or an implant based on tracked movement of a target; (4) increasing an accuracy of a pose, a position, or an orientation of a target (whether a tool, an instrument, an implant, a robotic arm, or any other object)” Zucker teaches a system with multiple robotic arms, each can be mounted with camera or surgical tool, camera-mounted arm can automatically move to identify and track a target (the target could be a tool, an instrument or other robotic arm))
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
CLAIM 9
In regards to Claim 9, Chen teaches a system for tracking a surgical tool in real-time (Chen, ¶ [0003]: “With further regard to the system…determining the position of the tip of the surgical tool and determining the orientation of the surgical tool in real-time”); a robotic arm. (Chen, ¶ [0018 and 0053]: “the surgeon is handling surgical tool 112 directly with the surgeon's hand and/or with a robotic arm … Implementations may be used with robotic arms during surgery (e.g., camera holder, tool holder, etc.)”)
Chen does not explicitly disclose a digital surgical microscope (DSM), a DSM comprising a DSM camera that is coupled to the robotic arm.
Zucker is in the same field of art of tracking surgical instruments. Further, Zucker teaches a digital surgical microscope (DSM) (Zucker, ¶ [0072]: “The imaging device 132 may be operable to image anatomical feature(s) (e.g., a bone, veins, tissue, etc.) and/or other aspects of patient anatomy, a target, a tracking marker, and/or any surgical instruments or tools within the field of view … The imaging device 132 may be or comprise … a microscope”), a DSM comprising a DSM camera that is coupled to the robotic arm (Zucker, ¶ [0072-0073]: “two imaging devices 132 may be supported on a single robotic arm 148 to capture images from different angles at the same time”; ¶ [0014, 0019 and 0029]: “a first imaging device secured to the first robotic arm”).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Chen by incorporating a robotic arm with attached microscope camera that is taught by Zucker, to make a surgical tool tracking system that has camera supported by robotic arm; thus, one of ordinary skilled in the art would be motivated to combine the references since among its several aspects, the present invention recognizes there is a need to improve target identification accuracy in tracking surgical instruments using robotic arm (Zucker, ¶ [0059-0060]: “a robotic arm may automatically change its position “around” a target to obtain more angles and improve an accuracy of an identified target position.”). In addition, Chen mentions his invention can be used with robotic arm to support camera and surgical tools. (Chen, ¶ [0053]: “Implementations may be used with robotic arms during surgery (e.g., camera holder, tool holder, etc.).”)
The combination of Chen and Zucker then teaches a processor (Chen, ¶ [0002]: “a system includes one or more processors”); and a memory device storing computer-executable instructions (Chen, ¶ [0002]: “one or more non-transitory computer-readable storage media for execution by the one or more processors”) that, when executed by the processor, causes the processor: receiving, in real-time, image data of a surgical video stream captured by a camera of the DSM (Zucker, ¶ [0073]: “The imaging device 132 may be operable to generate a stream of image data”), wherein the surgical video stream shows a tool of interest (Chen, ¶ [0017]: “a system receives an image frame from a camera…, the image frame may be one of a series of image frames of a video stream. The system detects a surgical instrument or tool in the at least one of the image frames”);
applying the image data to a first trained neural network model (Chen, ¶ [0047-0049]: “the system uses deep learning for tool detection and key points estimation on the surgical tool”) to determine a location (Chen, ¶ [0032]. Chen teaches determining location of surgical tool by key-point matching) for a bounding box around the tool of interest (Chen, ¶ [0021-0022]: “the system initially detects surgical tool … in image frame within boundary box”, see FIG. 3);
The combination of Chen and Zucker does not explicitly disclose generating augmented image data comprising a bounding box around the tool of interest;
Meglan is in the same field of art of tracking surgical instrument. Further, Meglan teaches generating augmented image data comprising a bounding box around the tool of interest. (Meglan, ¶ [0096]: “an image is input into a neural network … a region proposal network is slid over the convolutional feature map and generates proposals for the region of interest … Finally, the augmented image is output with bounding boxes …”; ¶ [0077-0079], Meglan teaches various ways to generate an augmented/enhanced image; see FIG. 5 for the overall process, see FIG. 9-10 for augmented image with bounding box)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Chen and Zucker by incorporating the region proposal network that is taught by Meglan, to make a surgical tool tracking system that can generate augmented image of surgical tool within bounding boxes; thus, one of ordinary skilled in the art would be motivated to combine the references since among its several aspects, the present invention recognizes there is a need to improve object detection by generating augmented/enhanced image with discernable features (Meglan, ¶ [0001]: “to enhancing aspects of discernable features of objects during surgical procedures”, ¶ [0077]: “the enhanced image may be fed as an input into the neural network of FIG. 7 for additional object detection.”).
The combination of Chen, Zucker and Meglan then teaches applying the augmented image data (Meglan, ¶ [0077]: “the enhanced image may be fed as an input into the neural network of FIG. 7 for additional object detection) to a second trained neural network model to determine a distal end point of the tool of interest (Chen, ¶ [0033]: “At block 508, the system determines the position of the tip of surgical tool based on key points…”, ¶ [0047-0049]: “the system uses deep learning for tool detection and key points estimation on the surgical tool”); and
causing, in real-time by the computing device, the DSM camera to track the distal end of the tool of interest by at least transmitting one or more robotic arm controls to the robotic arm. (Zucker, ¶ [0059-0064]: “a robotic system that contains two or more robotic arms. Each robotic arm may be mounted with a camera or otherwise support a camera … a robotic arm may automatically change its position “around” a target to obtain more angles and improve an accuracy of an identified target position … a robotic arm may support a tool and perform an action with the tool and one or more different robotic arms may track the action with one or more cameras … (1) determining a pose, a position, or an orientation of a target; (2) tracking movement of a target; (3) adjusting a path of a tool, an instrument, or an implant based on tracked movement of a target; (4) increasing an accuracy of a pose, a position, or an orientation of a target (whether a tool, an instrument, an implant, a robotic arm, or any other object)” Zucker teaches a system with multiple robotic arms, each can be mounted with camera or surgical tool, camera-mounted arm can automatically move to identify and track a target (the target could be a tool, an instrument or other robotic arm))
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
CLAIM 17
In regards to Claim 17, the combination of Chen, Zucker and Meglan teaches the system of Claim 9. In addition, the combination of Chen, Zucker and Meglan teaches the first trained neural network model is trained to differentiate between the tool of interest and one or more suction tools, and wherein the tool of interest is not any of the one or more suction tools. (Meglan, ¶ [0084]: “the video system 30 performs real-time image recognition on the first image to detect the object, classify the object and produce a first image classification probability value. For example, the video system 30 may detect a surgical instrument such as a stapler in the first image. For example, the detected object may include, but is not limited to, tissue, forceps, regular grasper, bipolar grasper, monopolar shear, suction, needle driver, and stapler”. Meglan teaches to detect and identify the surgical tool, the surgical tool could be suction tool or a plurality of other surgical tools.) (Chen, ¶ [0050]. Chen also teaches to detect and identify different surgical tool)
Claim(s) 2-3, 8, 10-11 and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen in view of Zucker, in view of Meglan and further in view of Venkataraman et al. (US-20200304753-A1, hereinafter Venka).
CLAIM 2
In regards to Claim 2, the combination of Chen, Zucker and Meglan teaches the method of claim 1.
The combination of Chen, Zucker and Meglan does not explicitly disclose identifying, by the computing device, based on a previously received image data of the surgical video stream, a displacement of a feature in the surgical video stream beyond a threshold distance.
Venka is in the same field of art of object tracking. Further, Holtz teaches identifying, by the computing device, based on a previously received image data of the surgical video stream, a displacement of a feature in the surgical video stream beyond a threshold distance (Venka, ¶ [0073]: “when the location of the tool tip is determined to be near an edge of the viewing window and about to go off-screen …” The Examiner note Venka’s distance from center to the edge of viewing window corresponds to the “threshold distance” of the invention), wherein the applying the image data, the generating the augmented image data, the applying the augmented image data, and the causing the DSM camera to track the distal end of the tool of interest is responsive to the identified displacement. (Venka, ¶ [0073-0074]: “… the system can automatically reposition the viewing window from the current location to the new ROI so that the tool tip is brought back to the center or closer to the center of the display. Moreover, after the initial repositioning of the viewing window, the system can start following the movement of the tool tip by continuously adjusting the position of the viewing window based on the movement of the tool tip”)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Chen, Zucker and Meglan by incorporating the method to refocus in case tracked object is displaced that is taught by Venka, to make an object tracking system that can perform autofocus; thus, one of ordinary skilled in the art would be motivated to combine the references since among its several aspects, the present invention recognizes there is a need for an automatic method to adjust the surgical microscope camera (Venka, ¶ []: [0007-0008]: “If the system detects that the tool tip is about to go off-screen, the system can automatically adjust/reposition the viewing window within the full-resolution endoscope video to keep the tool tip on the screen and visible, thereby preventing the surgeon from having to manually adjust the location of the viewing window or the endoscope camera inside the patient to keep the tool tip on the screen..”).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
CLAIM 3
In regards to Claim 3, the combination of Chen, Zucker, Meglan and Venka teaches the method of Claim 2. In addition, the combination of Chen, Zucker, Meglan and Venka teaches adjusting, by the computing device, a field of view of the DSM camera such that a focus point associated with the distal end of the tool of interest is at the center of the field of view (Venka, ¶ [0073], see rejection of claim 2. The distal end of the tool is kept at the center of viewing window), wherein the focus point is at a predetermined distance from the distal end of the tool of interest in a direction towards the displacement. (Venka, ¶ [0073], see rejection of claim 2. The distal end of the tool can move from center to edge of viewing screen, the camera maybe automatically adjusted to reposition the distal end to the center.
CLAIM 8
In regards to Claim 8, the combination of Chen, Zucker and Meglan teaches the method of claim 1.
The combination of Chen, Zucker and Meglan does not explicitly disclose applying, by the computing device, one or more image data of the surgical video stream to a third neural network model to detect a user intent; and altering, based on the user intent, one or more settings of the DSM camera.
Venka is in the same field of art of tracking surgical instrument. Further, Venka teaches applying, by the computing device, one or more image data of the surgical video stream to a third neural network model to detect a user intent (Venka, ¶ [0083-0084]: “the system uses one or more deep-learning models to determine the location of user's gaze on the display by analyzing the captured images of the user's eyes and head. After determining the initial location of the user's gaze, the system starts tracking a movement of the user's gaze from the initial location, e.g., by using a deep-learning-based gaze-tracking technique”); and altering, based on the user intent, one or more settings of the DSM camera. (Venka, ¶ [0083-0084]: “the system automatically repositions the viewing window from the current location to the new ROI to keep the user's gaze near the center of the display (step 612).”)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Chen, Zucker and Meglan by incorporating the method to refocus in case tracked object is displaced that is taught by Venka, to make an object tracking system that can perform autofocus; thus, one of ordinary skilled in the art would be motivated to combine the references since among its several aspects, the present invention recognizes there is a need for an automatic method to adjust the surgical microscope camera (Venka, ¶ [0007-0008]: “If the system detects that the tool tip is about to go off-screen, the system can automatically adjust/reposition the viewing window within the full-resolution endoscope video to keep the tool tip on the screen and visible, thereby preventing the surgeon from having to manually adjust the location of the viewing window or the endoscope camera inside the patient to keep the tool tip on the screen..”).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
CLAIM 10
In regards to Claim 10, the combination of Chen, Zucker and Meglan teaches the system of claim 9.
The combination of Chen, Zucker and Meglan does not explicitly disclose identifying, based on a previously received image data of the surgical video stream, a displacement of a feature in the surgical video stream beyond a threshold distance.
Venka is in the same field of art of object tracking. Further, Holtz teaches identifying, based on a previously received image data of the surgical video stream, a displacement of a feature in the surgical video stream beyond a threshold distance (Venka, ¶ [0073]: “when the location of the tool tip is determined to be near an edge of the viewing window and about to go off-screen …” The Examiner note Venka’s distance from center to the edge of viewing window corresponds to the “threshold distance” of the invention), wherein the applying the image data, the generating the augmented image data, the applying the augmented image data, and the causing the DSM camera to track the distal end of the tool of interest is responsive to the identified displacement. (Venka, ¶ [0073-0074]: “… the system can automatically reposition the viewing window from the current location to the new ROI so that the tool tip is brought back to the center or closer to the center of the display. Moreover, after the initial repositioning of the viewing window, the system can start following the movement of the tool tip by continuously adjusting the position of the viewing window based on the movement of the tool tip”)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Chen, Zucker and Meglan by incorporating the method to refocus in case tracked object is displaced that is taught by Venka, to make an object tracking system that can perform autofocus; thus, one of ordinary skilled in the art would be motivated to combine the references since among its several aspects, the present invention recognizes there is a need for an automatic method to adjust the surgical microscope camera (Venka, ¶ [0007-0008]: “If the system detects that the tool tip is about to go off-screen, the system can automatically adjust/reposition the viewing window within the full-resolution endoscope video to keep the tool tip on the screen and visible, thereby preventing the surgeon from having to manually adjust the location of the viewing window or the endoscope camera inside the patient to keep the tool tip on the screen..”).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
CLAIM 11
In regards to Claim 11, the combination of Chen, Zucker, Meglan and Venka teaches the system of Claim 10. In addition, the combination of Chen, Zucker, Meglan and Venka teaches adjusting a field of view of the DSM camera such that a focus point associated with the distal end of the tool of interest is at the center of the field of view (Venka, ¶ [0073], see rejection of claim 2. The distal end of the tool is kept at the center of viewing window), wherein the focus point is at a predetermined distance from the distal end of the tool of interest in a direction towards the displacement. (Venka, ¶ [0073], see rejection of claim 2. The distal end of the tool can move from center to edge of viewing screen, the camera maybe automatically adjusted to reposition the distal end to the center.
CLAIM 16
In regards to Claim 16, the combination of Chen, Zucker and Meglan teaches the system of claim 9.
The combination of Chen, Zucker and Meglan does not explicitly disclose applying, by the computing device, one or more image data of the surgical video stream to a third neural network model to detect a user intent; and altering, based on the user intent, one or more settings of the DSM camera.
Venka is in the same field of art of tracking surgical instrument. Further, Venka teaches applying, by the computing device, one or more image data of the surgical video stream to a third neural network model to detect a user intent (Venka, ¶ [0083-0084]: “the system uses one or more deep-learning models to determine the location of user's gaze on the display by analyzing the captured images of the user's eyes and head. After determining the initial location of the user's gaze, the system starts tracking a movement of the user's gaze from the initial location, e.g., by using a deep-learning-based gaze-tracking technique”); and altering, based on the user intent, one or more settings of the DSM camera. (Venka, ¶ [0083-0084]: “the system automatically repositions the viewing window from the current location to the new ROI to keep the user's gaze near the center of the display (step 612).”)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Chen, Zucker and Meglan by incorporating the method to refocus in case tracked object is displaced that is taught by Venka, to make an object tracking system that can perform autofocus; thus, one of ordinary skilled in the art would be motivated to combine the references since among its several aspects, the present invention recognizes there is a need for an automatic method to adjust the surgical microscope camera (Venka, ¶ [0007-0008]: “If the system detects that the tool tip is about to go off-screen, the system can automatically adjust/reposition the viewing window within the full-resolution endoscope video to keep the tool tip on the screen and visible, thereby preventing the surgeon from having to manually adjust the location of the viewing window or the endoscope camera inside the patient to keep the tool tip on the screen..”).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim(s) 4-7 and 12-15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen in view of Zucker in view of Meglan, and further in view of Herbwood (Herbwood “Fast R-CNN Paper Review” https://herbwood.tistory.com/8, published 2020, a translated copy of this document is attached, hereinafter Herbwood).
CLAIM 4
In regards to Claim 4, the combination of Chen, Zucker and Meglan teaches the method of Claim 1. In addition, the combination of Chen, Zucker and Meglan teaches CNN-based machine learning model to detect surgical instrument and distal end of surgical instrument. (Chen, FIG. 5, step 502 and 508) (Meglan, ¶ [0085-0086]: “the video system 30 may detect the object based on a convolutional neural network”)
The combination of Chen, Zucker and Meglan does not explicitly disclose generating a first input feature vector based on the image data; and applying the first input feature vector to the first trained neural network model to generate a first output feature vector, wherein the first output feature vector indicates the location for the bounding box around the tool of interest.
Herbwood is in the same field of art of CNN-based object detection. Further, Herbwood teaches generating a first input feature vector based on the image data (Herbwood, page 10-11, section 5: “we flatten the 7x7x512 (=25088) feature map for each region proposal and input it into the fc layer to obtain a feature vector of size 4096 through the fc layer…”, see figure in section 5 below, with annotation); and applying the first input feature vector to the first trained neural network model to generate a first output feature vector, wherein the first output feature vector indicates the location for the bounding box around the tool of interest. (Herbwood, page 11-12, section 7. Bounding box regressor: “A feature vector of size 4096 is input to an fc layer with (K+1) x 4 output units to predict the coordinates of bounding boxes for each class. It outputs bounding box coordinates adjusted for each class for one region proposal in one image”, see figure in section 5 below, with annotation)
PNG
media_image1.png
624
939
media_image1.png
Greyscale
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Chen, Zucker and Meglan by simply substituting their CNN-based model for object detection with the Fast R-CNN model that is taught by Herbwood, to make an object detection system using R-CNN model; thus, one of ordinary skilled in the art would be motivated to make the substitution since among its several aspects, the present invention recognizes there is a need to improve object detection speed of the machine learning model (Herbwood, page 1, first paragraph: “Fast R-CNN, as its name suggests, demonstrates a significant speed improvement over the existing R-CNN model.”).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
CLAIM 5
In regards to Claim 5, the combination of Chen, Zucker and Meglan teaches the method of Claim 1. In addition, the combination of Chen, Zucker and Meglan teaches CNN-based machine learning model to detect surgical instrument and distal end of surgical instrument. (Chen, FIG. 5, step 502 and 508) (Meglan, ¶ [0085-0086]: “the video system 30 may detect the object based on a convolutional neural network”)
The combination of Chen, Zucker and Meglan does not explicitly disclose generating a second input feature vector based on the augmented image data; and applying the second input feature vector to the second trained neural network model to generate a second output feature vector, wherein the second output feature vector indicates the location for the distal end of the tool of interest.
Herbwood is in the same field of art of CNN-based object detection. Further, Herbwood teaches generating a second input feature vector based on the augmented image data (Herbwood, page 10-11, section 5: “we flatten the 7x7x512 (=25088) feature map for each region proposal and input it into the fc layer to obtain a feature vector of size 4096 through the fc layer…”, see figure in section 5 below, with annotation); and applying the second input feature vector to the second trained neural network model to generate a second output feature vector, wherein the second output feature vector indicates the location for the distal end of the tool of interest. (Herbwood, page 11-12, section 7. Bounding box regressor: “A feature vector of size 4096 is input to an fc layer with (K+1) x 4 output units to predict the coordinates of bounding boxes for each class. It outputs bounding box coordinates adjusted for each class for one region proposal in one image”, see figure in section 5 below, with
PNG
media_image1.png
624
939
media_image1.png
Greyscale
annotation)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Chen, Zucker and Meglan by simply substituting their CNN-based model for object detection with the Fast R-CNN model that is taught by Herbwood, to make an object detection system using R-CNN model; thus, one of ordinary skilled in the art would be motivated to make the substitution since among its several aspects, the present invention recognizes there is a need to improve object detection speed of the machine learning model (Herbwood, page 1, first paragraph: “Fast R-CNN, as its name suggests, demonstrates a significant speed improvement over the existing R-CNN model.”).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
CLAIM 6
In regards to Claim 6, the combination of Chen, Zucker and Meglan teaches the method of Claim 1. In addition, the combination of Chen, Zucker and Meglan teaches CNN-based machine learning model to detect surgical instrument and distal end of surgical instrument. (Chen, FIG. 5, step 502 and 508) (Meglan, ¶ [0085-0086]: “the video system 30 may detect the object based on a convolutional neural network”); receiving a plurality of reference image data of a reference surgical video, wherein each reference image data represents a respective image frame showing a plurality of tools including the tool of interest. (Meglan, ¶ [0065]: “the networks be trained on clinical video. The anatomy seen in these videos can be complex as well as subtle and the surgical tool interaction with the anatomy equally challenging to yield the details of the interaction”, ¶ [0010]: “training images”; see FIG. 9 and 10 for different surgical tools)
The combination of Chen, Zucker and Meglan does not explicitly disclose generating, for each of the plurality of reference image data, a reference input feature vector indicating relevant features of the reference image data, and a reference output feature vector indicating a location for a bounding box around the tool of interest; and training, by associating each reference input feature vector with its respective reference output feature vector, a neural network to form the first trained neural network model.
PNG
media_image1.png
624
939
media_image1.png
Greyscale
Herbwood is in the same field of art of CNN-based object detection. Further, Herbwood teaches generating, for each of the plurality of reference image data, a reference input feature vector indicating relevant features of the reference image data (Herbwood, page 10-11, section 5: “we flatten the 7x7x512 (=25088) feature map for each region proposal and input it into the fc layer to obtain a feature vector of size 4096 through the fc layer…”, see figure in section 5 below, with annotation), and a reference output feature vector indicating a location for a bounding box around the tool of interest (Herbwood, page 11-12, section 7. Bounding box regressor: “A feature vector of size 4096 is input to an fc layer with (K+1) x 4 output units to predict the coordinates of bounding boxes for each class. It outputs bounding box coordinates adjusted for each class for one region proposal in one image”, see figure in section 5 below, with annotation);
and training, by associating each reference input feature vector with its respective reference output feature vector, a neural network to form the first trained neural network model. (Herbwood, see section Training Fast R-CNN, page 7-12. Page 12, subsection 8: “we train both models (classifier and bounding box regressor) simultaneously through backpropagation.”)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Chen, Zucker and Meglan by simply substituting their CNN-based model for object detection with the Fast R-CNN model that is taught by Herbwood, to make an object detection system using R-CNN model; thus, one of ordinary skilled in the art would be motivated to make the substitution since among its several aspects, the present invention recognizes there is a need to improve object detection speed of the machine learning model (Herbwood, page 1, first paragraph: “Fast R-CNN, as its name suggests, demonstrates a significant speed improvement over the existing R-CNN model.”).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
CLAIM 7
In regards to Claim 7, the combination of Chen, Zucker and Meglan teaches the method of Claim 1. In addition, the combination of Chen, Zucker and Meglan teaches CNN-based machine learning model to detect surgical instrument and distal end of surgical instrument. (Chen, FIG. 5, step 502 and 508) (Meglan, ¶ [0085-0086]: “the video system 30 may detect the object based on a convolutional neural network”); receiving a plurality of reference image data representing a plurality of respective reference images showing a plurality of respective reference surgical tools (Meglan, ¶ [0065]: “the networks be trained on clinical video. The anatomy seen in these videos can be complex as well as subtle and the surgical tool interaction with the anatomy equally challenging to yield the details of the interaction”, ¶ [0010]: “training images”; see FIG. 9 and 10 for different surgical tools) (Zucker, ¶ [0093-0094]: “The at least one target may be a reference marker, a marking on a patient anatomy, an anatomical element, an incision, a tool, an instrument, an implant, or any other object … feature recognition may be used to identify a feature of the at least one target. For example, a contour of a screw, port, tool, edge, instrument, or anatomical element may be identified in the first image. In other embodiments, an image processing algorithm may be based on artificial intelligence or machine learning. In such embodiments, a plurality of training images may be provided to the processor, and each training image may be annotated to include identifying information about a target in the image” Zucker teaches training images for each target are provided to train the algorithm for target identification, the target could be a screw, port, tool, edge, instrument, or anatomical element)
The combination of Chen, Zucker and Meglan does not explicitly disclose generating, for each of the plurality of reference image data, a reference input feature vector indicating relevant features of the reference image data, and a reference output feature vector indicating a location for a distal end of a reference surgical tool; and training, by associating each reference input feature vector with its respective reference output feature vector, a neural network to form the second trained neural network model.
Herbwood is in the same field of art of CNN-based object detection. Further, Herbwood teaches generating, for each of the plurality of reference image data, a reference input feature vector indicating relevant features of the reference image data (Herbwood, page 10-11, section 5: “we flatten the 7x7x512 (=25088) feature map for each region proposal and input it into the fc layer to obtain a feature vector of size 4096 through the fc layer…”, see figure in section 5 below, with annotation), and a reference output feature vector indicating a location for a distal end of a reference surgical tool (Herbwood, page 11-12, section 7. Bounding box regressor: “A feature vector of size 4096 is input to an fc layer with (K+1) x 4 output units to predict the coordinates of bounding boxes for each class. It outputs bounding box coordinates adjusted for each class for one region proposal in one image”, see figure in section 5 below, with annotation); and training, by associating each reference input feature vector with its respective reference output feature vector, a neural network to form the second trained neural network model. (Herbwood, see section Training Fast R-CNN, page 7-12. Page 12, subsection 8: “we train both models (classifier and bounding box regressor)
PNG
media_image1.png
624
939
media_image1.png
Greyscale
simultaneously through backpropagation.”)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Chen, Zucker and Meglan by simply substituting their CNN-based model for object detection with the Fast R-CNN model that is taught by Herbwood, to make an object detection system using R-CNN model; thus, one of ordinary skilled in the art would be motivated to make the substitution since among its several aspects, the present invention recognizes there is a need to improve object detection speed of the machine learning model (Herbwood, page 1, first paragraph: “Fast R-CNN, as its name suggests, demonstrates a significant speed improvement over the existing R-CNN model.”).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
CLAIM 12
In regards to Claim 12, the combination of Chen, Zucker and Meglan teaches the system of Claim 9. In addition, the combination of Chen, Zucker and Meglan teaches CNN-based machine learning model to detect surgical instrument and distal end of surgical instrument. (Chen, FIG. 5, step 502 and 508) (Meglan, ¶ [0085-0086]: “the video system 30 may detect the object based on a convolutional neural network”)
The combination of Chen, Zucker and Meglan does not explicitly disclose generating a first input feature vector based on the image data; and applying the first input feature vector to the first trained neural network model to generate a first output feature vector, wherein the first output feature vector indicates the location for the bounding box around the tool of interest.
PNG
media_image1.png
624
939
media_image1.png
Greyscale
Herbwood is in the same field of art of CNN-based object detection. Further, Herbwood teaches generating a first input feature vector based on the image data (Herbwood, page 10-11, section 5: “we flatten the 7x7x512 (=25088) feature map for each region proposal and input it into the fc layer to obtain a feature vector of size 4096 through the fc layer…”, see figure in section 5 below, with annotation); and applying the first input feature vector to the first trained neural network model to generate a first output feature vector, wherein the first output feature vector indicates the location for the bounding box around the tool of interest. (Herbwood, page 11-12, section 7. Bounding box regressor: “A feature vector of size 4096 is input to an fc layer with (K+1) x 4 output units to predict the coordinates of bounding boxes for each class. It outputs bounding box coordinates adjusted for each class for one region proposal in one image”, see figure in section 5 below, with annotation)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Chen, Zucker and Meglan by simply substituting their CNN-based model for object detection with the Fast R-CNN model that is taught by Herbwood, to make an object detection system using R-CNN model; thus, one of ordinary skilled in the art would be motivated to make the substitution since among its several aspects, the present invention recognizes there is a need to improve object detection speed of the machine learning model (Herbwood, page 1, first paragraph: “Fast R-CNN, as its name suggests, demonstrates a significant speed improvement over the existing R-CNN model.”).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
CLAIM 13
In regards to Claim 13, the combination of Chen, Zucker and Meglan teaches the system of Claim 9. In addition, the combination of Chen, Zucker and Meglan teaches CNN-based machine learning model to detect surgical instrument and distal end of surgical instrument. (Chen, FIG. 5, step 502 and 508) (Meglan, ¶ [0085-0086]: “the video system 30 may detect the object based on a convolutional neural network”)
The combination of Chen, Zucker and Meglan does not explicitly disclose generating a second input feature vector based on the augmented image data; and applying the second input feature vector to the second trained neural network model to generate a second output feature vector, wherein the second output feature vector indicates the location for the distal end of the tool of interest.
PNG
media_image1.png
624
939
media_image1.png
Greyscale
Herbwood is in the same field of art of CNN-based object detection. Further, Herbwood teaches generating a second input feature vector based on the augmented image data (Herbwood, page 10-11, section 5: “we flatten the 7x7x512 (=25088) feature map for each region proposal and input it into the fc layer to obtain a feature vector of size 4096 through the fc layer…”, see figure in section 5 below, with annotation); and applying the second input feature vector to the second trained neural network model to generate a second output feature vector, wherein the second output feature vector indicates the location for the distal end of the tool of interest. (Herbwood, page 11-12, section 7. Bounding box regressor: “A feature vector of size 4096 is input to an fc layer with (K+1) x 4 output units to predict the coordinates of bounding boxes for each class. It outputs bounding box coordinates adjusted for each class for one region proposal in one image”, see figure in section 5 below, with annotation)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Chen, Zucker and Meglan by simply substituting their CNN-based model for object detection with the Fast R-CNN model that is taught by Herbwood, to make an object detection system using R-CNN model; thus, one of ordinary skilled in the art would be motivated to make the substitution since among its several aspects, the present invention recognizes there is a need to improve object detection speed of the machine learning model (Herbwood, page 1, first paragraph: “Fast R-CNN, as its name suggests, demonstrates a significant speed improvement over the existing R-CNN model.”).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
CLAIM 14
In regards to Claim 14, the combination of Chen, Zucker and Meglan teaches the system of Claim 9. In addition, the combination of Chen, Zucker and Meglan teaches CNN-based machine learning model to detect surgical instrument and distal end of surgical instrument. (Chen, FIG. 5, step 502 and 508) (Meglan, ¶ [0085-0086]: “the video system 30 may detect the object based on a convolutional neural network”); receiving a plurality of reference image data of a reference surgical video, wherein each reference image data represents a respective image frame showing a plurality of tools including the tool of interest. (Meglan, ¶ [0065]: “the networks be trained on clinical video. The anatomy seen in these videos can be complex as well as subtle and the surgical tool interaction with the anatomy equally challenging to yield the details of the interaction”, ¶ [0010]: “training images”; see FIG. 9 and 10 for different surgical tools) (Zucker, ¶ [0093-0094]: “The at least one target may be a reference marker, a marking on a patient anatomy, an anatomical element, an incision, a tool, an instrument, an implant, or any other object … feature recognition may be used to identify a feature of the at least one target. For example, a contour of a screw, port, tool, edge, instrument, or anatomical element may be identified in the first image. In other embodiments, an image processing algorithm may be based on artificial intelligence or machine learning. In such embodiments, a plurality of training images may be provided to the processor, and each training image may be annotated to include identifying information about a target in the image” Zucker teaches training images for each target are provided to train the algorithm for target identification, the target could be a screw, port, tool, edge, instrument, or anatomical element)
The combination of Chen, Zucker and Meglan does not explicitly disclose generating, for each of the plurality of reference image data, a reference input feature vector indicating relevant features of the reference image data, and a reference output feature vector indicating a location for a bounding box around the tool of interest; and training, by associating each reference input feature vector with its respective reference output feature vector, a neural network to form the first trained neural network model.
Herbwood is in the same field of art of CNN-based object detection. Further, Herbwood teaches generating, for each of the plurality of reference image data, a reference input feature vector indicating relevant features of the reference image data (Herbwood, page 10-11, section 5: “we flatten the 7x7x512 (=25088) feature map for each region proposal and input it into the fc layer to obtain a feature vector of size 4096 through the fc layer…”, see figure in section 5 below, with annotation), and a reference output feature vector indicating a location for a bounding box around the tool of interest (Herbwood, page 11-12, section 7. Bounding box regressor: “A feature vector of size 4096 is input to an fc layer with (K+1) x 4 output units to predict the coordinates of bounding boxes for each class. It outputs bounding box coordinates adjusted for each class for one region proposal in one image”, see figure in
PNG
media_image1.png
624
939
media_image1.png
Greyscale
section 5 below, with annotation);
and training, by associating each reference input feature vector with its respective reference output feature vector, a neural network to form the first trained neural network model. (Herbwood, see section Training Fast R-CNN, page 7-12. Page 12, subsection 8: “we train both models (classifier and bounding box regressor) simultaneously through backpropagation.”)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Chen, Zucker and Meglan by simply substituting their CNN-based model for object detection with the Fast R-CNN model that is taught by Herbwood, to make an object detection system using R-CNN model; thus, one of ordinary skilled in the art would be motivated to make the substitution since among its several aspects, the present invention recognizes there is a need to improve object detection speed of the machine learning model (Herbwood, page 1, first paragraph: “Fast R-CNN, as its name suggests, demonstrates a significant speed improvement over the existing R-CNN model.”).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
CLAIM 15
In regards to Claim 15, the combination of Chen, Zucker and Meglan teaches the system of Claim 9. In addition, the combination of Chen, Zucker and Meglan teaches CNN-based machine learning model to detect surgical instrument and distal end of surgical instrument. (Chen, FIG. 5, step 502 and 508) (Zucker, page 8, first paragraph: “CNN architectures for object detection have been widely used with many deep learning frameworks [32,37–39]”; page 11: “Faster R-CNN: Towards real-time object detection with region proposal networks”); receiving a plurality of reference image data representing a plurality of respective reference images showing a plurality of respective reference surgical tools (Zucker, ¶ [0093-0094]: “The at least one target may be a reference marker, a marking on a patient anatomy, an anatomical element, an incision, a tool, an instrument, an implant, or any other object … feature recognition may be used to identify a feature of the at least one target. For example, a contour of a screw, port, tool, edge, instrument, or anatomical element may be identified in the first image. In other embodiments, an image processing algorithm may be based on artificial intelligence or machine learning. In such embodiments, a plurality of training images may be provided to the processor, and each training image may be annotated to include identifying information about a target in the image” Zucker teaches training images for each target are provided to train the algorithm for target identification, the target could be a screw, port, tool, edge, instrument, or anatomical element)
The combination of Chen, Zucker and Meglan does not explicitly disclose generating, for each of the plurality of reference image data, a reference input feature vector indicating relevant features of the reference image data, and a reference output feature vector indicating a location for a distal end of a reference surgical tool; and training, by associating each reference input feature vector with its respective reference output feature vector, a neural network to form the second trained neural network model.
Herbwood is in the same field of art of CNN-based object detection. Further, Herbwood teaches generating, for each of the plurality of reference image data, a reference input feature vector indicating relevant features of the reference image data (Herbwood, page 10-11, section 5: “we flatten the 7x7x512 (=25088) feature map for each region proposal and input it into the fc layer to obtain a feature vector of size 4096 through the fc layer…”, see figure in section 5 below, with annotation), and a reference output feature vector indicating a location for a distal end of a reference surgical tool (Herbwood, page 11-12, section 7. Bounding box regressor: “A feature vector of size 4096 is input to an fc layer with (K+1) x 4 output units to predict the coordinates of bounding boxes for each class. It outputs bounding box coordinates adjusted for each class for one region proposal in one image”, see figure in
PNG
media_image1.png
624
939
media_image1.png
Greyscale
section 5 below, with annotation); and
training, by associating each reference input feature vector with its respective reference output feature vector, a neural network to form the second trained neural network model. (Herbwood, see section Training Fast R-CNN, page 7-12. Page 12, subsection 8: “we train both models (classifier and bounding box regressor) simultaneously through backpropagation.”)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Chen, Zucker and Meglan by simply substituting their CNN-based model for object detection with the Fast R-CNN model that is taught by Herbwood, to make an object detection system using R-CNN model; thus, one of ordinary skilled in the art would be motivated to make the substitution since among its several aspects, the present invention recognizes there is a need to improve object detection speed of the machine learning model (Herbwood, page 1, first paragraph: “Fast R-CNN, as its name suggests, demonstrates a significant speed improvement over the existing R-CNN model.”).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim(s) 18 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen, in view of Zucker in view of Venka.
CLAIM 18
In regards to Claim 18, Chen teaches a method of tracking a surgical tool in real-time (Chen, ¶ [0003]: “With further regard to the system…determining the position of the tip of the surgical tool and determining the orientation of the surgical tool in real-time”)
Chen does not explicitly disclose a digital surgical microscope (DSM) connected to a robotic arm.
Zucker is in the same field of art of tracking surgical instruments. Further, Zucker teaches disclose a digital surgical microscope (DSM) (Zucker, ¶ [0072]: “The imaging device 132 may be operable to image anatomical feature(s) (e.g., a bone, veins, tissue, etc.) and/or other aspects of patient anatomy, a target, a tracking marker, and/or any surgical instruments or tools within the field of view … The imaging device 132 may be or comprise … a microscope”) connected to a robotic arm (Zucker, ¶ [0072-0073]: “two imaging devices 132 may be supported on a single robotic arm 148 to capture images from different angles at the same time”; ¶ [0014, 0019 and 0029]: “a first imaging device secured to the first robotic arm”).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Chen by incorporating a robotic arm with attached microscope camera that is taught by Zucker, to make a surgical tool tracking system that has camera supported by robotic arm; thus, one of ordinary skilled in the art would be motivated to combine the references since among its several aspects, the present invention recognizes there is a need to improve target identification accuracy using robotic arm (Zucker, ¶ [0059-0060]: “a robotic arm may automatically change its position “around” a target to obtain more angles and improve an accuracy of an identified target position.”). In addition, Chen mentions his invention can be used with robotic arm to support camera and surgical tools. (Chen, ¶ [0053]: “Implementations may be used with robotic arms during surgery (e.g., camera holder, tool holder, etc.).”)
The combination of Chen and Zucker then teaches the method comprising:
receiving, from one or more image sensors associated with the digital surgical microscope (Zucker, ¶ [0072]: “The imaging device 132 may be operable to image anatomical feature(s) (e.g., a bone, veins, tissue, etc.) and/or other aspects of patient anatomy, a target, a tracking marker, and/or any surgical instruments or tools within the field of view … The imaging device 132 may be or comprise … a microscope”), by a computing device having a processor (Chen, ¶ [0002]: “a system includes one or more processors”), image data of a first segment of a medical procedure, wherein the one or more image sensors are focused towards a first position of a surgical area of the medical procedure (Chen, ¶ [0017]: “a system receives an image frame from a camera…, the image frame may be one of a series of image frames of a video stream. The system detects a surgical instrument or tool in the at least one of the image frames”);
causing, in real-time, the DSM to track the tool (Chen, ¶ [0028]: “… the system tracks the movement of the surgical tool tip.”) by at least transmitting one or more robotic arm controls to the robotic arm. (Zucker, ¶ [0059-0064]: “a robotic system that contains two or more robotic arms. Each robotic arm may be mounted with a camera or otherwise support a camera … a robotic arm may automatically change its position “around” a target to obtain more angles and improve an accuracy of an identified target position … a robotic arm may support a tool and perform an action with the tool and one or more different robotic arms may track the action with one or more cameras … (1) determining a pose, a position, or an orientation of a target; (2) tracking movement of a target; (3) adjusting a path of a tool, an instrument, or an implant based on tracked movement of a target; (4) increasing an accuracy of a pose, a position, or an orientation of a target (whether a tool, an instrument, an implant, a robotic arm, or any other object)” Zucker teaches a system with multiple robotic arms, each can be mounted with camera or surgical tool, camera-mounted arm can automatically move to identify and track a target (the target could be a tool, an instrument or other robotic arm))
The combination of Chen and Zucker does not explicitly disclose identifying, by the computing device, and based on the location of the detected tool, a central region of the medical procedure, wherein the threshold distance is from the location of the detected tool to the edge of the central region tracking, by the computing device, movement of the tool through one or more subsequent segments after the first segment; after detecting a displacement of the tool beyond a threshold distance, refocusing the image sensors towards a second position of the surgical area of the medical procedure.
Venka is in the same field of art of tracking surgical instruments. Further, Venka teaches identifying, by the computing device, and based on the location of the detected tool, a central region of the medical procedure (Venka, ¶ [0072]: “the disclosed system displays the center portion of the high-resolution video images, which typically includes the end/tip of the tool (or simply “tool tip” hereafter)”), wherein the threshold distance is from the location of the detected tool to the edge of the central region tracking (Venka, ¶ [0073]: “when the location of the tool tip is determined to be near an edge of the viewing window and about to go off-screen … ” The Examiner note Venka’s distance from center to the edge of viewing window corresponds to the “threshold distance” of the invention), by the computing device, movement of the tool through one or more subsequent segments after the first segment (Venka, ¶ [0072]: “the disclosed system can also detect tool movement (assuming a tool is already displayed on the screen) and automatically reposition the viewing window within the full-resolution endoscope view (i.e., the endoscope video images) based on the detected tool movement”); and after detecting a displacement of the tool beyond a threshold distance, refocusing the image sensors towards a second position of the surgical area of the medical procedure. (Venka, ¶ [0073-0074]: “… the system can automatically reposition the viewing window from the current location to the new ROI so that the tool tip is brought back to the center or closer to the center of the display. Moreover, after the initial repositioning of the viewing window, the system can start following the movement of the tool tip by continuously adjusting the position of the viewing window based on the movement of the tool tip”)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Chen and Zucker by incorporating method to refocus the camera on tracked surgical tool that is taught by Venka, to make a surgical instrument tracking system that can automatically repositioning the viewing window; thus, one of ordinary skilled in the art would be motivated to combine the references since among its several aspects, the present invention recognizes there is a need to improve the efficiency of the surgeon by removing the manual camera adjustment (Venka, ¶ [0073]: “ Note that when the disclosed function of automatically adjusting the location of the viewing window is engaged, the surgeon no longer needs to manually change the location of the viewing window to follow the movement of the tool tip”).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
CLAIM 20
In regards to Claim 20, the combination of Chen, Zucker and Venka teaches the method of Claim 18. In addition, the combination of Chen, Zucker and Venka teaches applying, by the computing device, a second neural network to the image data of the one or more subsequent segments, to detect a user intent (Venka, ¶ [0083-0084]: “the system uses one or more deep-learning models to determine the location of user's gaze on the display by analyzing the captured images of the user's eyes and head. After determining the initial location of the user's gaze, the system starts tracking a movement of the user's gaze from the initial location, e.g., by using a deep-learning-based gaze-tracking technique”); and altering, based on the user intent, one or more settings of the image sensors. (Venka, ¶ [0083-0084]: “the system automatically repositions the viewing window from the current location to the new ROI to keep the user's gaze near the center of the display (step 612).”)
CLAIM 19
Claim(s) 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen in view of Zucker in view of Venka, and further in view of Herbwood.
In regards to Claim 19, the combination of Chen, Zucker and Venka teaches the method of claim 18. In addition, the combination of Chen, Zucker and Venka teaches receiving, by the computing device, a plurality of reference image data showing a plurality of reference tool markings (Zucker, ¶ [0093-0094]: “The at least one target may be a reference marker, a marking on a patient anatomy, an anatomical element, an incision, a tool, an instrument, an implant, or any other object … feature recognition may be used to identify a feature of the at least one target. For example, a contour of a screw, port, tool, edge, instrument, or anatomical element may be identified in the first image. In other embodiments, an image processing algorithm may be based on artificial intelligence or machine learning. In such embodiments, a plurality of training images may be provided to the processor, and each training image may be annotated to include identifying information about a target in the image” Zucker teaches training images for each target are provided to train the algorithm for target identification, the target could be a reference marker, screw, port, tool, edge, instrument, or anatomical element);
The combination of Chen, Zucker and Venka does not explicitly disclose generating, by the computing device, a plurality of feature vectors corresponding to the plurality of reference image data showing the plurality of reference tool markings, wherein each feature vector is generated via a convolution of the respective reference image data; associating, by the computing device, each of the plurality of feature vectors with their respective tool marking;
Herbwood is in the same field of art of CNN-based object tracking model. Further, Herbwood teaches generating, by the computing device, a plurality of feature vectors corresponding to the plurality of reference image data showing the plurality of reference tool markings, wherein each feature vector is generated via a convolution of the respective reference image data (Herbwood, page 10-11, section 5 and page 11-12, section 7. Herbwood teaches obtaining feature vector by fc layer, feed it into bounding box regressor to generate feature vector with bounding box information; see annotated figure below); associating, by the computing device, each of the plurality of feature vectors with their respective tool marking (see classifier head below, objects are classified based on their feature);
PNG
media_image1.png
624
939
media_image1.png
Greyscale
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Chen, Zucker and Venka by simply substituting their CNN-based model for object detection with the Fast R-CNN model that is taught by Herbwood, to make an object detection system using R-CNN model; thus, one of ordinary skilled in the art would be motivated to make the substitution since among its several aspects, the present invention recognizes there is a need to improve object detection speed of the machine learning model (Herbwood, page 1, first paragraph: “Fast R-CNN, as its name suggests, demonstrates a significant speed improvement over the existing R-CNN model.”).
The combination of Chen, Zucker, Venka and Herbwood then teaches training, by the computing device, the neural network, wherein the detecting the tool is based on a tool marking of the tools. (Geiger, ¶ [0059-0060]. Geiger teaches training model to detect surgical instruments based on their predefined markers) (Herbwood, see section Training Fast R-CNN, page 7-12. Page 12, subsection 8: “we train both models (classifier and bounding box regressor) simultaneously through backpropagation.”)
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NHUT HUY (JEREMY) PHAM whose telephone number is (703)756-5797. The examiner can normally be reached Mo - Fr. 8:30am - 6pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, O'Neal Mistry can be reached on (313)446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
NHUT HUY (JEREMY) PHAMExaminerArt Unit 2674
/ONEAL R MISTRY/Supervisory Patent Examiner, Art Unit 2674
Prosecution Insights