DETAILED ACTION
This Office Action is sent in response to Applicant’s Communication received 10/17/2025 for application number 18/479,813.
Claims 2-8, 10-19, and 21-23 are pending.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 2-6 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Mustafa et al. (US 2019/0231459 A1).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 2-8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Mustafa et al. (US 2019/0231459 A1) in view of Itkowitz et al. (US 2020/0163731 A1).
In reference to claim 2, Mustafa discloses a method (para. 0006) performed by a robotic surgical system that includes a robotic arm with an end effector (arms with surgical tools, fig. 1B, para. 0034) and a display (display, fig. 1C, para. 0038) for displaying a graphical user interface (GUI) (see GUI in fig. 3), the method comprising: displaying, on a display, a graphical user interface (GUI) that includes endoscopic images captured by an endoscopic camera and a floating user interface (UI) item overlaid on top of the endoscopic images, wherein the endoscopic images include a surgical site that has a surgical instrument with an end effector (image is endoscopic camera image of surgical site, para. 0037); using an input device to control movement of the surgical instrument with the end effector while a housing of the input device is in hand of a user and while the robotic surgical system is in teleoperation (user can use control device 36, see fig. 2, to control the movement of surgical tools in the robot’s arms in teleoperation, para. 0039-40, 0046); determining, based on the endoscopic images, that the surgical instrument is moving within a field of view of the endoscopic camera; moving the floating UI item to mirror the movement of the surgical instrument such that the floating UI item maintains a distance from the surgical instrument in the endoscopic images (floating menu is displayed proximate to the tool, para. 0042-43); switching the input device from controlling the movement of the robotic arm to controlling a user interface (UI) item in the GUI displayed on the display (clutch is activating which switches from controlling arm to controlling GUI menu, para. 0046); and receiving, via the input device, an input to control the UI item, wherein the input includes a rotational movement of the housing of the input device that is received about a same axis as the end effector (user can move selector through circular menu via rotating the input device by rotating around roll axis, para. 0042-50; moving about the roll axis maintains the orientation of the arm and input device, para. 0048, which is to say the rotational movement is about the same axis as the end effector).
However, Mustafa does not explicitly teach determining, based on the endoscopic images, that the surgical instrument is moving within a field of view of the endoscopic camera.
Itkowitz teaches determining, based on the endoscopic images, that the surgical instrument is moving within a field of view of the endoscopic camera (different indicators are displayed when tool is on-screen and off-screen, para. 0059-60, so a determination is made that the tool is moving in and out of view).
It would have been obvious to one of ordinary skill in art, having the teachings of Mustafa and Itkowitz before the earliest effective filing date, to modify the interface as disclosed by Mustafa to include the off-screen indicator as taught by Itkowitz.
One of ordinary skill in the art would have been motivated to modify the interface of Mustafa to include the off-screen indicator of Itkowitz because it can help users better switch between different surgical instruments (Itkowitz, para. 0004-05).
In reference to claim 3, Mustafa discloses the method of claim 2, wherein the rotational movement is received about the same axis as the end effector such that the input device remains aligned with the end effector while the input device controls the UI item (moving about the roll axis maintains the orientation of the arm and input device, para. 0048).
In reference to claim 4, Mustafa discloses the method of claim 2, wherein the input is a first input, wherein the method further comprises receiving, via the input device, a second input that is different than the first input, wherein the switching of the input device from controlling the movement to controlling the UI item occurs and is maintained while the second input is received via the input device (while holding clutch, user can control UI, para. 0046-48).
In reference to claim 5, Mustafa teaches the method of claim 4, wherein the input device comprises a clutch mechanism, wherein the method further comprises pausing robotic arm movement while the second input is received via the clutch mechanism of the input device (while holding clutch, user can control UI and movement of arm is paused, para. 0046-48).
In reference to claim 6, Mustafa discloses the method of claim 2, wherein the UI item comprises a menu with a plurality of selectable items that are arranged in a ring, wherein the rotational movement of the housing allows the user to navigate through the plurality of selectable items (see figs. 3, 4, and 6: menu is a plurality of selectable items in ring that user rotates input device to select, para. 0041-54).
In reference to claim 7, Mustafa teaches the method of claim 2, wherein the GUI includes a display panel that includes endoscopic images of a surgical site captured by an endoscopic camera during a surgical procedure (image is endoscopic camera image of surgical site, para. 0037).
However, Mustafa does not explicitly teach wherein the GUI further includes a floating UI item at an edge of the display panel that indicates that a surgical instrument is out of a field of view of the endoscopic camera and is at a location of the surgical site based on the edge at which the floating UI item is located.
Itkowitz teaches wherein the GUI further includes a floating UI item at an edge of the display panel that indicates that a surgical instrument is out of a field of view of the endoscopic camera and is at a location of the surgical site based on the edge at which the floating UI item is located (off-screen indicator at location to indicate the instrument is off-screen, para. 0059).
It would have been obvious to one of ordinary skill in art, having the teachings of Mustafa and Itkowitz before the earliest effective filing date, to modify the interface as disclosed by Mustafa to include the off-screen indicator as taught by Itkowitz.
One of ordinary skill in the art would have been motivated to modify the interface of Mustafa to include the off-screen indicator of Itkowitz because it can help users better switch between different surgical instruments (Itkowitz, para. 0004-05).
In reference to claim 8, Itkowitz further teaches the method of claim 8, wherein the floating UI item comprises a line that indicates an orientation of the endoscopic camera (line indicates orientation of camera relative to tool, fig. 6A, para. 0059).
In reference to claim 9, Mustafa teaches the method of claim 2, wherein the GUI comprises endoscopic images of a surgical site that includes a surgical instrument captured by an endoscopic camera (image is endoscopic camera image of surgical site, para. 0037), and a floating UI item overlaid on top of the endoscopic images, wherein the method further comprises: moving the floating UI item to mirror the movement of the surgical instrument such that the floating UI item maintains a distance from the surgical instrument in the endoscopic images (floating menu is displayed proximate to the tool, para. 0042-43).
However, Mustafa does not explicitly teach determining, based on the endoscopic images, that the surgical instrument is moving within a field of view of the endoscopic camera.
Itkowitz teaches determining, based on the endoscopic images, that the surgical instrument is moving within a field of view of the endoscopic camera (different indicators are displayed when tool is on-screen and off-screen, para. 0059-60, so a determination is made that the tool is moving in and out of view).
It would have been obvious to one of ordinary skill in art, having the teachings of Mustafa and Itkowitz before the earliest effective filing date, to modify the interface as disclosed by Mustafa to include the off-screen indicator as taught by Itkowitz.
One of ordinary skill in the art would have been motivated to modify the interface of Mustafa to include the off-screen indicator of Itkowitz because it can help users better switch between different surgical instruments (Itkowitz, para. 0004-05).
Allowable Subject Matter
Claims 12-19 and 22-23 allowed.
Claims 10-11 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: Please see Response to Arguments below.
Response to Arguments
Applicant’s arguments with respect to the 102/103 rejection of independent claims 12 and 22 have been fully considered and are persuasive. Specifically, Applicant argues that the combination of Mustafa and Azizian would not result in determining if a surgical task is completed based on endoscopic images, and in response, switching the input device from controlling a surgical instrument to controlling a UI item. After further consideration, the Examiner agrees. Mustafa teaches switching between controlling a surgical instrument to controlling a UI item using a clutch pedal, and Azizian teaches determining phases of a surgical procedure are complete. However, there is no teaching, suggestion, or other motivation for making the change in the prior art, nor is there another rationale for making the chainge. Therefore, the 102/103 rejection of claims 12 and 22 has been withdrawn.
Applicant's arguments with respect to the 103 rejection of independent claim 2 have been fully considered but they are not persuasive. Specifically, Applicant argues that Mustafa does not teach the menu mirroring the movement of a surgical instrument to maintain a distance from the instrument. Applicant’s argument seems to be that the menu is displayed at a fixed, default location. However, this interpretation of Mustafa does not make sense: Mustafa states that, “The graphical menu 110 is displayed proximate to image of the distal end of the medical instrument 26b so that the surgeon can easily tell that the co-located graphical menu 110 pertains to the medical instrument 26b,” (para. 0042), and corresponding fig. 3 shows a menu near (but not overlapping) the instrument. Mustafa further states that menu is moved to avoid obscuring the instrument (para. 0043). Together, these paragraphs and the figures teach that the menu follows, but does not obscure, the surgical instrument as it moves in the image. Otherwise, if it did were simply displayed at a default location, a user would not be able to, “easily tell that the co-located graphical menu 110 pertains to the medical instrument,” and the menu would not move to avoid obscuring the jaws of the instrument in response to the instrument being relatively small in the image.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Andrew T. Chiusano whose telephone number is (571)272-5231. The examiner can normally be reached M-F, 10am-6pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tamara Kyle can be reached at 571-272-4241. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANDREW T CHIUSANO/Primary Examiner, Art Unit 2144