DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed 08/07/2026 have been fully considered but they are not persuasive.
With respect to Applicant’s request to call Applicant’s representative for an interview before sending next action other than allowance (page 9), Examiner called and had an interview with attorney Travis J. Hill on August 28, 2026.
Applicant states claim 1 is amended to recite, in part:
display a graphical interface that includes the image and a visual overlay overlaid on the detected anatomical feature to indicate a location of the detected anatomical feature in the image;
determine whether the detected anatomical feature is selected as a target anatomical feature based at least in part on a position of the visual overlay relative to the image; and
track the anatomical feature responsive to determining that the detected anatomical feature is selected as a target anatomical feature.
Then, Applicant argues the Examiner has not cited to any language in any references that would disclose or suggest such features because Applicant’s specification describes that when an anatomical feature “is identified in image data, the scope camera image may be annotated/overlaid with various visual features that identity the features in the video image for the user, such as bounding boxes, highlighting, text, arrows, masking, or the like.” Specification, paragraph [0085]. Advantageously, the system provides “the ability to determine a physician’s/user intent regarding selection of a desired target anatomical feature among features…” (Specification, paragraphs [0131]-[0133]. (see page 8).
In response, according to MPEP 2111.04 “"reading a claim in light of the specification, to thereby interpret limitations explicitly recited in the claim, is a quite different thing from ‘reading limitations of the specification into a claim,’ to thereby narrow the scope of the claim by implicitly adding disclosed limitations which have no express basis in the claim."
In this case, the claims do not recite “when an anatomical feature is identified in image data, the scope camera image may be annotated/overlaid with various visual features that identity the features in the video image for the user, such as bounding boxes, highlighting, text, arrows, masking, or the like….The system provides the ability to determine a physician’s/user intent regarding selection of a desired target anatomical feature among features…”. Instead, amended claim 1 recites “display a graphical interface that includes the image and a visual overlay overlaid on the detected anatomical feature to indicate a location of the detected anatomical feature in the image;
determine whether the detected anatomical feature is selected as a target anatomical feature based at least in part on a position of the visual overlay relative to the image; and
track the anatomical feature responsive to determining that the detected anatomical feature is selected as a target anatomical feature.
The limitation “display a graphical interface that include image and a visual overlay overlaid on the detected anatomical feature to indicate a location of the detected anatomical feature in the image” is read on Ayvali’s disclosure of display a graphical interface that includes the image from the endoscope and visual features/icons overlain on or over the target anatomical feature of interest, such as a target papilla or other anatomical feature to indicate a location of the detected target anatomical feature in the image – see include, but are not limited to, figures 7, 11-12B, paragraphs 0162-0164, 0167).
The limitation “determine whether the detected anatomical feature is selected as a target anatomical feature based at least in part on a position of the visual overlay relative to the image” is read on Ayvali’s disclosure of either operator or the system determine whether the detected anatomical feature (e.g., papilla 579) is selected (tagged/registered/designated previously) as a target anatomical feature based at least in part on a position of the visual overlay such as icons or features 1150 relative to the image captured by the endoscope – see include, but are not limited to, figures 5-1, 6-1, 7, 11-12b, paragraphs 0059, 0090, 0118-0120, 0123, 0131, 0132, 0162-0164, 0167); and
limitation “track the anatomical feature responsive to determining that the detected anatomical feature is selected as a target anatomical feature” is read on Ayvali’s disclosure of tracking the anatomical feature (e.g., papilla) responsive to determining that the anatomical feature detected in captured image(s) is designated/registered or recorded or tagged as a target anatomical feature– see include, but are not limited to, figures 7, 11-12b, paragraphs 0121, 0123, 0128-0129, 0132-0133, 0136, 0148, 0162-0163).
For the reasons given above, rejection of claims 1-20 are discussed below.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-4 and 6-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Ayvali et al. (US 20210298590).
Regarding claim 1, Ayvali discloses a robotic system (robotic medical system – figure 1) comprising:
a robotic manipulator configured to manipulate an instrument having a camera associated therewith (robotic arms 12 comprises instrument device manipulator (IDM) configured to control a medical instrument/scope 32 having a camera associated therewith – see for example, figures 1-2, 7, 9, paragraphs 0015, 0063, 0073, 0085); and
control circuitry communicatively coupled to the robotic manipulator (control circuitry 202 and/or 340 coupled to the robotic arm with IDM – see figures 2-3, paragraphs 0015-0016, 0063, 0065), the control circuitry configured to:
receive an image depicting a field-of-view (FOV) of the camera associated with the instrument (receive an image depicting a field of view (FOV) captured by the camera associated with the instrument/scope – see for example, figures 12a-12b, 7, paragraphs 0063-0064, 0074, 0163, 0188);
detect an anatomical feature in the image (detect/localize an anatomical feature in the image – see for example, figure 5 (step 506), figures 7, 12a-12b, 18, paragraphs 0148, 0202, 0206);
display a graphical interface that includes the image and a visual overlay overlaid on the detected anatomical feature to indicate a location of the detected anatomical feature in the image (display a graphical interface that include the image and a visual overlay such as an icon, box, etc. over the detected/designed anatomical feature to indicate a location of the detected anatomical feature in the image – see for example, figures 7, 12a-12b, 14-1, 14-2, 18, paragraphs 0093, 0132, 0162-0164, 0167, 0187 and discussion in “response to argument” above);
determine whether the detected anatomical feature is selected as a target anatomical feature based at least in part on a position of the visual overlay relative to the image (determine whether the detected target anatomical feature (e.g., papilla) is designated/tagged/registered/recorded as target anatomical feature based on at least in part on a position of the visual overlay with indicator relative to the image - see for example, figures 7, 12a-12b, 14-1, 14-2, 18, paragraphs 0148, 0162, 0167, 0187 and discussion in “response to argument” above); and
track the anatomical feature responsive to determining that the detected anatomical feature is selected as a target anatomical feature (track the anatomical feature responsive to determining that the target anatomical feature is designated/recorded/registered/tagged as a target anatomical feature – see include, but are not limited to, figures 7, 12a-12b, 14-1, 14-2, 18, paragraphs 0112, 0128-0130, 0143, 0190 and discussion in “response to arguments” above).
Regarding claim 2, Ayvali discloses robotic system of claim 1, wherein the anatomical feature is detected based on a pretrained neural network (the anatomical feature is detected using a neural network or based on a pre-operative model data or computer models, machine learning processes – see include, but are not limited to, paragraphs 0080, 0084, 0087, 0141, 0202, 0204).
Regarding claim 3, Ayvali discloses the robotic system of claim 2, wherein the neural network is configured to detect the anatomical feature in the image based at least in part on position data indicating a position of the instrument, user input for controlling the instrument, or robotic command data that causes the robotic manipulator to manipulate the instrument (based at least in part on position data indicating a position of the instrument/scope, user input for controlling the instrument, or robotic command data that causes the arm with IDM to manipulate the instrument/scope – see include, but are not limited to, figures 7-9, 11-14-2, paragraphs 0063, 0073-0074, 0079-0080).
Regarding claim 4, Ayvali discloses the robotic system of claim 1, wherein the visual overlay comprises at least one of a bounding box, a binary mask, or an outline of the anatomical feature (e.g., bounding box, mask, or an outline of the anatomical – see include, but are not limited to, figures 11-12B, 14-1, 14-2, 18, paragraphs 0008, 0173).
Regarding claim 6, Ayvali discloses the robotic system of claim 1, wherein the determining of whether the detected anatomical feature is selected as a target anatomical feature comprises determining whether the visual overlay remains centered in relation to the FOV of the camera for at least a threshold duration (determining whether the visual overlay or icon maintains in the center in relation to the FOV of the camera for at least period of time or duration or cycle or breath-hold or period the overlay/icon is centered over the presentation of the target anatomical feature- figures 12A-12B, paragraphs 0008, 0015-0016, 0093, 0148, 0190, 0145-0146).
Regarding claim 7, Ayvali discloses the robotic system of claim 1, wherein the determining of whether the detected anatomical feature is selected as a target anatomical feature comprises:
receiving user input associated with a region of the graphical interface (receiving user/operator input associated with a region of the graphical interface from operator 5– see include, but are not limited to, figures 7, 9, 11-12b, 14a-14b, paragraphs 0074, 0162); and
determining whether the region coincides with the position of the visual overlay (determining whether the region contacts or is in the area or aligns with the position of the visual overlay see include, but are not limited to, figures 7, 9, 11-12b, 14a-14b, paragraphs 0074, 0162-0164).
Regarding claim 8, Ayvali discloses the robotic system of claim 1, wherein the determining of whether the detected anatomical feature is selected as a target anatomical feature comprises:
receiving user input (receiving user/operator input– see include, but are not limited to, figures 7, 9, 11-12b, 14a-14b, paragraphs 0074, 0162); and
determining whether the visual overlay is centered in the image responsive to receiving the user input (determining whether the visual overlay (with icons/features) is centered in the image based on receiving the input- see include, but are not limited to, figures 7, 9, 11-12b, 14a-14b, paragraphs 0163-0164).
Regarding claim 9, Ayvali discloses the robotic system of claim 1, wherein the tracking of the anatomical feature comprises:
receiving user input via an input mechanism for controlling the robotic manipulator to manipulate the instrument (receiving user input from an control mechanism for controlling the robotic manipulator of arm 12 to manipulate the instrument/scope – see figures 1, 9, paragraphs 0078-0079);
determining whether the user input causes the instrument to move in a direction away from the anatomical feature (determining whether the user input causes the instruction/scope to move in a direction away/retrack away from the anatomical feature – see include, but are not limited to, figure 7 (step 735), paragraphs 0006, 0008, 0170); and
providing haptic feedback via the input mechanism responsive to determining that the user input causes the instrument to move in a direction away from the anatomical feature (providing haptic feedback to the user via the control mechanism with message, visual feedback or audio feedback in response to direction of the instrument – see for example, paragraph 0079).
Regarding claim 10, Ayvali discloses the robotic system of claim 1, wherein the tracking of the anatomical feature comprises preventing the robotic manipulator from manipulating the instrument in a direction away from the anatomical feature (tracking to prevent/reduce the robotic manipulator from manipulating the instrument/scope to be parked too far away from the target anatomical feature - see for example, paragraphs 0167, 0171)
Regarding claim 11, Ayvali discloses the robotic system of claim 1, wherein the tracking of the anatomical feature comprises displaying an indicator on the graphical interface directing a user to move the instrument in a direction of the anatomical feature (displaying indicator such as icon/bounding box on the graphical interface directly a user to move the instrument in a direction of the anatomical feature – see include, but are not limited to, paragraphs 0171, 0173, figures 11-12b) .
Regarding claim 12, Ayvali discloses the robotic system of claim 1, wherein the tracking of the anatomical feature comprises:
displaying a first reticle in the graphical interface that is centered on the visual overlay (displaying an icon/features include crosshairs-style features which indicate a center point in the graphical interface that is centered on the visual overlay – see include, but are not limited to, figures 11-12b, paragraphs 0015, 0164);
displaying a second reticle in the graphical interface that is centered in relation to the FOV of the camera – see include, but are not limited to, figures 11-12b, paragraphs 0015, 0164); and
providing guidance via the graphical interface for manipulating the instrument so that the first reticle is aligned with the second reticle (providing a guidance via the graphical interface for manipulating the instrument/scope so that the first icon/feature/crosshair is aligned/positioned with the second features see include, but are not limited to, figures 11-13, paragraphs 0015, 0050, 0136, 0161-0165).
Regarding claim 13, Ayvali discloses the robotic system of claim 1, wherein the tracking of the anatomical feature comprises:
causing the robotic manipulator to manipulate the instrument in a series of poses (manipulate/control the instrument/scope in a series of positions/locations/orientations – see include, but are not limited to, figures 1, 7, 9, 11-15, paragraphs 0052, 0063, 0098, 0101) ;
capturing, via the camera, a series of images associated with the series of poses, respectively (capturing, via the camera, images associated with series of orientations/movements/positions – see include, but are not limited to, figures 1, 7, 9, 11-15, paragraphs 0052, 0063, 0098, 0101);
detecting the anatomical feature in each image of the series of images (detecting/tracking/identifying the anatomical feature in each image of the images – see include, but are not limited to, figures 7, 11-15, paragraphs 0143, 0148, 0202); and
determining a three-dimensional position of the anatomical feature based on positions of the anatomical feature in each image of the series of images (determining a 3D position of the anatomical features based on positions of the anatomical feature in each image - see include, but are not limited to, paragraphs 0089, 0165, 0190, 0202-0206, figures 18) .
Regarding claim 14, Ayvali discloses the robotic system of claim 13, wherein the control circuitry is further configured to provide guidance via the graphical interface for controlling the robotic manipulator to manipulate the instrument in the series of poses (e.g., scope guidance interface for controlling the robotic manipular to manipulate/direct the instrument in the series of movements/orientations - see include, but are not limited to, figures 9, 11-15, paragraphs 0091, 0093, 0132, 0152) .
Regarding claim 15, Ayvali discloses the robotic system of claim 14, wherein the guidance includes instructions to maintain the instrument in each pose of the series of poses for a duration associated with a respiration cycle (e.g., breath-hold, pulmonary cycle – see include, but are not limited to, paragraphs 0145-0146, 0072).
Regarding claim 16, Ayvali discloses the robotic system of claim 14, wherein the guidance includes a third-person point of view (POV) depicting the instrument in its current pose and further depicting the instrument in a next pose following the current pose in the series of poses (point in view of different angle/coordination depicting the instrument in its current position/point and further depicting the instrument in next position/point in the series of positions/locations of the path/trajectory/projection – see include, but are not limited to, figures 7, 9-15, 18, paragraphs 0070, 0090-0092), .
Regarding claim 17, Ayvali discloses the robotic system of claim 13, wherein the control circuitry causes the robotic manipulator to manipulate the instrument in the series of poses without user input (manipulate or control the instrument in the series of poses/position without using electronic user control or without author input or prompting or manipulate the instrument to different positions/poses automatically – paragraphs 0081, 0111, 0220).
Regarding claim 18, Ayvali discloses the robotic system of claim 17, wherein the causing of the robotic manipulator to manipulate the instrument in the series of poses without user input comprises: determining a difference between a current pose of the instrument and a next pose following the current pose in the series of poses (determining the difference between current location/point/pose of instrument and next pose/position/point based on distance, trajectory/projection – see include, but are not limited to, figures 7, 9-15, 18, paragraphs 0070, 0090-0092, 0168) ; and
causing the robotic manipulator to manipulate the instrument to the next pose based on the difference between the current pose and the next pose (see include, but are not limited to, figures 7, 9-15, 18, paragraphs 0070, 0090-0092, 0168-0171).
Regarding claim 19, limitations of a method as claimed that correspond to the limitations of the robotic system of claim 1 are analyzed as discussed in the rejection of claim 1. In particular, Ayvali discloses the method of target localization, comprising:
receiving an image depicting a field-of-view (FOV) of a camera associated with an instrument coup led to a robotic manipulator;
detecting an anatomical feature in the image;
displaying a graphical interface that includes the image and a visual overlay overlaid on the detected anatomical feature to identify the detected anatomical feature in the image;
determining whether the detected anatomical feature is selected as a target anatomical feature based at least in part on a position of the visual overlay relative to the image; and
tracking the anatomical feature responsive to determining that the detected anatomical feature is selected as a target anatomical feature (see similar explanation in the rejection of claim 1 above).
Regarding claim 20, limitations of a controller for a robotic system that correspond to the limitations of robotic system in claim 1 are analyzed as discussed in the rejection of claim 1. In particular, Ayvali discloses a controller for a robotic system, comprising:
a processing system (corresponding to “control circuitry” recited in claim 1 and discussed above); and
a memory storing instructions (computer-readable medium storing – see paragraphs 0065-0066) that, when executed by the processing system, cause the controller to:
receive an image depicting a field-of-view (FOV) of a camera associated with an instrument coupled to a robotic manipulator;
detect an anatomical feature in the image;
display a graphical interface that includes the image and a visual overlay overlaid on the detected anatomical feature to identify the detected anatomical feature in the image;
determine whether the detected anatomical feature is selected as a target anatomical feature based at least in part on a position of the visual overlay relative to the image; and
track the anatomical feature responsive to determining that the detected anatomical feature is selected as a target anatomical feature (see similar discussion in the rejection of claim 1 and include, but are not limited to, figures 2-3, 11-14-2, 18).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 5 are rejected under 35 U.S.C. 103 as being unpatentable over Ayvali et al. (US 20210298590) as applied to claim 4 and further in view of Yamada (US 20120114240).
Regarding claim 5, Ayvali discloses the robotic system of claim 4, comprising pixels of mask/overlay in the image and machine learning may be utilized to classify images and determine position information based on the size of features (pixels) in such images (see for example, figures 11-12B, 14-1, 14-2, 8, paragraphs 0173, 0204). However, Ayvali does not explicitly disclose each pixel of the binary mask has a confidence value indicating a confidence in the pixel being correctly classified.
Yamada discloses each pixel of a binary mask has a confidence value indicating confidence in the pixel being correctly classified (each pixel of a binary mask has pixel value of pixel set indicating confidence/probability in the pixel being correctly classified – see include, but are not limited to, figures 5-6, 8-9, paragraphs 0019-0021, 0102).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Ayvali with the teaching of each pixel of a binary mask has a confidence value indicating confidence in the pixel being correctly classified as taught by Yamada in order to yield predictable result of increasing the speed of a process of generating a binary mask image made up of an object region and a background region from an input image (paragraph 0013).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Azizian et al. (US 20200253673) discloses methods and systems for projecting an endoscopic image to a three-dimensional volume including overlaying/superimposing over the endoscopic image (see figures 3-6).
Cantrall et al. (US 11625825) discloses a marker indicative of the location of the area of interest is superimposed (see for example, figures 12-13).
Hod et al. (US 20240350204) discloses apparatus and method to overlay information on endoscope image (see figures 7, 9-10).
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AN SON P HUYNH whose telephone number is (571)272-7295. The examiner can normally be reached 9:00 am-6:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ANH TUAN T. NGUYEN can be reached at 571-272-4963. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AN SON P HUYNH/Primary Examiner, Art Unit 2426
August 28, 2026
Prosecution Insights