DETAILED ACTION
This office action is responsive to applicant’s communication filed 01/08/2026.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Applicant claims the benefit of US Provisional Application No. 63/119,549, filed 11/30/2020. Claims 1-10 and 32-41 have been afforded the benefit of this filing date.
Response to Arguments
Applicant’s arguments, see pg. 6-8, filed 01/08/2026, with respect to the rejection(s) of claim(s) 1, 3, 32, and 34 under 35 U.S.C. 102 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Seeber (US 20180228343 A1) and Itkowitz et al. (US 20180064499 A1). The new ground of rejection teaches the amended limitation of claim 1 (“wherein an orientation of the icon changes relative to the indicator body as the position of the instrument changes relative to the field of view”) according to the interpretation provided in the specification:
“The indicator body 250 may have at least one flat surface 253 and an icon 254 that may appear as a decal affixed along the flat surface 253. The icon 254 may include an identifier such as an identification for the manipulator arm to which the indicated tool is coupled or an identification for the indicated tool itself. As the indicator body 250 moves in three-dimensional space, the orientation of the icon 254 may rotate relative to the indicator body and directional portion 252 so that text or symbols on the icon 254 remains upright to the viewer.”
Claim Interpretation
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification.
The following terms in the claims have been given the following interpretations in light of the specification:
“Machine-readable media”: claims 32-41; pg. 15 lines 32-34 “In some embodiments, one or more of the processes may be implemented, at least in part, in the form of executable code stored on non-transitory, tangible, machine-readable media that when run by one or more processors…”
Thus, “machine-readable media” is defined as being non-transitory.
Should applicant wish different definitions, applicant should point to the portions of the specification that clearly show a different definition.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-7, 9-10, 32-38, and 40-41 is/are rejected under 35 U.S.C. 103 as being unpatentable over Seeber (US 20180228343 A1) in view of Itkowitz et al. (US 20180064499 A1, hereinafter "Itkowitz").
Regarding claim 1, Seeber teaches: A medical system ([0003] “The device is used in particular in a system for robot-assisted surgery, in particular for a telerobot-assisted procedure.”) comprising:
a display system ([0054] “FIG. 4 shows an image 1200 of the field of view captured by the endoscope 312, which image is displayed on the display unit 44.”); and
a control system, wherein the control system includes a processing unit including one or more processors ([0002] “At least one control unit processes the image data and outputs images corresponding to the image data on at least one display unit.”), and wherein the processing unit is configured to:
display, on the display system, an image of a field of view of a surgical environment, wherein the image is generated by an imaging component ([0054] “FIG. 4 shows an image 1200 of the field of view captured by the endoscope 312, which image is displayed on the display unit 44. In the displayed image 1200, the visible tissue structures 1210 in the surgical area 30 as well as further pieces of information are displayed.”);
generate a three-dimensional synthetic indicator for a position of an instrument outside of the field of view of the surgical environment (fig. 4, indicator 1250 points toward a position 1240 on a surgical instrument; [0054] “The end effector 310 is located outside the field of view of the endoscope 312 and is thus not illustrated in the image 1200 displayed by the display unit 44. A point of rotation 1240 at the front end of the bendable instrument shaft 309 is used as a reference point of the end effector 310 in the previous embodiment. In other embodiments also other points of the end effector can be used as reference points. Between the point of intersection 303b and the reference point 1240 the distance vector in the three-dimensional space, preferably in the three-dimensional device coordinate system X, Y, Z of the manipulator 12 is determined by the control unit 40. The value of the vector is optionally inserted into the displayed image 1200 as distance information 1260. The illustrated arrow 1250 extends in a section of the vector. The vector is defined by the line segment between the point of intersection 303b and the reference point 1240.”
[0057] teaches that the indicator can be three-dimensional: “If the endoscope 312 is a stereo endoscope, the images 1200, 1300 and 1400 may also be captured and output as three-dimensional images. The arrows 1250 and 1450 can then be inserted with a corresponding orientation along the course of the determined vector into the three-dimensional image 1200, 1300, 1400 such that also the illustration of the arrows 1250 and 1450 is made in a three-dimensional manner and thus the surgeon obtains a real three-dimensional direction information.”), wherein the three-dimensional synthetic indicator includes an indicator body and an icon on the indicator body (fig. 4, arrow body 1250 can be considered the “indicator body”, and numerical readout 1260 can be considered the “icon”); and
display the three-dimensional synthetic indicator with the image of the field of view of the surgical environment (displayed image 1200 includes tissue 1210 in the surgical area as well as indicator 1250; [0054] provides more detail),
Seeber does not explicitly teach: wherein an orientation of the icon changes relative to the indicator body as the position of the instrument changes relative to the field of view.
Itkowitz teaches a synthetic indicator for a surgical instrument including an indicator body and an icon on the indicator body (fig. 3D-3F show badge 218 with central portion 220 and orbiting portion 216, corresponding to the “indicator body” and “icon” respectively). Fig. 3D-3F show that the orbiting portion 216 rotates around the central portion 220 based on the orientation of the surgical tool and the user’s control device, as described in [0046]. However, the text on the icon remains upright regardless of the motion of the indicator body and the motion of the icon around the indicator body, meaning that an orientation of the icon changes relative to the indicator body. The indicator of Seeber always angles to face the position of the instrument, while the icon of Itkowitz always remains facing upright; therefore, if the indicator of Seeber is combined with the icon of Itkowitz, an orientation of the icon changes relative to the indicator body as the position of the instrument changes relative to the field of view.
Seeber and Itkowitz are both analogous to the claimed invention because they are in the same field of graphical displays to aid in robotic surgery by indicating the location of surgical instruments. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Seeber with the teachings of Itkowitz to automatically rotate the informative icon of Seeber (1260) to always remain upright, rather than following the direction of the indicator (1250). The motivation would have been to improve the user experience by making the icon easier for the surgeon to quickly read, regardless of the orientation of the associated indicator.
Regarding claim 2, the combination of Seeber in view of Itkowitz teaches: The medical system of claim 1 wherein the icon on the indicator body indicates a controlling manipulator arm (Itkowitz [0042] “The central portion 214 and orbiting portion 216 may contain different pieces of association information. For example, as shown in FIG. 2, the central portion 214 includes a number indicating the arm 54b (e.g. Arm “3”) to which the medical instrument 26b is attached.”).
Seeber and Itkowitz are both analogous to the claimed invention because they are in the same field of graphical displays to aid in robotic surgery by indicating the location of surgical instruments. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Seeber with the teachings of Itkowitz to include information in the icon indicating which arm the associated indicator points toward. The motivation would have been to improve the user experience by providing the surgeon with pertinent information in a convenient, easily visible location.
Regarding claim 3, the combination of Seeber in view of Itkowitz teaches: The medical system of claim 2, wherein an orientation of the icon changes relative to the indicator body as the field of view changes (Itkowitz fig. 3D-3F show the rotation of the icon 216 around the indicator body 220 based on the orientation of the surgical tool and the user’s control device, as described in [0046], and the icon remaining in the upright orientation to display the text upright; similar to what was discussed in claim 1, if the indicator of Seeber is combined with the icon of Itkowitz, as the field of view changes, the indicator will change to remain pointing toward the instrument while the icon will remain upright).
The rationale and motivation to combine the invention of Seeber with the teachings of Itkowitz would have been similar to the motivation described in claim 1 – keeping the icon text upright for legibility.
Regarding claim 4, the combination of Seeber in view of Itkowitz teaches: The medical system of claim 1, wherein the three-dimensional synthetic indicator includes a three-dimensional directional indicator portion oriented toward the position of the instrument (Seeber fig. 4, synthetic indicator 1250 points toward reference point 1240 located on the instrument;
[0054] “A point of rotation 1240 at the front end of the bendable instrument shaft 309 is used as a reference point of the end effector 310 in the previous embodiment. In other embodiments also other points of the end effector can be used as reference points. Between the point of intersection 303b and the reference point 1240 the distance vector in the three-dimensional space, preferably in the three-dimensional device coordinate system X, Y, Z of the manipulator 12 is determined by the control unit 40. The value of the vector is optionally inserted into the displayed image 1200 as distance information 1260. The illustrated arrow 1250 extends in a section of the vector. The vector is defined by the line segment between the point of intersection 303b and the reference point 1240.”;
[0057] teaches that the displayed arrow 1250 may be three-dimensional: “If the endoscope 312 is a stereo endoscope, the images 1200, 1300 and 1400 may also be captured and output as three-dimensional images. The arrows 1250 and 1450 can then be inserted with a corresponding orientation along the course of the determined vector into the three-dimensional image 1200, 1300, 1400 such that also the illustration of the arrows 1250 and 1450 is made in a three-dimensional manner and thus the surgeon obtains a real three-dimensional direction information.”).
Regarding claim 5, the combination of Seeber in view of Itkowitz teaches: The medical system of claim 4 wherein generating the three-dimensional synthetic indicator includes determining a ray between a centerline of the imaging component and a point on the instrument (Seeber fig. 4, indicator 1250 is positioned along a line between center point 303b and reference point 1240 on the instrument;
[0054] explains further: “The crossing dash-dotted lines in the center of the displayed image 1200 show the point of intersection 303b of the optical axis 316 of the endoscope 312 and the object plane 304b. The end effector 310 is located outside the field of view of the endoscope 312 and is thus not illustrated in the image 1200 displayed by the display unit 44. A point of rotation 1240 at the front end of the bendable instrument shaft 309 is used as a reference point of the end effector 310 in the previous embodiment. In other embodiments also other points of the end effector can be used as reference points. Between the point of intersection 303b and the reference point 1240 the distance vector in the three-dimensional space, preferably in the three-dimensional device coordinate system X, Y, Z of the manipulator 12 is determined by the control unit 40. The value of the vector is optionally inserted into the displayed image 1200 as distance information 1260. The illustrated arrow 1250 extends in a section of the vector.”).
Regarding claim 6, the combination of Seeber in view of Itkowitz teaches: The medical system of claim 1, wherein the three-dimensional synthetic indicator includes a height dimension, a width dimension, and a depth dimension (Seeber [0057] teaches a 3D indicator: “If the endoscope 312 is a stereo endoscope, the images 1200, 1300 and 1400 may also be captured and output as three-dimensional images. The arrows 1250 and 1450 can then be inserted with a corresponding orientation along the course of the determined vector into the three-dimensional image 1200, 1300, 1400 such that also the illustration of the arrows 1250 and 1450 is made in a three-dimensional manner and thus the surgeon obtains a real three-dimensional direction information.”; 3D models must necessarily be measurable in three spatial dimensions).
Regarding claim 7, the combination of Seeber in view of Itkowitz teaches: The medical system of claim 1 wherein the three-dimensional synthetic indicator is pivotable relative to position of the instrument as the field of view changes (Seeber [0053] teaches that the endoscope may be pivoted to change the field of view: “When the endoscope 312 is pivoted by the manipulator arm 16b such that the optical axis 316 is shifted or pivoted in the direction of the arrow P1…”; [0054] teaches that the direction of the synthetic indicator 1250 is dependent only on the location of the reference point 1240 located on the instrument, meaning that if the field of view changes, the synthetic indicator will also pivot to remain pointing towards the instrument).
Regarding claim 9, the combination of Seeber in view of Itkowitz teaches: The medical system of claim 1 wherein generating the three-dimensional synthetic indicator includes determining a depth range for structures in the field of view (Seeber [0020] “Further, it is advantageous when the distance of the point to the object plane is preset in the control unit as a parameter such that it lies within the depth of field. In an imaging optical system of an image capturing unit, the object plane is a plane running perpendicularly to the optical axis and in particular containing the object point. Only the points of the object plane are sharply imaged on the image sensor of the image capturing unit. The depth of field, often also referred to as field depth, is a measure for the extension of the sharp area in the object space of an imaging optical system. The depth of field describes the size of the distance range within which an object is imaged with sufficient sharpness.”; where generating a synthetic indicator depends on imaging the corresponding instrument).
Regarding claim 10, the combination of Seeber in view of Itkowitz teaches: The medical system of claim 1 wherein the three-dimensional synthetic indicator morphs to indicate a distance between the three-dimensional synthetic indicator and instrument (Seeber [0054] “Alternatively or additionally, the length of the displayed arrow 1250 can be dependent on the value of the vector. I.e. when the end effector 310 has a longer distance to the field of view of the endoscope 312 and thus a longer distance to the point of intersection 303b, a longer arrow 1250 is displayed and when the distance is shorter a shorter arrow 1250 is displayed in the displayed image 1200.”).
Regarding claim 32, it is rejected with the same references, rationale, and motivation to combine as claim 1 because its limitations substantially correspond to the limitations of claim 1, with the additional limitation of a machine-readable media storing instructions (Itkowitz [0066] “The program or code segments can be stored in a processor readable storage medium…”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Seeber with the teachings of Itkowitz to store necessary code in standard computer-readable storage media so that the invention is compatible with standard computational hardware.
Regarding claims 33-38 and 40-41, their limitations substantially correspond to the limitations of claims 2-7 and 9-10 respectively; therefore, they are rejected with the same references, rationale, and motivation to combine as their corresponding claims.
Claim(s) 8 and 39 is/are rejected under 35 U.S.C. 103 as being unpatentable over Seeber (US 20180228343 A1) in view of Itkowitz (US 20180064499 A1) as applied to claims 1 and 32 above, and further in view of Robinson et al. (GB 2560390 A, hereinafter "Robinson").
Regarding claim 8, the combination of Seeber in view of Itkowitz teaches: the medical system of claim 1, but does not explicitly teach: wherein the three-dimensional synthetic indicator indicates instrument status or control mode.
Robinson teaches: wherein the three-dimensional synthetic indicator indicates instrument status or control mode (fig. 10 icons 1002, 1004, 1006, and 1008 are synthetic indicators which may indicate instrument status; pg. 33 lines 10-14 “Referring to figure 10, the icons 1002, 1004, 1006, 1008 and the imaging device icon 1012 comprise a cluster of three circular sub-icons. Each of the sub-icons suitably provides a status associated with the respective arm. For example, each sub-icon suitably provides a status of a different aspect of the instrument or arm which that icon represents.”; fig. 11 further shows that the aforementioned icons are each associated with a particular instrument or manipulator arm).
Robinson and the combination of Seeber in view of Itkowitz are both analogous to the claimed invention because they are in the same field of graphical displays to aid in robotic surgery. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Seeber in view of Itkowitz with the teachings of Robinson to display information about the status of an associated instrument on the synthetic indicator. The motivation would have been to improve the user experience by providing the surgeon with pertinent information in a convenient, easily visible location.
Regarding claim 39, its limitations substantially correspond to the limitations of claim 8; therefore, it is rejected with the same references, rationale, and motivation to combine as claim 8.
References Cited
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Larkin et al. (US 20080004603 A1) teaches a three-dimensional synthetic indicator for a position of an instrument outside the field of view of a surgical environment. The indicator may change in size depending on the distance to the instrument, and it has an icon representing a controlling manipulator arm.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BENJAMIN STATZ whose telephone number is (571)272-6654. The examiner can normally be reached Mon-Fri 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tammy Goddard can be reached at (571)272-7773. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BENJAMIN TOM STATZ/ Examiner, Art Unit 2611
/TAMMY PAIGE GODDARD/ Supervisory Patent Examiner, Art Unit 2611