DETAILED ACTION
Notices to Applicant
This communication is a non-final rejection. Claims 2-21, as filed 10/20/2025, are currently pending and have been considered below.
Priority is generally acknowledged as shown on the filing receipt with the earliest priority date being 11/15/2019.
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon and the rationale supporting the rejection would be the same under either status.
Claim Objections
Claims 2-8 and 16-21 are objected to because of the following informalities. Claims 2 and 16 recite “offset from the center of the display” but the only antecedent is “a center of the display device”. This limitation is interpreted as “offset from the center of the display device.” Additionally, claims 20 and 21 are objected to because a carriage return is missing before claim 21 and thus the claims run together.
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 2-21 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claims no not fall within at least one of the four categories of patent eligible subject matter because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more.
Step 1
The claim(s) recite(s) subject matter within a statutory category as a process (claims 2-15), machine, and/or articles of manufacture (claims 16-20).
Step 2A Prong One
The broadest reasonable interpretation of these steps includes mental processes such as determining an area of interest from gaze data and determining that the AOI is off-center. A proctoring surgeon watching a trainee’s console could do this by observing where the trainee looks, judging the position, and making a note or telling the trainee. But for the generic computer and imaging devices determining an AOI and detecting that it is off-center in the context of this claim is analogous to steps a human would perform.
Dependent claims recite additional subject matter which further narrows or defines the abstract idea embodied in the claims. For example, claims 3-6, 10-15, 17-19, and 21 further narrow the abstract idea by giving more detail on how the determinations are made.
Step 2A Prong Two
This judicial exception is not integrated into a practical application. In particular, the additional elements do not integrate the abstract idea into a practical application, other than the abstract idea per se, because the additional elements:
amount to mere instructions to apply an exception. For example, using tools like digital cameras and computers and outputting notifications amounts to invoking computers as a tool to perform the abstract idea, see MPEP 2106.05(f))
add insignificant extra-solution activity to the abstract idea. For example, receiving image and sensor data amounts to mere data gathering, accessing a database amounts to selecting a particular data source or type of data to be manipulated, see MPEP 2106.05(g))
generally link the abstract idea to a particular technological environment or field of use such as the gaze being monitoring during a surgical procedure, see MPEP 2106.05(h))
Dependent claims recite additional subject matter which amount to limitations consistent with the additional elements in the independent claims. For example, claims 3, 8, 13, 16, 17, and 21 give more details on how computers are used as tools such as by generating visual notifications. Looking at the limitations as an ordered combination adds nothing that is not already present when looking at the elements taken individually. There is no indication that the combination of elements improves the functioning of a computer or improves any other technology. Their collective functions merely provide conventional computer implementation and do not impose a meaningful limit to integrate the abstract idea into a practical application.
Step 2B
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to discussion of integration of the abstract idea into a practical application, the additional elements amount to no more than mere instructions to apply an exception, add insignificant extra-solution activity to the abstract idea, and generally link the abstract idea to a particular technological environment or field of use. Additionally, the additional limitations, other than the abstract idea per se amount to elements that have been recognized as well-understood, routine, and conventional activity in particular fields such as receiving or transmitting data over a network, Symantec, MPEP 2106.05(d)(II)(i), performing repetitive calculations, Flook, MPEP 2106.05(d)(II)(ii), electronic recordkeeping, Alice Corp., MPEP 2106.05(d)(II)(iii), and/or storing and retrieving information in memory, Versata Dev. Group, MPEP 2106.05(d)(II)(iv). The Examiner further notes that the eye tracking software is well-understood, routine, and conventional as admitted in [0037] of applicant’s specification (as published): “Any suitable eye tracking system known in the art may be used as the eye tracking device 216.”
Dependent claims recite additional subject matter which, as discussed above with respect to integration of the abstract idea into a practical application, amount to invoking computers as a tool to perform the abstract idea. Dependent claims recite additional subject matter which amount to limitations consistent with the additional elements in the independent claims. Looking at the limitations as an ordered combination adds nothing that is not already present when looking at the elements taken individually. There is no indication that the combination of elements improves the functioning of a computer or improves any other technology. Their collective functions merely provide conventional computer implementation.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 2-4, 6-8, 16-18, 20, 21 are rejected under 35 U.S.C. 103 as being unpatentable over Jarc (US20170172675A1) in view of Kronman (US20170258295A1).
Claim interpretation – The Examiner interprets “a center of the display device” includes any non-perimeter region of the display and “offset from the center” includes when a gaze area of interest is not fully located within the perimeter region of the display.
Regarding claim 2, Jarc discloses: A computer-implemented method comprising:
--receiving sensor data indicating a gaze location of a user during a surgical procedure (“the eye tracking unit 200 may also include light emitters that can emit light to illuminate the surgeon's eyes, so that the gaze points of the surgeon's left and right eyes may be captured by the left and right eye trackers 204 a and 204 b respectively,” [0076]; “the left and right eye trackers 204 a, 204 b of the eye tracking unit 200 observe and measure an eye characteristic (e.g., a characteristic reflective of eye gaze) of the user,” [0082]);
--receiving image data of the surgical procedure from a camera (“the endoscope 112 located at the teleoperational assembly 100 can be manipulated to capture images of a surgical field during a surgery, and theses [sic] images are shown on the left and right image displays 202 a and 202 b,” [0074]; FIG. 2B);
--determining, based on the sensor data, an area of interest within the image data identifying the gaze location of the user on a display device, the display device displaying the image data (“At step 220, the processor 206 uses the measured eye gaze data from the eye trackers 204 a, 204 b to determine the 3D viewing location in the image on the image display 151 (within image frame 165) at which the user's eyes are directed,” [0082]; 220 in FIG. 2C; “the imaging device may “follow” the user's gaze and re-center its field of view (e.g., to position the center of its field of view at the user's gaze point),” [0084]; Step 230 in FIG. 2C);
Jarc teaches notifications in [0098]. Jarc does not expressly disclose but Kronman teaches:
--detecting that the area of interest is offset from a center of the display device (“the target area comprises the center of the display screen,” [0012]; “after every application of said transformation the system evaluates the latest position of said region of interest and in case said region of interest is not positioned at such second location, a new transformation is estimated based on the most recent location of the region of interest and the same is applied to cause said region of interest to move in the direction of said second location,” [0029]; “After applying the transformation, the new ROI location is detected in the image and the process is repeated in an iterative manner until the ROI is placed on top of the target,” [0080]).
--generating, based on detecting that the area of interest is offset from the center of the display, a notification recommending adjusting a position or orientation of the camera (“the user is guided by means of icons, guides, or other data displayed on a graphical user interface (GUI) which directs a user to manually move the distal tip of the insertion portion of the endoscope in order to position the abnormality in the target region of the display screen,” [0070]; “generating and electronically displaying a visual guide that directs a movement of the insertion portion such that the region of interest moves into the target area on the display screen,” claim 11. The distal tip caries the viewing element so directing its movement directs adjustment of camera position.). The Examiner further notes that Kronman teaches in [0066]: “the medical practitioner may mark the ROI/polyp using known methods such as by a mouse click, a touch screen, a hand gesture, or eye tracking.”
One of ordinary skill in the art before the effective filing date would have been motivated to expand Jarc’s re-centering based on gaze to include the user guidance notifications based on centeredness of Kronman because it solves the problem in which it is “difficult to maintain the display of the polyp constantly in the center of the display screen due to the movement of the body cavity, such as the colon, in which the polyp may be detected” (Kronman [0008]).
Additionally, it can be seen that each element is taught by either Jarc or Kronman. Kronman’s notifications do not affect the normal functioning of the elements of the claim which are taught by Jarc. Because the elements do not affect the normal functioning of each other, the results of their combination would have been predictable. Therefore, before the effective filing date of the claimed invention, it would have been obvious to combine the teachings of Kronman with the teachings of Jarc since the result is merely a combination of old elements, and, since the elements do not affect the normal functioning of each other, the results of the combination would have been predictable.
Regarding claim 3, Jarc discloses: wherein the notification is a visual notification (“blink red” in [0098]).
Regarding claim 4, Jarc discloses: wherein the sensor data comprises a time duration associated with the gaze location (“The user's gaze point in 3D space may be determined once the reflected infrared light reaches a particular threshold for a certain amount of time,” [0072]).
Regarding claim 6, Jarc discloses: wherein the surgical procedure is performed using a robotic surgical system (DA VINCI Surgical System in [0002]).
Regarding claim 7, Jarc discloses: wherein the camera is connected to a robotic arm of the robotic surgical system, and the method further comprises instructing the robotic arm to adjust a position of the camera to center the area of interest in a field of view of the camera (“arm 108 is a camera arm that supports and moves the endoscope,” [0052] ; “the user's eye gaze can be used to automatically center the view of the imaging device to correspond to the direction of the user's eye gaze,” [0066]).
Regarding claim 8, Jarc teaches: wherein the notification comprises an instruction to the user to adjust the camera to center the area of interest in a field of view of the camera (“, the proctor may send an instruction to the surgeon to correct the surgeon's actions and/or viewpoint in real-time…” [0098]).
Claims 16-18, 20, and 21 are substantially similar to claims 2-4, 7, and 8 and are rejected with the same reasoning. The examiner further notes that Jarc teaches in [0082]: “Any of the method steps described herein may be implemented, at least in part, in the form of executable code stored on non-transient, tangible, machine readable media that may be run by one or more processors.”
Claims 9, 10, 13, 15 are rejected under 35 U.S.C. 103 as being unpatentable over Jarc (US20170172675A1).
Regarding claim 9, Jarc teaches: A computer-implemented method comprising:
--receiving image data of a surgical procedure from a camera;
--presenting the image data on a display device (“the endoscope 112 located at the teleoperational assembly 100 can be manipulated to capture images of a surgical field during a surgery, and theses images are shown on the left and right image displays 202 a and 202 b,” [0074]);
--receiving sensor data identifying a gaze location of a surgeon during the surgical procedure (“At step 218, the left and right eye trackers 204 a, 204 b of the eye tracking unit 200 observe and measure an eye characteristic (e.g., a characteristic reflective of eye gaze) of the user,” [0082]);
--identifying, based on the sensor data, an area of interest within the image data on the display device (“At step 220, the processor 206 uses the measured eye gaze data from the eye trackers 204 a, 204 b to determine the 3D viewing location in the image on the image display 151 (within image frame 165) at which the user's eyes are directed,” [0082]);
--accessing a database of surgical procedure image data based on the area of interest (“acquiring baseline data as standards to evaluate and quantify the surgeon's skill level…by measuring an experienced surgeon's gaze point movement during a standard operation… the baseline data may be saved in a computer readable media to be used repeatedly in the future” [0103]; );
--determining an expected gaze location of the surgeon based on the database of surgical procedure image data (“the target P may be a predetermined location or region within display 450, or can even be established on the fly by the actual gaze location(s) accessed by the surgeon,” [0104]);
--determining that the gaze location of the surgeon differs from the expected gaze location (“both the proctor and trainee gaze points may be shown in one or both of the proctor and trainee displays, thereby allowing either one or both users to see any disparity between their gaze points,” [0086]; “The method 400 proceeds to step 412 by determining the surgeon's skill level by comparing the evaluation factor (e.g., the displacement ratio factor D, time ratio factor T, and/or gaze fixation time) against the baseline data (e.g., acquired at step 402),” [0114]); and
--generating a notification based on the gaze location of the surgeon differing from the expected gaze location (“the gaze point region of the surgeon's image displays may present a visual (e.g., blink red), audible (e.g., sound a buzzer), or tactile (e.g., vibrate a part of the manual controls) warning to warn the surgeon from proceeding with any incorrect operation,” [0098]; “a notice or an alarm may be given to prevent the surgeon from proceeding with any operation,” [0115]).
One of ordinary skill in the art before the effective filing date would have been motivated to expand Jarc’s stored baseline evaluations (FIGS. 4-5) to include its real-time disparity warning embodiments (FIGS. 2D and 3B) because this would improve training and patient care by giving feedback “so that the trainee can focus on the correct surgical site in real-time and avoid incorrect actions,” [0086].
Regarding claim 10, Jarc teaches: wherein identifying the area of interest is based on a portion of the sensor data comprising blink frequency (“A surgeon may look at a 3D location in the surgical field and confirm with a secondary action (e.g., by way of non-limiting example, by pushing a separate button, maintaining an extended gaze, or blinking in a particular pattern) to apply a virtual label in the surgical field and/or on the displayed image 150 to identify an anatomical area of interest,” [0067]; “Small lapses in gaze can be interpreted as blinks and are typically ignored,” [0072]).
The motivation to combine is the same as in claim 9.
Regarding claim 13, Jarc teaches: wherein the notification comprises an instruction identifying the expected gaze location (“the surgeon may be guided to complete the surgery using the proctor's gaze point intuitively,” [0097]; “the proctor may send an instruction to the surgeon to correct the surgeon's actions and/or viewpoint in real-time,” [0098]).
The motivation to combine is the same as in claim 9.
Regarding claim 15, Jarc teaches: wherein the notification comprises a graphical element identifying the expected gaze location (“the proctor's 3D eye gaze point may be shown as stereo images on the left and right image displays 244 a and 244 b of the trainee's eye tracking unit 240, so that the trainee may be assisted in real-time to complete the surgery using the proctor's gaze point as a visual guide to follow,” [0086]).
The motivation to combine is the same as in claim 9.
Claims 5 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Jarc (US20170172675A1) in view of Kronman (US20170258295A1) and Publicover (US20160085302A1).
Regarding claims 5 and 19, Jarc measures gaze locations and durations (“average fixation time of eye gaze” in [0113]) and selects the AOI from that data. Jarc does not expressly disclose but Publicover teaches: wherein determining the area of interest comprises:
--generating weight-averaged data of gaze locations based on the gaze location and the time duration; and selecting the area of interest based on the weight-averaged data (“combining this process with other processes and applying a lower weighting to the glint-reliant gaze-determination process, which in turn reduces its contribution to the top-level gaze-determination algorithm, ensures that a user's gaze is more accurately and stably calculated,” [0347]).
One of ordinary skill in the art before the effective filing date would have been motivated to expand Jarc and Kronman’s surgical display to include the weighting gaze data using time from Publicover because this would result in more accurate and stable gaze calculation (Publicover [0347]: “ensures that a user's gaze is more accurately and stably calculated”).
Claims 11 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Jarc (US20170172675A1) in view of Publicover (US20160085302A1).
Regarding claim 11, Jarc uses velocity of gaze (e.g., [0112]. Jarc does not expressly disclose but Publicover teaches: wherein identifying the area of interest is based on a velocity of the gaze location (“Eye velocities are measured based on eye-tracking within images at 333. If the eye velocity is greater than a minimum threshold for saccadic movements (typically 30° per second) at 334, then the system signifies that some form of saccadic movement has occurred at 335,” [0224]).
One of ordinary skill in the art before the effective filing date would have been motivated to expand Jarc’s surgical display to include the algorithmic approaches to discerning intent based on video images of eyes as taught by Publicover because this would help provide more accurate interpretations of what the surgeon is doing.
Additionally, it can be seen that each element is taught by either Jarc or Publicover. The calculations of Publicover do not affect the normal functioning of the elements of the claim which are taught by Jarc. Because the elements do not affect the normal functioning of each other, the results of their combination would have been predictable. Therefore, before the effective filing date of the claimed invention, it would have been obvious to combine the teachings of Jarc with the teachings of Publicover since the result is merely a combination of old elements, and, since the elements do not affect the normal functioning of each other, the results of the combination would have been predictable.
Regarding claim 12, Jarc does not expressly disclose but Publicover teaches: wherein identifying the area of interest is based on a portion of the sensor data describing pupil size (“the degree of pupil dilation can be used to discern emotional states such as fear, interest, or cognitive load,” [0098]; “Additional information to determine user intent may be determined from other factors such as pupil dilation, lens accommodation, eye lid motion, facial muscular movement, body and head motion, hand motion, and related contextual information,” [0088]).
The motivation to combine is the same as in claim 11.
Claim 14 is are rejected under 35 U.S.C. 103 as being unpatentable over Jarc (US20170172675A1) in view of Giataganas (US20190279765A1).
Regarding claim 14, Jarc does not expressly disclose but Giataganas teaches: wherein determining the expected gaze location is further based on a predictive model (“remote server 380 can process the live data (e.g., to identify and/or characterize a presence and/or position of one or more tools using a trained machine-learning model, to identify a procedural state using a trained machine-learning model or to train a machine-learning model),” [0068]).
One of ordinary skill in the art before the effective filing date would have been motivated to expand Jarc’s surgical display to include the machine learning of Giataganas because this would improve the display be facilitating the inclusion of more pertinent information about corresponding to the surgical procedure (Giataganas [0019]).
Additionally, it can be seen that each element is taught by either Jarc or Giataganas. The calculations of Giataganas do not affect the normal functioning of the elements of the claim which are taught by Jarc. Because the elements do not affect the normal functioning of each other, the results of their combination would have been predictable. Therefore, before the effective filing date of the claimed invention, it would have been obvious to combine the teachings of Jarc with the teachings of Giataganas since the result is merely a combination of old elements, and, since the elements do not affect the normal functioning of each other, the results of the combination would have been predictable.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOSHUA BLANCHETTE whose telephone number is (571)272-2299. The examiner can normally be reached on Monday - Thursday 7:30AM - 6:00PM, EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Shahid Merchant, can be reached on (571) 270-1360. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOSHUA B BLANCHETTE/Primary Examiner, Art Unit 3624
Prosecution Insights