DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Comments
The Amendment – After Non-Final Rejection filed on December 2, 2025 has been entered and made of record.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-4, 11, 17, 19, 20, 22 and 25 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Sanchez-Matilla et al. (U.S. Pub. No. 2024/0303984).
Re claim 1: Sanchez-Matilla et al. disclose a surgical system (i.e., “system 100”, Paragraph [0051]) comprising
a processing device (i.e., “machine-learning processing system 110”, Paragraph [0051]) configured to implement a trained machine learning model, the processing device being configured to:
receive first data (i.e., FIG. 5A, Paragraph [0091]) having a first data format (i.e., “receive, as input, surgical data 147 to be processed … the surgical data 147 can include data streams (e.g., an array of intensity, depth, and/or RGB values) for a single image or for each of a set of frames representing a temporal window of fixed or variable length in a video”, Paragraph [0060]) from a sensing device (See for example, “The surgical data 147 that is input can be received from a real-time data collection system 145”, Paragraph [0060]; “the one or more machine-learning models include or are associated with a preprocessing or augmentation (e.g., intensity normalization, resizing, cropping, etc.)”, Paragraph [0062]; and Paragraph [0113]);
receive additional data indicating a condition of the surgical system (i.e., “state detector 150 can use the output from the execution of the machine-learning model to identify a state within a surgical procedure (“procedure”). A procedural tracking data structure can identify a set of potential states that can correspond to part of a performance of a specific type of procedure. Different procedural data structures (e.g., and different machine-learning-model parameters and/or hyperparameters) may be associated with different types of procedures”, Paragraph [0063]);
based on the additional data, perform a determination of whether to process the first data or data derived therefrom by inputting the first data or data derived therefrom (i.e., Paragraph [0062]) to the trained machine learning model (i.e., what data to be processed according to state);
in dependence on the determination, input the first data or data derived therefrom to the trained machine learning model (i.e., Paragraph [0064]; “state and phase values determined by the state detector 150 can be fed back to be used by one or more machine-learning models executed by the model execution system 140. For example, a machine-learning model can be used to generate features for use by the state detector 150, and state or phase determinations of the state detector 150 can become inputs for other machine-learning models or networks executed by the model execution system 140”, Paragraph [0065]; and Paragraph [0066]); and
output second data (i.e., FIG. 5B, Paragraph [0091]) having a second data format (See for example, “An output of the one or more machine-learning models can include image-segmentation or probabilistic heatmap data that indicates which (if any) of a defined set of structures are predicted within the visual data, a location and/or position and/or pose of the structure(s) within the image data, and/or state of the structure(s) associated with a proposed region of interest. The location can be a set of coordinates in the image data. For example, the coordinates can provide a bounding box that defines the proposed region of interest”, Paragraph [0062]; “A modified visualization of the surgical procedure can include incorporating an image adjustment in a real-time output of the video of the surgical procedure, such as adjusting one or more of contrast, color, and focus in a region of interest. Further examples of adjustments can include full new image synthesis, recoloring, pastelization, and/or other enhancement techniques known in the art. The image adjustments may appear as a virtual light source within the image and can be concentrated with a greater intensity near the centroid of the region of interest, for example”, Paragraph [0068]; Paragraph [0069]; and Paragraph [0091]).
Re claim 2: Sanchez-Matilla et al. disclose wherein the second data format comprises an additional aspect to the first data format (i.e., “augmented visualization of the surgical view”, Paragraph [0091]).
Re claim 3: Sanchez-Matilla et al. disclose wherein the processing device is configured to generate the additional aspect in dependence on the first data using the trained machine learning model (i.e., “augmented visualization of the surgical view … According to one or more aspects, when adaptive visualization is activated, the one or more machine-learning models 702 of FIG. 7 can predict the proposed region of interest 440 in image 500B to generate a modified visualization of the surgical procedure by incorporating an image adjustment in a real-time output of the video of the surgical procedure … The image adjustment may appear, for example, as a change in contrast, brightness, focus, or other such parameters to enhance visibility for the surgeon”, Paragraph [0091]).
Re claim 4: Sanchez-Matilla et al. disclose wherein the trained machine learning model is a generative model (See for example, “The machine-learning models can include a fully convolutional network adaptation (FCN) and/or conditional generative adversarial network model”, Paragraph [0048]).
Re claim 11: Sanchez-Matilla et al. disclose wherein the surgical system further comprises a surgical instrument (i.e., “The surgical data 147 can include additional data streams, such as audio data, RFID data, textual data, measurements from one or more surgical instruments/sensors, etc., that can represent stimuli/procedural state from the operating room”, Paragraph [0060]).
Re claim 17: Sanchez-Matilla et al. disclose wherein the sensing device is an imaging device comprising one or more image sensors (i.e., “surgical data 147 can include data streams (e.g., an array of intensity, depth, and/or RGB values) for a single image or for each of a set of frames representing a temporal window of fixed or variable length in a video”, Paragraph [0060]; and see also, Paragraph [0113]).
Re claim 19: Sanchez-Matilla et al. disclose wherein the first data format and the second data format are respective representations of an image or a video (See for example, Figures 5A and 5B, Paragraph [0091]).
Re claim 20: Sanchez-Matilla et al. disclose wherein the first data represents a complete field of view or a fraction of a field of view of the imaging device (See for example, “image 500A depicts a surgical instrument 502 proximate to an anatomical structure 504. In this example, the surgical instrument 502 and the anatomical structure 504 are located in close physical proximity to a centroid 505 of the image 500A, while background structures 506 are further separated from the centroid 505”, Paragraph [0091]).
Re claim 22: Sanchez-Matilla et al. disclose wherein the second data output by the trained machine learning model is a predicted output (i.e., “The one or more machine-learning models, during execution, can receive, as input, surgical data 147 to be processed and generate one or more inferences according to the training”, Paragraph [0060]) and
wherein the sensing device is further configured to acquire true data (i.e., “labeled ground truth data”, Paragraph [0128]) having the second data format (i.e., Paragraph [0061]; “user input 148 may also support the addition of labels/annotations”, Paragraph [0092]; and Paragraph [0128]).
Re claim 25: Sanchez-Matilla et al. disclose a method for data processing in a surgical system (i.e., “computer-implemented method that predicts a proposed region of interest in an image from a video of a surgical procedure based on one or more contextual targets”, Abstract), the method comprising:
receiving first data (i.e., FIG. 5A, Paragraph [0091]) having a first data format (i.e., “receive, as input, surgical data 147 to be processed … the surgical data 147 can include data streams (e.g., an array of intensity, depth, and/or RGB values) for a single image or for each of a set of frames representing a temporal window of fixed or variable length in a video”, Paragraph [0060]) from a sensing device (See for example, “The surgical data 147 that is input can be received from a real-time data collection system 145”, Paragraph [0060]; and Paragraph [0113]);
receiving additional data indicating a condition of the surgical system (i.e., “state detector 150 can use the output from the execution of the machine-learning model to identify a state within a surgical procedure (“procedure”). A procedural tracking data structure can identify a set of potential states that can correspond to part of a performance of a specific type of procedure. Different procedural data structures (e.g., and different machine-learning-model parameters and/or hyperparameters) may be associated with different types of procedures”, Paragraph [0063]);
based on the additional data, perform a determination of whether to process the first data or data derived therefrom by inputting the first data or data derived therefrom (i.e., Paragraph [0062]) to the trained machine learning model (i.e., what data to be processed according to state);
in dependence on the determination, input the first data or data derived therefrom to the trained machine learning model (i.e., Paragraph [0064]; “state and phase values determined by the state detector 150 can be fed back to be used by one or more machine-learning models executed by the model execution system 140. For example, a machine-learning model can be used to generate features for use by the state detector 150, and state or phase determinations of the state detector 150 can become inputs for other machine-learning models or networks executed by the model execution system 140”, Paragraph [0065]; and Paragraph [0066]); and
outputting second data (i.e., FIG. 5B, Paragraph [0091]) having a second data format (See for example, “An output of the one or more machine-learning models can include image-segmentation or probabilistic heatmap data that indicates which (if any) of a defined set of structures are predicted within the visual data, a location and/or position and/or pose of the structure(s) within the image data, and/or state of the structure(s) associated with a proposed region of interest. The location can be a set of coordinates in the image data. For example, the coordinates can provide a bounding box that defines the proposed region of interest”, Paragraph [0062]; “A modified visualization of the surgical procedure can include incorporating an image adjustment in a real-time output of the video of the surgical procedure, such as adjusting one or more of contrast, color, and focus in a region of interest. Further examples of adjustments can include full new image synthesis, recoloring, pastelization, and/or other enhancement techniques known in the art. The image adjustments may appear as a virtual light source within the image and can be concentrated with a greater intensity near the centroid of the region of interest, for example”, Paragraph [0068]; Paragraph [0069]; and Paragraph [0091]).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 5 and 6 are rejected under 35 U.S.C. 103 as being unpatentable over Sanchez-Matilla et al. in view of Souza (U.S. Pub. No. 2022/0240879). The teachings of Sanchez-Matilla et al. have been discussed above.
As to claim 5, Sanchez-Matilla et al. does not explicitly disclose wherein the first data format represents data having a reduced quality relative to the second data format.
Souza teaches the first data format (i.e., “sparse image set”, Paragraph [0073]) represents data having a reduced quality relative to the second data format (See for example, “The low-dose sinogram—which may reflect, for example, an 87.5% reduction in dose as compared to a normal-dose sinogram—has inferior image quality to an up-sampled sinogram 208 such as that shown in FIG. 2B”, Paragraph [0068]).
Sanchez-Matilla et al. and Souza are combinable because they are from the field of digital image processing for medical imaging.
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Sanchez-Matilla et al. by incorporating the first data format represents data having a reduced quality relative to the second data format, as taught by Souza.
The suggestion/motivation for doing so would have been to expose the patient to a lower radiation when imaging (i.e., Paragraph [0060]).
Therefore, it would have been obvious to combine Souza with Sanchez-Matilla et al. to obtain the invention as specified in claim 5.
As to claim 6, Sanchez-Matilla et al. does not explicitly disclose where the second data format has a higher spatial, frequency or temporal resolution than the first data format.
Souza teaches the second data format (i.e., “up-sampled sinogram”, Paragraph [0076]) has a higher spatial, frequency or temporal resolution (i.e., Paragraph [0046]; and “applying a high-resolution filter kernel”, Paragraph [0079]) than the first data format (i.e., “sparse image set”, Paragraph [0073]).
Therefore, it would have been obvious to a person having ordinary skill in the art to which the claimed invention pertains before the effective filing date of the claimed invention to have modified Sanchez-Matilla et al. to include the second data format has a higher spatial, frequency or temporal resolution than the first data format, as taught by Souza, in order to enhance soft tissue edges represented in the image set (See for example, Souza at Paragraph [0079]).
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Sanchez-Matilla et al. in view of Kim et al. (U.S. Pub. No. 2021/0272339). The teachings of Sanchez-Matilla et al. have been discussed above.
As to claim 7, Sanchez-Matilla et al. does not explicitly disclose wherein first data format has fewer data channels than the second data format.
Kim et al. teaches a first data format that has fewer data channels than a second data format (i.e., “reconstructing an image from Ls (1≤LS<L) channels, where the reconstructed image is comparable with an image reconstructed from beamformed data using L channels. As an example, an image can be reconstructed and beamformed using 1 channel, where the resulting image has image quality similar to, or better than, an image reconstructed from 64 or 128 channel data, without requiring increased complexity and significant change in hardware or front-end architecture”, Paragraph [0022]).
Sanchez-Matilla et al. and Kim et al. are combinable because they are from the field of digital image processing for medical imaging.
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Sanchez-Matilla et al. by incorporating the first data format has fewer data channels than the second data format, as taught by Kim et al.
The suggestion/motivation for doing so would have been to reduce the size and cost for imaging systems without degrading image quality and performance (See for example, Kim et al. at Paragraph [0039]).
Therefore, it would have been obvious to combine Kim et al. with Sanchez-Matilla et al. to obtain the invention as specified in claim 7.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Sanchez-Matilla et al. in view of Krishnan et al. (U.S. Pub. No. 2025/0191124). The teachings of Sanchez-Matilla et al. have been discussed above.
As to claim 8, Sanchez-Matilla et al. does not explicitly disclose wherein the second data format comprises a greater number of frequency bands than the first data format.
Krishnan et al. teaches a second data format (i.e., “high resolution reconstructed image 268”, Paragraph [0063]) that comprises a greater number of frequency bands than a first data format (i.e., “low resolution image 264”, Paragraph [0063]; and “machine learning enabled restoration of low resolution images and improving the reconstruction of high frequency information”, Paragraph [0070]).
Sanchez-Matilla et al. and Krishnan et a. are combinable because they are from the field of digital image processing for medical imaging.
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Sanchez-Matilla et al. by incorporating the second data format comprises a greater number of frequency bands than the first data format, as taught by Krishnan et al.
The suggestion/motivation for doing so would have been to alleviate the degradation of high frequency details when increasing the resolution of low resolution images.
Therefore, it would have been obvious to combine Krishnan et al. with Sanchez-Matilla et al. to obtain the invention as specified in claim 8.
Claims 9, 10, and 12-15 are rejected under 35 U.S.C. 103 as being unpatentable over Sanchez-Matilla et al. in view of Meglan (U.S. Pub. No. 2023/0024362). The teachings of Sanchez-Matilla et al. have been discussed above.
As to claim 9, Sanchez-Matilla et al. does not explicitly disclose wherein the surgical system further comprises a robot arm having one or more joints.
Meglan teaches the surgical system (i.e., “surgical robotic system 10”, Paragraph [0027]) further comprises a robot arm having one or more joints (i.e., “each of the robotic arms 40 may include a plurality of links 42a, 42b, 42c, which are interconnected at joints 44a, 44b, 44c, respectively”, Paragraph [0033]; and Paragraphs [0035]-[0036]).
Sanchez-Matilla et al. and Meglan are combinable because they are from the field of digital image processing for medical imaging.
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Sanchez-Matilla et al. by incorporating the surgical system further comprises a robot arm having one or more joints, as taught by Meglan.
The suggestion/motivation for doing so would have been to enable rapid and effective completion of tissue manipulations tasks.
Therefore, it would have been obvious to combine Meglan with Sanchez-Matilla et al. to obtain the invention as specified in claim 9.
As to claim 10, Sanchez-Matilla et al. does not explicitly disclose wherein the additional data indicating the condition of the surgical system comprises data indicating a state of the robot arm.
Meglan teaches the additional data indicating the condition of the surgical system comprises data indicating a state of the robot arm (See for example, Paragraph [0038]; and “sensor data”, Paragraph [0047]).
Therefore, it would have been obvious to a person having ordinary skill in the art to which the claimed invention pertains before the effective filing date of the claimed invention to have modified Sanchez-Matilla et al. to include the additional data indicating the condition of the surgical system comprises data indicating a state of the robot arm, as taught by Meglan, in order to produce force feedback commands and provide haptic feedback to a surgeon (See for example, Meglan at Paragraph [0038]).
As to claim 12, Sanchez-Matilla et al. does not explicitly disclose wherein the additional data indicating the condition of the surgical system comprises data indicating a state of the surgical instrument.
Meglan teaches the additional data (i.e., “sensor data”, Paragraph [0047]) indicating the condition of the surgical system comprises data indicating a state of the surgical instrument (i.e., “the sensor data may be transmitted by the surgical instrument 50 to the learning system 100”, Paragraph [0048])
Therefore, it would have been obvious to a person having ordinary skill in the art to which the claimed invention pertains before the effective filing date of the claimed invention to have modified Sanchez-Matilla et al. to include the additional data indicating the condition of the surgical system comprises data indicating a state of the surgical instrument, as taught by Meglan, in order to enable the detection of surgical instrument failure (See for example, Meglan at Paragraph [0046]).
As to claim 13, Sanchez-Matilla et al. does not explicitly disclose wherein the surgical system is configured to input the first data into the trained machine learning model to output the second data if the data indicating the condition of the surgical system indicates that the surgical instrument is in operation.
Meglan teaches the surgical system is configured to input the first data into the trained machine learning model to output the second data (i.e., “failure probability”, Paragraph [0049]) if the data indicating the condition of the surgical system indicates that the surgical instrument is in operation (i.e., “The system 10 utilizes the learning system 100, once it has been trained, to determine an operational status of the surgical instrument 50 being used during the surgical procedure … The learning system 100 receives as input the following real-time data: video data, sensor data, user input data, service life data, and procedure data”, Paragraph [0047]).
Therefore, it would have been obvious to a person having ordinary skill in the art to which the claimed invention pertains before the effective filing date of the claimed invention to have modified Sanchez-Matilla et al. to include the surgical system is configured to input the first data into the trained machine learning model to output the second data if the data indicating the condition of the surgical system indicates that the surgical instrument is in operation, as taught by Meglan, in order to prevent tissue damage when a failure is detected (See for example, Meglan at Paragraph [0052]).
As to claim 14, Sanchez-Matilla et al. does not explicitly disclose wherein the sensing device is configured to sense data relating to the position of the one or more joints or forces at the one or more joints.
Meglan teaches the sensing device is configured to sense data relating to the position of the one or more joints or forces at the one or more joints (i.e., “The controller 21a also receives back the actual joint angles”, Paragraph [0038]).
Therefore, it would have been obvious to a person having ordinary skill in the art to which the claimed invention pertains before the effective filing date of the claimed invention to have modified Sanchez-Matilla et al. to include the sensing device is configured to sense data relating to the position of the one or more joints or forces at the one or more joints, as taught by Meglan, in order to produce force feedback commands and provide haptic feedback to a surgeon (See for example, Meglan at Paragraph [0038]).
As to claim 15, Sanchez-Matilla et al. does not explicitly disclose wherein the sensing device comprises a torque sensor and/or a position sensor.
Meglan teaches wherein the sensing device comprises a torque sensor and/or a position sensor (See for example, Paragraphs [0038] and [0040]).
Therefore, it would have been obvious to a person having ordinary skill in the art to which the claimed invention pertains before the effective filing date of the claimed invention to have modified Sanchez-Matilla et al. to include the sensing device comprises a torque sensor and/or a position sensor, as taught by Meglan, in order to produce force feedback commands and provide haptic feedback to a surgeon (See for example, Meglan at Paragraph [0038]).
Response to Arguments
Claim Interpretation
With respect to claim 1, Applicant’s arguments (Remarks dated December 2, 2025, pages 5-6) have been fully considered, and they are persuasive. After consideration of the claim, and upon review of the originally filed specification, the “processing device”, appears to have a sufficiently definite meaning as the name for the structure that performs the function. Thus, 35 USC § 112(f) is not invoked.
Claim Rejections - 35 USC §§ 102 and 103
With respect to claims 1 and 25, Applicant’s arguments (Remarks dated December 2, 2025, pages 6-8) have been fully considered, but they are not persuasive. Applicant respectfully submits that Sanchez-Matilla does not disclose any determination of whether to input the surgical data to a trained machine-learning model based on a different part of the surgical data (Remarks dated December 2, 2025, page 7). Examiner respectfully disagrees.
Sanchez-Matilla et al. disclose a computer-implemented method that predicts a proposed region of interest in an image from a video of a surgical procedure based on one or more contextual targets (i.e., Abstract). Sanchez-Matilla et al. receives (1) first data having first data format from a sensing device (See for example, surgical data 147 comprising data streams including images, measurements, information gathered from sensing devices, Paragraphs [0060], [0062], and [0113]), and (2) additional data indicating a condition of the surgical system via a state detector that identifies a state of a procedure (See for example, Paragraph [0063]). In addition, Sanchez-Matilla et al. disclose the outputting of second data having a second data format by at least an alert output system 170, and/or an augmented reality device 180, which can produce a modified visualization of the surgical procedure can include incorporating an image adjustment in a real-time output of the video of the surgical procedure, such as adjusting one or more of contrast, color, and focus in a region of interest (See for example, Paragraphs [0068]-[0069]). The outputting of second data having a second data format, as claimed, does not require the execution of a trained machine learning model.
The determination of whether to process the first data or data derived therefrom, as broadly claim, by inputting the data or data derived therefrom to the trained machine learning model based on the additional data (i.e., state of the procedure), and in dependence of the determination, input the data or data derived therefrom to the trained machine learning model, can be interpreted as the processing by inputting to a machine learning model image data or any data derived therefrom (i.e., images, measurements, user input on images (i.e., Paragraph [0061]) according to the state of the procedure. See for example, Paragraphs [0063]-[0064], for processing images or characteristics, which in turn are inputted to a ML model to further enhance machine-learning results. Thus, it can be shown that Sanchez-Matilla et al. disclose the claimed determination as recited in amended claims 1 and 25.
Therefore, the rejections are maintained.
With respect to claims 2-15, 17, 19, 20, and 22, Applicant’s arguments (Remarks dated December 2, 2025, pages 6-8) are no different from those previously presented with respect to claims 1 and 25, and already addressed above.
Therefore, the rejections are maintained.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOSE M TORRES whose telephone number is (571)270-1356. The examiner can normally be reached Monday thru Friday; 10:00 AM to 6:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at 571-272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOSE M TORRES/Examiner, Art Unit 2664 01/06/2025
/JENNIFER MEHMOOD/Supervisory Patent Examiner, Art Unit 2664
Prosecution Insights