DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 08/07/2026 has been entered.
Response to Amendment
Claims 1, 6, 9 and 13 have been cancelled. New Claims 19 and 20 have been added. Claims 2-3, 7, 10, 12, 14 and 16-17 have been amended. Claims 2-3, 7, 10, 12 and 14-20 remain pending.
Response to Arguments
In Pages 6-8 of Remarks, regarding new Claim 19, Applicant argues that reference Giataganas does not teach or suggest the newly claimed features. This argument is moot in view of the new grounds of rejection which relies on reference Lang et al (US 11553969 B1) to disclose these limitations in the claim.
Claim Objections
Claim 12 is objected to because of the following informalities:
Claim 12, Line 1, recites “The method according to claim 1”, which should be changed to “The method according to claim 19”.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 2-3, 7, 10, 12 and 14-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 19, Lines 10-12, recites “… the overlay … has at least some transparency”, and Lines 13-15, “… applying the overlay over … in which the one or more non-organic objects appear occluded by the augmentation information”. The two recited phrases are contradictory; specifically, with the overlay having some transparency, the non-organic objects should not appear occluded. For present purposes of examination, the recited phrase of “in which the one or more non-organic objects appear occluded by the augmentation information” is not considered.
Claim 20 has the same issue as discussed for Claim 19 above, and therefore is interpreted the same.
Claims 2-3, 7, 10, 12 and 14-18 are also rejected under 35 U.S.C. 112(b) because they inherit the indefiniteness of the claim(s) they respectively depend upon.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2-3, 7, 10, 12 and 14-20 are rejected under 35 U.S.C. 103 as being unpatentable over Giataganas (WO2022/195304 A1; hereafter Giataganas), in view of Lang et al (US 11553969 B1; hereafter Lang).
With regard to Claim 19, Giataganas discloses a method for providing a real-time augmented-reality image of a medical procedure, the method comprising the steps of:
capturing an image of the medical procedure (Giataganas, Abstract; “a surgical action to be performed during a surgical procedure […] An image/video capture device such as an endoscope, a wearable camera, a stationary camera, etc., can be used to capture the image(s)”);
automatically identifying and segmenting one or more non-organic objects in the captured image, wherein the one or more non-organic objects are automatically identified and segmented by a pretrained machine learning model (Giataganas, Para 0083; “the second machine learning model 400 further includes a decoder 406 that predicts and outputs scene segmentation 408 based on the feature space 404. The scene segmentation 408 provides semantic segmentation annotations […] to delineate with contours, masks, etc. one or more anatomical structures, surgical instruments, or other items (e.g., surgical staples, sutures, etc.) identified in the input window”);
creating a first mask comprising only the segmented one or more non-organic objects (Giataganas, Para 0087; “the segmentation masks 505 can represent separate parts of the surgical instruments and/or anatomical structures”);
creating an overlay comprising augmentation information relating to one or more body parts visible in the captured image (Giataganas, Para 0087; “the segmentation masks 505 can represent separate parts of the surgical instruments and/or anatomical structures”), wherein the overlay includes one or more respective 3D representations of the one or more body parts (Giataganas, Para 0091; “The machine learning model(s) track the provided input across the multiple images (e.g., surgical video) of the surgical procedure … Additionally, separate machine learning models are used to reconstruct real-world measures of the anatomical structures, or on-the-fly 3D “models” (or partial surfaces) of the anatomical structures.”; ; Para 0122; “… machine learning models are trained for depth reconstruction (mono/stereo) to recover and track three-dimensional structures of the features (e.g., surgical instruments, anatomical structures, other surgical items) in the video”) and has at least some transparency (Giataganas, Para 0099; “Various visual attributes of the graphical overlay 502, such as colors, transparency, visual-pattern, line thickness, etc., can be adjusted.”);
combining the captured image with the overlay by applying the overlay over the captured image (Giataganas, Para 0004; “generating a visualization comprising the video of the surgical procedure, and a first graphical overlay at a location of the anatomical structure …”); and
applying the first mask over the captured image (Giataganas, Para 0094; “In the augmented visualization 507, the surgical instruments are marked using graphic overlays 502.”).
Giataganas does not clearly and explicitly disclose
the segmented one or more non-organic objects being represented by pixels of the captured image, and
first applying the overlay over the captured image, followed by applying the first mask over the combined image.
Lang in the same field of endeavor discloses
the segmented one or more non-organic objects being represented by pixels of the captured image (Lang, Column 249, Lines 51-59; “In the example of FIG. 24D, a computer processor can track the physical screw driver 1021 … using direct video detection …, e.g. with an image capture … system …”. The disclosed physical screw driver 1021 as shown in Fig. 24D is a cropped group of pixels from captured video frames or images), and
first applying the overlay over the captured image, followed by applying the first mask over the combined image (Lang, Column 248, Line 62 to Column 249, Line 6, “FIGS. 24A-E provide illustrative, non-limiting examples of one or more augmented reality HMD displays including a virtual user interface 990 … the virtually displayed spine 993 of the patient, e.g. using a 3D model generated based on a pre-operative CT scan or an intra-operative O-arm scan.”. As Fig. 24A shows, the disclosed virtually displayed spine 993, corresponding to the claimed overlay, is displayed in the interface at early time point, then at a later point as demonstrated by Fig. 24D (cited below), the disclosed physical screw driver 1021 is displayed over the spine 993).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Giataganas, as suggested by Lang, in order to display a surgical tool with its real appearance and on top of other tissues at a surgical site. One of ordinary skill in the art would have been motivated to make the modification for the benefit of avoiding tissue injury in a surgical procedure by visualizing a surgical tool for entire duration of the procedure.
Fig. 24D of Lang
PNG
media_image1.png
448
672
media_image1.png
Greyscale
With regard to Claim 2, Giataganas and Lang disclose the method according to Claim 19. Giataganas further discloses wherein the pretrained machine learning model is trained to identify one or more surgical instruments as the one or more non-organic objects (Giataganas, Para 0083; “the second machine learning model 400 further includes a decoder 406 that predicts and outputs scene segmentation 408 based on the feature space 404. The scene segmentation 408 provides semantic segmentation annotations […] to delineate with contours, masks, etc. one or more anatomical structures, surgical instruments, or other items (e.g., surgical staples, sutures, etc.) identified in the input window”).
With regard to Claim 3, Giataganas and Lang disclose the method according to Claim 19. Giataganas further discloses wherein the pretrained machine learning model is trained (Giataganas, Para 0051; “Machine-learning processing system includes a data generator, […] to train a machine-learning model. […] Data generator can access (read/write) a data store with recorded data”) with models of a plurality of types of non-organic objects used in the medical procedure, said types comprising at least one or more of needles, gauze, wires, clamps, wires, trocars, forceps, scissors, catheter, drain, endograph elements, fibrillar, foam, clips, needle drivers, suction, hemostasis elements, vessel loops, gloves and patches (Giataganas, Para 0051; “Each of the images and/or videos included in the recorded data can be defined as a base image and can be associated with other data […] the other data can include image-segmentation data that identifies and/or characterizes one or more objects (e.g. tools, anatomical objects, etc) that are depicted in the image or video”).
With regard to Claim 7, Giataganas and Lang disclose the method according to Claim 19. Giataganas further discloses wherein the captured image is a stereoscopic image (Giataganas, Para 0109; “The machine learning model is trained using a pair of stereo frames that are captured using a stereo image capture device (not shown)”).
With regard to Claim 10, Giataganas and Lang disclose the method according to Claim 19. Giataganas further discloses comprising a step of resizing the overlay before applying the resized overlay over the captured image (Giataganas, Para 0099; “a user can adjust the attributes of the graphic overlays. For example, the user can select a type of highlighting, a color, a line thickness, a transparency, a shading pattern, a label, an outline, or any other such attributes to be used to generate and display the graphical overlay on the images”).
With regard to Claim 12, Giataganas and Lang disclose the method according to Claim 19, including the steps of the method. Giataganas further discloses wherein the captured image is provided to a capture card of a processing unit as a digital signal, and converted into an encoded bitstream by said processing unit, wherein said processing unit performs the steps of the method (Giataganas, Fig. 1 shows a system that contains machine learning training system (125), model execution system (140), augmentor (175), and real-time data collection system (145). This system is capable of collecting video stream, creating masks, combining masks and image for augmentation. More details are in 0050-0067).
With regard to Claim 20, Giataganas discloses a system for robot-assisted medical operations supported by real-time augmented reality support, the system comprising:
a. a surgical robot comprising at least one arm equipped with at least one surgical instrument (Giataganas, Para 0102; “The user feedback can further include generating and displaying a graphical overlay 502 on the surgical view […] can be integrated into a robotic workflow in response to the predictions described herein. For example, operating parameters of one or more surgical instruments are adjusted”) (Giataganas, Para 0131; “The surgical procedure support system 902 can also interface with a plurality of sensors 906 and effectors 908. […] The effectors 908 can be robotic components”. In surgical robotics, sensors and effectors are typically attached to end of robotic arms);
b. an image capturing device for capturing an image of an area affected by said at least one surgical instrument (Giataganas, Para 0131; “The surgical procedure support system 902 can acquire image data, such as images 302, using one or more cameras 904”); and
c. an image processing unit comprising at least an image feed input for receiving the image captured by the image capturing device, a processing element for processing the received image, and a memory element (Giataganas, Para 0131; “The surgical procedure support system 902 can include or may be coupled to the system 100 … The surgical procedure support system 902 can store, access, and/or update surgical data 914 associated with a training dataset and/or live data as a surgical procedure is being performed.”) (Giataganas, Para 0059; “the surgical data can include data streams (e.g., an array of intensity, depth, and/or RGB values) for a single image or …” The disclosed system can be configured to process a single image);
characterized in that, the memory element comprises:
a trained machine learning model (Giataganas, Para 0057; “The set of (learned) parameters can be stored as a trained machine-learning model data structure 130”), and
instructions for identifying and segmenting the surgical instrument from the captured image (Giataganas, Para 0174; “… include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to carry out aspects of the present invention.”),
the processing unit (Giataganas, Para 0131; “The surgical procedure support system 902 can include or may be coupled to the system 100.”) further configured for:
automatically identifying and segmenting the surgical instrument in the captured image, wherein the surgical instrument is automatically identified and segmented by a trained machine learning model (Giataganas, Para 0083; “the second machine learning model 400 further includes a decoder 406 that predicts and outputs scene segmentation 408 based on the feature space 404. The scene segmentation 408 provides semantic segmentation annotations […] to delineate with contours, masks, etc. one or more anatomical structures, surgical instruments, or other items (e.g., surgical staples, sutures, etc.) identified in the input window”);
creating a first mask comprising only the segmented surgical instrument (Giataganas, Para 0087; “the segmentation masks 505 can represent separate parts of the surgical instruments and/or anatomical structures”);
creating an overlay comprising augmentation information relating to one or more body parts visible in the captured image (Giataganas, Para 0087; “the segmentation masks 505 can represent separate parts of the surgical instruments and/or anatomical structures”), wherein the overlay includes at least one or more respective 3D representations of the one or more body parts (Giataganas, Para 0091; “The machine learning model(s) track the provided input across the multiple images (e.g., surgical video) of the surgical procedure … Additionally, separate machine learning models are used to reconstruct real-world measures of the anatomical structures, or on-the-fly 3D “models” (or partial surfaces) of the anatomical structures.”) and has at least some transparency (Giataganas, Para 0099; “Various visual attributes of the graphical overlay 502, such as colors, transparency, visual-pattern, line thickness, etc., can be adjusted.”);
combining the captured image with the overlay by applying the overlay over the captured image (Giataganas, Para 0004; “generating a visualization comprising the video of the surgical procedure, and a first graphical overlay at a location of the anatomical structure …”); and
applying the first mask over the captured image (Giataganas, Para 0094; “In the augmented visualization 507, the surgical instruments are marked using graphic overlays 502.”); and
adapting the one or more 3D representations in the overlay in real-time to a body reaction, a body function, or a response interaction with the surgical instrument (Giataganas, Para 0091; “The machine learning model(s) track the provided input across the multiple images (e.g., surgical video) of the surgical procedure … Additionally, separate machine learning models are used to reconstruct real-world measures of the anatomical structures, or on-the-fly 3D “models” (or partial surfaces) of the anatomical structures.”; Para 0122; “Further, machine learning is used for precise contour reconstruction and tracking of structures that are operated upon in the current step/phase of the surgical procedure.” These disclosures clearly indicate that 3D model of anatomical structure is reconstructed from surgical video, and is tracked so that when the structure is moved due to either interaction with surgical tool or physiologic movement such as respiration, the 3D model moves accordingly) (Giataganas, Para 0117; “to facilitate a real-time performance, the input window 320 is analyzed at a predetermined frequency, such as 5 times per second …”; Para 0119; “The graphical overlays 502 that are used to overlay the images 302 to represent the predicted and/or calculated features (surgical instruments, anatomical structures, calculated trajectories, etc.) are accordingly adjusted”; these disclosures explicitly disclose that the processing is performed in real time).
Giataganas does not clearly and explicitly disclose wherein:
storing augmentation information relating to one or more body parts visible in the captured image,
the segmented one or more non-organic objects being represented by pixels of the captured image, and
first applying the overlay over the captured image, followed by applying the first mask over the combined image.
Lang in the same field of endeavor discloses
storing augmentation information relating to one or more body parts visible in the captured image (Lang, Column 78, Lines 16-21; “The live surfaces of the patient that are detected and generated in this manner can be matched or aligned with virtual surfaces of the patient, for example obtained preoperatively using an imaging test such as x-ray imaging, ultrasound, CT or MRI or any other technique known in the art.”),
the segmented one or more non-organic objects being represented by pixels of the captured image (Lang, Column 249, Lines 51-59; “In the example of FIG. 24D, a computer processor can track the physical screw driver 1021 … using direct video detection …, e.g. with an image capture … system …”. The disclosed physical screw driver 1021 as shown in Fig. 24D is a cropped group of pixels from captured video frames or images), and
first applying the overlay over the captured image, followed by applying the first mask over the combined image (Lang, Column 248, Line 62 to Column 249, Line 6, “FIGS. 24A-E provide illustrative, non-limiting examples of one or more augmented reality HMD displays including a virtual user interface 990 … the virtually displayed spine 993 of the patient, e.g. using a 3D model generated based on a pre-operative CT scan or an intra-operative O-arm scan.”. As Fig. 24A shows, the disclosed virtually displayed spine 993, corresponding to the claimed overlay, is displayed in the interface at early time point, then at a later point as demonstrated by Fig. 24D, the disclosed physical screw driver 1021 is displayed over the spine 993).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Giataganas, as suggested by Lang, in order to store augmentation information relating to visible body parts, display a surgical tool with its real appearance and on top of other tissues at a surgical site. One of ordinary skill in the art would have been motivated to make the modification for the benefit of avoiding tissue injury in a surgical procedure by simultaneously and efficiently visualizing surgical tool and body part at a surgical site.
With regard to Claim 14, Giataganas and Lang disclose the system according to Claim 20. Giataganas further discloses wherein the image capturing device is an endoscope (Giataganas, Abstract; “A surgical action to be performed during a surgical procedure […] An image/video capture device such as an endoscope, a wearable camera, a stationary camera, etc., can be used to capture the image(s)”).
With regard to Claim 15, Giataganas and Lang disclose the system according to Claim 14. Giataganas further discloses wherein the endoscope is a stereoscopic endoscope (Giataganas, Para 0109; “The machine learning model is trained using a pair of stereo frames that are captured using a stereo image capture device”).
With regard to Claim 16, Giataganas and Lang disclose the system according to Claim 20. Giataganas further discloses wherein the system comprises a display unit for displaying a masked video feed produced by the image processing unit and a control unit for controlling the robotic arm, said display unit and said control unit being integrated together (Giataganas, Para 0102; “the user feedback can further include generating and displaying a graphical overlay 502 on the surgical view […] the user feedback can be integrated into a robotic workflow in response to the predictions described herein. For example, operating parameters of one or more surgical instruments are adjusted”).
With regard to Claim 17, Giataganas and Lang disclose the system according to Claim 20. Giataganas further discloses comprising a display unit for displaying the final composite image generated by the processing unit (Giataganas, Para 0067; “the augmented display can be presented at a non-wearable user device, such as at a computer or tablet”).
With regard to Claim 18, Giataganas and Lang disclose the system according to Claim 17, but as discussed above do not explicitly and clearly disclose wherein said display unit is a stereoscopic display.
Lang further discloses wherein said display unit is a stereoscopic display (Lang, Column 2, Lines 61-63; “… the 3D stereoscopic view is displayed by the at least one head mounted display”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Giataganas and Lang, as further suggested by Lang, in order to use a stereoscopic display as said display unit. One of ordinary skill in the art would have been motivated to make the modification for the benefit of visualizing depth information of all images and masks at surgical site for better surgical guidance.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LEI ZHANG whose telephone number is (571)272-7172. The examiner can normally be reached Monday-Friday 8am-5pm E.T..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pascal Bui-Pho can be reached at (571) 272-2714. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/L.Z./ Examiner, Art Unit 3798
/PASCAL M BUI PHO/ Supervisory Patent Examiner, Art Unit 3798
