Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
With respect to the newly amended limitations, it is noted that the Specification does not explicitly describe a 3D model being displayed at “position offset from the corresponding target organ or structure…such that the rendered representation does not coincide with the target organ or structure.” However, page 7, lines 10-25 of the Specification describe that the orientation, size, and/or location may be chosen such that the 3D model can be viewed from different perspectives, enlarged to view details, and/or located so as to minimize obstruction of the laparoscopic image (or in each of these cases, can be made to match the orientation, size, and/or location of the organ in the laparoscopic image). Further, claim 1 requires that the 3D model hovers above organs, which could seem to conflict with the 3D model being offset from the organs. Accordingly, for the purposes of examination, these limitations will be interpreted in a manner commensurate with the specification, such that the 3D model is displayed with the laparoscopic image, but not aligned with the organ in the laparoscopic image.
Claim Objections
Claims 1 and 9 are objected to because of a minor informality: each of these claims recites “configured to adding” which should be amended to read --configured to add--. Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3 and 5-10 are rejected under 35 U.S.C. 103 as being unpatentable over Zang et al. (U.S. Patent Application Publication No. 2022/0366649), referred herein as Zang, in view of Shademan et al. (U.S. Patent Application Publication No. 2023/0240764), referred herein as Shademan, and further in view of Popovic et al. (U.S. Patent Application Publication No. 2017/0007350), referred herein as Popovic.
Regarding claim 1, Zang teaches a laparoscopic image manipulation method (figs 2 and 3), the method comprising:
capturing a video stream of laparoscopic images of a patient using a laparoscope inserted into the patient during a laparoscopic procedure (paragraph 19, lines 1-7; paragraph 27; paragraph 39, lines 12-21; a laparoscopic video stream is captured of a patient during a procedure),
feeding the captured laparoscopic images to a video processor configured to adding additional information as overlay over the captured laparoscopic images (paragraph 19, lines 1-7; paragraph 27; paragraph 28, lines 1-5 and 10-14; paragraph 32, lines 1-6; paragraph 39, lines 12-21; additional information is added to overlay the laparoscopic video stream),
producing a composite image by rendering a representation of a 3D model of a target organ or structure at an arbitrary scale factor compared to the target organ or structure using a renderer and merging the rendered representation of the 3D model with the captured laparoscopic image to display the 3D model as overlaying organs visible in the laparoscopic image, and displaying the composite image on a monitor (paragraph 19, lines 1-7; paragraphs 27 and 31; paragraph 32, lines 1-6; paragraph 33, lines 1-11; paragraph 39, lines 12-37; a 3D model of a target organ/structure is rendered at an arbitrary scale and merged with the laparoscopic image to produce and display a composite image).
Zang teaches adjusting the position and orientation of the 3D model (see, for example, paragraph 20, the last 20 lines; paragraph 39, the last 12 lines), but does not explicitly teach rendering the 3D model at a different scale than the target organ or structure, wherein the rendered representation of the 3D model is displayed at a position offset from the corresponding target organ or structure in the image such that the rendered representation does not coincide with the target organ or structure.
However, in a similar field of endeavor, Shademan teaches a medical image manipulation method comprising capturing a video stream of medical images, such as endoscopic images, of a patient during a medical procedure, feeding the images to a video processor to add additional information to the images, and producing a composite image by rendering a representation of a 3D model of a target organ or structure using a renderer (figs 5A and 5B, endoscopic medical image 502, 3D model 510 of organ 504, composite image 502; paragraph 2; paragraph 50, lines 1-8 and the last 5 lines; paragraph 79, lines 1-12), and further comprising rendering the 3D model at a different scale than the target organ or structure, wherein the rendered representation of the 3D model is displayed at a position offset from the corresponding target organ or structure in the image such that the rendered representation does not coincide with the target organ or structure (figs 5B, 3D model 510; paragraph 50, lines 1-8; paragraph 52; paragraph 54, lines 1-12; the 3D model is rendered at a different scale from, and at a position offset from, the endoscopic image of the corresponding target organ such that it does not coincide with the target organ or structure).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the scaling and position offset of the 3D model, as taught by Shademan, with the 3D model manipulation of Zang because this helps improve the accuracy and efficiency of interacting with the images during the procedure by enabling independent manipulation of the 3D model and avoiding obstructing the view of the live endoscopic medical image (see, for example, Shademan, paragraph 23; paragraph 40, the last 16 lines; paragraph 46, the last 6 lines).
Although Zang in view of Shademan teaches overlaying the 3D models on the laparoscopic images (see, for example, Zang, paragraph 33, among others), Zang in view of Shademan does not explicitly teach displaying the 3D model as hovering above organs in the visible image.
However, in a similar field of endeavor, Popovic teaches a method for capturing laparoscopic images of a patient, feeding those images to a processor to add additional information as overlay over the images, producing a composite image by rendering a representation of a 3D model of a target organ or structure, and displaying the composite image (figs 2, 5-8, and 10; paragraphs 33 and 34; paragraph 36; paragraph 37, lines 1-6; paragraph 59), and further comprising displaying the 3D model as hovering above organs visible in the image (figs 5, 7, and 8; paragraph 41, lines 1-20; paragraph 44, lines 1-15; paragraphs 45, 48, and 50; paragraph 61, lines 1-4).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to display the 3D model hovering above the target, as taught by Popovic, with the composite image of Zang in view of Shademan, because this helps to reveal portions of the organ structure that may not be visible in the image, and conveys important location and depth information about such structures, thereby increasing procedural safety and situational awareness of the user, which is particularly important in endoscopic and laparoscopic procedures (see, for example Popovic, paragraphs 24-26; paragraph 44, lines 1-10; paragraphs 52 and 59).
Regarding claim 2, Zang in view of Shademan, further in view of Popovic teaches the laparoscopic image manipulation method of claim 1, wherein at least one of an orientation, a size and a location of the rendering of the 3D model inside the composite image is controlled by a manual controller connected to the renderer (Zang, paragraph 28, lines 1-5; paragraph 33, lines 1-4 and the last 8 lines; paragraph 35, the last 10 lines; paragraph 39, lines 22-37; paragraph 52, the last 12 lines).
Regarding claim 3, Zang in view of Shademan, further in view of Popovic teaches the laparoscopic image manipulation method of claim 1, wherein information of each individual frame of the video stream of laparoscopic images are input to the renderer, the renderer is configured to render the representation of the 3D model according to the input information of the individual frames (Zang, paragraph 19, lines 1-7; paragraph 27, lines 1-5 and the last 3 lines; paragraph 33, lines 1-15; paragraph 37, lines 1-9; paragraph 39, lines 12-21).
Regarding claim 5, Zang in view of Shademan, further in view of Popovic teaches the laparoscopic image manipulation method of claim 1, wherein the 3D model of the target organ or structure is derived from one or more of prior CT and MRI scan data of the patient (Zang, paragraph 19, lines 1-7; paragraph 25, lines 1-3; paragraph 32, lines 4-8).
Regarding claim 6, the limitations of this claim substantially correspond to the limitations of claim 1 (except for the at least one processor comprising hardware, which is disclosed by Zang, paragraph 28, lines 1-14); thus they are rejected on similar grounds.
Regarding claim 7, Zang in view of Shademan, further in view of Popovic teaches the laparoscopic image manipulation system of claim 6, further comprising a manual controller having a data link to the at least one processor, (Zang, paragraph 28, lines 1-25; paragraph 30), the at least one processor is further configured to change at least one of an orientation, a size and a location of the rendering of the 3D model of the target organ or structure inside the composite image in response to signals from the manual controller (Zang, paragraph 28, lines 1-5; paragraph 33, lines 1-4 and the last 8 lines; paragraph 35, the last 10 lines; paragraph 39, lines 22-37; paragraph 52, the last 12 lines).
Regarding claim 8, Zang in view of Shademan, further in view of Popovic teaches the laparoscopic image manipulation system of claim 6, further comprising a frame grabber configured to capture the laparoscopic video stream frame-by-frame and to produce one or more composite images by merging of the rendered representation of the 3D model with the captured laparoscopic images frame-by-frame (Zang, paragraph 19, lines 1-7; paragraph 27, lines 1-5 and the last 3 lines; paragraph 33, lines 1-15; paragraph 37, lines 1-9; paragraph 39, lines 12-21).
Regarding claim 9, Zang in view of Shademan, further in view of Popovic teaches the laparoscopic image manipulation system of claim 6, wherein the at least one processor is further configured to: capture the video stream of the laparoscopic images of the patient using the laparoscope inserted into the patient during a laparoscopic procedure (Zang, paragraph 19, lines 1-7; paragraph 27; paragraph 39, lines 12-21), and feed the captured laparoscopic images to the at least one processor to merge the rendered representation of the 3D model with the captured laparoscopic images (Zang, paragraph 19, lines 1-7; paragraph 27; paragraph 28, lines 10-14; paragraph 32, lines 1-6; paragraph 39, lines 12-21).
Regarding claim 10, the limitations of this claim substantially correspond to the limitations of claim 1 (except for the computer-readable medium, which is disclosed by Zang, paragraph 29); thus they are rejected on similar grounds.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Zang, in view of Shademan, further in view of Popovic, and further in view of Zhao et al. (U.S. Patent Application Publication No. 2009/0088634), referred herein as Zhao.
Regarding claim 4, Zang in view of Shademan, further in view of Popovic teaches the laparoscopic image manipulation method of claim 3, wherein the information of each individual frame comprises one or more identified features of each frame (Zang, paragraph 37, lines 1-15).
Although resolution and frame rate are inherent features of any video stream (including Zang’s laparoscopic video stream), Zang in view of Shademan, further in view of Popovic does not explicitly teach inputting one or more of an image resolution and a frame rate.
However, in a similar field of endeavor, Zhao teaches a laparoscopic image manipulation method comprising capturing a video stream of laparoscopic images of a patient and overlaying 3D models of structures with the video to produce and display a composite image by a renderer (paragraph 32; paragraph 39, the last 7 lines; paragraph 46, the last 7 lines; paragraph 191, the last 4 lines; paragraph 194, lines 1-18), wherein information for individual frames is input to the renderer, and wherein the information comprises one or more of an image resolution and a frame rate (paragraph 176, lines 1-5; paragraph 179, lines 1-5; paragraph 180, lines 1-5).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the image resolution input of Zhao with the frame processing of Zang in view of Shademan, further in view of Popovic because the quality of the rendered images depends on the particular features of each frame, and taking into consideration each frame’s image resolution or frame rate has a direct impact on, and can help increase the quality of, the combined image that is ultimately produced (see, for example, Zhao, paragraph 174; paragraph 175, lines 1-8).
Response to Arguments
Applicant’s arguments with respect to the 103 rejections have been fully considered, but they are not persuasive.
On page 7 of the Remarks, with respect to the 103 rejection of claim 1, the Applicant argues that 1) Popovic discloses an overlay that highlights internal anatomy / invisible subsurface structures of an organ, but does not disclose rendering a 3D model of the organ itself, and 2) Popovic’s intent is to align internal anatomy with visible anatomy, and thus does not teach the amended limitation describing that the 3D model is displayed at a position offset from the target organ in the laparoscopic image. The Examiner respectfully disagrees with these arguments, in part.
Regarding the first argument, it is respectfully submitted that the claim does not explicitly require a 3D model of the organ itself. Instead, the claim requires a 3D model “of a target organ or structure” and then rendering it to “display the 3D model as hovering above organs visible in the laparoscopic image.” Popovic discloses this in a number of ways, and one example is described in the Remarks themselves, which describe Popovic as displaying invisible subsurface structures of an organ on the laparoscopic image of the organ. Indeed, this appears to be directly analogous to an example given in Applicant’s specification, where it states that “the 3D model of the target organ or structure is displayed directly on the surgical main monitor 22 and can be used in a variety of ways to support intraoperative orientation of the surgeon. For example, it can be used to visualize subsurface structures that cannot be seen in the laparoscopic video image.” (Specification, page 7, lines 28-31). It is also separately noted that in any case, Zang teaches 3D models of an organ. Thus, it is respectfully submitted that this does not represent a patentable distinction in the claims.
Regarding the second argument, this argument is moot in view of the new grounds of rejection presented above. It is agreed that the previously applied art does not explicitly teach the amended portion of claim 1; but it respectfully submitted that the currently cited art teaches these limitations, as discussed above.
Conclusion
The following prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Devam (U.S. Patent Application Publication No. 2016/0249989); Reality-augmented morphological procedure.
Devam (U.S. Patent Application Publication No. 2019/0206134); Systems and methods for rendering immersive environments.
Jaskola (U.S. Patent No. 12,682,586); Laparoscopic image manipulation method and system.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID T WELCH whose telephone number is (571)270-5364. The examiner can normally be reached Monday-Thursday, 8:30-5:30 EST, and alternate Fridays, 9:00-2:30 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached at 571-272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
DAVID T. WELCH
Primary Examiner
Art Unit 2613
/DAVID T WELCH/Primary Examiner, Art Unit 2613
Prosecution Insights