DETAILED ACTION
This action is responsive to the Amendments and Remarks received 05/05/2026 in which claims 5, 6, and 16–18 are cancelled, claims 1–4, 7–15, and 19 are amended, and claims 20 and 21 are added as new claims.
Response to Arguments
In view of the amendments to claim 19, the rejection under 35 U.S.C. 101 is withdrawn. Remarks, 6.
In view of the amendments to the claims the rejections under 35 U.S.C. 112(a) and 112(b) are withdrawn. Remarks. 6.
On page 7 of the Remarks, Applicant contends the combination of Hasan and Chan fails to teach or suggest the feature of using an image representing a different view obtained from a different imaging device for inpainting occlusions. Examiner disagrees. The combination of the teachings of Hasan, combined with those of Chan, would teach or suggest the averred subject matter. The skilled artisan knows that taking background data from another perspective to use to inpaint occluded background regions in another image is a known technique for inpainting. Indeed, Hasan teaches such a technique. The skilled artisan knows it does not matter what camera takes the second perspective containing the missing information or even that there be a different perspective at all. The skilled artisan has within their basic knowledge that missing background information can be obtained (1) at a different perspective by a different camera; (2) at a different perspective with the same camera; or (3) taken at the same perspective with the same camera but while the occluding object is not present (e.g. not previously in the way). To leave no doubt about these most obvious variants of how to obtain occluded background image data, the rejection cites to the teachings of Chan, which clearly links for the reviewer the concepts of inpainting and occlusion. As Applicant correctly points out, the art describes missing information as “holes.” Just like that taught in Chan, Applicant obtains the missing background information (i.e. holes) from a different perspective view from different imaging device. When combined with the teachings of Hasan, the skilled artisan would understand that Hasan’s different perspective obtained by the same camera could be substituted by obtaining the different perspective from a different camera. There is no reasonable argument that, in this field one would find a true patentable distinction between capturing different camera perspectives by either moving a single camera to the new perspective or setting up an additional camera at a corresponding second perspective. Accordingly, Examiner is unpersuaded of patentability.
Claim Objections
Claims 7 and 10 are improperly dependent on cancelled claim 5. Appropriate correction is required. Examiner interprets this as a typographical omission wherein Applicant overlooked changing the dependency to the independent claim. Obviously, a 35 U.S.C. 112(d) rejection could also be made for improper dependency, but Examiner made a judgment call that this claim objection is better tailored for correcting the typographical oversight as opposed to citing the legal requirements of proper dependency under 35 U.S.C. 112(d).
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 19 and 20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Hasan et al., “Segmentation and Removal of Surgical Instruments for Background Scene Visualization from Endoscopic / Laparoscopic Video,” Medical Imaging 2021: Image-Guided Procedures, Robotic Interventions, and Modeling. SPIE, 2021. 5. 55–61 (herein “Hasan”).
Claim 19 lists the same elements as claim 20, but is drawn to a CRM rather than the system. Therefore, the rationale for the rejection of claim 20 applies to the instant claim.
Regarding claim 20, Hasan discloses a system comprising at least one medical imaging device, at least one image processing device, and at least one display device, the system performing: recording at least one digital image in a field of view of the medical imaging device; segmenting the at least one digital image into at least one foreground region; selecting the background region that is obscured by the at least one foreground region; overlaying the background region onto the at least one foreground region; and displaying the at least one digital image comprising the background region overlaying the at least one foreground region (Examiner interprets, consistent with the last wherein clause of original claim 1 and Applicant’s published paragraph [0009], a foreground surgical instrument that can be made partially or fully transparent so that the surgeon can see the underlying tissue that is being occluded by the foreground instrument; Consistent with paragraphs [0009]–[0012] of the published Specification, Examiner finds the claimed invention is drawn to segmenting out and making transparent a surgical instrument so that the underlying tissue can be viewed by the surgeon without occlusion using AI in-painting; Hasan, e.g. Abstract and Section 2: discloses a method for surgical tool segmentation wherein a medical image is recorded, the region representing the surgical tool is segmented, and then that region is in-painted with the background tissue; Examiner notes Applicant admits in paragraph [0050] of the published Specification that in-painting algorithms are an example of artificial intelligence algorithms).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1–4, 7, 9, and 12–15 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Hasan et al., “Segmentation and Removal of Surgical Instruments for Background Scene Visualization from Endoscopic / Laparoscopic Video,” Medical Imaging 2021: Image-Guided Procedures, Robotic Interventions, and Modeling. SPIE, 2021. 5. 55–61 (herein “Hasan”) and Chan (US 2017/0188002 A1).
Regarding claim 1, the combination of Hasan and Chan teaches or suggests a method for displaying a digital image comprising at least one foreground region overlaid with a background region (Examiner notes “overlay with” may be poor word choice, but is interpreted, consistent with the last wherein clause of original claim 1 and Applicant’s published paragraph [0009], as a foreground surgical instrument that can be made partially or fully transparent so that the surgeon can see the underlying tissue that is being occluded by the foreground instrument; In other words, it is the occluded “background region” that is being brought to the front that is being “overlaid” on the foreground surgical instrument so as to remove from view the occluding surgical instrument), comprising: recording, by a medical imaging device, at least one digital image in a field of view of the medical imaging device; segmenting, by an image processing device, the at least one digital image into the at least one foreground region; selecting, by the image processing device, the background region that is obscured by the at least one foreground region; overlaying, by the image processing device, the background region onto the at least one foreground region; and displaying, by a display device, the at least one digital image comprising the background region overlaying the at least one foreground region; and wherein the background region is extracted from at least one image recorded from a different perspective by a second medical imaging device (Consistent with paragraphs [0009]–[0012] of the published Specification, Examiner finds the claimed invention is drawn to segmenting out and making transparent a surgical instrument so that the underlying tissue can be viewed by the surgeon without occlusion using AI in-painting; Hasan, e.g. Abstract and Section 2: discloses a method for surgical tool segmentation wherein a medical image is recorded, the region representing the surgical tool is segmented, and then that region is in-painted with the background tissue; Examiner notes Applicant admits in paragraph [0050] of the published Specification that in-painting algorithms are an example of artificial intelligence algorithms; Hasan, Section 2.2: teaches the previous frames are of a perspective that does not include occluded portions of the background; In other words, the skilled artisan would interpret that the movement of the camera within the body (endoscope) means that reference frames along the temporal axis represent different viewpoint perspectives such that they may be matched and stitched together (or in-painted) to achieve the occluded background; A moving camera takes different perspective images along the temporal axis; see also Hasan, Section 3: teaching camera or anatomy movement allowing for differing perspectives; While Hasan teaches differing perspectives from a same camera moving to a new vantage point is alone sufficient to suggest the using a separate camera to get the other viewpoint, to expedite prosecution, the rejection additionally relies on the teachings of Chan; Chan, ¶ 0011: teaches that for stereo videos, data from another view can be used for inpainting holes due to disocclusion (i.e. uncovering covered background pixels)).
One of ordinary skill in the art, before the effective filing date of the claimed invention, would have been motivated to combine the elements taught by Hasan, with those of Chan, because both references are drawn to the same field of endeavor such that one wishing to practice the art of inpainting would be led to their relevant teachings and because Chan teaches that inpainting to fill the holes addressed by Hasan can be accomplished using another view, such as that provided by a stereoscopic camera system. Therefore, the combination is a mere combination of prior art elements, according to known methods, to yield a predictable result. This rationale applies to all combinations of Hasan and Chan used in this Office Action unless otherwise noted.
Regarding claim 2, the combination of Hasan and Chan teaches or suggests the method of claim 1, wherein the medical imaging device is an endoscope camera or an exoscope camera (Examiner notes it doesn’t matter what kind of imaging device took the picture; Hasan, Title: teaches the images can come from endoscopic video).
Regarding claim 3, the combination of Hasan and Chan teaches or suggests the method of claim 1, wherein the background region is extracted from at least one image which was recorded before the at least one digital image and contains the background region concealed by the at least one foreground region (Examiner interprets this as saying the occluded information is contained in a prior image; Hasan, Abstract: explains the tool removal algorithm in-paints using instrument-free reference frames to fill in the background tissue underneath).
Regarding claim 4, the combination of Hasan and Chan teaches or suggests the method of claim 1, wherein the background region is created by the image processing device with the aid of generative image processing methods based on artificial intelligence (Examiner notes Applicant admits in paragraph [0050] of the published Specification that in-painting algorithms are an example of artificial intelligence algorithms; Hasan, Abstract and Section 2: teaches inpainting to fill in occluded areas of the image with background tissue using reference frames).
Regarding claim 7, the combination of Hasan and Chan teaches or suggests the method of claim 5, wherein the image recorded from a different perspective is recorded following a movement of the medical imaging device (Hasan, Section 3: teaching camera or anatomy movement allowing for differing perspectives).
Regarding claim 9, the combination of Hasan and Chan teaches or suggests the method of claim 7, wherein the movement is implemented perpendicular to the main orientation of the information (Hasan teaches an endoscope having any type of movement possible; Applicant did not invent perpendicular movement with respect to an image and Examiner further finds images are taken perpendicular to a scene).
Regarding claim 12, the combination of Hasan and Chan teaches or suggests the method of claim 1, wherein the at least one foreground region is presented transparently, partly transparently, and/or as a contour (Hasan, e.g. Section 1: teaches surgical instruments made transparent; Hasan, Fig. 1: teaches partly transparent and outlined (i.e. contoured)).
Regarding claim 13, the combination of Hasan and Chan teaches or suggests the method of claim 12, wherein the type of presentation of the at least one foreground region depends on the reliability of the background region used (Hasan, Figs. 4 and 5: demonstrate that Poisson blended inpainting works well for an outlined tool (4b) whereas inpainting without blending is indicated for a transparent tool (5b)).
Regarding claim 14, the combination of Hasan and Chan teaches or suggests the method of claim 13, wherein the reliability of the background region used is determined by way of a plausibility check based on predefined metrics (Examiner interprets this limitation consistent with original claim 15 and paragraph [0015] of Applicant’s published Specification, which explains the plausibility check is a reliability metric that can be determined by comparing the artificial image with the real image; Examiner notes a DICE score or DICE coefficient measures similarity between two sets of data and is widely used in medical image analysis; Hasan, Section 3: describes several reliability metrics including DICE score for measuring inpainting vs. ground truth; Examiner notes reliability metrics for inpainting is widely represented in the art).
Regarding claim 15, the combination of Hasan and Chan teaches or suggests the method of claim 1, wherein the reliability of the overlay is determined by comparing the at least one foreground region with the background region (Examiner interprets this limitation consistent with original claim 15 and paragraph [0015] of Applicant’s published Specification, which explains the plausibility check is a reliability metric that can be determined by comparing the artificial image with the real image; Examiner notes a DICE score or DICE coefficient measures similarity between two sets of data and is widely used in medical image analysis; Hasan, Section 3: describes several reliability metrics including DICE score for measuring inpainting vs. ground truth; Examiner notes reliability metrics for inpainting is widely represented in the art).
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Hasan, Chan, and Kojo (US 2021/0169584 A1).
Regarding claim 8, the combination of Hasan, Chan, and Kojo teaches or suggests the method of claim 7, wherein the movement of the medical imaging device is implemented by a motorized controller (Kojo, ¶ 0050: teaches an “endoscope can have various motorized movement”; see also Itkowitz, used to reject other claims, ¶ 0064, teaching a motorized drive system for an endoscope).
One of ordinary skill in the art, before the effective filing date of the claimed invention, would have been motivated to combine the elements taught by Hasan and Chan, with those of Kojo, because both Hasan and Kojo are drawn to the same field of endeavor such that one wishing to practice the art of endoscopic imaging would be led to their relevant teachings and because Kojo teaches that endoscopes used by Hasan can include motorized controllers. Therefore, the combination is a mere combination of prior art elements, according to known methods, to yield a predictable result. This rationale applies to all combinations of Hasan, Chan, and Kojo used in this Office Action unless otherwise noted.
Claims 10 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Hasan, Chan, and Itkowitz (US 2025/0288367 A1).
Regarding claim 10, the combination of Hasan, Chan, and Itkowitz teaches or suggests the method of claim 5, wherein the medical imaging device comprises two objective lenses arranged next to one another and respectively used for recording a right and a left image, the at least one digital image is a three-dimensional image, and the image information, as two-dimensional information, is extracted from the left image and/or the right image (Itkowitz, e.g. ¶¶ 0069 and 0075: teaches stereoscopic endoscopes utilizing left and right images).
One of ordinary skill in the art, before the effective filing date of the claimed invention, would have been motivated to combine the elements taught by Hasan, with those of Itkowitz, because both references are drawn to the same field of endeavor such that one wishing to practice the art of endoscopic imaging would be led to their relevant teachings and because Itkowitz teaches that endoscopes used by Hasan can include stereoscopic imaging capability. Therefore, the combination is a mere combination of prior art elements, according to known methods, to yield a predictable result. This rationale applies to all combinations of Hasan and Itkowitz used in this Office Action unless otherwise noted.
Regarding claim 11, the combination of Hasan, Chan, and Itkowitz teaches or suggests the method of claim 10, wherein background region which cannot be extracted from the left image and/or the right image is, at least one of, (a) extracted from at least one image which was recorded before the at least one digital image and contains the background region concealed by the at least one foreground region or (b) created by the image processing device with the aid of generative image processing methods based on artificial intelligence (see treatment of claims 3 and/or 4).
Claim 21 is rejected under 35 U.S.C. 103 as being unpatentable over Hasan and Kim (US 2022/0343613 A1).
Regarding claim 21, the combination of Hasan and Kim teaches or suggests the system of claim 20, wherein the image processing device implements a discriminator to determine the reliability of the at least one foreground region (Examiner interprets this limitation consistent with original claim 15 and paragraph [0015] of Applicant’s published Specification, which explains the plausibility check is a reliability metric that can be determined by comparing the artificial image with the real image; Examiner notes a DICE score or DICE coefficient measures similarity between two sets of data and is widely used in medical image analysis; Hasan, Section 3: describes several reliability metrics including DICE score for measuring inpainting vs. ground truth; Examiner notes reliability metrics for inpainting is widely represented in the art; Kim, ¶¶ 0010 and 0013: teaches a deletion region in an image and inpainting that region using second information being performed through a deep learning network constituted by a generator and a discriminator).
One of ordinary skill in the art, before the effective filing date of the claimed invention, would have been motivated to combine the elements taught by Hasan, with those of Kim, because both references are drawn to the same field of endeavor such that one wishing to practice the art of deleting an image portion and inpainting would be led to their relevant teachings and because Kim teaches that inpainting used by Hasan can include inpainting achieved by a deep learning network constituted by a generator and a discriminator. Therefore, the combination is a mere combination of prior art elements, according to known methods, to yield a predictable result. This rationale applies to all combinations of Hasan and Kim used in this Office Action unless otherwise noted.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Newton (US 2010/0271466 A1) teaches hole filling is a known technique for dealing with occlusions resulting from stereoscopic imaging wherein the technique utilizes available information in an image for reconstructing the occluded visual information by referencing information in preceding and/or subsequent images (¶ 0009). The publication also teaches that occluded portions of the image can be segmented for determining the occlusion region (¶ 0011).
Rao (US 2022/0253592 A1) teaches solid or semi-transparent outlines or shading overlaid on a medical image (¶ 0291).
Bouhnik (US 2024/0324852 A1) teaches a neural network trained to perform inpainting (¶ 0137).
Valli (US 2021/0136354 A1) teaches hole filling and inpainting are synonymous (¶ 0350).
Karande (US 2017/0024864 A1) teaches removing an object from a multi-view image and inpainting to fill the holes by using the multi-view images (¶ 0030).
Mann (US 2024/0193835 A1) teaches a discriminator used to estimate confidence in the quality of the generated content (¶ 0078) and teaches the neural network can perform inpainting (¶ 0102).
Koreeda et al., “Virtually transparent surgical instruments in endoscopic surgery with augmentation of obscured regions,” International Journal of Computer Assisted Radiology and Surgery 11(10), 1927–1936 (2016). [PubMed: 27038964]. Teaches making surgical instruments transparent.
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Michael J Hess whose telephone number is (571)270-7933. The examiner can normally be reached Mon - Fri 9:00am-5:30pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, William Vaughn can be reached on (571)272-3922. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8933.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL J HESS/Examiner, Art Unit 2481
Prosecution Insights