DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This Office Action is in response to Applicant’s amendment filed 08/06/2026 which has
been entered and made of record. Claims 1-4, 9-13, and 16-18 have been amended. Claims 5-6, 14-15, and 19-20 have been cancelled. No claim newly added. Claims 1-4, 7-13 and 16-18 are pending in the application. Applicant’s amendments to the Claims have overcome each and every objection previously set forth in the Non-Final Office Action mailed May 12th, 2026.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1, 11 and 16 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument (due to applicant’s arguments directed to newly amend limitation(s) which is addressed by new prior art presented in this Office Action).
The arguments regarding dependent claims for the virtue of their dependency are moot
because the independent claims are not allowable.
Claim Rejections - 35 USC § 112
Previous 35 U.S.C. 112(b) rejection for claim 9 has been withdrawn.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 4, 7-10, 11, 13, 16 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Akimoto et al. (U.S. Patent Application Publication No. 2018/0214006), hereinafter referenced as Akimoto, in view of Mori et al. (A Method for Detecting Unobserved Regions in Virtual Endoscopy System), hereinafter referenced as Mori, and Zingaretti (U.S. Patent No. U.S. Patent No. 11,191,423), hereinafter referenced as Zingaretti.
Regarding claim 1, Akimoto teaches An image processing apparatus comprising: one or more processors comprising hardware, the one or more processors being configured to: (paragraph 67 teaches “image display apparatus configured to display an endoscope image generated by the video processor” and fig. 1 shows apparatus 7 with processor 4 comprising hardware); acquire a plurality of two-dimensional images of an object captured by an endoscope (paragraph 346 teaches “from 2D images inputted”, paragraph 113 teaches “3D shape data structuring section 42a to which two-dimensional image data is inputted generates 3D shape data corresponding to two-dimensional image data picked up (observed) by the image pickup section 25 of the endoscope” and paragraph 117 teaches “image update processing section 42o performs processing of updating a 3D model image generated by the 3D shape data structuring section 42a, based on change of two-dimensional data along with the insertion operation of the endoscope”); this shows plural/multiple/changing 2D image data/images acquired/captured/observed by endoscope; generate a partial three-dimensional model of the object (paragraph 121 teaches “a 3D model image corresponding to any region included in an endoscope image observed in the past from start of the 3D model image generation to the current observation state of the distal end portion 15 is displayed on the monitor 8. The display region of the 3D model image displayed on the monitor 8 increases with time elapse”); this shows partial 3D model (partial since region of 3D model increases with time elapse) of object generated based on the aforementioned images and corresponding to any region means this partial 3D model would be representing predetermined region of interest from the below combination.
However, Akimoto fails to explicitly teach determine whether a predetermined region of interest of the object exists in each of the plurality of two-dimensional images; generate, based on two or more images selected from the plurality of two- dimensional images in which the predetermined region of interest is determined to exist, a partial three-dimensional model corresponding to the predetermined region of interest of the object;
However, Zingaretti teaches determine whether a predetermined region of interest of the object exists in each of the plurality of two-dimensional images (Zingaretti, abstract teaches “provide not only a conventional real time image of the view obtained by an endoscope, but in addition, a near real time 3D model and/or a 2D flattened image of an interior surface of an organ, which model and image may be processed using AI software to highlight potential tissue abnormalities”, col. 1, lines 20-21 teach “Such abnormalities include polyps and various types of adenomas”, col. 5, lines 51-52 teaches “if a screening colonoscopy performed at the age of 50 finds no polyps”, col. 6 last line and col. 7 first two lines teach “artificial intelligence module, e.g., that automatically detects for polyps and other features in the real-time images”); abnormalities such as polyps are considered predetermined region of interest (ROI), and artificial intelligence (AI) model automatically detecting such as well as scenarios where no polyps are found shows a determination of whether this predetermined ROI exists in each of the real-time 2D images; generate, based on two or more images selected from the plurality of two- dimensional images in which the predetermined region of interest is determined to exist, a partial three-dimensional model corresponding to the predetermined region of interest of the object (Zingaretti, col. 12, lines 43-47 teach “window 51 also presents information about the presence of polyps determined by the artificial intelligence module. Window 53 displays the real-time incremental image of the 3D model and/or 2D flat image of the colon”); incremental image of 3D model shows generating partial 3D model which corresponds to information about presence of polyps thus corresponding to predetermined ROI of object/organ, also, this is based on the plurality of 2D images since that is what is aforementioned above to find where the predetermined ROI is determined to exist. Zingaretti is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of partial/incremental 3D models and determinations of predetermined ROI existing. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Akimoto's invention with the medical image determination and generation techniques of Zingaretti to ensure improving endoscopy procedures by enhancing the endoscopists' viewing, navigation and interpretation of the interior surface of an organ being examined (Zingaretti, col. 8, lines 18-20). This leads to a better user experience.
However, the combination of Akimoto and Zingaretti fails to explicitly teach and detect, based on the partial three-dimensional model, an unobserved region of the object that is not captured by the endoscope (although, paragraph 142 teaches “a boundary Ba between a structured region corresponding to an observation region that is observed and an unobserved region, and the boundary Ba portion is displayed in a color different from the color of the structured region”).
However, Mori explicitly teaches and detect, based on the partial three-dimensional model, an unobserved region of the object that is not captured by the endoscope (Mori, pages 136-137, section 3.1 (a) teaches “Detection of unobserved regions… target organ is represented using a set of polygons (triangular patches). The basic idea of the proposed method is to record triangle patches that were presented on the display terminal during the fly-through and to calculate the unobserved regions from the results obtained by recording unobserved regions. The method consists of three parts: (a) recording of observed triangle patches, (b) calculation of unobserved regions”); this shows detecting unobserved region of object that aren’t captured by endoscope (during fly-through) and this is based on partial 3D model (of recorded triangular/polygon patches presented during the fly-through). Mori is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of unobserved regions being detected using an endoscope in the medical imaging field. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Akimoto and Zingaretti's invention with the detection of unobserved region techniques of Mori to ensure improving the method for presenting unobserved regions, (Mori, page 144, section 5). This would be done by the advanced computation techniques of Mori to detect/calculate the unobserved regions.
Regarding claim 2, the combination of Akimoto, Zingaretti and Mori teaches wherein predetermined region of interest is a tissue protruding from an inner surface of a lumen (Mori, page 134, section 1 teaches “When we use the VES as a diagnostic tool, especially a tool to find
colonopolyps, it is very important for doctors to observe the whole target organ without unobserved regions. However it is difficult to view the whole of an organ.s inner wall without unobserved regions”; when finding a colonopolyp, that would be considered the predetermined ROI from Zingaretti, and a colonopolyp is a tissue protruding from an inner surface of a lumen. The same motivations used in claim 1 apply here in claim 2.
Regarding claim 4, the combination of Akimoto and Mori teaches and the generating of the partial three-dimensional model includes generating the partial three-dimensional model only from, among the plurality of two-dimensional images, images in which the predetermined region of interest is detected. (Akimotoko, paragraph 121 teaches “a 3D model image corresponding to any region included in an endoscope image observed in the past from start of the 3D model image generation to the current observation state of the distal end portion 15 is displayed on the monitor 8. The display region of the 3D model image displayed on the monitor 8 increases with time elapse”); this shows partial 3D model (partial since region of 3D model increases with time elapse) of object generated based on the aforementioned images and corresponding to any region means this partial 3D model would be representing predetermined ROI (thus only predetermined ROI when enough time has elapsed only for endoscope to be in predetermined ROI when predetermined ROI appears first).
Regarding claim 7, the combination of Akimoto, Zingaretti and Mori teaches wherein the detecting of the unobserved region includes detecting a missing part, as the unobserved region, in the partial three-dimensional model (Mori, page 135 paragraph 2 teaches “purpose of this paper is to describe a method that finds unobserved regions during fly-through. The method proposed here enables a VES to detect unobserved regions and to present them after the execution of the fly-thorough mode. This function is very useful and prevent doctors from missing abnormal regions in navigation-based diagnosis”); this shows find/detect unobserved regions and present them to prevent doctors from missing abnormal regions which shows the unobserved regions include detecting a missing part as the unobserved region, and as explained above in claim 1, since based on partial 3D model, this would be in the partial 3D model. The same motivations used in claim 1 apply here in claim 7.
Regarding claim 8, the combination of Akimoto and Mori teaches wherein the one or more processors are further configured to cause, when the unobserved region is detected, a display device to display in real time an indication indicating that the unobserved region is detected (Mori, page 137, last paragraph teaches “calculate regions consisting of connected ‘unobserved’ patches and consider them as ‘unobserved regions’. In the presentation step of the unobserved regions, the system displays the organ while coloring the unobserved regions”); displaying organ while coloring unobserved regions shows the display device (monitor 8 from fig. 1 of Akimoto) displays in real time an indication/color indicating that the unobserved region is detected and all of this is done when unobserved region detected. The same motivations used in claim 1 apply here in claim 8.
Regarding claim 9, the combination of Akimoto, Zingaretti and Mori teaches wherein the indication indicating that the unobserved region is detected includes an arrow superimposed on the partial three-dimensional model for indicating a position of the predetermined region of interest in the plurality of two-dimensional images (Mori, page 138, section (a) Specification of unobserved regions, fig. 3 and the description thereof teaches a point specification method for calculating location of voxels that are corresponding to pixels of the screen and shows these points labeled with arrows); fig. 3 shows the arrow superimposed in image to indicate position of part of organ (predetermined ROI from Zingaretti) which would be done in the plurality of 2D images from Akimoto on the partial 3D model and is for indication/specification of unobserved regions as the section header mentions. The same motivations used in claim 1 apply here in claim 9.
Regarding claim 10, the combination of Akimoto, Zingaretti and Mori teaches wherein the indication indicating that the unobserved region is detected includes a marker superimposed at a position corresponding to the unobserved region in the plurality of two-dimensional images (Mori, page 134, abstract teaches “the method labels those triangle patches or voxels that do not have ’observed’ marks as ’unobserved triangle patches or voxels” and fig. 2, right image on page 137 shows colored rendering with different color assigned to each triangle patch); this shows triangles marked (thus at a position superimposed) as unobserved and the color shows the indication which marks for each of the plurality of images from Akimoto. The same motivations used in claim 1 apply here in claim 10.
Regarding claim 11, the method claim 11 recites similar limitations as apparatus claim 1, and thus is rejected under similar rationale.
Regarding claim 13, the method claim 13 recites similar limitations as apparatus claim 4, and thus is rejected under similar rationale.
Regarding claim 16, the non-transitory computer-readable storage medium claim 16 recites similar limitations as apparatus claim 1, and thus is rejected under similar rationale. In addition, Akimoto, fig. 1 teaches memory 14 which acts as non-transitory computer-readable storage medium and paragraph 107 teaches “image processing…hardware configured by a computer program”.
Regarding claim 18, the non-transitory computer-readable storage medium claim 18 recites similar limitations as apparatus claim 4, and thus is rejected under similar rationale.
Claim(s) 3, 12, and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Akimoto, Zingaretti and Mori as applied to claims 1, 11 and 16 above, and further in view of Sharma et al. (U.S. Patent Application Publication No. 2013/0132054), hereinafter referenced as Sharma.
Regarding claim 3, the combination of Akimoto, Zingaretti and Mori teaches wherein the one or more processors are further configured to generate, based on the plurality of two-dimensional images, a whole three-dimensional model of the object, (Akimoto, paragraph 152 teaches “updating 3D shape data before the change with the generated 3D shape data, newly setting a polygon on the updated region appropriately by using the normal vector, and generating a 3D model image through addition (update)”); updated model here shows the whole 3D model of the object since it contains updates and since the display region of the 3D model image displayed on the monitor 8 increases with time elapse as mentioned in claim 1, also, this is based on plurality of images since comes in a step after such;
However, the combination of Akimoto, Zingaretti and Mori fails to explicitly teach
and the generating of the partial three-dimensional model includes extracting a portion corresponding to the predetermined region of interest from the whole three-dimensional model.
However, Sharma explicitly teaches and the generating of the partial three-dimensional model includes extracting a portion corresponding to the predetermined region of interest from the whole three-dimensional model (Sharma, paragraph 30 teaches “e.g., segments which contain narrowing plaque deposits are segmented using full 3D models…)”; this shows generating segment (partial 3D model) by using (thus extracted from) the full/whole 3D model, and this corresponds to first region since it is for region of interest as aforementioned. Sharma is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of medical imaging field and partial as well as whole 3D model generation thereof. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Akimoto, Zingaretti and Mori with the model generation techniques of Sharma to ensure reduced-order models produce reliable results…and lead to execution times which are more than two orders of magnitude smaller than the corresponding 3D computations (Sharma, paragraph 30). This means increased computational efficiency which would be due to the specific technique of model generation and generating partial model from full model instead of rendering a partial model from scratch every time.
Regarding claim 12, the method claim 12 recites similar limitations as apparatus claim 3, and thus is rejected under similar rationale. In addition, the combination of Akimoto, Zingaretti, Mori and Sharma teaches the determining whether the predetermined region of interest of the object exists in each of the plurality of two-dimensional images includes detecting the predetermined region of interest from at least one of the whole three-dimensional model and each of the plurality of two-dimensional images, (Mori, page 135, paragraph 2 teaches “The method proposed here enables a VES to detect unobserved regions and to present them after the execution of the fly-thorough mode. This function is very useful and prevent doctors from missing abnormal regions”); since this detection of abnormal/unobserved regions (predetermined ROI) is done after the fly-through of the probe it shows it would be from the whole 3D model that is incrementally completed during the fly-through, and as aforementioned in claim 1 above Zingaretti uses each of the plurality of two-dimensional images for the determination of whether predetermined ROI exists process.
Regarding claim 17, the non-transitory computer-readable storage medium claim 17 recites similar limitations as apparatus claim 3, and thus is rejected under similar rationale. In addition, the combination of Akimoto, Zingaretti, Mori and Sharma teaches the determining whether the predetermined region of interest of the object exists in each of the plurality of two-dimensional images includes detecting the predetermined region of interest from at least one of the whole three-dimensional model and each of the plurality of two-dimensional images, (Mori, page 135, paragraph 2 teaches “The method proposed here enables a VES to detect unobserved regions and to present them after the execution of the fly-thorough mode. This function is very useful and prevent doctors from missing abnormal regions”); since this detection of abnormal/unobserved regions (predetermined ROI) is done after the fly-through of the probe it shows it would be from the whole 3D model that is incrementally completed during the fly-through, and as aforementioned in claim 1 above Zingaretti uses each of the plurality of two-dimensional images for the determination of whether predetermined ROI exists process.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Yuille et al. (U.S. Patent Application Publication No. 2021/0012505) paragraph 34 teaches “the present disclosure can also be used to analyze two-dimensional images to determine whether the image depicts a tumor.”; this shows a determining step of whether predetermined ROI exists in each 2D image.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NAUMAN U AHMAD whose telephone number is (703)756-5306. The examiner can normally be reached Monday - Friday 9:00am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571) 272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/N.U.A./ Examiner, Art Unit 2611
/KEE M TUNG/ Supervisory Patent Examiner, Art Unit 2611
Prosecution Insights