Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 4-10, and 13-19 are rejected under 35 U.S.C. 103 as being unpatentable over Popovic et al. (U.S. Patent Application Publication No. 2017/0007350), referred herein as Popovic, in view of Komp et al. (U.S. Patent Application Publication No. 2021/0055537), referred herein as Komp.
Regarding claim 1, Popovic teaches an image processing device, the image processing device comprising: one or more processors comprising hardware, wherein the one or more processors being configured to (fig 1; paragraph 30, lines 1-11):
identify a status of observation of an observation target based on an image captured by an endoscope, wherein the observation target is a subject captured in the image (paragraph 32, lines 1-5; paragraph 33; an observation target captured by an endoscope, and its status, are identified);
determine, based on the status of observation, a processing mode concerning detection of an unobserved region that is a region of the observation target to be included in a three-dimensional model of the observation target and that has not been observed by the endoscope (paragraph 41, lines 1-10; paragraph 43, lines 3-8; an unobserved region to be included in that is a region of the target is detected that has not been observed by the endoscope); and
perform generation of the three-dimensional model of the observation target from the image and detection of the unobserved region according to the determined processing mode (paragraph 34; paragraph 37, lines 1-6; paragraph 38, lines 1-5; paragraph 41, lines 1-13; paragraph 43, the last 9 lines; a 3D model of the observation target and unobserved region is generated).
Popovic does not explicitly teach that the unobserved region is a region that is missing in a three-dimensional model of the observation target.
However, in a similar field of endeavor, Komp teaches a device configured to identify a status of an observation target based on an endoscopic image, and determine, based on the status, a processing mode concerning detection of an unobserved region, and generating a three-dimensional model of the observation target (paragraph 30; paragraphs 31 and 32; paragraph 36; paragraph 54; paragraph 70), wherein the unobserved region is a region of the observation target that is missing in a three-dimensional model of the observation target (paragraphs 34 and 55; paragraphs 70, 71, and 72).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the missing region identification of Komp with the unobserved region processing of Popovic because this provides the medical professional with more accurate and complete images, which increases situational awareness for the medical professional and safety for the patient (see, for example, Komp, paragraphs 26 and 31; paragraphs 34 and 35).
Regarding claim 4, Popovic in view of Komp teaches the image processing device according to claim 1, wherein the identifying the status of observation includes recognizing an observation site in the image (Popovic, paragraph 32, lines 1-5; paragraph 33; paragraph 38, lines 1-5; paragraph 41, lines 1-20).
Regarding claim 5, Popovic in view of Komp teaches the image processing device according to claim 1, wherein the identifying the status of observation includes identifying at least one of a lesion site, a treatment tool, an operation of a treatment tool, a staining solution, or special light, in the image, or movement of an observation site in the image (Popovic, paragraph 38; paragraph 44, lines 1-15; paragraph 50; paragraph 57, lines 10-21).
Regarding claim 6, Popovic in view of Komp teaches the image processing device according to claim 1, wherein the identifying the status of observation includes identifying at least a part of the endoscope in the image (Popovic, paragraph 57, lines 10-21).
Regarding claim 7, Popovic in view of Komp teaches the image processing device according to claim 1, wherein the identifying the status of observation includes identifying a moving speed of an observation site in the image (Popovic, paragraph 57, lines 1-13).
Regarding claim 8, Popovic in view of Komp teaches the image processing device according to claim 1, wherein the identifying the status of observation includes determining whether the image is appropriate for generating the three-dimensional model (Popovic, paragraph 34; paragraph 37, lines 1-6; paragraph 38, lines 1-5; paragraph 41, lines 1-13; paragraph 43, the last 9 lines).
Regarding claim 9, Popovic in view of Komp teaches the image processing device according to claim 8, wherein the determining whether the image is appropriate for generating the three-dimensional model includes determining whether the image includes at least one of an object covering the observation target, a red-out region, a halation region, a fuzzy region, or a blur region (Popovic, paragraph 41, lines 1-10; paragraph 43, lines 1-16).
Regarding claims 10 and 13-18, the limitations of these claims substantially correspond to the limitations of claims 1 and 4-9, respectively; thus they are rejected on similar grounds as their corresponding claims.
Regarding claim 19, the limitations of this claim substantially correspond to the limitations of claim 1; thus they are rejected on similar grounds.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2, 3, 11, and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Popovic, in view of Komp, and further in view of Leist ((U.S. Patent Application Publication No. 2021/0378748), referred herein as Leist).
Regarding claim 2, Popovic in view of Komp teaches the image processing device according to claim 1, wherein the processing mode includes an execution mode for executing detection of the unobserved region and an update mode for updating detection of the unobserved region, and the one or more processors being configured to: execute the generation of the three-dimensional model and the detection of the unobserved region where the processing mode is determined to be the execution mode, and update the generation of the three-dimensional model and the detection of the unobserved region where the processing mode is determined to be the update mode (paragraph 34; paragraph 38, lines 1-5; paragraph 41, lines 1-20; paragraph 43, the last 9 lines; paragraph 57, lines 1-21; an execution mode is activated when the user instantiates the detection and display of the combined endoscopic and 3D model images; an update mode is activated when movement of the endoscope causes the device to generate new imagery and 3D modeling based on the current view of the endoscope; see also Komp, paragraphs 36 and 52; paragraphs 56 and 57; paragraphs 59 and 72; varying modes for capturing the images and/or updating the models; the motivation to combine is substantially similar to that discussed with respect to claim 1).
As shown above, Popovic in view of Komp teaches that the user may start the detection and 3D display through commands entered in a user interface, or the like; this implies that the user also has the capability to stop the detection and display at any
point. Additionally, the real-time updating could be considered a stop mode, since the device will stop detecting and displaying structures at the previous position, and start detecting and displaying structures at the current position. Nevertheless, Popovic in view of Komp does not explicitly teach a stop mode for stopping the detection of the unobserved region and stopping the generation of the 3D model.
However, in a similar field of endeavor, Leist teaches a device comprising a processor configured to identify a status of an observation target of an endoscope, determine a processing mode for detecting an unobserved region that is not observed by the endoscope, and generating a 3D model of the observation target (paragraph 44, lines 1-5; paragraph 46, lines 1-12; paragraph 48, lines 1-6), wherein the processing mode includes an execution mode for detecting the unobserved region and generating the 3D model, and a stop mode for stopping the detection of the unobserved region and stopping the generation of the 3D model (paragraph 78; paragraph 79, lines 1-22).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the stop mode of Leist with the execution and stop modes of Popovic in view of Komp, because this provides a high degree of flexibility for the user such that the user can control what anatomy needs to be seen, and when the anatomy needs to be seen, which is incredibly important for such endoscopic processes in order to ensure the safety of the patient and improve the situational awareness of the operator (see, for example, Leist, paragraph 76).
Regarding claim 3, Popovic in view of Komp teaches the image processing device according to claim 1, but does not explicitly teach the device, wherein the processing mode includes a plurality of execution modes in which the unobserved region is detected with a different sensitivity in each of the plurality of execution modes.
However, in a similar field of endeavor, Leist teaches a device comprising a processor configured to identify a status of an observation target of an endoscope, determine a processing mode for detecting an unobserved region that is not observed by the endoscope, and generating a 3D model of the observation target (paragraph 44, lines 1-5; paragraph 46, lines 1-12; paragraph 48, lines 1-6), wherein the processing mode includes a plurality of execution modes in which the unobserved region is detected with a different sensitivity in each of the plurality of execution modes (paragraph 76, lines 1-18; paragraph 78; paragraph 79, lines 1-22).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the plurality of execution modes and sensitivity changes of Leist with the execution modes of Popovic in view of Komp, because this provides a high degree of flexibility for the user such that the user can control what anatomy needs to be seen, and when the anatomy needs to be seen, which is incredibly important for such endoscopic processes in order to ensure the safety of the patient and improve the situational awareness of the operator (see, for example, Leist, paragraph 76).
Regarding claims 11 and 12, the limitations of these claims substantially correspond to the limitations of claims 2 and 3, respectively; thus they are rejected on similar grounds as their corresponding claims.
Response to Arguments
On page 8 of the Applicant’s Remarks, with respect to the prior art rejection of claim 1, the Applicant argues that 1) Popovic does not disclose an observation target that is a subject captured in the image because Popovic discloses visualizing internal structures which are not visible to the camera, 2) Popovic does not disclose determining a processing mode based on the status of observation because Popovic only discloses moving the endoscope to detect the unobserved region, and 3) Popovic does not disclose the amended “missing” limitation. The Examiner respectfully disagrees with these arguments.
Regarding the first argument, it is respectfully submitted that Popovic describes an observation target such as an organ/structure (and its sub-surface anatomy/vessels), and clearly discloses that this target organ/structure is captured in the image by the endoscope. As pointed out by the Applicant, in Popovic, the sub-surface anatomy of the organ/structure cannot be directly/clearly seen by the endoscope, and therefore comprise “an unobserved region that is a region of the observation target” that “has not been observed by the endoscope” as the claim requires. Thus, the Examiner respectfully submits that Popovic teaches these broad limitations.
Regarding the second argument, it is respectfully submitted that a “processing mode” is a very broad term with a number of reasonable interpretations. In the citations, Popovic discloses, as just one example, a “processing mode” activated when the user instantiates detection and display of the combined endoscopic and 3D model images, and another “processing mode” activated when movement of the endoscope causes the device to generate new imagery and 3D modeling based on the current view. This clearly teaches determining a processing mode “based on the status of observation” as the claim requires. Thus, the Examiner respectfully submits that Popovic teaches these broad limitations.
Regarding the third argument, this argument has been fully considered, but is moot in view of the new ground(s) of rejection presented above.
On page 9 of the Applicant's Remarks, the Applicant argues that the remaining independent claims are not taught by the prior art for reasons similar to those discussed in regard to claim 1, and that the dependent claims are not taught by the prior art, insomuch as they depend from claims that are not taught by the prior art. The Examiner respectfully disagrees with these arguments, for the reasons discussed above.
Conclusion
The following prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Shamir (U.S. Patent Application Publication No. 2022/0237863); Methods, systems, and apparatuses for medical image enhancement to optimize transducer array placement.
Hane (U.S. Patent Application Publication No. 2022/0409030); Processing device, endoscope system, and method for processing captured image.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID T WELCH whose telephone number is (571)270-5364. The examiner can normally be reached Monday-Thursday, 8:30-5:30 EST, and alternate Fridays, 9:00-2:30 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached at 571-272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
DAVID T. WELCH
Primary Examiner
Art Unit 2613
/DAVID T WELCH/Primary Examiner, Art Unit 2613
Prosecution Insights