DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 12/13/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-3, 5, 7, 8, 12-17 and 19 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Oosake (US 2025/0095191).
Regarding claim 1:
Oosake discloses an image processing device (see Figs. 1-5 and 11)comprising: one or more processors (see Fig. 4, processor 58) comprising hardware and configured to:
input a captured image captured by an endoscope in a body cavity of a subject to a Convolutional Neural Network (CNN) using a trained model, the trained model having training data in which each of training images is associated with category information including a lumen direction of a region outside the each of training images in which the lumen is present (see Figs. 8-10 and paragraphs [0089]-[0095]);
estimate, based on the CNN, category information including a lumen direction of a region outside the captured image in which the lumen is likely to be present (see Fig. 9 and estimation result 118); and
output the category information estimated (see Fig. 10; display lumen direction 122A).
Regarding claim 2:
Oosake discloses the image processing device according to claim 1, wherein the one or more processors is configured to:
estimate absence of the lumen in the captured image (see Fig. 17; estimate result 118A includes absence probability);
in response to estimating the category information including absence of the lumen in the captured image, superimpose lumen information on the lumen direction of the region outside the captured image in which the lumen is likely to be present onto either the captured image or a peripheral region of the captured image (see Fig. 11; indication 122A or 122C is superimposed on the lumen direction to indicate the direction of the lumen which is outside the captured image); and
output either the captured image or the peripheral region of the captured image with the superimposed lumen information (see Fig. 11 and paragraph [0096]).
Regarding claim 3:
Oosake discloses the image processing device according to claim 1, wherein:
the category information further includes lumen directions of regions outside the captured image in which the lumen is likely to be present and a degree of reliability of each of the lumen directions (see paragraphs [0139]-[0142]); and
the one or more processors are configured to:
superimpose lumen information onto a position of the captured image that corresponds to the lumen direction for which the degree of reliability is the highest (see Fig. 16 and paragraph [0141]); and
output the captured image with the superimposed lumen information (see Fig. 16, indications 112E and 122D).
Regarding claim 5:
Oosake discloses the image processing apparatus according to claim 2, wherein the lumen information includes one or more of a figure, a character, and a frame that encloses the captured image (see Fig. 11, indicators 122A, 122B and 122C; also see paragraph [0096]; the lumen direction information can be in the form of an arrow or text).
Regarding claim 7:
Oosake discloses the image processing device according to claim 1, wherein:
the category information further includes a degree of reliability of the lumen direction (see paragraph [0140]); and
the one or more processors are configured to:
superimpose the lumen information and the degree of reliability onto the captured image (see Fig. 16); and
output the captured image with the superimposed lumen information and the superimposed degree of reliability (see Fig. 16 and paragraph [0141]; the indicators 122E and/or 122D is/are interpreted as lumen information and degree of reliability information).
Regarding claim 8:
Oosake discloses the image processing device according to claim 1, wherein:
the trained model is provided for each of regions in a subject; and
the one or more processors are further configured to:
input the captured image to a CNN using a region trained model having training data in which each of training images is associated with a captured image of a predetermined region (see paragraph [0090]; a plurality of image is inputted into the CNN for training; each of the image corresponds to a capture image of a predetermined region);
estimate, based on the CNN using the region trained model, that the captured image was captured in the predetermined region (see Fig. 9 and paragraphs [0091]-[0093]).
Regarding claim 12:
Oosake discloses the image processing device according to claim 1, wherein the one or more processors are configured to estimate absence of the lumen (see Fig. 17; “absence probability” in box 118A).
Regarding claims 13 and 14:
Claims 13 and 14 each recites similar limitations as in claim 1. Hence, claims 13 and 14 are rejected under the same reason as discussed above in claim 1.
Regarding claim 15:
Claim 15 recites similar limitation as in claim 12. Hence, claim 15 is rejected under the same reason as discussed above in claim 12.
Regarding claims 16, 17 and 19:
Claims 16, 17 and 19 recite similar limitations as in claims 2, 3 and 5. Hence, claims 16, 17 and 19 are rejected under the same reasons as discussed above in claims 2, 3, and 5, respectively.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 4 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Oosake in view of Jorgensen et al. (US 2023/0172428; hereinafter Jorgensen).
Regarding claim 4:
Oosake discloses all the features in claim 1. Oosake further discloses the image processing device, wherein:
the category information further includes lumen directions of regions outside the captured image in which the lumen is likely to be present and a degree of reliability of each of the lumen directions (see paragraphs [0139]-[0142]); and
the one or more processors are configured to:
determine whether each of the degrees of reliability is equal to or larger than a threshold (see Fig. 17, threshold value 128) and
and in response to determining that the each of the degrees of reliability is not equal to or larger than the threshold, set the result as absence of lumen direction (see Fig. 17; when it determines that the degree of reliability is not equal to or larger than the threshold, the system inherently set the result to absence of lumen).
Oosake does not disclose in response to determining that the each of the degrees of reliability is not equal to or larger than the threshold, output a warning for instructing a change of an image capturing direction in which the endoscope captures an image.
However, in the same field of endeavor, Jorgensen disclose an image processing device, wherein one or more processors are configured to:
output a warning for instructing a change of an image capturing direction in which the endoscope captures an image (see Fig. 8, step 138 and paragraph [0127]).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to combine the teaching of Oosake and Jorgensen such that the processor is configured to in response to determining that the each of the degrees of reliability is not equal to or larger than the threshold, output a warning for instructing a change of an image capturing direction in which the endoscope captures an image. The combination would have yielded a predictable result of informing the operator/user when absence of lumen is detected.
Regarding claim 18:
Claim 18 recites similar limitations as in claim 4. Hence, claim 18 is rejected under the same reason as discussed above in claim 4.
Claim(s) 6 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Oosake in view of Oyama (US 2023/0099565).
Regarding claim 6:
Oosake discloses all the features in claim 1. Oosake further discloses the image processing device, wherein the one or more processors are configured to:
acquire a plurality of captured images that are sequentially captured by the endoscope (see paragraph [0053]);
estimate, based on the CNN, the category information of each of the plurality of captured images sequentially (see paragraph [0090]); and
for each of the plurality of captured images:
superimpose lumen information on the lumen direction of the region outside the each of the plurality of captured images in which the lumen is likely to be present onto the captured image (see Fig. 11 or 16; the indicators 122A, 122C, 122D or 122E is superimposed lumen information); and
output the each of the plurality of captured images with the superimposed lumen information (see Fig. 11 or 16; indicators 122A, 122C, 122D or 122D).
Oosake dose not disclose while gradually decreasing at least one of a color and brightness with passage of time.
In the same field of endeavor, Oyama discloses an image processing device, wherein region of interest is displayed while changing the color and/or brightness (see paragraph [0089]).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to combine the teaching of Oosake and Oyama such that superimpose lumen information on the lumen direction of the region outside the each of the plurality of captured images in which the lumen is likely to be present onto the captured image while gradually decreasing at least one of a color and brightness with passage of time. The combination would have yielded a predictable result of displaying the pertinent information that is easily visually recognized by the user/operator.
Allowable Subject Matter
Claims 9-11 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
In regards to claim 9, none of the reference of record alone or in combination discloses or suggests the image processing device according to claim 1, wherein the one or more processors are configured to:
detect, as a plurality of corresponding points, corresponding pixel positions between chronologically successive captured images based on chronologically successive captured images;
identify coordinates of observation positions in the chronologically successive captured images based on the chronologically successive captured images,
convert a coordinate of an observation position identified in the captured image of a previous frame into a coordinate in a coordinate system of the captured image of a current frame based on the plurality of corresponding points;
determine whether the coordinates of the observation positions in chronologically successive captured images are trackable based on the corresponding points in the chronologically successive captured images;
determine whether it is possible to estimate the lumen direction based on the estimated category information, and
in response to determining the lumen direction cannot be estimated and it is possible to track the coordinates of the observation positions, covert the coordinate of the observation position of the previous frame into the coordinate in the coordinate system of the captured image of the current frame to estimate the lumen direction.
In regards to claim 10, none of the reference of record alone or in combination discloses or suggests the image processing device according to claim 9, wherein the one or more processors are configured to:
input the captured image to a CNN using a lumen detection trained model having training data in which each of the training images is associated with presence or absence of a lumen and a coordinate of a position of a lumen in each of the training images;
estimate, based on the CNN, presence or absence of a lumen in the captured image and a lumen coordinate indicating a position of the lumen; and
in response to estimating the presence of the lumen in the captured image, output the lumen coordinate.
In regards to claim 11, none of the reference of record alone or in combination discloses or suggests the image processing device according to claim 1, wherein the one or more processors are configured to:
input the captured image to a CNN using a trained model for each proficiency, the trained model having training data in which each of the training images is associated with lumen presence-absence information that differs depending on proficiency of an operator with respect to an endoscope; and
output the category information estimated in accordance with the proficiency of the operator.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Watanabe (US 2024/0087125) discloses a medical image process device and endoscope system that output diagnosis information through a diagnosis model.
Takenouchi (US 2022/0383533) discloses an endoscope system that displays operation assistance information that includes direction information in which the endoscope is to be moved.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LIXI CHOW SIMPSON whose telephone number is (571)272-7571. The examiner can normally be reached Mon-Fri 7:00am-3:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, William Boddie can be reached at 517-272-0666. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/LIXI C SIMPSON/Primary Examiner, Art Unit 2625
Prosecution Insights