Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) was filed on 09/13/2024. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-7 and 10-11 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Kitamura Makoto, hereafter Kitamura (US Pub No. 20100208047 A1).
As per claim 1, Kitamura teaches “An image processing device comprising: at least one memory configured to store instructions; and at least one processor configured to execute the instructions to:
acquire an endoscopic image obtained by photographing an examination target by an endoscope;” (See paragraph 40 “[0040] The capsule endoscope 3 is equipped with an imaging function, a wireless communication function, and the like. The capsule endoscope 3 is introduced into the subject 1 by being swallowed from a mouth of the subject 1, sequentially captures in-vivo images while moving through the inside of digestive tracts, and wirelessly transmits the captured in-vivo images to the outside of the body. The in-vivo images captured by the capsule endoscope 3 are color images having pixel values (RGB values) corresponding to R (red), G (green), and B (blue) color components at respective pixel positions in each image.” See also paragraphs 87-89. See also paragraphs 64-66, the examination target is the picture taken. “053] Then, the arithmetic unit 15 outputs a detection result of the lesion area with respect to the in-vivo image being the processing target (Step a11).” Kitamura)
“determine, based on the endoscopic image, a state of the examination target or a photographed part of the examination target; and” (See fig. 3, See paragraphs 8 and paragraphs 47-48 “[0047] The arithmetic unit 15 is realized by hardware such as a CPU, processes the series of in-vivo images acquired by the image acquiring unit 11, and performs various types of arithmetic processing for detecting a lesion area that appears in each in-vivo image. In the first embodiment, a reddish lesion area having a color property of red, e.g., bleeding or redness, is to be detected. The arithmetic unit 15 includes a converting unit 16, an extracting unit 17 as a body-tissue extracting unit, a criterion creating unit 18, and a lesion-area detecting unit 19 as a detecting unit. [0048] The converting unit 16 uses hue and saturation, which are examples of color elements, as feature data, and converts an in-vivo image into a color plane formed of the hue and the saturation. The extracting unit 17 extracts a mucous-membrane area as an example of a body tissue from an in-vivo image. The criterion creating unit 18 creates a criterion for detecting a lesion area based on the hue and the saturation of the mucous-membrane area extracted by the extracting unit 17. In the first embodiment, the criterion creating unit 18 creates a criterion for detecting a reddish lesion area such as bleeding or redness…” The endoscopy detects a state of the mucous-membrane of the subject (by extracting an image of the area) to detect if there is a lesion. See also paragraph 113-115. It also identifies the organ type based on the image. Kitamura)
“detect, based on the state or the photographed part and the endoscopic image, a part of interest to be noticed in the examination target.” (See paragraphs 47, 48 and 113-115, it shows that when the image is extracted an area of interest is detected such as a reddish lesion area or organ type. [0048] The converting unit 16 uses hue and saturation, which are examples of color elements, as feature data, and converts an in-vivo image into a color plane formed of the hue and the saturation. The extracting unit 17 extracts a mucous-membrane area as an example of a body tissue from an in-vivo image. The criterion creating unit 18 creates a criterion for detecting a lesion area based on the hue and the saturation of the mucous-membrane area extracted by the extracting unit 17. In the first embodiment, the criterion creating unit 18 creates a criterion for detecting a reddish lesion area such as bleeding or redness…” The endoscopy detects a state of the mucous-membrane of the subject (by extracting an image of the area) to detect if there is a lesion. See also paragraph 113-115. It also identifies the organ type based on the image. See also paragraphs 87-91 and 119 “[0089] Next, the lesion-area detection process performed by the lesion-area detecting unit 19 at Step a9 of FIG. 3 is described below. The lesion-area detecting unit 19 determines the data that has been identified as belonging to the mucous-membrane area through the mucous-membrane-area extraction process at Step a5 of FIG. 3, based on the criteria generated through the criterion creation process at Step a7, and detects the lesion area from the processing target image. In practice, the lesion-area detecting unit 19 determines whether the data is the lesion area or not, by using the center (Hi, Si) of the cluster of data calculated at Step c5 of FIG. 6 and determined as belonging to the mucous-membrane area at Step c7, and the determination criteria HThresh and SThresh. FIG. 13 is a flowchart of a detailed process procedure of the lesion-area detection process.” Kitamura)
Claim 10 is rejected under the same analysis as claim 1.
Claim 11 is rejected under the same analysis as claim 1. (See paragraphs 3 and 10. Kitamura)
As per claim 2, Kitamura teaches “The image processing device according to The image processing device according to wherein the at least one processor is configured to execute the instructions to determine, as the state of the examination target, at least, a background condition, which is a condition of the examination target other than a target condition to be detected as the part of interest, and wherein the at least one processor is configured to execute the instructions to detect the target condition based on the background condition and the endoscopic image.” (See paragraphs 76-89. Paragraph 79 specifically shows that a “reddish color” of the lesion area is used to detect it, therefore the background color is interpreted as the reddish color or hue/saturation of the area. “[0079] Subsequently, the saturation-threshold calculating unit 182 of the criterion creating unit 18 calculates a determination threshold TS in a saturation direction (direction of the saturation S) for detecting the lesion area (Step d3). Then, the hue-threshold calculating unit 181 calculates a determination threshold TH in a hue direction (direction of the hue H) for detecting the lesion area (Step d5). As described above, the mucous-membrane area that actually appears in the in-vivo image may be a mucous-membrane area colored in yellowish color, such as mucous membrane of a small intestine or mucous membrane of a large intestine, or a mucous-membrane area colored in reddish color, such as mucous membrane of a stomach. In the first embodiment, the lesion area to be detected is a lesion area in reddish color. Therefore, each determination threshold is calculated so that a determination threshold in the hue direction and a determination threshold in the saturation direction for the reddish mucous-membrane area, which is similar to a color property of the lesion area, are respectively made smaller than a determination threshold in the hue direction and a determination threshold in the saturation direction for the yellowish mucous-membrane area.” Kitamura)
As per claim 3 “The image processing device according to claim 1 wherein the at least one processor is configured to execute the instructions to determine, as the state of the examination target, at least, a presence or absence of a residue in the target examination, and wherein the at least one processor is configured to execute the instructions to detect the part of interest based on the presence or absence of the residue and the endoscope image.” (See paragraphs 69-71, there is a detection of residue such as bubbles and feces “[0069] Next, the mucous-membrane-area extraction process performed by the extracting unit 17 at Step a5 of FIG. 3 is described below. In the in-vivo images captured by the capsule endoscope 3, contents such as feces floating inside the body cavity, bubbles, and the like appear in addition to the mucous membrane. In the mucous-membrane-area extraction process, a bubble area and a contents area other than the mucous-membrane area are eliminated and the mucous-membrane area is extracted. FIG. 6 is a flowchart of a detailed process procedure of the mucous-membrane-area extraction process. [0070] As illustrated in FIG. 6, in the mucous-membrane-area extraction process, the extracting unit 17 identifies the bubble area (Step c1)… Then, a correlation value between the calculated edge intensity and a bubble model set in advance based on the feature of the bubble is calculated, and a portion highly correlated with the bubble model is detected as the bubble area. The applicable technique is not limited to the above, and any methods of identifying the bubble area may be applied appropriately.” Kitamura)
As per claim 4, Kitamura teaches “The image processing device according to claim 1, wherein the at least one processor is configured to execute the instructions to determine, as the state of the examination target, at least, a color of the examination target, and wherein the at least one processor is configured to execute the instructions to detect, based on the color and the endoscopic image, the part of interest.” (See paragraphs 76-89. Paragraph 79 specifically shows that a “reddish color” of the lesion area is used to detect it, therefore the background color is interpreted as the reddish color or hue/saturation of the area. “[0079] Subsequently, the saturation-threshold calculating unit 182 of the criterion creating unit 18 calculates a determination threshold TS in a saturation direction (direction of the saturation S) for detecting the lesion area (Step d3). Then, the hue-threshold calculating unit 181 calculates a determination threshold TH in a hue direction (direction of the hue H) for detecting the lesion area (Step d5). As described above, the mucous-membrane area that actually appears in the in-vivo image may be a mucous-membrane area colored in yellowish color, such as mucous membrane of a small intestine or mucous membrane of a large intestine, or a mucous-membrane area colored in reddish color, such as mucous membrane of a stomach. In the first embodiment, the lesion area to be detected is a lesion area in reddish color. Therefore, each determination threshold is calculated so that a determination threshold in the hue direction and a determination threshold in the saturation direction for the reddish mucous-membrane area, which is similar to a color property of the lesion area, are respectively made smaller than a determination threshold in the hue direction and a determination threshold in the saturation direction for the yellowish mucous-membrane area.” Kitamura)
As per claim 5, Kitamura teaches “The image processing device according to claims 1, the at least one processor is configured to further execute the instructions to set, based on the state or the photographed part, a threshold value for comparing with a score, the score indicating a confidence level as the part of interest, wherein the at least one processor is configured to execute the instructions to detect the part of interest, based on the score calculated based on the endoscopic image and the threshold value.” (See paragraphs 76-91 which shows that there is a threshold for the hue and saturation, with the score being the value of the hue/saturation which is used for detecting the lesion area. “[0087] Then, the criterion creating unit 18 creates a criterion HThresh in the hue direction and a criterion SThresh in the saturation direction, and sets them as criteria for detecting the lesion area in the processing target image (Step d7). Then, the process returns to Step a7 of FIG. 3 and proceeds to Step a9. More specifically, the criterion creating unit 18 creates the criterion HThresh in the hue direction according to the following Equation (8) and the criterion SThresh in the saturation direction according to the following Equation (9), by using the determination threshold TH in the hue direction and the determination threshold TS in the saturation direction.” In paragraphs 88-92 shows “0090] As illustrated in FIG. 13, in the lesion-area detection process, the lesion-area detecting unit 19 sets the data belonging to the mucous-membrane area as determination target data (Step e1). Then, the lesion-area detecting unit 19 compares Hi with HThresh, and when HThresh≦Hi, (YES at Step e3), the process proceeds to Step e7. On the other hand, when HTresh≦Hi is not satisfied (NO at Step e3) the lesion-area detecting unit 19 compares Si with SThresh. When SThresh≦Si (YES at Step e5) the process proceeds to Step e7. At Step e7, the lesion-area detecting unit 19 determines that the determination target data is data belonging to the lesion area. Namely, the lesion-area detecting unit 19 determines that data satisfying HThresh≦Hi or SThresh≦Si is the data belonging to the lesion area from among pieces of data belonging to the mucous-membrane area.” See also paragraph 139. Kitamura)
As per claim 6, Kitamura teaches “The image processing device according to claim 5, the at least one processor is configured to further execute the instructions to cause a display device to display at least one of information regarding the threshold value and/or information regarding the state or the photographed part to assist an examiner to perform a decision making.” (See paragraphs 53 “[0053] Then, the arithmetic unit 15 outputs a detection result of the lesion area with respect to the in-vivo image being the processing target (Step a11). As will be described later, the lesion-area detecting unit 19 generates label data indicating the lesion area. The arithmetic unit 15 displays the lesion area on the display unit 13 via the control unit 21 by creating an image or the like of the lesion area detected from the in-vivo image being the processing target based on the label data.” See also paragraphs 42 “[0042] The image processing apparatus 10 is used by a doctor or the like to observe and diagnose the series of in-vivo images captured by the capsule endoscope 3, and is realized by a general-purpose computer such as a workstation or a personal computer. The image processing apparatus 10 is configured to detachably attach the portable recording medium 7 thereto. The image processing apparatus 10 processes the series of in-vivo images stored in the portable recording medium 7, and displays the processed images in chronological order on a display such as an LCD or an EL display.” See also paragraph 5 “Observers such as doctors spend a lot of time to check the large number of in-vivo images transmitted to the receiving device outside the body for identifying a lesion area, by using diagnostic workstations or the like. Therefore, a technology for improving the efficiency of observation operations of the in-vivo images has been strongly desired.” Kitamura)
As per claim 7, Kitamura teaches “The image processing device according to claim 6 wherein the at least one processor is configured to execute the instructions to cause the display device to display, as the information regarding the photographed part, a model representing a whole of the examination target with a clear indication of the photographed part.” (See paragraph 53 “[0053] Then, the arithmetic unit 15 outputs a detection result of the lesion area with respect to the in-vivo image being the processing target (Step a11). As will be described later, the lesion-area detecting unit 19 generates label data indicating the lesion area. The arithmetic unit 15 displays the lesion area on the display unit 13 via the control unit 21 by creating an image or the like of the lesion area detected from the in-vivo image being the processing target based on the label data.” The model of the examination target is the created image based on taken images (of the processing target), it is a “whole” since it shows the whole examined detected area. Kitamura )
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 8 and 9 are rejected under 35 U.S.C. 103 as being unpatentable over Kitamura in view of Kamon (US Pub. No. 20200320702 A1) .
As per claim 8, Kitamura already teaches “The image processing device according to claim 1, wherein the at least one processor is configured to execute the instructions to detect the part of interest, based on input data based on the state or the photographed part and the endoscopic image and”, however Kitamura does not teach “a model to which the input data is inputted, and wherein the model is a model learned, through machine learning, a relation between the input data and a presence or absence of the part of interest in the endoscopic image included in the input data.”
Kamon teaches “a model to which the input data is inputted, and wherein the model is a model learned, through machine learning, a relation between the input data and a presence or absence of the part of interest in the endoscopic image included in the input data” (See paragraphs 60-63 “[0060] The lesion region detection unit 46 detects a lesion region from the input image (input image). The lesion region detection unit 46 comprises a first detection unit 46A, a second detection unit 46B, a third detection unit 46C, a fourth detection unit 46D, a fifth detection unit 46E, a sixth detection unit 46F, a seventh detection unit 46G, and an eighth detection unit 46H” “[0061] Each of the first detection unit 46A, the second detection unit 46B, the third detection unit 46C, the fourth detection unit 46D, the fifth detection unit 46E, the sixth detection unit 46F, the seventh detection unit 46G, and the eighth detection unit 46H is a learned model. The plurality of learned models are models learned using different data sets, respectively. Specifically, the plurality of learned models are models learned using data sets consisting of images captured at different positions in the body cavity, respectively.” “[0062] That is, the first detection unit 46A is a model learned using a data set consisting of images of the rectum, the second detection unit 46B is a model learned using a data set consisting of images of the sigmoid colon, the third detection unit 46C is a model learned using a data set consisting of images of the descending colon, the fourth detection unit 46D is a model learned using a data set consisting of images of the transverse colon, the fifth detection unit 46E is a model learned using a data set consisting of images of the ascending colon, the sixth detection unit 46F is a model learned using a data set consisting of images of the cecum, the seventh detection unit 46G is a model learned using a data set consisting of images of the ileum, and the eighth detection unit 46H is a model learned using a data set consisting of images of the jejunum.” See also paragraphs 118-119 and 6. Kamon )
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings of Kitamura with the teachings of Kamon to include a learning model to detect a presence or absence of the part of interest. The modification would have been motivated by the desire to have better accuracy, also it is preferable to use learning models to autonomously detect lesions, therefore it is an improvement, as suggested by Kamon (See paragraph 5 “[0005] With the technique, it is possible to match the in-vivo position of the endoscope and the position on a guide image with high accuracy.” See also paragraphs 6-10 and 13-15 “[0013] It is preferable that the plurality of region-of-interest detection units are a plurality of learned models. In this manner, it is possible to appropriately detect a lesion region.
[0014] It is preferable that the plurality of learned models are models learned using different data sets, respectively. In this manner, it is possible to perform different detection.
[0015] It is preferable that the plurality of learned models are models learned using data sets consisting of images captured at different in-vivo positions, respectively. In this manner, it is possible to detect a lesion region according to the in-vivo position of the image.” See also paragraphs 82, 92, 96, 104 and 126. Kamon )
As per claim 9, Kitamura already teaches “The image processing device according to claim 1, wherein the at least one processor is configured to execute the instructions to detect the part of interest,”, however Kitamura does not completely teach “based on a model selected based on the state or the photographed part and the endoscopic image, wherein the model is a model which learned, through machine learning, a relation between an endoscopic image inputted to the model and a presence or absence of the part of interest in the endoscopic image.”
Kamon teaches “based on a model selected based on the state or the photographed part and the endoscopic image, wherein the model is a model which learned, through machine learning, a relation between an endoscopic image inputted to the model and a presence or absence of the part of interest in the endoscopic image.” ((See paragraphs 60-63 “[0060] The lesion region detection unit 46 detects a lesion region from the input image (input image). The lesion region detection unit 46 comprises…” “[0061] Each of the first detection unit 46A, the second detection unit 46B, the third detection unit 46C, the fourth detection unit 46D, the fifth detection unit 46E, the sixth detection unit 46F, the seventh detection unit 46G, and the eighth detection unit 46H is a learned model. The plurality of learned models are models learned using different data sets, respectively. Specifically, the plurality of learned models are models learned using data sets consisting of images captured at different positions in the body cavity, respectively.” “[0062] That is, the first detection unit 46A is a model learned using a data set consisting of images of the rectum, the second detection unit 46B is a model learned using a data set consisting of images of the sigmoid colon, the third detection unit 46C is a model learned using a data set consisting of images of the descending colon, the fourth detection unit 46D is a model learned using a data set consisting of images of the transverse colon, the fifth detection unit 46E is a model learned using a data set consisting of images of the ascending colon, the sixth detection unit 46F is a model learned using a data set consisting of images of the cecum, the seventh detection unit 46G is a model learned using a data set consisting of images of the ileum, and the eighth detection unit 46H is a model learned using a data set consisting of images of the jejunum.” See also paragraphs 118-119 and 6. See also paragraphs 102-103 “[0102] The selection unit 52 selects any detection unit among the first detection unit 46A, the second detection unit 46B, the third detection unit 46C, the fourth detection unit 46D, the fifth detection unit 46E, the sixth detection unit 46F, the seventh detection unit 46G, and the eighth detection unit 46H on the basis of the acquired positional information I.” “[0103] The lesion region detection control unit 54 causes the selected detection unit to detect a lesion region from the image acquired by the image acquisition unit 44”. See also paragraph 7, 20 and 21. Kamon )
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings of Kitamura with the teachings of Kamon to include a selected learning model to detect a presence or absence of the part of interest. The modification would have been motivated by the desire to have better accuracy, also it is preferable to use learning models to autonomously detect lesions, therefore it is an improvement, as suggested by Kamon (See paragraph 5 “[0005] With the technique, it is possible to match the in-vivo position of the endoscope and the position on a guide image with high accuracy.” See also paragraphs 6-10 and 13-15 “[0013] It is preferable that the plurality of region-of-interest detection units are a plurality of learned models. In this manner, it is possible to appropriately detect a lesion region.
[0014] It is preferable that the plurality of learned models are models learned using different data sets, respectively. In this manner, it is possible to perform different detection.
[0015] It is preferable that the plurality of learned models are models learned using data sets consisting of images captured at different in-vivo positions, respectively. In this manner, it is possible to detect a lesion region according to the in-vivo position of the image.” See also paragraphs 82, 92, 96, 104 and 126. Kamon )
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DYLAN J MENDEZ MUNIZ whose telephone number is (703)756-5672. The examiner can normally be reached M-F, 8AM - 5PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Moyer can be reached at (571) 272-9523. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DYLAN JOHN MENDEZ MUNIZ/Examiner, Art Unit 2675
/ANDREW M MOYER/Supervisory Patent Examiner, Art Unit 2675
Prosecution Insights