DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
All amendments to the claims as filed on 8/17/26 have been entered and action follows:
Response to Arguments
Applicant's arguments filed have been fully considered but they are not persuasive.
Applicant argues “However, paragraph [0114] of Kitamura discloses how to detect bleeding source images: identifying an image captured at the most upstream anatomical location (oldest time- stamp) as the representative "bleeding source image" if its G/R color ratio satisfies a predetermined threshold. Kitamura does not assign probability values to image groups, nor does it evaluate or output probabilities representing the likelihood of bleeding among the classified groups.
Therefore, Hirakawa in view of Kitamura at least fails to teach or suggest: classify the plurality of images into a plurality of groups based on a result of comparison between at least one of the saturation, the hue, or the lightness acquired and a predetermined criterion; and assign, to each group of the plurality of groups, information indicating a probability that an image classified in a respective group contains a bleeding source.
as recited in amended claim 1.”, (see Remarks page 13).
Examiner respectfully disagrees, Hirakawa in combination with Kitamura discloses all the limitations as claimed and as rejected below. Also, claim as drafted not claiming “evaluate or output probabilities representing the likelihood of bleeding among the classified groups.”, as stated in the arguments.
Therefore, claims 1 and 3-19 stand rejected.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 and 3-19 are rejected under 35 U.S.C. 103 as being unpatentable over Hirakawa (JP 2006302043, and IDS document) in view of Kitamura et al (US Pub. 2017/0004620).
With respect to claim 1, Hirakawa discloses medical assistance system comprising a processor including hardware, wherein the processor is configured to:
acquire
classify the plurality of images into a plurality of groups based on criterion;
and display the plurality of groups distinguishably from each other, (see page 6, paragraph 4, wherein …That is, the image extraction unit 11 has a function of grouping image data according to imaging target parts such as the stomach, the small intestine, and the large intestine; see page 6 paragraph 6, wherein … the display control unit 16 performs grouping by the image extraction unit 11 and an all-image mode in which all of the image data stored in the captured image database 14 can be selected as the display mode of the image selection area 12 b provided in the display unit 12…), as claimed.
However, Hirakawa fails to explicitly disclose acquire at least one of saturation, hue, or lightness of an area in which bleeding is shown in each of a plurality of images taken inside a subject's body; classify the plurality of images into a plurality of groups based on a result of comparison between at least one of the saturation, the hue, or the lightness acquired and a predetermined criterion; assign, to each group of the plurality of groups, information indicating a probability that an image classified in a respective group contains a bleeding source (emphasis added), as claimed.
Kitamura teaches acquire at least one of saturation, hue, or lightness of an area in which bleeding is shown in each of a plurality of images taken inside a subject's body; classify the plurality of images into a plurality of groups based on a result of comparison between at least one of the saturation, the hue, or the lightness acquired and a predetermined criterion, (emphasis added, see paragraph 0045, wherein … The detecting unit 110 detects abnormal regions based on various features of the intraluminal images... detecting the abnormal region based on color features (color information) of the intraluminal image. Here, an abnormal region such as bleeding, redness, or vascular abnormality is indicated by a specific reddish color, and an abnormal region such as ulcer or aphtha is indicated by a specific whitish color. The detecting unit 110 uses color features, for example, color components (R component, G component, B component) of the pixel value, or values secondarily calculated by a known conversion from the color components (e.g., color difference calculated by YCbCr conversion, hue and saturation calculated by HSI conversion, color ratio such as G/R or B/G) to detect a region indicated by any of the specific colors in the intraluminal image, and defines the region as the abnormal region..); assign, to each group of the plurality of groups, information indicating a probability that an image classified in a respective group contains a bleeding source (emphasis added, see Kitamura paragraph 0114, wherein … the bleeding source detection unit 141a detects bleeding source images (abnormal image showing a bleeding source). Specifically, an abnormal image captured at an imaging position on the most upstream side in a lumen (i.e., the oldest time-series abnormal image) is detected, as the bleeding source image, from abnormal images including an abnormal region having a strong reddish color of the abnormal image group. Here, the abnormal region having a strong reddish color can be determined, for example, as a region having a color ratio G/R not more than a predetermined threshold…), as claimed.
It would have been obvious to one ordinary skilled in the art at the effective date of invention to combine the two references as they are analogous because they are solving similar problem of classifying endoscopic images and both are owned by the same assignee. Teaching of Kitamura using the image components in order to create or classify the images can be incorporated into Hirakawa as suggested (see page 7, paragraph 3, …analyzes the color tone of the pixels constituting the read image data…), for suggestion, modifying the system yields displaying images of the classified images (see Kitamura paragraph 0003), for motivation.
With respect to claim 3, combination of Hirakawa and Kitamura further discloses wherein the processor is configured to: display the image belonging to each of the plurality of groups, together with the information indicating the probability, (see Kitamura paragraph 0114, wherein … the bleeding source detection unit 141a detects bleeding source images (abnormal image showing a bleeding source). Specifically, an abnormal image captured at an imaging position on the most upstream side in a lumen (i.e., the oldest time-series abnormal image) is detected, as the bleeding source image, from abnormal images including an abnormal region having a strong reddish color of the abnormal image group. Here, the abnormal region having a strong reddish color can be determined, for example, as a region having a color ratio G/R not more than a predetermined threshold…), as claimed.
With respect to claims 4 and 5, combination of Hirakawa and Kitamura further discloses wherein the processor is configured to: acquire a selection operation for the group from a user and display the image belonging to the group selected by the selection operation, as a bleeding source candidate image that is likely to contain a bleeding source where blood is flowing out; and wherein the processor is configured to: display the images belonging to the group selected by the selection operation, in descending order of red saturation or in descending order of lightness, (see Kitamura paragraph 0114, wherein … the bleeding source detection unit 141a detects bleeding source images (abnormal image showing a bleeding source). Specifically, an abnormal image captured at an imaging position on the most upstream side in a lumen (i.e., the oldest time-series abnormal image) is detected, as the bleeding source image, from abnormal images including an abnormal region having a strong reddish color of the abnormal image group. Here, the abnormal region having a strong reddish color can be determined, for example, as a region having a color ratio G/R not more than a predetermined threshold. Note that, the threshold of the color ratio G/R used here is preferably set strictly (smaller value) relative to the determination criteria (color ratio G/R) used for detection of the abnormal regions in step S11), as claimed.
With respect to claim 6, combination of Hirakawa and Kitamura further discloses wherein the processor is configured to: judge whether a first image and a second image classified in the same group were shot in the same bleeding section or shot in different bleeding sections and display the first image and the second image in different manners when the first image and the second image were shot in different bleeding sections, (see Hirakawa page 6, paragraph 5, wherein …the color tone of the pixels constituting the image data is analyzed, the ratio of the area having the color tone corresponding to the large intestine is derived (step S107), and it is determined whether or not the derived area ratio exceeds a predetermined threshold value. (Step S108). When the area ratio exceeds the threshold (Yes at Step S108), the processing of the image extraction unit 11 is completed. When the area ratio does not exceed the threshold (No at Step S108), the image data to be processed It is determined whether or not the image data is the last imaged image (step S109)…), as claimed.
With respect to claim 7, combination of Hirakawa and Kitamura further discloses wherein the processor is configured to: judge whether the first image and the second image were shot in the same bleeding section or shot in different bleeding sections, based on at least one of the distance between shooting positions of the first image and the second image, the interval between shooting times of the first image and the second image, or the number of other images shot between shooting of the first image and shooting of the second image, (see Hirakawa page 7, paragraph 4, wherein …when the derived area ratio exceeds the predetermined threshold (step S103, Yes), the image data is read again (step S105), and the imaging target region is the small intestine with respect to the processed image data. A marking process to that effect is performed (step S106)…), as claimed.
With respect to claim 8, combination of Hirakawa and Kitamura further discloses wherein the processor is configured to: identify, as a representative image in each of the groups, the image with the highest saturation in an area in which blood is shown, the image with the highest lightness in the area, or the image with the largest area of the area and display the representative image of each of the plurality of groups, (see Hirakawa page 11, last paragraph wherein …Regarding the bleeding site “an area in which blood is shown”, a processing mode such as detecting the presence or absence of a region showing a color tone corresponding to blood can be considered…), as claimed.
With respect to claim 9, combination of Hirakawa and Kitamura further discloses wherein the processor is configured to: derive the percentage occupied by an area in which blood is shown in the image and classify the plurality of images into the plurality of groups based on a result of comparison between the percentage derived and the predetermined criterion, (see Hirakawa page 11, last paragraph wherein …Regarding the bleeding site, a processing mode such as detecting the presence or absence of a region showing a color tone corresponding to blood “the percentage occupied by an area in which blood is shown” can be considered…), as claimed.
With respect to claim 10, combination of Hirakawa and Kitamura further discloses wherein the processor is configured to: display the image included in the group in which the percentage is greater than or equal to a predetermined threshold among the plurality of groups, as a bleeding source candidate image that is likely to contain a bleeding source where blood is flowing out, (see Kitamura paragraph 0114, wherein …the abnormal region having a strong reddish color can be determined, for example, as a region having a color ratio G/R not more than a predetermined threshold. Note that, the threshold of the color ratio G/R used here is preferably set strictly (smaller value) relative to the determination criteria (color ratio G/R) used for detection of the abnormal regions…), as claimed.
With respect to claims 11 and 12, combination of Hirakawa and Kitamura further discloses wherein the processor is configured to: classify the plurality of images into the plurality of groups based on the shooting time of each of the images; and wherein the processor is configured to: classify the plurality of images into the plurality of groups based on at least one of the distance between shooting positions of two of the images, the interval between shooting times of two of the images, or the number of other images shot between shooting operations for two of the images, (see Hirakawa page 9, paragraph 5, wherein … As the image moves from the left end toward the right end, the image data sequentially corresponds to new image data. As described in steps S204 and S205 in FIG. 4, the selected image 20 to be displayed is switched according to the position of the pointer 12c on the image selection area 12b. For example, the pointer 12c is placed at the left end of the image selection area 12b. If it is positioned, the selected image 20 with the oldest imaging time is displayed on the image display area 12a, and if it is positioned at the right end, the selected image 20 with the newest imaging time is displayed), as claimed.
With respect to claim 13, combination of Hirakawa and Kitamura further discloses wherein the processor is configured to: display the images in descending order of the percentage of the area. (see Hirakawa page 9, paragraph 4, wherein …Specifically, the image display apparatus according to the present embodiment can display images in a pseudo moving image mode or the like in which a large amount of image data is continuously displayed while sequentially switching in the imaging order “descending order” …), as claimed.
With respect to claim 14, combination of Hirakawa and Kitamura further discloses wherein the processor is configured to: acquire a selection operation for the image from a user and display image in sequence from a selected image selected by the selection operation to a bleeding source candidate image that is likely to contain a bleeding source where blood is flowing out, (see Kitamura figure 1, numerical 30 input unit, paragraph 0040 for input from a user keyboard; paragraph 0081, wherein …When a signal for selecting a specific point on the color bar is input from the input unit 30 to the control unit 10, according to the user's operation to the input unit 30…; and paragraph 0110, wherein …representative-image extracting unit 141 preferentially extracts, as the representative image, an abnormal image showing a bleeding source for the abnormal region having a high degree of importance, from each of the abnormal image groups each including identical abnormal regions…), as claimed.
With respect to claim 15, combination of Hirakawa and Kitamura further discloses wherein the processor is configured to: acquire the number of images between the selected image and the bleeding source candidate image, or the time required until the bleeding source candidate image is displayed and display the number of images acquired or the time required for the image display acquired, (see Kitamura paragraph 0081, wherein …Specifically, through image processing in the calculation unit 100, average colors of the series of intraluminal images are calculated, and a color bar in which the average colors are arranged in arrangement order of intraluminal images (time-series order) is formed to be displayed on the display unit 40. A color difference (boundary) between average colors on this color bar corresponds to a boundary between organs in the series of intraluminal images. When a signal for selecting a specific point on the color bar is input from the input unit 30 to the control unit 10, according to the user's operation to the input unit 30, the control unit 10 inputs, to the calculation unit 100, an image number of an intraluminal image corresponding to the selected point), as claimed.
With respect to claim 16, combination of Hirakawa and Kitamura further discloses wherein the processor is configured to: acquire the number of images between the image being displayed and the bleeding source candidate image, or the time required until the bleeding source candidate image is displayed and display the number of images acquired or the time required for the image display acquired, (see Kitamura paragraph 0081, wherein …Specifically, through image processing in the calculation unit 100, average colors of the series of intraluminal images are calculated, and a color bar in which the average colors are arranged in arrangement order of intraluminal images (time-series order) is formed to be displayed on the display unit 40. A color difference (boundary) between average colors on this color bar corresponds to a boundary between organs in the series of intraluminal images. When a signal for selecting a specific point on the color bar is input from the input unit 30 to the control unit 10, according to the user's operation to the input unit 30, the control unit 10 inputs, to the calculation unit 100, an image number of an intraluminal image corresponding to the selected point), as claimed.
With respect to claim 17, combination of Hirakawa and Kitamura further discloses wherein the processor is configured to: perform predetermined notification when the bleeding source candidate image is displayed, (see Kitamura paragraph 0064, wherein …the following step S15, the calculation unit 100 outputs, as a result of extraction of the representative image, information indicating the representative image extracted from each of the abnormal image groups in step S14. Accordingly, the recording unit 50 adds information (flag) indicating the representative image to image data of an intraluminal image extracted as the representative image), as claimed.
Claims 18 and 19 are rejected for the same reasons as set forth in the rejection of claims 1 and 3, because claims 18 and 19 are claiming subject matter of similar scope as claimed in claims 1 and 3.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VIKKRAM BALI whose telephone number is (571)272-7415. The examiner can normally be reached Monday-Friday 7:00AM-3:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached at 571-272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/VIKKRAM BALI/ Primary Examiner, Art Unit 2663
Prosecution Insights