Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. JP2022-165970, filed on 10/17/2022.
Response to Amendment
The amendment filed on 12/08/2025 has been entered.
Claims 1 and 12 were amended. Claims 1-13 remain pending in the application.
The title objection remains as any generic machine learning algorithm involving images and bounding boxes can specify region of interest.
Response to Arguments
Applicant’s arguments have been considered but are not fully persuasive.
Applicant makes the following argument on pages 9-10 of the remarks, reproduced below:
PNG
media_image1.png
512
717
media_image1.png
Greyscale
Upon further review of the reference and in light of applicant's argument, the examiner respectfully disagrees as follows: first of all, In response to applicant's argument that the references fail to show certain features of the invention, it is noted that the features upon which applicant relies (i.e., disposing ROI near the arrow) are not recited in the rejected claim(s). Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993). Further, the claims are written broad enough as to be encompassed by Gupta.
In response to applicant's argument that the references fail to show certain features of the invention, it is noted that the features upon which applicant relies (i.e., []) are not recited in the rejected claim(s). Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993).
Applicant further argues on pg 10, reproduced below:
PNG
media_image2.png
592
730
media_image2.png
Greyscale
Upon further review of the reference and in light of applicant's argument, the examiner respectfully disagrees as follows: first of all, In response to applicant's argument that the references fail to show certain features of the invention, it is noted that the features upon which applicant relies (i.e., generative approach where the system automatically generates and disposes one or more region-of-interest candidates based on the arrows vector information) are not recited in the rejected claim(s). Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993). Further, the claims are written broad enough as to be encompassed by Gupta.
Applicant further argues on pg 11, reproduced below:
PNG
media_image3.png
808
730
media_image3.png
Greyscale
Upon further review of the reference and in light of applicant's argument, the examiner respectfully disagrees as follows: first of all, In response to applicant's argument that the references fail to show certain features of the invention, it is noted that the features upon which applicant relies (i.e., interest degree calculation to evaluate and rank multiple ROI candidates located at different, non-overlapping positions in the image) are not recited in the rejected claim(s). Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993). Further, the claims are written broad enough as to be encompassed by Gupta. Further, as applicant has stated, NMS involves a “highest confidence score”, which involves an evaluation and ranking of bounding boxes; the instant specification does not appear to specifically disavow confidence score as interest degree using broad language such as “interest degree may be” (bold and underline for emphasis). Additionally, “dispose” does not appear to be specifically defined in the instant specifications, the definition, as one with ordinary skill in the art would know, can include arrangement or removing. See *note below.
Regarding independent claim 12, which cites similar limitations as claim 1, are rejected for similar reasons.
Regarding dependent claims 2-11 and 13, depends on the independent claims above, and are rejected for similar reasons.
Accordingly, the claims, as they are currently written, remain rejected.
Potential pathway towards advancing prosecution: Specifying the limitations with language consistent from the specifications into the claims would advance prosecution as the specifications appear to disclose a potentially allowable inventive concept.
*Note: “dispose” may mean to adjust by arrange the parts or “to throw out or away” (to name a few definitions, more available at the source: https://www.dictionary.com/browse/dispose).
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed. The title objection remains as any generic machine learning algorithm specifies a region of interest from images, if this is the goal of the machine learning algorithm. Although the title does not necessarily need to be as long; applicant may draw inspiration from the DERWENT version of the title: “Medical Image Analysis Apparatus For Use In Medical Image Analysis System For Using Image In Which Region Of Interest Is Annotated With Arrow In Training Learning Model, Has Processor That Specifies Region Of Interest From Among Region-of-interest Candidates Based On Interest Degree”.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-7, 9-13 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Gupta (“Information Extraction from Hand-marked Industrial Inspection Sheets”, 2018).
Regarding claims 1 and 12, Gupta teaches An image analysis apparatus (Gupta, abstract: “In this paper, we propose a novel pipeline to build an information extraction system for such machine inspection sheets, utilizing state-of-the-art deep learning and computer vision techniques”) comprising:
at least one processor (Gupta, abstract: “utilizing state-of-the-art deep learning and computer vision techniques”. Which is being interpreted as using at least one processor); and
at least one memory in which an instruction to be executed by the at least one processor is stored (Gupta, abstract: “utilizing state-of-the-art deep learning and computer vision techniques”. Which is being interpreted as using at least one memory with instructions),
wherein the at least one processor (Gupta, abstract: “utilizing state-of-the-art deep learning and computer vision techniques”. Which is being interpreted as using at least one processor) is configured to:
receive an image in which an arrow is assigned to a region of interest (Gupta, pg 3, column 1, subsection 2, reproduced below:
PNG
media_image4.png
932
1168
media_image4.png
Greyscale
. “Inspection sheets” is being interpreted as an image with an arrow. “handwritten text” is being interpreted as a “region of interest”);
specify the arrow (Gupta, see subsection 2 image above: “learn the arrow structure” shows “specify the arrow”);
dispose one or more region-of-interest candidates that are candidates of the region of interest (Gupta, pg 4, column 1, ¶1, reproduced below:
PNG
media_image5.png
886
1164
media_image5.png
Greyscale
. “Overlapping regions” is being interpreted as “one or more region-of-interest candidates”) a direction of the arrow (Gupta, see subsection 2 image above: “The handwritten text is present at the arrow tail”. Which shows the text is in a direction of the arrow. The instant claims do not specify arrow head or tail; as the direction may simply be in the vicinity of the arrow) in accordance with a distance from the arrow (Gupta, see subsection 2 image above: “The handwritten text is present at the arrow tail”. “Present at the arrow tail” shows “in accordance with a distance from the arrow”.);
calculate an interest degree for each region-of-interest candidate (Gupta, see the pg 4, column 1, ¶1 image above: “Non-Maximum Suppression (NMS)” is being interpreted as involve calculating an “interest degree”. As one with ordinary skill in the art would know, NMS uses intersection-over-union to remove overlapping regions, which is being interpreted as an interest degree); and
specify the region of interest from among the region-of-interest candidates (Gupta, see the pg 4, column 1, ¶1 image above: “Overlapping regions”) based on the interest degree (Gupta, see the pg 4, column 1, ¶1 image above: “Non-Maximum Suppression (NMS)”. NMS is being interpreted as “specify the region of interest” amongst the region-of-interest candidates. NMS is being interpreted as involve calculating an “interest degree”. As one with ordinary skill in the art would know, NMS uses intersection-over-union to remove overlapping regions, which is being interpreted as an interest degree).
Regarding claim 2, Gupta teaches The image analysis apparatus according to claim 1, wherein the region-of-interest candidate has a shape with which a size (Gupta, pg 4, column 1, ¶1: “we apply empirically chosen upper and lower threshold of 4000 and 100 square pixels”. The pixel range is being interpreted as size) and a position of the region of interest is specifiable (Gupta, see subsection 2 image above: “text is present at the tail” is being interpreted as position that is specifiable. The pixel range from pg 4, column 1, ¶1 shows specifiable pixel range).
Regarding claim 3, Gupta teaches The image analysis apparatus according to claim 2, wherein the region-of-interest candidate is a rectangle (Gupta, pg 4, column 1, ¶1: “to get bounding boxes for all text segments”. “Bounding boxes” is being interpreted as “a rectangle”) or a circle.
Regarding claim 4, Gupta teaches The image analysis apparatus according to claim 1, wherein the at least one processor is configured to dispose the region-of-interest candidate in accordance with a first distance that is a first distance from a tip end of the arrow (Gupta, pg 3, column 1, subsection 2: “The handwritten text is present at the arrow tail”) and that is in a normal direction of the direction of the arrow (Gupta, pg 3, column 1, subsection 2: “The handwritten text is present at the arrow tail”. Which is being interpreted as part of a normal direction of the direction of the arrow).
Regarding claim 5, Gupta teaches The image analysis apparatus according to claim 1, wherein the at least one processor is configured to dispose the region-of-interest candidate (Gupta, see Fig 5 image below, the text boxes near the top of Fig 5b) in accordance with a second distance (Gupta, see Fig 5b image below, the text boxes are a second distance away from the tip end of the arrow) that is a second distance from a tip end of the arrow (Gupta, see Fig 5b image below, the text boxes are a second distance away from the tip end of the arrow) and that is in a direction parallel to the direction of the arrow (Gupta, pg 4, Figure 5, reproduced below:
PNG
media_image6.png
632
1172
media_image6.png
Greyscale
. Fig 5b, which shows the text boxes above the arrow are in a direction parallel to the direction of the arrow).
Regarding claim 6, Gupta teaches The image analysis apparatus according to claim 1, wherein the at least one processor is configured to:
acquire accessory information of the image (Gupta, see Figure 5 image above, the detected text is being interpreted as “accessory information”); and
dispose the region-of-interest candidate in accordance with the accessory information (Gupta, see Figure 5 image above; pg 4, column 2, lines 2-3: “If we get multiple clusters
in this sector, the closest one from the tail is chosen”. “Multiple clusters” is being interpreted as region-of-interest candidates that were then disposed when the closest one from the tail was chosen).
Regarding claim 7, Gupta teaches The image analysis apparatus according to claim 6, wherein the accessory information includes a sentence in which a content of the image is described (Gupta, see Figure 5b image above, the detected text is being interpreted as “accessory information includes a sentence”).
Regarding claim 9, Gupta teaches The image analysis apparatus according to claim 1,
wherein the at least one processor is configured to calculate the interest degree for each region-of-interest candidate (Gupta, pg 4, column 1, ¶1: “Non-Maximum Suppression (NMS)”. Which is being interpreted as calculate the interest degree for each region-of-interest candidate)
using a first interest degree calculation model that outputs the interest degree of the input region-of-interest candidate in a case where a feature of the image and the region of interest are input (Examiner note: as this is an “or” statement, the second of the choices will be mapped), or
a second interest degree calculation model that outputs the interest degree of the input region-of-interest candidate in a case where the image and the region-of-interest candidate are input (Gupta, pg 4, column 1, ¶1: “Once we have a pre-processed image, we use connected components analysis to get bounding boxes for all text segments and objects present in the inspection sheet. Overlapping regions are removed using Non-Maximum Suppression (NMS)”. Which shows an image and region-of-interest candidate input, then NMS outputs an interest degree as part of the process).
Regarding claim 10, Gupta teaches The image analysis apparatus according to claim 9,
wherein the first interest degree calculation model (Gupta, pg 5, column 1, Section B, ¶2: “The Faster-RCNN [2] is trained on the manually annotated arrow images from the complete training set”) and the second interest degree calculation model (Gupta, pg 4, column 1, ¶1: “Once we have a pre-processed image, we use connected components analysis to get bounding boxes for all text segments and objects present in the inspection sheet. Overlapping regions are removed using Non-Maximum Suppression (NMS)”) are trained models that are trained by disposing a plurality of regions in an image having a known region of interest (Gupta, pg 5, column 1, Section B, ¶2: “The Faster-RCNN is trained on the manually annotated arrow images from the complete training set”. “Manually annotated arrow images” are being interpreted to include “known region of interest”) and by using an interest degree between the disposed region and the known region of interest as correct answer data (Gupta, pg 5, column 1, Section B, ¶2: “By keeping the confidence threshold greater than 0:9 and Non Maximal Suppression (NMS) threshold less than 0:05”. “Manually annotated arrow images” are being interpreted to include “known region of interest”. “Confidence” is being interpreted as “interest degree”. “Disposed region” is being interpreted as being output by the model with a confidence value, as one with ordinary skill in the art would know. “Known region of interest as the correct answer data” is being interpreted as the ground truth data needed for training; the Faster-RCNN is interpreted as having ground truth data for training).
Regarding claim 11, Gupta teaches The image analysis apparatus according to claim 1
wherein the at least one processor (Gupta, abstract: “utilizing state-of-the-art deep learning and computer vision techniques”. Which is being interpreted as using at least one processor) is configured to:
dispose a plurality of the region-of-interest candidates (Gupta, pg 4, column 1, ¶1: “Once we have a pre-processed image, we use connected components analysis to get bounding boxes for all text segments and objects present in the inspection sheet. Overlapping regions are removed using Non-Maximum Suppression (NMS)”. “Bounding boxes” are being interpreted as “plurality of region-of-interest candidates”); and
specify the region-of-interest candidate having a highest interest degree among the plurality of region-of-interest candidates as the region of interest (Gupta, pg 4, column 1, ¶1: “Non-Maximum Suppression (NMS)”. NMS is being interpreted to include “highest interest degree” as part of the process of specifying the region-of-interest candidate among the plurality of region-of-interest candidates).
Regarding claim 13, Gupta teaches A non-transitory, computer-readable tangible recording medium (Gupta, abstract: “utilizing state-of-the-art deep learning and computer vision techniques”. Which is being interpreted as having a non-transitory, computer-readable tangible recording medium) which records thereon a program for causing, when read by a computer, the computer to execute the image analysis method according to claim 12 (Gupta, abstract: “utilizing state-of-the-art deep learning and computer vision techniques”. Which is being interpreted as having a non-transitory, computer-readable tangible recording medium with a program that executes the image analysis method according to claim 12).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gupta, in view of Santosh (“Overlaid Arrow Detection for Labeling Regions of Interest in Biomedical Images”, 2016) and QIBA (“Segmentation and Markup Formats”, Aug 2022).
Regarding claim 8, Gupta teaches The image analysis apparatus according to claim 6 (Gupta, abstract: “In this paper, we propose a novel pipeline to build an information extraction system for such machine inspection sheets, utilizing state-of-the-art deep learning and computer vision techniques”),
However, Gupta does not appear to explicitly teach medical image.
Pertaining to the same field of endeavor, Santosh teaches
wherein the image is a medical image (Santosh, pg 67, Figure 1b—which shows a sample medical image),
the region of interest is a lesion (Santosh, pg 67, Figure 1b—which shows a sample medical image with the top-left black arrow pointing at a region of interest that is being interpreted as a lesion), and
Gupta and Santosh are considered to be analogous art because they are directed to medical image analysis. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method and system for arrow and non-arrow object detection (as taught by Gupta) to include medical images (as taught by Santosh) because the combination provides an improvement arrow detection that leads to highlighting meaningful ROIs (Santosh, pg 66, column 1, ¶1).
However, Gupta and Santosh does not appear to explicitly teach “size of the lesion”.
Pertaining to the same field of endeavor, QIBA teaches
the accessory information includes information about a size of the lesion (QIBA, pg 4, lines 1-3: “A common use is to encode the extent of a single lesion segmentation in one segmentation object, and reference it from a DICOM Structured Report that contains the meta-data (esp. measurements) of the segmented lesion.” “Measurements” are being interpreted as “size of the lesion”)
Gupta, Santosh, and QIBA are considered to be analogous art because they are directed to medical image analysis. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method and system for arrow detection in medical images (as taught by Gupta and modified by Santosh) to include a size of the lesion (as taught by QIBA) because the combination provides an improvement arrow detection that leads to highlighting meaningful ROIs (Santosh, pg 66, column 1, ¶1).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Ortiz et al (US 2018/0293438 A1) discloses an arrow and associated region detection method.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOHNNY B DUONG whose telephone number is (571)272-1358. The examiner can normally be reached Monday - Thursday 10a-9p (ET).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at (571)272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.B.D./Examiner, Art Unit 2667
/MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667