DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
The IDS filed on 1/19/2024 has been received and considered.
Claims 1 – 7, all of the claims pending in this application, have been rejected.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13. The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1 – 7 are provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1 – 7 of a copending Application No. 18/417,288. Although the claims at issue are not identical, they are not patentably distinct from each other because they overlap in scope as shown in the table below.
This is a provisional nonstatutory double patenting rejection because the patentably indistinct claims have not in fact been patented. While claim 1 of the present invention is limited to endoscopic procedures and claim 1 of the ‘288 application is not, Claim 5 (dependent from 1) of the ‘288 application does require endoscopic procedures. Claim 5 also (in light of the specification) has decision making based on a detection or non-detection.
#
US 20240202916 A1 (18/417,144)
(Present Application)
#
US 20240169535 A1 (18/417,288)
(Co-pending)
1
A lesion detection apparatus comprising: a memory storing instructions;
1
An object detection apparatus comprising: a memory storing instructions;
1
at least one processor configured to execute the instructions to: acquire one or more images captured by endoscopic examination, the one or more images including a first image;
1
at least one processor configured to execute the instructions to: acquire one or more images, the one or more images including a main image;
1
calculate a first map from the first image with use of a first model;
1
calculate a first map from the main image with use of a first model;
1
detect a lesion with reference to at least the first map;
1
detect an object with reference to at least the first map;
1
determine whether a second image that is captured by a past endoscopic examination is present, the second image indicating the same place of a same subject as the first image, wherein no lesion is detected from the second image;
1
determine whether a background image is present;
(where no lesion being detected would essentially be the same as only a background since the lesion is would be an object present in a foreground image)
1
in a case where the second image is present, calculate, with use of a second model, a second map from the second image or from both the first image and the second image; and
1
in a case where the background image is present, calculate, with use of a second model, a second map from the background image or from both the main image and the background image; and
1
in a case where the second image is present, detect the lesion with reference to both the first map and the second map.
1
in a case where the background image is present, detect the object with reference to not only the first map but also the second map.
2
The lesion detection apparatus according to claim 1, wherein the at least one processor is further configured to execute the instructions to: in a case where the second image is present, detect the lesion with reference to a third map obtained by multiplying the first map by the second map.
2
The object detection apparatus according to claim 1, wherein the at least one processor is further configured to execute the instructions to: in a case where the background image is present, detect the object with reference to a third map obtained by multiplying the first map by the second map.
3
The lesion detection apparatus according to claim 1, wherein the determining comprises referring to a flag indicating whether the first image is present or whether the first image and the second image are present.
3
The object detection apparatus according to claim 1, wherein the determining comprises referring to a flag indicating whether the main image is present or whether the main image and the background image are present.
4
The lesion detection apparatus according to claim 1, wherein the at least one processor is further configured to execute the instructions to: acquire training data which includes at least one first image, at least one second image, and label information indicative of a lesion included in the at least one first image;
4
The object detection apparatus according to claim 1, wherein the at least one processor is further configured to execute the instructions to: acquire training data which includes at least one main image, at least one background image, and label information indicative of an object included in the at least one main image;
4
train the first model by machine learning with reference to the at least one first image and the label information which are included in the training data; and
4
train the first model by machine learning with reference to the at least one main image and the label information which are included in the training data; and
4
train the first model and the second model by machine learning with reference to the at least one first image, the at least one second image, and the label information which are included in the training data.
4
train the first model and the second model by machine learning with reference to the at least one main image, the at least one background image, and the label information which are included in the training data.
5
The lesion detection apparatus according to claim 1, wherein the detection result of the lesion supports decision making by a medical worker.
5
The object detection apparatus according to claim 1, wherein the at least one processor further is configured to execute the instructions to: detect the object that is a lesion which is capable of being detected from an image captured by carrying out an endoscopic examination with respect to a subject, and output a result of detection of the lesion for supporting decision making by a medical worker, the result being obtained by the detecting.
(where the endoscopic examination that captures an image is claimed in claim 1 of present application)
6
A lesion detection method comprising: acquiring one or more images captured by endoscopic examination, the one or more images including a first image;
6
An object detection method comprising: acquiring one or more images, the one or more images including a main image;
6
calculating a first map from the first image with use of a first model;
6
calculating a first map from the main image with use of a first model;
6
detecting a lesion with reference to at least the first map;
6
detecting an object with reference to at least the first map;
6
determining whether a second image that is captured by a past endoscopic examination is present, the second image indicating the same place of a same subject as the first image, wherein no lesion is detected from the second image,
6
determining whether a background image is present;
(where no lesion being detected would essentially be the same as only a background since the lesion is would be an object present in a foreground image)
6
in a case where the second image is present, calculating, with use of a second model, a second map from the second image or from both the first image and the second image; and
6
in a case where the background image is present, calculating, with use of a second model, a second map from the background image or from both the main image and the background image; and
6
in a case where the second image is present, detecting the lesion with reference to both the first map and the second map.
6
in a case where the background image is present, detecting the object with reference to not only the first map but also the second map.
7
A non-transitory tangible computer-readable storage medium storing therein a lesion detection program causing a computer to execute the processing comprising:
7
A non-transitory tangible computer-readable storage medium storing therein an object detection program causing a computer execute the processing comprising:
7
acquiring one or more images captured by endoscopic examination, the one or more images including a first image;
7
acquiring one or more images, the one or more images including a main image;
7
calculating a first map from the first image with use of a first model;
7
calculating a first map from the main image with use of a first model;
7
detecting a lesion with reference to at least the first map;
7
detecting an object with reference to at least the first map;
7
determining whether a second image that is captured by a past endoscopic examination is present, the second image indicating the same place of a same subject as the first image, wherein no lesion is detected from the second image;
7
determining whether a background image is present;
(where no lesion being detected would essentially be the same as only a background since the lesion is would be an object present in a foreground image)
7
in a case where the second image is present, calculating, with use of a second model, a second map from the second image or from both the first image and the second image; and
7
in a case where the background image is present, calculating, with use of a second model, a second map from the background image or from both the main image and the background image; and
7
in a case where the second image is present, detecting the lesion with reference to both the first map and the second map.
7
in a case where the background image is present, detecting the object with reference to not only the first map but also the second map.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3 – 4, and 6 – 7 are rejected under 35 U.S.C. 103 as being unpatentable over JP Publication No. 2021-65606 A to MASASHI et al. (hereinafter MASASHI) in view of Non-Patent Literature "Background Subtraction With Real-Time Semantic Segmentation" to Zeng et al. (hereinafter Zeng).
Examiner notes that a translated version of JP 2021-65606 A (hereinafter “Translated Document”) will be used in reference to the respective JP publication throughout this Office Action.
Claim 1
Regarding Claim 1, MASASHI teaches a lesion detection apparatus comprising: a memory storing instructions (Translated Document lines 531 - 532);
at least one processor configured to execute the instructions (Translated Document lines 519 - 520) to: acquire one or more images captured by endoscopic examination, the one or more images including a first image ("In step S1 (first acquisition step), the light source device 5 is operated with the endoscope 2 inserted into the observation target (for example, the lower gastrointestinal organ), and the image acquisition unit 6b acquires the endoscope image. The endoscopic image is used as a first medical image", Translated Document - lines 82 - 85);
determine whether a second image that is captured by a past endoscopic examination is present, the second image indicating the same place of a same subject as the first image, wherein no lesion is detected from the second image ("In the present embodiment, the second medical image is acquired at a certain timing, and the first medical image is acquired after a predetermined period (1 year) has elapsed. Then, when a lesion is found in the first medical image, a first region of interest is set in the lesion portion, and the feature amount of the first medical image is compared with the feature amount of the second medical image for the second medical use. A second region of interest corresponding to the first region of interest (lesion) in the image is specified. If the second region of interest does not include the lesion, the lesion has occurred by the time the predetermined period elapses. That is, the tissue of the second region of interest is in the pre-stage of lesion development.", Translated Document - lines 265 - 273); and
display a detection result of the lesion to an output device ("In step S55, the prediction result is displayed on the monitor 4. The mode of notifying the user of the prediction result is not particularly limited.", Translated Document lines 332 - 333).
Although MASASHI teaches the detection of a lesion in a first and second image, MASASHI does not explicitly teach calculate a first map from the first image with use of a first model;
detect a lesion with reference to at least the first map;
in a case where the second image is present, calculate, with use of a second model, a second map from the second image or from both the first image and the second image; and
in a case where the second image is present, detect the lesion with reference to both the first map and the second map.
However, Zeng teaches calculate a first map from the first image with use of a first model (Figure 1; "To reduce network computational complexity, a multi-resolution cascade network architecture was proposed. Its core idea is to let the low-resolution image go through the full semantic network first for a coarse segmentation map.", Section III - Background Subtraction with Real-Time Semantic Segmentation, Part B. - ICNet for Real-Time Semantic Segmentation);
PNG
media_image1.png
331
340
media_image1.png
Greyscale
detect a lesion with reference to at least the first map ("The BGS segmenter B aims to construct background (BG) models and segments FG objects. The real-time semantic segmenter S is used to refine the FG segmentation outputs as feedbacks for improving the model updating accuracy. B and S work in parallel on two threads. For each input frame It, the BGS segmenter B computes a preliminary FG/BG mask Bt. At the same time, the real-time semantic segmenter S extracts the object-level semantics St. Then, some specific rules are applied on Bt and St to generate the final detection Dt.", Abstract), wherein the object detection (Foreground detection) and segmentation for object extraction is taught, but not explicitly for a lesion;
in a case where the second image is present, calculate, with use of a second model, a second map from the second image or from both the first image and the second image (Rejected as applied above pertaining to Figure 1 of Zeng), where each image is input into multiple models to produce segmentation maps; and
in a case where the second image is present, detect the lesion with reference to both the first map and the second map (Rejected as applied above), where the model can be used to specifically detect a lesion present in the image if needed.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of MASASHI to incorporate the detection of a lesion, but by the use of a first and second map generated by segmentation that is associated with their respective images to detect change, as disclosed by Zeng. The suggestion/motivation for doing so would have been automating the process to highlight the detected changes relevant to the second image.
Claim 3
Regarding Claim 3, dependent on claim 1, MASASHI, in view of Zeng, teaches the invention as claimed in claim 1.
MASASHI further teaches wherein the determining comprises referring to a flag indicating whether the first image is present or whether the first image and the second image are present (Figure 9; "If there is a lesion in the first medical image (YES in step S28), the process proceeds to step S29. In step S29 (first extraction step), the feature amount extraction unit 6d extracts the feature amount of the first medical image. In this embodiment, the feature amount is a blood vessel pattern. In step S30 (first setting step), the region of interest setting unit 6e sets the first region of interest on the first medical image. In the present embodiment, the region of interest setting unit 6e sets the first region of interest so as to surround the lesion. The first region of interest may be set automatically or manually. In step S31 (comparison step), the comparison unit 6g compares the feature amount of the first medical image with the feature amount of the second medical image with reference to the past data D2. If there is no same feature amount (NO in step S32), the process ends because there is no second medical image containing the same part as the first medical image. When there is the same feature amount (YES in step S32), the first medical image and the second medical image containing the same feature amount can be regarded as images containing the same part. Subsequently, the process proceeds to step S33 (interest region identification step), and the interest region identification unit 6h identifies the second interest region corresponding to the first interest region in the second medical image based on the comparison result in step S31.", Translated Document lines 241 - 259), where in order to perform the comparison there would necessarily need to be an indication in the computer code of the presence of the images before they could be compared.
Claim 4
Regarding Claim 4, dependent on claim 1, MASASHI, in view of Zeng, teaches the invention as claimed in claim 1.
MASASHI further teaches wherein the at least one processor is further configured to execute the instructions to: acquire training data which includes at least one first image, at least one second image, and label information indicative of a lesion included in the at least one first image ("Therefore, by performing the flow shown in FIGS. 8 and 9 for a plurality of subjects, the second medical image to which the second region of interest is added is accumulated as a set of teacher data D3 (learning data set), and the machine. By performing the learning, it is possible to predict the occurrence of the lesion after a predetermined period based on the image of the tissue in which the lesion has not occurred, as described below.", Translated Document, lines 274 - 278);
train the first model by machine learning with reference to the at least one first image and the label information which are included in the training data ("First, in step S71, the first region of interest is surgically excised to obtain the tissue of the lesion. Subsequently, in step S72, the resected tissue is inspected and pathological diagnostic information is acquired. Pathological diagnostic information includes information regarding the extent of lesions and non-lesions in the first region of interest. Subsequently, in step S73, the pathological diagnosis information is added to the composite enlarged image to create teacher data. In step S74, machine learning is performed using the generated teacher data. As a result, in step S75, a trained model is generated.", Translated Document lines 494 - 502); and
train the first model and the second model by machine learning with reference to the at least one first image, the at least one second image, and the label information which are included in the training data ("Subsequently, in step S34, the teacher data generation unit 6j adds the second area of interest specified in step S33 to the second medical image (past data D2) to generate teacher data D3 for machine learning.", Translated Document lines 260 -264).
Claim 6 and 7, both independent claims, are rejected for the same reasons as applied to claim 1.
Claims 2 is rejected under 35 U.S.C. 103 as being unpatentable over JP Publication No. 2021-65606 A to MASASHI et al. (hereinafter MASASHI) in view of Non-Patent Literature "Background Subtraction With Real-Time Semantic Segmentation" to Zeng et al. (hereinafter Zeng) in further view of US Publication No. 2020/0342589 A1 to Heindl et al. (hereinafter Heindl).
Claim 2
Regarding Claim 2, dependent on claim 1, MASASHI, in view of Zeng, teaches the invention as claimed in claim 1.
Neither MASASHI, or Zeng, or the combination teach in a case where the second image is present, detect the lesion with reference to a third map obtained by multiplying the first map by the second map.
However, Heindl teaches in a case where the second image is present, detect the lesion with reference to a third map obtained by multiplying the first map by the second map (Figure 3, #305 - #307; "During a prediction, an image is analysed using both the CNN-generated malignancy mask and the FCN-generated probability map. During run time, the malignancy model and the lesion segmentation model process the image simultaneously. The probability map is then used to select one or more relevant parts of the malignancy mask. Such a selection may be provided through multiplying the malignancy mask 305 with the one or more binary masks 306. The result then undergoes a post-processing stage 307, during which an overlay is generated. The overlay may comprise any markings one or more parts of the original image, for example by outlining different areas of human breast tissue, regions of interest, and/or marking one or more levels of malignancy if a lesion is detected and can be stored in the DICOM image 308.", Paragraph [0047]).
PNG
media_image2.png
799
487
media_image2.png
Greyscale
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify the teachings of MASASHI, in view of Zeng, to incorporate the detection of a lesion, but by the use of a first and second map generated by segmentation which is then multiplied together to produce a third map, as disclosed by Heindl. The suggestion/motivation for doing so would have been because simply multiplying the maps would be a more efficient process that would also save on computational processing.
Claims 5 is rejected under 35 U.S.C. 103 as being unpatentable over JP Publication No. 2021-65606 A to MASASHI et al. (hereinafter MASASHI) in view of Non-Patent Literature "Background Subtraction With Real-Time Semantic Segmentation" to Zeng et al. (hereinafter Zeng) in further view of US Publication No. 2021/0272277 to OGINO et al. (hereinafter OGINO).
Claim 5
Regarding Claim 5, dependent on claim 1, MASASHI, in view of Zeng, teaches the invention as claimed in claim 1.
Neither MASASHI, or Zeng, or the combination teach wherein the detection result of the lesion supports decision making by a medical worker.
However, OGINO teaches wherein the detection result of the lesion supports decision making by a medical worker ("A processing result of the diagnosis support processing unit 230 may be output to the output unit 120 provided in the image processing apparatus 20, or may be sent to the medical imaging apparatus to which the image data is sent, a facility in which the medical imaging apparatus is placed, a database in another medical institution, etc.", Paragraph [0128]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify the teachings of MASASHI, in view of Zeng, to incorporate outputting the detection result to a medical personnel or institute, as disclosed by OGINO. The suggestion/motivation for doing so would have been to allow for a medical specialists to analyze the results so the best treatment can be recommended for the client/patient.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ronde Miller whose telephone number is (703) 756-5686 The examiner can normally be reached Monday-Friday 8:00-4:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor Gregory Morse can be reached on (571) 272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RONDE LEE MILLER/Examiner, Art Unit 2663
/GREGORY A MORSE/Supervisory Patent Examiner, Art Unit 2698
Prosecution Insights