DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Prior arts cited in this office action:
Ishi et al. (US 20200074231 A1, hereinafter “Ishi”)
Ohba (WO 2019059343 A1, hereinafter “Ohba”)
Boddington et al. (US 20210015560 A1, hereinafter “Boddington”)
Response to Arguments
Applicant's arguments filed 04/13/2026 have been fully considered but they are not persuasive.
Applicant’s Arguments/Remarks: Applicant argues that Independent claims 1 and 14 require, and the combination of Ishii and Ohba fails to disclose, inter alia, "acquir[ing] the object recognition results as annotation data, the annotation data outputted by input of the captured images into the first recognition model." These limitations require a specific auto-annotation workflow, namely that captured images of the detection target are input into the first recognition model, and the resulting object recognition results are acquired as annotation data for those captured images.
Examiner’s Response: Examiner disagrees with applicant assertion above that the combination of the cited prior arts does not teach or suggest applicant invention as claimed. As cited in the previous office action Ishi teach is using and a machine learning model and output from captured images that contains annotation and then further update the model by performing a second training suing the annotated output and the captured images. Ishi teaches “data usage determiner 150 determines the composite image provided to the recognition model as an image having the object unrecognizable by the recognition model, and makes the second determination to determine the training data for the recognition model based on the composite image. According to the second determination result, data usage determiner 150 determines the composite image as the training data. Data usage determiner 150 also determines, as the training data, a corresponding image having the same or similar visual characteristics as the composite image. The corresponding image may be selected or generated from the images stored in training data storage 160. Data usage determiner 150 stores the image determined as the training data, in training data storage 160 as new training data (Ishi [0050] and [0080], [0086], [0096])”. Ishi Further teaches “ For example, when data usage determiner 150 makes the second determination to make the first determination, similar scene retriever 190 selects an image having the same or similar visual characteristics as the composite image from among the captured images stored in training data storage 160. The same or similar visual characteristics as the composite image include, for example, the object synthesis position in the image, the image background, the aspect of the synthesized object such as the posture of a person, statistical characteristics of the image parameter such as image color tone and image edge, and qualitative characteristics such as the weather condition, the wet load surface, and occlusion. The captured image may be a captured image selected by sampling unit 112 and stored as new training data, or may be a captured image included in pre-stored DB. It should be noted that the pre-stored DB includes images of various scenes stored as default in the information processing system (Ishi [0113]). As can be seen above and throughout Ishi teaches determining a first output using the model and the composite/synthesized/annotated image and performing update of the model using the composite/synthesized/annotated output and corresponding captured images from the training data storage. As a result, examiner is not convinced by the applicant argument presented with regard to the independent claims.
Applicant’s Arguments/Remarks: actual medical images captured during surgery, Boddington does not teach or suggest that the 3D shape data comprises CAD data of the detection target as required by amended claim 15.
Furthermore, none of the cited references teach or suggest that at least one of the captured
images are captured based on an imaging guide for capturing the captured images, and that the
imaging guide is based on the 3D shape data of the detection target. Amended claim 15 requires
an integrated workflow where CAD data of a specific detection target is used both to generate
composite images for training the first recognition model and to provide an imaging guide for
capturing images of that same detection target. This integrated workflow is not taught or suggested by any of the cited references, alone or in combination.
Examiner’s Response. Examiner disagrees with applicant assertion above that the combination of the cited prior arts does not teach or suggest applicant invention as claim. Ohba teaches as described above, the learning data creation unit 3 artificially creates various training data from CAD data, not from actual photographed images. The learning data creation unit 3 is a program for performing at least one of the above-mentioned translation, rotation, reduction, and enlargement, the change of the color of the work surface, the change of the direction of the light to be irradiated, and the intensity. It is a combination of programs that perform at least one process of change, and the order of processes can be arbitrarily changed…. The training data consists of three-dimensional coordinates (X, Y, Z) of each sample point on the work surface, the color (R, G, B) of the work, and the brightness I, which are stored in an array and associated…. That is, the training data learned by the work learning unit 5 includes the first image data and the second image data. The first preparation procedure is a procedure for generating a large number of first image data for training from design data (CAD data). And Boddington teaches an artificial intelligent intra-operative surgical system and method wherein the system estimates three-dimensional anatomical shape information using the 3D Shape Modeling Module 10 followed by a registration (mapping) of an alignment grid to annotated image using the Image Registration Module 9. The system produces composite image display of any combination of aligned preoperative image, 3D model, and alignment grid using image composition module (11) (Boddington [0007], [0008], [0014]-[0115], fig. 5B).
Therefore, one of ordinary skill in the art before the effective filing date of the application would find it obvious to generate 3D shape data comprising CAD data and to capture images based on an imaging guide for capturing the captured images, the imaging guide based on the 3D shape data of the detection target, in order to obtain good training data for the desired model.
Applicant is reminded that the test for obviousness is not whether the features of a secondary reference may be bodily incorporated into the structure of the primary reference; nor is it that the claimed invention must be expressly suggested in any one or all of the references. Rather, the test is what the combined teachings of the references would have suggested to those of ordinary skill in the art. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981).
Applicant tries to argue the cited references separately wherein the rejection was based on the combination of the references. Applicant is also reminded that one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986).
Claim 2-13 depend at least in part on claim 1 and are therefore not allowable for the same reason given above.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-3, 5-7, 11-14 are rejected under 35 U.S.C. 103 as being unpatentable over Ishi et al. (US 20200074231 A1, hereinafter “Ishi”) in view of Ohba (WO 2019059343 A1, hereinafter “Ohba”).
Regarding claims 1, 14 and 15:
Ishi teaches a recognition model generation method (Ishi Abstract, [0006], where Ishi teaches a recognition model generation method) comprising:
acquiring composite images depicting a detection target (Ishe [0030], [0049], [0094]-[0095], figs. 7 and 8 where Ishi teaches an image synthesis unit 140 that provides composite images a detection processing unit 140);
performing a first training, based on the plurality of composite images, to create a first recognition model configured to output object recognition results for input of an image (Ishi [0049]-[0050], [0056], figs. 7 and 8, wherein Ishi teaches training the recognition model base on the composite image and output object recognition results for the input image);
acquiring captured image images of the detection target (Ishi [0056], [0074], [0110]-[0111], figs. 7 and 8, where Ishi teaches using the captured images from sampling unit 112 and the composite images);
acquiring the object recognition results as annotation data, the annotation
data outputted by input of the captured images into the first recognition model (Ishi [0056], [0074], [0110]-[0111], figs. 7 and 8, where Ishi teaches using the captured images from sampling unit 112 and the recognition results as annotation data from the detection processing unit 140); and
performing a second training, based on the captured images and the annotation data, to create a second recognition model (Ishi [0006], [0056], [0074], [0088], [0110]-[0111], figs. 7 and 8, where Ishi teaches using the captured images from sampling unit 112 and the recognition results as annotation data from the detection processing unit 140 to perform second training to update the recognition model).
Although Ishi teaches using and updating the same model for recognition of the target object, he fails to explicitly teach using the first model as teacher.
However, Ohba discloses a recognition model generation apparatus that generates a second recognition model by training a first recognition model using a captured image of a detection target as teacher data, wherein the first recognition model is a recognition model generated by training an original recognition model used in object recognition using a composite image generated on the basis of three-dimensional shape data of the detection target as teacher data (Ohba [0024]-[0027] and [0037]-[0038], [0065], figs. 4 and 5).
Therefore, taking the teachings of Ishi and Ohba as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the application to receive composite images, perform annotation and object detection by training model, update the model by learning from the previous training and further train the model again accordingly, in order to properly annotate and detect the object in the image.
Regarding claim 2:
Ishi in view of Ohba teaches wherein in the second training, the first recognition model is retrained (Ishi [0056], [0074], [0110]-[0111], figs. 7 and 8, where Ishi teaches the first recognition model is retrained.
Regarding claim 3:
Ishi in view of Ohba teaches wherein the second training is performed with the captured images fewer in number than the composite images used during the first training (Ishi [0056], [0074], [0110]-[0111], figs. 7 and 8).
Regarding claim 5:
Ishi in view of Ohba teaches wherein in the second training, the second recognition model is generated using the captured images to which the annotation data are provided (Ishi [0056], [0074], [0110]-[0111], figs. 7 and 8 wherein the annotation is provided for the captured images).
Regarding claim 6:
Ishi in view of Ohba teaches Wherein in the second training, the first recognition model is retrained by performing domain adaptation using first captured images of the detection target to which annotation data have not been provided, and second captured image images of the detection target to which the annotation data are provided are used to evaluate the second recognition model provided (Ishi [0056], [0074], [0110]-[0111], figs. 7 and 8)
Regarding claim 7:
Ishi in view of Ohba teaches Wherein in a case in which a degree of confidence in annotation of a captured image is equal to or less than a threshold, a composite image of the detection target is generated so as to have an identical feature as the captured image, and
the composite image is used in the second training (Ishi [0050]).
Regarding claim 11:
Ishi in view of Ohba wherein in the annotation, the annotation data are acquired by having the first recognition model recognize removed images yielded by removing noise from the captured images, and in the second training, the first recognition model is trained retrained using the captured images (Ishi [0053] fig. 4).
Regarding claim 12:
Ishi in view of Ohba teaches wherein the composite images are generated using a texture corresponding to a material of the detection target identified based on an image of the detection target captured by imaging means, or a texture selected from a template corresponding to any material (Ishi [0023], [0047]-[0048]).
Regarding claim 13:
Ishi in view of Ohba teaches wherein the annotation data have at least one of a mask image of the detection target and a bounding box surrounding the detection target in a captured image that is acquired (Ishi [0053] fig. 4).
Claims 4, 8-10 are rejected under 35 U.S.C. 103 as being unpatentable over Ishi et al. (US 20200074231 A1, hereinafter “Ishi”) in view of Ohba (WO 2019059343 A1, hereinafter “Ohba”) and in view of Boddington et al. (US 20210015560 A1, hereinafter “Boddington”).
Regarding claim 4:
Ishi in view of Ohba fails to explicitly teach wherein the composite images are generated based on 3D shape data of the detection target using an image guide for the captured of the image of the subject.
However, Boddington teaches an artificial intelligent intra-operative surgical system and method wherein the system estimates three-dimensional anatomical shape information using the 3D Shape Modeling Module 10 followed by a registration (mapping) of an alignment grid to annotated image using the Image Registration Module 9. The system produces composite image display of any combination of aligned preoperative image, 3D model, and alignment grid using image composition module (11) (Boddington [0007], [0008], [0014]-[0115], fig. 5B).
Therefore, taking the teachings of Ishi, Ohba and Boddington as whole, taking the teachings of Ishi and Boddington as whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the application to generate the composite images based on 3D shape data of the target and using image guide to guide in the capturing of the image since it is a well-known technique in the art and when use produce predictable result such as obtaining composite images with enough details that would allow for viewing relevant information regarding the object of interest.
Regarding claim 8:
Ishi in view of Ohba and in view of Boddington teaches wherein at least one of the captured image images is captured based on an imaging guide for capturing the at least one of the captured images, the imaging guide being provided based on 3D shape data (Boddington [0007], [0008], [0014]-[0115], fig. 5B).
Regarding claim 9:
Ishi in view of Ohba and in view of Boddington teaches wherein the at least one of the captured image images is captured by controlling, based on the imaging guide, a robot having attached thereto an imaging apparatus that acquires configured to acquire the at least one of the captured images of the detection target (Boddington [0006]- [0008], [0073], [0114]-[0115], fig. 5B).
Regarding claim 10:
Ishi in view of Ohba and in view of Boddington teaches wherein the imaging guide includes an imaging direction of the detection target as determined based on the 3D shape data (Boddington [0007], [0008], [0014]-[0115], fig. 5B).
Regarding claim 15:
Ishi in view of Ohba and in view of Boddington teaches A recognition model generation apparatus for generating a second recognition model by training a first recognition model using captured images of a detection target as teacher data, wherein the first recognition model is a recognition model generated by training an original recognition model used for object recognition, using composite images generated based on 3D shape data of the detection target as teacher data, the 3D shape data comprising CAD data of the detection target, and at least one of the captured images is captured based on an imaging guide for capturing the captured images, the imaging guide based on the 3D shape data of the detection target. Claim 15 contains the limitations of claim 1 and 4 and are therefore rejected on the same ground as claims 1 and 4 above and see also response to arguments above.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WEDNEL CADEAU whose telephone number is (571)270-7843. The examiner can normally be reached Mon-Fri 9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chieh Fan can be reached at 571-272-3042. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WEDNEL CADEAU/Primary Examiner, Art Unit 2632 June 15, 2026
Prosecution Insights