DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The Amendment filed October 15, 2025 has been entered. Claims 1-10 remain pending in the application. Applicant’s amendments to the Specification and Claims have overcome each and every objection previously set forth in the Non-Final Office Action mailed May 15, 2025. In addition, the filed Terminal Disclaimer overcomes the provisional nonstatutory double patenting rejection.
Specification
The disclosure is objected to because of the following informalities:
On page 14 lines 16-17, “Based on the source image (x1) and the second generated image a2” should read “Based on the target image (x2) and the second generated image a2”
On page 14 lines 23-24, “a loss value at a position corresponds to a pixel position of the source image (x1) and of the second generated image (a2)” should read “a loss value at a position corresponds to a pixel position of the target image (x2) and of the second generated image (a2)”
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1-4, 6 and 9-10 are rejected under 35 U.S.C. 103 as being unpatentable over Mejjati et al. (NPL "Unsupervised Attention-guided Image-to-Image Translation") in view of Strohmann et al. (NPL "Semantic segmentation of synchrotron tomography of multiphase Al-Si alloys using a convolutional neural network with a pixel-wise weighted loss function").
Regarding claim 1, Mejjati discloses a computer-implemented method for training a machine learning system (page 2: “The whole network is trained end-to-end with no additional supervision”), the method comprising the following steps: providing a source image (page 4: “input image s Є S”) from a source domain (page 3: “a source image domain S”) and a target image (page 5: “target sample t”) of a target domain (page 3: “a target image domain T”); determining a first generated image (Equation 1, page 4: “the mapped image s’ ”) based on the source image using a first generator of the machine learning system (page 4: “the generator FS→T , which maps s to the target domain T”), and determining a first reconstruction based on the first generated image using a second generator of the machine learning system (page 4: “ s” is obtained from s’ via FT→S and AT”); determining a second generated image based on the target image using the second generator (page 9: “ t’ via the inverse mapping [FT→S]”), and determining a second reconstruction based on the second generated image using the first generator (Figure 2: “The roles of S and T are symmetric in our network, so that data also flows in the opposite direction T→S”; therefore, it is understood t” is obtained from t’ via FS→T); determining a first loss value, wherein the first loss value characterizes a first difference of the source image and of the first reconstruction (Equation 3, page 4: “a cycle-consistency loss to the overall framework by enforcing a one-to-one mapping between s and the output of its inverse mapping s”: Lscyc”), and determining a second loss value, wherein the second loss value characterizes a second difference of the target image and of the second reconstruction (Equation 4: it is known in the art that the backward cycle-consistency loss Ltcyc characterizes a difference between t and t”, as evidenced by supporting NPL document “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks” – see conclusion); training the machine learning system by training the first generator (page 3: “The training of the transfer network FS→T”) and/or the second generator (page 3: “the inverse map FT→S…[is] simultaneously trained”) based on the first loss value and/or the second loss value (Equation 4, page 4: “We obtain the final energy to optimize by combining the adversarial and cycle-consistency losses for both source and target domains” including Lscyc and Ltcyc). Mejjati additionally discloses a first attention map (page 3: “Sa…the attention map[s] induced from S”) and a second attention map (page 3: “Ta…the attention map[s] induced from T”). These maps are applied to the output of the respective generator to constrain it to relevant image regions (Figure 2). However, Mejjati fails to explicitly disclose the first difference is weighted according to a first attention map and the second difference is weighted according to a second attention map. In the related art of training neural networks, Strohmann discloses the first difference is weighted according to a first attention map and the second difference is weighted according to a second attention map (Strohmann Figure 3, page 4: “The loss is weighted on a pixel-wise level to the purpose of highlighting the interfaces (wInt) and silicon (wSi) pixels during training”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Mejjati to incorporate the teachings of Strohmann to account for features which are hard to identify in the images and play a relevant role for the correct description of the architecture of the object investigated (Strohmann page 2).
Regarding claim 2, Mejjati, modified by Strohmann, discloses the method claimed in claim 1, wherein: (i) the first attention map respectively characterizes for each pixel of the source image whether or not the pixel belongs to an object depicted in the source image (Mejjati Figures 2 and 3, page 4: “input [s] is fed to the attention network AS, resulting in the attention map sa = AS(s)…for AS to focus on the objects or areas that the corresponding discriminator thinks are the most descriptive within its domain (i.e., the horses)”), and/or (ii) the second attention map respectively characterizes for each pixel of the target image whether or not the pixel belongs to an object depicted in the target image (Mejjati page 3: “AT: T → Ta, where…Ta [is] the attention map induced from T”; Ta is the equivalent of sa for the target image).
Regarding claim 3, Mejjati, modified by Strohmann, discloses the method claimed in claim 1, wherein: (i) the first attention map is determined based on the source image (Mejjati Figure 3, page 4: “input [s] is fed to the attention network AS, resulting in the attention map sa = AS(s)”) using an object detector (Mejjati Figure 2, page 4: AS is trained “to focus on the objects or areas that the corresponding discriminator thinks are the most descriptive within its domain (i.e., the horses)”), and/or (ii) the second attention map is determined based on the target image using the object detector (Mejjati page 3: “AT: T → Ta, where…Ta [is] the attention map induced from T”; AT is the equivalent of AS for the target image).
Regarding claim 4, Mejjati, modified by Strohmann, discloses the method claimed in claim 3, wherein the steps of the method are performed iteratively (Mejjati Algorithm 1: the training procedure is performed iteratively via the for loops) and the object detector determines a first attention map for a source image in each iteration (Mejjati Algorithm 1: for each iteration, either s’ or s’new is computed, both requiring AS to determine sa) and/or determines a second attention map for a target image in each iteration (Mejjati page 5: “training FT→S is similar”; thus, each iteration also involves AT determining Ta).
Regarding claim 6, Mejjati, modified by Strohmann, discloses the method claimed in claim 1, wherein the machine learning system characterizes a CycleGAN (Mejjati page 4: “If the attention map sa was replaced by all ones, to mark the entire image as relevant, then we obtain CycleGAN”).
Regarding claim 9, it is the corresponding device configured to execute the method claimed in claim 1. Therefore, Mejjati, modified by Strohmann, discloses the limitations of claim 9 as it does the limitations of claim 1.
Regarding claim 10, it is the corresponding non-transitory machine-readable storage medium storing a computer program configured to execute the method claimed in claim 1. Therefore, Mejjati, modified by Strohmann, discloses the limitations of claim 10 as it does the limitations of claim 1.
Claim(s) 5 is rejected under 35 U.S.C. 103 as being unpatentable over Mejjati and Strohmann in view of Hu et al. (NPL "Utilising Visual Attention Cues for Vehicle Detection and Tracking").
Regarding claim 5, Mejjati, modified by Strohmann, discloses the method claimed in claim 4, wherein the object detector is configured to determine objects in images (Mejjati Figure 2, page 4: AS is trained “to focus on the objects or areas that the corresponding discriminator thinks are the most descriptive within its domain (i.e., the horses)”). However, Mejjati fails to disclose the object detector is configured to determine objects in images of traffic scenes. In the related art of generating attention maps, Hu discloses the object detector is configured to determine objects in images of traffic scenes (Hu Figs. 7 and 8, page 5536: “using one VGG16 model as backbone for saliency generation, objectness map generation” where “the generated saliency map does not only give attention to vehicles but also to surrounding objects” and “The objectness map separates foreground and background and thus identifies possible coarse locations for objects”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Mejjati to incorporate the teachings of Hu to incorporate a derived attention map that provides a probabilistic map of the most visually important regions in a video to improve the efficiency and accuracy of object detection and tracking in video for Advanced Driver Assistance Systems (ADAS) (Hu page 5535).
Claim(s) 7 is rejected under 35 U.S.C. 103 as being unpatentable over Mejjati and Strohmann in view of Li et al. (NPL "Generalisable Cardiac Structure Segmentation via Attentional and Stacked Image Adaptation").
Regarding claim 7, Mejjati, modified by Strohmann, discloses a computer-implemented method, the method comprising the following steps: determining an intermediate image using a first generator of a machine learning system (Mejjati Figure 5: Translation results for different input source images) trained by the method claimed in claim 1 (see rejection of claim 1). However, Mejjati fails to disclose the machine learning system is used in training an object detector and the method additionally comprises: providing an input image and an annotation, wherein the annotation characterizes a position of at least one object depicted in the input image; and training the object detector in such a way that for the intermediate image as input, the object detector predicts the object or objects that are characterized by the annotation. In the related art of addressing the problem of domain shift, Li discloses a computer-implemented method for training an object detector (Li page 5: “Dilated Residual U-Net (DRUNet)…used for both brain and cardiac segmentation”), the method comprising the following steps: providing an input image and an annotation (Li page 3: “labelled vendor A and labelled vendor B {(xA, yA), (xB, yB)}”), wherein the annotation characterizes a position of at least one object depicted in the input image (Li page 5: “The CMR scans have been segmented by experienced clinicians, with contours for the left (LV) and right ventricle (RV) blood pools, and the left ventricular myocardium (MYO)”); and training the object detector (Li page 6: “trained the DRUNet”) in such a way that for the intermediate image (Li page 3: “the image style transfer module translated the images from vendor A and vendor B to unlabeled vendor C”) as input, the object detector predicts the object or objects that are characterized by the annotation (Li page 3: “further augmented the training set with synthetic vendor-C-like images and the annotations from vendor A and B”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Mejjati to incorporate the teachings of Li to apply the domain adaptation methods to augment training samples and generalize a cardiac segmentation model to unseen domains for diagnosis of multiple heart diseases (Li pages 1-2).
Claim(s) 8 is rejected under 35 U.S.C. 103 as being unpatentable over Mejjati, Strohmann and Li in view of Tschirhart (US 2018/0334100 A1).
Regarding claim 8, Mejjati, modified by Strohmann and Li, discloses a computer-implemented method, the method comprising the following steps: providing a second input image (Li Table 1, page 5: “25 unannotated scans from a third vendor [C]”); determining, using a trained object detector (Li page 6: “we trained the DRUNet”), objects depicted in the second input image (Li Fig. 4, page 6: “On the unlabeled vendor C, we achieved average Dice score of 86.8% for the left ventricle, 83.4% for the myocardium, and 83.2% for the right ventricle”), wherein the object detector is trained by the method claimed in claim 7 (see rejection of claim 7). However, Mejjati fails to disclose the object detector is used in determining a control signal for controlling an actuator and/or a display device and the method additionally comprises: determining the control signal based on the determined objects; and controlling the actuator and/or the display device according to the control signal. In the related art of controlling an actuator, Tschirhart discloses a computer-implemented method for determining a control signal for controlling an actuator and/or a display device, the method comprising the following steps: determining the control signal based on the determined objects (Tschirhart paragraph 0036: “a field of view associated with the captured images is changed dependent upon the detected location of the object behind the motor vehicle”); and controlling the actuator and/or the display device according to the control signal (Tschirhart paragraph 0036: “processor 226 controlling actuator 228 to move camera 232 so that its field of view 314 is in a more downward direction in order to provide a better viewpoint”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Mejjati to incorporate the teachings of Tschirhart to apply the object detector to facilitate driver awareness of relative vehicle positioning (e.g., while parallel parking, etc.) (Tschirhart paragraph 0007).
Response to Arguments
Applicant’s arguments with respect to the rejection under 35 U.S.C. 101 have been fully considered and are persuasive. The 101 rejection has been withdrawn in view of Applicant’s arguments.
Applicant's arguments with respect to the rejection under 35 U.S.C. 103 have been fully considered but they are not persuasive.
Regarding the argument that “Strohmann does not disclose attention maps at all”, MPEP 2111.01 Plain Meaning sections I, II, and III disclose the words of a claim must be given their “plain meaning” unless such meaning is inconsistent with the specification, it is improper to import claim limitations from the specification, and “plain meaning” refers to the ordinary and customary meaning given to the term by those of ordinary skill in the art, respectively. Under broadest reasonable interpretation, one of ordinary skill art would understand an attention map to be a visualization that shows which parts of an input a model is to be focused on in making a decision or prediction. Strohmann teaches generating pixel-wise weights to highlight microstructural features, such as the interfaces and silicon pixels, during training to accurately perform segmentation. Therefore, the map of pixel-wise weights (shown in Figure 3 of Strohmann) correspond to the claimed attention map.
Regarding the arguments that “Strohmann would have to disclose first and second attention maps” and “there is no basis to suppose that the "loss" mentioned in the blurb from Strohmann characterizes either of the claimed first and second differences”, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986). Mejjati teaches determining a first loss value, wherein the first loss value characterizes a first difference of the source image and of the first reconstruction (Mejjati Equation 3, page 4: Lscyc), and determining a second loss value, wherein the second loss value characterizes a second difference of the target image and of the second reconstruction (Mejjati Equation 4: Ltcyc). Strohmann teaches weighting a loss by an attention map. The combined teachings of Mejjati and Strohmann would have suggested to one of ordinary skill in the art that the first and second losses in Mejjati can each be weighted by an attention map.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHRISTINE ZHAO whose telephone number is (703)756-5986. The examiner can normally be reached Monday - Friday 9:00am - 5:00pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached at (571)270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/C.Z./ Examiner, Art Unit 2677
/ANDREW W BEE/ Supervisory Patent Examiner, Art Unit 2677