DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is responsive to the claims dated 3/14/2026.
Claims 1-30 are presented for examination.
Priority and Benefit
Acknowledgment is made of Applicant's claim to the benefit of provisional application 63/454,570, filed 3/24/2023, and provisional application 63/452,028, filed 3/14/2023.
Information Disclosure Statement
The information disclosure statement filed 8/7/2024 has been received and considered.
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they do not include the following reference sign(s) mentioned in the description: 103 (first occurring at specification [0037] "cloud computing system 103") and 105 (first occurring at specification [0031] "Domain adaptation system 105"). Reference character 105 appears to be a typographical error for domain adaptation system 104, which is shown in FIGS. 1 and 2. Corrected drawing sheets in compliance with 37 CFR 1.121(d), or an amendment to the specification in compliance with 37 CFR 1.121(b) deleting or correcting these reference characters in the description, are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either "Replacement Sheet" or "New Sheet" pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because reference character "514" has been used to designate both the ground-truth binary mask (the bracketed element labeled "Binary Mask GT" in FIG. 6) and the binary mask predicted by refinement neural network 234 (labeled "Binary Mask" in FIGS. 5 and 6, and described as the predicted "correct mask" 514 in [0109]), and because reference character "516" has been used to designate both the source label (labeled "Source Label" in FIG. 5) and the original source logits (FIG. 6; [0110]). Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either "Replacement Sheet" or "New Sheet" pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
The drawings are objected to because
(1) in FIG. 5, the legend "Target Pred (Stundet)" contains a misspelling and should read "Target Pred (Student)";
(2) in FIG. 2, the rectangular box designated by reference character 202 (described as memory 202 in [0036]) is unlabeled and should be provided with a descriptive text label, for example "Memory"; and
(3) in FIG. 2, the box designated by reference character 232 bears only the legend "P2", which is not explained anywhere in the specification, and should be provided with a descriptive text label, for example "Pseudo-Labels" (see pseudo-labels 232, [0058]).
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as "amended." If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either "Replacement Sheet" or "New Sheet" pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The disclosure is objected to because of the following informalities:
(1) in [0010] "a method or unsupervised domain adaptation" should read "a method for unsupervised domain adaptation";
(2) in [0029] the final sentence, ending "generalizes better to the target domain", lacks a terminal period;
(3) in [0032] "in such a way as to computing devices to exchange information" is grammatically incomplete (for example, "in such a way as to allow computing devices to exchange information");
(4) in [0036] "memory 202 for executing a domain adaptation system 104 that hosts and/or implements domain adaptation system 104" is circular, describing domain adaptation system 104 as hosting itself;
(5) in [0040] "retain information after activate/off cycles" appears to contain a typographical error (for example, "power on/off cycles");
(6) [0047] begins with a lowercase letter ("domain adaptation system 104"), and its reference to "training data 106" is inconsistent with the use of reference character 106 elsewhere to designate training data repository 106 ([0031] through [0033]; FIG. 1);
(7) in [0105] reference character 404 is applied to the contrastive loss coefficient, whereas reference character 404 designates the contrastive loss itself elsewhere ([0087] and [0107]; FIG. 4);
(8) in [0110] reference character 610 is described both as "original target pseudo-label 610" and as "original target logits 610", and consistent terminology is required;
(9) in [0112] "In mode operation 700" should read "In mode of operation 700", and "domain adaptation system 104 component of the domain adaptation system 104 may assign" is garbled;
(10) in [0113] the description of FIG. 8 refers to "mode of operation 700", whereas FIG. 8 depicts mode of operation 800 (see [0114]); and
(11) in [0114] (page 28), "In mode operation 800" should read "In mode of operation 800".
Appropriate correction is required.
Claim Objections
Claims 9 and 11 are objected to because of the following informalities:
In claim 9 the recited "the source model" and "the target model" should read "the source machine learning model" and "the target machine learning model", respectively, for consistency with the terminology of claim 8, from which claim 9 depends (compare claims 2, 16 and 24).
In claim 11 the recited "The method of claim 10" should read "The system of claim 10". Claim 10 is directed to a computing system, and the body of claim 11 further limits the machine learning system recited in claims 8 and 10, so the preamble's reference to a method is inconsistent with the claim from which claim 11 depends. For purposes of examination, claim 11 is treated as a system claim depending from claim 10.
Appropriate correction is required.
Claim Rejections - 35 U.S.C. 112(b)
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 12-13, 20-21, and 28-29 are rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 12, the claim recites the machine learning system configured to train the target machine learning model is further configured to adapt the machine learning model to new domain data without access to original training data. Claim 8, from which claim 12 depends, recites two different machine learning models: a source machine learning model, which is used to generate the plurality of pseudo-labels, and a target machine learning model, which is trained. The later recitation further configured to adapt the machine learning model in claim 12 does not identify which of the two models is intended. A claim is indefinite when it contains words or phrases whose meaning is unclear, and where two different elements are recited earlier in the claims, a later reference to one of them with a definite article is unclear where it is uncertain which of the two was intended.
Both readings are reasonable. On one reading, the model adapted is the target machine learning model, because claim 12 ties the adapting to the machine learning system configured to train the target machine learning model, and parallel method claim 5 recites adapting the target machine learning model. On the other reading, the model adapted is the source machine learning model. The phrase without access to original training data most naturally describes the data on which the adapted model was originally trained, and claim 8 identifies no training data for the target machine learning model other than the selected pseudo-labels and the corresponding unlabeled data. The specification likewise states that the goal of the domain adaptation system may be to adapt the source model to perform well on the target domain without accessing the source data during adaptation (specification [0085]), identifies the teacher model as the source model (specification [0086]), and describes updating the teacher model weights during each training iteration with an exponential moving average (specification [0058], [0095]), while it also describes the student model as the model being adapted to the target domain (specification [0086], [0089]).
The uncertainty is material to the scope of the claim. If the machine learning model is the target machine learning model, claim 12 reaches a system that adapts only the target machine learning model. If it is the source machine learning model, claim 12 requires adaptation of the source machine learning model and does not reach a system that leaves the source machine learning model unchanged while training a separate target machine learning model. Accordingly, the broadest reasonable interpretation consistent with the specification does not make the metes and bounds of claim 12 clear: a person of ordinary skill in the relevant art would read the claim with more than one reasonable interpretation, and claim 12 is therefore indefinite.
For purposes of examination, the machine learning model in claim 12 is interpreted as either the source machine learning model or the target machine learning model, so that the limitation is met by adapting either model to new domain data without access to the data on which the source machine learning model was originally trained. This interpretation is consistent with the specification, which describes adapting both the source (teacher) model and the target (student) model without access to the source data, which it also calls the original training data (specification [0028], [0085], [0086], [0089], [0095]).
Claim 13 depends from claim 12; it incorporates and does not cure this defect, and is rejected for the same reason.
Regarding claims 20 and 28, claim 20 recites applying a Fourier transform to each of the plurality of source image logits and target image logits to separate a phase component and an amplitude component, and claim 28 recites apply a Fourier transform to each of the plurality of source image logits and target image logits to separate a phase component and an amplitude component. Claim 20 recites the limitation the plurality of source image logits and target image logits in lines 2-3 of the claim, and claim 28 recites the same limitation in lines 3-4 of the claim. There is insufficient antecedent basis for this limitation in the claims.
The lack of antecedent basis here leaves the scope of the claims uncertain. The only logits recited earlier in the chain of claim 20 are the image logits from the teacher decoder recited in claim 18, and the chain of claim 28 likewise recites only the image logits from the teacher decoder in claim 26. Those logits are inputted into the refinement neural network and carry no source or target qualification. The definite article indicates that the source image logits and target image logits refer back to logits already recited, but it cannot be determined whether the Fourier transform must be applied to the image logits from the teacher decoder or to a different set of logits. A claim that refers with a definite article to a qualified element when only an unqualified element was recited earlier is indefinite, because it is uncertain which element is intended.
The uncertainty is material. The specification describes the teacher model generating pseudo-labels for the unlabeled target domain data, with the image logits 506 from the teacher decoder 508 inputted into the refinement neural network 234 (specification [0109]; FIG. 5). The source logits that are Fourier transformed in training the refinement neural network are instead the original source logits 516, which FIG. 5 labels as the source label, and whose amplitude component is mixed with the amplitude of a target image so that training supervision may be provided by source labels (specification [0110]; FIGS. 5 and 6). If the claimed source image logits and target image logits are the image logits from the teacher decoder, claims 20 and 28 require the teacher decoder to produce the source image logits and do not reach a refinement neural network trained on source label maps; if they are different logits, claims 20 and 28 reach logits of any origin. Accordingly, the broadest reasonable interpretation consistent with the specification does not make the metes and bounds of claims 20 and 28 clear: a person of ordinary skill in the relevant art would read each of those claims with more than one reasonable interpretation, and claims 20 and 28 are therefore indefinite.
For purposes of examination, the plurality of source image logits and target image logits in claims 20 and 28 is interpreted as a plurality of logits for source domain images and a plurality of logits for target domain images, that is, pixel-wise class scores taken before the final activation function, or non-binary label maps, associated with each image, whether or not they are produced by the teacher decoder recited in claims 18 and 26. This interpretation is consistent with the specification, which describes logits as the values before the activation function is applied in the final layer of a segmentation model and describes the original source logits 516 as a non-binary mask likely containing pixel-wise probabilities or confidence scores for different classes (specification [0059], [0110]; FIG. 6).
Claim 21 depends from claim 20 and claim 29 depends from claim 28; they incorporate and do not cure this defect, and are rejected for the same reason.
Claim Rejections - 35 U.S.C. 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraph(s) of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office Action:
A person shall be entitled to a patent unless - (a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1, 3-5, 8 and 10-12 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Ding et al. (hereinafter Ding) "MAPS: A Noise-Robust Progressive Learning Approach for Source-Free Domain Adaptive Keypoint Detection" 2023.
Regarding independent claim 1, Ding discloses a method for unsupervised domain adaptation, the method comprising (Ding: page 1, 1. Introduction, "Unsupervised domain adaptation (UDA), a paradigm that aims to transfer knowledge from a label-rich source domain to another unlabeled target domain, is proposed to reduce the burden of manual annotations"; Ding adapts a keypoint detector trained on a labeled source domain to an unlabeled target domain): generating a plurality of pseudo-labels for a dataset of unlabeled data using a source machine learning model (Ding: page 1, 1. Introduction, "Therefore, we resort to pseudo-labeling [22], a popular semi-supervised learning technique that generates the pseudo labels for unlabeled data"; pseudo labels (a plurality of pseudo-labels) are generated for the unlabeled target data (a dataset of unlabeled data) instead of being supplied by annotation, page 5, Algorithm 1, "Input: Target dataset Xt; a well-trained source model fs"; the well-trained source model (a source machine learning model) named in that input line, fs, is the one model the adaptation is given, page 5, Algorithm 1, "Initialize the student ft and teacher f′t with fs"; the first step of Algorithm 1 loads the parameters of that well-trained source model into each member of the pair, so the network that predicts on the target data in the first round carries the parameters of the source model itself, page 4, Progressive Selection, "which is the 2D coordinate of the maximum activation of the student output"; the pseudo label of each target sample is the position of the maximum activation in the output of that network for that sample, page 5, Algorithm 1, "3: Fix ft, update v by Eq. (11); // sample selection 4: Update ft by Eq. (12); // model updating"; Algorithm 1 selects on those pseudo labels before it makes any update to the parameters, so in the first round nothing has yet displaced the parameters the well-trained source model supplied, page 3, Fig. 1, "Given the well-trained source model …, MAPS progressively selects easy samples from target samples"; Fig. 1 likewise shows that well-trained source model as the first of the successive parameter states the target training produces and the selection of the easy samples as what is performed at each state, so the pseudo labels the first round acts on are generated using a source machine learning model); estimating a reliability of each pseudo-label of the plurality of pseudo-labels using one or more reliability measures (Ding: page 4, Progressive Selection, "The self-paced loss for each sample … is defined by the mean squared error, i.e., pseudo-label regression loss"; a quantity is computed for every target sample from its own pseudo label, page 4, Progressive Selection, "The model parameters are fixed during selection, therefore, … can be seen as a difficulty measurer, and the age parameter … is a threshold"; that quantity is computed with the model held fixed and serves as a difficulty measurer (one or more reliability measures), so it measures how far the model already is from each pseudo label, page 4, Progressive Selection, "We introduce the self-paced learning regime [24] to progressively select samples with high-confident pseudo labels"; Ding treats the pseudo labels the measure ranks as easy as the high-confident ones, so the measure is an estimate of the reliability of each pseudo label); selecting a subset of the plurality of pseudo-labels having estimated reliabilities that satisfy a reliability threshold (Ding: page 4, Progressive Selection, "This regime progressively selects the easy samples by assigning the weight 1 for them"; the easy samples (a subset of the plurality of pseudo-labels) are the ones a weight of one picks out of the whole set, page 5, "where S(m, L) denotes the m-th smallest value of loss function L"; the age parameter (a reliability threshold) is set at the m-th smallest difficulty score of the round, so a pseudo label is selected exactly when its estimated reliability satisfies that threshold); and training, using one or more curriculum learning techniques, a target machine learning model starting with the selected subset of the plurality of pseudo-labels and the corresponding unlabeled data (Ding: page 3, left column, "Self-paced learning (SPL) [24], [61] is a typical example of automatic curriculum learning"; Ding identifies the self-paced learning regime it applies (one or more curriculum learning techniques) as a curriculum learning technique, page 3, left column, "The easiest samples with the lowest training loss are used to train the models, this portion of the easiest samples gradually increases according to a specified scheduler"; training begins with the selected easiest pseudo labels and the target samples (the corresponding unlabeled data) they belong to, and the selected portion then grows, page 5, Algorithm 1, "Output: New target model ft and f′t"; the two models so trained are the ones Algorithm 1 returns and designates new target models, and the new target model (a target machine learning model) the training updates is one of the two).
Regarding dependent claim 3, Ding discloses the method of claim 1, wherein estimating a reliability comprises: assigning a score to each pseudo-label of the plurality of pseudo-labels indicating reliability of the pseudo-label (Ding: page 4, Progressive Selection, "The self-paced loss for each sample … is defined by the mean squared error, i.e., pseudo-label regression loss"; a pseudo-label regression loss (a score) is computed individually for every sample against that sample's own pseudo label, page 4, Progressive Selection, "can be seen as a difficulty measurer, and the age parameter … is a threshold"; the score so assigned is what ranks the pseudo label from easy to hard and is compared against the threshold, so it indicates the reliability of that pseudo label, page 4, Progressive Selection, "We introduce the self-paced learning regime [24] to progressively select samples with high-confident pseudo labels, and the pseudo-label regression loss is imposed on them"; the pseudo labels Ding treats as the high-confident ones are the ones that score lowest on that same regression loss, so the score assigned to each pseudo label is Ding's own measure of how far that pseudo label can be trusted).
Regarding dependent claim 4, Ding discloses the method of claim 3, wherein selecting a subset comprises: selectively grouping the plurality of pseudo-labels into two or more groups based on the assigned score (Ding: page 3, Fig. 1, "assigns weight, 1 for easy samples and 0 for the remaining samples, the weights are notated as symbol v"; the pseudo labels are divided into the easy samples and the remaining samples (two or more groups) by the weight each one receives, page 4, Progressive Selection, "This regime progressively selects the easy samples by assigning the weight 1 for them"; the weight a sample receives is fixed by comparing its assigned difficulty score against the age parameter, so the grouping is made on the assigned score).
Regarding dependent claim 5, Ding discloses the method of claim 1, wherein training the target machine learning model comprises adapting the target machine learning model to new domain data without access to original training data (Ding: page 1, Abstract, "only the well-trained source model is provided to the target domain"; the target domain (new domain data) and the source model are all that the adaptation is given; page 1, right column, "no longer requires access to the source data"; the labeled source dataset on which that model was trained (original training data) is not accessed while the target model is adapted).
Regarding independent claim 8, this is a computer system claim that is substantially the same as the method of claim 1. Thus, claim 8 is rejected for the same reason as claim 1. In addition, Ding discloses a computing system for unsupervised domain adaptation comprising (Ding: page 1, Abstract, "only the well-trained source model is provided to the target domain"; Ding adapts a model to an unlabeled target domain using nothing but the model trained on the labeled source domain, which is the unsupervised domain adaptation the preamble recites): processing circuitry in communication with storage media, the processing circuitry configured to execute a machine learning system configured to (Ding: page 5, "via PyTorch and report the average accuracies"; MAPS (a machine learning system) is run as a PyTorch program, and such a program cannot be run at all except by processing circuitry that fetches its instructions and its network weights from storage media in communication with that circuitry, so the recited platform necessarily and not merely probably attends the implementation Ding reports).
Regarding dependent claim 10, the rejection of claim 3 is applied in the same manner to the corresponding system form recitation of claim 10.
Regarding dependent claim 11, the rejection of claim 4 is applied in the same manner to the corresponding system (as interpreted per the Claim Objections set forth above) form recitation of claim 11.
Regarding dependent claim 12, Ding discloses the system of claim 8, wherein the machine learning system configured to train the target machine learning model is further configured to adapt the source machine learning model or the target machine learning model (as interpreted per the 35 U.S.C. 112(b) rejections set forth above) to new domain data without access to original training data (Ding: page 1, Abstract "only the well-trained source model is provided to the target domain"; the target domain (new domain data) and that model are all the adaptation is given, page 1, right column, "no longer requires access to the source data"; the labeled source dataset (original training data) is not accessed while the target model is adapted).
Claims 15 and 23 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Corbiere et al. (hereinafter Corbiere) "Confidence Estimation via Auxiliary Models" 2020.
Regarding independent claim 15, Corbiere discloses a method for unsupervised domain adaptation, the method comprising (Corbiere: page 6, 5 Application to Self-Training in Semantic Segmentation with Domain Adaptation, "Unsupervised domain adaptation for semantic segmentation aims to adapt a segmentation model trained on a labeled source domain to a target domain devoid of annotation"; the ConDA framework Corbiere presents carries out that adaptation, so the field and purpose the preamble recites are met and the recited acts are the steps by which it proceeds): generating a plurality of pseudo-labels for a dataset of unlabeled data using a source machine learning model (Corbiere: page 6, 5 Application to Self-Training in Semantic Segmentation with Domain Adaptation, "This network is learned with full supervision from the source-domain samples in Ds, using a cross-entropy loss, while leveraging a set Dt of unlabelled target-domain examples"; the only labeled supervision the segmentation network F (a source machine learning model) receives is that of the first-domain samples, the unlabelled second-domain examples it also draws on supplying no labels of their own, page 6, Fig. 3, "This network F is fixed during this phase and its weights are not updated"; that network's weights are held fixed while it supplies the target predictions, so those predictions are its own, page 6, 5.1 Self-training for unsupervised domain adaptation, "2) Collect pseudo-labels among the predictions that this network makes on the target-domain training images"; the predictions that network makes on the unlabelled target-domain training images (a dataset of unlabeled data) are collected as the target pseudo-labels (a plurality of pseudo-labels)); refining the plurality of pseudo-labels to reduce noise in the plurality of pseudo-labels (Corbiere: page 7, left column, "If too many of them correspond to erroneous predictions of the current segmentation network, the performance of the whole UDA can deteriorate"; Corbiere identifies the erroneous predictions among the collected labels as the noise those labels carry, page 7, left column, "Thus, a measure of confidence should be used in order to only gather reliable predictions as pseudo-labels and to reject the others"; a confidence measure is then applied to the collected labels for the purpose of keeping the reliable label content and rejecting the rest, page 7, 5.2 Selecting pseudo-labels with a confidence model, "we label it with its predicted class … otherwise it is masked out"; every pixel whose predicted confidence clears the chosen threshold keeps its predicted class and every pixel that does not is masked out, which lowers the proportion of incorrect content the labels carry and so reduces the noise in them); and training, using the refined pseudo-labels, a target machine learning model (Corbiere: page 7, 5.2 Selecting pseudo-labels with a confidence model, "these incomplete segmentation maps, constitute target pseudo-labels that are used to train a new semantic-segmentation network"; the incomplete segmentation maps (the refined pseudo-labels) that survive the refinement are the labels on which the new semantic-segmentation network (a target machine learning model) is trained, page 6, 5.1 Self-training for unsupervised domain adaptation, "1) Train a new semantic-segmentation network from scratch using the chosen UDA technique"; the model so trained is a network separate from the fixed network that produced the labels).
Regarding independent claim 23, this is a computer system claim that is substantially the same as the method of claim 15. Thus, claim 23 is rejected for the same reason as claim 15. In addition, Ding discloses Corbiere discloses a computing system for unsupervised domain adaptation comprising (Corbiere: page 6, 5 Application to Self-Training in Semantic Segmentation with Domain Adaptation, "Unsupervised domain adaptation for semantic segmentation aims to adapt a segmentation model trained on a labeled source domain to a target domain devoid of annotation"; the ConDA framework Corbiere presents carries out that adaptation, which is the unsupervised domain adaptation the preamble recites): processing circuitry in communication with storage media, the processing circuitry configured to execute a machine learning system configured to (Corbiere: page 10, 6.2.1 Experimental setup, "All the implementations are done with the PyTorch framework"; ConDA (a machine learning system) is implemented in the PyTorch framework, page 10, 6.2.2 Comparison with state of the art, "ConDA outperforms all those methods by reaching 49.9% mIoU"; that implementation was run and its results measured, and a program so implemented cannot be executed at all except by processing circuitry that reads its instructions and its network weights from storage media in communication with that circuitry, so the recited platform necessarily attends the implementation).
Claim Rejections - 35 U.S.C. 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office Action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2 and 9 are rejected under 35 U.S.C. 103 as being unpatentable over Ding, as applied in the rejection of claims 1 and 8, respectively above, in view of Kim et al. (hereinafter Kim) "Domain Adaptation without Source Data" 2021.
Regarding dependent claim 2, Ding teaches the method of claim 1.
Ding does not expressly teach wherein the source machine learning model comprises a teacher model and the target machine learning model comprises a student model.
However, Kim teaches wherein the source machine learning model comprises a teacher model and the target machine learning model comprises a student model (Kim: page 3, 2.2 SFDA framework, "Our SFDA framework contains two types of models: a pre-trained source model and a trainable target model"; the framework is built of two networks, a pre-trained source model (a teacher model) and a trainable target model (a student model), page 3, 2.2 SFDA framework, "all parameters of these modules are fixed after being pre-trained from the source domain"; the pre-trained source model is held frozen for the whole of the adaptation, so it only supplies labels and never learns, page 2, "The first loss uses source-oriented pseudo labels of all target samples from the pre-trained source model; this prevents the target model from a self-biasing problem caused by the second self-learning loss"; the labels that frozen network produces for the target samples enter a loss of their own, and that loss is added for the purpose of keeping the other network from biasing itself on the labels it produces for itself, page 3, 2.2 SFDA framework, "Overall, we adapt the target model to the target domain using pseudo labels … in a self-learning manner, while using source knowledge … from the pre-trained source model as a regularizer"; the network being trained goes on learning from the labels it generates for itself, and the frozen network's labels are added alongside them as a regularizer rather than put in their place, page 3, 2.2 SFDA framework, "the parameters of each module are initialized by the parameter of pre-trained source model"; the trainable target model begins as a copy of that frozen network and thereafter departs from it, page 2, "One is a pre-trained model from the source domain where all the weights are frozen, and the other is a target model that is initialized from the pre-trained source model but evolves progressively by optimizing two losses"; the second of the two networks is the only one the training changes, and it is changed by optimizing both losses together, which is the teacher-and-student split the limitation recites).
Because Ding and Kim are analogous art in the same field of endeavor, both addressing the adaptation of a model trained on a labeled source domain to an unlabeled target domain when the source data can no longer be accessed, accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Kim's two-model arrangement to the method of Ding, with a reasonable expectation of success, by keeping the well-trained source model Ding is given frozen and running it on each target sample as a further, source-oriented source of pseudo labels on which the model under training is also trained, while retaining the pseudo labels Ding's own model generates and the loss taken on them, so that the source machine learning model is the teacher that supplies labels it never learns from and the model the rounds update is the student, to teach wherein the source machine learning model comprises a teacher model and the target machine learning model comprises a student model. This modification would have been motivated by the desire to prevent the model being trained from the self-biasing problem that training on its own labels alone causes (Kim: page 2).
Regarding dependent claim 9 (as interpreted per the Claim Objections set forth above), the rejection of claim 2 is applied in the same manner to the corresponding system form recitation of claim 9.
Claims 6 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Ding, as applied in the rejection of claims 5 and 12, respectively above, in view of Kothandaraman et al. (hereinafter Kothandaraman) "SS-SFDA: Self-Supervised Source-Free Domain Adaptation for Road Segmentation in Hazardous Environments" 2021.
Regarding dependent claim 6, Ding teaches the method of claim 5.
Ding does not expressly teach wherein the new domain data comprises unlabeled nighttime image data.
However, Kothandaraman teaches wherein the new domain data comprises unlabeled nighttime image data (Kothandaraman: page 1, right column, "the pre-trained model is directly used to adapt to the unlabeled target domain"; the second-domain images on which Kothandaraman adapts the pre-trained model carry no labels, page 8, "Table 7: Results on the night driving dataset, Dark Zurich"; the second-domain data of that adaptation is a night driving image dataset; page 8, left column, "SS-SFDA first trains on the twilight images, and then on the night images"; the unlabeled second-domain images the self-training runs on are night images, which is the unlabeled nighttime image data recited).
Because Ding and Kothandaraman are analogous art with both addressing the adaptation of a trained model to an unlabeled target domain by generating pseudo-labels for that domain and training on the more reliable ones first, accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Kothandaraman's night driving target data to the method of Ding, with a reasonable expectation of success, by taking the unlabeled target data on which Ding's progressive selection operates to be night images, to teach wherein the new domain data comprises unlabeled nighttime image data. This modification would have been motivated by the desire to recover the segmentation accuracy a model trained on clear daytime data loses at night without collecting any new annotations (Kothandaraman: page 1).
Regarding dependent claim 13, the rejection of claim 6 is applied in the same manner to the corresponding system form recitation of claim 13.
Claims 7 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Ding in view of Ge et al. (hereinafter Ge) "Self-paced Contrastive Learning with Hybrid Memory for Domain Adaptive Object Re-ID" 2020.
Regarding dependent claim 7, Ding teaches the method of claim 1.
Ding does not expressly teach wherein the one or more curriculum learning techniques include unsupervised contrastive learning (CL).
However, Ge teaches wherein the one or more curriculum learning techniques include unsupervised contrastive learning (CL) (Ge: page 1, Abstract, "we propose a novel self-paced contrastive learning framework with hybrid memory"; the training framework is at once self-paced, that is ordered from the more reliable material to the less reliable, and contrastive, page 1, Abstract, "the proposed self-paced method gradually creates more reliable clusters to refine the hybrid memory and learning targets"; the self-paced schedule starts from the most reliable pseudo-labelled clusters and admits further material as it becomes reliable, which is the easier-first ordering a curriculum requires, page 4, 3.1 Constructing and Updating Hybrid Memory for Contrastive Learning, "Given the target-domain training samples … without any ground-truth label, we employ the self-paced clustering strategy"; the material that schedule orders carries no ground-truth label, so the learning it governs is unsupervised, page 4, 3.1 Constructing and Updating Hybrid Memory for Contrastive Learning, "we design a novel contrastive loss to fully exploit available data by treating all the source-domain classes, target-domain clusters and target-domain un-clustered instances as independent classes"; the loss that schedule governs is a contrastive loss, so the self-paced schedule includes contrastive learning carried out without labels).
Because Ding and Ge are analogous art with both addressing the ordering of noisy pseudo-labelled training material from more reliable to less reliable while a model is adapted to an unlabeled target domain, accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Ge's contrastive loss to the method of Ding, with a reasonable expectation of success, by adding that loss to the objective Ding optimizes on the progressively selected target samples so that representation learning proceeds on the same easier-first schedule, to teach wherein the one or more curriculum learning techniques include unsupervised contrastive learning (CL). This modification would have been motivated by the desire to draw training signal from the target samples that carry no reliable pseudo-label instead of discarding them (Ge: page 4).
Regarding dependent claim 14, the rejection of claim 7 is applied in the same manner to the corresponding system form recitation of claim 14.
Claims 16-17 and 24-25 are rejected under 35 U.S.C. 103 as being unpatentable over Corbiere, as applied in the rejections of claims in view of Kwon et al. (hereinafter Kwon) "Semi-supervised Semantic Segmentation with Error Localization Network" 2022.
Regarding dependent claim 16, Corbiere teaches the method of claim 15.
Corbiere does not expressly teach wherein the source machine learning model comprises a teacher model and the target machine learning model comprises a student model.
However, Kwon teaches wherein the source machine learning model comprises a teacher model and the target machine learning model comprises a student model (Kwon: page 2, Figure 1, "It employs two segmentation networks, the student (s), which will be our final model, and the teacher (t) used for generating pseudo labels"; the network that produces the pseudo labels is the teacher network and the network trained on them is the student network, which becomes the final model, page 2, Figure 1, "Then the teacher is updated by an exponential moving average (EMA) of the student"; the two are maintained as a teacher and student pair throughout training, the teacher's weights tracking the student's).
Because Corbiere and Kwon are analogous art with both addressing the training of a semantic segmentation network on pseudo labels whose erroneous parts must be located and withheld, accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Kwon's semi-supervised training scheme, in which a teacher and student pair is maintained and an error localization network marks the parts of the teacher's pseudo labels that are likely to be wrong, to the method of Corbiere, with a reasonable expectation of success, by maintaining the network that supplies the target predictions as a teacher whose weights track the network being trained rather than as a network frozen once and for all, and by withholding the marked parts of its pseudo labels from the loss that trains that network, to teach wherein the source machine learning model comprises a teacher model and the target machine learning model comprises a student model. This modification would have been motivated by the desire to obtain more stable and effective training of the network being trained on those pseudo labels (Kwon: page 3).
Regarding dependent claim 17, Corbiere, in view of Kwon, teach the method of claim 16, wherein refining the plurality of pseudo-labels comprises refining, using a refinement neural network, one or more of the plurality of pseudo-labels to improve accuracy of the one or more of the plurality of pseudo-labels (Corbiere: page 7, 5.2 Selecting pseudo-labels with a confidence model, "a confidence network C is learned at step (2) to predict the confidence of the UDA-trained semantic segmentation network F and used to select only trustworthy pseudo-labels on target-domain images"; the confidence network C (a refinement neural network) is a learned network, separate from the segmentation network, whose output decides which of the collected labels are kept, page 6, Fig. 3, "the trainable head of the confidence network C, which includes a multi-scale ConfidNet module"; that network is a trainable neural network rather than a fixed rule, page 7, left column, "Thus, a measure of confidence should be used in order to only gather reliable predictions as pseudo-labels and to reject the others"; keeping the reliable predictions and rejecting the others raises the proportion of the retained label content that is correct, which improves the accuracy of those pseudo-labels).
Regarding dependent claim 24, the rejection of claim 16 is applied in the same manner to the corresponding system form recitation of claim 24.
Regarding dependent claim 25, the rejection of claim 17 is applied in the same manner to the corresponding system form recitation of claim 25.
Claims 18-19, 22, 26-27 and 30 are rejected under 35 U.S.C. 103 as being unpatentable over Corbiere in view of Kwon, as applied in the rejections of claims 17 and 25, and further in view of Kirillov et al. (hereinafter Kirillov) "PointRend: Image Segmentation as Rendering" 2020.
Regarding dependent claim 18, Corbiere, in view of Kwon, teach the method of claim 17, wherein the teacher model comprises a teacher encoder and a teacher decoder (Kwon: page 5, 4.1. Network Architecture, "The proposed model mainly consists of two types of networks, the main segmentation network and the ELN. Each network is formed with an encoder and decoder"; the segmentation networks Kwon maintains are each formed with an encoder and a decoder, page 2, Figure 1, "the teacher (t) used for generating pseudo labels"; the network so formed that supplies the pseudo labels is the teacher, so that teacher comprises an encoder and a decoder).
Corbiere and Kwon do not expressly teach wherein one or more features from the teacher encoder and image logits from the teacher decoder are inputted into the refinement neural network.
However, Kirillov teaches wherein one or more features from the teacher encoder and image logits from the teacher decoder are inputted into the refinement neural network (Kirillov: page 3, Figure 3, "The MLP uses interpolated features computed at these points"; the refining network Kirillov applies is a small multi-layer perceptron fed point-wise inputs, page 3, Figure 3, "from (1) a fine-grained feature map of the backbone CNN and (2) from the coarse prediction mask"; that network receives two inputs, a feature map taken from the backbone CNN (a teacher encoder) and the coarse prediction, page 5, Coarse prediction features, "the second feature type is a coarse segmentation prediction from the network"; the second input is the network's own dense class scores (image logits) rather than a further feature map, page 5, Coarse prediction features, "a K-dimensional vector at each point in the region (box) representing a K-class prediction"; that input carries one value for each of the K classes at every point, so what is supplied is a score for each class and not a feature, page 5, Coarse prediction features, "The coarse resolution, by design, provides more globalized context, while the channels convey the semantic classes"; the channels of that input are the semantic classes themselves, so the values supplied are the class scores the network computes for the image, page 8, Implementation details, "Coarse prediction features come from the (already coarse) output of the semantic segmentation model"; in the semantic segmentation setting those dense scores are taken from the output of the semantic segmentation model (a teacher decoder), page 8, Figure 9, "PointRend refines prediction scores for areas where a coarser prediction is not sufficient"; what that input carries are the prediction scores of the coarse prediction, page 8, Figure 9, "To visualize the scores at each step we take arg max at given resolution without bilinear interpolation"; the class assignment is made from those scores by taking the largest of them, so what is supplied to the refining network is the decoder's score for each class at each point taken before the class assignment is made on it, and an encoder feature map is supplied alongside it).
Because Corbiere, in view of Kwon, and Kirillov are analogous art with all three addressing the correction of a semantic segmentation network's own dense predictions by an auxiliary trained network, accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Kirillov's two-part point-wise input to the method of Corbiere, in view of Kwon, with a reasonable expectation of success, by feeding the auxiliary network the teacher decoder's dense class scores alongside the encoder feature map it already receives, to teach wherein one or more features from the teacher encoder and image logits from the teacher decoder are inputted into the refinement neural network. This modification would have been motivated by the desire to give that network a feature source carrying more contextual and semantic information than the encoder feature map alone supplies (Kirillov: page 5).
Regarding dependent claim 19, Corbiere, in view of Kwon and Kirillov, teach the method of claim 18, further comprising: training the refinement neural network (Corbiere: page 7, 5.2 Selecting pseudo-labels with a confidence model, "To train the segmentation confidence network C, we propose to jointly optimize two objectives"; the confidence network is itself trained (the refinement neural network) rather than merely applied, page 7, 5.3 Confidence training with adversarial loss, "the first one supervises the confidence prediction on annotated source-domain examples using the known true class probabilities for the predictions from F"; that training is supervised by the labels of the annotated first-domain examples).
Regarding dependent claim 22, Corbiere, in view of Kwon and Kirillov, teach the method of claim 19, wherein training the refinement neural network comprises: using a predicted mask as a weighting factor in training the target machine learning model (Kwon: page 4, 3.1. Learning ELN Using Labeled Data, "Given an image and its segmentation prediction as input, ELN is trained to localize errors on the prediction through supervised learning"; the auxiliary network is trained to output a binary mask (a predicted mask) marking where the prediction is wrong, page 4, 3.2. Semi-supervised Learning with ELN, "but is applied only to valid pixels identified by ELN"; the loss that trains the student network is applied only where that map marks the pseudo label valid, page 5, left column, "Through the rounded binary mask, the main segmentation network can be trained on valid pixels only"; the map therefore scales the contribution each part of the pseudo label makes to the training of the student network (as a weighting factor in training the targe machine learning model), the weight being one where the map marks the label correct and zero where it does not, page 4, right column, "The total loss minimized in the first stage for labeled data is as follows"; that total is stated as the sum of the supervised segmentation loss, the auxiliary decoder loss and the loss that trains the auxiliary network itself, page 5, left column, "Note that labeled data are also involved in training through Llabeled"; that labeled objective remains in the training while the map is gating the loss on the unlabeled images, so the gating is carried out within the training of the auxiliary network (wherein training the refinement neural network) and not after that training has ended, page 5, "When training is completed, only the student network is used at inference since the others, including ELN, are all auxiliary modules that support semi-supervised learning of the student"; the auxiliary network is released only once the training is over).
Regarding dependent claim 26, the rejection of claim 18 is applied in the same manner to the corresponding system form recitation of claim 26.
Regarding dependent claim 27, the rejection of claim 19 is applied in the same manner to the corresponding system form recitation of claim 27.
Regarding dependent claim 30, the rejection of claim 22 is applied in the same manner to the corresponding system form recitation of claim 30.
Allowable Subject Matter
Claims 20-21 and 28-29 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims and overcoming the 35 U.S.C. 112(b) rejections set forth in this Office action.
The following is a statement of reasons for the indication of allowable subject matter:
The closest prior arts found when taken individually or in combination do not expressly teach or render obvious the limitations recited in dependent claims 20-21 and 28-29 when taken in the context of the claims as a whole.
At best the closest prior arts uncovered, specifically, Yang "FDA: Fourier Domain Adaptation for Semantic Segmentation" discloses: We describe a simple method for unsupervised domain adaptation, whereby the discrepancy between the source and target distributions is reduced by swapping the low-frequency spectrum of one with the other. We illustrate the method in semantic segmentation, where densely annotated images are aplenty in one domain (e.g., synthetic data), but difficult to obtain in another (e.g., real images). Current state-of-the-art methods are complex, some requiring adversarial optimization to render the backbone of a neural network invariant to the discrete domain selection variable. Our method does not require any training to perform the domain alignment, just a simple Fourier Transform and its inverse. Despite its simplicity, it achieves state-of-the-art performance in the current benchmarks, when integrated into a relatively standard semantic segmentation model. Our results indicate that even simple procedures can discount nuisance variability in the data that more sophisticated methods struggle to learn away (Abstract); Kundu "Amplitude Spectrum Transformation for Open Compound Domain Adaptive Semantic Segmentation" discloses: Open compound domain adaptation (OCDA) has emerged as a practical adaptation setting which considers a single labeled source domain against a compound of multi-modal unlabeled target data in order to generalize better on novel unseen domains. We hypothesize that an improved disentanglement of domain-related and task-related factors of dense intermediate layer features can greatly aid OCDA. Prior-arts attempt this indirectly by employing adversarial domain discriminators on the spatial CNN output. However, we find that latent features derived from the Fourier-based amplitude spectrum of deep CNN features hold a more tractable mapping with domain discrimination. Motivated by this, we propose a novel feature space Amplitude Spectrum Transformation (AST). During adaptation, we employ the AST auto-encoder for two purposes. First, carefully mined source-target instance pairs undergo a simulation of cross-domain feature stylization (AST-Sim) at a particular layer by altering the AST-latent. Second, AST operating at a later layer is tasked to normalize (AST-Norm) the domain content by fixing its latent to a mean prototype. Our simplified adaptation technique is not only clustering-free but also free from complex adversarial alignment. We achieve leading performance against the prior arts on the OCDA scene segmentation benchmarks (Abstract); and Kwon "Semi-supervised Semantic Segmentation with Error Localization Network" discloses: This paper studies semi-supervised learning of semantic segmentation, which assumes that only a small portion of training images are labeled and the others remain unlabeled. The unlabeled images are usually assigned pseudo labels to be used in training, which however often causes the risk of performance degradation due to the confirmation bias towards errors on the pseudo labels. We present a novel method that resolves this chronic issue of pseudo labeling. At the heart of our method lies error localization network (ELN), an auxiliary module that takes an image and its segmentation prediction as input and identifies pixels whose pseudo labels are likely to be wrong. ELN enables semi-supervised learning to be robust against inaccurate pseudo labels by disregarding label noises during training and can be naturally integrated with self-training and contrastive learning. Moreover, we introduce a new learning strategy for ELN that simulates plausible and diverse segmentation errors during training of ELN to enhance its generalization. Our method is evaluated on PASCAL VOC 2012 and Cityscapes, where it outperforms all existing methods in every evaluation setting (Abstract).
In addition, references uncovered have not provided a basis of evidence for asserting a motivation for one of ordinary skill in the art, before the effective filing date of the claimed invention and knowing the teachings of the prior art of record, to combine them to arrive at the present invention in the context of claims 20-21 and 28-29 as a whole.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Feng et al. “Semi-FedSER: Semi-supervised Learning for Speech Emotion Recognition On Federated Learning using Multiview Pseudo-Labeling” 2022 (Abstract Speech Emotion Recognition (SER) application is frequently associated with privacy concerns as it often acquires and transmits speech data at the client-side to remote cloud platforms for further processing. These speech data can reveal not only speech content and affective information but the speaker’s identity, demographic traits, and health status. Federated learning (FL) is a distributed machine learning algorithm that coordinates clients to train a model collaboratively without sharing local data. This algorithm shows enormous potential for SER applications as sharing raw speech or speech features from a user’s device is vulnerable to privacy attacks. However, a major challenge in FL is limited availability of high-quality labeled data samples. In this work, we propose a semi-supervised federated learning framework, Semi-FedSER, that utilizes both labeled and unlabeled data samples to address the challenge of limited labeled data samples in FL. We show that our Semi-FedSER can generate desired SER performance even when the local label rate l = 20% using two SER benchmark datasets: IEMOCAP and MSP-Improv).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KUANG FU CHEN whose telephone number is (571)272-1393. The examiner can normally be reached M-F 9:00-5:30pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Welch can be reached on (571) 272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KC CHEN/Primary Patent Examiner, Art Unit 2143
