Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is in response to the amendment and remarks filed June 11th, 2026. In the
amendment, claims 1, 4, 15, and 18 were amended, claims 7, 12, and 16-17 were cancelled, and no claims were added. As such, claims 1-6, 8-11, 13-15, and 18-19 are pending.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on June 23rd, 2026, has been entered.
Response to Arguments
Applicant’s arguments with respect to the remaining rejections are not persuasive for the
following reasons:
35 U.S.C 103:
Applicant argues that Kolouri adds a data sample when a prediction confidence exceeds a threshold and therefore fails to disclose adding a data sample to a dataset based on a distance associated with a similarity between an attribute of the data sample and a corresponding attribute of the dataset, that Duan merely classifies handwritten digits and fails to disclose determining a scene identity by inputting a subset of the dataset or the data sample into a scene understanding model and determining the diversity score based on an output of the scene understanding model, and that the remaining references do not cure the alleged deficiencies of Kolouri and Duan (Pages 11 to 13 of Remarks). The Examiner respectfully disagrees. Applicant’s arguments have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Applicant argues that independent claim 18 recites features similar to claim 1 and is patentable for the same reasons (Pages 13 to 14 of Remarks). The Examiner respectfully disagrees. Claim 18 is rejected under the same new grounds and the same combination as claim 1, and Applicant's arguments as to claim 18 are moot for the same reasons stated above. The rejection of claim 18 is maintained.
35 U.S.C 101:
Applicant argues that claims 1 and 18 are eligible at Step 2A Prong Two under Ex parte Desjardins because the combination of adding a data sample based on a distance, determining a diversity score using a scene understanding model, and aggregating models weighted by the diversity score results in an improved training method for a shadow detection model that is more suitable for lightweight devices, citing specification paragraphs 7-9, 16, 47, 115, and 143 (Pages 8 to 10 of Remarks). The Examiner respectfully disagrees. Ex parte Desjardins was deemed a practical application because the improvement to the functioning of the machine learning model itself was recited in the claim and identified with specificity in the specification, such as adjusting parameters to protect performance on a prior task in order to overcome catastrophic forgetting. Here, the limitations on which Applicant relies as allegedly providing the practical application are in fact parts of the recited abstract idea itself, namely the mental processes of determining the diversity score and the scene identity, and the additional element of aggregating shadow detection machine learning models weighted by the diversity score merely limits the abstract idea to the field of shadow detection and links it to a technological environment, which does not integrate the exception into a practical application (see MPEP 2106.05(h)). A benefit described only in the specification does not confer eligibility unless the corresponding improvement to the functioning of the model or the computer is reflected in the claim language. Therefore, the 35 U.S.C. 101 rejection is maintained.
Specification
The disclosure is objected to because of the following informalities:
Paragraph 108: "for example, fore further image analysis" should read "for example, for further image analysis."
Paragraph 121: "Referring in particular to Figure 2D" should read "Referring in particular to Figure 22D."
Paragraph 141: "The receiving client 44 may be one of the n clients" should read "The receiving client 47 may be one of the n clients."
Appropriate correction is required.
Claim Objections
Claims 1-6, 8-11, 13-15, and 19 are objected to because of the following informalities:
Claim 13 recites "adding the data sample to the dataset if the confidence score is above a threshold." Claim 1, from which claim 13 depends, previously recites "a threshold" in connection with the distance between an attribute of the data sample and a corresponding attribute of the dataset. Examiner suggests amending the limitations to recite “a first threshold” for claim 1’s distance threshold, and “a second threshold” for claim 13’s confidence threshold.
Claim 2 recites "generating… models corresponding to each of the clients," which should read "generating… shadow detection machine learning models corresponding to each of the clients" for consistency.
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-6, 8-11, 13-15, and 18-19 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claim 1
Step 1: The claim recites a method; therefore, it is directed to the statutory category of a process.
Step 2A Prong 1: The claim recites, inter alia:
adding a data sample to a dataset corresponding to that client based on determining a distance associated with a similarity between an attribute of the data sample and a corresponding attribute of the dataset is above a threshold: This limitation corresponds to determining whether to add a data sample to the dataset by comparing an attribute of the data sample to an attribute of the dataset, which is a mental process.
determining a diversity score of the dataset … wherein the diversity score is a measure of dataset variability: This limitation recites a mental process because it involves the evaluation/judgement/opinion of determining a diversity score from a dataset, which can be performed mentally or by pen and paper.
wherein the determining the diversity score of the dataset further comprises: determining a scene identity for a subset of the dataset or the data sample included in the dataset…: This limitation recites a mental process because it involves determining an identity for a data sample or subset of a dataset.
and determining the diversity score based on an output of the … model: This limitation recites a mental process because it involves the evaluation/judgement/opinion of determining a score based on the output of a model.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the additional elements are as follows:
for each of a number (N) of clients… the dataset corresponding to that client for training a shadow detection machine learning model… aggregating, weighted by the respective diversity score, shadow detection machine learning models corresponding to each of the clients to generate an aggregated shadow detection machine learning model: The limitation amounts to merely indicating a field of use or technological environment in which to apply a judicial exception. This does not amount to significantly more than the exception itself which cannot provide inventive concept (MPEP 2106.05(h)).
sending the aggregated shadow detection machine learning model to at least one receiving client: Insignificant extra-solution activity as the limitation amounts to transmitting data (MPEP 2106.05(g)(3)).
…by inputting the subset of the dataset or the data sample into a scene understanding model: Mere data gathering recited at a high level of generality, and thus is an insignificant extra-solution activity (MPEP 2106.05(g)).
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements are as follows:
for each of a number (N) of clients… that client for training a shadow detection machine learning model… aggregating, weighted by the respective diversity score, shadow detection machine learning models corresponding to each of the clients to generate an aggregated shadow detection machine learning model: The limitation amounts to merely indicating a field of use or technological environment in which to apply a judicial exception. This does not amount to significantly more than the exception itself (MPEP 2106.05(h)).
sending the aggregated shadow detection machine learning model to at least one receiving client: Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone) (MPEP 2106.05(d)(II)).
…by inputting the subset of the dataset or the data sample into a scene understanding model: The additional element of “inputting” does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. As discussed above with respect to integration of the abstract idea into a practical application, the additional element of receiving steps amounts to no more than mere data gathering. This element amounts to receiving data over a network and are well-understood, routine, conventional activity. See MPEP 2106.05(d), subsection II (i). This cannot provide an inventive concept.
The elements in combination as an ordered whole do not amount to significantly more than the judicial exception (i.e., the abstract ideas of determining whether to add a data sample to a dataset, determining a scene identity, and evaluating dataset variability and the diversity score). The claim merely uses the abstract idea to analyze data and generate an aggregated model. The remaining steps recite conventional data processing operations such as adding data samples to a dataset from multiple clients, training machine learning models using generic computing components, inputting data into a scene understanding model, aggregating model parameters at a server, and transmitting the aggregated model to client devices. These elements merely implement the abstract idea on generic computer components and represent routine data gathering, analysis, and output operations without improving the functioning of a computer or any other technology. The claim as a whole is directed to the abstract idea of the diversity score being determined.
Therefore, the claim as a whole remains focused on the abstract idea and fails Step 2B of the eligibility analysis.
Claim 2
Step 1: A process, as above.
Step 2A Prong 1: The claim recites:
assigning each client to a cluster from a plurality of clusters based on one or more dataset attributes: This limitation is a mental process because it involves the assignment of a client to a cluster, which can be performed mentally or by pen and paper.
and aggregating the aggregated cluster weights: This limitation is a mental process because it involves aggregating clustered weights, which can be performed mentally or by pen and paper.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the additional elements are as follows:
the aggregating the shadow detection machine learning models corresponding to each of the N clients comprises: for each of the: Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea (MPEP 2106.05(f)).
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements are as follows:
the aggregating the shadow detection machine learning models corresponding to each of the N clients comprises: for each of the: Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea and cannot provide inventive concept (MPEP 2106.05(f)).
Even when considered in combination, these additional elements represent mere instructions to apply an exception and therefore do not provide an inventive concept. The claim is ineligible.
Claim 3
Step 1: A process, as above.
Step 2A Prong 1: The claim recites:
assigning each client to the cluster from the plurality of clusters comprises: assigning a cluster identity to the dataset used…: This limitation corresponds to an assignment of a client to a cluster based on the identity of the dataset, which recites a mental process.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the additional elements are as follows:
…to train the model shadow detection machine learning model: Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea (MPEP 2106.05(f)).
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements are as follows:
…to train the model shadow detection machine learning model: Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea and cannot provide inventive concept (MPEP 2106.05(f)).
Even when considered in combination, these additional elements represent mere instructions to apply an exception and therefore do not provide an inventive concept. The claim is ineligible.
Claim 4
Step 1: A process, as above.
Step 2A Prong 1: The claim recites:
assigning the cluster identity to the dataset comprises calculating a vector of softmax probabilities or extracting an embedding vector from a classification model: This limitation involves calculating a softmax probability or extracting an embedding vector, which is a mathematical concept.
Step 2A Prong 2 and Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B. Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2. The claim is ineligible.
Even when considered in combination, these additional elements represent mere instructions to apply an exception and therefore do not provide an inventive concept. The claim is ineligible.
Claim 5
Step 1: A process, as above.
Step 2A Prong 1: This claim does not recite an additional abstract idea, but the claim depends on claim 1 which recites an abstract idea.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the additional elements are as follows:
the dataset has been used for training a local model cached on the client: This limitation is merely a post-solution step of storing the data—a nominal addition to the claim that does not meaningfully limit the claim. The method storing is recited at a high level of generality. Simply implementing the abstract idea in a generic method is not a practical application of the abstract idea. Therefore, storing step is an insignificant extra-solution activity. See MPEP 2106.05(g).
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements are as follows:
the dataset has been used for training a local model cached on the client: These elements amount to storing… information in memory, Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93; See MPEP 2106.05(d) (II)(iv). The courts have recognized the computer functions of storing as well‐understood, routine, and conventional function when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity.
Even when considered in combination, these additional elements represent mere instructions to apply an exception and therefore do not provide an inventive concept. The claim is ineligible.
Claim 6
Step 1: A process, as above.
Step 2A Prong 1: The claim recites:
for each of the N clients: applying a differential privacy function to the diversity score weighted model weight: This limitation corresponds to a mathematical concept because it involves applying a function to the diversity scire for each client. Paragraph 138 of the instant specification describes applying Equation 3 to perform the steps of applying the function to the diversity score.
Step 2A Prong 2 and Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B. Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2. The claim is ineligible.
Even when considered in combination, these additional elements represent mere instructions to apply an exception and therefore do not provide an inventive concept. The claim is ineligible.
Claim 8
Step 1: A process, as above.
Step 2A Prong 1: This claim does not recite an additional abstract idea, but the claim depends on claim 1 which recites an abstract idea.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the additional elements are as follows:
wherein the aggregating the shadow detection machine learning models are performed on a central server: The limitation amounts to merely indicating a field of use or technological environment in which to apply a judicial exception. This does not amount to significantly more than the exception itself (MPEP 2106.05(h)).
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements are as follows:
wherein the aggregating the shadow detection machine learning models are performed on a central server: The limitation amounts to merely indicating a field of use or technological environment in which to apply a judicial exception. This does not amount to significantly more than the exception itself which cannot provide inventive concept (MPEP 2106.05(h)).
Even when considered in combination, these additional elements represent mere instructions to apply an exception and therefore do not provide an inventive concept. The claim is ineligible.
Claim 9
Step 1: A process, as above.
Step 2A Prong 1: This claim does not recite an additional abstract idea, but the claim depends on claim 1 which recites an abstract idea.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the additional elements are as follows:
wherein the dataset used to train the shadow detection machine learning model comprises image data: The limitation amounts to merely indicating a field of use or technological environment in which to apply a judicial exception. This does not amount to significantly more than the exception itself (MPEP 2106.05(h)).
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements are as follows:
wherein the dataset used to train the shadow detection machine learning model comprises image data: The limitation amounts to merely indicating a field of use or technological environment in which to apply a judicial exception. This does not amount to significantly more than the exception itself which cannot provide inventive concept (MPEP 2106.05(h)).
Even when considered in combination, these additional elements represent mere instructions to apply an exception and therefore do not provide an inventive concept. The claim is ineligible.
Claim 10
Step 1: A process, as above.
Step 2A Prong 1: This claim does not recite an abstract idea, but the claim depends on claim 1 which recites an abstract idea.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the additional elements are as follows:
the dataset used to train the shadow detection machine learning model comprises an input provided by a user: Data Gather- Mere data gathering recited at a high level of generality, and thus is an insignificant extra-solution activity (MPEP 2106.05(g)).
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements are as follows:
the dataset used to train the shadow detection machine learning model comprises an input provided by a user: Mere data gathering recited at a high level of generality, and thus is an insignificant extra-solution activity. See MPEP 2106.05(g). The additional element of “receiving” does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. As discussed above with respect to integration of the abstract idea into a practical application, the additional element of receiving steps amounts to no more than mere data gathering. This element amounts to receiving data over a network and are well-understood, routine, conventional activity. See MPEP 2106.05(d), subsection II (i). This cannot provide an inventive concept.
Even when considered in combination, these additional elements represent mere instructions to apply an exception and therefore do not provide an inventive concept. The claim is ineligible.
Claim 11
Step 1: A process, as above.
Step 2A Prong 1: This claim does not recite an additional abstract idea, but the claim depends on claim 1 which recites an abstract idea.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the additional elements are as follows:
wherein the dataset used to train the model comprises a mask.: The limitation merely describes the type of data being processed and thus amounts to merely indicating a field of use or technological environment in which to apply a judicial exception. This does not amount to significantly more than the exception itself (MPEP 2106.05(h)).
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements are as follows:
wherein the dataset used to train the model comprises a mask: The limitation merely describes the type of data being processed and thus amounts to merely indicating a field of use or technological environment in which to apply a judicial exception. This does not amount to significantly more than the exception itself which cannot provide inventive concept (MPEP 2106.05(h)).
Even when considered in combination, these additional elements represent mere instructions to apply an exception and therefore do not provide an inventive concept. The claim is ineligible.
Claim 13
Step 1: A process, as above.
Step 2A Prong 1: The claim recites:
in response to a trigger condition the method further comprises, for each data sample: determining a confidence score of that data sample: adding the data sample to the dataset if the confidence score is above a threshold; and discarding the data sample if the confidence score is below the threshold: This claim encompasses a mental process because it involves determining whether to add or remove a data sample to the dataset based on a threshold, which can be performed mentally or by pen and paper.
Step 2A Prong 2 and Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B. Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2. The claim is ineligible.
Even when considered in combination, these additional elements represent mere instructions to apply an exception and therefore do not provide an inventive concept. The claim is ineligible.
Claim 14
Step 1: A process, as above.
Step 2A Prong 1: The claim recites:
in response to a trigger condition, the method further comprises, for each data sample: determining a first confidence score for that data sample to generate an augmented data sample: augmenting that data sample, determining a second confidence score for the augmented data sample: This limitation is a mental process because it involves determining confidence scores to generate augmented data samples, which can be performed mentally or by pen and paper.
discarding that data sample if the first and second confidence scores are above a first threshold distance: based on determining the first confidence score is above a second threshold, adding that data sample to the dataset: and based on determining the first confidence score is below the second threshold, discarding that data sample: This limitation is a mental process because it involves determining whether to add or discard a data sample to the dataset based on the determined confidence scores meeting a threshold.
Step 2A Prong 2 and Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B. Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2. The claim is ineligible.
Even when considered in combination, these additional elements represent mere instructions to apply an exception and therefore do not provide an inventive concept. The claim is ineligible.
Claim 15
Step 1: A process, as above.
Step 2A Prong 1: The claim recites inter alia:
the determining the confidence score of the data sample further comprises determining a softmax probability for the data sample: This limitation recites a mental process because it involves determining a score based on the softmax probability.
Step 2A Prong Two and Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B. Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2. The claim is ineligible.
Even when considered in combination, these additional elements represent mere instructions to apply an exception and therefore do not provide an inventive concept. The claim is ineligible.
Claim 18
Step 1: The claim recites a system; therefore, it is directed to the statutory category of a machine.
Step 2A Prong 1: The claim recites, inter alia:
adding a data sample to a dataset corresponding to that client based on determining a distance associated with a similarity between an attribute of the data sample and a corresponding attribute of the dataset is above a threshold: This limitation corresponds to determining whether to add a data sample to the dataset by comparing an attribute of the data sample to an attribute of the dataset, which is a mental process.
determining a diversity score of the dataset … wherein the diversity score is a measure of dataset variability: This limitation recites a mental process because it involves the evaluation/judgement/opinion of determining a diversity score from a dataset, which can be performed mentally or by pen and paper.
wherein the determining the diversity score of the dataset further comprises: determining a scene identity for a subset of the dataset or the data sample included in the dataset…: This limitation recites a mental process because it involves determining an identity for a data sample or subset of a dataset.
and determining the diversity score based on an output of the … model: This limitation recites a mental process because it involves the evaluation/judgement/opinion of determining a score based on the output of a model.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the additional elements are as follows:
[a] computer system comprising: a memory configured to store a dataset; and it least one processor; wherein the at least one processor is configured to: Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea (MPEP 2106.05(f)).
for each of a number (N) of clients… the dataset corresponding to that client for training a shadow detection machine learning model… aggregating, weighted by the respective diversity score, shadow detection machine learning models corresponding to each of the clients to generate an aggregated shadow detection machine learning model: The limitation amounts to merely indicating a field of use or technological environment in which to apply a judicial exception. This does not amount to significantly more than the exception itself (MPEP 2106.05(h)).
sending the aggregated shadow detection machine learning model to at least one receiving client: Insignificant extra-solution activity as the limitation amounts to transmitting data (MPEP 2106.05(g)(3)).
…by inputting the subset of the dataset or the data sample into a scene understanding model: Mere data gathering recited at a high level of generality, and thus is an insignificant extra-solution activity (MPEP 2106.05(g)).
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements are as follows:
[a] computer system comprising: a memory configured to store a dataset; and it least one processor; wherein the at least one processor is configured to: Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea and cannot provide an inventive concept (MPEP 2106.05(f)).
for each of a number (N) of clients… that client for training a shadow detection machine learning model… aggregating, weighted by the respective diversity score, shadow detection machine learning models corresponding to each of the clients to generate an aggregated shadow detection machine learning model: The limitation amounts to merely indicating a field of use or technological environment in which to apply a judicial exception. This does not amount to significantly more than the exception itself which cannot provide inventive concept (MPEP 2106.05(h)).
sending the aggregated shadow detection machine learning model to at least one receiving client: Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone) (MPEP 2106.05(d)(II)).
…by inputting the subset of the dataset or the data sample into a scene understanding model: The additional element of “inputting” does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. As discussed above with respect to integration of the abstract idea into a practical application, the additional element of receiving steps amounts to no more than mere data gathering. This element amounts to receiving data over a network and are well-understood, routine, conventional activity. See MPEP 2106.05(d), subsection II (i). This cannot provide an inventive concept.
The elements in combination as an ordered whole do not amount to significantly more than the judicial exception (i.e., the abstract ideas of determining whether to add a data sample to a dataset, determining a scene identity, and evaluating dataset variability and the diversity score). The claim merely uses the abstract idea to analyze data and generate an aggregated model. The remaining steps recite conventional data processing operations such as adding data samples to a dataset from multiple clients, training machine learning models using generic computing components, inputting data into a scene understanding model, aggregating model parameters at a server, and transmitting the aggregated model to client devices. These elements merely implement the abstract idea on generic computer components and represent routine data gathering, analysis, and output operations without improving the functioning of a computer or any other technology. The claim as a whole is directed to the abstract idea of the diversity score being determined.
Therefore, the claim as a whole remains focused on the abstract idea and fails Step 2B of the eligibility analysis.
Claim 19
Step 1: A process, as above.
Step 2A Prong 1: This claim does not recite an additional abstract idea, but the claim depends on claim 1 which recites an abstract idea.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the additional elements are as follows:
the at least one receiving client is a smartphone or a tablet.: The limitation amounts to merely indicating a field of use or technological environment in which to apply a judicial exception. This does not amount to significantly more than the exception itself (MPEP 2106.05(h)).
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements are as follows:
the at least one receiving client is a smartphone or a tablet.: The limitation amounts to merely indicating a field of use or technological environment in which to apply a judicial exception. This does not amount to significantly more than the exception itself which cannot provide inventive concept (MPEP 2106.05(h)).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim 1-3, 5, 8-10, and 18-19 is rejected under 35 U.S.C. 103 as being unpatentable over Duan (“FedGroup: Efficient Federated Learning via Decomposed Similarity-Based Clustering”, 2021) in view of Towal (US 11334789 B2), in further view of Wang ("REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets", 2020), and in further view of Adiri (US 20220217287 A1).
Regarding claim 1,
Duan teaches [a] computer-implemented federated learning method comprising: (Page 2 Background of Duan, "we introduce the most widely adopted FL algorithm FedAvg and briefly explain the optimization method FedProx. McMahan et al. first introduce federated learning [2] and the vanilla FL optimization method FedAvg, which is designed to provide privacy-preserving support for distributed machine learning model training." Duan discloses a federated learning method carried out across client devices and a server.)
for each of a number (N) of clients: (Page 2 Background of Duan, "Where N is the number of clients", Page 1 Introduction of Duan, "In practice, a FL server first distributes the global model to a random subset of participating clients (e.g. mobile and IoT devices). Then each client optimizes its local model by gradient descent based on its local data in parallel." Duan defines N as the number of participating clients and performs the federated operations for each client.)
aggregating, weighted by the respective … [value], … machine learning models corresponding to each of the clients to generate an aggregated … machine learning model (Page 1 Introduction of Duan, "Finally, the FL server averages all local models' updates or parameters and aggregates them to construct a new global model", See Figure 2 and its description in Page 4 Section III of Duan, "Specifically, each group broadcasts its model parameters to their clients and then aggregates the updates from these clients using FedAvg in parallel, we call this aggregation procedure the intra-group aggregation. After all federated trainings of groups complete, the models are aggregated using a certain weight, which we call inter-group aggregation.",
Duan aggregates the client models into an aggregated global model and performs that aggregation using a weight (inter-group aggregation), which corresponds to the claimed aggregating of models corresponding to each of the clients to generate an aggregated model, weighted by a respective value.)
sending the aggregated … machine learning model to at least one receiving client (Page 1 Introduction of Duan, "In practice, a FL server first distributes the global model to a random subset of participating clients (e.g. mobile and IoT devices). Then each client optimizes its local model by gradient descent based on its local data in parallel." Duan distributes the aggregated global model to participating clients, which corresponds to sending the aggregated model to at least one receiving client.)
Duan does not teach adding a data sample to a dataset based on determining a distance associated with a similarity between an attribute of the data sample and a corresponding attribute of the dataset is above a threshold. Towal, in the same field of endeavor, teaches adding a data sample to a dataset corresponding to that client based on determining a distance associated with a similarity between an attribute of the data sample and a corresponding attribute of the dataset is above a threshold (Col. 5 Lines 20-27 of Towal, "In accordance with aspects of the present disclosure, a data sample may be added to a training set based on a similarity metric. The data samples may be stored or may be provided sequentially via a data stream from an external source (e.g., a mobile device, digital camera, video recording device, a streaming media input or other data source). In some aspects, data samples that are not added may be discarded, for data storage considerations, for example.", Col. 5 Lines 34-36, "The similarity metric may, for example, comprise a distance metric relative to other samples, either within the same class or different classes of the existing training set.", Col. 5 Lines 41-47, “...new feature vectors may be added until the limit or maximum number is reached… the determination may be made based on the similarity of the new feature vector as compared with existing feature vectors.”, Col. 6 Lines 17-22, "If the new sample's closest distance is smaller than the preexisting minimum distance, the new sample may be determined to be redundant and may be rejected. On the other hand, if the new sample's closest distance is larger than the preexisting minimum distance, the new sample may be stored."
Towal computes a distance-based similarity metric between a new data sample and the existing training samples and adds the new sample to the training set when that distance is larger than the pre-existing minimum distance (the threshold), while discarding it when the distance is smaller (redundant). The new sample's feature vector corresponds to the attribute of the data sample, and the existing training samples correspond to the corresponding attribute of the dataset.) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Duan's federated learning method with Towal's similarity and distance based selective addition of data samples in order to retain informative, non-redundant training samples on the client while reducing the memory required to store the training set (Col. 6 Lines 17-22 of Towal).
Duan in view of Towal does not teach determining a diversity score of a dataset that is a measure of dataset variability. Wang, in the same field of endeavor, teaches determining a diversity score of the dataset corresponding to that client for training a … machine learning model, wherein the diversity score is a measure of dataset variability (Page 6 Section 4.1.5 of Wang, "Building on quantifying the common context of an object, we additionally strive to measure the scene diversity directly. To do so, for every object class we consider the entropy of scene categories in which the object appears. We use a ResNet-18 (He et al., 2016) trained on Places … to classify every image into one of 16 scene groups", Page 7 Section 4.1.6, “Finally, we consider the appearance diversity (i.e., intra class variation) of each object class, which is a primary challenge in object detection… We first validate that distances in this feature space correspond to semantically meaningful measures of diversity. To do so, on the COCO dataset we compute the average distance with n = 500,000 between two object instances of the same class… This metric allows us to identify individual object instances that contribute the most to the diversity of an object class” Wang determines a scene diversity of a dataset directly from the entropy of the scene categories output by the scene classifier, which corresponds to determining a diversity score that is a measure of dataset variability.)
wherein the determining the diversity score of the dataset further comprises: determining a scene identity for a subset of the dataset or the data sample included in the dataset by inputting the subset of the dataset or the data sample into a scene understanding model (Page 6 Section 4.1.5 of Wang, "Building on quantifying the common context of an object, we additionally strive to measure the scene diversity directly. To do so, for every object class we consider the entropy of scene categories in which the object appears. We use a ResNet-18 (He et al., 2016) trained on Places … to classify every image into one of 16 scene groups", Page 6 Section 4.1.6, “Finally, we consider the appearance diversity (i.e., intra class variation) of each object class, which is a primary challenge in object detection… We first validate that distances in this feature space correspond to semantically meaningful measures of diversity. To do so, on the COCO dataset we compute the average distance with n = 500,000 between two object instances of the same class… This metric allows us to identify individual object instances that contribute the most to the diversity of an object class” Wang inputs each image of the dataset into a places-trained scene recognition network and classifies it into one of a plurality of scene groups, which corresponds to determining a scene identity for the data sample by inputting the data sample into a scene understanding model.)
and determining the diversity score based on an output of the scene understanding model (See Figure 5, Page 6 Section 4.1.5 of Wang, "To do so, for every object class we consider the entropy of scene categories in which the object appears. We use a ResNet-18 … trained on Places … to classify every image into one of 16 scene groups, and identify objects like person that appear in a higher diversity of scenes versus objects like baseball glove that appear in fewer kinds of scenes (almost all baseball fields)."
Wang then determines the scene diversity from the entropy of those scene group classifications, which corresponds to determining the diversity score based on an output of the scene understanding model.)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Duan in view of Towal's federated learning in further view of Wang's scene-understanding-based diversity measurement in order to quantify the scene diversity of a client's dataset from the output of a scene classifier and thereby reduce over representation of any single scene and improve generalization of the trained model (Section 4.1.5 of Wang).
Duan in view of Towal and in further view of Wang does not teach a shadow detection machine learning model. Adiri, in the same field of endeavor, teaches aggregating… shadow detection machine learning models corresponding to … the clients to generate an aggregated shadow detection machine learning model (Paragraph 34, "...the operations may further include detecting... the presence of a shadow in the plurality of frames...", Paragraph 128, "...by combining the two or more other artificial neural networks into a single artificial neural network.", Paragraph 252, "The user rating may include a score, a percentage, or any other metric indicative of the user's actual positioning and/or movement of the mobile device relative to the desired positions and/or directions indicated by the mobile device", Paragraph 254, “In some examples, a machine learning model may be trained using training examples to detect presence of shadows in images and/or videos.”, Paragraph 7, “the field of wound care could greatly benefit from new and improved systems and methods implementing real-time overlays on video feeds of mobile devices would provide great benefits in the field of wound care.”
Adiri discloses a machine learning model trained to detect the presence of a shadow in images and/or videos, and further discloses combining two or more neural networks into a single artificial neural network, which corresponds to the claimed shadow detection machine learning models and aggregating them to generate an aggregated shadow detection machine learning model.) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Duan in view of Towal and in further view of Wang's teaching with Adiri's shadow detection machine learning model and its combination of networks into a single aggregated network in order to enhance the accuracy of the model (Paragraphs 34 and 254 of Adiri).
Regarding claim 2,
Duan teaches for each of the N clients: assigning each client to a cluster from a plurality of clusters based on one or more dataset attributes (See Equation 1 in Section II Background, “Where N is the number of clients”, Pg. 4 Section III B, “Each client and group have a one-to-one correspondence, but there are also possible to have a group without clients in a communication round (empty group) or a client is not in any groups (e.g. a newcomer joins the training later)”, Page 7 Section IV A, “We evaluate FedGroup and FedGrouProx on four federated datasets, which including two image classification tasks, a synthetic dataset, a sentiment analysis task.”
Duan assigns each client to one of a plurality of groups (clusters) based on the client's dataset and optimization direction.);
for each cluster from a plurality of clusters, generating aggregated cluster weights by aggregating, weighted by the respective [value], models corresponding to each of the clients; and aggregating the aggregated cluster weights (Page 4 Under Figure 2, “Specifically, each group broadcasts its model parameters to their clients and then aggregates the updates from these clients using FedAvg in parallel, we call this aggregation procedure the intra-group aggregation”, Page 7 Section IV A, “Synthetic. It’s a synthetic federated dataset proposed by Shamir et. al [36]. Our hyperparameter settings of this data-generated algorithm are α = 1, β = 1, which control the statistical heterogeneity among clients. We study a MCLR model based on it”
Duan performs an intra-group aggregation to generate per-cluster aggregated weights and then an inter-group aggregation of those cluster weights, which corresponds to generating aggregated cluster weights and aggregating the aggregated cluster weights.).
Duan in view of Towal does not teach the diversity score.
Wang, in the same field of endeavor, teaches the diversity score [used in training the machine learning model] (Page 6 Section 4.1.5 of Wang, "Building on quantifying the common context of an object, we additionally strive to measure the scene diversity directly. To do so, for every object class we consider the entropy of scene categories in which the object appears. We use a ResNet-18 (He et al., 2016) trained on Places … to classify every image into one of 16 scene groups"
Wang determines a scene diversity of a dataset directly from the entropy of the scene categories output by the scene classifier, which corresponds to determining a diversity score that is a measure of dataset variability.)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Duan in view of Towal's federated learning in further view of Wang's scene-understanding-based diversity measurement in order to quantify the scene diversity of a client's dataset from the output of a scene classifier and thereby reduce over representation of any single scene and improve generalization of the trained model (Section 4.1.5 of Wang).
Duan in view of Towal in further view of Wang does not teach a shadow detection machine learning model.
Adiri, in the same field of endeavor, teaches aggregating the shadow detection machine learning models corresponding to each of the clients (Paragraph 34, "...the operations may further include detecting... the presence of a shadow in the plurality of frames...", Paragraph 128, "...by combining the two or more other artificial neural networks into a single artificial neural network.", Paragraph 252, “The user rating may include a score, a percentage, or any other metric indicative of the user's actual positioning and/or movement of the mobile device relative to the desired positions and/or directions indicated by the mobile device”)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Duan in view of Towal and in further view of Wang's teaching with Adiri's shadow detection machine learning model and its combination of networks into a single aggregated network in order to enhance the accuracy of the model (Paragraphs 34 and 254 of Adiri).
Regarding claim 3,
Duan teaches assigning each client to the cluster from the plurality of clusters comprises: assigning a cluster identity to the dataset used to train the… machine learning model (Page 4 Section III B, “Each client and group have a one-to-one correspondence, but there are also possible to have a group without clients in a communication round (empty group) or a client is not in any groups (e.g. a newcomer joins the training later). The newcomer device uses a cold start algorithm to determine the assigned group (we call the unassigned client is cold), we will explain the details of the cold start algorithm later. For the dataset, each client ci has a training set and a test set according to its local data distribution p(ci)data.”, Page 6 Section III D, “Our client cold start strategy is to assign clients to the groups that are most closely related to their optimization direction…”
Duan assigns each client's local dataset to a group based on data similarity and optimization direction, and that group functions as the cluster identity for the dataset.)
Duan in view of Towal in further view of Wang does not teach a shadow detection machine learning model.
Adiri, in the same field of endeavor, teaches shadow detection machine learning model (Paragraph 34, "...the operations may further include detecting... the presence of a shadow in the plurality of frames...", Paragraph 128, "...by combining the two or more other artificial neural networks into a single artificial neural network.", Paragraph 252, “The user rating may include a score, a percentage, or any other metric indicative of the user's actual positioning and/or movement of the mobile device relative to the desired positions and/or directions indicated by the mobile device”)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Duan in view of Towal and in further view of Wang's teaching with Adiri's shadow detection machine learning model and its combination of networks into a single aggregated network in order to enhance the accuracy of the model (Paragraphs 34 and 254 of Adiri).
Regarding claim 5,
Duan teaches the dataset is used for training a local model cached on the client (Pg. 228 Introduction, “Then each client optimizes its local model by gradient descent based on its local data in parallel. Finally, the FL server averages all local models’ updates or parameters and aggregates them to construct a new global model.”
Duan trains a local model on each client using that client's local data.)
Regarding claim 8,
Duan teaches the aggregating the… machine learning models is performed on a central server (Page 4 Section III B of Duan, “The federated training procedure of FedGroup is shown in Fig. 2…. the clients can be connected mobiles or IoT devices, the auxiliary server can be deployed in the cloud (FL server), and the group can be deployed on the FL server or the mobile edge computing (MEC) server. To ease the discussion, we assume that all groups are deployed on the same cloud server, and all clients are connected mobile devices…. Specifically, each group broadcasts its model parameters to their clients and then aggregates the updates from these clients using FedAvg in parallel, we call this aggregation procedure the intra-group aggregation. After all federated trainings of groups complete, the models are aggregated using a certain weight, which we call inter-group aggregation.”
Duan performs the model aggregation on a cloud/FL server, which corresponds to the aggregating being performed on a central server.).
Duan and Li do not teach a shadow detection machine learning model.
Adiri, in the same field of endeavor, teaches shadow detection machine learning model (Paragraph 34, "...the operations may further include detecting... the presence of a shadow in the plurality of frames...", Paragraph 128, "...by combining the two or more other artificial neural networks into a single artificial neural network.", Paragraph 252, “The user rating may include a score, a percentage, or any other metric indicative of the user's actual positioning and/or movement of the mobile device relative to the desired positions and/or directions indicated by the mobile device”)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Duan in view of Towal and in further view of Wang's teaching with Adiri's shadow detection machine learning model and its combination of networks into a single aggregated network in order to enhance the accuracy of the model (Paragraphs 34 and 254 of Adiri).
Regarding claim 9,
Duan teaches the dataset used to train the… machine learning model comprises image data (Page 7 Section IV A, “We evaluate FedGroup and FedGrouProx on four federated datasets, which including two image classification tasks, a synthetic dataset, a sentiment analysis task.”
Duan trains on federated datasets that include image classification tasks).
Duan and Li do not teach a shadow detection machine learning model.
Adiri, in the same field of endeavor, teaches shadow detection machine learning model (Paragraph 34, "...the operations may further include detecting... the presence of a shadow in the plurality of frames...", Paragraph 128, "...by combining the two or more other artificial neural networks into a single artificial neural network.", Paragraph 252, “The user rating may include a score, a percentage, or any other metric indicative of the user's actual positioning and/or movement of the mobile device relative to the desired positions and/or directions indicated by the mobile device”)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Duan in view of Towal and in further view of Wang's teaching with Adiri's shadow detection machine learning model and its combination of networks into a single aggregated network in order to enhance the accuracy of the model (Paragraphs 34 and 254 of Adiri).
Regarding claim 10,
Duan teaches the dataset used to train the… machine learning model comprises an input provided by a user (Page 1 Introduction of Duan, “…which enables multiple end-users to collaboratively train a shared neural network model while keeping the training data decentralized. In practice, a FL server first distributes the global model to a random subset of participating clients (e.g. mobile and IoT devices). Then each client optimizes its local model by gradient descent based on its local data in parallel.”.
Duan discloses that the training dataset is derived from the local data on each end-user's client device, which corresponds to the dataset comprising an input provided by a user.)
Duan and Li do not teach a shadow detection machine learning model.
Adiri, in the same field of endeavor, teaches shadow detection machine learning model (Paragraph 34, "...the operations may further include detecting... the presence of a shadow in the plurality of frames...", Paragraph 128, "...by combining the two or more other artificial neural networks into a single artificial neural network.", Paragraph 252, “The user rating may include a score, a percentage, or any other metric indicative of the user's actual positioning and/or movement of the mobile device relative to the desired positions and/or directions indicated by the mobile device”)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Duan in view of Towal and in further view of Wang's teaching with Adiri's shadow detection machine learning model and its combination of networks into a single aggregated network in order to enhance the accuracy of the model (Paragraphs 34 and 254 of Adiri).
Regarding claim 18,
Duan teaches for each of a number (N) of clients: (Page 2 Background of Duan, "Where N is the number of clients", Page 1 Introduction of Duan, "In practice, a FL server first distributes the global model to a random subset of participating clients (e.g. mobile and IoT devices). Then each client optimizes its local model by gradient descent based on its local data in parallel." Duan defines N as the number of participating clients and performs the federated operations for each client.)
aggregating, weighted by the respective … [value], … machine learning models corresponding to each of the clients to generate an aggregated … machine learning model (Page 1 Introduction of Duan, "Finally, the FL server averages all local models' updates or parameters and aggregates them to construct a new global model", See Figure 2 and its description in Page 4 Section III of Duan, "Specifically, each group broadcasts its model parameters to their clients and then aggregates the updates from these clients using FedAvg in parallel, we call this aggregation procedure the intra-group aggregation. After all federated trainings of groups complete, the models are aggregated using a certain weight, which we call inter-group aggregation.",
Duan aggregates the client models into an aggregated global model and performs that aggregation using a weight (inter-group aggregation), which corresponds to the claimed aggregating of models corresponding to each of the clients to generate an aggregated model, weighted by a respective score.)
sending the aggregated … machine learning model to at least one receiving client (Page 1 Introduction of Duan, "In practice, a FL server first distributes the global model to a random subset of participating clients (e.g. mobile and IoT devices). Then each client optimizes its local model by gradient descent based on its local data in parallel." Duan distributes the aggregated global model to participating clients, which corresponds to sending the aggregated model to at least one receiving client.)
Duan does not teach adding a data sample to a dataset based on determining a distance associated with a similarity between an attribute of the data sample and a corresponding attribute of the dataset is above a threshold. Towal, in the same field of endeavor, teaches adding a data sample to a dataset corresponding to that client based on determining a distance associated with a similarity between an attribute of the data sample and a corresponding attribute of the dataset is above a threshold (Col. 5 Lines 20-27 of Towal, "In accordance with aspects of the present disclosure, a data sample may be added to a training set based on a similarity metric. The data samples may be stored or may be provided sequentially via a data stream from an external source (e.g., a mobile device, digital camera, video recording device, a streaming media input or other data source). In some aspects, data samples that are not added may be discarded, for data storage considerations, for example.", Col. 5 Lines 34-36, "The similarity metric may, for example, comprise a distance metric relative to other samples, either within the same class or different classes of the existing training set.", Col. 5 Lines 41-47, “...new feature vectors may be added until the limit or maximum number is reached… the determination may be made based on the similarity of the new feature vector as compared with existing feature vectors.”, Col. 6 Lines 17-22, "If the new sample's closest distance is smaller than the preexisting minimum distance, the new sample may be determined to be redundant and may be rejected. On the other hand, if the new sample's closest distance is larger than the preexisting minimum distance, the new sample may be stored."
Towal computes a distance-based similarity metric between a new data sample and the existing training samples, and adds the new sample to the training set when that distance is larger than the pre-existing minimum distance (the threshold), while discarding it when the distance is smaller (redundant). The new sample's feature vector corresponds to the attribute of the data sample, and the existing training samples correspond to the corresponding attribute of the dataset.) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Duan's federated learning method with Towal's similarity and distance based selective addition of data samples in order to retain informative, non-redundant training samples on the client while reducing the memory required to store the training set (Col. 6 Lines 17-22 of Towal).
Duan in view of Towal does not teach determining a diversity score of a dataset that is a measure of dataset variability. Wang, in the same field of endeavor, teaches determining a diversity score of the dataset corresponding to that client for training a … machine learning model, wherein the diversity score is a measure of dataset variability (Page 6 Section 4.1.5 of Wang, "Building on quantifying the common context of an object, we additionally strive to measure the scene diversity directly. To do so, for every object class we consider the entropy of scene categories in which the object appears. We use a ResNet-18 (He et al., 2016) trained on Places … to classify every image into one of 16 scene groups", Page 7 Section 4.1.6, “We first validate that distances in this feature space correspond to semantically meaningful measures of diversity. To do so, on the COCO dataset we compute the average distance with n = 500, 000 between two object instances of the same class (5.91 ± 1.44), and verify that it is smaller than the average distance between two object instances belonging to different classes but the same supercategory (6.24 ± 1.42), with a Cohen’s D effect size of .23 and further smaller than the average distance between two unrelated objects (6.48 ± 1.44), with a Cohen’s D effect size of .17.” Wang determines a scene diversity of a dataset directly from the entropy of the scene categories output by the scene classifier, which corresponds to determining a diversity score that is a measure of dataset variability.)
wherein the determining the diversity score of the dataset further comprises: determining a scene identity for a subset of the dataset or the data sample included in the dataset by inputting the subset of the dataset or the data sample into a scene understanding model (Page 6 Section 4.1.5 of Wang, "Building on quantifying the common context of an object, we additionally strive to measure the scene diversity directly. To do so, for every object class we consider the entropy of scene categories in which the object appears. We use a ResNet-18 (He et al., 2016) trained on Places … to classify every image into one of 16 scene groups", Wang inputs each image of the dataset into a places-trained scene recognition network and classifies it into one of a plurality of scene groups, which corresponds to determining a scene identity for the data sample by inputting the data sample into a scene understanding model.)
and determining the diversity score based on an output of the scene understanding model (Page 6 Section 4.1.5 of Wang, "We use a ResNet-18 … trained on Places … to classify every image into one of 16 scene groups, and identify objects like person that appear in a higher diversity of scenes versus objects like baseball glove that appear in fewer kinds of scenes (almost all baseball fields)."
Wang then determines the scene diversity from the entropy of those scene group classifications, which corresponds to determining the diversity score based on an output of the scene understanding model.)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Duan in view of Towal's federated learning in further view of Wang's scene-understanding-based diversity measurement in order to quantify the scene diversity of a client's dataset from the output of a scene classifier and thereby reduce over representation of any single scene and improve generalization of the trained model (Section 4.1.5 of Wang).
Duan in view of Towal and in further view of Wang does not teach a shadow detection machine learning model. Adiri, in the same field of endeavor, teaches [a] computer system comprising: a memory configured to store a dataset and at least one processor; wherein the at least one processor is configured to: (Paragraph 10, “One aspect of the present disclosure is directed to a non-transitory computer readable medium storing data and computer implementable instructions that, when executed by at least one processor, may cause the at least one processor to perform operations for generating cross section views of a wound.”, Paragraph 11, “The system may include a memory storing instructions; and at least one processor configured to execute the instructions to receive 3D information of a wound based on information captured using an image sensor associated with an image plane substantially parallel to the wound;”)
aggregating… shadow detection machine learning models corresponding to … the clients to generate an aggregated shadow detection machine learning model (Paragraph 34, "...the operations may further include detecting... the presence of a shadow in the plurality of frames...", Paragraph 128, "...by combining the two or more other artificial neural networks into a single artificial neural network.", Paragraph 252, "The user rating may include a score, a percentage, or any other metric indicative of the user's actual positioning and/or movement of the mobile device relative to the desired positions and/or directions indicated by the mobile device", Paragraph 254, “In some examples, a machine learning model may be trained using training examples to detect presence of shadows in images and/or videos.”, Paragraph 7, “the field of wound care could greatly benefit from new and improved systems and methods implementing real-time overlays on video feeds of mobile devices would provide great benefits in the field of wound care.”
Adiri discloses a machine learning model trained to detect the presence of a shadow in images and/or videos, and further discloses combining two or more neural networks into a single artificial neural network, which corresponds to the claimed shadow detection machine learning models and aggregating them to generate an aggregated shadow detection machine learning model.) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Duan in view of Towal and in further view of Wang's teaching with Adiri's shadow detection machine learning model and its combination of networks into a single aggregated network in order to enhance the accuracy of the model (Paragraphs 34 and 254 of Adiri).
Regarding claim 19,
Duan teaches the at least one receiving client is a smartphone or a tablet. (Page 1 Introduction, “In practice, a FL server first distributes the global model to a random subset of participating clients (e.g. mobile and IoT devices)”).
Claims 4 and 6 are rejected under 35 U.S.C. 103 as being unpatentable over Duan (“FedGroup: Efficient Federated Learning via Decomposed Similarity-Based Clustering”, 2021), in view of Towal (US 11334789 B2), in view of Wang ("REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets", 2020), in view of Adiri (US 20220217287 A1) and in further view of Adnan (“Federated learning and differential privacy for medical image analysis”, February 04, 2022).
Regarding claim 4,
Duan teaches assigning the cluster identity to the dataset (Pg 228 Para 3 In Introduction, “which clusters clients into multiple groups based on a new decomposed data-driven measure between their parameter updates.”).
Duan in view of Towal, Wang, and in further view of Adiri does not teach calculating a vector of softmax probabilities.
Adnan, in the same field of endeavor, teaches comprises calculating an vector of softmax probabilities or extracting an embedding vector from a classification model (“A higher attention value represents a higher “importance” of the corresponding element of the input sequence. Essentially, it captures the relationships among different elements of the input… The memory vectors are stacked to form a matrix Ui = [ui0, ..., uin] of shape (n d). The relative degree of correlations among the memory vectors are computed using cross-correlation followed by a column-wise softmax and then taking a row-wise average”, See Equations 3 and 4
Adnan computes a column-wise softmax over stacked memory vectors extracted from a model.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Duan in view of Towal Wang and in further view of Adiri's clustered federated learning with Adnan's use of softmax vector representations in order to enhance the accuracy and precision of client clustering for image analysis (Page 1 Paragraph 2 of Adnan).
Regarding claim 6,
Duan teaches for each of the N clients: applying a [function] to a [value] weighted model weight (See Equation 1 in Section II Background, “Where N is the number of clients”, Pg. 234 Section A, “Synthetic. It’s a synthetic federated dataset proposed by Shamir et. al [36]. Our hyperparameter settings of this data-generated algorithm are α = 1, β = 1, which control the statistical heterogeneity among clients. We study a MCLR model based on it”, Page 3 Section II B, “…right y-axis is the arithmetic mean of norm difference between the clients’ model weights and the latest global model weights.”):
Duan in view of Towal, Wang, and in further view of Adiri does not teach applying a differential privacy function.
Adnan, in the same field of endeavor, teaches applying a differential privacy function (Pg. 2 Under Differential privacy, “The goal of private data analysis is to extract as much useful information as possible while consuming the least privacy. To formalize this concept, consider a database D, which is simply a set of datapoints, and a probabilistic function M acting on databases, called a mechanism. The mechanism is said to be (ε, δ)-differentially private if for all subsets of possible outputs, and for all pairs of databases D and D′ that differ by one element”, See Equation 2
Adnan applies a differentially private mechanism to data shared during federated learning.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Duan in view of Towal, Wang, and in further view of Adiri's diversity-score-weighted model weight with Adnan's differential privacy mechanism in order to limit data leakage during aggregation while maintaining model accuracy (Page 1 Paragraph 2 of Adnan).
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Duan (“FedGroup: Efficient Federated Learning via Decomposed Similarity-Based Clustering”, 2021), in view of Towal (US 11334789 B2), in view of Wang ("REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets", 2020), in view of Adiri (US 20220217287 A1) and in further view of Yoo (US 11934555 B2).
Regarding claim 11,
Duan does not teach the dataset comprising a mask.
Yoo, in the same field of endeavor, teaches the dataset comprises a mask (Col. 5 Lines 40-44 of Yoo, “The collaborator 131 masks residual anatomy which anonymizes the patient data. The sample data is transmitted to a curation server 141 that is configured to perform data curation of the masked data.”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Duan in view of Towal in view of Wang and in further view of Adiri's teaching with Yoo's dataset comprising a mask in order to improve privacy and ensure that sensitive information remains obscured when sharing model parameters in a federated learning environment (Col. 1 Lines 31-42 of Yoo).
Claims 13 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Duan (“FedGroup: Efficient Federated Learning via Decomposed Similarity-Based Clustering”, 2021), in view of Towal (US 11334789 B2), in view of Wang ("REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets", 2020), in view of Adiri (US 20220217287 A1) and in further view of Kolouri (US 12468949 B2).
Regarding claim 13,
Duan in view of Towal, Wang, and in further view of Adiri does not teach a trigger condition in determining whether a data sample is added to the dataset if the confidence score is above a threshold.
Kolouri, in the same field of endeavor, teaches in response to a trigger condition the method further comprises, for each data sample: determining a confidence score of that data sample: adding the data sample to the dataset if the confidence score is above a threshold; and discarding the data sample if the confidence score is below the threshold (Col. 10 Lines 37-38 of Kolouri, “where each sample x.sub.i can be labeled with one or more of K possible categories or classifications”, Col. 19 Lines 31-33, “In operation 1024, if the confidence of a prediction exceeds a threshold, that example is added to a current pseudo-labeled dataset.”
Kolouri determines a prediction confidence for each sample and adds the sample to the dataset when the confidence exceeds a threshold, and otherwise does not add it, which corresponds to adding the data sample if the confidence score is above a threshold and discarding it if below.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Duan in view of Towal in view of Wang and in further view of Adiri's teaching with Kolouri's confidence-threshold trigger condition in order to improve the quality of the dataset by selectively incorporating high-confidence samples (Col. 1 Lines 21-31 of Kolouri).
Regarding claim 15,
Duan in view of Towal, Wang, and in further view of Adiri does not teach a confidence score comprising a softmax probability.
Kolouri, in the same field of endeavor, teaches determining the confidence score of the data sample further comprises determining a softmax probability for the data sample (Col. 19 Lines 7-9, “The classifier is a softmax layer, and a decoder, with an inverse structure of the feature extractor, completes an autoencoder.”, Col. 19 Lines 31-33, “In operation 1024, if the confidence of a prediction exceeds a threshold, that example is added to a current pseudo-labeled dataset.”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Duan in view of Towal in view of Wang and in further view of Adiri's teaching with Kolouri's softmax-based confidence score in order to provide a standardized, interpretable measure of prediction confidence (Paragraph 2 of Kolouri).
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Duan (“FedGroup: Efficient Federated Learning via Decomposed Similarity-Based Clustering”, 2021), in view of Towal (US 11334789 B2), in view of Wang ("REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets", 2020), in view of Adiri (US 20220217287 A1) , in view of Kolouri (US 12468949 B2), in view of Price (US 20200380285 A1), and in further view of Hua (CN-112836340-A).
Regarding claim 14,
Duan in view of Towal, in view of Wang, and in further view of Adiri does not teach a trigger condition based on the first and second confidence scores to determine whether a data sample is added to the dataset.
Kolouri, in the same field of endeavor, teaches for each data sample and based on determining… adding that data sample to the dataset… based on determining… discarding that data sample (Col. 10 Lines 37-38, “where each sample x.sub.i can be labeled with one or more of K possible categories or classifications”, Col. 19 Lines 31-33, “In operation 1024, if the confidence of a prediction exceeds a threshold, that example is added to a current pseudo-labeled dataset.”
Kolouri adds a sample to the dataset when the confidence exceeds a threshold and otherwise discards it, which corresponds to the claimed adding and discarding of the data sample based on the confidence determinations.)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Duan in view of Towal, in view of Wang, and in further view of Adiri’s teaching with Kolouri’s threshold trigger condition of adding a data sample to the dataset in order to improve the quality of the dataset (Paragraph 2 of Kolouri).
Duan in view of Towal, in view of Wang, in view of Adiri, and in further view of Kolouri does not teach in response to a trigger condition, the method further comprises… determining a first confidence score for that data sample; augmenting that data sample to generate an augmented data sample; determining a second confidence score for the augmented data sample.
Price, in the same field of endeavor, in response to a trigger condition, the method further comprises… determining a first confidence score for that data sample; augmenting that data sample to generate an augmented data sample; determining a second confidence score for the augmented data sample (Paragraph 28, “The image may be analyzed and first predicted identity and first confidence values may be determined. The first confidence value may be compared to a predetermined threshold. The processors may further select a processing technique for modifying the image and further analyze the modified image determining a second predicted identity of the vehicle. A second confidence value may be determined, and the system may further compare the second confidence value to the predetermined threshold to select the first or second predicted identity for transmission to a user.”, Paragraph 78, “At step 722, when the first identification confidence is below the quality threshold, outside the threshold range, or system 100 otherwise determines that a given identification is insufficiently strong (e.g., based on an analysis of one or more confidence index value distributions as described elsewhere herein, system 100 may modify an attribute of the image with a preprocessing augmentation tool, e.g., preprocessing augmentation tool 319”
Price determines a first confidence for an image, augments the image, and determines a second confidence for the augmented image, which corresponds to determining a first confidence score, augmenting the data sample, and determining a second confidence score for the augmented data sample. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Duan in view of Towal in view of Wang in view of Adiri in further view of Kolouri's teaching with Price's dual-confidence augmentation mechanism in order to improve the reliability of sample selection and produce a higher-quality dataset for more accurate predictions (Paragraph 1 of Price).
Duan in view of Towal, in view of Wang, in view of Adiri, in view of Kolouri, and in further view of Price does not teach a confidence score exceeding a second threshold.
Hua, in the same field of endeavor, teaches based on determining the first confidence score is above a second threshold (6th to Last Paragraph of Page 3 of Hua, “the confidence interval is greater than the second threshold value”),
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Duan in view of Towal, in view of Wang, in view of Adiri, in view of Kolouri, and in further view of Price’s teaching with Hua's secondary threshold in order to distinguish acceptable samples from reliable samples and improve the precision of sample selection (Page 2 Paragraph 2 of the Background in Hua).
Conclusion
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MAJD MAHER HADDAD whose telephone number is (571)272-2265. The examiner can normally be reached Mon-Friday 8-5 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kamran Afshar, can be reached at (571) 272-7796. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/M.M.H./Examiner, Art Unit 2125
/KAMRAN AFSHAR/Supervisory Patent Examiner, Art Unit 2125
Prosecution Insights