DETAILED ACTION
This action is in response to the submission filed 06 December 2023 for application 18/531,097. Currently claims 1-12 are pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55. However, no translation has been provided. Hence, Applicant has not perfected the right of priority.
Information Disclosure Statement
Information disclosure statements (IDS) were submitted on 06 December 2023, 20 June 2025, and 15 January 2026. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
Claim 9 is being interpreted under 35 U.S.C. 112(f) because of the language “a clustering device, configured for performing clustering” on Page 31 (line 12), “balanced sampling device, configured for calculating sub-clusters” on Page 31 (line 15), and “a multimedia recognition training device, configured for performing the incremental learning” on Page 31 (line 18). However, there is structure found at least in the specification (Page 11) and in Figure 12 and therefore no 35 U.S.C. 112(a) or 35 U.S.C. 112(b) is made.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 2 and 3 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 2 and 3 recite the limitation "…the step…" in line 1. There is insufficient antecedent basis for this limitation in the claim.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-12 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claims 1-8:
According to the first step (Step 1) of the 101 analysis, claims 1-8 are directed to a method for incremental learning of a multimedia recognition model, the method comprising certain steps (process) and falls within one of the four statutory categories (i.e., process, machine, manufacture, or composition of matter).
Regarding claim 1:
In the next step (Step 2A, prong 1) of the analysis, the limitations of:
performing clustering according to a plurality of features of a plurality of input multimedia objects and a plurality of multimedia features in a present dataset to generate clustered samples;
calculating sub-clusters of each cluster in a plurality of clusters in the clustered samples;
performing balanced sampling on each of the sub-clusters to generate balanced samples;
and wherein the dynamic margin is determined according to a number of samples of each of the clusters and a number of samples of each of the sub-clusters.
Under the broadest reasonable interpretation, the above limitations are process steps that cover mental processes including an observation, evaluation, judgment or opinion that could be performed in the mind or with the aid of pencil and paper but for the recitation of a generic computer component. If a claim, under its broadest reasonable interpretation, covers a mental process but for the recitation of generic computer components, then it falls within the “Mental Process” grouping of abstract ideas.
In the next step (Step 2A, prong 2) of the analysis, the limitation, and performing the incremental learning of the multimedia recognition model according to the balanced samples, wherein a loss function used in the incremental learning comprises a dynamic margin between the clusters, is considered to be an additional element and it does not integrate the abstract idea into a practical application because the additional element is recited so generically (no details whatsoever are provided other than that it is a method that performs the incremental learning of the multimedia recognition model according to the balanced samples, wherein a loss function used in the incremental learning comprises a dynamic margin between the clusters) that it represents no more than mere instructions to apply the judicial exception on a computer. As discussed in MPEP 2106.05(f), mere instructions to implement an abstract idea on a computer as a tool to perform an abstract idea is not indicative of integration into a practical application.
In the last step (Step 2B) of the analysis, the additional element does not amount to significantly more than the judicial exceptions. As explained with respect to Step 2A Prong Two, the method that performs the incremental learning of the multimedia recognition model according to the balanced samples, wherein a loss function used in the incremental learning comprises a dynamic margin between the clusters, is at best the equivalent of merely adding the words “apply it” to the judicial exception. See MPEP 2106.05(f). Even when considered in combination, mere instructions to apply an exception cannot provide an inventive concept and does not amount to significantly more than the judicial exception. The claim is not patent eligible.
Regarding claim 2:
In the next step (Step 2A, prong 1) of the analysis, the limitations of:
wherein the step of performing the clustering according to the features comprises: among the features of the input multimedia objects, excluding a portion of the features whose interval distances are less than a corresponding threshold;
performing clustering on the features of the input multimedia objects that are not excluded to generate a plurality of input clusters;
and merging the input clusters and a plurality of present clusters formed by the multimedia features of the present dataset, wherein the clusters in the clustered samples are a result of the merging of the input clusters and the present clusters.
Under the broadest reasonable interpretation, the above limitations are process steps that cover mental processes including an observation, evaluation, judgment or opinion that could be performed in the mind or with the aid of pencil and paper but for the recitation of a generic computer component. If a claim, under its broadest reasonable interpretation, covers a mental process but for the recitation of generic computer components, then it falls within the “Mental Process” grouping of abstract ideas.
In the next step (Step 2A, prong 2) of the analysis, it does not integrate into a practical application because it does not add any additional elements that integrate the abstract idea into practical application.
In the last step (Step 2B) of the analysis, it does not add any additional elements that amount to significantly more than the abstract idea and thus fails to add an inventive concept. The claim is not patent eligible.
Regarding claim 3:
In the next step (Step 2A, prong 1) of the analysis, the limitation of:
wherein the step of merging the input clusters and the present clusters comprises: merging at least two clusters in the input clusters and the present clusters whose interval distance is less than a corresponding threshold, wherein the interval distance is a distance between centroids of the at least two clusters or is calculated according to distances between samples of the at least two clusters.
Under the broadest reasonable interpretation, the above limitation is a process step that covers mental processes including an observation, evaluation, judgment or opinion that could be performed in the mind or with the aid of pencil and paper but for the recitation of a generic computer component. If a claim, under its broadest reasonable interpretation, covers a mental process but for the recitation of generic computer components, then it falls within the “Mental Process” grouping of abstract ideas.
In the next step (Step 2A, prong 2) of the analysis, it does not integrate into a practical application because it does not add any additional elements that integrate the abstract idea into practical application.
In the last step (Step 2B) of the analysis, it does not add any additional elements that amount to significantly more than the abstract idea and thus fails to add an inventive concept. The claim is not patent eligible.
Regarding claim 4:
In the next step (Step 2A, prong 1) of the analysis, the limitation of:
wherein the sub-clusters of each of the clusters are obtained by performing clustering on the features of each of the clusters.
Under the broadest reasonable interpretation, the above limitation is a process step that covers mental processes including an observation, evaluation, judgment or opinion that could be performed in the mind or with the aid of pencil and paper but for the recitation of a generic computer component. If a claim, under its broadest reasonable interpretation, covers a mental process but for the recitation of generic computer components, then it falls within the “Mental Process” grouping of abstract ideas.
In the next step (Step 2A, prong 2) of the analysis, it does not integrate into a practical application because it does not add any additional elements that integrate the abstract idea into practical application.
In the last step (Step 2B) of the analysis, it does not add any additional elements that amount to significantly more than the abstract idea and thus fails to add an inventive concept. The claim is not patent eligible.
Regarding claim 5:
In the next step (Step 2A, prong 1) of the analysis, the limitation of:
wherein the clusters in the clustered samples are clusters selected from all clusters in the clustered samples by balanced sampling with a same probability for each cluster in the clustered samples.
Under the broadest reasonable interpretation, the above limitation is a process step that covers mental processes including an observation, evaluation, judgment or opinion that could be performed in the mind or with the aid of pencil and paper but for the recitation of a generic computer component. If a claim, under its broadest reasonable interpretation, covers a mental process but for the recitation of generic computer components, then it falls within the “Mental Process” grouping of abstract ideas.
In the next step (Step 2A, prong 2) of the analysis, it does not integrate into a practical application because it does not add any additional elements that integrate the abstract idea into practical application.
In the last step (Step 2B) of the analysis, it does not add any additional elements that amount to significantly more than the abstract idea and thus fails to add an inventive concept. The claim is not patent eligible.
Regarding claim 6:
In the next step (Step 2A, prong 1) of the analysis, the limitation of:
wherein the balanced samples comprise features selected from the features in each of the sub-clusters by balanced sampling with a same probability for each of the sub-clusters.
Under the broadest reasonable interpretation, the above limitation is a process step that covers mental processes including an observation, evaluation, judgment or opinion that could be performed in the mind or with the aid of pencil and paper but for the recitation of a generic computer component. If a claim, under its broadest reasonable interpretation, covers a mental process but for the recitation of generic computer components, then it falls within the “Mental Process” grouping of abstract ideas.
In the next step (Step 2A, prong 2) of the analysis, it does not integrate into a practical application because it does not add any additional elements that integrate the abstract idea into practical application.
In the last step (Step 2B) of the analysis, it does not add any additional elements that amount to significantly more than the abstract idea and thus fails to add an inventive concept. The claim is not patent eligible.
Regarding claim 7:
In the next step (Step 2A, prong 2) of the analysis, the limitations of:
wherein the incremental learning of the multimedia recognition model comprises: using the multimedia recognition model to calculate a forward propagation result of the balanced samples;
using the loss function to calculate a training loss of the forward propagation result;
performing backward propagation on the training loss to obtain a gradient result;
and using the gradient result to optimize weights of the multimedia recognition model
are considered to be additional elements and it does not integrate the abstract idea into a practical application because the additional element is recited so generically (no details whatsoever are provided other than that it is a method wherein the incremental learning of the multimedia recognition model comprises: using the multimedia recognition model to calculate a forward propagation result of the balanced samples; using the loss function to calculate a training loss of the forward propagation result; performing backward propagation on the training loss to obtain a gradient result; and using the gradient result to optimize weights of the multimedia recognition model) that it represents no more than mere instructions to apply the judicial exception on a computer. As discussed in MPEP 2106.05(f), mere instructions to implement an abstract idea on a computer as a tool to perform an abstract idea is not indicative of integration into a practical application.
In the last step (Step 2B) of the analysis, the additional element does not amount to significantly more than the judicial exceptions. As explained with respect to Step 2A Prong Two, the method wherein the incremental learning of the multimedia recognition model comprises: using the multimedia recognition model to calculate a forward propagation result of the balanced samples; using the loss function to calculate a training loss of the forward propagation result; performing backward propagation on the training loss to obtain a gradient result; and using the gradient result to optimize weights of the multimedia recognition model, is at best the equivalent of merely adding the words “apply it” to the judicial exception. See MPEP 2106.05(f). Even when considered in combination, mere instructions to apply an exception cannot provide an inventive concept and does not amount to significantly more than the judicial exception. The claim is not patent eligible.
Regarding claim 8:
In the next step (Step 2A, prong 1) of the analysis, the limitation of:
wherein the dynamic margin comprises a product of a first value and a second value, wherein the first value decreases as a number of samples of the cluster corresponding to the loss function increases, and wherein the second value decreases as a number of samples of the sub-cluster corresponding to the loss function increases.
Under the broadest reasonable interpretation, the above limitation is a process step that covers mental processes including an observation, evaluation, judgment or opinion that could be performed in the mind or with the aid of pencil and paper but for the recitation of a generic computer component. If a claim, under its broadest reasonable interpretation, covers a mental process but for the recitation of generic computer components, then it falls within the “Mental Process” grouping of abstract ideas.
In the next step (Step 2A, prong 2) of the analysis, it does not integrate into a practical application because it does not add any additional elements that integrate the abstract idea into practical application.
In the last step (Step 2B) of the analysis, it does not add any additional elements that amount to significantly more than the abstract idea and thus fails to add an inventive concept. The claim is not patent eligible.
Regarding claims 9-11:
According to the first step (Step 1) of the 101 analysis, claims 9-11 are directed to a system (manufacture) and falls within one of the four statutory categories (i.e., process, machine, manufacture, or composition of matter).
Regarding claim 9:
In step (Step 2A, prong 2) of the analysis, the limitation of:
A system for incremental learning of a multimedia recognition model, the system comprising: a clustering device, a balanced sampling device, and a multimedia recognition training device configured for performing steps for incremental learning, is considered to be an additional element and it does not integrate the abstract idea into a practical application because the additional element is recited so generically (no details whatsoever are provided other than that it is a system for incremental learning of a multimedia recognition model, the system comprising: a clustering device, a balanced sampling device, and a multimedia recognition training device configured for performing steps for incremental learning) that it represents no more than mere instructions to apply the judicial exception on a computer. As discussed in MPEP 2106.05(f), mere instructions to implement an abstract idea on a computer as a tool to perform an abstract idea is not indicative of integration into a practical application.
The rest of the limitations of claim 9 are substantially similar to claim 1 and therefore is rejected on similar grounds as claim 1 as explained above.
Regarding claim 10:
Claim 10 is substantially similar to claim 2 and therefore is rejected on similar grounds as claim 2.
Regarding claim 11:
Claim 11 is substantially similar to claim 7 and therefore is rejected on similar grounds as claim 7.
Regarding claim 12:
According to the first step (Step 1) of the 101 analysis, claims 12 is directed to a A non-transitory computer readable storage medium, storing instructions therein, to execute a method for incremental learning of a multimedia recognition model (manufacture) and falls within one of the four statutory categories (i.e., process, machine, manufacture, or composition of matter).
In step (Step 2A, prong 2) of the analysis, the limitation of:
A non-transitory computer readable storage medium, storing instructions therein, to execute a method for incremental learning of a multimedia recognition model, the method comprising some steps, is considered to be an additional element and it does not integrate the abstract idea into a practical application because the additional element is recited so generically (no details whatsoever are provided other than that it is a non-transitory computer readable storage medium, storing instructions therein, to execute a method for incremental learning of a multimedia recognition model, the method comprising steps for incremental learning) that it represents no more than mere instructions to apply the judicial exception on a computer. As discussed in MPEP 2106.05(f), mere instructions to implement an abstract idea on a computer as a tool to perform an abstract idea is not indicative of integration into a practical application.
The rest of the limitations of claim 12 are substantially similar to claim 1 and therefore is rejected on similar grounds as claim 1 as explained above.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-6, 9, 10, and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Hirsch et al (US 10341701 B2) in view of Zhang et al (A Survey on Class Imbalance in Federated Learning, 2023).
Regarding claim 1:
Hirsch teaches: A method for incremental learning of a multimedia recognition model, the method comprising ([Column 32, Lines 9-10] The system processor 128 can provide for incremental learning):
performing clustering according to a plurality of features of a plurality of input multimedia objects and a plurality of multimedia features in a present dataset to generate clustered samples ([Abstract] The system processor can formulate an input dataset from the retrieved data, perform nonlinear clustering on the input dataset to formulate subscriber and multimedia content clusters having similarities between elements therein, and determine the recommendation of multimedia content based on a metric distance between vector elements of the formulated subscriber and multimedia content clusters and the metric distance crossing a threshold. [Column 39, Lines 1-4] Multimedia content text features (i.e., metadata) associated with multimedia content can include those drawn from bulk text in online data bases, published reviews, wiki's, blogs, social media content, etc. [Column 40, Lines 9-16] A text feature vector can be a multidimensional vector of numeric features that represent a text element or group of text elements, for example, a collection of attributes and/or groups of attributes such as soccer moms, millennials, etc. For multimedia content, a text feature vector can include non-zero text elements comprised of, for example, genre, title, actors, director, producer/production company, synopsis, screenplay, language, etc. [Column 44, Lines 27-30] The lower dimensional vector of synthesized text features can be used in the formation of the linear clustering, nonlinear clustering and/or nonlinear manifold clustering. [Column 45, Lines 10-13] o make use of text features in the clustering performed by the clustering, visualization & data seeding modules 1312 and 1322, the text features can be coded in numeric form);
calculating sub-clusters of each cluster in a plurality of clusters in the clustered samples ([Column 54, Lines 3-16] However, the system processor 128 can determine a metric, between the centroids, as an indicator of the closeness between clusters and the affinity between those clusters. The system processor 128 can add and/or subtract cluster members without the re-computation of the linear, nonlinear, and nonlinear manifold clusters. The system processor 128 can use the metric, centroids and metric variances, to find the affinity between members of a cluster or between different groups and subsets of clusters. The system processor 128 can also partition a cluster into sub-clusters. The system processor 128 can determine the recommendation of multimedia content based on such affinity between members of a cluster or between different groups of clusters);
and performing the incremental learning of the multimedia recognition model ([Column 32, Lines 9-10] The system processor 128 can provide for incremental learning).
However, Hirsch does not explicitly disclose: performing balanced sampling on each of the sub-clusters to generate balanced samples; according to the balanced samples, wherein a loss function used in the incremental learning comprises a dynamic margin between the clusters, and wherein the dynamic margin is determined according to a number of samples of each of the clusters and a number of samples of each of the sub-clusters.
Zhang teaches, in an analogous system: performing balanced sampling on each of the sub-clusters to generate balanced samples ([Page 9, Column 1, Paragraph 4] Hybrid sampling: generating a balanced dataset via the combination of over-sampling and the undersampling. [Page 12, Column 2, Paragraph 2] Weighted per-cluster model aggregation is performed on the server side. Each cluster is assigned with a different weight to balance the contribution of each class of global training data. In addition, the weight of each cluster is optimized through the convergence rate analysis);
according to the balanced samples, wherein a loss function used in the incremental learning comprises a dynamic margin between the clusters, and wherein the dynamic margin is determined according to a number of samples of each of the clusters and a number of samples of each of the sub-clusters ([Page 6, Column 2, Last but one Paragraph] For dataset with high data imbalance, the minority class usually have much higher loss than the majority class. In [75], the value of loss function of one client is considered to reflect the usefulness of the client’s dataset during a training round. Therefore, the clients that are considered more useful (i.e., having more samples belong to the global minority class) will be more likely to be selected in the subsequent training rounds in order to handle the issue of class imbalance. [Page 9, Column 1, Paragraph 4] Hybrid sampling: generating a balanced dataset via the combination of over-sampling and the undersampling. [Page 12, Column 1, Paragraph 1] As shown in Fig. 7, the method encourages a classification model to have a larger margin between a minority class and other classes. Taking dynamic margin softmax as an example, it adds margin for minority class. [Page 12, Column 1, Paragraph 2] It encourages each participating client’s local classification model to have large margins for minority classes, a dynamic term is skillfully added to the loss function which guide the model to learn more separable features from data. [Page 12, Column 2, Paragraph 2] Weighted per-cluster model aggregation is performed on the server side. Each cluster is assigned with a different weight to balance the contribution of each class of global training data. In addition, the weight of each cluster is optimized through the convergence rate analysis).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Hirsch to incorporate the teachings of Zhang to use performing balanced sampling on each of the sub-clusters to generate balanced samples; according to the balanced samples, wherein a loss function used in the incremental learning comprises a dynamic margin between the clusters, and wherein the dynamic margin is determined according to a number of samples of each of the clusters and a number of samples of each of the sub-clusters. One would have been motivated to do this modification because doing so would give the benefit of guiding the model to learn more separable features from data as taught by Zhang [Page 12, Column 1, Paragraph 2].
Regarding claim 2:
The system of Hirsch and Zhang teaches: The method according to claim 1 (as shown above).
Hirsch further teaches: wherein the step of performing the clustering according to the features comprises: among the features of the input multimedia objects, excluding a portion of the features whose interval distances are less than a corresponding threshold ([Column 55, Lines 6-20] Multiple recommendation thresholds can be set based on the inverse distance (or inverse metric) and/or the distance (or metric) between a subscriber 105 (or subscriber cluster centroid) and particular content (multimedia content or advertising content) or cluster of content. Using the inverse metric, closeness or affinity can be defined if the inverse metric is greater than or equal to a threshold. In another example, using the metric, closeness or affinity can be defined if the metric is less than a threshold. The example embodiment described hereafter uses the inverse metric exceeding (or crossing) a threshold floor as a basis for determining a recommendation. Likewise, another equivalent embodiment can use the metric not exceeding (or crossing) a threshold ceiling as a basis for determining a recommendation);
performing clustering on the features of the input multimedia objects that are not excluded to generate a plurality of input clusters ([Column 55, Lines 21-23] The recommendation thresholds can change as subscribers 105 and multimedia content are added to and/or deleted from subscriber clusters and/or multimedia clusters);
and merging the input clusters and a plurality of present clusters formed by the multimedia features of the present dataset, wherein the clusters in the clustered samples are a result of the merging of the input clusters and the present clusters ([Column 37, Lines 8 and 9] Data subsets can be merged to create new subsets).
Regarding claim 3:
The system of Hirsch and Zhang teaches: The method according to claim 2 (as shown above).
Hirsch further teaches: wherein the step of merging the input clusters and the present clusters comprises: merging at least two clusters in the input clusters and the present clusters whose interval distance is less than a corresponding threshold, wherein the interval distance is a distance between centroids of the at least two clusters or is calculated according to distances between samples of the at least two clusters ([Column 53, Lines 52-67] The mean of a cluster of subscribers 105, multimedia content, advertising content or an aggregate and/or subset thereof can be represented by a centroid of that cluster in the N-dimensional space. For example, the centroid of a cluster of subscribers can be used to define a representative subscriber for that cluster. Likewise, a representative multimedia content, representative advertising, and/or representative aggregates of any combination of subscribers, multimedia content, and advertising can be defined. The system processor 128 can determine variances of a cluster of subscribers 105, a cluster of multimedia content, a cluster of advertising content, and/or a subset and/or an aggregate thereof using a metric of each member of the cluster relative to the centroid of that cluster. The system processor can use these variances to estimate the similarities between members of a given cluster and determine cluster membership and cluster [Column 54, Lines 1- 32] boundaries. The variances can be used to weight the feature components or vector components, as well. A centroid is a concept and not actually a member of that cluster. However, the system processor 128 can determine a metric, between the centroids, as an indicator of the closeness between clusters and the affinity between those clusters. The system processor 128 can add and/or subtract cluster members without the re-computation of the linear, nonlinear, and nonlinear manifold clusters. The system processor 128 can use the metric, centroids and metric variances, to find the affinity between members of a cluster or between different groups and subsets of clusters. [Column 55, Lines 6-20] Multiple recommendation thresholds can be set based on the inverse distance (or inverse metric) and/or the distance (or metric) between a subscriber 105 (or subscriber cluster centroid) and particular content (multimedia content or advertising content) or cluster of content. Using the inverse metric, closeness or affinity can be defined if the inverse metric is greater than or equal to a threshold. In another example, using the metric, closeness or affinity can be defined if the metric is less than a threshold. The example embodiment described hereafter uses the inverse metric exceeding (or crossing) a threshold floor as a basis for determining a recommendation. Likewise, another equivalent embodiment can use the metric not exceeding (or crossing) a threshold ceiling as a basis for determining a recommendation).
Regarding claim 4:
The system of Hirsch and Zhang teaches: The method according to claim 1 (as shown above).
Hirsch further teaches: wherein the sub-clusters of each of the clusters are obtained by performing clustering on the features of each of the clusters ([Column 53, Lines 65-67] The system processor can use these variances to estimate the similarities between members of a given cluster and determine cluster membership and cluster [Column 54, Lines 1-13] boundaries. The variances can be used to weight the feature components or vector components, as well. A centroid is a concept and not actually a member of that cluster. However, the system processor 128 can determine a metric, between the centroids, as an indicator of the closeness between clusters and the affinity between those clusters. The system processor 128 can add and/or subtract cluster members without the re-computation of the linear, nonlinear, and nonlinear manifold clusters. The system processor 128 can use the metric, centroids and metric variances, to find the affinity between members of a cluster or between different groups and subsets of clusters. The system processor 128 can also partition a cluster into sub-clusters).
Regarding claim 5:
The system of Hirsch and Zhang teaches: The method according to claim 1 (as shown above).
However Hirsch is not relied upon to teach: wherein the clusters in the clustered samples are clusters selected from all clusters in the clustered samples by balanced sampling with a same probability for each cluster in the clustered samples.
Zhang teaches, in an analogous system: wherein the clusters in the clustered samples are clusters selected from all clusters in the clustered samples by balanced sampling with a same probability for each cluster in the clustered samples ([Page 7, Column 1, Section 3.3, Paragraph 1] Based on the clustering results, the server can then use different strategies for different clusters in order to alleviate data balance issue, such as generating a personalized model for each cluster, data sampling and client selection, which aim to address global data imbalance. [Page 9, Column 1, Paragraph 4] Hybrid sampling: generating a balanced dataset via the combination of over-sampling and the undersampling. [Page 9, Column 2, Paragraph 3] In the early training rounds, in order to converge faster, it makes all clients to have more similar label sampling probabilities. In later rounds, to allow each client to better learn the special knowledge from its local datasets, it makes all clients to use their the original label sampling probability, which is a similar to local sampling).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Hirsch to incorporate the teachings of Zhang wherein the clusters in the clustered samples are clusters selected from all clusters in the clustered samples by balanced sampling with a same probability for each cluster in the clustered samples. One would have been motivated to do this modification because doing so would give the benefit of using different strategies for different clusters in order to alleviate data balance issue, such as generating a personalized model for each cluster as taught by Zhang [Page 7, Column 1, Section 3.3, Paragraph 1].
Regarding claim 6:
The system of Hirsch and Zhang teaches: The method according to claim 1 (as shown above).
However Hirsch is not relied upon to teach: wherein the balanced samples comprise features selected from the features in each of the sub-clusters by balanced sampling with a same probability for each of the sub-clusters.
Zhang further teaches, in an analogous system: wherein the balanced samples comprise features selected from the features in each of the sub-clusters by balanced sampling with a same probability for each of the sub-clusters ([Page 7, Column 2, Paragraph 2] Every client computes a feature vector, using weight updates and the label-wise gradients of the global model and sends it to the server. CIC-FL employs a top-down hierarchical clustering process. Then, the server iteratively conducts bi-partitioning to partition these clients into two clusters).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Hirsch to incorporate the teachings of Zhang wherein the balanced samples comprise features selected from the features in each of the sub-clusters by balanced sampling with a same probability for each of the sub-clusters. One would have been motivated to do this modification because doing so would give the benefit of using different strategies for different clusters in order to alleviate data balance issue, such as generating a personalized model for each cluster as taught by Zhang [Page 7, Column 1, Section 3.3, Paragraph 1].
Regarding claim 9:
Claim 9 is substantially similar to claim 1 and therefore is rejected on similar grounds as claim 1.
Regarding claim 10:
Claim 10 is substantially similar to claim 12and therefore is rejected on similar grounds as claim 2.
Regarding claim 12:
Claim 12 is substantially similar to claim 1 and therefore is rejected on similar grounds as claim 1.
Claims 7 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Hirsch et al (US 10341701 B2) in view of Zhang et al (A Survey on Class Imbalance in Federated Learning, 2023) and further in view of Li et al (Camel: Managing Data for Efficient Stream Learning, 2022).
Regarding claim 7:
The system of Hirsch and Zhang teaches: The method according to claim 1 (as shown above).
However Hirsch is not relied upon to teach: wherein the incremental learning of the multimedia recognition model comprises: using the multimedia recognition model to calculate a forward propagation result of the balanced samples; using the loss function to calculate a training loss of the forward propagation result; performing backward propagation on the training loss to obtain a gradient result; and using the gradient result to optimize weights of the multimedia recognition model.
Li teaches, in an analogous system: wherein the incremental learning of the multimedia recognition model comprises: using the multimedia recognition model to calculate a forward propagation result of the balanced samples; using the loss function to calculate a training loss of the forward propagation result; performing backward propagation on the training loss to obtain a gradient result; and using the gradient result to optimize weights of the multimedia recognition model ([Page 1271, Column 2, Figure 1] Note: Shows forward propagation, backward propagation, gradient, and training loss. [Page 1273, Column 2, Paragraph 3] Then, Camel updates the model 𝜃𝑡−1 via weighted gradient descent (Eq.1). [Page 1275, Column 2, Paragraph 1] As a result, the selected subset can be extremely class-imbalanced, which will degrade the models’ generalization performance. Therefore, we separately select a subset from each class, which better exploits the intra-class data distribution. [Page 1280, Column 1, Last Paragraph] Implementation Details. We set the streaming batch size |𝐵𝑡 | = 1, 000 for the three class-incremental datasets).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combined method of Hirsch and Zhang to incorporate the teachings of Li wherein the incremental learning of the multimedia recognition model comprises: using the multimedia recognition model to calculate a forward propagation result of the balanced samples; using the loss function to calculate a training loss of the forward propagation result; performing backward propagation on the training loss to obtain a gradient result; and using the gradient result to optimize weights of the multimedia recognition model. One would have been motivated to do this modification because doing so would give the benefit of better exploiting the intra-class data distribution as taught by Li [Page 1275, Column 2, Paragraph 1].
Regarding claim 11:
Claim 11 is substantially similar to claim 7 and therefore is rejected on similar grounds as claim 7.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Hirsch et al (US 10341701 B2) in view of Zhang et al (A Survey on Class Imbalance in Federated Learning, 2023) and further in view of Ran et al (Dynamic Margin for Federated Learning with Imbalanced Data, 2021).
Regarding claim 8:
The system of Hirsch and Zhang teaches: The method according to claim 1 (as shown above).
However Hirsch is not relied upon to teach: wherein the dynamic margin comprises a product of a first value and a second value, wherein the first value decreases as a number of samples of the cluster corresponding to the loss function increases, and wherein the second value decreases as a number of samples of the sub-cluster corresponding to the loss function increases.
Ran teaches, in an analogous system: wherein the dynamic margin comprises a product of a first value and a second value, wherein the first value decreases as a number of samples of the cluster corresponding to the loss function increases, and wherein the second value decreases as a number of samples of the sub-cluster corresponding to the loss function increases ([Pages 3, Section III, Column 2, Paragraph 2] Note: Where n is the size of samples. [Page 4, Column 1, Paragraph 1] We design Dynamic Margin Softmax loss function to improve the performance in federated learning with non-IID data. Note: See Equations on Pages 3 and 4 that show products of two values and the inverse relationships with regard to n. Also, see Figure 4 that Illustrates dynamic margin).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combined method of Hirsch and Zhang to incorporate the teachings of Ran wherein the dynamic margin comprises a product of a first value and a second value, wherein the first value decreases as a number of samples of the cluster corresponding to the loss function increases, and wherein the second value decreases as a number of samples of the sub-cluster corresponding to the loss function increases. One would have been motivated to do this modification because doing so would give the benefit of improving the performance as taught by Ran [Page 4, Column 1, Paragraph 1].
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Aggarwal et al (MINORITY CLASS ORIENTED ACTIVE LEARNING FOR IMBALANCED DATASETS, 2022) discloses a new active learning method which is designed for imbalanced datasets. It favors samples likely to be in minority classes so as to reduce the imbalance of the labeled subset and create a better representation for these classes. We also compare two training schemes for active learning: (1) the one commonly deployed in deep active learning using model fine tuning for each iteration and (2) a scheme which is inspired by transfer learning and exploits generic pre-trained models and train shallow classifiers for each iteration. Evaluation is run with three imbalanced datasets. Results show that the proposed active learning method outperforms competitive baselines. Equally interesting, they also indicate that the transfer learning training scheme outperforms model fine tuning if features are transferable from the generic dataset to the unlabeled one. This last result is surprising and should encourage the community to explore the design of deep active learning methods.
He et al (Learning from Imbalanced Data, 2009) discloses a comprehensive review of the development of research in learning from imbalanced data. Our focus is to provide a critical review of the nature of the problem, the state-of-the-art technologies, and the current assessment metrics used to evaluate learning performance under the imbalanced learning scenario. Furthermore, in order to stimulate future research in this field, we also highlight the major opportunities and challenges, as well as potential important research directions for learning from imbalanced data.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHAITANYA RAMESH JAYAKUMAR whose telephone number is (571)272-3369. The examiner can normally be reached Mon-Fri 9am-1pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Omar Fernandez Rivas can be reached at (571)272-2589. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/C.R.J./Examiner, Art Unit 2128
/KYLE R STORK/Primary Examiner, Art Unit 2128
Prosecution Insights