DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 05/15/2026 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Amendment
The amendments filed 02/09/2026 have been entered.
Claims 1-3, 5, 7-10, 12, 14-18 and 20 remain pending within the application.
The amendments and remarks filed 02/09/2026 are sufficient to overcome the 101 rejections previously set forth in the Non-Final Office Action mailed 11/07/2025. The rejections have been withdrawn.
The 35 USC § 103 rejections set forth in the Non-Final Office Action mailed 11/07/2025 are updated in light of the amendments filed 02/09/2026. Applicant's amendment necessitated the updated ground(s) of rejection presented in this Office action.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 18 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 18 recites the limitation " wherein the expert hierarch has multiple expert layers including an initial expert layer and one or more augmented expert layers; … each of the one or more augmented expert layers includes at least one augmented expert.". It is unclear whether these initial and augmented expert layers are the “initial expert layers” and “augmented expert layers” recited in claim 15, or if they refer to different initial and augmented expert layers. There is insufficient antecedent basis for this limitation.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 5, 7-10, 12, 14-18 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Shazeer et al. ("OUTRAGEOUSLY LARGE NEURAL NETWORKS: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER"), hereafter Shazeer, in view of Eigen et al. ("Learning Factored Representations in a Deep Mixture of Experts"), hereafter Eigen, in further view of Shen et al. ("SAR-Net: A Scenario-Aware Ranking Network for Personalized Fair Recommendation in Hundreds of Travel Scenarios"), as disclosed in the office action mailed 03/21/2025 as prior art made of record and not relied upon, hereafter Shen.
Regarding claim 1, Shazeer discloses:
A method implemented on at least one processor, a memory, and a communication platform for integrating heterogeneous experts, comprising (Shazeer, page 4, paragraph 5, line 3 “parameters are synchronized through a set of parameter servers” and page 16, paragraph 4, lines 1-2 “Models are trained on a cluster of 32 Tesla K40 GPUs, except for the last two models, which are trained on clusters of 64 and 128 GPUs” teaches GPUs as processors and memories and servers as communication platforms),
configuring a nonlinear integration model for combining individual expert predictions from a plurality of experts of an expert hierarch (Shazeer, Figure 1 and “a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a … combination of “experts", each of which is itself a secondary mixture-of-experts with its own gating network” teaches combining predictions from a hierarchical mixture-of-experts, i.e. MoE, within a language model),
wherein the expert hierarch includes an initial expert layer and one or more augmented expert layers, wherein the initial expert layer has a plurality of initial experts and each of the one or more augmented expert layers has at least one augmented expert for prediction (Shazeer, page 14, last paragraph, lines 7-9 “For the hierarchical MoE layers, the first level branching factor was 16, corresponding to the number of GPUs in our cluster. We use Noisy-Top-K Gating … for the ordinary MoE layers and k=2 at each level of the hierarchical MoE layers” and page 14, last paragraph, lines 4-10 “We varied the number of experts between models, using … hierarchical MoE layers with 256, 1024 and 4096 experts…Thus, each example is processed by exactly 4 experts…” teaches first level layers as an initial expert layer having a plurality of initial experts and subsequent levels of the hierarchical MoE layers as one or more augmented expert layers having at least one augmented expert for prediction).
wherein each of the experts at a layer is trained based on training data directed thereto and an individual expert output from … individual expert at any lower layer of the expert hierarch so that an expert at any of the one or more augmented expert layers augments the expertise of experts at any lower layer of the expert hierarchy (Shazeer, Figure 1, page 14, paragraph 2, lines 2-4 “In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of “experts", each of which is itself a secondary mixture-of-experts with its own gating network” teaches training each of the experts at a layer based on training data and an individual expert output so that each augmented expert at an augmented expert layer augments experts at any lower expert layer in the expert hierarchy),
wherein the nonlinear integration model is characterized by a plurality of parameters and maps the individual expert predictions to … integrated expert prediction (Shazeer, Figure 1 and “a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a … combination of “experts", each of which is itself a secondary mixture-of-experts with its own gating network” teaches combining predictions from a hierarchical mixture-of-experts, i.e. MoE, within a language model, which is a nonlinear integration model characterized by a plurality of parameters and maps the individual expert predictions to integrated expert predictions, and the expert hierarchy layers exist within each MOE layer as shown in Figure 1),
learning values of the plurality of parameters based on the training data as well as outputs from all the experts in the expert hierarch, wherein learned values of the plurality of parameters create a learned nonlinear integration model (Shazeer, Figure 1 and page 3, paragraph 5, lines 1-3 “The Mixture-of-Experts (MoE) layer consists of a set of “expert networks”… whose output is a sparse n-dimensional vector. Figure 1 shows an overview of the MoE module. The experts are themselves neural networks, each with their own parameters.” and page 15, paragraph 2, lines 8-9 “For each model, we performed a hyper-parameter search” teaches learning values of parameters based on training data and outputs from the respective plurality of experts generated based on the training data, which creates the language model in Figure 1),
receiving individual expert predictions from the plurality of experts generated based on a given input (Shazeer, Figure 1 and page 3, paragraph 6, lines 1-2 “output of the i-th expert network for a given input x.” teaches receiving individual expert predictions from the plurality of experts generated based on a given input),
combining, via the learned nonlinear integration model, the individual expert predictions to derive … integrated expert prediction in response to the given input (Shazeer, Figure 1, Equation 12, and page 15, paragraph 2, line 7 “The Softmax output layer was trained efficiently” teaches combining, in Figure 1 and Equation 12, expert predictions via the learned nonlinear integration model to derive integrated expert prediction at the softmax output layer).
Shazeer discloses wherein each of the experts at a layer is trained based on training data directed thereto and an individual expert output from … individual expert at any lower layer of the expert hierarch so that an expert at any of the one or more augmented expert layers augments the expertise of experts at any lower layer of the expert hierarchy, but does not explicitly disclose training experts at a layer based on expert output from each individual expert at any lower layer.
Eigen discloses:
wherein each of the experts at a layer is trained based on … an individual expert output from each individual expert at any lower layer (Eigen, Figure 1(b), ), page 3, paragraph below Figure 1, lines 2-3 “After training with the constraint in place, we lift it and further train in a second fine-tuning phase”, page 2, section 3, paragraph 1, lines 2-3 “The final output is produced by composing the mixtures at each layer” and page 2, paragraph 1, lines 2-3 "g(x) is a distribution over experts i = 1, ... , N that sums to 1." discloses training experts at one layer using all outputs from the previous layer of experts, as the gating used here produces a distribution over all experts at each layer).
Shazeer and Eigen are analogous art because they are from the same field of endeavor, mixture of expert models and machine learning.
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Shazeer, in view of Shen, to include wherein each of the experts at a layer is trained based on … an individual expert output from each individual expert at any lower layer, based on the teachings of Eigen. The motivation for doing so would have been to be more efficient even with an increased the number of experts (Eigen, page 1 , paragraph 3, lines 2-5 “increases the number of effective experts…Thus it can be both large and efficient at the same time”).
Shazeer teaches wherein the nonlinear integration model is characterized by a plurality of parameters and maps the individual expert predictions to .. integrated expert prediction, but does not explicitly teach the integrated expert prediction to be a single prediction.
Eigen teaches:
an integrated expert prediction (Eigen, Figure 1, and page 2, section 3 Approach, paragraph 1, lines 2-3 “The final output is produced by composing the mixtures at each layer” teaches an integrated expert prediction from the mixture of experts in Figure 1).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Shazeer to include an integrated expert prediction, based on the teachings of Eigen. The motivation for doing so would have been to be more efficient even with an increased the number of experts (Eigen, page 1 , paragraph 3, lines 2-5 “increases the number of effective experts…Thus it can be both large and efficient at the same time”).
Shazeer teaches combining, via the learned nonlinear integration model, the individual expert predictions to derive … integrated expert prediction in response to the given input, but does not explicitly teach the integrated expert prediction to be a single prediction.
Eigen teaches:
an integrated expert prediction (Eigen, Figure 1, and page 3, section 4.1, paragraph 4, line 2 “concatenating the output of all experts” teaches an integrated expert prediction from the mixture of experts in Figure 1).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Shazeer to include an integrated expert prediction, based on the teachings of Eigen. The motivation for doing so would have been to be more efficient even with an increased the number of experts (Eigen, page 1 , paragraph 3, lines 2-5 “increases the number of effective experts…Thus it can be both large and efficient at the same time”).
Shazeer, in view of Eigen, discloses wherein each of the experts at a layer is trained based on training data directed thereto and an individual expert output from each individual expert at any lower layer of the expert hierarch so that an expert at any of the one or more augmented expert layers augments the expertise of experts at any lower layer of the expert hierarchy, but does not disclose wherein different expert layers … are directed with different training data, respectively.
Shen discloses:
wherein different expert layers … are directed with different training data, respectively (Shen, Figure 1, elements “Shared experts”, “Specific experts”, “vb” and “ v’ ” and page 5, right column, section 4.5, paragraph 2, lines 4-8 “Main net takes user interest transfer vector…. user basic profiles vector… target item feature vector… and scenario context feature vector … as input… Bias net receives the input of Fairness Coefficient vb” teaches different layers of expert nets to be trained with different sets of training data).
Shazeer, Eigen, and Shen are analogous art because they are from the same field of endeavor, mixture of expert models and machine learning.
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Shazeer, in view of Eigen, to include wherein different expert layers … are directed with different training data, respectively, based on the teachings of Shen. The motivation for doing so would have been to alleviate the influence of intervention bias on model prediction (Shen, page 5, right column, section 4.5, paragraph 2, lines 1-2 “…to alleviate the influence of intervention bias on model prediction…”).
Regarding claim 2, Shazeer, in view of Eigen, in further view of Shen, discloses the method of claim 1. Shazeer further discloses:
wherein the learned nonlinear integration model captures nonlinear relationships among the plurality of experts (Shazeer, Figure 1 and page 15, paragraph 2, lines 1-2 “The models were trained … using the synchronous method described” teaches capturing the nonlinear relationship among the experts using training of the method).
Regarding claim 3, Shazeer, in view of Eigen, in further view of Shen, discloses the method of claim 1. Shazeer further discloses:
wherein the nonlinear integration model is configured as an artificial neural network (ANN) with the plurality of parameters related to the ANN, including embeddings of the ANN (Shazeer, page 14, Section C.1, paragraph 1, lines 1-3 “Our model consists of five layers: a word embedding layer, a recurrent Long Short-Term Memory (LSTM) layer …, a MoE layer, a second LSTM layer, and a softmax layer.” teaches layers of an ANN to learn the plurality of parameters related to the ANN, including embeddings of the ANN).
Regarding claim 5, Shazeer, in view of Eigen, in further view of Shen, discloses the method of claim 1. Shazeer further discloses:
wherein the initial expert layer includes a plurality of heterogeneous experts (Shazeer, page 4, paragraph 5, lines 7-9 “set of devices …each hosting a subset of the experts…” teaches the initial expert layer to include a plurality of heterogenous experts from different devices),
Regarding claim 7, Shazeer, in view of Eigen, in further view of Shen, discloses the method of claim 1. Shazeer further discloses:
initializing the values of the plurality of parameters (Shazeer, page 13, last two paragraphs, lines 1-3 “initialize the network …To accomplish this, we initialize the matrices…” teaches initializing the values of the parameters),
receiving the training data … includes an input feature vector … (Shazeer, page 19, paragraph 2, lines 1-2 “To force each expert to receive the exact same number of examples … batches of input vectors…” recites receiving training data including an input feature vector),
for each of … the training data, receiving the outputs from the respective plurality of experts generated based on the input feature vector… (Shazeer, Figure 1 and page 19, paragraph 2, line 4 “each example is sent to an average of k experts” teaches receiving outputs, in Figure 1, for each of the training data from the plurality of experts),
generating an integrated output of the received outputs based on current values of the plurality of parameters of the nonlinear integration model (Shazeer, Figure 1 and page 6, paragraph 5, lines 1-6 “we trained a series of MoE models… Each expert had about 1 million parameters” teaches generating an integrated output from the MOE model based on current values of the plurality of parameters of the nonlinear integration model),
determining a loss based on a discrepancy … the integrated output…(Shazeer, page 5, last paragraph, line 3 “…overall loss function for the model…” teaches determining a loss based on a discrepancy of the integrated output from the model),
updating the current values of the plurality of parameter based on the loss (Shazeer, page 4, paragraph 4, line 2 “parameter loads and updates” and page 14, paragraph 1, lines 1-2 ”All the combinations containing at least one the two losses” teaches updating plurality of parameters based on loss),
repeating the steps of receiving, generating, determining, and updating until a convergence condition is satisfied (Shazeer, page 5, paragraph 5, line 1 “network tends to converge to a state” and page 6, paragraph 5, line 1-2 “we trained a series of MoE models” teaches training the MoE model until convergence as repeating the steps of receiving, generating, determining, and updating until a convergence condition is satisfied).
Shazeer discloses receiving training data with input feature vectors, but does not teach the training data having pairs of data, wherein each of the pairs includes an input feature vector and a corresponding ground truth label.
Eigen discloses:
the training data having pairs of data, wherein each of the pairs includes an input feature vector and a corresponding ground truth label (Eigen, page 4, paragraph 1, lines 5-6 “Each input was fed to the network as a 440-dimensional vector. There were 40 possible output phoneme classes.” teaches an input vector with phenome classes as training data having pairs of data, wherein each of the pair includes an input feature vector and a corresponding ground truth label).
Shazeer teaches for each of … the training data, receiving the outputs from the respective plurality of experts generated based on the input feature vector … but does not teach the training data to contain pairs.
Eigen discloses:
for each of the pairs in the training data, receiving the outputs … based on the input feature vector in the pair (Eigen, page 4, paragraph 1, lines 5-6 to paragraph 2, line 1 “Each input was fed to the network as a 440-dimensional vector. There were 40 possible output phoneme classes. We trained a model with 4 experts” teaches receiving the outputs based on the input feature vector in the pair for each of the pairs in the training data),
Shazeer teaches determining a loss based on a discrepancy … the integrated output…, but does not teach the discrepancy to be between the … output and the ground truth label in the pair.
Eigen discloses:
determining a loss based on a discrepancy between the … output and the ground truth label in the pair (Eigen, page 4, paragraph 1, lines 5-6 “Each input was fed to the network as a 440-dimensional vector. There were 40 possible output phoneme classes” and paragraph 3, line 1 “Table 1 shows the error on the training and test sets for each model size” teaches determining an error on the training and test sets as a loss based on a discrepancy between the output and the ground truth, i.e., output classes).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Shazeer to include: the training data having pairs of data, wherein each of the pair includes an input feature vector and a corresponding ground truth label; for each of the pairs in the training data, receiving the outputs based on the input feature vector in the pair; and determining a loss based on a discrepancy between the output and the ground truth label in the pair, based on the teachings of Eigen. The motivation for doing so would have been to be more efficient even with an increased the number of experts (Eigen, page 1 , paragraph 3, lines 2-5 “increases the number of effective experts…Thus it can be both large and efficient at the same time”).
Claims 8-10, 12 and 14 are substantially similar to claims 1-3, 5, and 7 respectively, and thus are rejected on the same basis as 1-3, 5, and 7 respectively.
Claims 15-17 are substantially similar to claims 1-3, and thus are rejected on the same basis as claims 1-3.
Regarding claim 18, Shazeer, in view of Eigen, in further view of Shen, discloses the system of claim 15. Shazeer further discloses:
wherein the expert hierarch has multiple expert layers including an initial expert layer and one or more augmented expert layers (Shazeer, page 14, last paragraph, lines 7-9 “For the hierarchical MoE layers, the first level branching factor was 16, corresponding to the number of GPUs in our cluster. We use Noisy-Top-K Gating … for the ordinary MoE layers and k=2 at each level of the hierarchical MoE layers” teaches first level layers as an initial expert layer and subsequent levels of the hierarchical MoE layers as one or more augmented expert layers),
the initial expert layer includes a plurality of heterogeneous experts (Shazeer, page 4, paragraph 5, lines 7-9 “set of devices …each hosting a subset of the experts…” teaches the initial expert layer to include a plurality of heterogenous experts from different devices),
each of the one or more augmented expert layers includes at least one augmented expert (Shazeer, Figure 1 and page 14, last paragraph, lines 4-10 “We varied the number of experts between models, using … hierarchical MoE layers with 256, 1024 and 4096 experts…Thus, each example is processed by exactly 4 experts…” teaches including at least one augmented expert at the hierarchical layers).
Claim 20 is substantially similar to claim 7, and thus is rejected on the same basis as claim 7.
Response to Arguments
Applicant's arguments filed 02/09/2026 have been fully considered with regards to the 35 U.S.C. 101 rejection, and they are persuasive.
Applicant's arguments filed 02/09/2026 have been fully considered with regards to the 35 U.S.C. 102/103 rejection.
The applicant asserts of page 18-19 of the remarks that the prior art does not disclose training each of the experts at a layer based on different training data directed to the different expert layers and an individual expert output from each individual expert at any lower layer. In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986). The examiner refers to the rejection under 35 USC § 103 in the current office action for more details.
The applicant also states in page 19, “Eigen does not teach or suggest that each expert in the second layer expert f2 is trained based on an individual expert output of each individual expert f1.” The examiner respectfully disagrees, as Eigen explicitly discloses each expert in the second layer expert f2 is trained based on an individual expert output of each individual expert f1 through the gating mechanism g (Figure 1(b), page 3, paragraph below Figure 1, lines 2-3 “After training with the constraint in place, we lift it and further train in a second fine-tuning phase”, page 2, section 3, paragraph 1, lines 2-3 “The final output is produced by composing the mixtures at each layer” and page 2, paragraph 1, lines 2-3 "g(x) is a distribution over experts i = 1, ... , N that sums to 1.").
The applicant also states in page 21 that Shen does not disclose training each of the experts at a layer based on different training data directed to the different expert layers and an individual expert output from each individual expert at any lower layer. Once more, the applicant is reminded that one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. The applicant states “Shen does not teach or suggest … wherein different expert layers… are directed with different training data, respectively”. The examiner respectfully disagrees. Shen discloses different expert layers are directed with different training data, respectively, through different layers of expert nets trained with different sets of training data (Shen, Figure 1, elements “Shared experts”, “Specific experts”, “vb” and “ v’ ” and page 5, right column, section 4.5, paragraph 2, lines 4-8 “Main net takes user interest transfer vector…. user basic profiles vector… target item feature vector… and scenario context feature vector … as input… Bias net receives the input of Fairness Coefficient vb”). Shazeer, Eigen, and Shen are analogous art because they are from the same field of endeavor, mixture of expert models and machine learning. It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Shazeer, in view of Eigen, to include wherein different expert layers … are directed with different training data, respectively, based on the teachings of Shen. The motivation for doing so would have been to alleviate the influence of intervention bias on model prediction (Shen, page 5, right column, section 4.5, paragraph 2, lines 1-2 “…to alleviate the influence of intervention bias on model prediction…”).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's
disclosure.
Nie et al. (“DENSE-TO-SPARSE GATE FOR MIXTURE-OF-EXPERTS”) teaches training hierarchical mixture of experts.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HUMAIRA ZAHIN MAUNI whose telephone number is (703)756-5654. The examiner can normally be reached Monday - Friday, 9 am - 5 pm (ET).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, MATT ELL can be reached at (571) 270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/H.Z.M./Examiner, Art Unit 2141
/ANDREW L TANK/Primary Examiner, Art Unit 2141
Prosecution Insights