GENERATING IMAGE DESCRIPTIONS USING A MACHINE LEARNING MODEL

Application No. 18/827,521

• Filed: Sep 06, 2024
• Priority: Apr 29, 2024 — provisional 63/639,969
• Examiner: SITTNER, MATTHEW T
• Art Unit: 3629
• Tech Center: 3600 — Transportation & Electronic Commerce
• Assignee: Lemon Inc.
• OA Round: 1 (Non-Final)

Office Action

DETAILED ACTION
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on XXXXXXXXXXXXXX has been entered.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims X are canceled.
Claims X are new.
Claims 1–20 are pending. Claims 1, 12, and 17 are independent. Claims 1–20 are examined in this action.
This action is in reply to the papers filed on 09/06/2024 (effective filing date 04/29/2024).
Information Disclosure Statement
No Information Disclosure Statement has been filed.
The information disclosure statement(s) submitted: xxxxxxxx, has/have been considered by the Examiner and made of record in the application file.
Claim Interpretation
During examination, claims are given their broadest reasonable interpretation consistent with the specification. MPEP 2111. The following terms are construed as follows for purposes of this action.
"Sub-model" is construed as a component neural network of the overall machine learning model that performs a defined stage of processing (for example, the vision encoder, the connector, and the large language model described at ¶[0027] of the specification – US 2025/0335797 A1).
"Mixture of Experts (MoE) block" is construed as a network layer or block that contains a plurality of expert networks and a router or gating function that selects a subset of the experts for a given input (specification, ¶[0026], ¶[0033]–[0034]).
"Visual tokens" is construed as the sequence of feature vectors produced by the first sub-model from the input image (specification, ¶[0029], ¶[0039]).
"Input space of a third sub-model" is construed as the embedding space whose dimensionality and representation the third sub-model is able to consume as input (specification, ¶[0030], describing projection "into a latent input space of the third sub-model 106 so that they are consumable by the third sub-model 106").
"Router" is construed as any gating component that selects which experts receive an input and assigns weights to the selected experts (specification, ¶[0033]–[0034]).
Amendment

The present Office Action is based upon the original patent application filed on xxx as modified by the amendment filed on xxx.
Reasons For Allowance
Prior-Art Rejection withdrawn
Claims xxx are potentially allowable over the prior-art, but, are still subject to the 35 USC §101 and/or 35 USC §112 rejections herein. The closest prior art (See PTO-892, Notice of References Cited) does not teach the claimed:
Claims xxx are allowed. Independent claims X, Y, and Z all contain the same inventive scope. The closest prior art (See PTO-892, Notice of References Cited) does not teach the claimed:

The invention teaches… and the prior-art teaches…, however, the prior-art does not teach…
The closest prior-art (xxx) teach the features as disclosed in Non-final Rejection (xxxx), however, these cited references do not teach and the prior-art does not teach at least the following combination of features and/or elements:
determining, at a second time after associating the information corresponding to the first loyalty card with the logged location, that a second user computing device is located within a specified distance of the logged location using a second positioning system of the second user computing device; in response to determining that the second user computing device is located within the specified distance of the logged location of the first user computing device at the first time of detecting: retrieving information corresponding to a second loyalty card, the second loyalty card being associated with the merchant and the second user computing device; and displaying, by the second user computing device, data describing the second loyalty card.

Claim Rejections - 35 USC §101 - Withdrawn
Per Applicant’s amendments and arguments and considering new guidance in the MPEP, the rejections are withdrawn. Specifically, in Applicant’s Remarks (dated 03/14/2017, pgs. 8-11), Applicant traverses the 35 USC §101 rejections arguing that the amended claims recite new limitations that are not abstract, amount to significantly more, are directed to a practical application, etc… For example, Applicant argues….
In support of their arguments, Applicant cites to the following recent Fed. Cir. court cases (i.e., Alice Corp. v. CLS Bank Int’l, SRI Int’l, Inc. v. Cisco Systems, Inc., Ultramercial, Inc. v. Hulu, LLC, Berkheimer, Core Wireless, McRO, Enfish, Bascom, DDR, etc…).
Claim Rejections - 35 USC § 101
35 U.S.C. § 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.

Claims 1–20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (an abstract idea) without significantly more.
The claims have been evaluated under the framework of Alice Corp. Pty. Ltd. v. CLS Bank Int'l, 573 U.S. 208 (2014), and Mayo Collaborative Servs. v. Prometheus Labs., Inc., 566 U.S. 66 (2012), as set forth in MPEP 2106.
Step 1 – Statutory Category
Claims 1–11 recite a method and are directed to a process. Claims 12–16 recite a system comprising "at least one processor" and "at least one memory" and are directed to a machine. Claims 17–20 recite a "non-transitory computer-readable storage medium" and are directed to an article of manufacture. The word "non-transitory" excludes transitory propagating signals from the scope of claims 17–20, so no signal-per-se issue is present. MPEP 2106.03(I). No claim is directed to software per se. Each claim falls within a statutory category (Step 1: Yes). The analysis proceeds to Step 2A.
Step 2A, Prong One – Independent Claim 1
Claim 1 recites the following limitations, which set forth mathematical concepts:
• "configuring the machine learning model by incorporating Mixture of Experts (MoE) blocks into a plurality of sub-models of the machine learning model, wherein a first sub-model of the machine learning model comprises at least one first MoE block, wherein the at least one first MoE block comprises a first plurality of experts, wherein a second sub-model of the machine learning model comprises at least one second MoE block, and wherein the at least one second MoE block comprises a second plurality of experts"
• "generating visual tokens by the first sub-model, wherein only a subset of the first plurality of experts is activated to generate the visual tokens based on an input image"
• "projecting the visual tokens by the second sub-model, wherein only a subset of the second plurality of experts is activated to project the visual tokens into an input space of a third sub-model of the machine learning model"
A claim does not have to recite the word "calculating" to recite a mathematical calculation. A limitation recites a mathematical calculation when its broadest reasonable interpretation, in light of the specification, encompasses one. MPEP 2106.04(a)(2)(I)(C).
The specification defines the claimed MoE block and the activation of "only a subset" of its experts in purely mathematical terms. A router "adopts a linear layer to compute the normalized weight matrix W," computed as a Softmax of a linear function of the inputs. The experts are then selected by a Top-K operation on that matrix, the selected weights are re-normalized by a second Softmax, and "the final output can be a re-weighted sum" of the selected experts' outputs (specification, ¶[0034], ¶[0037]). The Top-K selection is what activates "only a subset" of the experts; the remaining experts are those not selected by it.
"Generating visual tokens" in light of the specification is the computation of numerical feature vectors from pixel values through the expert computations just described (¶[0029], ¶[0033]–[0034]). "Projecting the visual tokens … into an input space of a third sub-model" is the mapping of one set of vectors into another vector space by a weighted sum of expert outputs (¶[0030], ¶[0057]). Each of these is a mathematical relationship or calculation. MPEP 2106.04(a)(2)(I)(A), (C).
The "configuring" step specifies which mathematical operations the model applies. Arranging mathematical operations into a model is itself a mathematical concept.
Claim 1 therefore recites an abstract idea in the mathematical-concepts grouping (Step 2A, Prong One: Yes).
Step 2A, Prong Two – Independent Claim 1
The additional elements of claim 1, beyond the mathematical concepts identified above, are: (i) the "machine learning model" and its "first sub-model," "second sub-model," and "third sub-model"; (ii) the "input image"; and (iii) "outputting a text description of the input image by the third sub-model … wherein the third sub-model is configured to generate descriptions of input images based on tokens projected in the input space of the third sub-model."
The machine learning model and its sub-models are recited at a high level of generality, as generic components in which the mathematical operations run. The claim does not recite how the sub-models are structured apart from the MoE computations themselves. It does not recite how the router selects experts, how many experts are selected, or how the selected outputs are combined. Reciting that a mathematical calculation is performed by a generic "machine learning model" is no more than an instruction to apply the exception using a computer. MPEP 2106.05(f).
Receiving an "input image" is mere data gathering, which is insignificant extra-solution activity. MPEP 2106.05(g).
"Outputting a text description" by a third sub-model that "is configured to generate descriptions of input images" recites only the result of the processing, without any particular way of generating the description. It is insignificant post-solution activity. MPEP 2106.05(g). It also limits the abstract idea to the field of image captioning, which does not integrate the exception. MPEP 2106.05(h).
The claim does not reflect an improvement to the functioning of a computer or to another technology. MPEP 2106.04(d)(1); MPEP 2106.05(a). The benefit the specification attributes to the claimed arrangement is computational: fewer experts are evaluated per token, so fewer calculations are performed for the same number of parameters (¶[0029]–[0030], ¶[0039]). Performing fewer mathematical operations is an improvement, if at all, in the mathematical concept itself. "[A]n improvement in the abstract idea itself … is not an improvement in technology." MPEP 2106.05(a)(II); SAP America, Inc. v. InvestPic, LLC, 898 F.3d 1161, 1168 (Fed. Cir. 2018).
To the extent the specification describes a technical improvement, it attributes the improvement to details that claim 1 does not recite. Those details are the Top-K router trained from scratch (¶[0026], ¶[0034]); co-upcycling each MoE block from a pre-finetuned MLP (¶[0041]–[0043]); the three-stage training with auxiliary losses (¶[0040], ¶[0044]); and the specific expert counts and routers evaluated (¶[0076]–[0078]). An improvement described in the specification must be reflected in the claim. MPEP 2106.04(d)(1); MPEP 2106.05(a).
Claim 1 differs from the claims found eligible in Ex parte Desjardins, Appeal No. 2024-000567 (PTAB Sept. 26, 2025) (Appeals Review Panel) (precedential). There, the claims recited a particular manner of training a machine learning model that addressed an identified technical problem, and the Panel found the claims reflected that improvement. Here, claim 1 recites a machine learning model generically, in which conventional sparse MoE computations are placed in each sub-model, and then recites the result: a text description. This is analogous to Recentive Analytics, Inc. v. Fox Corp., 134 F.4th 1205 (Fed. Cir. 2025), cert. denied, No. 25-505 (U.S. Dec. 8, 2025). The court there held that "patents that do no more than claim the application of generic machine learning to new data environments, without disclosing improvements to the machine learning models to be applied, are patent ineligible under § 101" (slip op. at 18). Sparse mixture-of-experts computation was a known machine learning technique before the effective filing date (see, e.g., Shazeer, US 2023/0419079 A1, ¶[0023], ¶[0027]; Riquelme, WO 2022/251717 A1, ¶[0005], ¶[0040]). Applying it to each stage of a known image-captioning pipeline applies an established machine learning technique to a new data environment.
Considered individually and as an ordered combination, the additional elements do not integrate the abstract idea into a practical application (Step 2A, Prong Two: No). Claim 1 is directed to an abstract idea.
Step 2B – Independent Claim 1
The additional elements do not amount to significantly more than the abstract idea. As discussed above, the generic machine learning model and sub-models amount to no more than instructions to apply the exception on a computer. MPEP 2106.05(f). Receiving an input image and outputting a text description are insignificant extra-solution activity.
Re-evaluated under Step 2B, receiving data and outputting a result of the processing are well-understood, routine, and conventional activities. MPEP 2106.05(d)(II) recognizes receiving or transmitting data over a network, and storing and retrieving information in memory, as such activities. The specification itself describes the hardware on which the model runs as "a conventional server computer, workstation, desktop computer, laptop, tablet, network appliance, PDA, e-reader, digital cellular phone, or other computing node" (¶[0081]).
The arrangement of a vision encoder, a projecting connector, and a language model that outputs a description of an image was itself well-understood, routine, and conventional before the effective filing date, as shown by Li, US 2024/0160853 A1, ¶[0026], ¶[0050]–[0051], and Dai, US 2024/0160858 A1, ¶[0024]–[0027]. Sparse MoE blocks with top-k routing were likewise conventional, as shown by Shazeer, US 2023/0419079 A1, ¶[0023]–[0027], and Riquelme, WO 2022/251717 A1, ¶[0039]–[0042], ¶[0060]. See Berkheimer v. HP Inc., 881 F.3d 1360, 1368–69 (Fed. Cir. 2018); MPEP 2106.07(a)(III).
Considered as an ordered combination, the additional elements add nothing that is not already present when they are considered individually. Claim 1 does not include an inventive concept (Step 2B: No). Claim 1 is ineligible.

Independent Claims 12 and 17
Claim 12 recites the same operations as claim 1, performed by "at least one processor" executing instructions stored in "at least one memory." Claim 17 recites the same operations stored as instructions on a "non-transitory computer-readable storage medium." Each recites the same mathematical concepts identified for claim 1 (Prong One: Yes). The processor, memory, and storage medium are generic computer components recited at a high level of generality. The specification describes them as conventional (¶[0081]–[0093]). Implementing an abstract idea on generic computer components does not integrate it into a practical application or provide significantly more. MPEP 2106.05(f); Alice, 573 U.S. at 223–24. Claims 12 and 17 are ineligible for the reasons given for claim 1.
Dependent Claims
The dependent claims either recite further mathematical concepts or add additional elements that do not integrate the exception into a practical application or amount to significantly more.
Claims 2, 13, and 18. These claims add "generating representations of the input image based on performing self-attention and normalization" and "routing the representations of the input image to the activated subset of the first plurality of experts by a router." Self-attention is a weighted combination of vectors computed from dot products and a softmax. Normalization is a statistical rescaling of values. Routing by a router is the Top-K selection described at ¶[0034]. Each is a mathematical calculation. MPEP 2106.04(a)(2)(I)(C). Claims 13 and 18 also recite the weighted sum addressed for claim 3.
Claims 3, 5, 14, and 19. These claims recite "calculating a weighted sum of outputs from the activated subset" of the first or second plurality of experts. Claims 5, 14, and 19 also recite routing the visual tokens by a router and processing them by the activated experts. A weighted sum is a mathematical calculation, and routing and processing by the experts are the mathematical operations identified for claim 1. MPEP 2106.04(a)(2)(I)(C).
Claim 4. This claim adds "dividing the input image into patches," "generating a pyramid of high-resolution visual features," "down-sampling the high-resolution visual features," and "performing channel-wise concatenation." Partitioning an array of pixel values, computing features at multiple scales, down-sampling (averaging or pooling values), and concatenating vectors are mathematical operations on numerical data. MPEP 2106.04(a)(2)(I)(C).
Claim 6. This claim adds a third MoE block in the third sub-model and "generating the text description of the input image by only a subset of the third plurality of experts." This applies the same Top-K selection and weighted-sum computation to the third sub-model. It is a further mathematical concept and does not add an additional element beyond those addressed for claim 1.
Claim 7. This claim adds "receiving an input text query," which is mere data gathering and insignificant extra-solution activity. MPEP 2106.05(g). It also adds "converting the input text query into an embedding in the input space of the third sub-model," which maps text to a numerical vector and is a mathematical calculation. Generating the description "based on the projected visual tokens and the embedding" specifies the inputs to the mathematical operations. None of these limitations integrates the exception or amounts to significantly more. Receiving a text query is well-understood, routine, and conventional (Li, ¶[0055]; MPEP 2106.05(d)(II)).
Claims 8, 15, and 20. These claims add a "three-stage training process": "pre-training the second sub-model while freezing the first sub-model and the third sub-model," "pre-finetuning parameters of the machine learning model," and "adding at least one MoE block into each of the first sub-model, the second sub-model, and the third sub-model and training the machine learning model on visual instruction tuning data." Training a machine learning model is the iterative adjustment of numerical parameters to minimize a loss function, which is a mathematical calculation. The claims recite the stages at a high level of generality, without any particular way of performing them. The Federal Circuit has held that requiring a machine learning model to be "iteratively trained" does not "represent a technological improvement." Recentive, slip op. at 12. Freezing parameters, fine-tuning, and training on instruction data were conventional training practices (Li, ¶[0029]; Lin, "MoE-LLaVA," arXiv:2401.15947, p. 2). These limitations do not integrate the exception or amount to significantly more.
Claims 9, 10, and 16. These claims add generating an initial expert from the pre-finetuned parameters, replicating it to form an initial expert block, and "iteratively training" the initial expert blocks to obtain the MoE blocks. Copying numerical parameter values and iteratively adjusting them are mathematical operations, recited without any particular technical implementation. Recentive, slip op. at 12. Initializing experts by copying trained dense weights was a conventional practice (Komatsuzaki, "Sparse Upcycling," arXiv:2212.05055, p. 3). These limitations do not integrate the exception or amount to significantly more.
Claim 11. This claim adds "applying a loading balancing loss and a router-z loss to each of the first sub-model, the second sub-model, and the third sub-model to maintain a load balance between experts." A loss function is a mathematical formula. The specification sets it out as L = Lce + αb·Lb + αz·Lz (¶[0044]). This is a further mathematical concept. MPEP 2106.04(a)(2)(I)(B).
Claims 13–16 and 18–20 are the system and storage-medium counterparts of claims 2, 3, 5, 8, 9, and 10 and are ineligible for the same reasons. The generic processor, memory, and storage medium do not change the analysis.
No dependent claim, considered individually or as an ordered combination with the claims from which it depends, integrates the abstract idea into a practical application or amounts to significantly more than the abstract idea. Claims 1–20 are rejected under 35 U.S.C. 101.
Suggested Amendments
The rejection may be overcome by amending the independent claims to recite, with particularity, the technical implementation that the specification credits for the improvement. The amendment must reflect that improvement in the claim itself, rather than reciting the use of MoE blocks and the resulting description. For example, Applicant may consider reciting in each independent claim the following elements together:
• the Top-K router structure and its operation in each of the vision encoder and the MLP connector (¶[0033]–[0037]);
• the co-upcycling of each MoE block from a pre-finetuned MLP of the corresponding sub-model, followed by iterative training during which different experts are activated for different data (¶[0041]–[0043]);
• the three-stage training with auxiliary balancing losses applied to each sub-model separately (¶[0040], ¶[0044]).
Applicant should also explain on the record how the claimed arrangement improves the operation of the multimodal model itself, with citations to the specification. MPEP 2106.04(d)(1); MPEP 2106.05(a). Any such amendment must find support in the application as filed.

Claim Rejections - 35 USC §112
The following is a quotation of 35 U.S.C. §112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.

35 U.S.C. 112(b) – Claim 4 (Term of Degree)
Claim 4 is rejected under 35 U.S.C. 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
Claim 4 recites "generating a pyramid of high-resolution visual features based on the patches" and "down-sampling the high-resolution visual features." The term "high-resolution" is a relative term of degree. The claim provides no standard for measuring what resolution is "high," and the specification does not supply one. The specification uses the term only comparatively and by example: it states that "high-resolution inputs are essential" (¶[0039], ¶[0054]), gives 672×672 inputs as an example in contrast to 336×336 inputs (¶[0039]), and reports a "combination of 1008 (3×) and 336 (1×)" as empirically best (¶[0079]). Those examples do not define a boundary, and elsewhere the specification describes the same features as "a pyramid of multi-resolution visual features" (¶[0039], ¶[0054]), which differs from "high-resolution." A person of ordinary skill in the art would not be informed with reasonable certainty which visual features fall within the claim. Nautilus, Inc. v. Biosig Instruments, Inc., 572 U.S. 898, 901 (2014); Interval Licensing LLC v. AOL, Inc., 766 F.3d 1364, 1370–71 (Fed. Cir. 2014); MPEP 2173.05(b).
Suggested amendment: replace "a pyramid of high-resolution visual features" with language tied to a measurable reference, for example "a pyramid of multi-resolution visual features, including visual features at a resolution greater than a resolution of the input image," and replace "the high-resolution visual features" with "the visual features at the resolution greater than the resolution of the input image," consistent with ¶[0039] and ¶[0054].
35 U.S.C. 112(b) – Claims 17–20 (Antecedent Basis)
Claims 17–20 are rejected under 35 U.S.C. 112(b) as being indefinite.
Claim 17 recites "configuring the machine learning model by incorporating Mixture of Experts (MoE) blocks." There is insufficient antecedent basis for "the machine learning model" in the claim. Unlike claims 1 and 12, whose preambles introduce "a machine learning model," the preamble of claim 17 does not introduce any machine learning model. MPEP 2173.05(e). Claims 18–20 depend from claim 17, do not cure the deficiency, and are rejected for the same reason. For purposes of the prior-art rejections below, the phrase is interpreted as "a machine learning model."
Suggested amendment: in claim 17, change "configuring the machine learning model" to "configuring a machine learning model."

Claim Objection – Claim 11
Claim 11 is objected to because of the following informality: "a loading balancing loss" should read "a load balancing loss," and "a router-z loss" should read "a router z-loss," consistent with ¶[0044] and ¶[0070] of the specification. Appropriate correction is required.

Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.

The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim 1 (Li in view of Riquelme and Shazeer)
Claim 1 is rejected under 35 U.S.C. 103 as being unpatentable over Li (US 2024/0160853 A1) in view of Riquelme (WO 2022/251717 A1) and Shazeer (US 2023/0419079 A1).
Claim 1 recites "A method of generating image descriptions using a machine learning model." Li teaches this feature: "a vision-language model may be trained to receive an input image and generate a text caption of the input image" (Li, ¶[0005]). To the extent the preamble limits the claim, Li meets it.
Claim 1 recites "configuring the machine learning model by incorporating Mixture of Experts (MoE) blocks into a plurality of sub-models of the machine learning model." Li teaches a machine learning model made of a plurality of sub-models: "The multi-modal vision-language model that comprises an image encoder 110, a query Transformer 120 and a (large) language model (LLM) 130 may be trained by the vision-language pretraining framework 100" (Li, ¶[0026]). Under the claim interpretation above, Li's image encoder 110 is the first sub-model; Li's query Transformer 120 together with its fully connected projection layer 410 (Li, ¶[0050]) is the second sub-model; and Li's LLM 130 is the third sub-model. Primary reference Li does not explicitly teach "incorporating Mixture of Experts (MoE) blocks into a plurality of sub-models," however, secondary references Riquelme and Shazeer teach this feature. Riquelme teaches incorporating MoE blocks into an image-encoding network: "At least one of the network blocks is an expert network block that includes a set of multiple different expert modules" (Riquelme, ¶[0005]). Shazeer teaches incorporating MoE blocks into any network: "To increase the computational capacity of the neural network 102, the neural network 102 includes one or more Mixture of Experts (MoE) subnetworks between the stacked neural network layers" (Shazeer, ¶[0023]). Incorporating Riquelme's expert blocks in Li's image encoder and Shazeer's MoE subnetworks in Li's connector is incorporating MoE blocks into a plurality of sub-models. Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Li's vision-language model by incorporating Riquelme's sparse expert blocks into Li's image encoder and Shazeer's MoE subnetworks into Li's connector, because Li identifies that the "traditional end-to-end framework would incur significant computational cost, resulting in low scalability of the vision-language model" (Li, ¶[0021]). Riquelme teaches that sparse expert blocks significantly improve "the time and computational efficiency of the processing of the image" (Riquelme, ¶[0007]), and Shazeer teaches that sparse MoE "can significantly increase the computational capacity of the neural network 102 without a proportional increase in computation" (Shazeer, ¶[0036]). The rationale is MPEP 2143(I)(D), because Li's transformer-based sub-models are known networks ready for the known improvement of sparse MoE layers, and the predictable result is a vision-language model with greater capacity in its sub-models at approximately constant per-input computation. KSR, 550 U.S. at 416.
Claim 1 recites "wherein a first sub-model of the machine learning model comprises at least one first MoE block, wherein the at least one first MoE block comprises a first plurality of experts." Primary reference Li does not explicitly teach this feature, however, secondary reference Riquelme teaches this feature as follows: "The updated representations of the embedded patches 152 can be provided to a routing module 170 that is configured to assign, for each updated representation of a respective embedded patch 152, the updated representation to one or more expert modules 180 of a set of multiple expert modules of the network block 150" (Riquelme, ¶[0094]), and "each expert module 180 is a multi-layer perceptron (MLP) that includes one or more feedforward neural network layers" (Riquelme, ¶[0096]). Riquelme's expert network block is a first MoE block, and its expert modules are a first plurality of experts. Riquelme's network is itself an image encoder whose output "can represent an embedding of the input image" provided "to one or more downstream machine learning models for further processing" (Riquelme, ¶[0100]). Riquelme's network blocks therefore fit directly into the image-encoder position of Li. The motivation, rationale (MPEP 2143(I)(D)), and predictable result are those stated for the preceding limitation.
Claim 1 recites "wherein a second sub-model of the machine learning model comprises at least one second MoE block, and wherein the at least one second MoE block comprises a second plurality of experts." Primary reference Li does not explicitly teach this feature, however, secondary reference Shazeer teaches this feature as follows: "The MoE subnetwork 130 includes multiple expert neural networks, e.g., expert neural networks 114-122" (Shazeer, ¶[0025]); "Each expert neural network can be a feed-forward neural network with its own parameters" (Shazeer, ¶[0026]); and "In some implementations, one or both of the first neural network layer 104 and the second neural network layer 108 can be other MoE subnetworks" (Shazeer, ¶[0024]). Li's connector contains feed-forward layers, namely "a feed forward layer 213" (Li, ¶[0037]) and the fully connected projection layer 410 (Li, ¶[0050]). Placing a Shazeer MoE subnetwork at a feed-forward layer of Li's connector yields a second MoE block with a second plurality of experts. Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to place a Shazeer MoE subnetwork at a feed-forward layer of Li's connector, because Shazeer teaches that the capacity-per-computation benefit of sparse experts is available at any feed-forward position of a network (Shazeer, ¶[0023], ¶[0036]), and Riquelme confirms that the benefit carries over to networks that process image representations (Riquelme, ¶[0007]). A person of ordinary skill building the Li–Riquelme model would have had the same reason to sparsify the connector that processes the encoder's visual tokens. The rationale is MPEP 2143(I)(D), and the predictable result is a connector with more parameters but approximately the same per-token computation. KSR, 550 U.S. at 416.
Claim 1 recites "generating visual tokens by the first sub-model, wherein only a subset of the first plurality of experts is activated to generate the visual tokens based on an input image." Li teaches generating visual tokens by the first sub-model based on an input image: "an input image 105a may be encoded by the image encoder 110 into image representations" (Li, ¶[0032]). The image encoder may be one of the "vision transformer models, such as ViT-L/14 from CLIP" (Li, ¶[0033]), whose features are a sequence of feature vectors, "e.g. 257×1024 for ViT-L/14" (Li, ¶[0037]), that is, 257 visual tokens. Primary reference Li does not explicitly teach "only a subset of the first plurality of experts is activated," however, secondary reference Riquelme teaches this feature as follows: "each patch can be assigned to the k expert modules 130a-e that have the highest routing scores for the patch, where k is the same for all patches" (Riquelme, ¶[0040]), and "for some input images 102, some expert modules 130a-e of the jth network block 110j are idle for the block input 112 generated from the input image 102" (Riquelme, ¶[0042]). Riquelme's patch representations are visual tokens, and assignment to the top-k experts with the others idle is activation of only a subset of the first plurality of experts. Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to generate Li's visual tokens with Riquelme's top-k expert routing, so that only the selected experts of each first MoE block are activated, for the reasons stated above for incorporating MoE blocks into Li's sub-models (Li, ¶[0021]; Riquelme, ¶[0007]). The rationale is MPEP 2143(I)(D), and the predictable result is an image encoder whose visual tokens are generated by only the selected experts of each MoE block. KSR, 550 U.S. at 416.
Claim 1 recites "projecting the visual tokens by the second sub-model, wherein only a subset of the second plurality of experts is activated to project the visual tokens into an input space of a third sub-model of the machine learning model." Li teaches projecting the visual tokens by the second sub-model into the input space of the third sub-model. The query Transformer processes the image encoder's output: "The self-attention outputs then interact with frozen image features, e.g., the image representation from the frozen image encoder 110, through cross-attention layers 212 to produce cross-attention outputs" (Li, ¶[0036]). The result is then projected: "a fully-connected (FC) layer 410 is connected to linearly project the output query embeddings Z 408 into embeddings 412 having the same dimension as the word embedding of the LLM 130" (Li, ¶[0050]). Embeddings having the same dimension as the LLM's word embeddings, which the LLM consumes as input (Li, ¶[0051]), are in the input space of the third sub-model. Primary reference Li does not explicitly teach that "only a subset of the second plurality of experts is activated" in the projecting sub-model, however, secondary reference Shazeer teaches this feature as follows: "Although the MoE subnetwork includes a large number of expert neural networks, only a small number of them are selected during the processing of any given network input by the neural network 102" (Shazeer, ¶[0027]). An expert whose gating weight is zero "does not need to compute the respective expert output" (Shazeer, ¶[0032]), and the gating keeps only "the k highest values" (Shazeer, ¶[0050]). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to activate only the top-k experts of the MoE block in Li's connector, as Shazeer teaches, for the reasons stated above for the second MoE block (Shazeer, ¶[0023], ¶[0036]). The rationale is MPEP 2143(I)(D), and the predictable result is a connector whose projection is computed by only the selected experts. KSR, 550 U.S. at 416.
Claim 1 recites "outputting a text description of the input image by the third sub-model of the machine learning model, wherein the third sub-model is configured to generate descriptions of input images based on tokens projected in the input space of the third sub-model." Li teaches this feature: "the LLM decoder 130b receives the projected embedding 412, and generate a decoded output text 415 conditioned on the visual representation 412 from the query Transformer 120" (Li, ¶[0051]). In response to a request such as "describe what is in this picture," "The LLM 130 may then generate an output text 118 conditioned on the visual representation of the input image 115 and the guided text 116" (Li, ¶[0055]). No secondary reference is needed to meet this limitation.
Claim 2 (Li in view of Riquelme and Shazeer)
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Riquelme and Shazeer.
Claim 2 recites "generating representations of the input image based on performing self-attention and normalization by the first sub-model." Li teaches that the first sub-model is a vision transformer (Li, ¶[0033]), but primary reference Li does not explicitly teach performing "self-attention and normalization," however, secondary reference Riquelme teaches this feature as follows: "the network block 150 can first apply a layer normalization layer 154 to the embedded patches 152 before providing the output of the layer normalization layer 154 to the self-attention neural network layer 160" (Riquelme, ¶[0087]), and "the network block 150 can include a layer normalization layer 164 that applies layer normalization to the output of the self-attention neural network layer 160 (or the residual connection layer 162)" (Riquelme, ¶[0093]). "The output of the self-attention neural network layer 160 (or the residual connection layer 162 or the layer normalization layer 164) includes an updated representation of the embedded patches 152" (Riquelme, ¶[0094]). Those updated representations are representations of the input image generated by self-attention and normalization. The motivation, rationale (MPEP 2143(I)(D)), and predictable result are those stated for claim 1, because Riquelme's expert block is incorporated as a whole, including its normalization and self-attention layers.
Claim 2 recites "routing the representations of the input image to the activated subset of the first plurality of experts by a router of the at least one first MoE block." Primary reference Li does not teach a router, however, secondary reference Riquelme teaches this feature as follows: "The updated representations of the embedded patches 152 can be provided to a routing module 170 that is configured to assign, for each updated representation of a respective embedded patch 152, the updated representation to one or more expert modules 180 of a set of multiple expert modules of the network block 150" (Riquelme, ¶[0094]). "The routing module 120 can determine to assign a patch to one or more expert modules 130a-e corresponding to the highest routing scores for the patch" (Riquelme, ¶[0039]). Riquelme's routing module is a router of the first MoE block. The motivation, rationale (MPEP 2143(I)(D)), and predictable result are those stated for claim 1.
Claim 3 (Li in view of Riquelme and Shazeer)
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Riquelme and Shazeer.
Claim 3 recites "generating the visual tokens by calculating a weighted sum of outputs from the activated subset of the first plurality of experts." Primary reference Li does not teach experts, however, secondary reference Riquelme teaches this feature as follows: "the combination module 140 combines, for each patch of the block input, the corresponding module outputs by computing a sum of the module outputs for the patch. In some such implementations, to generate the patch output for each patch, the combination module 140 can weight each module output using the routing score corresponding to the patch and the expert module 130a-e that generated the module output" (Riquelme, ¶[0060]). The patch outputs are the visual tokens output by the first MoE block. The motivation, rationale (MPEP 2143(I)(D)), and predictable result are those stated for claim 1. KSR, 550 U.S. at 416.
Claim 4 (Li in view of Riquelme, Shazeer, and Shi)
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Riquelme and Shazeer, and further in view of Shi (Baifeng Shi et al., "When Do We Not Need Larger Vision Models?," arXiv:2403.13043).
Claim 4 recites "dividing the input image into patches; generating a pyramid of high-resolution visual features based on the patches by the activated subset of the first plurality of experts; and down-sampling the high-resolution visual features and performing channel-wise concatenation on the down-sampled high-resolution visual features." As noted in the rejection under 35 U.S.C. 112(b), "high-resolution" is indefinite. For purposes of this rejection it is interpreted as visual features computed from the image at a resolution greater than the encoder's base input resolution.
Riquelme teaches dividing the input image into patches: "The neural network system can be configured to segment the images 210-250 into multiple image patches, and process the image patches using a sequence of network blocks that includes one or more expert network blocks" (Riquelme, ¶[0102]). Primary reference Li and secondary reference Riquelme do not explicitly teach the pyramid, down-sampling, and channel-wise concatenation, however, secondary reference Shi teaches this feature as follows: "S2-Wrapper extends a pre-trained model to multiple image scales (e.g., 224², 448²) by splitting different scales of images to the same size as seen in pre-training. Specifically, given the image at 224² and 448² scales, S2-Wrapper first divides the 448² image into four 224² sub-images, which along with the original 224² image are fed to the same pre-trained model. The features of four sub-images are merged back to the large feature map of the 448² image, which is then average-pooled to the same size as the feature map of 224² image. Output is the concatenation of feature maps across scales" (Shi, p. 3, § 3.1). Shi also states that "Feature maps of different scales are average-pooled to the original spatial size (16²) and concatenated together. The final multi-scale feature has the same spatial shape as single-scale feature while having higher channel dimension (e.g., 1536 vs. 768)" (Shi, p. 2, Fig. 1 caption). Splitting each scaled image into sub-images is dividing the image into patches. The feature maps at the larger scales are a pyramid of higher-resolution visual features, average pooling is down-sampling, and concatenation that increases the channel dimension is channel-wise concatenation. In the combination of claim 1, the features are generated by the activated subset of the first plurality of experts. Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to apply Shi's multi-scale S2-Wrapper processing to the MoE image encoder of Li as modified by Riquelme, because Shi teaches that multi-scale features improve visual understanding while "interpolating the large feature map into the regular size makes sure the number of output tokens stays the same, preventing computational overhead in downstream applications such as MLLMs" (Shi, p. 3, § 3.1). That addresses the same computational-cost concern Li identifies (Li, ¶[0021]). The rationale is MPEP 2143(I)(C), because Shi's multi-scale technique improves a similar vision encoder in the same way, and the predictable result is an MoE image encoder that outputs multi-scale features without increasing the number of visual tokens passed to the connector. KSR, 550 U.S. at 416.
Claims 5, 14, and 19 (Li in view of Riquelme and Shazeer)
Claims 5, 14, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Riquelme and Shazeer.
Claim 5 recites "routing the visual tokens to the activated subset of the second plurality of experts by a router of the at least one second MoE block; processing the visual tokens by the activated subset of the second plurality of experts; and calculating a weighted sum of outputs from the activated subset of the second plurality of experts." Claim 14 recites the same operations in the system of claim 12 and claim 19 recites the same operations in the storage medium of claim 17. Li teaches that the second sub-model receives and processes the visual tokens (Li, ¶[0036]–[0037]). Primary reference Li does not teach routing to, processing by, and weighted summation of a subset of experts, however, secondary reference Shazeer teaches this feature as follows: "The gating subsystem 110 then provides the first layer output as input to each of the selected expert neural networks" (Shazeer, ¶[0027]). "Each of the selected expert neural networks is configured to process the first layer output in accordance with the respective current values of parameters of the selected expert neural network to generate a respective expert output" (Shazeer, ¶[0041]). "The system then sums the weighted expert outputs to generate the MoE output" (Shazeer, ¶[0043]). Shazeer's gating subsystem is a router of the second MoE block, and in the combination of claim 1 its input is the visual tokens from the image encoder. The motivation, rationale (MPEP 2143(I)(D)), and predictable result are those stated for the second MoE block of claim 1. KSR, 550 U.S. at 416.
Claim 6 (Li in view of Riquelme, Shazeer, and Lin)
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Riquelme and Shazeer, and further in view of Lin (Bin Lin et al., "MoE-LLaVA: Mixture of Experts for Large Vision-Language Models," arXiv:2401.15947).
Claim 6 recites "wherein the third sub-model comprises at least one third MoE block, wherein the at least one third MoE block comprises a third plurality of experts, and wherein the method further comprises: generating the text description of the input image by only a subset of the third plurality of experts." Li teaches that the third sub-model generates the text description (Li, ¶[0051], ¶[0055]). Primary reference Li does not teach an MoE block in the language model, however, secondary reference Lin teaches this feature as follows. MoE-LLaVA "uniquely activates only the top-k experts through routers during deployment, keeping the remaining experts inactive" (Lin, p. 1, Abstract). "For each MoE layer, only two experts are activated for each token, while the other experts remain silent" (Lin, p. 2, Fig. 2 caption), and "When the top-k experts are activated, the remaining experts remain silent" (Lin, p. 5). Lin's MoE layers are in the LLM of a vision-language model that generates text about an image. Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to incorporate Lin's top-k MoE layers into the LLM of Li as modified by Riquelme and Shazeer, so that only a subset of the LLM's experts generates the text description, because Lin teaches that sparse MoE layers "effectively scale model capacity by using fixed activated parameters to process data" (Lin, p. 2) in the language model of a vision-language model of the kind Li describes. The rationale is MPEP 2143(I)(C), because Lin's technique improves a similar vision-language LLM in the same way, and the predictable result is an LLM that generates the text description using only the selected subset of its experts. KSR, 550 U.S. at 416.
Claim 7 (Li in view of Riquelme and Shazeer)
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Riquelme and Shazeer.
Claim 7 recites "receiving an input text query; converting the input text query into an embedding in the input space of the third sub-model; and generating the text description of the input image by the third sub-model based on the projected visual tokens and the embedding." Li teaches this feature: "a text 116 may be received accompanying the input image 115, providing guidance on text generation. For example, the text 116 may comprise a question or a request, such as “describe what is in this picture”. The text 116 may then be prepended to the projected embeddings 412 and input to the LLM 130. The LLM 130 may then generate an output text 118 conditioned on the visual representation of the input image 115 and the guided text 116" (Li, ¶[0055]). Li also teaches that the text is embedded in the LLM's input space: "The projected query embeddings 412 are then prepended to the input text embeddings" (Li, ¶[0050]). No secondary reference is needed to meet the limitations added by claim 7.
Claims 8, 15, and 20 (Li in view of Riquelme, Shazeer, and Lin)
Claims 8, 15, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Riquelme and Shazeer, and further in view of Lin.
Claim 8 recites "wherein the machine learning model is trained by utilizing a three-stage training process." Claim 15 recites the same limitation in the system of claim 12 and claim 20 recites it in the storage medium of claim 17, adding that the three steps occur "in a first training stage," "in a second training stage," and "in a third training stage," respectively. Primary reference Li teaches two-stage training (Li, ¶[0028]–[0029]) but does not explicitly teach three stages, however, secondary reference Lin teaches this feature as follows: "we introduce a simple yet effective three-stage training strategy MoE-Tuning" (Lin, p. 2).
Claims 8, 15, and 20 recite "pre-training the second sub-model while freezing the first sub-model and the third sub-model." Li teaches this feature: "During the second stage, again only the query Transformer 120 and the queries 106 are updated while the image encoder 110 and the LLM 130 are frozen" (Li, ¶[0029]). No secondary reference is needed to meet this limitation. Lin also teaches it: "In stage I, only the MLP is trained" (Lin, p. 2, Fig. 2 caption).
Claims 8, 15, and 20 recite "pre-finetuning parameters of the machine learning model." Primary reference Li does not teach this step, however, secondary reference Lin teaches this feature as follows: "In stage II, all parameters are trained except for the Vision Encoder (VE)" (Lin, p. 2, Fig. 2 caption). "MoE-LLaVA utilizes the weights from the second stage as initialization for the third stage to alleviate the learning difficulty of the sparse model" (Lin, p. 4). Lin's second stage fine-tunes parameters of the model before the MoE layers are added, which is pre-finetuning parameters of the machine learning model.
Claims 8, 15, and 20 recite "adding at least one MoE block into each of the first sub-model, the second sub-model, and the third sub-model and training the machine learning model on visual instruction tuning data." Primary reference Li does not teach this step, however, secondary reference Lin teaches this feature as follows: "in stage III we replicate the FFN as the initialization weights for the experts and only train the MoE layers" (Lin, p. 2), and "For the third stage, we utilize the same data pipeline as LLaVA-mix-665k (Liu et al., 2023b)" (Lin, p. 5). Lin also states that "Tuning with multi-modal instruction data is a key technique to enhance the capabilities and controllability of large models" (Lin, p. 4). Lin adds the MoE blocks in the third stage and trains them on multimodal instruction data, which is visual instruction tuning data. Lin adds MoE blocks to the language model. In the combination of claims 1 and 6, MoE blocks are present in each of the first, second, and third sub-models, and applying Lin's third-stage addition to each MoE-equipped sub-model adds at least one MoE block into each. Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to train the combined Li–Riquelme–Shazeer model using Lin's three-stage process, so that the MoE blocks are added to each sub-model only after the connector is pre-trained and the model is fine-tuned, because Lin teaches that "simultaneously converting LLM to LVLM and sparsifying the model leads to significant performance degradation" and that "proper initialization is crucial for sparsifying the LVLM" (Lin, p. 2). That gives a reason to add the MoE blocks of the combined model only after the connector is pre-trained and the model is fine-tuned. The rationale is MPEP 2143(I)(C), because Lin's staged training improves a similar sparse vision-language model in the same way, and the predictable result is a stable, staged training of the combined MoE model. KSR, 550 U.S. at 416.
Claim 9 (Li in view of Riquelme, Shazeer, Lin, and Komatsuzaki)
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Riquelme, Shazeer, and Lin, and further in view of Komatsuzaki (Aran Komatsuzaki et al., "Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints," arXiv:2212.05055).
Claim 9 recites "generating an initial expert for each of the first sub-model, the second sub-model, and the third sub-model based on the pre-finetuned parameters; and generating at least one initial expert block for each of the first sub-model, the second sub-model, and the third sub-model by replicating the initial expert." Primary reference Li does not teach this feature. Secondary reference Lin teaches it for the language model: "Stage III: As an initialization, we replicate the FFN multiple times to initialize the experts" (Lin, p. 5), using "the weights from the second stage as initialization for the third stage" (Lin, p. 4). Lin does not explicitly teach replicating a trained expert for the vision encoder or the connector, however, secondary reference Komatsuzaki teaches this feature as follows: "We apply the same sparse upcycling recipe to both language and vision tasks, focusing on the T5 (encoder-decoder) (Raffel et al., 2020; Narang et al., 2021) and Vision Transformer (encoder) (Dosovitskiy et al., 2021) architectures, respectively" (Komatsuzaki, p. 3, § 2.2). "Each MoE layer contains a fixed number of experts. Each expert is initialized as a copy of the original MLP. In addition, we add a router whose weights are randomly initialized" (Komatsuzaki, p. 3, § 3). The trained MLP of each sub-model is an initial expert based on the pre-finetuned parameters and copying it into each expert slot generates an initial expert block by replicating the initial expert. Applying that recipe to the MoE block of each of the three sub-models in the combination of claims 1, 6, and 8 meets the claim. Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to initialize the experts of each sub-model's MoE block in the combined model by replicating that sub-model's pre-finetuned MLP, as Komatsuzaki teaches, because Komatsuzaki teaches that initializing experts from a trained dense checkpoint gives an MoE model "more performant than the original model, at a smaller cost than was used to train the original model" (Komatsuzaki, p. 2), and that the same recipe works for a Vision Transformer encoder as well as a language model. The rationale is MPEP 2143(I)(C), because Komatsuzaki's initialization improves similar transformer sub-models in the same way, and the predictable result is that each sub-model's experts start from its own trained feed-forward weights. KSR, 550 U.S. at 416.
Claim 10 (Li in view of Riquelme, Shazeer, Lin, and Komatsuzaki)
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Riquelme, Shazeer, and Lin, and further in view of Komatsuzaki.
Claim 10 recites "generating the at least one first MoE block by iteratively training the at least one initial expert block for the first sub-model; generating the at least one second MoE block by iteratively training the at least one initial expert block for the second sub-model; and generating at least one third MoE block by iteratively training the at least one initial expert block for the third sub-model." Primary reference Li teaches iterative training in general: "The backpropagation from the last layer to the input layer may be conducted for a number of training samples in a number of training epochs" (Li, ¶[0065]). Li does not teach training initial expert blocks, however, secondary reference Komatsuzaki teaches this feature as follows: "After the new model is loaded and initialized, we continue training it for a number of additional steps depending on the available budget and resources" (Komatsuzaki, p. 3, § 3). Lin likewise teaches that in stage III the model will "only train the MoE layers. Finally, the sparse model gradually transitions from a general LVLM initialization to sparse mixture of experts" (Lin, p. 2). Continuing to train the replicated experts over additional steps is iteratively training the initial expert blocks to generate the MoE blocks. Applying this to the initial expert block of each sub-model in the combination of claim 9 generates the first, second, and third MoE blocks. The motivation, rationale (MPEP 2143(I)(C)), and predictable result are those stated for claim 9. KSR, 550 U.S. at 416.
Claim 11 (Li in view of Riquelme, Shazeer, Lin, and Zoph)
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Riquelme, Shazeer, and Lin, and further in view of Zoph (Barret Zoph et al., "ST-MoE: Designing Stable and Transferable Sparse Expert Models," arXiv:2202.08906v2).
Claim 11 recites "applying a loading balancing loss and a router-z loss to each of the first sub-model, the second sub-model, and the third sub-model to maintain a load balance between experts in each of the first sub-model, the second sub-model, and the third sub-model during training of the machine learning model." Primary reference Li does not teach these losses. Secondary reference Riquelme teaches balancing losses applied to each expert block of the image encoder. It teaches an "importance loss" that encourages, "for each expert network block in the sequence of network blocks 110a-l, the routing scores corresponding to each expert module 130a-e in the expert network block to have the same mean" (Riquelme, ¶[0066]). It also teaches a "load loss" that encourages, "for each expert network block and for each patch, the likelihood that each expert module 130a-e of the expert network block is assigned the patch to have the same mean" (Riquelme, ¶[0069]). Secondary reference Lin teaches a load balancing loss for the language model: "We incorporate differentiable load balancing loss (Fedus et al., 2022) into each MoE layer to encourage experts to handle tokens in a balanced manner" (Lin, p. 5). Li, Riquelme, and Lin do not explicitly teach "a router-z loss," however, secondary reference Zoph teaches this feature as follows: "To fix this, we introduce the router z-loss" (Zoph, p. 7), which "penalizes large logits into the gating network" (Zoph, p. 7). "The total loss is a linearly weighted combination of the cross entropy loss (LCE), the auxiliary load balance loss (LB), and the router z-loss (LZ)" (Zoph, p. 8). Applying both losses to each MoE-equipped sub-model of the combined model, as Riquelme applies its losses to "each expert network block" and Lin applies its loss "into each MoE layer," is applying them to each of the first, second, and third sub-models to maintain load balance. Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to apply Zoph's router z-loss, together with the load-balancing losses of Riquelme and Lin, to each MoE-equipped sub-model of the combined model during training, because Zoph recommends "a new auxiliary loss, the router z-loss, which significantly improves training stability with no quality degradation" (Zoph, p. 5), and Lin teaches that "it is necessary to impose load balancing constraints on the MoE layer" (Lin, p. 5). The rationale is MPEP 2143(I)(A), because Each auxiliary loss performs the same known function when added to the training objective and the predictable result is balanced and stable training of the experts in every sub-model. KSR, 550 U.S. at 416.
Claim 12 (Li in view of Riquelme and Shazeer)
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Riquelme and Shazeer.
Claim 12 recites "A system of generating image descriptions using a machine learning model, comprising: at least one processor; and at least one memory communicatively coupled to the at least one processor and comprising computer-readable instructions that upon execution by the at least one processor cause the at least one processor to perform operations." Li teaches this feature: "computing device 600 includes a processor 610 coupled to memory 620" (Li, ¶[0058]). "memory 620 may include non-transitory, tangible, machine readable media that includes executable code that when run by one or more processors (e.g., processor 610) may cause the one or more processors to perform the methods described in further detail herein" (Li, ¶[0061]). No secondary reference is needed to meet this limitation.
Claim 12 further recites the operations of "configuring the machine learning model by incorporating Mixture of Experts (MoE) blocks into a plurality of sub-models," "generating visual tokens by the first sub-model, wherein only a subset of the first plurality of experts is activated," "projecting the visual tokens by the second sub-model, wherein only a subset of the second plurality of experts is activated," and "outputting a text description of the input image by the third sub-model." These operations are identical to the steps of claim 1. They are met by Li in view of Riquelme and Shazeer for the reasons given for claim 1, which are incorporated here, with the same motivation, rationale (MPEP 2143(I)(D)), and predictable result.
Claims 13 and 18 (Li in view of Riquelme and Shazeer)
Claims 13 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Riquelme and Shazeer.
Claim 13 recites "generating representations of the input image based on performing self-attention and normalization by the first sub-model; routing the representations of the input image to the activated subset of the first plurality of experts by a router of the at least one first MoE block; and generating the visual tokens by calculating a weighted sum of outputs from the activated subset of the first plurality of experts." Claim 18 recites the same operations in the storage medium of claim 17. These limitations combine those of claims 2 and 3. They are met by Li in view of Riquelme and Shazeer for the reasons given for claims 2 and 3 (Riquelme, ¶[0039], ¶[0060], ¶[0087], ¶[0093]–[0094]), which are incorporated here, including the motivation and rationale (MPEP 2143(I)(D)) stated there.
Claim 16 (Li in view of Riquelme, Shazeer, Lin, and Komatsuzaki)
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Riquelme, Shazeer, and Lin, and further in view of Komatsuzaki.
Claim 16 recites "generating an initial expert for each of the first sub-model, the second sub-model, and the third sub-model based on the pre-finetuned parameters; generating at least one initial expert block for each of the first sub-model, the second sub-model, and the third sub-model by replicating the initial expert; and obtaining the at least one first MoE block of the first sub-model, the at least one second MoE block of the second sub-model, and at least one third MoE block of the third sub-model by iteratively training the at least one initial expert block for each of the first sub-model, the second sub-model, and the third sub-model." These limitations combine those of claims 9 and 10. They are met by Li in view of Riquelme, Shazeer, Lin, and Komatsuzaki for the reasons given for claims 9 and 10, which are incorporated here, including the motivation and rationale (MPEP 2143(I)(C)) stated there.
Claim 17 (Li in view of Riquelme and Shazeer)
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Riquelme and Shazeer.
Claim 17 recites "A non-transitory computer-readable storage medium, storing computer-readable instructions that upon execution by a processor cause the processor to implement operations." Li teaches this feature: "memory 620 may include non-transitory, tangible, machine readable media that includes executable code that when run by one or more processors (e.g., processor 610) may cause the one or more processors to perform the methods described in further detail herein" (Li, ¶[0061]). No secondary reference is needed to meet this limitation. The remaining operations of claim 17 are "configuring the machine learning model by incorporating Mixture of Experts (MoE) blocks into a plurality of sub-models" (interpreted as "a machine learning model," see the rejection under 35 U.S.C. 112(b)), "generating visual tokens by the first sub-model," "projecting the visual tokens by the second sub-model," and "outputting a text description of the input image by the third sub-model." These operations are identical to the steps of claim 1 and are met by Li in view of Riquelme and Shazeer for the reasons given for claim 1, which are incorporated here.

Any comments considered necessary by applicant must be submitted no later than the payment of the issue fee and, to avoid processing delays, should preferably accompany the issue fee. Such submissions should be clearly labeled “Comments on Statement of Reasons for Allowance.”

Conclusion
Prior Art of Record
Li et al., US 2024/0160853 A1, published May 16, 2024; Appl. No. 18/160,722, filed January 27, 2023; provisional No. 63/424,413, filed November 10, 2022; first-named inventor Junnan Li; assignee Salesforce, Inc. Prior art under 35 U.S.C. 102(a)(2). Li describes a multimodal vision-language model made of three component networks in series: a pretrained vision-transformer image encoder; a lightweight query Transformer followed by a fully connected layer that projects the visual representation into the word-embedding dimension of a large language model; and a large language model that generates output text such as image captions and answers to questions about the image. Li also describes staged training in which the connecting query Transformer is trained while the image encoder and the language model are frozen. Li is relevant because it discloses the encoder–connector–LLM pipeline that the claims recite, apart from the mixture-of-experts features.
Riquelme Ruiz et al., WO 2022/251717 A1, published December 1, 2022; International Application No. PCT/US2022/031466, filed May 27, 2022; priority to U.S. Provisional Application No. 63/194,926, filed May 28, 2021; first-named inventor Carlos Riquelme Ruiz; applicant Google LLC. Prior art under 35 U.S.C. 102(a)(1). The U.S. national stage of this application published as US 2024/0289926 A1 with the same text and paragraph numbering. Riquelme describes a vision transformer whose network blocks apply layer normalization and self-attention to image-patch representations, then route each patch representation to the k expert modules with the highest routing scores. Each expert is an MLP, unselected experts stay idle, and the selected experts' outputs are combined as a sum weighted by the routing scores. It also describes importance and load losses that balance the use of experts in each expert block, and states that the resulting image embedding can be passed to downstream machine learning models. Riquelme is relevant because it teaches the claimed sparse MoE block inside an image encoder.
Shazeer et al., US 2023/0419079 A1, published December 28, 2023; Appl. No. 18/244,171, filed September 8, 2023 (continuation of Appl. Nos. 16/879,187 and 16/393,063 and PCT/US2017/059909); first-named inventor Noam M. Shazeer; assignee Google LLC. Prior art under 35 U.S.C. 102(a)(1). Shazeer describes a sparsely gated mixture-of-experts subnetwork that may be inserted between any layers of a neural network, and whose neighboring layers may themselves be MoE subnetworks. A gating subsystem selects the top k feed-forward experts for each input, only the selected experts compute outputs, and the outputs are combined as a weighted sum. Shazeer is relevant because it teaches that sparse MoE is a general-purpose replacement for feed-forward layers in any network, including the feed-forward layers of the connector.
Komatsuzaki et al., "Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints," arXiv:2212.05055 (v1, December 2022); first-named author Aran Komatsuzaki; co-authors of Google Research. Prior art under 35 U.S.C. 102(a)(1). Komatsuzaki describes converting a pretrained dense Transformer, including a Vision Transformer, into a sparse MoE model. Each expert in a new MoE layer is initialized as a copy of the original MLP, a new router is added, and training then continues. Komatsuzaki is relevant to the expert-replication and iterative-training steps of claims 9, 10, and 16, applied to a vision encoder as well as a language model.
Lin et al., "MoE-LLaVA: Mixture of Experts for Large Vision-Language Models," arXiv:2401.15947 (v1, January 2024); first-named author Bin Lin. Prior art under 35 U.S.C. 102(a)(1). Lin describes a large vision-language model with a vision encoder, an MLP projection layer, and an LLM in which top-k sparse MoE layers replace feed-forward layers. It trains the model in three stages: the projection layer first, then fine-tuning of the model, then initialization of the MoE experts by replicating the trained feed-forward weights, followed by further training with a load-balancing loss. Lin is relevant because it applies sparse MoE to the language model of the same type of multimodal model the claims recite, and teaches the claimed staged training.
Zoph et al., "ST-MoE: Designing Stable and Transferable Sparse Expert Models," arXiv:2202.08906v2, April 29, 2022; first-named author Barret Zoph. Prior art under 35 U.S.C. 102(a)(1). Zoph describes training sparse mixture-of-experts transformers with an auxiliary load-balancing loss and introduces the router z-loss for training stability, combining both with the cross-entropy loss. Zoph is relevant to the two auxiliary losses recited in claim 11.
Shi et al., "When Do We Not Need Larger Vision Models?," arXiv:2403.13043 (v1, March 2024); first-named author Baifeng Shi. Prior art under 35 U.S.C. 102(a)(1). Shi describes an "S2-Wrapper" that runs a pretrained vision encoder over multiple image scales. The larger-scale images are split into sub-images of the pretraining size, their features are merged and average-pooled back to the base feature-map size, and the features of the different scales are concatenated along the channel dimension, so the number of output tokens passed to a multimodal LLM is unchanged. Shi is relevant to the multi-resolution feature processing recited in claim 4.

PERTINENT PRIOR ART
The prior art made of record and not relied upon is considered pertinent to Applicant's disclosure.
Mustafa et al., US 2025/0217641 A1, published July 3, 2025; Appl. No. 18/867,170 (§ 371 of PCT/US2023/022977, PCT filed May 19, 2023); provisional No. 63/344,036, filed May 19, 2022; first-named inventor Basil Mustafa; assignee Google LLC. Mustafa describes a multi-modal neural network in which image and text inputs are embedded by modality-specific embedding engines and processed by a shared backbone with sparsely gated mixture-of-experts layers. It uses an entropy-based regularization scheme that "enhances training stability and encourages balanced expert utilization" (Mustafa, ¶[0027]). Mustafa teaches sparse MoE applied to image tokens in a multimodal model and expert-balancing losses relevant to claim 11. Mustafa does not teach a separate projecting connector between an image encoder and a language model, generation of a text description by a language model, or staged training with replicated experts.
Jenatton et al., US 2023/0107409 A1, published April 6, 2023; Appl. No. 17/960,780, filed October 5, 2022; provisional No. 63/252,614, filed October 5, 2021; first-named inventor Rodolphe Jenatton; assignee Google LLC. Jenatton describes neural networks, including Vision Transformer networks that process image patches, with expert blocks whose routers assign inputs to the experts with the highest routing scores and sum the expert outputs. It states that "selectively activating subsets of the parameters of the neural network based on the network input, significantly improving the time and computational efficiency of the neural network" (Jenatton, ¶[0006]). Jenatton is directed to ensembling several representations through the expert blocks. It does not teach a multimodal encoder–connector–LLM model or image description generation, and it is cumulative of Riquelme as to sparse MoE in a vision transformer.
Dai et al., US 2024/0160858 A1, published May 16, 2024; Appl. No. 18/505,982, filed November 9, 2023; continuation-in-part of Appl. No. 18/160,664; provisional Nos. 63/500,551 (May 5, 2023) and 63/424,413 (November 10, 2022); first-named inventor Wenliang Dai; assignee Salesforce, Inc. Dai describes instruction tuning of a vision-language model in which a query-transformer "multimodal encoder" extracts an instruction-aware image representation from a frozen image encoder and passes it, together with the text instruction, to a large language model that generates output text. For example, given the instruction "Can you tell me about this image in detail?" the model generates a detailed description of the image (Dai, ¶[0024]). Dai also converts captioning datasets into instruction data with templates such as "<image> Write a short description for the image" (Dai, ¶[0039]). Dai teaches visual instruction tuning data and text-query-conditioned image description relevant to claims 7 and 8. Dai does not teach mixture-of-experts blocks in any sub-model, sparse activation of experts, or the replication of trained experts, and it is cumulative of Li as to the encoder–connector–LLM structure.
Shazeer et al., US 2019/0251423 A1, published August 15, 2019; Appl. No. 16/393,063, filed April 24, 2019 (issued as U.S. Patent No. 10,719,761); continuation of PCT/US2017/059909; first-named inventor Noam M. Shazeer; assignee Google LLC. This earlier publication in the same family as the Shazeer reference applied above discloses the same sparsely gated MoE subnetwork. That includes the multi-level hierarchical MoE in which "the gating subsystem 110 can include a parent gating subnetwork and multiple child gating subnetworks" (Shazeer '423, ¶[0035]), and the teaching that conditional computation increases capacity "without a proportional increase in computation" (Shazeer '423, ¶[0036]). It establishes that the sparse MoE technique was publicly available by August 2019. It does not teach application of MoE to vision encoders, projection connectors, or multimodal language models, or any staged training for MoE initialization.

THIS ACTION IS MADE FINAL
Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
THIS ACTION IS MADE FINAL
Applicant’s amendment necessitated new grounds of rejection and FINAL Rejection.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.

Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW T. SITTNER whose telephone number is [phone] and email: [e-mail]. The examiner can normally be reached on Monday-Friday, 8:00am - 5:00pm (Mountain Time Zone). Please schedule interview requests via email: [e-mail]
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sarah M. Monfeldt can be reached on [phone].
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW T SITTNER/
Primary Examiner, Art Unit 3629b

Prosecution Timeline

• Sep 06, 2024: Application Filed
• Sep 28, 2026: Non-Final Rejection mailed — §101, §102, §103 (current)