Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is responsive to the application filed 03/29/2024. Claims 12-14 & 21-22 are canceled and 1-11, 15-20, & 23 are presented for examination.
Priority
Applicant’s claim for the benefit of prior filed application PCT/US2022/045805, filed 10/05/2022, and provisionally filed application 63/252593, filed 10/05/2021, is acknowledged.
Information Disclosure Statement
The information disclosure statement (IDS) submitted 04/28/2025 has been considered by the examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefore, subject to the conditions and requirements of this title.
Claims 1-11, 15-20, & 23 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claim 1
Step 1: The claim recites “A method of training a neural network to process respective input sequences corresponding to each of a plurality of modalities and to generate respective network outputs for the input sequences, wherein the neural network is configured to execute, for each of the plurality of modalities, one or more machine learning tasks corresponding to the modality, the training comprising:”; therefore, it is directed to the statutory category of a process.
Step 2A Prong 1: The claim recites, inter alia:
determining a plurality of patches of the network input, wherein each patch comprises a different subset of the elements of the network input: These limitations recite a mentally performable process with the aid of pen and paper of using observation and judgement to determine a plurality of patches of the network input, wherein each patch comprises a different subset of the elements of the network input.
processing the plurality of patches to generate an input sequence comprising a respective input element at each of a plurality of input positions, where some or all of the input elements correspond to respective different patches; processing the input sequence using the neural network to generate, for at least one of the one or more machine learning tasks corresponding to the particular modality, a respective predicted network output; and determining an update to a plurality of parameters of the neural network according to an error in the respective predicted network outputs: These limitations recite mathematical relationships similar to organizing information and manipulating information through mathematical correlations per MPEP 2106.04(a)(2)(I)(A)(iv).
Thus, the claim recites a judicial exception.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The additional elements of the claim are as follows:
A method of training a neural network to process respective input sequences corresponding to each of a plurality of modalities and to generate respective network outputs for the input sequences, wherein the neural network is configured to execute, for each of the plurality of modalities, one or more machine learning tasks corresponding to the modality, the training comprising: These additional elements are recited at a high level of generality and merely indicate a field of use or technological environment in which to apply a judicial exception, e.g. a method of training a neural network to process respective input sequences corresponding to each of a plurality of modalities and to generate respective network outputs for the input sequences, to a particular technological environment or field of use, e.g. wherein the neural network is configured to execute, for each of the plurality of modalities, one or more machine learning tasks corresponding to the modality, the training comprising. See MPEP 2106.05(h). Elements that use or interact with the judicial exception do not integrate the judicial exception into a practical application.
obtaining a network input corresponding to a particular modality and comprising a plurality of elements: These additional elements amount to insignificant extra-solution activity in the form of mere data gathering MPEP § 2106.05(g).
wherein the neural network comprises one or more self-attention neural network layers that are each configured to apply a self-attention mechanism to the input sequence or an intermediate representation of the input sequence: These additional elements are recited at a high level of generality and merely indicate a field of use or technological environment in which to apply a judicial exception, e.g. the neural network comprises one or more self-attention neural network layers, to a particular technological environment or field of use, e.g. that are each configured to apply a self-attention mechanism to the input sequence or an intermediate representation of the input sequence. See MPEP 2106.05(h). Elements that use or interact with the judicial exception do not integrate the judicial exception into a practical application.
and wherein at least a subset of the self-attention neural network layers are shared across the plurality of modalities: These additional elements are recited at a high level of generality and merely indicate a field of use or technological environment in which to apply a judicial exception, e.g. at least a subset of the self-attention neural network layers, to a particular technological environment or field of use, e.g. are shared across the plurality of modalities. See MPEP 2106.05(h). Elements that use or interact with the judicial exception do not integrate the judicial exception into a practical application.
Thus, the way in which the additional elements use or interact with the judicial exception do not integrate the judicial exception into a practical application.
Step 2B: The additional elements from Step 2A Prong 2 include generally linking the use of the judicial exception to indicate a field of use or technological environment and insignificant extra-solution activity of data gathering recited by “obtaining a network input corresponding to a particular modality and comprising a plurality of elements” which are well-understood routine and conventional activities similar to presenting offers and gathering statistics per MPEP 2106.05(d)(II). Thus, the additional elements, viewed individually or in combination, do not provide an inventive concept or otherwise amount to significantly more than the abstract idea itself. See MPEP § 2106.05.
Claim 2
Step 1: A process, as above.
Step 2A Prong 1: The claim recites the abstract ideas as the judicial exception of claim 1.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The additional elements of the claim are as follows:
wherein the plurality of modalities comprises one or more of an image modality, a video modality, or an audio modality: These additional elements are recited at a high level of generality and merely indicate a field of use or technological environment in which to apply a judicial exception, e.g. the plurality of modalities, to a particular technological environment or field of use, e.g. comprises one or more of an image modality, a video modality, or an audio modality. See MPEP 2106.05(h). Elements that use or interact with the judicial exception do not integrate the judicial exception into a practical application.
Thus, the way in which the additional elements use or interact with the judicial exception do not integrate the judicial exception into a practical application.
Step 2B: The additional elements from Step 2A Prong 2 include generally linking the use of the judicial exception to indicate a field of use or technological environment. Thus, the additional elements, viewed individually or in combination, do not provide an inventive concept or otherwise amount to significantly more than the abstract idea itself. See MPEP § 2106.05.
Claim 3
Step 1: A process, as above.
Step 2A Prong 1: The claim recites the abstract ideas of claim 1, as well as, inter alia:
processing the input sequence using one or more modality-specific network blocks corresponding to the particular modality to generate a first intermediate representation of the input sequence; processing the first intermediate representation using one or more shared network blocks to generate a second intermediate representation of the input sequence; and for each of the at least one machine learning tasks, processing the second intermediate representation using a respective task-specific network block to generate the respective predicted network output for the machine learning task: These limitations recite mathematical calculations similar to organizing information and manipulating information through mathematical correlations per MPEP 2106.04(a)(2)(I)(A)(iv).
Thus, the claim recites a judicial exception.
Step 2A Prong 2 & Step 2B: There are no additional elements recited so the claim does not provide a
practical application and is not considered to be significantly more. As such, the claim is patent
ineligible.
Claim 4
Step 1: A process, as above.
Step 2A Prong 1: The claim recites the abstract ideas of claim 1, as well as, inter alia:
identifying a set of training hyperparameters corresponding to the particular modality or the at least one machine learning task: These limitations recite a mentally performable process with the aid of pen and paper of using observation and judgement to identify a set of training hyperparameters corresponding to the particular modality or the at least one machine learning task.
and determining the update according to the identified set of training hyperparameters: These limitations recite a mentally performable process with the aid of pen and paper of using observation and judgement to determine the update according to the identified set of training hyperparameters.
Thus, the claim recites a judicial exception.
Step 2A Prong 2 & Step 2B: There are no additional elements recited so the claim does not provide a
practical application and is not considered to be significantly more. As such, the claim is patent
ineligible.
Claim 5
Step 1: A process, as above.
Step 2A Prong 1: The claim recites the abstract ideas as the judicial exception of claim 1.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The additional elements of the claim are as follows:
wherein at least 50%, at least 60%, at least 70%, at least 80%, or at least 90% of the plurality of parameters of the neural network are shared across the plurality of modalities: These additional elements are recited at a high level of generality and merely indicate a field of use or technological environment in which to apply a judicial exception, e.g. at least 50%, at least 60%, at least 70%, at least 80%, or at least 90% of the plurality of parameters of the neural network, to a particular technological environment or field of use, e.g. are shared across the plurality of modalities. See MPEP 2106.05(h). Elements that use or interact with the judicial exception do not integrate the judicial exception into a practical application.
Thus, the way in which the additional elements use or interact with the judicial exception do not integrate the judicial exception into a practical application.
Step 2B: The additional elements from Step 2A Prong 2 include generally linking the use of the judicial exception to indicate a field of use or technological environment. Thus, the additional elements, viewed individually or in combination, do not provide an inventive concept or otherwise amount to significantly more than the abstract idea itself. See MPEP § 2106.05.
Claim 6
Step 1: A process, as above.
Step 2A Prong 1: The claim recites the abstract ideas of claim 1, as well as, inter alia:
determining a respective update to the plurality of parameters of the neural network using each of a plurality of batches of network inputs: These limitations recite mathematical relationships similar to organizing information and manipulating information through mathematical correlations per MPEP 2106.04(a)(2)(I)(A)(iv).
Thus, the claim recites a judicial exception.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The additional elements of the claim are as follows:
wherein for each batch of network inputs, each network input in the batch corresponds to the same modality of the plurality of modalities: These additional elements are recited at a high level of generality and merely indicate a field of use or technological environment in which to apply a judicial exception, e.g. each batch of network inputs, to a particular technological environment or field of use, e.g. corresponds to the same modality of the plurality of modalities. See MPEP 2106.05(h). Elements that use or interact with the judicial exception do not integrate the judicial exception into a practical application.
Thus, the way in which the additional elements use or interact with the judicial exception do not integrate the judicial exception into a practical application.
Step 2B: The additional elements from Step 2A Prong 2 include generally linking the use of the judicial exception to indicate a field of use or technological environment. Thus, the additional elements, viewed individually or in combination, do not provide an inventive concept or otherwise amount to significantly more than the abstract idea itself. See MPEP § 2106.05.
Claim 7
Step 1: A process, as above.
Step 2A Prong 1: The claim recites the abstract ideas as the judicial exception of claim 6.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The additional elements of the claim are as follows:
wherein for each batch of network inputs, each network input in the batch corresponds to the same machine learning task of the one or more machine learning tasks corresponding to the modality of the batch: These additional elements are recited at a high level of generality and merely indicate a field of use or technological environment in which to apply a judicial exception, e.g. each network input in the batch, to a particular technological environment or field of use, e.g. corresponds to the same machine learning task of the one or more machine learning tasks corresponding to the modality of the batch. See MPEP 2106.05(h). Elements that use or interact with the judicial exception do not integrate the judicial exception into a practical application.
Thus, the way in which the additional elements use or interact with the judicial exception do not integrate the judicial exception into a practical application.
Step 2B: The additional elements from Step 2A Prong 2 include generally linking the use of the judicial exception to indicate a field of use or technological environment. Thus, the additional elements, viewed individually or in combination, do not provide an inventive concept or otherwise amount to significantly more than the abstract idea itself. See MPEP § 2106.05.
Claim 8
Step 1: A process, as above.
Step 2A Prong 1: The claim recites the abstract ideas of claim 6, as well as, inter alia:
wherein the training comprises determining a same number of updates to the plurality of parameters of the neural network for each of the plurality of modalities: These limitations recite mathematical relationships similar to organizing information and manipulating information through mathematical correlations per MPEP 2106.04(a)(2)(I)(A)(iv).
Thus, the claim recites a judicial exception.
Step 2A Prong 2 & Step 2B: There are no additional elements recited so the claim does not provide a
practical application and is not considered to be significantly more. As such, the claim is patent
ineligible.
Claim 9
Step 1: A process, as above.
Step 2A Prong 1: The claim recites the abstract ideas of claim 6, as well as, inter alia:
wherein the training comprises, for each modality of the plurality of modalities, determining a same number of updates to the plurality of parameters of the neural network for each machine learning task corresponding to the modality: These limitations recite mathematical relationships similar to organizing information and manipulating information through mathematical correlations per MPEP 2106.04(a)(2)(I)(A)(iv).
Thus, the claim recites a judicial exception.
Step 2A Prong 2 & Step 2B: There are no additional elements recited so the claim does not provide a
practical application and is not considered to be significantly more. As such, the claim is patent
ineligible.
Claim 10
Step 1: A process, as above.
Step 2A Prong 1: The claim recites the abstract ideas of claim 6, as well as, inter alia:
the training further comprises generating, for each machine learning task corresponding to each modality of the plurality of modalities, one or more batches of network inputs from a training data set corresponding to the machine learning task; and during training, batches of network inputs are sampled according to a size of the corresponding training data sets from which the batches were generated: These limitations recite mathematical relationships similar to organizing information and manipulating information through mathematical correlations per MPEP 2106.04(a)(2)(I)(A)(iv).
Thus, the claim recites a judicial exception.
Step 2A Prong 2 & Step 2B: There are no additional elements recited so the claim does not provide a
practical application and is not considered to be significantly more. As such, the claim is patent
ineligible.
Claim 11
Step 1: A process, as above.
Step 2A Prong 1: The claim recites the abstract ideas of claim 1, as well as, inter alia:
wherein the training further comprises determining a single update to the plurality of parameters of the neural network using multiple batches of network inputs corresponding to respective different modalities of the plurality of modalities: These limitations recite mathematical relationships similar to organizing information and manipulating information through mathematical correlations per MPEP 2106.04(a)(2)(I)(A)(iv).
Thus, the claim recites a judicial exception.
Step 2A Prong 2 & Step 2B: There are no additional elements recited so the claim does not provide a
practical application and is not considered to be significantly more. As such, the claim is patent
ineligible.
Claim 15
Step 1: The claim recites “A system comprising a first neural network that is configured to process input sequences corresponding to a single particular modality of a plurality of modalities and to generate respective outputs for the input sequences, the first neural network having been trained by performing operations comprising:”; therefore, it is directed to the statutory category of a machine.
Step 2A Prong 1: The claim recites, inter alia:
determining a plurality of patches of the network input, wherein each patch comprises a different subset of the elements of the network input: These limitations recite a mentally performable process with the aid of pen and paper of using observation and judgement to determine a plurality of patches of the network input, wherein each patch comprises a different subset of the elements of the network input.
processing the plurality of patches to generate an input sequence comprising a respective input element at each of a plurality of input positions, where some or all of the input elements correspond to respective different patches; processing the input sequence using the second neural network to generate, for at least one of one or more machine learning tasks corresponding to the particular modality, a respective predicted network output; and modifying an architecture of the second neural network to generate the first neural network: These limitations recite mathematical relationships similar to organizing information and manipulating information through mathematical correlations per MPEP 2106.04(a)(2)(I)(A)(iv).
Thus, the claim recites a judicial exception.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The additional elements of the claim are as follows:
A system comprising a first neural network that is configured to process input sequences corresponding to a single particular modality of a plurality of modalities and to generate respective outputs for the input sequences, the first neural network having been trained by performing operations comprising: These additional elements are recited at a high level of generality and merely indicate a field of use or technological environment in which to apply a judicial exception, e.g. A system comprising a first neural network, to a particular technological environment or field of use, e.g. that is configured to process input sequences corresponding to a single particular modality of a plurality of modalities and to generate respective outputs for the input sequences, the first neural network having been trained by performing operations comprising. See MPEP 2106.05(h). Elements that use or interact with the judicial exception do not integrate the judicial exception into a practical application.
training a second neural network to process respective input sequences corresponding to each of the plurality of modalities: These additional elements recite only the idea of a training a second neural network to process respective input sequences corresponding to each of the plurality of modalities and attempts to cover any implementation of training a second neural network without any restriction as to the specific type of neural network or how this process of training a second neural network to process input sequences that are corresponding to each of the plurality of modalities is implemented. These additional elements do not meaningfully limit the claim and does not integrate the judicial exception into a practical application because this type of recitation is equivalent to the words “apply it”. See MPEP 2106.05(f). Thus, the way in which the additional elements use or interact with the judicial exception do not integrate the judicial exception into a practical application.
obtaining a network input corresponding to a particular modality and comprising a plurality of elements: These additional elements amount to insignificant extra-solution activity in the form of mere data gathering MPEP § 2106.05(g).
wherein the second neural network comprises one or more self- attention neural network layers that are each configured to apply a self-attention mechanism to the input sequence or an intermediate representation of the input sequence: These additional elements are recited at a high level of generality and merely indicate a field of use or technological environment in which to apply a judicial exception, e.g. the second neural network, to a particular technological environment or field of use, e.g. comprises one or more self- attention neural network layers that are each configured to apply a self-attention mechanism to the input sequence or an intermediate representation of the input sequence. See MPEP 2106.05(h). Elements that use or interact with the judicial exception do not integrate the judicial exception into a practical application.
and wherein at least a subset of the self-attention neural network layers are shared across the plurality of modalities: These additional elements are recited at a high level of generality and merely indicate a field of use or technological environment in which to apply a judicial exception, e.g. at least a subset of the self-attention neural network layers, to a particular technological environment or field of use, e.g. are shared across the plurality of modalities. See MPEP 2106.05(h). Elements that use or interact with the judicial exception do not integrate the judicial exception into a practical application.
Thus, the way in which the additional elements use or interact with the judicial exception do not integrate the judicial exception into a practical application.
Step 2B: The additional elements from Step 2A Prong 2 include generally linking the use of the judicial exception to indicate a field of use or technological environment, adding words equivalent to "apply it" with the judicial exception, and insignificant extra-solution activity of data gathering recited by “obtaining a network input corresponding to a particular modality and comprising a plurality of elements” which are well-understood routine and conventional activities similar to presenting offers and gathering statistics per MPEP 2106.05(d)(II). Thus, the additional elements, viewed individually or in combination, do not provide an inventive concept or otherwise amount to significantly more than the abstract idea itself. See MPEP § 2106.05.
Claim 16
Step 1: A machine, as above.
Step 2A Prong 1: The claim recites the abstract ideas of claim 15, as well as, inter alia:
wherein modifying an architecture of the second neural network to generate the first neural network comprises removing one or more modality-specific network blocks corresponding to respective modalities that are different from the particular modality: These limitations recite mathematical relationships similar to organizing information and manipulating information through mathematical correlations per MPEP 2106.04(a)(2)(I)(A)(iv).
Thus, the claim recites a judicial exception.
Step 2A Prong 2 & Step 2B: There are no additional elements recited so the claim does not provide a
practical application and is not considered to be significantly more. As such, the claim is patent
ineligible.
Claim 17
Step 1: A process, as above.
Step 2A Prong 1: The claim recites the abstract ideas as the judicial exception of claim 15.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The additional elements of the claim are as follows:
wherein the first neural network is configured to execute a plurality of machine learning tasks corresponding to the particular modality: These additional elements are recited at a high level of generality and merely indicate a field of use or technological environment in which to apply a judicial exception, e.g. the first neural network, to a particular technological environment or field of use, e.g. is configured to execute a plurality of machine learning tasks corresponding to the particular modality. See MPEP 2106.05(h). Elements that use or interact with the judicial exception do not integrate the judicial exception into a practical application.
Thus, the way in which the additional elements use or interact with the judicial exception do not integrate the judicial exception into a practical application.
Step 2B: The additional elements from Step 2A Prong 2 include generally linking the use of the judicial exception to indicate a field of use or technological environment. Thus, the additional elements, viewed individually or in combination, do not provide an inventive concept or otherwise amount to significantly more than the abstract idea itself. See MPEP § 2106.05.
Claim 18
Step 1: A process, as above.
Step 2A Prong 1: The claim recites the abstract ideas as the judicial exception of claim 15.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The additional elements of the claim are as follows:
wherein the first neural network is configured to execute a single machine learning task corresponding to the particular modality: These additional elements are recited at a high level of generality and merely indicate a field of use or technological environment in which to apply a judicial exception, e.g. the first neural network, to a particular technological environment or field of use, e.g. is configured to execute a single machine learning task corresponding to the particular modality. See MPEP 2106.05(h). Elements that use or interact with the judicial exception do not integrate the judicial exception into a practical application.
Thus, the way in which the additional elements use or interact with the judicial exception do not integrate the judicial exception into a practical application.
Step 2B: The additional elements from Step 2A Prong 2 include generally linking the use of the judicial exception to indicate a field of use or technological environment. Thus, the additional elements, viewed individually or in combination, do not provide an inventive concept or otherwise amount to significantly more than the abstract idea itself. See MPEP § 2106.05.
Claim 19
Step 1: A machine, as above.
Step 2A Prong 1: The claim recites the abstract ideas of claim 15, as well as, inter alia:
wherein modifying an architecture of the second neural network to generate the first neural network comprises removing one or more task-specific network blocks corresponding to respective tasks for which the first neural network is not configured: These limitations recite mathematical relationships similar to organizing information and manipulating information through mathematical correlations per MPEP 2106.04(a)(2)(I)(A)(iv).
Thus, the claim recites a judicial exception.
Step 2A Prong 2 & Step 2B: There are no additional elements recited so the claim does not provide a
practical application and is not considered to be significantly more. As such, the claim is patent
ineligible.
Claim 20
Step 1: A machine, as above.
Step 2A Prong 1: The claim recites the abstract ideas of claim 15, as well as, inter alia:
modifying the architecture of the second neural network to generate an initial first neural network; and fine-tuning a plurality of parameters of the initial first neural network to generate the first neural network: These limitations recite mathematical relationships similar to organizing information and manipulating information through mathematical correlations per MPEP 2106.04(a)(2)(I)(A)(iv).
Thus, the claim recites a judicial exception.
Step 2A Prong 2 & Step 2B: There are no additional elements recited so the claim does not provide a
practical application and is not considered to be significantly more. As such, the claim is patent
ineligible.
Claim 23
Step 1: The claim is directed to “A method performed by one or more computers, the method comprising:”; therefore, it is directed the statutory category of a process.
Step 2A Prong 1: Claim 23 recites the same judicial exception as Claim 1.
Step 2A Prong 2: The judicial exception recited in the claim is not integrated into a practical application. The analysis at this step for Claim 23 mirrors that of Claim 1.
Step 2B: The additional elements from Step 2A Prong 2 do not contain significantly more than the judicial exception for the claim. The analysis at this step for Claim 23 mirrors that of Claim 1.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim 23 is rejected under 35 U.S.C. 102(a)(1) as being anticipated by Akbari et al. ("VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text," Paper, Presented at the 35th Conference on Neural Information Processing Systems, Virtual Event, December 6-14, 2021; Advances in Neural Information Processing Systems, December 2021, 34:1-20), hereafter Akbari.
Akbari was cited in the IDS submitted 04/28/2025.
Regarding independent claim 23, Akbari teaches a method performed by one or more computers ([Sec. 4] discusses implementing the method on a computer) comprising:
receiving a respective network input corresponding to each of a plurality of modalities, wherein each network input includes a respective plurality of elements ([Sec. 3-4] discusses receiving a network input corresponding to a plurality of modalities, and the input includes a plurality of elements);
and processing the respective network inputs using a neural network to generate respective network outputs for each of the network inputs ([Sec. 4] discusses producing an output by processing the input using a neural network), the processing comprising, for each network input: determining a plurality of patches of the network input, wherein each patch comprises a different subset of the elements of the network input ([Sec. 3.1 & 4] discusses determining patches of the input comprising a subset of the elements of input);
processing the plurality of patches to generate an input sequence comprising a respective input element at each of a plurality of input positions, where some or all of the input elements correspond to respective different patches ([Sec. 3.1-3.2] discusses mapping each patch into a sequence, wherein different input elements correspond to different patches);
and processing the input sequence using the neural network to generate, for at least one of one or more machine learning tasks corresponding to the modality corresponding to the network input, a respective predicted network output ([Sec. 3.2-3.3 & Sec. 4] discusses processing the input sequence using the neural network to generate an output corresponding to a particular modality);
wherein the neural network comprises one or more self-attention neural network layers that are each configured to apply a self-attention mechanism to the input sequence or an intermediate representation of the input sequence ([Sec. 3.2] discusses applying a self-attention layer on the input sequence);
and wherein at least a subset of the self-attention neural network layers are shared across the plurality of modalities ([Sec. 3] discusses applying the same backbone layer to each of the plurality of modalities).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 5-8, & 10 are rejected under 35 U.S.C. 103 as being unpatentable over Akbari et al. ("VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text," Paper, Presented at the 35th Conference on Neural Information Processing Systems, Virtual Event, December 6-14, 2021; Advances in Neural Information Processing Systems, December 2021, 34:1-20), hereafter Akbari, in view of Hu et al. ("UniT: Multimodal Multitask Learning with a Unified Transformer," Paper, Presented at Proceedings of the IEEE/CVF Conference: Computer Vision, Montreal, Canada, October 10-17, 2021, pages 1439-1449), hereafter Hu.
Akbari and Hu were cited in the IDS submitted 06/09/2026.
Regarding independent claim 1, Akbari teaches a method of training a neural network to process respective input sequences corresponding to each of a plurality of modalities and to generate respective network outputs for the input sequences ([Abstract & Sec. 1] discusses a method for training a neural network end to end to process input corresponding to modalities and generate output for the input), wherein the neural network is configured to execute, for each of the plurality of modalities, one or more machine learning tasks corresponding to the modality ([Sec. 3-4] discusses executing machine learning tasks for each specific modality), comprising:
obtaining a network input corresponding to a particular modality and comprising a plurality of elements ([Sec. 3.1] discusses obtaining a particular input modality comprising a plurality of elements);
determining a plurality of patches of the network input, wherein each patch comprises a different subset of the elements of the network input ([Sec. 3.1] discusses partitioning input into patches, wherein each patch is a different subset of the elements of input);
processing the plurality of patches to generate an input sequence comprising a respective input element at each of a plurality of input positions, where some or all of the input elements correspond to respective different patches ([Sec. 3.1-3.2] discusses mapping each patch into a sequence, wherein different input elements correspond to different patches);
processing the input sequence using the neural network to generate, for at least one of the one or more machine learning tasks corresponding to the particular modality, a respective predicted network output ([Sec. 3.2-3.3 & Sec. 4] discusses processing the input sequence using the neural network to generate an output corresponding to a particular modality);
wherein the neural network comprises one or more self-attention neural network layers that are each configured to apply a self-attention mechanism to the input sequence or an intermediate representation of the input sequence ([Sec. 3.2] discusses applying a self-attention layer on the input sequence);
and wherein at least a subset of the self-attention neural network layers are shared across the plurality of modalities ([Sec. 3] discusses applying the same backbone layer to each of the plurality of modalities);
and determining an update according to an error in the respective predicted network outputs ([Sec. 3.4] discusses calculating a loss to update a softness according to the loss in the predicted network output).
Akbari does not explicitly teach determining an update to a plurality of parameters of the neural network according to an error in the respective predicted network outputs.
However, in a similar field of endeavor, Hu teaches a method for multimodal multitask learning wherein an update to a plurality of parameters is done according to a calculated error in the outputs ([Sec. 3.4-3.5] discusses calculating an error on predicted network outputs; updates are then determined for a plurality of parameters according to the error).
Because Akbari teaches obtaining input, determining patches of input, processing the patches to generate an input sequence, processing the input sequence using a neural network built on self-attention layers that are shared across modalities to generate output, and determining an update according to an error; and Hu teaches determining an update to a plurality of parameters according to an error in the predicted network output, accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate determining an update to a plurality of parameters according to an error in the predicted network output as taught by Hu into Akbari’s method, with a reasonable expectation of success, to teach obtaining a network input corresponding to a particular modality and comprising a plurality of elements; determining a plurality of patches of the network input, wherein each patch comprises a different subset of the elements of the network input; processing the plurality of patches to generate an input sequence comprising a respective input element at each of a plurality of input positions, where some or all of the input elements correspond to respective different patches; processing the input sequence using the neural network to generate, for at least one of the one or more machine learning tasks corresponding to the particular modality, a respective predicted network output, wherein the neural network comprises one or more self-attention neural network layers that are each configured to apply a self-attention mechanism to the input sequence or an intermediate representation of the input sequence, and wherein at least a subset of the self-attention neural network layers are shared across the plurality of modalities; and determining an update to a plurality of parameters of the neural network according to an error in the respective predicted network outputs. This combination would have been motivated by the desire to allow for training of different detection datasets due to varying class, box, and attribute heads (Hu [Sec. 3]).
Regarding dependent claim 2, the combination of Akbari and Hu teaches the claimed invention as claimed in claim 1, including wherein the plurality of modalities comprises one or more of an image modality, a video modality, or an audio modality ([Sec. 3.1] discusses the plurality of modalities comprise audio and video modalities).
Regarding dependent claim 3, the combination of Akbari and Hu teaches the claimed invention as claimed in claim 1, including:
processing the input sequence using one or more modality-specific network blocks corresponding to the particular modality to generate a first intermediate representation of the input sequence (Hu [Sec. 3] discusses processing the sequence using modality-specific blocks to produce a hidden-state sequence; Akbari [Sec. 3.1] discusses generating a first representation of the input sequence with positional encoding);
processing the first intermediate representation using one or more shared network blocks to generate a second intermediate representation of the input sequence (Hu [Sec. 3.3] discusses using a shared network block on the first representation to generate a second representation of the input sequence);
and for each of the at least one machine learning tasks, processing the second intermediate representation using a respective task-specific network block to generate the respective predicted network output for the machine learning task (Hu [Sec. 3.4] discusses applying a task-specific head to each task to generate a predicted network output).
Regarding dependent claim 5, the combination of Akbari and Hu teaches the claimed invention as claimed in claim 1, including wherein at least 50%, at least 60%, at least 70%, at least 80%, or at least 90% of the plurality of parameters of the neural network are shared across the plurality of modalities (Akbari [Sec. 4] discusses at least 50% of the parameters are shared across the plurality of modalities; Hu [Abstract] discusses all the parameters are shared across all modalities).
Regarding dependent claim 6, the combination of Akbari and Hu teaches the claimed invention as claimed in claim 1, including determining a respective update to the plurality of parameters of the neural network using each of a plurality of batches of network inputs (Hu [Sec. 3.5] discusses determining an update to the parameters based on the plurality of batches);
wherein for each batch of network inputs, each network input in the batch corresponds to the same modality of the plurality of modalities (Hu [Sec. 3.5, & 4.1-4.2] discusses that for one task or modality, the inputs in the batch correspond to the selected task or modality).
Regarding dependent claim 7, the combination of Akbari and Hu teaches the claimed invention as claimed in claim 6, including wherein for each batch of network inputs, each network input in the batch corresponds to the same machine learning task of the one or more machine learning tasks corresponding to the modality of the batch (Hu [Sec. 3.5, & 4.1-4.2] discusses the network inputs are selected for the same machine learning task that corresponds to the modality of the batch).
Regarding dependent claim 8, the combination of Akbari and Hu teaches the claimed invention as claimed in claim 6, including wherein the training comprises determining a same number of updates to the plurality of parameters of the neural network for each of the plurality of modalities (Akbari [Sec. 4] discusses training steps update all modalities used; thus, the same number of updates is used for parameters for the plurality of modalities).
Regarding dependent claim 10, the combination of Akbari and Hu teaches the claimed invention as claimed in claim 6, including wherein: the training further comprises generating, for each machine learning task corresponding to each modality of the plurality of modalities, one or more batches of network inputs from a training data set corresponding to the machine learning task (Hu [Sec. 3.5] discusses generating batches of inputs based the machine learning task);
and during training, batches of network inputs are sampled according to a size of the corresponding training data sets from which the batches were generated (Hu [Sec. 3.5] discusses sampling based on the size of the datasets from which batches were generated).
Claims 4 & 9 are rejected under 35 U.S.C. 103 as being unpatentable over Akbari, in view of Hu, as applied in claims 1 & 6, and further in view of Lu et al. ("12-in-1: Multi-Task Vision and Language Representation Learning," Paper, Presented at Proceedings of the IEEE/CVF Conference: Computer Vision and Pattern Recognition, Seattle, WA, June 13-19, 2020, pages 10437-10446), hereafter Lu.
Lu was cited in the IDS submitted 06/09/2026.
Regarding dependent claim 4, the combination of Akbari and Hu teaches the claimed invention as claimed in claim 1, including identifying a set of training hyperparameters corresponding to the particular modality (Akbari [Sec. 4] discusses specifying settings and hyperparameters for each different modality).
The combination of Akbari and Hu does not explicitly teach determining the update according to the identified set of training hyperparameters.
However, in a similar field of endeavor, Lu teaches a method for multitask learning, wherein a set of training hyperparameters are identified ([Sec. 3.3] discusses identifying training hyperparameters for a machine learning task); and an update is determined according to the hyperparameters ([Sec. 3.3] discusses determining an update to the set of training hyperparameters).
Because the combination of Akbari and Hu teaches identifying a set of training hyperparameters; and Lu teaches determining an update according to the hyperparameters, accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate determining an update according to the hyperparameters as taught by Lu into the combination of Akbari and Hu’s method, with a reasonable expectation of success, to teach identifying a set of training hyperparameters corresponding to the particular modality or the at least one machine learning task; and determining the update according to the identified set of training hyperparameters. This combination would have been motivated by the desire to reduce the burden of searching for joint training hyper-parameters (Lu [Sec. 3]).
Regarding dependent claim 9, the combination of Akbari and Hu teaches the claimed invention as claimed in claim 6, including determining a respective update to the plurality of parameters of the neural network using each of a plurality of batches of network inputs (Hu [Sec. 3.5] discusses determining an update to the parameters based on the plurality of batches).
The combination of Akbari and Hu does not explicitly teach determining a same number of updates to the plurality of parameters of the neural network for each machine learning task corresponding to the modality.
However, in a similar field of endeavor, Lu teaches a method for multitask learning, wherein parameters are updated for a single machine learning task at once for each task ([Sec. 3.3] discusses updating parameters for a single machine task at once for each task corresponding to a modality; thus, the same number of updates are carried out for each machine learning task).
Because the combination of Akbari and Hu teaches updating a plurality of parameters; and Lu teaches determining a same number of updates to the parameters for each machine learning task, accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate determining a same number of updates to the parameters for each machine learning task as taught by Lu into the combination of Akbari and Hu’s method, with a reasonable expectation of success, to teach wherein the training comprises, for each modality of the plurality of modalities, determining a same number of updates to the plurality of parameters of the neural network for each machine learning task corresponding to the modality. This combination would have been motivated by the desire to avoid overfitting by reducing the chance to overtrain smaller tasks (Lu [Sec. 3]).
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Akbari, in view of Hu, as applied in claims 1, and further in view of Aghajanyan et al. ("Muppet: Massive Multi-task Representations with Pre-Finetuning", Facebook, arXiv) (Year: 2021), hereafter Aghajanyan.
Regarding dependent claim 11, the combination of Akbari and Hu teaches the claimed invention as claimed in claim 1, including determining an update to a plurality of parameters of the neural network (Akbari [Sec. 3.4] discusses calculating a loss to update a softness according to the loss in the predicted network output; Hu [Sec. 3.4-3.5] discusses calculating an error on predicted network outputs; updates are then determined for a plurality of parameters according to the error).
The combination of Akbari and Hu does not explicitly teach determining a single update to the plurality of parameters of the neural network using multiple batches of network inputs corresponding to respective different modalities of the plurality of modalities.
However, in a similar field of endeavor, Aghajanyan teaches a method for multi-task representation with fine-tuning, wherein a single update is determined to the plurality of parameters using multiple batches of input ([Sec. 5] discusses determining a single update to parameters using batches of multiple tasks corresponding to multiple modalities).
Because the combination of Akbari and Hu teaches determining an update to a plurality of parameters of the neural network; and Aghajanyan teaches determining a single update to the plurality of parameters of the neural network using multiple batches of input, accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate determining a single update to the plurality of parameters of the neural network using multiple batches of input as taught by Aghajanyan into the combination of Akbari and Hu’s method, with a reasonable expectation of success, to teach wherein the training further comprises determining a single update to the plurality of parameters of the neural network using multiple batches of network inputs corresponding to respective different modalities of the plurality of modalities. This combination would have been motivated by the desire to improve stability during training (Aghajanyan [Sec. 1, 2, & 5]).
Claims 15-20 are rejected under 35 U.S.C. 103 as being unpatentable over Akbari et al. ("VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text," Paper, Presented at the 35th Conference on Neural Information Processing Systems, Virtual Event, December 6-14, 2021; Advances in Neural Information Processing Systems, December 2021, 34:1-20), hereafter Akbari, in view of Hu et al. ("UniT: Multimodal Multitask Learning with a Unified Transformer," Paper, Presented at Proceedings of the IEEE/CVF Conference: Computer Vision, Montreal, Canada, October 10-17, 2021, pages 1439-1449), hereafter Hu, and further in view of Chen et al. ("Pre-Trained Image Processing Transformer", Key Lab of Machine Perception, arXiv) (Year: 2021), hereafter Chen.
Akbari and Hu were cited in the IDS submitted 06/09/2026.
Regarding independent claim 15, Akbari teaches a method comprising a first neural network that is configured to process input sequences corresponding to a single particular modality of a plurality of modalities and to generate respective outputs for the input sequences ([Sec. 4] discusses taking a modality and generating an output for the input sequence using a neural network), comprising:
training a second neural network to process respective input sequences corresponding to each of the plurality of modalities ([Sec. 1-2 & 4] discusses a shared backbone neural network is trained by initializing and training a fine-tuned model to process input sequences corresponding to modalities), the training comprising: obtaining a network input corresponding to a particular modality and comprising a plurality of elements ([Sec. 3.1] discusses obtaining a particular input modality comprising a plurality of elements);
determining a plurality of patches of the network input, wherein each patch comprises a different subset of the elements of the network input ([Sec. 3.1] discusses partitioning input into patches, wherein each patch is a different subset of the elements of input);
processing the plurality of patches to generate an input sequence comprising a respective input element at each of a plurality of input positions, where some or all of the input elements correspond to respective different patches ([Sec. 3.1-3.2] discusses mapping each patch into a sequence, wherein different input elements correspond to different patches);
processing the input sequence using the second neural network to generate, for at least one of one or more machine learning tasks corresponding to the particular modality, a respective predicted network output ([Sec. 3.2-3.3 & Sec. 4] discusses processing the input sequence using the neural network to generate an output corresponding to a particular modality);
wherein the second neural network comprises one or more self- attention neural network layers that are each configured to apply a self-attention mechanism to the input sequence or an intermediate representation of the input sequence ([Sec. 3.2] discusses applying a self-attention layer on the input sequence);
and wherein at least a subset of the self-attention neural network layers are shared across the plurality of modalities ([Sec. 3] discusses applying the same backbone layer to each of the plurality of modalities);
and determining an update of the second neural network according to an error in the respective predicted network outputs ([Sec. 3.4] discusses calculating a loss to update a softness according to the loss in the predicted network output);
and modifying an architecture ([Sec. 3.4] discusses changes the architecture for fine-tuning and resizing position embeddings).
Akbari does not explicitly teach determining an update to a plurality of parameters of the second neural network according to an error in the respective predicted network outputs.
However, in a similar field of endeavor, Hu teaches a method for multimodal multitask learning wherein an update to a plurality of parameters is done according to a calculated error in the outputs ([Sec. 3.4-3.5] discusses calculating an error on predicted network outputs; updates are then determined for a plurality of parameters according to the error).
Because Akbari teaches obtaining input, determining patches of input, processing the patches to generate an input sequence, processing the input sequence using a neural network built on self-attention layers that are shared across modalities to generate output, determining an update according to an error, and modifying an architecture; and Hu teaches determining an update to a plurality of parameters according to an error in the predicted network output, accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate determining an update to a plurality of parameters according to an error in the predicted network output as taught by Hu into Akbari’s method, with a reasonable expectation of success, to teach training a second neural network to process respective input sequences corresponding to each of the plurality of modalities, comprising obtaining a network input corresponding to a particular modality and comprising a plurality of elements; determining a plurality of patches of the network input, wherein each patch comprises a different subset of the elements of the network input; processing the plurality of patches to generate an input sequence comprising a respective input element at each of a plurality of input positions, where some or all of the input elements correspond to respective different patches; processing the input sequence using the second neural network to generate, for at least one of one or more machine learning tasks corresponding to the particular modality, a respective predicted network output, wherein the second neural network comprises one or more self-attention neural network layers that are each configured to apply a self-attention mechanism to the input sequence or an intermediate representation of the input sequence, and wherein at least a subset of the self-attention neural network layers are shared across the plurality of modalities; and determining an update to a plurality of parameters of the second neural network according to an error in the respective predicted network outputs; and modifying an architecture. This combination would have been motivated by the desire to allow for training of different detection datasets due to varying class, box, and attribute heads (Hu [Sec. 3]).
The combination of Akbari and Hu does not explicitly teach modifying an architecture of the second neural network to generate the first neural network.
However, in a similar field of endeavor, Chen teaches a method for modifying the architecture of a neural network to generate a first neural network ([Sec. 3] discusses modifying an initial network’s architecture by dropping network blocks to generate a new network).
Because the combination of Akbari and Hu teaches obtaining input, determining patches of input, processing the patches to generate an input sequence, processing the input sequence using a neural network built on self-attention layers that are shared across modalities to generate output, determining an update to a plurality of parameters according to an error in the predicted network output, and modifying an architecture; and Chen teaches modifying an architecture of a second neural network to generate the first neural network, accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate modifying an architecture of a second neural network to generate the first neural network as taught by Chen into the combination of Akbari and Hu’s method, with a reasonable expectation of success, to teach training a second neural network to process respective input sequences corresponding to each of the plurality of modalities, comprising obtaining a network input corresponding to a particular modality and comprising a plurality of elements; determining a plurality of patches of the network input, wherein each patch comprises a different subset of the elements of the network input; processing the plurality of patches to generate an input sequence comprising a respective input element at each of a plurality of input positions, where some or all of the input elements correspond to respective different patches; processing the input sequence using the second neural network to generate, for at least one of one or more machine learning tasks corresponding to the particular modality, a respective predicted network output, wherein the second neural network comprises one or more self-attention neural network layers that are each configured to apply a self-attention mechanism to the input sequence or an intermediate representation of the input sequence, and wherein at least a subset of the self-attention neural network layers are shared across the plurality of modalities; and determining an update to a plurality of parameters of the second neural network according to an error in the respective predicted network outputs; and modifying an architecture of the second neural network to generate the first neural network. This combination would have been motivated by the desire to save computation costs in execution of the network (Chen [Sec. 3]).
Regarding dependent claim 16, the combination of Akbari, Hu, and Chen teaches the claimed invention as claimed in claim 15, including removing one or more modality-specific network blocks corresponding to respective modalities that are different from the particular modality (Chen [Sec. 3] discusses dropping blocks of modalities that are unnecessary, or corresponding to a different modality).
Regarding dependent claim 17, the combination of Akbari, Hu, and Chen teaches the claimed invention as claimed in claim 15, including wherein the first neural network is configured to execute a plurality of machine learning tasks corresponding to the particular modality (Chen [Sec. 3-4] discusses executing a plurality of machine learning tasks corresponding to a particular modality).
Regarding dependent claim 18, the combination of Akbari, Hu, and Chen teaches the claimed invention as claimed in claim 15, including wherein the first neural network is configured to execute a single machine learning task corresponding to the particular modality (Akbari [Sec. 4] discusses the network executing a single machine learning task for a particular modality).
Regarding dependent claim 19, the combination of Akbari, Hu, and Chen teaches the claimed invention as claimed in claim 15, including wherein modifying an architecture of the second neural network to generate the first neural network comprises removing one or more task-specific network blocks corresponding to respective tasks for which the first neural network is not configured (Chen [Sec. 3-4] discusses dropping network blocks that are unnecessary; Akbari [Sec. 3-4] discusses the architecture of the multimodal network).
Regarding dependent claim 20, the combination of Akbari, Hu, and Chen teaches the claimed invention as claimed in claim 15, including modifying the architecture of the second neural network to generate an initial first neural network (Akbari [Sec. 3-4] discusses resizing position embeddings to generate an initial neural network);
and fine-tuning a plurality of parameters of the initial first neural network to generate the first neural network (Akbari [Sec. 3-4] discusses further fine-tuning the entire initial neural network, including resized buckets, end-to-end to generate the first neural network).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Bao et al. ("BEIT: BERT Pre-Training of Image Transformers", Microsoft, arXiv) (Year: 2021) ([Abstract] We introduce a self-supervised vision representation model BEIT, which stands for Bidirectional Encoder representation from Image Transformers. Following BERT (Devlin et al., 2019) developed in the natural language processing area, we propose a masked image modeling task to pretrain vision Transformers. Specifically, each image has two views in our pre-training, i.e., image patches (such as 16 × 16 pixels), and visual tokens (i.e., discrete tokens). We first “tokenize” the original image into visual tokens. Then we randomly mask some image patches and fed them into the backbone Transformer. The pre-training objective is to recover the original visual tokens based on the corrupted image patches. After pre-training BEIT, we directly fine-tune the model parameters on downstream tasks by appending task layers upon the pretrained encoder).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RILEY S ACOSTA whose telephone number is (571)272-8714. The examiner can normally be reached Monday-Thursday 6am-4pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer N Welch can be reached at (571)272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RILEY S ACOSTA/Examiner, Art Unit 2143
/JENNIFER N WELCH/Supervisory Patent Examiner, Art Unit 2143
Prosecution Insights