Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1, 12, 14-15, 17-20 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Anandkumar (US 20240265690 A1).
Regarding claims 1 and 20 Anandkumar teaches
(claim 1) A multimodal data generation method, comprising:
(claim 20) An electronic device, comprising: one or more processors ([0028] For instance, various functions may be carried out by a processor executing instructions stored in memory.); a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for:
obtaining a query data sequence, wherein the query data sequence comprises at least one data segment, and wherein each data segment of the at least one data segment corresponds to one data modality (Anandkumar ¶[0035] For example, given multi-modal tokens (encoded from an input image and corresponding multi-modal labels) and a question [query data sequence] as the prefix, the expert ensemble vision-language system 100 generates an answer for the visual question answering task; given the multi-modal tokens, the expert ensemble vision-language system 100 generates a caption for the image captioning task. In response to a prefix prompt, the predicted text [response data sequence] may be sampled in an autoregressive manner , as in an open-ended setting; or a ranking of the log-likelihood may be performed from a fixed set of completions, as in a closed-ended setting.); and
inputting the query data sequence into a multimodal model, to obtain a plurality of tokens in a response data sequence output sequentially by the multimodal model, wherein a current token among the plurality of tokens is generated through the following operations (Anandkumar ¶[0035] For example, given multi-modal tokens (encoded from an input image and corresponding multi-modal labels) and a question [query data sequence] as the prefix, the expert ensemble vision-language system 100 generates an answer for the visual question answering task; given the multi-modal tokens, the expert ensemble vision-language system [multimodal model] 100 generates a caption for the image captioning task. In response to a prefix prompt, the predicted text [response data sequence] may be sampled in an autoregressive manner , as in an open-ended setting; or a ranking of the log-likelihood may be performed from a fixed set of completions, as in a closed-ended setting.):
in response to determining that the current token belongs to a first data modality, inputting the query data sequence and a current response data sequence into the multimodal model, so that the multimodal model generates the current token based on the query data sequence and the current response data sequence, (Anandkumar ¶[0035] For example, given multi-modal tokens (encoded from an input image and corresponding multi-modal labels) and a question [query data sequence] as the prefix, the expert ensemble vision-language system 100 generates an answer for the visual question answering task; given the multi-modal tokens, the expert ensemble vision-language system [multimodal model] 100 generates a caption for the image captioning task. In response to a prefix prompt, the predicted text [response data sequence; generate current token] may be sampled in an autoregressive manner , as in an open-ended setting; or a ranking of the log-likelihood may be performed from a fixed set of completions, as in a closed-ended setting)
wherein values of unit data of the first data modality are discrete (Anandkumar ¶[0035] For example, given multi-modal tokens (encoded from an input image and corresponding multi-modal labels) and a question [query data sequence] as the prefix, the expert ensemble vision-language system 100 generates an answer for the visual question answering task; given the multi-modal tokens, the expert ensemble vision-language system [multimodal model] 100 generates a caption for the image captioning task. In response to a prefix prompt, the predicted text [response data sequence; generate current token] may be sampled in an autoregressive manner , as in an open-ended setting; or a ranking of the log-likelihood may be performed from a fixed set of completions, as in a closed-ended setting) (Anandkumar ¶[0035] For example, given multi-modal tokens (encoded from an input image and corresponding multi-modal labels) and a question [query data sequence] as the prefix, the expert ensemble vision-language system 100 generates an answer for the visual question answering task; given the multi-modal tokens, the expert ensemble vision-language system [multimodal model] 100 generates a caption for the image captioning task. In response to a prefix prompt, the predicted text [response data sequence; generate current token] may be sampled in an autoregressive [autoregressive=discrete]manner , as in an open-ended setting; or a ranking of the log-likelihood may be performed from a fixed set of completions, as in a closed-ended setting)
or in response to determining that the current token belongs to a second data modality, inputting the query data sequence and the current response data sequence into the multimodal model, so that the multimodal model denoises an initial token sequence based on the query data sequence and the current response data sequence, to generate a result token sequence, wherein values of unit data of the second data modality are continuous, wherein the initial token sequence comprises a preset quantity of initial tokens, and wherein the result token sequence comprises the preset quantity of tokens starting from the current token. Examiner Note: This limitation is not mapped because of “or”.
With respect to claim 12 Anandkumar teaches wherein the first data modality comprises any one of the following: natural language text, tables, code, SMILES molecular formulas, or protein sequences (Anandkumar ¶[0035] For example, given multi-modal tokens (encoded from an input image and corresponding multi-modal labels) and a question [query data sequence] as the prefix, the expert ensemble vision-language system 100 generates an answer for the visual question answering task;); and
the second data modality comprises any one of the following: images, videos, audio, or point cloud data (Anandkumar ¶[0035] For example, given multi-modal tokens (encoded from an input image and corresponding multi-modal labels) and a question [query data sequence] as the prefix, the expert ensemble vision-language system 100 generates an answer for the visual question answering task;).
With respect to claim 14, Anandkumar teaches A multimodal model training method, comprising: obtaining a sample data sequence, wherein the sample data sequence comprises at least one data segment, and wherein each data segment of the at least one data segment corresponds to one data modality (Anandkumar ¶[0035] For example, given multi-modal tokens (encoded from an input image and corresponding multi-modal labels) and a question [query data sequence] as the prefix, the expert ensemble vision-language system 100 generates an answer for the visual question answering task; given the multi-modal tokens, the expert ensemble vision-language system 100 generates a caption for the image captioning task. In response to a prefix prompt, the predicted text [response data sequence] may be sampled in an autoregressive manner , as in an open-ended setting; or a ranking of the log-likelihood may be performed from a fixed set of completions, as in a closed-ended setting.);
inputting the sample data sequence into a multimodal model, to obtain a plurality of tokens in a predicted data sequence output sequentially by the multimodal model, wherein a current token among the plurality of tokens is generated through the following steps (Anandkumar ¶[0035] For example, given multi-modal tokens (encoded from an input image and corresponding multi-modal labels) and a question [query data sequence] as the prefix, the expert ensemble vision-language system 100 generates an answer for the visual question answering task; given the multi-modal tokens, the expert ensemble vision-language system 100 generates a caption for the image captioning task. In response to a prefix prompt, the predicted text [response data sequence] may be sampled in an autoregressive manner , as in an open-ended setting; or a ranking of the log-likelihood may be performed from a fixed set of completions, as in a closed-ended setting.):
in response to determining that the current token belongs to a first data modality, inputting the sample data sequence and a current predicted data sequence into the multimodal model, so that the multimodal model generates the current token based on the sample data sequence and the current predicted data sequence, wherein values of unit data of the first data modality are discrete (Anandkumar ¶[0035] For example, given multi-modal tokens (encoded from an input image and corresponding multi-modal labels) and a question [query data sequence] as the prefix, the expert ensemble vision-language system 100 generates an answer for the visual question answering task; given the multi-modal tokens, the expert ensemble vision-language system 100 generates a caption for the image captioning task. In response to a prefix prompt, the predicted text [response data sequence] may be sampled in an autoregressive manner , as in an open-ended setting; or a ranking of the log-likelihood may be performed from a fixed set of completions, as in a closed-ended setting.);
or in response to determining that the current token belongs to a second data modality, inputting the sample data sequence and the current predicted data sequence into the multimodal model, so that the multimodal model denoises an initial token sequence based on the sample data sequence and the current predicted data sequence, to generate a result token sequence, wherein values of unit data of the second data modality are continuous, wherein the initial token sequence comprises a preset quantity of initial tokens, and wherein the result token sequence comprises the preset quantity of tokens starting from the current token Examiner Note: This limitation is not mapped because of “or”. ;and
adjusting a parameter of the multimodal model based on a difference between the predicted data sequence and a target data sequence corresponding to the sample data sequence (Anandkumar ¶[0032] In an embodiment, the expert ensemble vision-language system 100 is trained in a supervised manner using a training dataset that includes images (and optional input questions) paired with ground truth text (reference text). A loss function 150 receives the predicted text output by the language decoder 140 and reference text corresponding to the image. The loss function 150 compares the predicted text with the reference text and updates parameters of the experts resampler 120 and adaptors within the vision encoder 130 and the language decoder 140. Because the parameters of the pre-trained experts 120 are already trained, the number of parameters that are modified during training of the expert ensemble vision-language system 100 is significantly reduced compared with a conventional monolithic neural network. In an embodiment, the trainable parameters are approximately 20% of the total number of parameters for the expert ensemble vision-language system 100 populated with the pre-trained experts 110.[Note: training adjusts weights]).
With respect to claim 15 Anandkumar teaches wherein the sample data sequence comprises at least two of the following: a first sample data sequence that only comprises a data segment of the first data modality; a second sample data sequence that only comprises a data segment of the second data modality; or a third sample data sequence that comprises interleaved data segments of the first data modality and the second data modality ((Anandkumar ¶[0035] For example, given multi-modal tokens (encoded from an input image and corresponding multi-modal labels) and a question [query data sequence] as the prefix, the expert ensemble vision-language system 100 generates an answer for the visual question answering task; given the multi-modal tokens, the expert ensemble vision-language system 100 generates a caption for the image captioning task. In response to a prefix prompt, the predicted text [response data sequence] may be sampled in an autoregressive manner , as in an open-ended setting; or a ranking of the log-likelihood may be performed from a fixed set of completions, as in a closed-ended setting.)
With respect to claim 17 Anandkumar teaches wherein the sample data sequence only comprises a data segment of the first data modality, and wherein, the target data sequence is the sample data sequence itself (Anandkumar ¶[0032] In an embodiment, the expert ensemble vision-language system 100 is trained in a supervised manner using a training dataset that includes images (and optional input questions) paired with ground truth text [target](reference text). A loss function 150 receives the predicted text output by the language decoder 140 and reference text corresponding [first modality ] to the image. The loss function 150 compares the predicted text with the reference text and updates parameters of the experts resampler 120 and adaptors within the vision encoder 130 and the language decoder 140. Because the parameters of the pre-trained experts 120 are already trained, the number of parameters that are modified during training of the expert ensemble vision-language system 100 is significantly reduced compared with a conventional monolithic neural network. In an embodiment, the trainable parameters are approximately 20% of the total number of parameters for the expert ensemble vision-language system 100 populated with the pre-trained experts 110.)
With respect to claim 18 Anandkumar teaches wherein the sample data sequence comprises at least a data segment of the second data modality, and the method further comprises: obtaining a first data sequence, wherein the first data sequence comprises at least a first data segment of the second data modality; performing a plurality of noise addition operations on the first data segment, to obtain a plurality of second data segments corresponding to the plurality of noise addition operations respectively, thereby obtaining a plurality of second data sequences corresponding to the plurality of noise addition operations respectively, wherein the sample data sequence is any one of the plurality of second data sequences, and the target data sequence is the first data sequence(Anandkumar ¶ [0060] In an embodiment, results also show that the ensemble vision-language system 100 maintains performance even when including experts that predict noise, as shown in graph 350 of FIG. 3D. Interestingly, adding noise [noise addition] can even result in a non-trivial improvement compared to training on RGB images alone, which can be considered as a form of implicit regularization. This property allows the ensemble vision-language system 100 to safely include many experts without degrading the performance, even when the expert is not necessarily informative. Therefore, the ensemble vision-language system 100 presents a more effective learning strategy than the standard multi-task or auxiliary learning methods, which either require exploring task relationships or designing more advanced optimization procedures.)
With respect to claim 19 Anandkumar teaches wherein the initial token sequence is obtained by segmenting a second data segment in the sample data sequence (Anandkumar ¶[0035] For example, given multi-modal tokens (encoded from an input image and corresponding multi-modal labels) and a question [second data segment] as the prefix, the expert ensemble vision-language system 100 generates an answer for the visual question answering task; given the multi-modal tokens, the expert ensemble vision-language system 100 generates a caption for the image captioning task. In response to a prefix prompt, the predicted text [initial token sequence] may be sampled in an autoregressive manner , as in an open-ended setting; or a ranking of the log-likelihood may be performed from a fixed set of completions, as in a closed-ended setting.)
.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 2 is (are) rejected under 35 U.S.C. 103 as being unpatentable over Anandkumar in further view of Hwang (US 12217003 B2) and Li (US 20090281791 A1)
With respect to claim2 Anandkumar does not explicitly disclose however Hwang teaches
wherein the plurality of tokens comprise a modality tag pair, the modality tag pair comprising a modality data start tag and a modality data end tag that indicate a same data modality, and there being at least one token belonging to a corresponding data modality between the modality data start tag and the modality data end tag, and the method further comprises (Hwang ¶Col4 ll41-48 Here, a tag is a grammatical mark that makes up the HTML document, and consists of angle brackets (<keyword>) that enclose a keyword indicating an attribute [modality] of the corresponding tag. And, most HTML elements are written with a start tag and an end tag [modality tag pair], and content thereof is placed between the start tag and the end tag. In this case, the end tag is distinguished from the start tag by putting a slash (</keyword>) in the angle brackets.):
data start tag with modality (Hwang ¶Col4 ll41-48 Here, a tag is a grammatical mark that makes up the HTML document, and consists of angle brackets (<keyword>) that enclose a keyword indicating an attribute [modality] of the corresponding tag. And, most HTML elements are written with a start tag and an end tag [modality tag pair], and content thereof is placed between the start tag and the end tag. In this case, the end tag is distinguished from the start tag by putting a slash (</keyword>) in the angle brackets.)
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the invention to modify token determination of Anandkumar to include the modality determination of Hwang in order to provide robust multimodal token generation that provides structural context.
None of Anandkumar and Hwang exp0lictly disclose however Li teaches determining a data modality to which the current token belongs based on a last modality [[data start tag]] in the current response data sequence (Li ¶[0008] Raw input text is received, and divided into sequences of tokens. Each token is marked with a text normalization tag that identifies a text normalization operation to be performed on the token during text normalization. The tags are assigned to the tokens by determining a most likely tag sequence [modality], given the sequence of tokens being processed [current response data]. The text normalization operations are performed on the tokens in order to provide clean output text, which can be output for further natural language processing.)
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the invention to modify token determination of Anandkumar in view of the modality determination of Hwang to include data modality determination of Li in order to provide robust multimodal token generation that provides structural context.
Claim(s) 3 is(are) rejected under 35 U.S.C. 103 as being unpatentable over Anandkumar, Hwang and Li in further view of Motoyama (US 12217003 B2)
With respect to claim 3 Li further teaches wherein the determining a data modality to which the current token belongs based on a last modality data start tag in the current response data sequence comprises: [[in response to determining that the last modality data start tag in the current response data sequence indicates the first data modality,]] determining that the current token belongs to the first data modality (Li ¶[0008] Raw input text is received, and divided into sequences of tokens. Each token is marked with a text normalization tag that identifies a text normalization operation to be performed on the token during text normalization. The tags are assigned to the tokens by determining a most likely tag sequence [modality], given the sequence of tokens being processed [current response data]. The text normalization operations are performed on the tokens in order to provide clean output text, which can be output for further natural language processing.);
None of Anandkumar, Hwang and Li explicitly disclose however Motoyama teaches in response to determining that the last modality data start tag in the current response data sequence indicates the first data modality, [[determining that the current token belongs to the first data modality]] (Motoyama ¶Col10ll22-31 In FIG. 9B, step 248 processes the start tag in order to determine the attributes [last modality data start tag], if any, of the start tag. For example, see lines 4, 10, etc. of FIG. 2. Step 250 calls the process to move down a hierarchical level which is illustrated in FIG. 10. The document being translated is arranged into different hierarchical levels and when a new hierarchical level is encountered which is indicated by a new start tag, it is necessary to adjust the data structures such as the data structures illustrated in FIG. 6 so that the proper dictionary and rule data base are created, if necessary, and utilized.)
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the invention to modify token determination of Anandkumar in view of the modality determination of Hwang in view of data modality determination of Li to include last modality start tag of Motoyama in order to properly align each token with its correct data.
or in response to determining that the last modality data start tag in the current response data sequence indicates the second data modality, determining that the current token belongs to the second data modality. Examiner Note: This limitation is not mapped because of “or”.
Claim(s) 4 is(are) rejected under 35 U.S.C. 103 as being unpatentable over Anandkumar, Hwang, Li and Motoyama in further view of Wong (US 20120254157 A1).
With respect to claim 4, none of Anandkumar, Hwang, Li and Motoyama explicitly disclose however Wong teaches further comprising: after that the result token sequence is generated, sequentially generating a modality data end tag indicating the second data modality, and a modality data start tag indicating the first data modality (Wong¶ [0046] where the markup data includes a plurality of elements each of which has a start tag, an end tag and content therebetween, as shown in block 42.)
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the invention to modify token determination of Anandkumar in view of the modality determination of Hwang in view of data modality determination of Li in view of last modality start tag of Motoyama to include modality tags of different types of Wong in order to seamlessly carry information of two different modalities in one stream.
Claim(s) 5 is (are) rejected under 35 U.S.C. 103 as being unpatentable over Anandkumar, Hwang and Li in further view of Cheslow (US 20060036631 A1) and Monro (US 20090019070 A1)
With respect to claim 5, none of Anandkumar, Hwang and Li explicitly disclose however Cheslow teaches wherein the modality data start tag and the modality data end tag are both tokens of the first data modality (Cheslow¶[0026] Token types include, for example, "Element name in start tag", "Element name in end tag", "Attribute Name", "Attribute Value", and CDATA (text), etc.),
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the invention to modify token determination of Anandkumar in view of the modality determination of Hwang in view of data modality determination of Li to include tags of first modality of Cheslow in order to identify different modalities.
None of Anandkumar, Hwang, Li and Cheslow explicitly disclose however Monro teaches and the method further comprises: in response to determining that the current response data sequence is empty, determining that the current token belongs to the first data modality (Monro ¶Claim 4. The method of claim 1 wherein the second symbol string code is generated for one or more empty symbol strings within the data grouping.)
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the invention to modify token determination of Anandkumar in view of the modality determination of Hwang in view of data modality determination of Li in view of tags of first modality of Cheslow to include assignment of default tags of Monro in order to initialize streams with default modality.
Claims 6-9 are not mapped as they are dependent on “or” of Claim 1 which is not mapped.
Claim(s) 10 is (are) rejected under 35 U.S.C. 103 as being unpatentable over Anandkumar in further view of Li and Cheslow.
With respect to claim 10 Anandkumar teaches obtaining an initial query data sequence entered by a user, wherein the initial query data sequence comprises at least one initial data segment, and wherein each initial data segment of the at least one initial data segment corresponds to one data modality (Anandkumar ¶[0035] For example, given multi-modal tokens (encoded from an input image and corresponding multi-modal labels) and a question [query data sequence] as the prefix, the expert ensemble vision-language system 100 generates an answer for the visual question answering task; given the multi-modal tokens, the expert ensemble vision-language system 100 generates a caption for the image captioning task. In response to a prefix prompt, the predicted text [response data sequence] may be sampled in an autoregressive manner , as in an open-ended setting; or a ranking of the log-likelihood may be performed from a fixed set of completions, as in a closed-ended setting.); and
Anandkumar does not explicitly disclose however Li teaches adding, for each initial data segment of the at least one initial data segment, a modality data start tag indicating a data modality of the initial data segment before the initial data segment (Li ¶[0008] Raw input text is received, and divided into sequences of tokens. Each token is marked with a text normalization tag that identifies a text normalization operation to be performed on the token during text normalization. The tags are assigned to the tokens by determining a most likely tag sequence [modality determined before the initial data segment], given the sequence of tokens being processed [current response data]. The text normalization operations are performed on the tokens in order to provide clean output text, which can be output for further natural language processing.)
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the invention to modify token determination of Anandkumar to include data modality determination of Li in order to provide robust multimodal token generation that provides structural context.
None of Anandkumar and Li explicitly disclose however Cheslow teaches and adding a modality data end tag indicating a data modality of the initial data segment after the initial data segment, to obtain the query data sequence (Cheslow¶[0026] Token types include, for example, "Element name in start tag", "Element name in end tag", "Attribute Name", "Attribute Value", and CDATA (text), etc.),
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the invention to modify token determination of Anandkumar in view of data modality determination of Li to include addition of tag of Cheslow in order to delineate different modalities.
Claim(s) 11 is (are) rejected under 35 U.S.C. 103 as being unpatentable over Anandkumar in further view of Upadhyay (US 20250053391 A1).
With respect to claim 11, Anandkumar does not explicitly disclose however Upadhyay teaches wherein the multimodal model is a transformer model that comprises only a decoder (Upadhyay ¶[0041] Transformers can include language transformers (e.g., BERT, GPT, T5, XLNet, ROBERTa, ALBERT, DistilBERT, ERNIE, etc.), vision transformers (e.g., VIT, data-efficient image transformers [DeiT], Swin transformers, convolutional vision transformers [CvT], multimodal transformers (e.g., contrastive language-image pre-training), DALL-E, VisualBERT, VideoBERT, encoder-only transformers, decoder-only transformers [decoder only]], encoder-decoder transformers, long range transformers, sparse transformers, and/or transformers of any other suitable type of transformer architecture.)
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the invention to modify token to include decoder only of Upadhyay in order to simplify model structure to speed up training.
Claim(s) 13 is (are) rejected under 35 U.S.C. 103 as being unpatentable over Anandkumar in further view of Baevski ( US 11551668 B1).
With respect to claim 13 Anandkumar does not explicitly disclose however Baevski teaches wherein the initial token sequence is a random noise sequence (Col10ll3-7 After pre-training we fine-tune the learned representations on labeled data and add a randomly initialized output layer on top of the Transformer to predict characters (Librispeech/Libri-light) or phonemes (TIMIT).)
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the invention to modify token to include randomized output of Baevski in order to trigger the autoregressive output to generate the sequence.
Claim(s) 16 is (are) rejected under 35 U.S.C. 103 as being unpatentable over Anandkumar in further view of Han (US 20220253678 A1).
With respect to claim 16 Anandkumar does not explicitly disclose however Han teaches wherein the method is performed by a plurality of electronic devices, any one of the plurality of electronic devices is configured to train the multimodal model by using sample data sequences of a same type, and a loss value of the multimodal model is a sum of respective local loss values of the plurality of electronic devices (Han ¶[0082] In an embodiment of the present disclosure, after the model training operation is delivered to the second device for distributed model training, the model training results of the second device are aggregated in the first device, so that the transmission of training data between the first device and the second device is avoided, which makes the method to be suitable for model training under multiple application scenarios (e.g., when the device manufacturer does not open data), and reduces the occupied bandwidth and reduces the difficulty of data security management, and at the same time, fully utilizes the parallel computing capability of multiple second devices to realize the scalability of the model training system, ¶ [0061]D.sub.ij is the number of training data corresponding to the jth training step uploaded by an ith second device, N is the number of second devices participating in the model training, and L.sub.ij is a loss function value corresponding to the jth training step uploaded by the ith device [sum of local loss].)
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the invention to modify token to include local loss values of Han in order to minimize hardware resources for training by doing distributed training.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ATHAR N PASHA whose telephone number is (408)918-7675. The examiner can normally be reached on Monday-Thursday Alternate Fridays, 7:30-4:30 PT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Washburn can be reached on (571)272-5551. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ATHAR N PASHA/Primary Examiner, Art Unit 2657
Prosecution Insights