Notice of Pre-AIA or AIA Status
The present application, filed on or after December 04, 2024, is being examined under the first inventor to file provisions of the AIA .
Priority
Applicant claims the benefit U.S. Provisional Patent Application No. 63/602,301 filed on November 22, 2023. Claims 1-20 have been afforded the benefit of November 22, 2023 filing date.
Information Disclosure Statement
The IDS dated November 22, 2024 has been considered and placed in the application file.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim [ 1, 2, 3, 4, 11, 12, 13, 14, 15 ] are rejected under 35 U.S.C. 103 as being
unpatentable over Menefee (US 10,489,639 B2) in view of Hu (US 11,908,461 B2).
Regarding claim 1, Menefee teaches
A method comprising: providing first training data to a translation system configured to
translate between sign language and language data,
[Column 6, lines 8-10 "FIG . 5 illustrates a configurable automated translation
system in accordance with an example embodiment of the disclosed technology"]; [Column 8, lines 43-49 "To perform the neural network learning process, a set of training data can be used to carry out training algorithms such as supervised training of the neural network. In some embodiments, as part of feedback for the learning process, the translated sign language is used to further train and modify the neural network to improve its identification and translation capabilities"];[Column3, lines 51-59 "One specific application of using devices for machine-assisted interpersonal communication is sign language communication and translation. Sign languages are extremely complex, and generally do not have a linguistic relation to the spoken languages of the lands in which they arise. The correlation between sign and spoken languages is complex and varies depending on the country more than the spoken language.
However, Menefee does not teach the translation system includes a plurality of stages and each of the plurality of stages including one or more machine learning models;
But Hu teaches the translation system includes a plurality of stages and each of the plurality of stages including one or more machine learning models;
[Column 5, lines 38-47 "To capitalize on the quality of a non-steaming E2E LAS model, implementations herein are directed toward a two-pass speech recognition system (e.g., shown in FIG. 2A) that includes a first-pass component of an RNN-T network followed by a second-pass component of a LAS network. With this design, the two-pass model benefits from the streaming nature of an RNN-T model with low latency while improving the accuracy of the
RNN-T model through the second-pass incorporating the LAS network" where RRN-T and LAS network are specific subsets of deep learning machine models that convert raw speech and the two-pass model indicates a plurality of processing stages by sequentially dividing the speech task into two distinct components”].
obtaining a first hypothesis output from the translation system based on the first training data;
[Column 10, lines 60-61 "Here, the first pass hypothesis 222 is generated by a RNN decoder 220 for the encoded acoustic frame 212"]; [Column 10, lines 14- 15 "During the first step of the training process, the RNN-T decoder 220 is trained"].
modifying one or more of the machine learning models based on the first hypothesis output;
[Column 10, lines 35-38 "Following either of these approaches, the speech recognizer 200 may be further trained using a minimum WER(MWER) loss to optimize the expected word error rate by using n-best hypotheses" where using MWER is a specific way of modifying a machine learning model based on hypotheses].
providing second training data to a first set of the plurality of stages without providing the second training data to other of the plurality of stages not included in the first set of the plurality of stages;
[Column 10, lines 15-19 " After the RNN-T decoder 220 has been trained, parameters for the RNN-T decoder 220 are fixed and only the deliberation decoder 240 and additional encoder layers (e.g., the deliberation encoder 242 and the acoustic encoder 250) are trained" where training one while fixing the other implies a first set and a second set].
obtaining a second hypothesis output from the first set of the plurality of stages based on the second training data;
[Column 11, lines 2-6 “At operations 308, the method 300 includes decoding the first context vector 247 and the second context vector 245 at a context vector decoder 230 to form a second-pass hypothesis 248."].
and modifying one or more of the machine learning models of the first set of the plurality of stages based on the second hypothesis output
[Column 2, lines 8-16 “In some implementations, the operations also include training the RNN decoder model and training a deliberation decoder while parameters of the trained RNN decoder model remain fixed. The deliberation decoder includes the hypothesis
encoder, the first attention mechanism, the second attention mechanism, and the context vector decoder.”]; [Column 13, lines 59-61 “decoding the first context vector and the second context vector at a context vector decoder to form a second-pass hypothesis”]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine teachings of Menefee into the teachings of Hu because Menefee teaches a sign language translation system that utilizes training data, while Hu teaches a plurality of machine-learning stages that generate first-pass and second-pass hypothesis outputs and train or update machine-learning models based on those hypothesis outputs. Combining the multi-stage hypothesis generation system and model training techniques of Hu into the sign language translation system would predictably improve the accuracy and performance of the sign language translation system by enabling refinement of translation models through multi-stage hypothesis generation and selective model training.
Regarding claim 2, Menefee teaches the method of claim 1, wherein the translation system is configured for sign language recognition or sign language generation
[Column 3, lines 20-22 "FIG. 11 illustrates an example system for sign language recognition using a device with multiple input and output modalities"].
Regarding claim 3, Menefee does not teach the method of claim 1, wherein the one or more of the machine learning models of the first set of the plurality of stages modified based on the second hypothesis output is the same one or more of the machine learning models modified based on the first hypothesis output.
However, Hu teaches the method of claim 1, wherein the one or more of the machine learning models of the first set of the plurality of stages modified based on the
second hypothesis output is the same one or more of the machine learning models modified based on the first hypothesis output.
[Column 2, lines 8-16 “In some implementations, the operations also include training the RNN decoder model and training a deliberation decoder while parameters of the trained RNN decoder model remain fixed. The deliberation decoder includes the hypothesis
encoder, the first attention mechanism, the second attention mechanism, and the context vector decoder.”]; [Column 13, lines 59-61 “decoding the first context vector and the second context vector at a context vector decoder to form a second-pass hypothesis”]
[Column 10, lines 8- 14 “Here, the speech recognizer 200 may be trained using either a cross entropy loss approach or a joint training approach. In a cross entropy loss approach, a deliberation model, such as the speech recognizer 200 with the deliberation decoder 240 (i.e., deliberation-based recognizer 200), is trained in a two-step training process”].
[Column 10, lines 20-34 “In contrast, sometimes training the deliberation decoder 240 while fixing parameters of the RNN-T decoder 220 is not optimal since components of a deliberation-based recognizer 200 are not jointly updated. As an alternative training approach, the deliberation-based recognizer 200 may be jointly trained using a combined loss approach represented by the following equation:L joint(θe,θ1,θ2)=L RNNT(θe,θ1)+λL CE(θe,θ2) (1)where LRNNT(−) is the RNN−T loss and λLCE(−) is the cross entropy loss for the deliberation decoder 240. θe, θ1, and θ2 denote the parameters of the encoder 210, the RNN-T decoder 220, and the deliberation decoder 230, respectively. Here, joint training is similar to the concept of “deep fine tuning” but without a pre-trained decoder”];
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine teachings of Menefee into the teachings of Hua because combining the machine-learning models based on first and second hypothesis outputs with the sign language translation system would predictably improve translation accuracy by allowing the same machine learning models to be progressively refined using successive hypothesis outputs while also reducing training overhead.
Regarding claim 4, Menefee does not teach the method of claim 1, wherein the modifying the one or more of the machine learning models based on the first hypothesis output includes modifying all the machine learning models in the translation system based on the first hypothesis output.
However, Hu teaches the method of claim 1, wherein the modifying the one or more of the machine learning models based on the first hypothesis output includes modifying all the machine learning models in the translation system based on the first hypothesis output.
[Column 10, lines 20-34 “In contrast, sometimes training the deliberation decoder 240 while fixing parameters of the RNN-T decoder 220 is not optimal since components of a deliberation-based recognizer 200 are not jointly updated. As an alternative training approach, the deliberation-based recognizer 200 may be jointly trained using a combined loss approach represented by the following equation:L joint(θe,θ1,θ2)=L RNNT(θe,θ1)+λL CE(θe,θ2) (1)where LRNNT(−) is the RNN−T loss and λLCE(−) is the cross entropy loss for the deliberation decoder 240. θe, θ1, and θ2 denote the parameters of the encoder 210, the RNN-T decoder 220, and the deliberation decoder 230, respectively. Here, joint training is similar to the concept of “deep fine tuning” but without a pre-trained decoder”];
[Column 9, lines 18-27 "By attending to both acoustics (e.g., the output 212 represented as e) and the first-pass hypotheses, the deliberation decoder 20 240 generates the output 248 (e.g., a prediction sequence). Here, each attention mechanism 244, 246 forms a context vector 245, 247 (e.g., an acoustic context vector 247 and a hypothesis context vector 245) that is input into the LAS decoder 230 of the deliberation decoder 240. These context vectors 245, 247 may be concatenated as inputs into the LAS decoder 230" where the LAS decoder is part of the Deliberation Decoder 240 which is being trained jointly with RRN-T decoder based on the first hypothesis input].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine teachings of Menefee into the teachings of Hua because Hu teaches generating a first hypothesis output within a multi-stage machine learning architecture and training it based on generated hypothesis output and applying this first hypothesis based training throughout Menefee’s sign language translation system would improve overall translation performance by allowing each stage of the system to benefit from information contained in the first hypothesis output.
Regarding claim 11, Menefee teaches at least one non-transitory computer-readable media configured to store one or more instructions that, in response to being executed by a system, cause or direct the system to perform the method of claim 1
[Column 31, lines "1-5 "Embodiments of a non-transitory machine-readable storage
medium according to the present invention may include a processor capable of executed instructions to perform operations. The instructions may be stored in memory,
including machine-readable storage media."].
Regarding claim 12, Menefee teaches
A system comprising: one or more computer readable mediums including instructions;
[Column 7, lines 2-7 “Furthermore, the methods disclosed herein may be implemented in hardware, software, or both. If implemented in software, the functions may be stored or transmitted as one or more computer-readable instructions ( e.g., software code) on a computer readable medium”];
one or more computing systems coupled to the one or more computer readable mediums and configured to execute the instructions to cause or direct the system to perform operations, the operations comprising:
[Column 21, lines 53- 56 “Thus, the processor 820 may include one or more processors that coordinate the communication between the various devices as well as execute instructions stored in 55 computer-readable media of the memory device 830”].
providing first training data to a translation system configured to translate between sign language and language data,
[Column 3, line 7 "A method of providing automated translation is disclosed"]; [Column 17, lines 17- 25 Thus, the training station 301 may be configured to receive the video data that includes the sign language content as well as the translated output (as synthesized audio and/or text) to review the accuracy of the automatic translation. If errors are identified, the trainer may enter the corrections into the training station 301, which then transmits the corrections to the AI servers 112 for updating the AI translation database 250 to be used in future calls facilitated by the video relay service 106"].
However, Menefee does not teach the translation system includes a plurality of stages and each of the plurality of stages including one or more machine learning models;
But Hu teaches the translation system includes a plurality of stages and each of the plurality of stages including one or more machine learning models;
[Column 5, lines 38-47 "To capitalize on the quality of a non-steaming E2E LAS model, implementations herein are directed toward a two-pass speech recognition system (e.g., shown in FIG. 2A) that includes a first-pass component of an RNN-T network followed by a second-pass component of a LAS network. With this design, the two-pass model benefits from the streaming nature of an RNN-T model with low latency while improving the accuracy of the
RNN-T model through the second-pass incorporating the LAS network" where RRN-T and LAS network are specific subsets of deep learning machine models that convert raw speech and the two-pass model indicates a plurality of processing stages by sequentially dividing the speech task into two distinct components”].
obtaining a first hypothesis output from the translation system based on the first training data;
[Column 10, lines 60-61 "Here, the first pass hypothesis 222 is generated by a RNN decoder 220 for the encoded acoustic frame 212"]; [Column 10, lines 14- 15 "During the first step of the training process, the RNN-T decoder 220 is trained"].
modifying one or more of the machine learning models based on the first hypothesis output;
[Column 10, lines 35- 38 "Following either of these approaches, the speech recognizer 200 may be further trained using a minimum WER(MWER) loss to optimize the expected word error rate by using n-best hypotheses" where using MWER is a specific way of modifying a machine learning model based on hypotheses].
providing second training data to a first set of the plurality of stages without providing the second training data to other of the plurality of stages not included in the first set of the plurality of stages;
[Column 10, lines 15-19 " After the RNN-T decoder 220 has been trained, parameters for the RNN-T decoder 220 are fixed and only the deliberation decoder 240 and additional encoder layers (e.g., the deliberation encoder 242 and the acoustic encoder 250) are trained" where training one while fixing the other implies a first set and a second set].
obtaining a second hypothesis output from the first set of the plurality of stages based on the second training data;
[Column 11, lines 2-6 “At operations 308, the method 300 includes decoding the first context vector 247 and the second context vector 245 at a context vector decoder 230 to form a second-pass hypothesis 248."].
and modifying one or more of the machine learning models of the first set of the plurality of stages based on the second hypothesis output
[Column 2, lines 8-16 “In some implementations, the operations also include training the RNN decoder model and training a deliberation decoder while parameters of the trained RNN decoder model remain fixed. The deliberation decoder includes the hypothesis
encoder, the first attention mechanism, the second attention mechanism, and the context vector decoder.”]; [Column 13, lines 59-61 “decoding the first context vector and the second context vector at a context vector decoder to form a second-pass hypothesis”].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine teachings of Menefee into the teachings of Hu because Menefee teaches a sign language translation system that utilizes training data, while Hu teaches a plurality of machine-learning stages that generate first-pass and second-pass hypothesis outputs and train or update machine-learning models based on those hypothesis outputs. Combining the multi-stage hypothesis generation system and model training techniques of Hu into the sign language translation system would predictably improve the accuracy and performance of the sign language translation system by enabling refinement of translation models through multi-stage hypothesis generation and selective model training.
Regarding claim 13, Menefee teaches the system of claim 12,wherein the translation system is configured for sign language recognition or sign language generation.
Claim 13 is rejected for the same reasons as claim 2.
Regarding claim 14, Menefee teaches the system of claim 12, wherein the one or more of the machine learning models of the first set of the plurality of stages modified based on the second hypothesis output is the same one or more of the machine learning models modified based on the first hypothesis output.
Claim 14 is rejected for the same reasons as claim 3.
Regarding claim 15, Menefee teaches the system of claim 12, wherein the modifying the one or more of the machine learning models based on the first hypothesis output includes modifying all the machine learning models in the translation system based on the first hypothesis output.
Claim 15 is rejected for the same reasons as claim 4.
Claim [5 , 8, 16, 19] are rejected under 35 U.S.C. 103 as being unpatentable
over Menefee (US 10,489,639 B2) in view of Hu (US 11,908,461 B2) and in further view of Shen
(US 12,461,991 B2).
Regarding claim 5, Menefee in view of Hu do not teach the method of claim 1, wherein the second training data is a subset of the first training data.
However, Shen teaches the method of claim 1, wherein the second training data is a subset of the first training data.
[Column 5, lines 31-38 "In at least one embodiment, each machine learning client has access to a different set of training data that is not accessible to other machine learning clients. In at least one embodiment, first machine learning client 106 is able to access first training data subset 114, second machine learning client 108 is able to access second training data subset 116, third machine learning client 110 is able to access third training data subset 118…" where each machine learning has access to a different set of training data (one training dataset being a subset of a larger dataset].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine teachings of Menefee in view Hua with the teaching of Shen because Shen further teaches utilizing subsets of training data for selecting training operations. Using a subset of the original training data for subsequent training stages would improve training efficiency and reduce computation overhead while continuing to refine the translation models.
Regarding claim 8, Menefee does not teach the method of claim 1, wherein the steps of providing first training data, obtaining the first hypothesis output, and modifying based on the first hypothesis output comprises end-to-end training and is iteratively repeated and the steps of providing the second training data, obtaining the second hypothesis output, and modifying based on the second hypothesis output comprises sub-training and is iteratively repeated.
However, Hu teaches the steps of providing first training data, obtaining the first hypothesis output, and modifying based on the first hypothesis output comprises end-to-end training
[Column 10, lines 60-61 "Here, the first pass hypothesis 222 is generated by a RNN decoder 220 for the encoded acoustic frame 212"]; [Column 10, lines 14- 15 "During the first step of the training process, the RNN-T decoder 220 is trained"]; [Column 10, lines 35- 38 "Following either of these approaches, the speech recognizer 200 may be further trained using a minimum WER(MWER) loss to optimize the expected word error rate by using n-best hypotheses" where using MWER is a specific way of modifying a machine learning model based on hypotheses].
[Column 2, lines 17-21 “In other implementations, the operations include jointly training the RNN decoder model and a deliberation decoder that includes the hypothesis encoder, the first attention mechanism, the second attention mechanism, and the context vector decoder" where training one while fixing the other implies a first set and a second set and shows sub-part training while another model is fixed]. [Column 4, lines 14-16 “ With an integrated structure, all components of a model may be trained jointly as a single end-to-end (E2E) neural network”]
[Column 10, lines 11- 19 “In a cross entropy loss approach, a deliberation model, such as the speech recognizer 200 with the deliberation decoder 240 (i.e., deliberation-based recognizer 200), is trained in a two-step training process. During the first step of the training process, the RNN-T decoder 220 is trained. After the RNN-T decoder 220 has been trained, parameters for the RNN-T decoder 220 are fixed and only the deliberation decoder 240 and additional encoder layers (e.g., the deliberation encoder 242 and the acoustic encoder 250) are trained” where this implies sub-training]; [Column 11, lines 1-6 “The second attention mechanism 244 attends to the encoded first-pass hypothesis 243. At operations 308, the method 300 includes decoding the first context vector 247 and the second context vector 245 at a context vector decoder 230 to form a second-pass hypothesis 248”];
[Column 10, lines 8- 14 “Here, the speech recognizer 200 may be trained using either a cross entropy loss approach or a joint training approach. In a cross entropy loss approach, a deliberation model, such as the speech recognizer 200 with the deliberation decoder 240 (i.e., deliberation-based recognizer 200), is trained in a two-step training process”].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine teachings of Menefee in view of Hua with the teaching of Thomas because Thomas further teaches training machine learning models using communication session derived training data. Combining this would allow both end-end training of the overall translation architecture and sub training of selected machine learning stages with the sign language translation system so the translation models could be refined at both at the system level and stage level using hypothesis outputs and communication derived session data, thereby improving translation accuracy and training effectiveness.
However, Menefee in view of Hu do not teach the process being iteratively repeated.
But Shen teaches the process is iteratively repeated
[Column 14, lines 19-23 " In at least one embodiment, training framework 1004 trains untrained neural network 1006 repeatedly while adjust weights to refine an output of untrained neural network 1006 using a loss function and adjustment algorithm, such as stochastic gradient descent" where this process is repeated and therefore is teaching a iterative process].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine teachings of Menefee in view Hu with the teachings of Shen because Shen further teaches repeatedly training a neural network by adjusting model weights to refine generated outputs. Combining this iterative process to repeat the end to end training and sub training process would progressively refine the machine learning models over successive cycle training and improve translation accuracy.
Regarding claim 16, Menefee in view of Hu and in further view of Shen do teach the system of claim 12, wherein the second training data is a subset of the first training data.
Claim 16 is rejected for the same reasons as claim 5.
Regarding claim 19, Menefee in view of Hu does teach the system of claim 12, wherein the steps of providing first training data, obtaining the first hypothesis output, and modifying based on the first hypothesis output comprises end-to-end training and is iteratively repeated and the steps of providing the second training data, obtaining the second hypothesis output, and modifying based on the second hypothesis output comprises sub-training and is iteratively repeated.
Claim 19 is rejected for the same reasons as claim 8.
Claim [ 6, 7, 17, 18 ] are rejected under 35 U.S.C. 103 as being unpatentable
over Menefee (US 10,489,639 B2) in view of Hu (US 11,908,461 B2) and in further view of
Thomson (US 11,170,761 B2) and in further view of Fox (US 11,373,044 B2).
Regarding claim 6, Menefee in view of Hu do not teach the method of claim 1, wherein the first training data is obtained from a communication session between devices and deleted before the communication session ends and the second training data is stored before, during, and after the communication session.
However, Thomson teaches the first training data is obtained from a communication session between devices and deleted before the communication session ends
[Column 126, lines 28-31 "The method 3000 may begin at block 3002, where first audio data originating at a first device during a communication session between the first device and a second device may be obtained"]; [Column 49, lines 25-33 " In some embodiments, the ASR system models in the CA profile 908 may be trained on-the-fly. Training on-the-fly may indicate that the ASR system models are trained on a data sample (e.g., audio and/or text) as it is captured. In some embodiments, the data sample may deleted after it is used for training. In some embodiments, the data sample may be deleted before a processor performing training using a first batch of samples including the data sample begins training using a second batch of samples including other data samples not in the first batch"]; [Column 49, lines 33-36 "In some embodiments, the data sample may be deleted at or near the end of the communication session in which the data sample is captured."]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine teachings of Menefee in view Hu with the teaching of Thomson because Thomson further teaches obtaining training data from a communication device and deleting the training data after is has been used. Combining this would lead to preserving user privacy, reduce storage requirements and improve management of communication session data.
Menefee in view of Hu and in further with of Thomson do not teach the second training data is stored before, during, and after the communication session.
However, Fox teaches the second training data is stored before, during, and after the communication session
[Column 9, lines 42- 45 "In some implementations, updates to the session data and machine learning model data may be stored to the data store throughout the session" where machine learning data is used as machine learning training (second training ) data that is stored]; [Column 9, lines 6-8 "At step 440, session data and machine learning model data are saved to a data store so that the data may be reused in future sessions"]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine teachings of Menefee in view Hu and in further view of Thomson with the teaching of Fox because Fox further teaches storing machine-learning data in a data store during a communication session and retaining the data for reuse in further communication sessions. Retaining second training data while deleting session-specific data would preserve learned information for future training, improve model performance and reduce training overhead.
Regarding claim 7, Menefee in view of Hu do not teach the method of claim 1, the method of claim 1, wherein the second training data is obtained from a communication session between devices and deleted substantially at an end of the communication session and the first training data is stored before, during, and after the communication session.
However, Thomson teaches the method of claim 1, wherein the second training data is obtained from a communication session between devices and deleted substantially at an end of the communication session
[Column 126, lines 28-31 The method 3000 may begin at block 3002, where first audio data originating at a first device during a communication session between the first device and a second device may be obtained."]; [ Column 49, lines 25-33 “In some embodiments, the ASR system models in the CA profile 908 may be trained on-the-fly. Training on-the-fly may indicate that the ASR system models are trained on a data sample (e.g., audio and/or text) as it is captured. In some embodiments, the data sample may be deleted after it is used for training. In some embodiments, the data sample may be deleted before a processor performing training using a first batch of samples including the data sample begins training using a second batch of samples including other data samples not in the first batch"]; [Column 49, lines 33-36 "In some embodiments, the data sample may be deleted at or near the end of the communication session in which the data sample is captured."].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine teachings of Menefee in view Hu with the teaching of Thomson because Thomson further obtaining training data from a communication session between devices and deleting the training data at the conclusion of session. Combining this would preserve user privacy, reduce storage consumption, and prevent unnecessary retention of session-specific information while still allowing translation models to be trained.
Menefee in view of Hu and in further with of Thomas do not teach the first training data is stored before, during, and after the communication session.
However, Fox teaches the first training data is stored before, during, and after the communication session
[Column 9, lines 42- 45 "In some implementations, updates to the session data and machine learning model data may be stored to the data store throughout the session" where session data is used as first training data that is stored]; [Column 9, lines 6-8 "At step 440, session data and machine learning model data are saved to a data store so that the data may be reused in future sessions"].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine teachings of Menefee in view Hu and in further view of Thomson with the teaching of Fox because Fox further teaches storing machine-learning data in a data store during a communication session and retaining the data for reuse in further communication sessions. Retaining first training data while deleting session-specific data would preserve learned information for future training, improve model performance and reduce training overhead.
Regarding claim 17, Menefee in view of Hu in view of Thomas and in further view of Fox teach the system of claim 12, wherein the first training data is obtained from a communication session between devices and deleted before the communication session ends and the second training data is stored before, during, and after the communication session.
Claim 17 is rejected for the same reasons as claim 6.
Regarding claim 18, Menefee in view of Hu in view of Thomas and in further view of Fox teach the system of claim 12, wherein the second training data is obtained from a communication session between devices and deleted substantially at an end of the communication session and the first training data is stored before, during, and after the communication session.
Claim 18 is rejected for the same reasons as claim 7.
Claim [ 9, 20 ] are rejected under 35 U.S.C. 103 as being unpatentable
over Menefee (US 10,489,639 B2) in view of Hu (US 11,908,461 B2) in further view of Shen
(US 12,461,991 B2) and in further view of Shi (US 11,024,009 B2).
Regarding claim 9, Menefee in view of Hu in further view of Shen do not teach the method of claim 8, wherein a number of iterations for the sub-training are different than a number of iterations for the end-to-end training.
However, Shi teaches the method of claim 8, wherein a number of iterations for the sub-training are different than a number of iterations for the end-to-end training
[Column 10, lines 66-67 "The SRRES networks were trained with a learning rate of 10-4 and 106 update iterations"]; [Column 11, lines 3-6 " All SRGAN network variants were trained with 100,000 update iterations at a learning rate of 10-4, and another 100,000 iterations at a lower 5 learning rate of 10-5 " where these training phases have different iteration counts].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine teachings of Menefee in view Hu and in further view of Thomson with the teaching of Shi because Shi further teaches different machine learning training process may be trained using different numbers of update iterations. Combining these different iteration counts for the end-to end training and sub training with the combined sign language translation system would have predictably optimized convergence of each training process and improve overall translation model performance.
Regarding claim 20, Menefee in view of Hu in further view of Shen and in further view of Shi do teach the system of claim 19, wherein a number of iterations for the sub-training are different than a number of iterations for the end-to-end training.
Claim 20 is rejected for the same reasons as claim 9.
Claim [ 10 ] is rejected under 35 U.S.C. 103 as being unpatentable
over Menefee (US 10,489,639 B2) in view of Hu (US 11,908,461 B2) in further view of Shen
(US 12,461,991 B2) and in further view of Ban (Ban, Hao, and Pengtao Xie. "Interleaving learning, with application to neural architecture search." arXiv preprint arXiv:2103.07018 (2021).
Regarding claim 10, Menefee in view of Hu in further view of Shen do teach the method of claim 8,
Hue teaches the method of sub-training and end to end training
[Column 10, lines 8-19 “Here, the speech recognizer 200 may be trained using either a cross entropy loss approach or a joint training approach. In a cross entropy loss approach, a deliberation model, such as the speech recognizer 200 with the deliberation decoder 240 (i.e., deliberation-based recognizer 200), is trained in a two-step training process. During the first step of the training process, the RNN-T decoder 220 is trained. After the RNN-T decoder 220 has been trained, parameters for the RNN-T decoder 220 are fixed and only the deliberation decoder 240 and additional encoder layers (e.g., the deliberation encoder 242 and the acoustic encoder 250) are trained.].
[Column 10, lines 20- 23 “In contrast, sometimes training the deliberation decoder 240 while fixing parameters of the RNN-T decoder 220 is not optimal since components of a deliberation-based recognizer 200 are not jointly updated. As an alternative training approach, the deliberation-based recognizer 200 may be jointly trained using a combined loss approach represented by the following equation..”]; [Column 2, lines 17-25 “In other implementations, the operations include jointly training the RNN decoder model and a deliberation decoder that includes the hypothesis encoder, the first attention mechanism, the second attention mechanism, and the context vector decoder”]; [Column 4, lines 14-16 “ With an integrated structure, all components of a model may be trained jointly as a single end-to-end (E2E) neural network”];
[Column 10, lines 8- 10 “Here, the speech recognizer 200 may be trained using either a cross entropy loss approach or a joint training approach.”].
However, Menefee in view of Hu in further view of Shen do not teach wherein the iterations for the sub-training are intermixed between iterations for the end-to-end training.
However, Ban teaches the method of claim 8, wherein the iterations for the sub-training are intermixed between iterations for the end-to-end training.
[Page 2, Figure 1: "Comparison between interleaving learning and block learning. In interleaving learning, we perform task 1 for a short while, then move to task 2, then task
3. Afterwards, we move from task 3 back to task 1. This process iterates where
each task is performed for a short time period before switching to another task" where iterations of one training process occurs between iterations of another training process].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine teachings of Menefee in view Hua and in further view of Shen with the teaching of Ban because Ban further teaches interleaving different training processes by alternating among training tasks over multiple rounds. Combining the intermix iterations of the different training process -of sub-training process with iterations of the end to end training process would improving training efficiency, promote knowledge transfer between training stages, accelerate convergence and improve overall model performance.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHEZA ABDUL AZIZ whose telephone number is (571)272-9610. The examiner can normally be reached Monday-Friday 7:30am-5pm Alternate Fridays off.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Washburn can be reached at (571) 272-5551. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHEZA ABDUL AZIZ/Examiner, Art Unit 2657
/DANIEL C WASHBURN/Supervisory Patent Examiner, Art Unit 2657
Prosecution Insights