Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 9 and 19 objected to because of the following informality: wherein the second machine learning model including a third recursive neural network that uses the at least one second biometric signal as an input should read “wherein the second machine learning model includes a third recursive neural network that uses the at least one second biometric signal as an input”. Appropriate correction is required.
Claim Rejections - 35 USC § 112a
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claim 2, 3, 12, and 13 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the enablement requirement. The claim contains subject matter which was not described in the specification in such a way as to enable one skilled in the art to which it pertains, or with which it is most nearly connected, to make and/or use the invention.
Regarding claim 2,
Claim 2 is drawn to the use of the at least one word constituting the thoughts of a user provided by two machine learning models, recited earlier within claim 1, to create a machine command to control a user terminal. The specification as a whole does not provide a skilled artisan with the requisite detail to make and/or use the invention as described.
Compliance with enablement is set forth per MPEP 2164 which recites the test for enablement. In order to answer the factual inquiry of whether or not an application requires undue experimentation, and thus is not enabled, consideration must analyze the undue experimentation factors which are known as the In re Wands factors, or more simply, the Wands factors. While MPEP 2164 further details that it is not necessary to discuss each factor, the consideration should focus on the factors, reason, and evidence leading to the conclusion that the application fails to teach how to make and/or use the claimed invention without undue experimentation, or that the scope of any enablement provided to one skilled in the art is not commensurate with the scope of protection sought by the claims.
The In re Wands factors are applied below to determine that the invention and the limitations as recited in claim 2 are not described in the specification in such a way as to enable one skilled in the art to which it pertains, or which it is most nearly connected, to make and/or use the invention. The Wands factors and analysis as they pertain to claim 2 are as follows:
The breadth of the claims: The machine learning models themselves, as recited in claim 1, are generic, with no specific detail regarding how they function apart from using biometric signal features as input and producing at least one word describing the thoughts of a user as an output. There are no limitations on the at least one word produced apart from there needing to be at least one. There are no limitations on what sort of machine commands can be created. The process of converting the at least one word into a machine command is limited to being done using natural language processing (NLP), although NLP is recited generically with no further detail.
The nature of the invention: The nature of the invention pertains to both artificial intelligence and neuroscience, specifically the incorporation of the two fields in order to produce a brain-machine interface (BMI), which are alternatively known as brain-computer interfaces (BCI). Particularly, the invention appears to utilize brain wave or gaze signals (the specification states at [0041]: “A biometric signal may include at least one of a brain wave signal and a gaze signal, but is not limited thereto”), with the brain wave signals being electroencephalogram (EEG), Magnetoencephalogram (MEG), or Electrocorticogram (ECoG) (the specification states at [0043]: “In the present specification, the brain wave signals such as EEG, MEG, and ECoG are described by way of example. The brain wave signal is not limited to the specific type of a brain wave signal”, and machine learning models to analyze the brain wave or gaze signals)
The state of the prior art: The machine learning model and its use to determine words from thoughts, as recited in claim 1, is grounded in the prior art, as found by Examiner. However, its use in claim 2 to in turn determine machine commands from the words determined from thoughts does not appear to be grounded within the prior art. The prior art discloses that numerous challenges exist in the integration of biometric signals related to neural activity with brain-machine interfaces. Rashid et al. “Current Status, Challenges, and Possible Solutions of EEG-Based Brain-Computer Interface: A Comprehensive Review” states on page 11: “It is worth noting that although conventional BCI systems have made tremendous advances in the past few decades, nonetheless, the research still faces significant challenges in EEG classification. The challenges include various biological and environmental artifacts in EEG, a low SNR, and dependency on human expertise for extracting meaningful features. In addition, most existing machine learning research, if not all, centers on static data and, hence, is not able to classify rapidly changing brain signals accurately (Lotte et al., 2018)”. Rashid et al. further states on page 22: “The most preferred EEG modalities, namely MI, SSVEP, and P300, are continuously facing signal processing issues, especially the identification of the most applicable approaches for feature extraction and feature reduction. This is primarily owing to the nature of the EEG signals, namely extremely non-linear, non-stationary, and artifact-prone. Other notable issues involve data fusion, in particular how the data from numerous electrodes are merged to be able to lessen the data dimensionality as well as to make improvements to the classification results”. Thus, issues relating to reliably processing brain wave signals and maintaining a continually working BMI/BCI system are pertinent in the art. Further, Makin et al. “Machine translation of cortical activity to text with an encoder-decoder framework” states on page 1: “In the last decade, brain-machine interfaces (BMIs) have transitioned from animal models into human subjects, demonstrating that some amount of motor function can be restored to tetraplegics— typically, continuous movements with two degrees of freedom (Nuyujukian et al., 2018; Gilja et al., 2015; Jarosiewicz et al., 2015). Although this type of control can be used in conjunction with a virtual keyboard to produce text, even under ideal cursor control (not currently achievable), the word rate would still be limited to that of typing with a single finger”, i.e. that the primary existing techniques commonly within use within the art are not the use of biometric signals which are analyzed to create words reflecting thoughts, but rather the use of biometric signals to operate a virtual keyboard in order to type words. Further, although Makin et al. purports to, uniquely, enable direct translation of brain wave activity to words, they acknowledge on page 7: “We have shown that spoken speech can be decoded reliably from ECoG data, with WERs as low as 3% on data sets with 250-word vocabularies. But there are several provisos. First, the speech to be decoded was limited to 30-50 sentences. The decoder learns the structure of the sentences and uses it to improve its predictions. This can be seen in the errors the decoder makes, which frequently include pieces or even the entirety of other valid sentences from the training set (see Table 1). Although we should like the decoder to learn and to exploit the regularities of the language, it remains to show how many data would be required to expand from our tiny languages to a more general form of English”. Therefore, it can be seen that within the prior art, direct translation of biometric signals to words for operation of BMIs/BCIs is uncommon, and an issue of a restricted vocabulary of words and sentences that is possible to translate exists.
The level of ordinary skill in the art: A machine learning model for translating biometric signals into words, as recited in claim 1, would be enabled for one of ordinary skill in the arts of machine learning and neuroscience. Translation of words into machine commands could also be known to one more generally skilled in the computer arts. However, reliably and accurately translating biometric signals into words, especially with an unrestricted or arbitrary vocabulary of words, does not appear to be enabled to one of ordinary skill in the art, based on a consideration of the existing prior art.
The level of predictability in the art: There is a moderate level of predictability in the art related to BMI/BCI technology, specifically in regards to brain wave analysis. Rashid et al. states on page 3: “BCI aims to identify the specific neurophysiological signals of a given subject in order to associate a command to each of these signals. Some of these control signals are relatively easy to identify, as well as being relatively easy to control by the user. The extensively utilized EEG control signals include SCP, P300, MI, MRCP, ErrP, SSVEP, SSAEP, and SSSEP”. However Rashid et al. also states on page 22: “The most preferred EEG modalities, namely MI, SSVEP, and P300, are continuously facing signal processing issues, especially the identification of the most applicable approaches for feature extraction and feature reduction. This is primarily owing to the nature of the EEG signals, namely extremely non-linear, non-stationary, and artifact-prone”.
The level of direction provided by the inventor: With regards to the biometric signals that can be used by the machine learning model to produce the words that are translated into machine commands, the inventor provides some direction within the specification in paragraph [0041]: “A biometric signal may include at least one of a brain wave signal and a gaze signal, but is not limited thereto”. Possible brain wave signals are elaborated on within paragraphs [0042] and [0043] of the specification. With regards to the translation of the biometric signals into words, a great deal of direction is provided on the structure of the machine learning models, but not much direction is provided on how the machine learning models are trained, i.e. learned, including their learning algorithm (although the specification discloses at paragraph [0099]: “For example, a linear regression or pseudo inverse matrix method may be used as the linear learning algorithm”) and what data is used for training (although the specification discloses at paragraph [0028]: “The decoding system 100 may communicate with at least one server 110a, 110b,110c, 110d, ..., or 110n to collect learning data for learning”, and the specification further discloses at paragraph [0037]: “Furthermore, according to an embodiment of the inventive concept, learning data for machine learning may be generated based on a U-Net-dhSegment model”). With regards to the translation of the words into machine commands, little direction is provided by the inventor, apart from that a natural language processing (NLP) scheme, without elaboration, is used, which is recited at multiple locations within the specification. With regards to the reliability issues known within the art, little direction is provided by the inventor in addressing these issues apart from recitation of some preprocessing techniques within paragraphs [0053], [0054], and [0079] of the specification.
The existence of working examples: No working examples are disclosed.
The quantity of experimentation needed to make or use the invention based on the content of the disclosure: An unreasonable amount of experimentation would be needed for a person of ordinary skill in the art to use the words output by the machine learning models of claim 1 to reliably create machine commands as recited in claim 2. The human brain and the signals it generates are very complex, and issues with reliably measuring signals from the human brain are well-known in the art. Issues with acquiring appropriate amounts of training data to train models generating words from brain wave signals are also well-known within the art. Solving these issues in order to reliably translate words derived from biometric signals into machine commands, and address situations where errors occur, would require a large amount of experimentation.
In reference to dependent claim 3, claim 3 does not cure the deficiencies noted in the rejection of claim 2. Therefore, claim 3 is rejected under the same rationale as claim 2.
Regarding claim 12,
Claim 12 recites a method for performing the function of the system of claim 2. All other limitations in claim 12 are substantially the same as those in claim 2, therefore the claim is considered to not be enabled by the specification with an equivalent rationale as for claim 2.
In reference to dependent claim 13, claim 13 does not cure the deficiencies noted in the rejection of claim 12. Therefore, claim 3 is rejected under the same rationale as claim 12.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a No therefor, subject to the conditions and requirements of this title.
Claims 1-4, 9, 11-14, and 19 are rejected under 35 U.S.C. 101 because the claimed invention is directed to abstract ideas without significantly more.
Regarding claim 1,
Step 1 - “Is the claim to a process, machine, manufacture or composition of matter?”
Yes, the claim is directed towards a machine.
Step 2A, Prong 1 - “Is the claim directed to a law of nature, a natural phenomenon (product of nature) or an abstract idea?”:
The limitation of extract at least one first biometric signal feature from the preprocessed biometric signal; recites an evaluation of a biometric signal to determine a feature, which is a mental process, which is an abstract idea, regardless of whether it’s implemented on a generic computer.
The limitation of determine at least one second biometric signal feature necessary to identify thoughts of the user among the at least one first biometric signal feature by using a first machine learning model learned to identify thoughts of the user; recites an evaluation of a biometric signal feature that would be useful to identify thoughts, which is a mental process, which is an abstract idea, regardless of whether it’s implemented on a generic computer or using a generic machine learning model.
The limitation of and derive at least one word constituting the thoughts of the user by using the learned first machine learning model and the learned second machine learning model recites an evaluation of a word that reflects the thoughts of a user, which is a mental process, which is an abstract idea, regardless of whether it’s implemented on a generic computer or using generic machine learning models.
Step 2A, Prong 2 - “Does the claim recite additional elements that integrate the judicial exception into a practical application?”:
The limitation of preprocess a measured biometric signal; recites mere instructions to apply preprocessing, MPEP 2106.05(d) and 2106.05(f).
The limitation of learn a second machine learning model while using the at least one second biometric signal as an input and using a word constituting the thoughts of the user as an output in the learned first machine learning model; recites mere instructions to apply training of a generic machine learning model, MPEP 2106.05(d) and 2106.05(f).
Step 2B - “Does the claim recite additional elements that amount to significantly more than the judicial exception?”:
The limitation of preprocess a measured biometric signal; recites mere instructions to apply preprocessing, MPEP 2106.05(f).
The limitation of learn a second machine learning model while using the at least one second biometric signal as an input and using a word constituting the thoughts of the user as an output in the learned first machine learning model; recites mere instructions to apply training of a generic machine learning model, MPEP 2106.05(f).
Therefore, claim 1 is found to be ineligible subject matter under 35 U.S.C. 101.
Regarding claim 2,
Claim 2 adds the additional limitations to claim 1:
convert the derived at least one word into a machine command for controlling a user terminal by using a natural language processing scheme (NLP) recites mere instructions to apply natural language processing to convert at least one word into a command, MPEP 2106.05(d) and 2106.05(f).
Therefore, claim 2 is found to be ineligible subject matter under 35 U.S.C. 101.
Regarding claim 3,
Claim 3 adds the additional limitations to claim 2:
execute the converted machine command on the user terminal recites mere instructions to execute a command on a generic computer, MPEP 2106.05(d) and 2106.05(f).
Therefore, claim 3 is found to be ineligible subject matter under 35 U.S.C. 101.
Regarding claim 4,
Claim 4 adds the additional limitations to claim 1:
wherein the learned first machine learning model extracts the at least one second biometric signal feature by using an advanced variational autoencoder recites mere instructions to apply an advanced variational autoencoder to extract a signal feature, MPEP 2106.05(d) and 2106.05(f).
Therefore, claim 4 is found to be ineligible subject matter under 35 U.S.C. 101.
Regarding claim 9,
Claim 9 adds the additional limitations to claim 1:
wherein the second machine learning model including a third recursive neural network that uses the at least one second biometric signal as an input and uses the word constituting the thoughts of the user as an output in the learned first machine learning model recites mere instructions to apply a third generic neural network, MPEP 2106.05(d) and 2106.05(f).
Therefore, claim 9 is found to be ineligible subject matter under 35 U.S.C. 101.
Regarding claims 11-14 and 19,
Claims 11-14 and 19 disclose a method that implements the function of the system of claims 1-4 and 9 respectively, with substantially the same limitations. Therefore the same analysis and rejection applied to claims 1-4, and 9 applies to claims 11-14 and 19.
Therefore, claims 11-14 and 19 are found to be ineligible subject matter under 35 U.S.C. 101.
Prior Art
The following references are used for prior art claim rejections:
Bi et al. “EEG-Based Adaptive Driver-Vehicle Interface Using Variational Autoencoder and PI-TSVM”
Makin et al. “Machine translation of cortical activity to text with an encoder-decoder framework”
Krishna et al. “Constrained Variational Autoencoder for improving EEG based Speech Recognition Systems”
Cao et al. “An Initial Study on the Relationship Between Meta Features of Dataset and the Initialization of NNRW”
Ozdenizci et al. “Transfer Learning in Brain-Computer Interfaces with Adversarial Variational Autoencoders”
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4 and 11-14 are rejected under 35 U.S.C. 103 as being unpatentable over Bi et al. “EEG-Based Adaptive Driver-Vehicle Interface Using Variational Autoencoder and PI-TSVM”, hereinafter Bi, in view of Makin et al. “Machine translation of cortical activity to text with an encoder-decoder framework”, hereinafter Makin.
Regarding claim 1,
Bi teaches An artificial intelligence system for decoding thoughts of a user, the system comprising a processor configured to: ((Bi Abstract) “A driver-vehicle interface (DVI) based on electroencephalogram (EEG) signals is the core part of a brain-controlled vehicle, which generally includes stimuli interface and EEG decoding module translating EEG signals into commands and sending these translated commands to vehicles”)
preprocess a measured biometric signal; ((Bi Pg. 2) “In the proposed semi-supervised algorithm, EEG signals are first preprocessed with the algorithm that we proposed in [20] to improve the SNR”)
extract at least one first biometric signal feature from the preprocessed biometric signal; ((Bi Pg. 4) “the signals are downsampled to 125 Hz and 512 ms EEG signals are segmented after every flashing stimulus as an epoch. The epochs of same letters are superimposed and averaged to further improve the SNR. Finally, multichannel signals have been vectorized by concatenating the multichannel signals into a vector”, segmented EEG signals are a biometric signal feature)
determine at least one second biometric signal feature necessary to identify thoughts of the user among the at least one first biometric signal feature by using a first machine learning model ((Bi Pg. 4) “The VAE, as a good generative model, provides the opportunity to map the time-domain features to a latent space. In the latent space, the samples can be represented as low dimensional and robust features, which may lead to better classification performance. Thus, after the EEG signals are preprocessed, the VAE is applied to provide a low dimensional and robust feature representation of the EEG signals”) learned to identify thoughts of the user; ((Bi Pg. 2) “As shown in Fig.1, the users select commands by focusing on the desired letter on the user interface. The DVI system uses the EEG decoding model to translate the EEG signals into commands”, a user focus on a letter is a thought of the user)
learn a second machine learning model while using the at least one second biometric signal as an input ((Bi Pg. 5) “Once we obtain the lower-dimension and robust features, we propose a prior information-based transductive support vector machine (PI-TSVM) to classify the EEG signals…Training a TSVM is to solve the optimization objective (14) with training data”, learning a machine learning model corresponds to training the machine learning model)
Makin teaches the following further limitations that Bi does not teach:
and using a word constituting the thoughts of the user as an output in the learned first machine learning model; ((Makin Pg. 3) “A second RNN is therefore initialized at this state, and then trained to emit at each time step either a word or the end-of-sequence token”)
and derive at least one word constituting the thoughts of the user ((Makin Abstract) “we train a recurrent neural network to map neural signals directly to word sequences (sentences). In particular, the network first encodes a sentence-length sequence of neural activity into an abstract representation, and then decodes this representation, word by word, into an English sentence”) by using the learned first machine learning model and the learned second machine learning model (Makin Pg. 3, Fig. 1 shows that their system consists of at least two machine learning models, the encoder RNN and the decoder RNN)
PNG
media_image1.png
783
982
media_image1.png
Greyscale
At the time of filing, one of ordinary skill in the art would have motivation to combine Bi and Makin by taking the system for decoding a user’s thoughts, including preprocessing and extracting a biometric signal feature, using a machine learning model to determine at least a second biometric signal feature from the first, and learning a second machine learning model with the second biometric signal feature, taught by Bi, and adding an extracted word as the final output of the two machine learning models working in tandem, as Makin teaches: (Makin Pg. 1) “In the last decade, brain-machine interfaces (BMIs) have transitioned from animal models into human subjects, demonstrating that some amount of motor function can be restored to tetraplegics— typically, continuous movements with two degrees of freedom (Nuyujukian et al., 2018; Gilja et al., 2015; Jarosiewicz et al., 2015). Although this type of control can be used in conjunction with a virtual keyboard to produce text, even under ideal cursor control (not currently achievable), the word rate would still be limited to that of typing with a single finger”, that is, that decoding thoughts directly into words increases the rate at which commands can be output, restoring more control to a disabled user. Such a combination would be obvious.
Regarding claim 2,
Bi and Makin jointly teach The system of claim 1,
Bi further teaches:
wherein the processor is configured to: convert the derived at least one [word] into a machine command for controlling a user terminal [by using a natural language processing scheme (NLP)] ((Bi Pg. 2) “As shown in Fig.1, the users select commands by focusing on the desired letter on the user interface. The DVI system uses the EEG decoding model to translate the EEG signals into commands”, neither converting words nor natural language processing is taught by Bi)
Makin further teaches:
wherein the processor is configured to: convert the derived at least one word…by using a natural language processing scheme ((Makin Abstract) “Taking a cue from recent advances in machine translation and automatic speech recognition, we train a recurrent neural network to map neural signals directly to word sequences (sentences). In particular, the network first encodes a sentence-length sequence of neural activity into an abstract representation, and then decodes this representation, word by word, into an English sentence”, machine translation and automatic speech recognition are natural language processing schemes)
At the time of filing, one of ordinary skill in the art would have motivation to combine the system jointly taught by Bi and Makin for the parent claim of claim 2, claim 1. No new embodiments are introduced, so the reason to combine is the same as for the parent claim.
Regarding claim 3,
Bi and Makin jointly teach The system of claim 2,
Bi further teaches:
wherein the processor is configured to: execute the converted machine command on the user terminal ((Bi Pg. 1) “A driver-vehicle interface (DVI) based on electroencephalogram (EEG) signals is the core part of a brain-controlled vehicle, which generally includes stimuli interface and EEG decoding module translating EEG signals into commands and sending these translated commands to vehicles…For task-level DVIs based on EEG signals, users first use a DVI to choose a task from a predefined list. Then, the selected task is sent to an automatic control system, which is responsible for performing the selected task”, a driver-vehicle interface of a brain-controlled vehicle with an automatic control system corresponds to a user terminal)
At the time of filing, one of ordinary skill in the art would have motivation to combine the system jointly taught by Bi and Makin for the parent claim of claim 3, claim 2. No new embodiments are introduced, so the reason to combine is the same as for the parent claim.
Regarding claim 4,
Bi and Makin jointly teach The system of claim 1,
Bi further teaches:
wherein the learned first machine learning model extracts the at least one second biometric signal feature by using an advanced variational autoencoder ((Bi Pg. 2) “In the proposed semi-supervised algorithm, EEG signals are first preprocessed…After that, a deep generative model called variational autoencoder (VAE) [21] is used to provide a low dimensional and robust feature representation of the EEG signals”)
At the time of filing, one of ordinary skill in the art would have motivation to combine the system jointly taught by Bi and Makin for the parent claim of claim 4, claim 1. No new embodiments are introduced, so the reason to combine is the same as for the parent claim.
Regarding claims 11-14,
Claims 11-14 recite a method for performing the function of the system of claims 1-4, respectively. All other limitations in claims 11-14 are substantially the same as those in claims 1-4, respectively, therefore the same rationale for rejection applies.
Claims 5 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Bi, in view of Makin, further in view of Krishna et al. “Constrained Variational Autoencoder for improving EEG based Speech Recognition Systems”, hereinafter Krishna.
Regarding claim 5,
Bi and Makin jointly teach The system of claim 1,
Bi further teaches:
wherein the learned first machine learning model is composed of an artificial neural network having a form of an advanced variational autoencoder, ((Bi Pg. 2) “In the proposed semi-supervised algorithm, EEG signals are first preprocessed…After that, a deep generative model called variational autoencoder (VAE) [21] is used to provide a low dimensional and robust feature representation of the EEG signals”)
Makin further teaches:
wherein the artificial neural network is composed of a first recursive neural network acting as an encoder and a second recursive neural network acting as a decoder, (Makin Pg. 3, Fig. 1 shows that their neural network consists of an encoder RNN and a decoder RNN)
Krishna teaches the following further limitations more explicitly than Bi and that Makin does not teach:
wherein the learned first machine learning model…and is learned such that the at least one first biometric signal feature is input to the artificial neural network and the at least one first biometric signal feature is output as a final result, (Krishna Pg. 3, Fig. 1 shows an EEG signal is input into the variational autoencoder and is output by the variational autoencoder)
PNG
media_image2.png
392
350
media_image2.png
Greyscale
wherein an input of the first recursive neural network is the at least one first biometric signal feature, and an output of the first recursive neural network is the at least one second biometric signal feature, ((Krishna Pg. 2) “As seen from Figure 1, the encoder in the VAE model takes EEG features recorded in parallel with speech as input, transforms it into latent representation”, latent representation of a biometric signal feature corresponds to another biometric signal feature)
and wherein an input of the second recursive neural network is the at least one second biometric signal feature, and an output of the second recursive neural network is the at least one first biometric signal feature ((Krishna Pg. 2) “As seen from Figure 1,…and the decoder model reconstructs the EEG features from the latent space”, latent representation of a biometric signal feature corresponds to another biometric signal feature)
At the time of filing, one of ordinary skill in the art would have motivation to combine Bi, Makin, and Krishna by taking the system of claim 1, including a variational autoencoder including an encoder recurrent neural network and a decoder recurrent neural network, taught jointly by Bi and Makin, and combining it with the variational autoencoder wherein an EEG signal, a biometric signal is both the input and the ultimate output, and latent variables based on the EEG signal are an intermediate input and output, taught by Krishna, as variational autoencoders are well-known in the art for being generative models that produce an output that attempts to mimic an input as closely as possible after encoding the input into a latent space that attempts to filter out useful features from noise, and when applied to biometric EEG data results in the predictable benefit of producing a model that can be used to produce new synthetic data with important properties learned from input data, mitigating problems related to lack of adequate training data. Such a combination would be obvious.
Regarding claim 15,
Claim 15 recites a method for performing the function of the system of claim 5. All other limitations in claim 15 are substantially the same as those in claim 5, therefore the same rationale for rejection applies.
Claims 6 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Bi, in view of Makin, further in view of Krishna, further in view of Cao et al. “An Initial Study on the Relationship Between Meta Features of Dataset and the Initialization of NNRW”, hereinafter Cao.
Regarding claim 6,
Bi, Makin, and Krishna jointly teach The system of claim 5,
Bi further teaches:
wherein each of a connection between the at least one first biometric signal feature and a unit of the first recursive neural network, and a connection between the at least one second biometric signal feature and a unit of the second recursive neural network is an ALL-TO-ALL linear connection, ((Bi Pg. 4) “Once we obtain the optimization objective, we can start to determine the specific neural network structure of the VAE. Here, we choose the multi-layer perceptron (MLP) to approximate the pθ(x|z) and q∅(z|x) owing to its strong approximate ability for solving extremely complex problems. The MLP is a kind of neural network, which consists of several layers of nodes. Once the pθ(x|z) and q∅(z|x) are approximated by neural networks, they compose the architecture of the VAE. The network architecture of the VAE is shown in Fig. 5”, Bi Pg. 5, Fig. 5 shows their variational autoencoder structure, including fully connected layers between the input layer and the hidden layer, the broadest reasonable interpretation of an ALL-TO-ALL linear connection between a feature and a neural network includes connections between every input node and every hidden layer node)
PNG
media_image3.png
527
471
media_image3.png
Greyscale
Cao teaches the following further limitations that neither Bi, nor Makin, nor Krishna teach:
wherein a connection weight is randomly determined by a uniform distribution, ((Cao Pg. 3) “Usually, researchers prefer to use the Uniform distribution to initialize the NNRW model, where the range of randomly generated input weights is (-1,1) and the range of hidden bias is (0,1)”)
and wherein a value of the connection weight is fixed during an initialization procedure and is not changed afterward ((Cao Pg. 1) “However, in NNRW, not all weights need to be trained. For example, for an NNRW with a single hidden layer, the input weights (i.e., the weights between the input layer and hidden layer) and hidden biases (i.e., the thresholds of hidden layer nodes) are randomly generated and remain unchanged throughout the whole training process, while the output weights (i.e., the weights between hidden layer and output layer) are obtained by solving a system of linear matrix equations”)
At the time of filing, one of ordinary skill in the art would have motivation to combine Bi, Makin, Krishna, and Cao by taking the system of claim 5, including ALL-TO-ALL linear connections between units of the first and second neural networks and the biometric signal features, taught jointly by Bi, Makin, and Krishna, and combining it with the initialization of weights via random sampling of a uniform distribution which are fixed and not changed during training, taught by Cao, as Cao teaches: (Cao Pg. 1) “In BP-based neural networks, all weights need to be iteratively fine-tuned until the model meets the expected accuracy. However, in NNRW, not all weights need to be trained…The training process of NNRW is non-iterative and thus it can often achieve faster learning speed and comparable accuracy than BP-based neural networks on some issues”, that is, that fixing some weights can cause the network to learn faster without necessarily suffering a loss of accuracy. Such a combination would be obvious.
Regarding claim 16,
Claim 16 recites a method for performing the function of the system of claim 6. All other limitations in claim 16 are substantially the same as those in claim 6, therefore the same rationale for rejection applies.
Claims 7, 8, 17, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Bi, in view of Makin, further in view of Krishna, further in view of Cao, further in view of Ozdenizci et al. “Transfer Learning in Brain-Computer Interfaces with Adversarial Variational Autoencoders”, hereinafter Ozdenizci.
Regarding claim 7,
Bi, Makin, Krishna, and Cao jointly teach The system of claim 6,
Bi further teaches:
and wherein a value of a connection weight of the connection between the unit of the first recursive neural network and the at least one second biometric signal feature and the connection between the unit of the second recursive neural network and the at least one first biometric signal feature is changed while being learned by a linear learning algorithm ((Bi Pg. 4) “Once we obtain the optimization objective, we can start to determine the specific neural network structure of the VAE. Here, we choose the multi-layer perceptron (MLP) to approximate the pθ(x|z) and q∅(z|x) owing to its strong approximate ability for solving extremely complex problems. The MLP is a kind of neural network, which consists of several layers of nodes. Once the pθ(x|z) and q∅(z|x) are approximated by neural networks, they compose the architecture of the VAE. The network architecture of the VAE is shown in Fig. 5. We need to determine the parameters of the VAE network by optimizing (13). Considering (13) is a non-convex function, we use the stochastic gradient descent (SGD) to optimize the objective”, pθ(x|z) and q∅(z|x) are the encoder and decoder of the variational autoencoder of Bi, parameters of a multi-layer perceptron including layers of nodes correspond to connection weights, stochastic gradient descent is a training algorithm that can be used to train linear machine learning algorithms such as linear regression)
Ozdenizci teaches the following further limitations that neither Bi, nor Makin, nor Krishna, nor Cao teaches:
wherein each of a connection between the unit of the first recursive neural network and the at least one second biometric signal feature and a connection between the unit of the second recursive neural network and the at least one first biometric signal feature is an ALL-TO-ALL linear connection, (Ozdenizci Pg. 2, Fig. 1 and Ozdenizci Pg. 3, Table 1 show that there are fully-connected layers within the encoder neural network and the latent variable, as well as between the decoder neural network and the EEG time series output, fully-connected layers correspond to ALL-TO-ALL linear connections, the latent variable corresponds to the second biometric signal feature, and the EEG signal correspond to the first biometric signal feature)
PNG
media_image4.png
583
1152
media_image4.png
Greyscale
PNG
media_image5.png
512
567
media_image5.png
Greyscale
At the time of filing, one of ordinary skill in the art would have motivation to combine Bi, Makin, Krishna, Cao, and Ozdenizci by taking the system of claim 6, including changing weights using a linear learning algorithm, taught jointly by Bi, Makin, Krishna, and Cao, and combining it with ALL-TO-ALL connections within fully-connected layers between a first neural network and a second biometric signal feature, as well as a second neural network and a first biometric signal feature, taught by Ozdenizci, as fully-connected layers consolidate and combine the information from all earlier units, allowing the use of all this information to make determinations, increasing the accuracy of the determinations. Such a combination would be obvious.
Regarding claim 8,
Bi, Makin, Krishna, Cao, and Ozdenizci jointly teach The system of claim 7,
Bi further teaches:
wherein the value of the connection weight of the connection between the unit of the first recursive neural network and the at least one second biometric signal feature and the connection between the unit of the second recursive neural network and the at least one first biometric signal feature [is randomly determined by a uniform distribution and then] is changed while being learned by the linear learning algorithm ((Bi Pg. 4) “Once we obtain the optimization objective, we can start to determine the specific neural network structure of the VAE. Here, we choose the multi-layer perceptron (MLP) to approximate the pθ(x|z) and q∅(z|x) owing to its strong approximate ability for solving extremely complex problems. The MLP is a kind of neural network, which consists of several layers of nodes. Once the pθ(x|z) and q∅(z|x) are approximated by neural networks, they compose the architecture of the VAE. The network architecture of the VAE is shown in Fig. 5. We need to determine the parameters of the VAE network by optimizing (13). Considering (13) is a non-convex function, we use the stochastic gradient descent (SGD) to optimize the objective”, pθ(x|z) and q∅(z|x) are the encoder and decoder of the variational autoencoder of Bi, parameters of a multi-layer perceptron including layers of nodes correspond to connection weights, stochastic gradient descent is a training algorithm that can be used to train linear machine learning algorithms such as linear regression, Bi does not teach initialization of weights using a uniform distribution)
Cao further teaches:
wherein the value of the connection weight of the connection…is randomly determined by a uniform distribution ((Cao Pg. 3) “Usually, researchers prefer to use the Uniform distribution to initialize the NNRW model, where the range of randomly generated input weights is (-1,1) and the range of hidden bias is (0,1)”)
At the time of filing, one of ordinary skill in the art would have motivation to combine the system jointly taught by Bi, Makin, Krishna, Cao, and Ozdenizci for the parent claim of claim 8, claim 7. No new embodiments are introduced, so the reason to combine is the same as for the parent claim.
Regarding claims 17 and 18,
Claims 17 and 18 recite a method for performing the function of the system of claims 7 and 8, respectively. All other limitations in claims 17 and 18 are substantially the same as those in claims 7 and 8, respectively, therefore the same rationale for rejection applies.
Claims 9 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Bi, in view of Makin, further in view of Ozdenizci.
Regarding claim 9,
Bi and Makin jointly teach The system of claim 1,
Makin further teaches:
wherein the second machine learning model…and uses the word constituting the thoughts of the user as an output in the learned first machine learning model ((Makin Pg. 3) “A second RNN is therefore initialized at this state, and then trained to emit at each time step either a word or the end-of-sequence token”)
Ozdenizci teaches the following further limitations that neither Bi nor Makin teaches:
wherein the second machine learning model including a third recursive neural network that uses the at least one second biometric signal as an input (Ozdenizci Pg. 2, Fig. 1 shows a third neural network using the latent variables produced by the autoencoder as inputs, latent variables produced by a neural network based on EEG signals corresponds to a second biometric signal)
At the time of filing, one of ordinary skill in the art would have motivation to combine Bi, Makin, and Ozdenizci by taking the system of claim 1, including a second machine learning model that outputs at least one word constituting the thoughts of a user, taught jointly by Bi and Makin, and combining it with a third recurrent neural network that uses a second biometric signal as an input, taught by Ozdenizci, as Ozdenizci teaches: (Ozdenizci Pg. 1) “Particularly, the proposed approach [9], [13] aims to learn subject-invariant representations by simultaneously training a conditional VAE and an adversarial network to enforce invariance of the learned data representations with respect to subject identity. This adversarial training procedure, with VAEs based on CNN architectures, yields data representations that work as features that are disentangled from subject-specific nuisance variations, which enables decoding for unseen BCI subjects”, that is, the addition of the additional neural network enables penalizing details that do not reflect important general concepts, creating a model that can adapt better to new environments and users. Such a combination would be obvious.
Regarding claim 19,
Claim 19 recites a method for performing the function of the system of claim 9. All other limitations in claim 19 are substantially the same as those in claim 9, therefore the same rationale for rejection applies.
Claims 10 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Bi, in view of Makin, further in view of Ozdenizci, further in view of Cao.
Regarding claim 10,
Bi, Makin, and Ozdenizci jointly teach The system of claim 9,
Ozdenizci further teaches:
wherein a connection between the at least one second biometric signal feature and a unit of the third recursive neural network is an ALL-TO-ALL linear connection, (Ozdenizci Pg. 2, Fig. 1 shows the input of the third, adversary network is to a fully-connected layer, which corresponds to an ALL-TO-ALL linear connection)
wherein a connection between the unit of the third recursive neural network and a unit [corresponding to the word constituting the thoughts of the user] is an ALL-TO-ALL linear connection, (Ozdenizci Pg. 2, Fig. 1 shows there are several fully-connected layers, which correspond to an ALL-TO-ALL linear connections, between units of the third neural network and the output of the network, output of a network being a word constituting the thoughts of the user taught by Makin)
and wherein a value of a connection weight of the connection between the unit of the third recursive neural network and the unit [corresponding to the word constituting the thoughts of the user] is changed while being learned by a linear learning algorithm ((Ozdenizici Pg. 2, Fig 1 Caption) “A-cVAE is trained to minimize the loss function in Eq. (2), while the adversary is also individualy trained to minimize its softmax cross-entropy loss. Parameter updates were performed alternatingly among the A-cVAE and the adversary once per batch”, training the adversary network to minimize loss including parameter updates corresponds to changing a value of a connection weight in relation to an output while being learned by a linear learning algorithm, output of a network being a word constituting the thoughts of the user taught by Makin)
Cao teaches the following further limitations that neither Bi, nor Makin, nor Ozdenizci teaches:
a connection weight is randomly determined by a uniform distribution, ((Cao Pg. 3) “Usually, researchers prefer to use the Uniform distribution to initialize the NNRW model, where the range of randomly generated input weights is (-1,1) and the range of hidden bias is (0,1)”)
and a value of the connection weight is fixed during an initialization procedure and is not changed afterward, ((Cao Pg. 1) “However, in NNRW, not all weights need to be trained. For example, for an NNRW with a single hidden layer, the input weights (i.e., the weights between the input layer and hidden layer) and hidden biases (i.e., the thresholds of hidden layer nodes) are randomly generated and remain unchanged throughout the whole training process, while the output weights (i.e., the weights between hidden layer and output layer) are obtained by solving a system of linear matrix equations”)
At the time of filing, one of ordinary skill in the art would have motivation to combine Bi, Makin, Ozdenizci, and Cao by taking the system of claim 9, including ALL-TO-ALL linear connections between units of the third neural network and training of connection weights in the third neural network, taught jointly by Bi, Makin, and Ozdenizci, and combining it with the initialization of weights via random sampling of a uniform distribution which are fixed and not changed during training, taught by Cao, as Cao teaches: (Cao Pg. 1) “In BP-based neural networks, all weights need to be iteratively fine-tuned until the model meets the expected accuracy. However, in NNRW, not all weights need to be trained…The training process of NNRW is non-iterative and thus it can often achieve faster learning speed and comparable accuracy than BP-based neural networks on some issues”, that is, that fixing some weights can cause the network to learn faster without necessarily suffering a loss of accuracy. Such a combination would be obvious.
Regarding claim 20,
Claim 20 recites a method for performing the function of the system of claim 10. All other limitations in claim 20 are substantially the same as those in claim 10, therefore the same rational