DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Drawings
The drawings are objected because it is unclear from Applicant’s multiple submissions which figure is intended to replace which previous submitted figure. For clarity of the record, Applicant is advised to submit all figures that Applicant intended to rely upon, with the figures that clearly identified and numbered. Any structural detail that is essential for a proper understanding of the disclosed invention should be shown in the drawing. MPEP § 608.02(d). Corrected drawing sheets in compliance with 37 CFR 1.121(d) is required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and were necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1 – 2 are rejected under 35 U.S.C. 103 as being unpatentable over Zhao, et al. "A multi-branch 3D convolutional neural network for EEG-based motor imagery classification" in view of Navaneet et al., "Capnet: Continuous approximation projection for 3d point cloud reconstruction using 2d supervision" and Han et al., Pub. No.: CN115130664A.
Regarding claim 1, Zhao teaches: A method for recognizing a motor imagery electroencephalography (MI-EEG) signal
((Zhao, page: 2165), “Based on this algorithm, BSML-based MI-EEG classification algorithm [recognizing a motor imagery electroencephalography (MI-EEG) signal] is proposed which exhibits strong robustness and competitive classification accuracies. In [20], Principal Component Analysis (PCA) transform is applied to decompose and retain the data information of two EEG sampling channels. Then Neural Network Classifier is applied to achieve the final results. This work indicates the feasibility of combining PCA transform with neural network.”)
Si: mapping an electroencephalography (EEG) time series of an MI-EEG signal into a three-dimensional (3D) array form based on a spatial electrode distribution;
((Zhao, page: 2164), “In this study, for EEG-based MI classification tasks, a new 3D representation of EEG [Si: mapping an electroencephalography (EEG) time series of an MI-EEG signal into a three-dimensional (3D) array] is first introduced, which preserves both spatial information and temporal information [form based on a spatial electrode distribution]. Based on this 3D representation, a multi-branch 3D CNN is employed to extract MI related features.”)
a three-dimensional capsule network (3D-CapsNet) model for recognizing the MI-EEG signal, and using an EEG signal in the 3D array form described in the Si as an input of the 3D-CapsNet for recognizing the MI-EEG signal,
((Zhao, page: 2165 – 2166), “In this study, for EEG-based MI classification tasks, a new 3D representation of EEG is first introduced [a three-dimensional capsule network (3D-CapsNet) model for recognizing the MI-EEG signal], which preserves both spatial information and temporal information. Based on this 3D representation, a multi-branch 3D CNN is employed to extract MI related features. The strength of applying 3D CNN in EEG classification is that the spatial and temporal features in EEG signals can be extracted simultaneously and the relationship between them can be fully utilized [using an EEG signal in the 3D array form described in the Si as an input of the 3D-CapsNet for recognizing the MI-EEG signal]. By combining 3D representation of EEG and multi-branch 3D CNN, we achieve state-of-the-art performance on the BCI competition IV-2a data set, significantly enhancing the classification performance with higher average kappa and lower standard deviation of different subjects.”)
the 3D convolution module performs feature extraction on the input EEG signal in the 3D array form from both a temporal dimension and an inter-channel spatial dimension through a plurality of layers of 3D convolution to obtain a low-level feature; and
((Zhao, page: 2165), “In this study, for EEG-based MI classification tasks, a new 3D representation of EEG is first introduced, which preserves both spatial information and temporal information. Based on this 3D representation, a multi-branch 3D CNN is employed to extract MI related features [the 3D convolution module performs feature extraction on the input EEG signal in the 3D array form]. The strength of applying 3D CNN in EEG classification is that the spatial and temporal features in EEG signals can be extracted simultaneously and the relationship between them can be fully utilized [from both a temporal dimension and an inter-channel spatial dimension through a plurality of layers of 3D convolution to obtain a low-level feature].”)
by using a dynamic routing algorithm, connecting a primary capsule and a motor capsule through dynamic routing, and finally outputting a classification result through a nonlinear activation function squash.
((Zhao, page: 2171), “1) FBCSP: FBCSP [12] first uses a group of band-pass filters and CSP algorithm [by using a dynamic routing algorithm, connecting a primary capsule and a motor capsule through dynamic routing] to extract the optimal spatial features from the subject-specific frequency band and then train a classifier to output the final classification results [finally outputting a classification result through a nonlinear activation function squash]. FBCSP was the best performing method for the BCI competition IV dataset 2a in the competition, and it also won other similar EEG decoding competitions BCI IV dataset 2b [41].”)
Zhao teaches:
based on a capsule network (CapsNet)
S2: constructing, by using a CapsNet and 3D convolution,
wherein a 3D-CapsNet comprises a 3D convolution module and a CapsNet module;
the CapsNet module has a spatial detection capability, and the low-level feature output by the 3D convolution module is integrated through the CapsNet to obtain a high-level spatial vector containing an inter-feature relationship; and
S3: training the CapsNet module
Navaneet teaches:
based on a capsule network (CapsNet),
((Navaneet, page 1 – 2), “We propose CAPNet [based on a capsule network (CapsNet)], a continuous approximation projection module for a differentiable and accurate rendering of 3D point clouds, to enable weakly supervised 3D object reconstruction. The proposed rendering module generates smooth, artifact-free projections, while also overcoming the lack of gradients that can exist in a naive discretization based approach.”)
S2: constructing, by using a CapsNet and 3D convolution,
((Navaneet, page 1 – 2), “We propose CAPNet [S2: constructing, by using a CapsNet], a continuous approximation projection module for a differentiable and accurate rendering of 3D point [and 3D convolution] clouds, to enable weakly supervised 3D object reconstruction. The proposed rendering module generates smooth, artifact-free projections, while also overcoming the lack of gradients that can exist in a naive discretization based approach.”)
wherein a 3D-CapsNet comprises a 3D convolution module and a CapsNet module;
((Navaneet, page 1 – 2), “We propose CAPNet [wherein a 3D-CapsNet], a continuous approximation projection module for a differentiable and accurate rendering of 3D point [comprises a 3D convolution module and a CapsNet module] clouds, to enable weakly supervised 3D object reconstruction. The proposed rendering module generates smooth, artifact-free projections, while also overcoming the lack of gradients that can exist in a naive discretization based approach.”)
Navaneet and Zhao are related to the same field of endeavor (i.e.: analysis of electroencephalograms). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of Navaneet with teachings of Zhao to add techniques for addressing sparse 2D spatial representations including generating a more continuous spatial representation to improve feature extraction. (Navaneet, Abstract).
Zhao in view of Navaneet do not teach:
the CapsNet module has a spatial detection capability, and the low-level feature output by the 3D convolution module is integrated through the CapsNet to obtain a high-level spatial vector containing an inter-feature relationship; and
S3: training the CapsNet module
Han teaches:
the CapsNet module has a spatial detection capability, and the low-level feature output by the 3D convolution module is integrated through the CapsNet to obtain a high-level spatial vector containing an inter-feature relationship; and
(Han, “[0004] Based on this, the purpose of the present invention is to provide a method, device, equipment, and storage medium for emotion analysis of EEG signals based on a capsule network model. By combining the capsule network model [the CapsNet module has a spatial detection capability] and the attention routing mechanism, the positional relationship between high-level features and low-level features is considered during the emotion analysis process [the low-level feature output by the 3D convolution module is integrated through the CapsNet to obtain a high-level spatial vector containing an inter-feature relationship], thereby preserving temporal and spatial features, improving the efficiency and accuracy of emotion analysis of EEG signals, and reducing labor and equipment costs. [0005] In a first aspect, embodiments of this application provide a method for emotion analysis of EEG signals based on a capsule network model, comprising the following steps: acquiring an EEG signal dataset, performing channel data selection”)
S3: training the CapsNet module
(Han, “[0058] S7: Input the several high-level capsules corresponding to the several EEG signal samples into the output capsule module for connection processing to obtain the output capsules corresponding to the several EEG signal samples. Based on the initial capsules, output capsules, and label sets corresponding to the several EEG signal samples, construct the loss function of the capsule network model, perform optimization training [S3: training the CapsNet module], and obtain the target capsule network model.”)
Han, Zhao and Navaneet are related to the same field of endeavor (i.e.: analysis of electroencephalograms). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of Han with teachings of Zhao and Navaneet to add attention routing mechanism for processing EEG representation to account for the positional relationships between lower-level and higher-level features to improve classification accuracy and robustness. (Han, Abstract).
Regarding claim 2, Zhao in view of Navaneet and Han teach the method of claim 1.
Zhao further teaches: intercepting the EEG signal by frame, obtaining a value of a current frame, transforming a value of each frame into an X * Y 2D matrix (2D-map) based on a general spatial distribution of a sampled electrode, and filling an unused electrode position with 0; and expanding TP 2D-maps into an X * Y * TP 3D matrix based on temporal information of the EEG signal, wherein TP represents a quantity of sampling points for each channel, and TP is a natural number.
((Zhao, page: 2166), “A. 3D Representation of EEG In this study, we design a 3D representation of EEG. The first step of obtaining this 3D representation is transforming each frame of EEG into a 2D array [intercepting the EEG signal by frame, obtaining a value of a current frame, transforming a value of each frame into an X * Y 2D matrix (2D-map) based on a general spatial distribution of a sampled electrode, and filling an unused electrode position with 0] according to the general spatial distribution of sampling electrodes and meanwhile padding the point where there is no electrode with 0. The rectangular shape of this 2D array will make the following feature extraction process implemented much easier. Then, we expand this 2D array to 3D array by using EEG’s temporal information [expanding TP 2D-maps into an X * Y * TP 3D matrix based on temporal information of the EEG signal, wherein TP represents a quantity of sampling points for each channel, and TP is a natural number]. Note that this representation approach is easy-to use and general enough to be applied in any other EEG-based classification tasks.”)
Allowable subject matter
Claim(s) 3 – 4 are objected to as being dependent upon a rejected base claim and would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. The prior art made of record does not teach, make obvious, or suggest the claim limitations as disclosed in applicant's claims.
Claim 3 recites:
The method for recognizing an MI-EEG signal based on a CapsNet according to claim 2,
wherein the step S2 is executed as follows: constituting the 3D convolution module by encapsulating five 3D convolution layers to extract a basic feature of the input EEG signal in the 3D array form at a plurality of levels to provide local perceptual information for a main capsule layer, and gradually increasing a quantity of convolution kernels to ensure that increasingly rich features 19 are correctly extracted;
performing batch normalization (BN) after each convolution to accelerate convergence and reduce overfitting;
inputting the input into the convolution module to generate 128 4*5*6 outputs, converting the outputs into a 128*4*5*6 tensor, and sending the 128*4*5*6 tensor to the main capsule layer, such that the main capsule layer outputs 384 4-dimensional capsules, wherein the main capsule stores spatial features of different forms for the MI-EEG signal;
connecting the main capsule layer and a motor capsule layer through the dynamic routing; aggregating, by the dynamic routing algorithm, predicted capsules that are similar to each other, and obtaining, through abstraction, a motor capsule capable of representing an inter-class difference; and outputting the classification result through the nonlinear activation function squash.
Closest prior arts:
Zhao, et al. "A multi-branch 3D convolutional neural network for EEG-based motor imagery classification"
Zhao teaches preserving both temporal and spatial information when representing EEG signals for motor imagery classification. It converts EEG signals into a sequence of 2D arrays that preserve the spatial distribution of the electrodes and processes the resulting 3D representation using a multi-branch 3D CNN and corresponding classification strategy. However, Zhao does not teach processing a 3D representation of an EEG signal using five 3D convolution layers to progressively extract increasingly rich spatial and temporal features, with batch normalization applied after each layer. The resulting feature tensor is converted into capsules that preserve spatial feature relationship and dynamic routing aggregates similar predicted capsules into higher level motor capsules representing differences between EEG classes with a squash activation function producing the final classification result.
Navaneet et al., "Capnet: Continuous approximation projection for 3d point cloud reconstruction using 2d supervision"
Navaneet proposes reconstructing a 3D point cloud from a single 2D image using multiple foregrounds makes as supervisory data, reducing the need for large scale 3D training datasets. It introduces CAPNet, a differentiable projection module that projects a predicted 3D point cloud into 2D masks, along with an affinity loss to address sparse projection maps and produce cleaner reconstructions. However, Navaneet does not teach processing a 3D representation of an EEG signal using five 3D convolution layers to progressively extract increasingly rich spatial and temporal features, with batch normalization applied after each layer. The resulting feature tensor is converted into capsules that preserve spatial feature relationship and dynamic routing aggregates similar predicted capsules into higher level motor capsules representing differences between EEG classes with a squash activation function producing the final classification result.
Han et al., Pub. No.: CN115130664A.
Han proposes an EEG emotion analysis method using a capsule network with an attention based routing mechanism. By considering positional relationships between low and high level features, the method preserves both temporal and spatial EEG features. However, Han does not teach processing a 3D representation of an EEG signal using five 3D convolution layers to progressively extract increasingly rich spatial and temporal features, with batch normalization applied after each layer. The resulting feature tensor is converted into capsules that preserve spatial feature relationship and dynamic routing aggregates similar predicted capsules into higher level motor capsules representing differences between EEG classes with a squash activation function producing the final classification result.
Dependent claim 4 is allowable because of its dependency to claim 3.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Yang, Lie, et al. "Two-branch 3D convolutional neural network for motor imagery EEG decoding." (2021).
Lie adopts a concise 3D representation for the MI-EEG data to take full advantage of the spatial features and propose a two-branch 3D CNN (TB-3D CNN)for the 3D representation of MI-EEG data.
Kwon-Woo, Ha, and Jeong Jin-Woo. "Motor imagery EEG classification using capsule networks." (2019).
Ha proposes to apply a capsule network (CapsNet) for learning various properties of EEG signals, thereby achieving better and more robust performance than previous CNN methods. The proposed CapsNet-based framework classifies the two-class motor imagery, namely right-hand and left-hand movements. The motor imagery EEG signals are first transformed into 2D images using the short-time Fourier transform (STFT) algorithm and then used for training and testing the capsule network.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATIYAS T MARU whose telephone number is (571)270-0902 or via email: matiyas.maru@uspto.gov. The examiner can normally be reached Monday 8:00am - Friday 4:00pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle Bechtold can be reached on (571)431-0762. The fax phone number for the organization were this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from
Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/M.T.M./Examiner, Art Unit 2148
/MICHELLE T BECHTOLD/Supervisory Patent Examiner, Art Unit 2148
Prosecution Insights