DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . This action is responsive to the Application filed on 12/28/2023. Claims 1-20 are pending in the case. Claims 1 and 8 are independent claims.
Claim Rejections - 35 U.S.C. § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. §§ 102 and 103 (or as subject to pre-AIA 35 U.S.C. §§ 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. § 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant are advised of the obligation under 37 C.F.R. § 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. § 102(b)(2)(C) for any potential 35 U.S.C. § 102(a)(2) prior art against the later invention.
Claims 1 and 8 are rejected under 35 U.S.C. § 103 as being unpatentable over Novikov et al. (Novikov, Alexander, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P. Vetrov. "Tensorizing neural networks." Advances in neural information processing systems 28 (2015), hereinafter Novikov) in view of Chen et al. (Chen, Tianshi, Zidong Du, Ninghui Sun, Jia Wang, Chengyong Wu, Yunji Chen, and Olivier Temam. "Diannao: A small-footprint high-throughput accelerator for ubiquitous machine-learning." ACM SIGARCH Computer Architecture News 42, no. 1 (2014): 269-284, hereinafter Chen).
As to independent claims 1 and 8, Novikov teaches a computational framework, comprising:
execution of techniques or procedures to deploy a computational framework (Page 1, "We call the resulting layer a TT-layer and refer to a network with one or more TT-layers as TensorNet." TensorNet reads on the claimed framework);
creation of mathematical structures for utilization of a type of machine learning (Page 1, "We use a compact multiliniear format – Tensor-Train (TT-format) [17] – to represent the dense weight matrix of the fully-connected layers using few parameters while keeping enough flexibility to perform signal transformations" Tensor-Train (TT-format) reads on the claimed mathematical structures);
utilization of a type of deep learning for faster convergence and training, better precision (Page 7, "We use deep the CNNs vgg-16 and vgg-19 [21] as the reference models2." Page 7, "In each network we substitute the first fully-connected layer with the TT-layer." Page 8, "All in all a wide and shallow TensorNet can become a time and memory efficient model to use in real time applications and on mobile devices.");…
extraction of data from a collection of data (Page 1, "We apply our method to popular network architectures proposed for several datasets of different scales: MNIST [15], CIFAR-10 [12], ImageNet [13]." Page 6, "we preprocess the images by subtracting the mean and performing global contrast normalization and ZCA whitening");
arrangement of computational units in layers (Page 1, "we consider probably the most frequently used layer of the neural networks: the fully-connected layer." Page 4, "we derive the gradients required to use the back-propagation algorithm with the TT-layer.");
substitution of numerical arrays in deep learning module by mathematical structures (Abstract, "we convert the dense weight matrices of the fully-connected layers to the Tensor Train [17] format such that the number of parameters is reduced by a huge factor and at the same time the expressive power of the layer is preserved.");
usage of specific type of mathematical structures (Page 1, "Tensor-Train (TT-format)". Page 3, "matrix product".);
application of optimization technique to adjust adjustable elements (Page 8, "We train all the networks with stochastic gradient descent");
reduction of performance measure for a subset of dataset (Page 4, "Back-propagation allows to compute the gradient of a loss-function L with respect to all the parameters of the network"); and
generation of output of the model for unseen data (Page 6, "As a baseline we use a neural network with two fully-connected layers (1024 hidden units) and rectified linear unit (ReLU) achieving 1:9% error on the test set". The test set reads on the claimed unseen data).
Novikov does not appear to expressly teach employment of hardware components to implement the model.
Chen teaches employment of hardware components to implement the model (Abstract, "we design an accelerator for large-scale CNNs and DNNs, with a special emphasis on the impact of memory on accelerator design, performance and energy").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the tensorizing neural networks of Novikov to include the accelerator techniques of Chen to provide high throughput in a small footprint thereby opening up the usage of state-of-the-art machine-learning algorithms in a broad set of systems and for a broad set of applications (see Chen at abstract).
Claims 2-7 and 9-14 are rejected under 35 U.S.C. § 103 as being unpatentable over Novikov in view of Chen and Lopez-Piqueres et al. (Lopez-Piqueres, Javier, Jing Chen, and Alejandro Perdomo-Ortiz. "Symmetric tensor networks for generative modeling and constrained combinatorial optimization." Machine Learning: Science and Technology 4, no. 3 (2023): 035009, hereinafter Lopez-Piqueres).
As to dependent claim 2, the rejection of claim 1 is incorporated.
Novikov does not appear to expressly teach the mathematical structures are symmetric tensor networks.
Lopez-Piqueres teaches the mathematical structures are symmetric tensor networks (Abstract, "symmetric tensor networks").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the tensorizing neural networks of Novikov to include the symmetric tensor network techniques of Lopez-Piqueres to find novel and better solutions to combinatorial optimization problems (see Lopez-Piqueres at abstract).
As to dependent claim 3, Novikov does not appear to expressly teach the symmetric tensor networks are built using the symmetries of the dataset.
Lopez-Piqueres teaches the symmetric tensor networks are built using the symmetries of the dataset (Page 1, "we show that by exploiting U(1) symmetry in TN states it is possible to encode arbitrary integer-valued equalities of the form A⃗x =⃗b in a probabilistic model").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the tensorizing neural networks of Novikov to include the symmetric tensor network techniques of Lopez-Piqueres to find novel and better solutions to combinatorial optimization problems (see Lopez-Piqueres at abstract).
As to dependent claim 4, Novikov teaches the weight matrices in the deep learning module are replaced by the… tensor networks (Page 6, "replace both fully-connected layers by the TT-layers").
Novikov does not appear to expressly teach symmetric tensor networks.
Lopez-Piqueres teaches symmetric tensor networks (Abstract, "symmetric tensor networks").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the tensorizing neural networks of Novikov to include the symmetric tensor network techniques of Lopez-Piqueres to find novel and better solutions to combinatorial optimization problems (see Lopez-Piqueres at abstract).
As to dependent claim 5, Novikov teaches the system includes a classical optimization algorithm that fine-tunes the parameters of the symmetric tensor deep learning network to minimize a cost function for a training set (Page 8, "We train all the networks with stochastic gradient descent". Page 4, "Back-propagation allows to compute the gradient of a loss-function L with respect to all the parameters of the network").
As to dependent claim 6, Novikov teaches the system includes an inference module that makes predictions over a new set of datapoints (Page 6, "As a baseline we use a neural network with two fully-connected layers (1024 hidden units) and rectified linear unit (ReLU) achieving 1:9% error on the test set". The test set reads on the claimed new set of datapoints).
As to dependent claim 7, Chen teaches the model can be implemented on deep learning chips (Abstract, "we design an accelerator for large-scale CNNs and DNNs, with a special emphasis on the impact of memory on accelerator design, performance and energy").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the tensorizing neural networks of Novikov to include the accelerator techniques of Chen to provide high throughput in a small footprint thereby opening up the usage of state-of-the-art machine-learning algorithms in a broad set of systems and for a broad set of applications (see Chen at abstract).
As to dependent claim 9, the rejection of claim 8 is incorporated.
Novikov does not appear to expressly teach the mathematical structures are symmetric tensor networks.
Lopez-Piqueres teaches the mathematical structures are symmetric tensor networks (Abstract, "symmetric tensor networks").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the tensorizing neural networks of Novikov to include the symmetric tensor network techniques of Lopez-Piqueres to find novel and better solutions to combinatorial optimization problems (see Lopez-Piqueres at abstract).
As to dependent claim 10, Lopez-Piqueres teaches the symmetric tensor networks are built using the symmetries of the dataset (Page 1, "we show that by exploiting U(1) symmetry in TN states it is possible to encode arbitrary integer-valued equalities of the form A⃗x =⃗b in a probabilistic model").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the tensorizing neural networks of Novikov to include the symmetric tensor network techniques of Lopez-Piqueres to find novel and better solutions to combinatorial optimization problems (see Lopez-Piqueres at abstract).
As to dependent claim 11, Novikov teaches the weight matrices in the deep learning module are replaced by the… tensor networks (Page 6, "replace both fully-connected layers by the TT-layers").
Novikov does not appear to expressly teach symmetric tensor networks.
Lopez-Piqueres teaches symmetric tensor networks (Abstract, "symmetric tensor networks").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the tensorizing neural networks of Novikov to include the symmetric tensor network techniques of Lopez-Piqueres to find novel and better solutions to combinatorial optimization problems (see Lopez-Piqueres at abstract).
As to dependent claim 12, Novikov teaches the system includes a classical optimization algorithm that fine-tunes the parameters of the symmetric tensor deep learning network to minimize a cost function for a training set (Page 8, "We train all the networks with stochastic gradient descent". Page 4, "Back-propagation allows to compute the gradient of a loss-function L with respect to all the parameters of the network").
As to dependent claim 13, Novikov teaches the system includes an inference module that makes predictions over a new set of datapoints (Page 6, "As a baseline we use a neural network with two fully-connected layers (1024 hidden units) and rectified linear unit (ReLU) achieving 1:9% error on the test set". The test set reads on the claimed new set of datapoints).
As to dependent claim 14, Chen teaches the model can be implemented on deep learning chips (Abstract, "we design an accelerator for large-scale CNNs and DNNs, with a special emphasis on the impact of memory on accelerator design, performance and energy").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the tensorizing neural networks of Novikov to include the accelerator techniques of Chen to provide high throughput in a small footprint thereby opening up the usage of state-of-the-art machine-learning algorithms in a broad set of systems and for a broad set of applications (see Chen at abstract).
Claims 15-20 are rejected under 35 U.S.C. § 103 as being unpatentable over Novikov in view of Chen, Lopez-Piqueres, and Guo et al. (Guo, Chu, and Dario Poletti. "Matrix product states with adaptive global symmetries." Physical Review B 100, no. 13 (2019): 134304, hereinafter Guo).
As to dependent claim 15, the rejection of claim 14 is incorporated.
Novikov does not appear to expressly teach the symmetric tensor networks are symmetric matrix product operators.
Guo teaches the symmetric tensor networks are symmetric matrix product operators (Page 134304-2, "Adaptively symmetric matrix product operators can be treated in a similar manner as they form a chain of four-dimensional symmetry-protected tensors").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the tensorizing neural networks of Novikov to include the matrix techniques of Guo to allow a superposition of different total quantum numbers (see Guo at page 134304-1).
As to dependent claim 16, Novikov teaches the classical optimization algorithm uses backpropagation or similar (Page 4, "Back-propagation allows to compute the gradient of a loss-function L with respect to all the parameters of the network").
As to dependent claim 17, Novikov teaches the parameters are fine-tuned to minimize a cost function for a training set (Page 8, "We train all the networks with stochastic gradient descent". Page 4, "Back-propagation allows to compute the gradient of a loss-function L with respect to all the parameters of the network").
As to dependent claim 18, Novikov teaches the inference module makes predictions over a new set of datapoints (Page 6, "As a baseline we use a neural network with two fully-connected layers (1024 hidden units) and rectified linear unit (ReLU) achieving 1:9% error on the test set". The test set reads on the claimed new set of datapoints).
As to dependent claim 19, Novikov teaches the predictions are made over a new set of datapoints (Page 6, "As a baseline we use a neural network with two fully-connected layers (1024 hidden units) and rectified linear unit (ReLU) achieving 1:9% error on the test set". The test set reads on the claimed new set of datapoints).
As to dependent claim 20, Novikov teaches the new set of datapoints are unseen data (Page 6, "As a baseline we use a neural network with two fully-connected layers (1024 hidden units) and rectified linear unit (ReLU) achieving 1:9% error on the test set". The test set reads on the claimed unseen data).
Citation of Pertinent Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Stoudenmire et al. (Stoudenmire, Edwin, and David Schwab. "Supervised learning with tensor networks." Advances in neural information processing systems 29 (2016)) teaches supervised learning with tensor networks. Liu et al. (Liu, Ding, Jiaqi Yao, Zekun Yao, and Quan Zhang. "Quantum-classical machine learning by hybrid tensor networks." arXiv preprint arXiv:2005.09428 (2020)) teaches quantum-classical machine learning by hybrid tensor networks.
Conclusion
The prior art made of record and not relied upon is considered pertinent to Applicant's disclosure. Applicant is required under 37 C.F.R. § 1.111(c) to consider these references fully when responding to this action.
It is noted that any citation to specific pages, columns, lines, or figures in the prior art references and any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331, 1332-33, 216 U.S.P.Q. 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 U.S.P.Q. 275, 277 (C.C.P.A. 1968)).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Casey R. Garner whose telephone number is 571-272-2467. The examiner can normally be reached Monday to Friday, 8am to 5pm, Eastern Time.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey Shmatov can be reached on 571-270-3428. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center and the Private Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from Patent Center or Private PAIR. Status information for unpublished applications is available through Patent Center and Private PAIR to authorized users only. Should you have questions about access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) Form at https://www.uspto.gov/patents/uspto-automated- interview-request-air-form.
/Casey R. Garner/Primary Examiner, Art Unit 2123
Prosecution Insights