DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) filed on 1/4/2024 and 5/20/2024 were considered and placed on the file of record by the examiner.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 20-32 are rejected under 35 U.S.C. 103 as being unpatentable over Harris et al., “Architectures and algorithms for on-device user customization of CNNs” in view of Yehezkel Rohekar (US 2018/0322365).
Regarding claim 20, Harris teaches a recognition method comprising: obtaining a feature extractor including a first neural network having a fixed parameter dependent on pretraining and a second neural network having an adjustable parameter dependent on an on-device training (see figure 3, figure 4, section 3, Harris discusses an architecture that comprises a large CNN trained on general data and a small CNN re-trained on-device using user-specific inputs, training a network on general data to obtain fixed parameters and a second network with adjustable parameters trained on user data).
Yehezkel teaches performing the on-device training on the feature extractor based on user data corresponding to a valid user and reference data corresponding to generalized users (see figure 8A, figure 8B, figure 8C, figure 15, para. 0045, 0049, Yehezkel discusses training a network on general user independent data and a second network trained on specific user dependent data),
when the on-device training is completed, performing user recognition on test data using the feature extractor (see figure 8A, figure 8B, figure 8C, figure 15, Yehezkel discusses user-dependent classifier/model referring to a classifier which, once trained, has an accuracy level that is guaranteed only for a specific user; see para. 0183, Yehezkel discusses training the second network for a specific user using any user - independent appearance features from the first network).
Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Harris with Yehezkel to derive at the invention of claim 20. The result would have been expected, routine, and predictable in order to perform neural network transfer learning.
The determination of obviousness is predicated upon the following: One skilled in the art would have been motivated to modify Harris in this manner in order to improve AI image recognition by using an offline pretrained neural network based on general user data, and transferring learned data to a device adapted and fine-tuned to specific users, thereby increasing network efficiency. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in this manner explained using known engineering design, interface and/or programming techniques, without changing a fundamental operating principle of Harris, while the teaching of Yehezkel continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of implementing a pretrained neural network to a second neural network to transfer learned knowledge from a large dataset to a specific neural network thereby allowing faster and more efficient performance. The Harris and Yehezkel systems perform neural network training, therefore one of ordinary skill in the art would have reasonable expectation of success in the combination. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Regarding claim 21, Yehezkel teaches wherein the adjustable parameter of the second neural network is adjusted by the on-device training (see para. 0251-0253, 0255, Yehezkel discusses training and adapting a second neural network).
The same motivation of claim 20 is applied to claim 21. Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Harris with Yehezkel to derive at the invention of claim 21. The result would have been expected, routine, and predictable in order to perform neural network transfer learning.
Regarding claim 22, Yehezkel teaches wherein performing the on-device training comprises: inputting the user data to the first neural network (see figure 8A, figure 8B, figure 8C, figure 15, para. 0045, 0049, Yehezkel discusses training a network on general user independent data and a second network trained on specific user dependent data),
inputting, to the second neural network, the reference data and an output from the first neural network in response to the input of the user data (see figure 8A, figure 8B, figure 8C, figure 15, para. 0045, 0049, Yehezkel discusses inputting into the second network the output of the first neural network); and
performing the on-device training based on an output from the second neural network (see para. 0251-0253, 0255, Yehezkel discusses training and adapting the second neural network).
The same motivation of claim 20 is applied to claim 22. Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Harris with Yehezkel to derive at the invention of claim 22. The result would have been expected, routine, and predictable in order to perform neural network transfer learning.
Regarding claim 23, Harris teaches an on-device training method for a feature extractor provided in a user device, and the feature extractor including a pretrained first neural network having a fixed parameter and a second neural network having an adjustable parameter dependent on an on-device training, the on-device training method comprising: obtaining user data input by a valid user; inputting the user data to the first neural network having the fixed parameter (see figure 3, figure 4, section 3, Harris discusses an architecture that comprises a large CNN trained on general data and a small CNN re-trained on-device using user-specific inputs, training a network on general data to obtain fixed parameters and a second network with adjustable parameters trained on user data).
Yehezkel teaches adjusting the adjustable parameter of the second neural network by inputting, to the second neural network, preset reference data and an output from the first neural network in response to the input of the user data (see figure 8A, figure 8B, figure 8C, figure 15, para. 0045, 0049, Yehezkel discusses training a network on general user independent data and a second network trained on specific user dependent data).
Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Harris with Yehezkel to derive at the invention of claim 23. The result would have been expected, routine, and predictable in order to perform neural network transfer learning.
The determination of obviousness is predicated upon the following: One skilled in the art would have been motivated to modify Harris in this manner in order to improve AI image recognition by using an offline pretrained neural network based on general user data, and transferring learned data to a device adapted and fine-tuned to specific users, thereby increasing network efficiency. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in this manner explained using known engineering design, interface and/or programming techniques, without changing a fundamental operating principle of Harris, while the teaching of Yehezkel continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of implementing a pretrained neural network to a second neural network to transfer learned knowledge from a large dataset to a specific neural network thereby allowing faster and more efficient performance. The Harris and Yehezkel systems perform neural network training, therefore one of ordinary skill in the art would have reasonable expectation of success in the combination. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Regarding claim 24, Yehezkel teaches wherein the reference data includes 1000 or fewer feature vectors (see para. 0206, 0209-0210, Yehezkel discusses a mini-batch buffer that contains some reference feature vectors associated with independent user data).
The same motivation of claim 23 is applied to claim 24. Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Harris with Yehezkel to derive at the invention of claim 24. The result would have been expected, routine, and predictable in order to perform neural network transfer learning.
Regarding claim 25, Yehezkel teaches wherein the reference data includes 500 or fewer feature vectors (see para. 0206, 0209-0210, Yehezkel discusses a mini-batch buffer that contains some reference feature vectors associated with independent user data).
The same motivation of claim 23 is applied to claim 25. Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Harris with Yehezkel to derive at the invention of claim 25. The result would have been expected, routine, and predictable in order to perform neural network transfer learning.
Regarding claim 26, Yehezkel teaches wherein the reference data includes 100 or fewer feature vectors (see para. 0206, 0209-0210, Yehezkel discusses a mini-batch buffer that contains some reference feature vectors associated with independent user data).
The same motivation of claim 23 is applied to claim 26. Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Harris with Yehezkel to derive at the invention of claim 26. The result would have been expected, routine, and predictable in order to perform neural network transfer learning.
Regarding claim 27, Yehezkel teaches wherein the reference data includes generalized feature vectors corresponding to the generalized users (see para. 0206, 0209-0210, Yehezkel discusses a mini-batch buffer that contains some reference feature vectors associated with independent user data).
The same motivation of claim 23 is applied to claim 27. Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Harris with Yehezkel to derive at the invention of claim 27. The result would have been expected, routine, and predictable in order to perform neural network transfer learning.
Claim 28 is rejected as applied to claim 20 as pertaining to a corresponding apparatus.
Claim 29 is rejected as applied to claim 21 as pertaining to a corresponding apparatus.
Claim 30 is rejected as applied to claim 22 as pertaining to a corresponding apparatus.
Regarding 31, Harris teaches a method comprising: after the first neural network is pretrained, providing the feature extractor to a device, the feature extractor comprising the first neural network having a fixed parameter dependent on the pretraining and a second neural network having an adjustable parameter dependent on-device training (see figure 3, figure 4, section 3, Harris discusses an architecture that comprises a large CNN trained on general data and a small CNN re-trained on-device using user-specific inputs, training a network on general data to obtain fixed parameters and a second network with adjustable parameters trained on user data).
Yehezkel teaches performing pretraining of a first neural network of a feature extractor at a server end (see figure 7, para. 0180, Yehezkel discusses training a first neural network offline at a server);
performing the on-device training of the second neural network of the feature extractor on the device using data input to the device (see figure 7, para. 0180, Yehezkel discusses on the fly learning training of a second neural network at a user’s device); and
performing user recognition on test data input to the device using the feature extractor (see figure 8A, figure 8B, figure 8C, figure 15, para. 0212, Yehezkel discusses user-dependent classifier/model referring to a classifier which, once trained, has an accuracy level that is guaranteed only for a specific user).
Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Harris with Yehezkel to derive at the invention of claim 31. The result would have been expected, routine, and predictable in order to perform neural network transfer learning.
The determination of obviousness is predicated upon the following: One skilled in the art would have been motivated to modify Harris in this manner in order to improve AI image recognition by using an offline pretrained neural network based on general user data, and transferring learned data to a device adapted and fine-tuned to specific users, thereby increasing network efficiency. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in this manner explained using known engineering design, interface and/or programming techniques, without changing a fundamental operating principle of Harris, while the teaching of Yehezkel continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of implementing a pretrained neural network to a second neural network to transfer learned knowledge from a large dataset to a specific neural network thereby allowing faster and more efficient performance. The Harris and Yehezkel systems perform neural network training, therefore one of ordinary skill in the art would have reasonable expectation of success in the combination. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Regarding claim 32, Yehezkel teaches wherein the data input to the device includes user data input by a valid user for user registration and reference data corresponding to generalized users (see figure 8A, figure 8B, figure 8C, figure 15, para. 0045, 0049, Yehezkel discusses training a network on general user independent data and a second network trained on specific user dependent data).
The same motivation of claim 31 is applied to claim 32. Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Harris with Yehezkel to derive at the invention of claim 32. The result would have been expected, routine, and predictable in order to perform neural network transfer learning.
Claim 33 is rejected under 35 U.S.C. 103 as being unpatentable over Harris et al., “Architectures and algorithms for on-device user customization of CNNs” in view of Yehezkel Rohekar (US 2018/0322365) in view of Mostafa et al. (US 2019/0042835).
Regarding claim 33, Harris and Yehezkel do not expressly disclose further comprising performing the user recognition by comparing a registration feature vector corresponding to the user data to a test feature vector corresponding to the test data inputs of the user data and the reference data.
However, Mostafa teaches further comprising performing the user recognition by comparing a registration feature vector corresponding to the user data to a test feature vector corresponding to the test data inputs of the user data and the reference data (see claim 1, para. 0076, 0110, Mostafa discusses comparing the first feature vector to one or more second reference templates stored in the memory of the device to obtain a second matching score).
Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Harris and Yehezkel with Mostafa to derive at the invention of claim 33. The result would have been expected, routine, and predictable in order to perform neural network transfer learning.
The determination of obviousness is predicated upon the following: One skilled in the art would have been motivated to modify Harris and Yehezkel this manner in order to improve AI image recognition by using an offline pretrained neural network based on general user data, and transferring learned data to a device adapted and fine-tuned to recognize specific users, thereby increasing network efficiency. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in this manner explained using known engineering design, interface and/or programming techniques, without changing a fundamental operating principle of Harris and Yehezkel, while the teaching of Mostafa continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of implementing a pretrained neural network to a second neural network to transfer learned knowledge from a large dataset to a specific neural network thereby allowing faster and more efficient performance. The Harris, Yehezkel, and Mostafa systems perform neural network training, therefore one of ordinary skill in the art would have reasonable expectation of success in the combination. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Mao et al. (US 2020/0202168) discusses an object classifier neural network with a first set of training data, and training, in a second phase after completion of the first phase, the object classifier neural network with a second set of training data.
Mu et al. (US 2016/0358043) discusses training the neural network by iteratively adjusting parameters of the neural network based on multiple loss functions to the subset of images.
Choi et al., “TrainWare: A Memory Optimized Weight Update Architecture for On-Device Convolutional Neural Network Training” discusses an on-device training neural network system.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KENNY A CESE whose telephone number is (571) 270-1896. The examiner can normally be reached on Monday – Friday, 9am – 4pm.
If attempts to reach the primary examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached on (571) 272-3838. The fax phone number for the organization where this application or proceeding is assigned is (571) 273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Kenny A Cese/
Primary Examiner, Art Unit 2663