Detailed Action
This action is in response to the original filing on 01/10/2024. Claims 1-12 are pending and are considered below.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-12 are rejected under 35 U.S.C. 103 as being unpatentable over CROXFORD et al. (US 20200342285 A1, hereinafter Croxford) in view of MUKHOPADHYAY et al. (US 20230367995 A1, hereinafter Mukhopadhyay).
Regarding Claim 1, Croxford teaches an information processing apparatus comprising one or more memories storing instructions and one or more processors that execute the instructions to: (Paragraph [0001] The present disclosure relates to methods and apparatus for processing data with a neural network system. Paragraph [0078] Computer executable code that includes instructions for performing various operations of the controller 340 described herein can be stored in the memory.)
acquire a learned parameter of the shared layer based on the acquired integration ratio and learned parameters of the replacement layers acquired by the process for learning. (Paragraph [0038] FIG. 2 shows an example in which a neural network system comprises a merged layer 103 shareable between a first neural network 100 and a second neural network 200. Paragraph [0040] The merged layer 103 is configured to apply a superset of weights 120 to the input data 110 to generate intermediate feature data 130 representative of at least one feature of the input data 110. Paragraph [0041] Implementing the merged layer 103 to apply the superset of weights 120 may allow for fewer memory accesses compared to reading the first and second sets of weights separately from storage, and thus allow for more efficient processing of the neural network system. Paragraph [0042] For example, a first layer serving the first neural network 100 may be merged with a second layer serving the second neural network 200 to form the merged layer 103. The merging may involve combining the first and second sets of weights, e.g. stored in storage, to form the superset of weights as described herein. The first and second layers may then be replaced with the merged layer 103 shared between the first and second neural networks 100, 200. Paragraph [0045] The kernel corresponding to the superset of weights may thus be associated with the said same activation function when used to generate the intermediate feature data 130 at the merged layer 103. For example, a merged layer formed from three respective layers across three neural networks will have a superset of weight data combined from three sets of weights and could have a superset of bias data formed from three sets of biases and/or be associated with three different activation functions. (The merged layer (integrated) is shared between two or more neural networks. This layer applies weights (in the form of a kernel applied to the layer) in order to generate feature data. Furthermore, a superset of weights is acquired from the merging (integration) of these primary layers. The specification defines a learned parameter as "memory used for holding weights" (Spec [0044]). Paragraph [0041] of the reference makes clear one benefit of merging layers comprises fewer memory accesses when applying these weights, and so this comprises a 'learned parameter' in light of the specification.))
Croxford does not teach: acquire, by a process for learning a plurality of tasks, an integration ratio of output of replacement layers for which a shared layer, which is shared by a plurality of tasks in a hierarchical neural network, is replaced by a neural network layer for each task; and
In the same field of endeavor, Mukhopadhyay teaches: acquire, by a process for learning a plurality of tasks, an integration ratio of output of replacement layers for which a shared layer, which is shared by a plurality of tasks in a hierarchical neural network, is replaced by a neural network layer for each task; and (Paragraph [0151] In some configurations, the modifying of the CNN further comprises: identifying, via the processor, at least one layer of the CNN corresponding to the at least one drop-off point; identifying, via the processor within a database of neural network layers, at least one replacement layer associated which resonates with the pre-defined features; and replacing, via the processor, the at least one layer of the CNN with the at least one replacement layer, resulting in the modified CNN. In such configurations, the pre-defined features can be associated with recognition of a particular type of object having a pre-defined scale, and the replacement layer is associated with a distinct pre-defined scale. (Respondent to a detected drop-off in a CNN's layer, the system identifies the layer and replaces it with one that is more suitable to the task at hand (fitting the correct pre-defined scale). This new layer is already predetermined to be resonant with the predefined features, and so integrates accordingly. In light of the instant specification, the 'integration ratio' can be expressed as a contribution to a learning task [0061]. The modification determines the contribution and selects accordingly))
It would have been obvious to one having ordinary skill in the art before the effective filing date to have incorporated the method of acquiring an integration ratio analogous to the output of weights in shared layers as described by Mukhopadhyay into Croxford as they are both in the same field of neural network data analysis and doing so would provide obvious benefits, one such example of which being the increased capacity for neural networks to quickly recognize subjects in an image or other types of non-imaging data. Currently, when compared to a human, a neural network needs far more training data to recognize images. This combination would thus represent an improvement in the state of the art by allowing for development in this area (Mukhopadhyay Paragraphs [0003] - [0004]).
Regarding Claims 9 and 11, Claims 9 and 11 are corresponding method and computer readable storage claims to the apparatus claim of Claim 1, and so are rejected for the same reasons.
Regarding Claim 2, the combination of Croxford and Mukhopadhyay teaches: wherein the one or more processors execute the instructions to acquire, as a shared layer replacement model, a hierarchical neural network in which a shared layer in the hierarchical neural network is replaced by replacement layers for each task, and (Croxford Paragraph [0042] For example, a first layer serving the first neural network 100 may be merged with a second layer serving the second neural network 200 to form the merged layer 103. Paragraph [0044] The first and second sets of weights, which are combined to form the superset of weights, may be applicable to the input data in the form of respective first and second kernels. (The example given is of a convolutional neural network, a form of hierarchical neural network. The shared primary layers are replaced by a merged primary layer, which is also shared by two neural networks.))
acquire, as the integration ratio, a weight that is updated by performing a process for learning the hierarchical neural network, which is made to contain an integration network in which products of output from each replacement layer corresponding to the shared layer in the shared layer replacement model and the weight are integrated and output. (Croxford Paragraph [0040] The merged layer 103 is configured to apply a superset of weights 120 to the input data 110 to generate intermediate feature data 130 representative of at least one feature of the input data 110. The superset of weights 120 is combined from a first set of weights associated with the first neural network 100 and a second set of weights associated with the second neural network 200. The first set of weights may be an outcome of training the first neural network 100, and the second set of weights may be an outcome of training the second neural network 200, for example. In some cases, the merged layer 103 may have been produced by merging respective layers from the first and second neural networks 100, 200, with the said layers being respectively associated with the first and second sets of weights. The respective layers may have been configured to receive the same input data. The respective layers may have been trained independently of one another, as part of their respective neural network 100, 200, to produce the first and second sets of weights. (Acquiring weights updated for the process, of which two neural networks share an 'integrated' (merged) layer.))
Regarding Claim 3, the combination of Croxford and Mukhopadhyay teaches: wherein the one or more processors execute the instructions to acquire the learned parameter of the shared layer such that output from the replacement layers is equivalent to output of the integration network. (Croxford Paragraph [0031] Hence, the output of the training may be a plurality of kernels associated with a predetermined neural network architecture (for example with different kernels being associated with different respective layers of a multi-layer neural network architecture). The kernel data may be considered to correspond to weight data representative of weights to be applied to image data, as each element of a kernel may be considered to correspond to a weight, respectively. Paragraph [0040] The merged layer 103 is configured to apply a superset of weights 120 to the input data 110 to generate intermediate feature data 130 representative of at least one feature of the input data 110. (Acquiring the weights through the output))
Regarding Claim 4, the combination of Croxford and Mukhopadhyay teaches: wherein, in a case where the integration ratio of output of the replacement layers is acquired, the one or more processors execute the instructions to acquire the learned parameter of the shared layer based on the integration ratio. (Croxford Paragraph [0040] The first set of weights may be an outcome of training the first neural network 100, and the second set of weights may be an outcome of training the second neural network 200, for example. In some cases, the merged layer 103 may have been produced by merging respective layers from the first and second neural networks 100, 200, with the said layers being respectively associated with the first and second sets of weights. The respective layers may have been configured to receive the same input data. (Acquiring the weights (learned parameter) based on a condition, in this case the outcome of training))
Regarding Claim 5, the combination of Croxford and Mukhopadhyay teaches: wherein, in a case where an integration ratio of output of the replacement layers corresponding to a respective shared layer is acquired, the one or more processors execute the instructions to acquire the learned parameter of that shared layer based on that integration ratio and the learned parameters of the replacement layers. (Croxford Paragraph [0040] The first set of weights may be an outcome of training the first neural network 100, and the second set of weights may be an outcome of training the second neural network 200, for example. In some cases, the merged layer 103 may have been produced by merging respective layers from the first and second neural networks 100, 200, with the said layers being respectively associated with the first and second sets of weights. The respective layers may have been configured to receive the same input data. (As the shared layer and the replaced layer are one in the same in the referenced system, the acquired weights for the merged layer fulfill both roles in the claimed limitation))
Regarding Claim 6, the combination of Croxford and Mukhopadhyay teaches: wherein the task is a plurality of sub-tasks corresponding to one task. (Mukhopadhyay Paragraph [0027] In addition, the system disclosed herein can perform recognition on an object using the original subset (lions, tigers, and bears), and recognize that the object is not any of the known objects. (The task (image recognition) is divided into sub-tasks (identifying multiple unknown animals)))
Regarding Claim 7, the combination of Croxford and Mukhopadhyay teaches: wherein the replacement layer comprises a neural network structure that is the same as the shared layer. (Croxford Paragraph [0042] The merging may involve combining the first and second sets of weights, e.g. stored in storage, to form the superset of weights as described herein. The first and second layers may then be replaced with the merged layer 103 shared between the first and second neural networks 100, 200. (Replacement layers are the same structure as the original layer/layers as the replacement becomes the primary layer of both neural networks))
Regarding Claim 8, Croxford teaches an information processing apparatus comprising one or more memories storing instructions and one or more processors that execute the instructions to (Paragraph [0001] The present disclosure relates to methods and apparatus for processing data with a neural network system. Paragraph [0078] In some examples, the controller 340 includes a processor and a memory. Computer executable code that includes instructions for performing various operations of the controller 340 described herein can be stored in the memory.) including:
acquire a learned parameter of a layer that integrates two or more unique layers based on the acquired integration ratio and a learned parameter of the unique layers. (Croxford Paragraph [0042] The merging may involve combining the first and second sets of weights, e.g. stored in storage, to form the superset of weights as described herein. In examples, the first and second layers may be the first unique layers of the first and second neural networks 100, 200. Paragraph [0044] The first and second sets of weights, which are combined to form the superset of weights, may be applicable to the input data in the form of respective first and second kernels.)
Croxford does not teach: acquire, by learning processing, an integration ratio for output of a unique layer that is unique to each task in a hierarchical neural network;
In the same field of endeavor, Mukhopadhyay teaches: acquire, by learning processing, an integration ratio for output of a unique layer that is unique to each task in a hierarchical neural network; (Mukhopadhyay Paragraph [0112] That is, after training, the layers will be transferred to an untrained classification network and frozen during training to ensure the network uses only the previously learned weights at those layers. Multiple classification networks are built using the GAN layers each measuring the applicability at that layer. (Determining the accuracy of weights in learning tasks (measuring the applicability)))
It would have been obvious to one having ordinary skill in the art before the effective filing date to have incorporated acquiring a learned parameter of a layer which integrates two or more layers as described by Mukhopadhyay into Croxford as they are both in the same field of neural network data analysis and doing so would provide obvious benefits, one such being the improved capacity for data analysis in the field of image recognition. Currently, neural networks need far more training data than a human in accomplishing the same task. Thusly, this combination would represent an improvement in the state of the art by further helping to accomplish this goal (Mukhopadhyay Paragraphs [0003] - [0004]).
Regarding Claims 10 and 12, Claims 10 and 12 are corresponding method and computer readable storage claims to the apparatus claim of Claim 8, and so are rejected for the same reasons.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure.
Lan et al. (US 20230274158 A1) encompasses weight matrices which were deemed similar to the acquired parameters of the instant application
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JUSTIN A CARDOSO whose telephone number is (571)272-8512. The examiner can normally be reached M-F 7:30 - 5:00, alternate Friday's off.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Welch can be reached at (571) 272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JUSTIN CARDOSO/
Patent Examiner, Art Unit 2143
/JENNIFER N WELCH/ Supervisory Patent Examiner, Art Unit 2143
Prosecution Insights