DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The Information Disclosure Statement (IDS) submitted by Applicant on 12/10/2025 has been considered.
Status of Claims
Applicant submitted a Pre-Appeal Brief Request for Review on 08/25/2025. Pursuant to Notice of Panel Decision form Pre-Appeal Brief Review, dated 10/31/2025, prosecution has been reopened, the rejection of claims 1-20 have been withdrawn and a new Non-Final Office Action is being issued herein.
No claims have been amended, cancelled or added by Applicant. Claims 1-20 are currently pending.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 3, 4, 16, 19, and 20 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Li et al. (U.S. Patent No. 10,748,057 filed Sept. 21, 2016 and published Aug. 18, 2020).
Regarding claim 1, Li teaches an apparatus for managing a program (Li, Col. 7, lines 2-6 teaches Some implementations are directed to using version identifiers of neural network modules in determining whether and/or how to combine multiple neural network modules to generate a combined neural network model for use by a robot and/or other apparatus.), the apparatus comprising:
at least one storage device (Li, Col. 21, lines 62-67 and Col. 22, lines 1-4 teaches FIG. 11 is a block diagram of an example computing device 1110 that may optionally be utilized to perform one or more aspects of techniques described herein. Computing device 1110 typically includes at least one processor 1114 which communicates with a number of peripheral devices via bus subsystem 1112. These peripheral devices may include a storage subsystem 1124, including, for example, a memory subsystem 1125 and a file storage subsystem 1126, user interface output devices 1120, user interface input devices 1122, and a network interface subsystem 1116.) storing a common neural network, a first program comprising a first neural network, a second program comprising a second neural network, and mapping information, wherein the mapping information indicates that the first program utilizes the common neural network, and the second program utilizes the common neural network (Li, Col. 5, lines 24-40 teaches a method is provided that includes identifying a neural network model stored in one or more computer readable media of a robot and in use by the robot. The neural network model includes a first neural network module [i.e. a first program comprising a first neural network] and a second neural network module [i.e., a second program comprising a second neural network] the first neural network module and the second neural network module each having a plurality of endpoints, and in the neural network model a first endpoint of the endpoints of the first neural network module is joined to a second endpoint of the endpoints of the second neural network module.; Li, Fig. 7, 752 and corresponding description in column 18, lines 38-43 teaches at block 752, the system identifies existing neural network modules of a combined neural network model, and identifies version identifiers of those existing modules.; Li, column 9, lines 25-27 further teaches optionally generating a request to include version identifier(s) of endpoints of other modules that are joined to the given module in the existing combined model of robot 180A.; Li, Col. 10, lines 31-36 teaches In some implementations, assigning a version identifier to an endpoint of a version of a neural network module includes storing, in one or more computer readable media (e.g., database 130), the version identifier and a mapping (e.g., data defining an association) of the version identifier to the endpoint. [Note: Li, Col. 10, lines 31-36, Li, Fig. 7, 752, Col. 18, lines 38-43, and Col. 9, lines 25-27 have been understood to teach stored identifiers of the modules/sub-components needed to use in the combined neural network model, the identifiers of the modules reading on mapping information indicates that the first program utilizes the common neural network, and the second program utilizes the common neural network, as claimed); and
at least one processor (Li, Col. 21, lines 62-67 and Col. 22, lines 1-4 teaches FIG. 11 is a block diagram of an example computing device 1110 that may optionally be utilized to perform one or more aspects of techniques described herein. Computing device 1110 typically includes at least one processor 1114 which communicates with a number of peripheral devices via bus subsystem 1112.) configured to:
based on execution of the first program: load the common neural network using the mapping information, and perform, using the common neural network and the first neural network, a first processing for first data of the first program, and obtain a first execution result of the first processing for the first data based on the common neural network and the first neural network (Li, Col. 1, lines 36-39, teaches a combined neural network model (also referred to herein as a “combined model”) joins a plurality of individual neural network modules (also referred to herein as “modules” to perform a particular task [i.e., to perform a particular task reading on execution of a program]; Li, Col. 1, lines 39-49 further teaches each of the neural network modules [i.e., neural network programs] is utilized in performing a “subtask” [i.e., execution of a first program] in performing the overall task of the combined neural network program. For example, a combined model may be utilized to perform the overall task of determining a classification of an object in a raw image by utilizing: a first module that can be utilized to perform a firs subtask of extracting image features from the raw image [i.e., Li, Col. 1, lines 39-49, as disclosed, understood to read on the loaded common neural network using the mapping information, and perform, using the common neural network and the first neural network]; Li, Col. 1, lines 50-52 further teach each neural network module includes defined endpoints. Each of the endpoints includes one or more input(s) and/or output(s) and has a shape and a datatype. [i.e., the outputs teaching “obtain a first execution result of the first processing…”, as claimed; Li, Col. 10, lines 31-36 teaches In some implementations, assigning a version identifier to an endpoint of a version of a neural network module includes storing, in one or more computer readable media (e.g., database 130), the version identifier and a mapping (e.g., data defining an association) of the version identifier to the endpoint.); and
based on execution of the second program: load the common neural network using the mapping information, and perform, using the common neural network and the second neural network, a second processing for second data of the second program, and obtain a second execution result of the second processing for the second data based on the common neural network and the second neural network (Li, Col. 1, lines 36-39, teaches a combined neural network model (also referred to herein as a “combined model”) joins a plurality of individual neural network modules (also referred to herein as “modules” to perform a particular task [i.e., to perform a particular task reading on execution of a program]; Li, Col. 1, lines 39-49 further teaches each of the neural network modules [i.e., neural network programs] is utilized in performing a “subtask” [i.e., execution of a first program] in performing the overall task of the combined neural network program. For example, a combined model may be utilized to perform the overall task of determining a classification of an object in a raw image by utilizing:.. a second module that can be utilized to perform a second subtask [i.e., execution of a second program] of determining an object class based on the image features extracted from the raw image by the first neural first neural network module. [i.e., Li, Col. 1, lines 39-49, as disclosed, understood to read on the loaded common neural network using the mapping information, and perform, using the common neural network and the second neural network]; Li, Col. 1, lines 50-52 further teach each neural network module includes defined endpoints. Each of the endpoints includes one or more input(s) and/or output(s) and has a shape and a datatype. [i.e., the outputs teaching “obtain a second execution result of the second processing…”, as claimed]; Li, Col. 10, lines 31-36 teaches In some implementations, assigning a version identifier to an endpoint of a version of a neural network module includes storing, in one or more computer readable media (e.g., database 130), the version identifier and a mapping (e.g., data defining an association) of the version identifier to the endpoint.).
Regarding claim 3, Li teaches all of the limitations of claim 1, and Li further teaches wherein the at least one processor is further configured to: determine a third neural network of the first program or a fourth neural network of the second program as the common neural network (Li, Col. 8, lines 47-67 teaches The request engine 112 generates and/or receives various requests related to combining multiple neural network modules in a combined neural network model. The request engine 112 provides those requests to matching engine 114 to enable matching engine 114 to identify module(s) that conform to the requests. As one example, a user may utilize a graphical user interface of one of the client computing device(s) 105 to select a module of neural network modules database 130. In response, the request engine 112 may receive or generate a request to locate other module(s) of neural network modules database 130 that have an endpoint that is compatible and/or fully compatible with any of the endpoints of the selected module. For instance, a user may select a module that has an outputs endpoint with a data type of “image features” and a shape of 64×1. The request engine 112 may generate a request to locate other modules that have inputs endpoints with a data type of “image features” and a shape of 64×1. In some of those instances, the request engine 112 may optionally generate the request to locate only other modules that have inputs endpoints with a version identifier that matches that of the outputs endpoint of the selected module (e.g., if user interface input is provided that indicates a desire for only modules fully compatible with the outputs endpoint of the selected module).; Li, Col. 10, lines 10-35, teaches the version identifier engine 118 assigns version identifiers to endpoints of versions of neural network modules in response to training performed by training engine 116. For example, the version identifier engine 118 assigns a version identifier to an endpoint of a version of a module to reflect that the version of the module has been trained when the endpoint was joined to an additional endpoint of an additional neural network module. The version identifier assigned to the endpoint may match (e.g., is the same as or otherwise identifies) a version identifier assigned to the additional module, such as a version identifier assigned to the additional endpoint to which the endpoint was joined during training of the module. The version identifier can be used to “tie” the version of the module to the additional module. For example, the version identifier can be used to restrict the version of the module to use in combined models that join the endpoint of the version of the module to the additional endpoint of the additional module. For instance, the neural network module system 110 and/or the robots 180A and 180B may prevent joined modules from being used if the endpoints that join the modules do not have matching version identifiers. In some implementations, assigning a version identifier to an endpoint of a version of a neural network module includes storing, in one or more computer readable media (e.g., database 130), the version identifier and a mapping (e.g., data defining an association) of the version identifier to the endpoint.) [Note: the combined models in Li are matched and/or tied to the respective identified modules].
Regarding claim 4, Li teaches all of the limitations of claim 1, and Li further teaches wherein the at least one processor is further configured to control to store in the at least one storage device first files constituting the first program and second files constituting the second program (Li, Col. 22, lines 45-48 teaches the modules [i.e., first and second programs] implementing the functionality of certain implementations may be stored by file storage subsystem 1126 in the storage subsystem 1124, or in other machines accessible by the processor(s) 1114.).
Regarding claim 16, Li teaches all of the limitations of claim 3, and Li further teaches wherein the at least one processor is further configured to: obtain update data related to the third neural network of the first program, obtain a fifth neural network corresponding to the update data, determine that the fifth neural network and the common neural network do not structurally correspond to each other, and store the fifth neural network as a neural network of the first program in the at least one storage device (Li, Col. 3, lines 7-22 teaches in response to determining that the first version identifier does not match the second version identifier, training at least the first neural network module to generated a refined version of the combined neural network model; and using the refined version of the combined neural network model. Training the first neural network module includes applying training examples to the combined neural network model. Li, Col. 3, lines 34-37 further teaches in response to training at least the first neural network module to generate the refined version: replacing the first version identifier with the new version identifier.; Li, Col. 10, lines 31-36 further teaches assigning a version identifier to an endpoint of ta version of a neural network module includes storing, in one or more computer readable media (e.g., a database) the version identifier and the mapping of the version identifier to the endpoint. [Note: the “refined version” of the first neural network model being understood as obtain update data (i.e., through training) and obtaining a fifth neural network corresponding to the update data.; It is further understood that replacing the first version identifier with the new version identifier teaches storing the fifth neural network as the neural network of the first program])
Regarding claim 19,
Claim 19 (directed towards a method) recites the same and/or analogous limitations as claim 1, and thus it is rejected based on the same rationale as claim 1.
Regarding claim 20,
Claim 20 recites the same and/or analogous limitations as claim 1, and thus it is rejected based on the same rationale as claim 1.
Claim 20 further recites a non-transitory computer-readable medium having embodied thereon a program for executing a method of an apparatus for managing the program.
Li teaches a non-transitory computer-readable medium having embodied thereon a program for executing a method of an apparatus for managing the program (Li, Col. 6, lines 12-16 teaches implementations may include a non-transitory computer readable storage medium storing instructions executable by at least one processor ( e.g., a central processing unit (CPU) and/or graphics processing unit (GPU)) to perform a method such as one or more of the methods described above.)
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Li et al. (U.S. Patent No. 10,748,057 filed Sept. 21, 2016 and published Aug. 18, 2020), as applied to claim 1, in view of Abolmaesumi et al. (US 20190125298 A1, filed Oct. 22, 2018 and published May 2, 2019)
Regarding claim 2, Li teaches all of the limitations of claim 1, however Li does not distinctly disclose wherein the at least one processor is further configured to: based on execution of the first program, obtain a first file related to the first neural network and a third file related to the common neural network for performing the first processing, and based on execution of the second program, obtain a second file related to the second neural network and the third file related to the common neural network for performing the second processing.
Nevertheless, Abolmaesumi teaches wherein the at least one processor is further configured to: based on execution of the first program, obtain a first file related to the first neural network and a third file related to the common neural network for performing the first processing, and based on execution of the second program, obtain a second file related to the second neural network and the third file related to the common neural network for performing the second processing (Abolmaesumi, [0094] Block 404 may direct the analyzer processor 100 to read the image files received at block 202 from the location 140 of the storage memory 104 and to read the common neural network record 320 and the view category specific neural network record 340 from the location 146 of the storage memory, and to input the image data from the image files into a neural network that includes the shared layers 362 and the view category specific layers 374 shown in FIG. 8, which are defined by the common neural network record 320 and the view category specific neural network record 340, to generate or determine a view category specific quality assessment value as an output of the neural network.; See Fig. 9 showing the common neural network record [i.e., common neural network file] and Fig. 10 showing an exemplary View category specific neural network record [i.e., one of the plurality of view category specific neural network records [i.e., first and second files]; [Note: common neural network record and view category specific neural network record understood to read on files “a first file related to the first neural network and a third file related to the common neural network”).
Before the effective filing date of the claimed invention it would have been obvious to one of ordinary skill in the art to have modified the method and apparatus related to combining neural network modules based on version identifiers assigned to the neural network modules, as taught by Li, to further include the neural network processing by a common/shared neural network and view category specific neural networks, as taught by Abolmaesumi. Splitting the neural network 360 into a common portion and view category specific portions may facilitate more efficient training of the neural networks. In some embodiments, splitting the neural network 360 into a common portion and view category specific portions may result in requiring fewer learning parameters than would be required if using fully separate neural networks, which may help facilitate easier transferring of a neural network to a new machine, and/or may reduce memory usage. (Abolmaesumi, Paragraph [0089])
[EXAMINER NOTE: Li, Col. 22, lines 45-48 teaches the modules implementing the functionality of certain implementations may be stored by file storage subsystem 1126 in the storage subsystem 1124, or in other machines accessible by the processor(s) 1114.]
Claims 5 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Li et al., as applied to claim 3, in view of Okuno et al. (US 20170344881 A1, filed May 25, 2016, and published Nov. 30, 2017)
Regarding claim 5, Li teaches all of the limitations of claim 3 and however Li does not distinctly disclose wherein the at least one processor is further configured to: obtain first structure information of the third neural network from a first metafile included in the first program, obtain second structure information of the fourth neural network from a second metafile included in the second program, and determine the third neural network or the fourth neural network as the common neural network by comparing the first structure information with the second structure information.
Nevertheless, Okuno teaches wherein the at least one processor is further configured to: obtain first structure information of the third neural network from a first metafile included in the first program, obtain second structure information of the fourth neural network from a second metafile included in the second program, and determine the third neural network or the fourth neural network as the common neural network by comparing the first structure information with the second structure information (Okuno, Paragraph [0086] teaches “the learning apparatus 10 may also cause the structure of the DCNN evaluated …”; Okuno, Paragraph [0088], teaches determining whether to adopt the shared layer candidate SLi for each of the recognition tasks with use of the shared layer candidate adoption/rejection determination unit.; Okuno, Paragraph [0089] teaches determining the multi-task DCNN structure based on the shared layer candidate adoption/rejection determination result with use of the sharing structure determination unit; Okuno, Paragraph [0137], teaches the shared layer candidate adoption/rejection determination unit compares the candidate multi-task DCNN accuracy and the allowable accuracy with respect to the recognition task.; Okuno, Paragraph [0048] teaches storing a predetermined number of sets (pairs) of image data and supervised data is the format of the learning data 30. In the present exemplary embodiment, assume that there is a plurality of recognition tasks, and GTs of the plurality of recognition tasks are associated with a single piece of image data. More specifically, this association can be prepared by expressing this association as a text file such as a table in which a filename of the image and the GTs of the plurality of recognition tasks are enumerated. [Note: Li, Col. 22, lines 45-48 teaches the modules implementing the functionality of certain implementations may be stored by file storage subsystem 1126 in the storage subsystem 1124, or in other machines accessible by the processor(s) 1114. - Li, Col. 22, lines 45-48 reading on metafiles])..
[EXAMINER NOTE: Li, Col. 1, lines 63-67 and Col. 2, lines 1-6, teaches Two neural network modules may be joined in a combined neural network model if they have compatible endpoints. Two compatible endpoints have the same data type, the same shape, and one includes output(s) while the other includes input(s). For example, an outputs endpoint of a first neural network module may be joined to an inputs endpoint of a second neural network module if both of those endpoints are of the same data type and of the same shape.]
Before the effective filing date of the claimed invention it would have been obvious to one of ordinary skill in the art to have modified the method and apparatus related to combining neural network modules based on version identifiers assigned to the neural network modules, as taught by Li, to further include the improved multi-task learning and shared layer determination techniques and apparatus, as taught by Okuno, in order to provide a learning apparatus that determines and considers the neural network structure itself to provide capability of simultaneously carrying out a plurality of recognition tasks on same image data under a single calculation environment (e.g., a personal computer (PC)), while reducing the memory capacity required when the plurality of recognition processes is performed. (Okuno, Paragraphs [0013], [0148] and [0149]).
Regarding claim 15, Li teaches all of the limitations of claim 3, however, Li does not distinctly disclose wherein the at least one processor is further configured to: store a first internal parameter used to process the first data in the third neural network of the first program as a common parameter in the at least one storage device, determine that the common parameter is different than a second internal parameter used to process the second data in the fourth neural network of the second program, determine difference information between the common parameter and the second internal parameter, and determine whether to store the difference information in the least one storage device, based on a result obtained by comparing a size of the difference information with a size of the second internal parameter.
Nevertheless, Okuno teaches wherein the at least one processor is further configured to: store a first internal parameter used to process the first data in the third neural network of the first program as a common parameter in the at least one storage device, determine that the common parameter is different than a second internal parameter used to process the second data in the fourth neural network of the second program, determine difference information between the common parameter and the second internal parameter (Okuno, [0113] further teaches in the next loop with i set to i=4, the IDs of the recognition tasks R1 and R2 are left in the layer sharing task list 35. For example, suppose that the recognition task R2 is determined not to adopt the sharing layer candidate SL4 in step S8 in the loop with i set to i=4 (the same also applies in a case where the recognition task R1, instead of the recognition task R2, is determined not to adopt the shared layer candidate SL4, or both the recognition tasks R1 and R2 are determined not to adopt the shared layer candidate SL4); Okuno [0114] teaches in this case, such a structure that the recognition tasks R1 and R2 do not share the shared layer candidate SL4 is generated as the final structure of the multi-task DCNN, and this DCNN is stored into the learned DCNN 33.; [Note: R1 and R2 being understood to read on first internal parameter and second internal parameter and SL4 is the common parameter, wherein the DCNN stored as a result is the difference information [i.e., the structure that the recognition tasks R1 and R2 do not share is stored as the final structure of the multi-task DCNN]]), and determine whether to store the difference information in the least one storage device, based on a result obtained by comparing a size of the difference information with a size of the second internal parameter (Okuno, Paragraph [0029] teaches as described in the description of the related art, the deep convolutional NN (DCNN) has the issues of making it difficult to thoroughly search for the optimum (desirable) network structure, requiring the large-capacity memory for holding a large number of learning parameters, and taking a long processing time in the recognition phase. One possible solution to mitigate or solve these issues is to reduce the number of learning parameters required when the recognition task is carried out.;Okuno, Paragraph [0031] further teaches employing the multi-task DCNN structure [i.e., common neural network with common parameters] leads to shared use of the learned parameter and a result of a calculation using this parameter between or among the plurality of recognition tasks at the layers close to the input, which reduces the memory capacity and the calculation time period required at the time of the recognition processing.; Okuno, Paragraph [0137] teaches the shared layer candidate adoption/rejection determination unit 24 compares the candidate multi-task DCNN accuracy 38 in the second memory 15 and the allowable accuracy 41 acquired in step S31 with respect to the recognition task having the task ID t.; Okuno, Paragraph [0148] further teaches the learning apparatus 10 can reduce the memory capacity required when the plurality of recognition processes is performed compared with the case where distinct DCNNs for the individual recognition tasks is held, and can also speed up the processing speed. Okuno, Paragraph [0152] further teaches the learning apparatus 10 can calculate the structure of the multi-task multi-layer NN that can achieve high-speed processing with a further small memory capacity while maintaining a performance similar to the performance when learning each of the tasks alone. In other words, the learning apparatus 10 learns so as to efficiently search for the desirable structure of the multi-task DCNN that carries out the plurality of recognition tasks simultaneously. In this manner, the learning apparatus 10 allows the recognition tasks to share the intermediate layer in the DCNN between or among them without reducing the accuracies of the recognition tasks, thereby succeeding in saving the memory usage and speeding up the processing speed at the time of the recognition processing.[Note: Okuno compares the memory capacity where distinct DCNNs for the individual recognition tasks is held versus where the tasks share the intermediate layer in the DCNN without reducing the accuracies of the recognition tasks, thereby succeeding in saving memory usage. This understanding of Okuno has been interpreted on reading on determining to store the difference information…based on a result obtained by comparing the size of the difference information with the size of the second internal parameter].).
Claims 6 and 7 are rejected under 35 U.S.C. 103 as being unpatentable over Li et al. in view of Okuno et al., as applied to claim 5, and in further view of Xiao et al. (U.S. Patent No. 10106153, published Oct. 23, 2018)
Regarding claim 6, the combination of Li in view of Okuno teaches all of the limitations of claim 5, however, the combination does not distinctly disclose wherein the first structure information comprises first types of first layers included in the third neural network and first connection relations between the first layers included in the third neural network, and wherein the second structure information comprises second types of second layers included in the fourth neural network and second connection relations between the second layers included in the fourth neural network.
Xiao teaches wherein the first structure information comprises first types of first layers included in the third neural network and first connection relations between the first layers included in the third neural network, and wherein the second structure information comprises second types of second layers included in the fourth neural network and second connection relations between the second layers included in the fourth neural network (Xiao, Figure 1 teaches Deep Learning Engine 112, comprising 1st, 2nd, 3rd, and 4th neural networks at 114, 116, 118, and 120.; Xiao, Col. 12, lines 20-34, teaches “the deep learning engine 112 can maintain, manage, store, update, tune, or configure the one or more neural networks 114, 116, 118, and 120. The deep learning engine 112 can use different parameters, weights, training sets, or configurations for each of the neural networks 114, 116, 118, and 120 to allow the neural networks to efficiently and accurately process a type of input and generate a type of output.”; Xiao, Col. 12, lines 35-48, teaches the first neural network 114 can be configured as or include a convolution neural network. The convolution neural network can include one or more convolution cells (or pooling layers) and kernels, that can each serve a different purpose; Xiao, Col. 14, lines 38-65, further teaches second neural network maintained by the deep learning engine 112 which can include one or more component or functionality of the first neural network 114, can be a same type of neural network as the first neural network 114, or can differ from the first neural network 114 in that the second neural network 116 can be tuned, or trained on different data sets, configured to receive different inputs, and configured to generate a different output.; Xiao, Col. 17, lines 1-20, teaches third neural network maintained by the deep learning engine 112, which can include one or more component or functionality of the first neural network 114 or second neural network 116, can be a same type of neural network as the first neural network 114 or second neural network 116, and can include a convolution neural network.; Xiao, Col. 18 teaches fourth neural network and corresponding structure and/or configuration.)
Before the effective filing date of the claimed invention it would have been obvious to one of ordinary skill in the art to have modified the method and apparatus related to combining neural network modules based on version identifiers assigned to the neural network modules, as taught by Li in view of Okuno, to further include the deep learning engine that can use different parameters, weights, training sets, or configurations for each of the neural networks, as taught by Xiao, in order to allow the neural networks to efficiently and accurately process a type of input and generate a type of output. (Xiao, Col. 12, lines 20-34).
Regarding claim 7, the combination of Li in view of Okuno teaches all of the limitations of claim 5, however the combination does not distinctly disclose wherein the first structure information comprises first sizes of first input data and first output data of first layers included in the third neural network, and wherein the second structure information comprises second sizes of second input data and second output data of the second layers included in the fourth neural network.
Nevertheless, Xiao teaches wherein the first structure information comprises first sizes of first input data and first output data of first layers included in the third neural network, and wherein the second structure information comprises second sizes of second input data and second output data of the second layers included in the fourth neural network (Xiao, Col. 6, lines 16-46, teaches each neural network can be tuned to process a specific type of input and generate a specific type of output with a higher level of accuracy and reliability as compared to a different neural network that is either at the baseline model or tuned or trained for a different objective or purpose...; Fig. 1, 112 teaches 1st neural network 114, 2nd neural network 116, 3rd neural network 118, and 4th neural network 120 and corresponding Col. 6, lines 19-46 teaches tuning can refer to or include training or processing of the neural network to allow the neural network to improve accuracy. Tuning the neural network can include, for example, designing the neural network using architectures for that have proven to be successful for the type of problem or objective desired for the neural network (e.g., first neural network 114, second neural network 116, third neural network 118 or fourth neural network 120). In some cases, the one or more neural networks may initiate at a same or similar baseline model, but during the tuning (or training or learning process), the result neural networks can be sufficiently different such that each neural network can be tuned to process a specific type of input and generate a specific type of output with a higher level of accuracy and reliability as compared to a different neural network that is either at the baseline model or tuned or trained for a different objective or purpose. Tuning the neural network can include setting different parameters for each network, fine-tuning parameters differently for each neural network, or assigning different weights (e.g., hyperparameters, or learning rates), tensor flows. Thus, by setting different parameters for each the neural networks based on a tuning or training process and the objective of the neural network, the data processing system can improve performance of the overall path generation process .[Note: Okuno teaches four neural networks wherein each are tuned with different parameters and specific types of input to generate a specific type of output, and each type of output has a higher level of accuracy and reliability as compared to a different neural network that is at the baseline model. This comparison between types of inputs to the different neural networks for obtaining a different types of output, wherein the comparison is against a baseline model has been understood to read on the limitation as claimed.]).
Motivation to combine same as stated in claim 6.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Li et al. in view of Okuno et al., as applied to claim 5, and further in view of Liu (US 20170330069 A1, filed May 11, 2016 and published Nov. 16, 2017)
Regarding claim 8, the combination of Li in view of Okuno teaches all of the limitations of claim 5, however, the combination does not distinctly disclose wherein the first structure information comprises first numbers of nodes per layer included in first fully connected layers of the third neural network, and wherein the second structure information comprises second numbers of nodes per layer included in second fully connected layers of the fourth neural network.
Nevertheless, Liu teaches wherein the first structure information comprises first numbers of nodes per layer included in first fully connected layers of the third neural network, and wherein the second structure information comprises second numbers of nodes per layer included in second fully connected layers of the fourth neural network (Liu, Paragraph [0040] teaches the controller 154 is designed as operating according to a configuration file adaptively determined based on the property of external data. Please refer to FIG. 5. The main difference between the artificial neural networks 300 and 400 is that the artificial neural network 400 further includes an input analyzer 156. The input analyzer 156 is used for analyzing external data to be processed by the artificial neural network 400 and accordingly determines a configuration for the artificial neural network 400. For example, the configuration determined by the input analyzer 156 can includes a total number of layers in the artificial neural network 400 and how many convolutional layers and/or fully-connected layers should be included in this network. As described above, as long as the controller 154 changes parameters provided from the storage device 152 to the reconfigurable artificial neurons, those neurons can then be reconfigured to work for a different layer.; Liu, Paragraph [0041] further teaches the artificial neural network 400 provides a more flexible solution that one or more of the following factors can be adjusted: the network structure, the number of layers, the number of artificial neurons in each layer, the connections between artificial neurons. Therefore, the artificial neural network 400 can be not bonded to a specific application. Aiming at external data with pure properties, the input analyzer 156 can determine a configuration with fewer layers for the artificial neural network 400, so as to save computation resources and prevent overfitting. On the contrary, aiming at external data with complicated properties, the input analyzer 156 can determine a configuration with more layers for the artificial neural network 400, so as to make the judgments more matching to the external data.; Liu, Abstract and Paragraph [0009], teaches the plurality of artificial neurons are used for performing computation based on plural parameters. The storage device is used for storing plural sets of parameters, each set of parameters being corresponding to a respective layer. At a first time instant, the controller controls the storage device to provide a set of parameters corresponding to a first layer to the plurality of artificial neurons so that the plurality of artificial neurons form at least part of the first layer. At a second time instant, the controller controls the storage device to provide a set of parameters corresponding to a second layer to the plurality of artificial neurons so that the plurality of artificial neurons format least part of the second layer.).
Before the effective filing date of the claimed invention it would have been obvious to one of ordinary skill in the art to have modified the method and apparatus related to combining neural network modules based on version identifiers assigned to the neural network modules, as taught by Li in view of Okuno, to further include the multi-layer artificial neural network method and controlling method, as taught by Liu. Compared with prior arts, artificial neural networks according to the invention obviously can utilize fewer artificial neurons while generate the same computation results. Thereby, the hardware cost is significantly reduced. (Liu, Paragraph [0032])
Claims 9 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Li et al. in view of Okuno et al., as applied to claim 5, and in further view of Abolmaesumi et al.
Regarding claim 9, the combination of Li in view of Okuno teaches all of the limitations of claim 5, however, the combination does not distinctly disclose wherein the first structure information comprises first convolution layers and first numbers of first filter kernels of the third neural network, first sizes of the first filter kernels of the third neural network, and first strides of the first convolution layers of the third neural network, and wherein the second structure information comprises second convolution layers and second numbers of second filter kernels of the fourth neural network, second sizes of the second filter kernels of the fourth neural network, and second strides of the second convolution layers of the fourth neural network.
Nevertheless, Abolmaesumi teaches wherein the first structure information comprises first convolution layers and first numbers of first filter kernels of the third neural network, first sizes of the first filter kernels of the third neural network, and first strides of the first convolution layers of the third neural network, and wherein the second structure information comprises second convolution layers and second numbers of second filter kernels of the fourth neural network, second sizes of the second filter kernels of the fourth neural network, and second strides of the second convolution layers of the fourth neural network (Abolmaesumi, Paragraph [0079] teaches Referring to FIG. 8, the neural network 360 includes 5 image quality assessment neural networks, each including the same shared layers 362 but including a different set of view category specific layers 370, 372, 374, 376, and 378. In various embodiments, the shared layers 362 and the view category specific layers 370, 372, 374, 376, and 378 may each be considered neural networks and it will be understood that a neural network may include more than one neural network within.; Abolmaesumi, Fig. 9 teaches Common neural network record comprising details of the layer number, layer type, stride, and number of kernels; Fig. 10 teaches an exemplary view category specific neural network record comprising the view category ID, layer number, layer type, stride and number of kernels; Abolmaesumi, Paragraph [0086] teaches a representation of a portion of an exemplary common neural network record for storing a set of parameters defining the shared layers 362 of the neural network 360 shown in FIG. 8, is shown at 320 in FIG. 9. Referring to FIG. 9, the common neural network record 320 includes first, second, third, fourth, fifth and sixth sets of fields 324, 326, 328, 330, 332, and 334 defining the parameters for the six layers of the shared layers 362 of the neural network 360 shown in FIG. 8. For ease of reference, not all kernel fields of the common neural network record 320 are shown in FIG. 9 and the content of the kernels is shown as [ . . . ], though it will be understood that there are 8 kernels in layer 1, 16 kernels in layer 3 and 32 kernels in layer 5 and that each kernel field stores a 3×3 matrix of values.; Abolmaesumi, Paragraph [0087] further teaches a representation of a portion of an exemplary view category specific neural network record for storing a set of parameters defining the set of view category specific layers 374 of the neural network 360 shown in FIG. 8, is shown at 340 in FIG. 10. Referring to FIG. 10, the view category specific neural network record 340 includes a view category identifier field 342 for storing a view category identifier identifying which view category the record is associated with and seventh, eighth, ninth, and tenth sets of fields 344, 346, 348, and 350 for storing parameters defining the set of view category specific layers 374 of the neural network 360 shown in FIG. 8. For ease of reference, not all kernel fields are shown in FIG. 10 and the content of the kernels and LSTM parameters are shown as [ . . . ]; Abolmaesumi, Paragraph [0089] teaches splitting the neural network 360 into a common portion and view category specific portions may facilitate more efficient training of the neural networks. In some embodiments, splitting the neural network 360 into a common portion and view category specific portions may result in requiring fewer learning parameters than would be required if using fully separate neural networks, which may help facilitate easier transferring of a neural network to a new machine, and/or may reduce memory usage.)
Before the effective filing date of the claimed invention it would have been obvious to one of ordinary skill in the art to have modified the method and apparatus related to combining neural network modules based on version identifiers assigned to the neural network modules, as taught by Li in view of Okuno, to further include the neural network processing by a common/shared neural network and view category specific neural networks, as taught by Abolmaesumi. Splitting the neural network 360 into a common portion and view category specific portions may facilitate more efficient training of the neural networks. In some embodiments, splitting the neural network 360 into a common portion and view category specific portions may result in requiring fewer learning parameters than would be required if using fully separate neural networks, which may help facilitate easier transferring of a neural network to a new machine, and/or may reduce memory usage. (Abolmaesumi, Paragraph [0089])
Regarding claim 10, the combination of Li in view of Okuno teaches all of the limitations of claim 5, however the combination does not distinctly disclose wherein the first structure information comprises first pooling layers and first sizes of first filter kernels of the third neural network and first strides of the first pooling layers of the third neural network, and wherein the second structure information comprises second pooling layers and second sizes of second filter kernels of the fourth neural network and second strides of the second pooling layers of the fourth neural network (Abolmaesumi, Figures 8, 9, and 10, and corresponding Paragraphs [0085]-[0088] teach neural network structure information relating to a common neural network and view specific neural networks, including kernel size(s), layer numbers, and stride(s); Abolmaesumi, Paragraph [0066] teaches max-pooling layers; Abolmaesumi Paragraph [0170] further teaches in some embodiments, the neural network 950 may include one convolutional layer with 12 kernels of each 11×11 pixels, one pooling layer with a kernel of 3×3 and stride of 2, one convolutional layer with 24 kernels of each 7×7 pixels, one pooling layer with a kernel of 3×3 and stride of 2, one convolutional layer with 48 kernels of each 3×3 pixels, one pooling layer with a kernel of 3×3 and stride of 2, a fully connected layer with 2048 outputs, a fully connected layer with 1024 outputs, and a fully connected layer with 5 outputs).
Motivation to combine same as stated in claim 9.
Claims 11, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Li et al. in view of Toba et al. (U.S. Patent No. US 11341398 B2, filed Mar. 8, 2017 and published May 24, 2022- Toba further claiming the benefit of foreign application filed Oct. 3, 2016)
Regarding claim 11, Li teaches all of the limitations of claim 1, and Li further teaches wherein the at least one processor is further configured to: store a first internal parameter used to process the first data in the third neural network of the first program as a common parameter in the at least one storage device, determine that the common parameter is different than a second internal parameter used to process the second data in the fourth neural network of the second program (Li, Col. 1, lines 36-49 teaches a combined neural network model joins a plurality of individual neural network modules to perform a particular task. Generally, each of the neural network modules is utilized in performing a “subtask” in performing the overall task of the combined neural network model. For example, a combined model may be utilized to perform the overall task of determining a classification of an object in a raw image by utilizing: a first module that can be utilized to perform a first subtask of extracting image features from the raw image [i.e., additionally and/or alternatively teaching process first data]; and a second module that can be utilized to perform a second subtask of determining an object class based on the image features extracted from the raw image by the first neural network module [i.e., additionally and/or alternatively teaching process second data].; Col. lines 43-46 further teaches in some implementations, the method further includes: receiving a request to combine the first neural network module and the second neural network module and in response to the request identifying the refined version of the first module based on the version identifier of the first neural network module identifying the second neural network module or the refined version of the second neural network module; Col. 4, lines 57-60 further teaches in some implementations, the method further includes: identifying an additional combined neural network model that combines at least the first neural network module and a third neural network module.; Li, [claim 1] further teaches wherein generating the combined neural network model comprises joining the first endpoint of the first neural network module and the second endpoint of the second neural network module; identify a first version identifier based on a previously stored mapping, in the one or more computer readable media, of the first version identifier to at least the first endpoint of the first neural network module, wherein the previously stored mapping comprises data defining an association, and wherein the first version identifier reflects that the first neural network module has been trained [i.e., process first data]; identify a second version identifier based on an additional previously stored mapping, in the one or more computer readable media, of the second version identifier to at least the second endpoint of the second neural network module, wherein the additional previously stored mapping comprises data defining an additional association, and wherein the second version identifier reflects that the second neural network module has been trained [i.e., process second data];), …
However, Li does not distinctly teach …determine difference information between the common parameter and the second internal parameter, and store the difference information in the at least one storage device.
Nevertheless, Toba teaches …determine difference information between the common parameter and the second internal parameter, and store the difference information in the at least one storage device (Toba Col. 5, lines 61-67 teaches Fig. 3 is a diagram illustrating an example of the structures of the neural networks 12b stored in the reconfiguration data memory 15. The reference symbols “00”, “01”, … “N” in Fig. 3 denote identifiers for identifying each structure of the neural network 12b. The identifiers are transmitted from the host system together with the structures of the neural network 12b.; Col. 6, lines 1-14 teaches the hatched portions in FIG. 3 represent portions that are different from the structure of the neural network 12a [Note: hatched portion of the NN that are different reading on an “difference information”, as claimed] . In each layer, a part (i.e., the hatched portion) of the structures of the neural network 12b is different from that of the neural network 12a. From the host system, (N+1) different structures (i.e., information on the structures) of the neural network 12b are transmitted, and stored in the reconfiguration data memory 15. It is not necessary for all of the structures of the neural network 12b to be stored in the reconfiguration data memory 15. For example, only the parts that are different from those of the structure of the neural network 12a (i.e., only the hatched portions) may be stored in the reconfiguration data memory 15.).
Before the effective filing date of the claimed invention it would have been obvious to one of ordinary skill in the art to have modified the method and apparatus related to combining neural network modules based on version identifiers assigned to the neural network modules, as taught by Li, to further include the features of the recognition apparatus and learning system using neural networks, as taught by Toba, in order to enable the learning apparatus to learn the image data having a feature, and to generate a neural network structure that can handle new automobile designs and new dangers [i.e., usage environments] that my change day by day. (Toba, Col. 4,, lines 16-25)
Regarding claim 17, Li teaches all of the limitations of claim 16, however, Li does not distinctly disclose wherein the at least one processor is further configured to: determine that the fifth neural network and the common neural network structurally correspond to each other, determine a common parameter and an internal parameter obtained based on the update data, and store difference information between the common parameter and the internal parameter in the at least one storage device.
Nevertheless, Toba teaches wherein the at least one processor is further configured to: determine that the fifth neural network and the common neural network structurally correspond to each other, determine a common parameter and an internal parameter obtained based on the update data, and store difference information between the common parameter and the internal parameter in the at least one storage device (Toba, Col. 5, lines 61-67 teaches Fig. 3 is a diagram illustrating an example of the structures of the neural networks 12b stored in the reconfiguration data memory 15. The reference symbols “00”, “01”, … “N” in Fig. 3 denote identifiers for identifying each structure of the neural network 12b. The identifiers are transmitted from the host system together with the structures of the neural network 12b.; Col. 6, lines 1-14 teaches the hatched portions in FIG. 3 represent portions that are different from the structure of the neural network 12a [Note: hatched portion of the NN that are different understood as the “difference information”] . In each layer, a part (i.e., the hatched portion) of the structures of the neural network 12b is different from that of the neural network 12a. From the host system, (N+1) different structures (i.e., information on the structures) of the neural network 12b are transmitted, and stored in the reconfiguration data memory 15. It is not necessary for all of the structures of the neural network 12b to be stored in the reconfiguration data memory 15. For example, only the parts that are different from those of the structure of the neural network 12a (i.e., only the hatched portions) may be stored in the reconfiguration data memory 15.; Toba, Col. 4, lines 10-25 teaches the learning apparatus is able to learn the image data having a feature, and to generate a neural network structure that can handle new automobile designs and new dangers [i.e., usage environments] that my change day by day [i.e., as in “based on the update data”, as claimed])
Before the effective filing date of the claimed invention it would have been obvious to one of ordinary skill in the art to have modified the method and apparatus related to combining neural network modules based on version identifiers assigned to the neural network modules, as taught by Li, to further include the features of the recognition apparatus and learning system using neural networks, as taught by Toba, in order to enable the learning apparatus to learn the image data having a feature, and to generate a neural network structure that can handle new automobile designs and new dangers [i.e., usage environments] that my change day by day. (Toba, Col. 4,, lines 16-25)
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Li et al. in view of Toba et al., as applied to claim 11, and further in view of Monirul et al., “A New Adaptive Merging and Growing Algorithm for Designing Artificial Neural Networks” (June, 2009)
Regarding claim 12, the combination of Li in view of Toba teaches all of the limitations of claim 11, however, the combination does not distinctly disclose wherein the at least one processor is further configured to: restore the second internal parameter, and determine at least one of a residual parameter added to the common parameter, a transformation parameter multiplied by the common parameter, or a selective parameter for selecting a part of the common parameter as the difference information.
Nevertheless, Monirul teaches wherein the at least one processor is further configured to: restore the second internal parameter, and determine at least one of a residual parameter added to the common parameter, a transformation parameter multiplied by the common parameter, or a selective parameter for selecting a part of the common parameter as the difference information (Monirul, pg. 710, “step 9” teaches merge each S-labeled hidden neuron with its most correlated unlabeled counterpart…Thus, AMGA produces one new hidden neural [i.e., a residual parameter] by merging the S-labeled hidden neuron with its unlabeled counterpart.; Monirul, “step 10” further teaches retrain the modified ANN (i.e., merged and/or combined ANN), which is obtained after merging hidden neurons, until its previous error level has been reached. If the modified ANN [i.e., merged ANN] is able to reach its previous error level, continue. Otherwise, restore the unmodified ANN [i.e., restore the second internal parameter]).
Before the effective filing date of the claimed invention it would have been obvious to one of ordinary skill in the art to have modified the method and apparatus related to combining neural network modules based on version identifiers assigned to the neural network modules, as taught by Li in view of Toba, to further include the adaptive merging and growing algorithm for designing artificial neural networks, as taught by Monirul, in order to obtain compact ANN architectures with good generalization ability compared to other algorithms. (Monirul, Abstract and pg. 719, Section V )
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Toba, as applied in claim 11, and further in view of Baidyk et al., “Neural Network with Ensembles”, IEEE (2010)
Regarding claim 13, the combination of Li in view of Toba teaches all of the limitations of claim 11, and the combination further teaches wherein the at least one processor is further configured to: determine a plurality of pieces of difference information between the common parameter and the second internal parameter according to a plurality of calculation methods, and store a … difference … from among the plurality of pieces of difference information in the at least one storage device (Toba Col. 5, lines 61-67 teaches Fig. 3 is a diagram illustrating an example of the structures of the neural networks 12b stored in the reconfiguration data memory 15. The reference symbols “00”, “01”, … “N” in Fig. 3 denote identifiers for identifying each structure of the neural network 12b. The identifiers are transmitted from the host system together with the structures of the neural network 12b.; Col. 6, lines 1-14 teaches the hatched portions in FIG. 3 represent portions that are different from the structure of the neural network 12a [Note: hatched portion of the NN that are different reading on an “difference information”, as claimed] . In each layer, a part (i.e., the hatched portion) of the structures of the neural network 12b is different from that of the neural network 12a. From the host system, (N+1) different structures (i.e., information on the structures) of the neural network 12b are transmitted, and stored in the reconfiguration data memory 15. It is not necessary for all of the structures of the neural network 12b to be stored in the reconfiguration data memory 15. For example, only the parts that are different from those of the structure of the neural network 12a (i.e., only the hatched portions) may be stored in the reconfiguration data memory 15. [Note: Figs. 3 and 11 in Toba understood to read on store a difference information from among the plurality of pieces of difference information).
However, the combination does not distinctly disclose …store a smallest difference information having a smallest data size…
Nevertheless, Baidyk teaches …store a smallest difference information having a smallest data size… (Baidyk, pg. 3, col. 1 paragraphs 1 and 2, teach The neural ensemble is the basic information element of all hierarchical levels of the neural network. It is formed from the elements of lower hierarchical levels and can correspond to the feature, to the description of an object, to the description of a situation, to the relation between the objects, and so forth…[T]he neural ensemble corresponding to the car can be formed at the higher hierarchical level. The neurons that describe car’s speed and acceleration at the lower level will enter into the ensemble that corresponds to the car at the higher level. To ensure that the size of the ensemble at the higher level is not too large only part of the neurons from the lower level ensembles falls into the higher level ensemble. For example, during the construction of the ensemble that corresponds to a car, only parts of each of the ensembles describing speed, acceleration, type of the car and so on are included in the ensemble “car”. We term the procedure for selecting a part of the neurons to be transferred to the higher level as normalization of the neural ensemble. The ensemble is formed in such a way that, using the neurons that entered into the ensemble at the higher level, it would be possible to restore the ensembles of the lower level due to the associative restoration of the entire ensemble from its part. This type of organization of the associative-projective neural networks makes it possible to form the hierarchy of “part – whole.”)
Before the effective filing date of the claimed invention it would have been obvious to one of ordinary skill in the art to have modified the method and apparatus related to combining neural network modules based on version identifiers assigned to the neural network modules, as taught by Li in view of Toba, to further the neural network ensembles method, as taught by Baidyk, in order to make it possible to restore the ensembles of the lower level dure to the associative restoration of the entire ensemble from its part, taking into consideration the demands of memory storage for recognition of known situations. (Baidyk, pg. 3, col. 1 paragraphs 1 and 2 and Section IV).
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Li et al. in view of Toba et al., as applied to claim 11, and further in view of Okuno et al.
Regarding claim 14, the combination of Li in view of Toba teaches all of the limitations of claim 11, however, the combination does not distinctly disclose wherein the at least one processor is further configured to: when the second program is executed, restore the second internal parameter from the common parameter based on the difference information, and load the second internal parameter.
Nevertheless, Okuno teaches wherein the at least one processor is further configured to: when the second program is executed, restore the second internal parameter from the common parameter based on the difference information, and load the second internal parameter (Okuno, Paragraphs [0082] and [0092] teach relearning only an 1+1-th layer and higher layers without updating the interchanged shared layer candidate SLi and lower layers. [i.e., relearning understood to read on “restoring”]; Okuno, Paragraph [0093] further teaches the relearning can be carried out by setting the parameter of the DCNN in the stored in the learned DCNN 33 after the processing exits the loop described in Okuno, paragraph [0091]; Okuno, Paragraph [0095] teaches interchangeable lower layer portions of the DCNN which has individually learned plurality of recognition tasks; Okuno, Paragraphs [0106]-[0107], teach supposing a recognition task is determined to share the layer with other recognition tasks as far as a determined shared layer candidate, but uses CL.sub.4k (k.gtoreq.2) in the learned DCNN 33 stored in the previous loop for the second layer or a higher layer (CL.sub.4k, k.gtoreq.2) [Note: using CL.sub.4k (k.gtoreq.2) stored in the previous loop for the second layer understood to read on “loading the second internal parameter”]; Okuno, [0113] further teaches in the next loop with i set to i=4, the IDs of the recognition tasks R1 and R2 are left in the layer sharing task list 35. For example, suppose that the recognition task R2 is determined not to adopt the sharing layer candidate SL4 in step S8 in the loop with i set to i=4 (the same also applies in a case where the recognition task R1, instead of the recognition task R2, is determined not to adopt the shared layer candidate SL4, or both the recognition tasks R1 and R2 are determined not to adopt the shared layer candidate SL4); Okuno [0114] teaches in this case, such a structure that the recognition tasks R1 and R2 do not share the shared layer candidate SL4 is generated as the final structure of the multi-task DCNN, and this DCNN is stored into the learned DCNN 33. ).
Before the effective filing date of the claimed invention it would have been obvious to one of ordinary skill in the art to have modified the method and apparatus related to combining neural network modules based on version identifiers assigned to the neural network modules, as taught by Li in view of Toba, to further include the improved multi-task learning and shared layer determination techniques and apparatus, as taught by Okuno, in order to provide a learning apparatus that determines and considers the neural network structure itself to provide capability of simultaneously carrying out a plurality of recognition tasks on same image data under a single calculation environment (e.g., a personal computer (PC)), while reducing the memory capacity required when the plurality of recognition processes is performed. (Okuno, Paragraphs [0013], [0148] and [0149]).
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Li et al. in view of Toba et al., as applied to claim 17, and further in view of Arbel et al. (US 20090326886 A1, filed Jun. 30, 2008 and published Dec. 31, 2009)
Regarding claim 18, the combination of Li in view of Toba teaches all of the limitations of claim 17, and the combination further teaches wherein the at least one processor is further configured to: determine that the fifth neural network and the common neural network structurally correspond to each other (Li, Col. 3, lines 8-22, teaches determining whether a first version identifier assigned to an endpoint of the first neural network module matches a second version identifier assigned to the second endpoint of the second neural network module. The method further includes, in response to determining that the first version identifier does match the second version identifier, using the combined neural network model.; Li, Col. 3 lines 34-42 further teaches in some implementations, the method further includes: in response to training at least the first neural network module to generate the refined version [i.e., the refined version understood as the fifth neural network]: replacing the first version identifier with a new version identifier. In some versions of those implementations, the new version identifier matches the second version identifier when training at least the first neural network module to generate the refined version occurs without any training of the second neural network module. [Note: Because there is a match between the refined version of the first model and the second version identifier of the second neural network model it is understood from Li, Col. 3, lines 8-22 that the combined neural network will be used]),
However the combination does not distinctly disclose: and not store the fifth neural network as the neural network of the first program in the at least one storage device.
Nevertheless, Arbel et al. teaches and not store the fifth neural network as the neural network of the first program in the at least one storage device (Arbel, Paragraph [0022] teaches the term "Model-Reduction Algorithm", "MRA", and "model reducer" as used herein include, for example, an algorithm or module able to reduce a size of a model, able to reduce complexity of a model, able to reduce the number of elements of a model, and/or able to reduce the state space corresponding to a model [i.e., not store the model]; an algorithm or module able to find and/or eliminate redundant elements in a model; an algorithm or module able to find and/or eliminate invariants in a model; or other suitable algorithms or modules…In some embodiments, for example, a MRA may eliminate duplicate logic, e.g., using "BDD sweeping" algorithms in order to identify two or more pieces of logic having identical functionality; only one copy is then maintained, and is connected to other places to which the other copies were previously connected. [Note: Arbel [0022] has been understood to teach not storing duplicate models reading on not store the fifth neural network as the network of the first program in response to determining that the fifth neural network is the same as the common neural network]).
Before the effective filing date of the claimed invention it would have been obvious to one of ordinary skill in the art to have modified the method and apparatus related to combining neural network modules based on version identifiers assigned to the neural network modules, as taught by Li in view of Okuno, to further include the verification of models using concurrent model-reduction and model-checking, as taught by Arbel, in order to simplify the model being checked and thus improve performance. (Arbel, [0040])
Conclusion
The following prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Brown et al. (US 20040122785 A1) disclosing a compiler to map application program code to object code capable of being executed on an operating system platform. A first neural network module is trained to generate characteristic output based on input information describing attributes of the application program. A second neural network module is rained to receive as input the application program code and the characteristic output and, in response, generate object code. The first and second neural network modules are used to convert the application program code to object code. [0038] FIG. 5 illustrates further detail of the components of the Application Characterizer Neural Network (ACNN) 10 to generate the characteristic output vector 40, which includes a separate neural network for each possible characterization type of the Applet bytecodes 8.
Szyperski et al. (US 20130066925 A1) teaching accessing different application data via a common data structure. A common data type structure can be used to correlate access requests between applications that implement data in accordance with different types or type structures. In one implementation, a common data structure includes schemes for operations, sequences, records, and atoms (i.e., undefined). The system can then map any type structure to the schemes of the common data structure. In operation, a request for data by an application can involve identifying one or more proxies used by an application to map the data to the common data structure. The proxies map the data to the common data structure based on the shape of the data (to the extent it can be identified). The proxies then can return one or more data structures that comprise the identified mapping information. The application can then perform operations directly on the received data structures.
Bazrafkan et al., “Merging deep networks to accelerate edge artificial intelligence in consumer electronics devices and systems”, IEEE (2018) – See Fig. 1 teaching concatenation of the last layer into a single layer (a convolution case). And further teaches (in pg. 59 – Merging the Networks) the main purpose of the semi-parallel DNN (SPDNN) model is to mix and merge several deep architectures and produce a final model that takes advantage of the specialized layers of each architecture but is significantly smaller than the combined sizes of these networks. The approach we adopted is based on graph optimization.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BEATRIZ RAMIREZ BRAVO whose telephone number is 571-272-2156. The examiner can normally be reached Mon. - Fri. 7:30a.m.-5:00p.m..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, USMAAN SAEED can be reached at 571-272-4046. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/B.R.B./Examiner, Art Unit 2146
/USMAAN SAEED/Supervisory Patent Examiner, Art Unit 2146