DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This Office Action is in response to Applicant Amendment and Arguments filed on 20 July, 2026.
Claims 1-4, 6, 9-13, 15-18 and 20 are pending for examination. Claims 5, 7-8, 14 and 19 were cancelled.
Claim Rejections - 35 USC § 103
The following is a quotation of pre-AIA 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 6, 9-13, 15-18 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over HWANG et al. (US Pub. 2020/0196024 A1) in view of Ferreira Moreno et al. (US Pub. 2019/0370634 A1; hereafter Moreno) and further in view of SHI (US Pub. 2020/0412521 A1) and Ling et al. (US Pub. 2017/0103298 A1).
HWANG, Moreno and SHI were cited in the previous Office Action.
As per claim 1, HWANG teaches the invention substantially as claimed including A neural network model scheduling method (HWANG, Fig. 6, 140, 606, 230 and 240 rendering engine model; [0007] lines 14-15, execute the neural network to process the media data and generate the particular media data output), comprising:
storing at least one pre-trained neural network models in a memory (HWANG, Fig. 4, 230 and 410; Fig. 6, 140, 606, 230 and 240 rendering engine model; [0091] lines 1-4, The neural network 410 can be pre-trained to process the features from the data in the input layer 402 using the different hidden layers 404 in order to provide the output through the output layer 406; [0111] lines 1-11, the remote system 140 can have the infrastructure and capabilities, such as storage and/or compute capabilities, to compute and/or maintain a large number of rendering engine models. For example, the remote system 140 can be a server or cloud service containing a large repository of rendering engine models and having the compute capabilities to generate and store rendering engine models having increased complexity, customization, size, etc. The remote system 140 can also train rendering engine models and make the trained rendering engine models available; [0112] lines 3-7, the address 604 to rendering engine model 606 on the remote system 140. The address 604 can include, for example, a uniform resource identifier (URI), a link, a path, a resource locator, or any other address such as a network, storage; also see [0119] The rendering engine models can be pre-configured by the manufacturer, and can describe rendering engines (e.g., neural networks) which have been trained and tuned as described in flow 700; [0142] The storage device may include any of a variety of distributed or locally accessed data storage media such as…flash memory, volatile or non-volatile memory), and acquiring a base address of the pre-trained neural network models (HWANG, Fig. 6, see arrow from 140 rendering engine model to be collected by 110 compute engine; [0061] lines 1-10, the compute engine 110 can receive media data (e.g., image data, video data, audio data, etc.) captured by any of the data capture devices 102, 104, 152, 154 in the computing environment 100, and receive, generate, and/or retrieve from storage one or more rendering engine models for the media data. The compute engine 110 can embed the one or more rendering engine models in a media item containing the media data, and/or embed an address (e.g., a uniform resource identifier (URI); a link; a path; a network, storage or destination address; a resource locator; etc.) to one or more rendering engine models in a media item containing the media data; also see Fig. 6, 604 address for rendering engine model (as acquiring a base address of the pre-trained neural network model and embedded into media item); [0067] lines 1-4, The compute engine 110 can generate or obtain one or more rendering engine models 230, 240 configured to process and render the media data 210; [0091] lines 1-4, The neural network 410 can be pre-trained to process the features from the data in the input layer 402 using the different hidden layers 404 in order to provide the output through the output layer 406),
acquiring base addresses of the corresponding neural network models selected from the plurality of pre-trained neural network models according to a task type, the corresponding neural network models stored at the base addresses (HWANG, [0116] lines 1-9, address (e.g., 604) can include such information (e.g., description information) even if it is the only address in the media item 602. The media processing system 102 (or associated user) can use this information to determine whether the rendering engine model associated with the address is suitable for the particular instance or comports with the desired processing or rendering outcome; [0112] lines 1-12, the media item 602 includes the media data 210 and the address 604 to rendering engine model 606 on the remote system 140. The address 604 can include, for example, a uniform resource identifier (URI), a link, a path, a resource locator, or any other address such as a network, storage or destination address. The address 604 indicates where the rendering engine model 606 is located and/or how it can be retrieved. When the media processing system 102 processes the media item 602 (e.g., as described in flow 300 illustrated in FIG. 3A), it can use the address 604 to retrieve the from the remote system 140 (as to acquiring base addresses of corresponding rendering engine model 606 neural network models according to a task type); also see [0131] the media processing system 102 can include multiple rendering engine models (e.g., 230, 240) and/or addresses (e.g., 604) in the media item. For example, the media processing system 102 can embed an additional rendering engine model (e.g., 240) in the media item (e.g., 220). The additional rendering engine model can include an additional description of an additional neural network configured to process the media data (e.g., 210) and generate a different media data output. The additional description can define a different neural network architecture for the additional neural network. The different neural network architecture can be customized for a different operational outcome based on different neural network layers, filters, activation functions, parameters, etc; [0136] a address (e.g., 604) to a remote rendering engine model or a remote location of the remote rendering engine model. The remote rendering engine model can include a respective description of a neural network configured to process the media data and generate a respective media data output. The media item with the address can be sent to a recipient, which can use the address to retrieve the remote rendering engine model from the remote location and, based on the respective description in the remote rendering engine model, generate the neural network associated with the remote rendering engine model and process the media data in the media item using the neural network to generate the respective media data output (e.g., the rendering of the media data).); and
invoking, on the basis of the base addresses of the corresponding neural network models, the corresponding neural network models; computing the data to obtain a computation result, and outputting the computation result. (HWANG, Fig. 7, 410 neural network, 704 output, [0007] lines 14-15, execute the neural network to process the media data and generate the particular media data output; [0112] lines 1-12, The address 604 can include, for example, a uniform resource identifier (URI), a link, a path, a resource locator, or any other address such as a network, storage or destination address. The address 604 indicates where the rendering engine model 606 is located and/or how it can be retrieved. When the media processing system 102 processes the media item 602 (e.g., as described in flow 300 illustrated in FIG. 3A), it can use the address 604 to retrieve the rendering engine model 606 from the remote system 140; [0109] lines 3-12, The rendering engine model 606 can be a particular rendering engine model capable of rendering the media data 210. In this example, the rendering engine model 606 is stored in the remote system 140. The remote system 140 can store one or more rendering engine models 606, 230, 240 for the media data 210 (and other media data), which can be access by the media processing system 102 (and any other device) to process and render media data).
HWANG fails to specifically teach when storing, it is loading at least one pre-trained neural network models to a model storage area in a memory, wherein the memory further comprises a common data storage area, reading data in the common data storage area and when computing, it is computing the data read in the common data storage area to obtain a computation result.
However, Moreno teaches loading a at least one pre-trained neural network models to a model storage area in a memory (Moreno, [0048] lines 8-9, The trained models may be loaded on the memory 304; Fig. 3, 304 memory, machine learning models (as model storage area), wherein the memory further comprises a common data storage area, reading data in the common data storage area (Moreno, Fig. 3, 304 memory, data units (as common data storage area in the memory); [0027] lines 1-4, A data unit 202 describes an input data embedded in a smart contract. There may be a plurality of data units 202. A data unit 202 may be received from a data provider 204 and stored in a memory device or storage device; [0029] lines 1-12, Machine learning algorithms 206 are used to generate machine learning models 208. There may be new or existing machine learning models. A machine learning algorithm, for example, trains a machine learning model, for example, based on a data set contained or referenced in a data unit 202. A machine learning model trained by running a machine learning algorithm or technique may include parameters or weights which are generated during the training phase. A machine learning model takes data units as input (e.g., input feature vectors) in order to produce an outcome such as a prediction, classification, and/or another outcome) and
when computing, it is computing the data read in the common data storage area to obtain a computation result (Moreno, [0029] lines 1-12, Machine learning algorithms 206 are used to generate machine learning models 208. There may be new or existing machine learning models. A machine learning algorithm, for example, trains a machine learning model, for example, based on a data set contained or referenced in a data unit 202. A machine learning model trained by running a machine learning algorithm or technique may include parameters or weights which are generated during the training phase. A machine learning model takes data units as input (e.g., input feature vectors) in order to produce an outcome such as a prediction, classification, and/or another outcome; [0038] lines 1-7, data unit's data blocks may be used as a training set to train a model of a given machine learning algorithm. Multiple models may be trained based on different data unit's data blocks. The trained models may be run with input data and the outcomes or output data produced by the trained models may be verified against the ground-truth data (or what the actual outcome associated with the input data should be). In this way, the data platform 210 may infer which data units provide the best value for a given service. For example, a selection of data units relating to coarse-resolution satellite scenes may have been selected to train a model used by a service that computes statistics on deforestation).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of HWANG with Moreno because Moreno’s teaching of utilizing the data that is stored in the memory to train the machine learning model in the memory to produce the output would have provided HWANG’s system with the advantage and capability to allow the system to build an accurate machine learning model in order to improving the system performance and efficiency (see Moreno, [0003] “building of an accurate machine learning model, that data set should be recognized as needing improvement”).
HWANG and Moreno fail to specifically teach loading the corresponding neural network models into a processor.
However, SHI teaches loading the corresponding neural network models into a processor (SHI, [0022] lines 25-28, the processor can be able to efficiently implement custom data processing applications by loading pre-trained neural network models into the AI/neuromorphic core 314).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of HWANG and Moreno with SHI because SHI’s teaching of loading the pre-trained neural network models into core/processer would have provided HWANG and Moreno’s system with the advantage and capability to allow the system to be able to efficiently implement custom data processing applications in order to improving the system performance and efficiency (see SHI, [0022] “efficiently implement custom data processing applications”).
HWANG, Moreno and SHI fail to specifically teach configuring corresponding hardware resources according to network structures of the corresponding neural network models; and when computing the data read in the common data storage area, it is based on the corresponding hardware resources to obtain a computation result, and outputting the computation result, wherein the hardware resources comprise one or more of a computing unit, a storage unit and a pipeline acceleration unit.
However, Ling teaches configuring corresponding hardware resources according to network structures of the corresponding neural network models (Ling, [0025] CNN accelerator is configured on the target. According to an embodiment of the present invention, the target is programmed using the compiled design of the CNN accelerator. The target is further configured to implement a variation of the CNN accelerator to support execution of a desired CNN algorithm. During runtime, further configurations may be applied to implement other variations of the CNN accelerator to support execution of other CNN algorithms; [0026] identifying the characteristics for the CNN accelerator may include identifying characteristics of one or more CNN algorithms that the CNN accelerator is desired to support. The characteristics of a CNN algorithm may include a number and sequence of stages of layers, such as convolution and noise filtering layers. The noise filtering layers may include, for example, pooling, normalization, and rectified linear (ReLU) layers. The characteristics of the CNN algorithm may also include sizes and coefficients of filters, and sizes, strides, and padding of images to be processed; [0029] generating a design for a CNN accelerator according to an embodiment of the present invention. The procedures described in FIG. 3 may be used to implement procedure 102 in FIG. 1 and may be performed by an EDA tool executed on a computer system. At 301, buffers on the CNN accelerator are assigned an appropriate size to support a size of images to be processed by the CNN accelerator. According to an embodiment of the present invention, the design for the CNN accelerator architecture is structured such that there is one read port and one write port for each buffer); and
when computing the data read in the common data storage area, it is based on the corresponding hardware resources to obtain a computation result, and outputting the computation result (Ling, [0030] computation units are generated to support the CNN algorithms to be performed by the CNN accelerator. According to an embodiment of the present invention, the computation units include kernels that perform convolution and noise filtering. The computation units may be generated to maximum performance utilizing resources available on a target implementing the CNN accelerator. The resources available on the target that may be utilized may include DSP blocks, memory blocks, shift registers, and adders. In one embodiment, dot product computations performed by kernels are organized to leverage vector modes supported by reconfigurable DSP blocks on the target; [0049] FIG. 7 is a block diagram of an exemplary CNN accelerator 700 that may be used to implement a CNN according to an embodiment of the present invention. The CNN accelerator 700 allows for efficient computation of forward propagation of convolution and other layers. The CNN accelerator 700 accepts an input image and may apply multiple convolution and other layers in succession; [0050] Input image pixels are transmitted into the processing element (PE) arrays 701-704 which may perform independent dot-product operations in a convolution procedure; also see [0051] – [0054] for computation result),
wherein the hardware resources comprise one or more of a computing unit, a storage unit and a pipeline acceleration unit (Ling, [0028] resources available on a target to implement the CNN accelerator are identified. According to an embodiment of the present invention the target may include one or more target devices of one or more target device types. The resources identified may include a number and type of memory blocks, digital signal processors (DSPs), and other components and processing units on a target device).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of HWANG, Moreno and SHI with Ling because Ling’s teaching of configuring the CNN accelerator based on identified neural network models (i.e., CNN algorithms including layers) would have provided HWANG, Moreno and SHI’s system with the advantage and capability to allow the system to improving the accuracy of complex recognition tasks such as large-category image classification, automatic speech recognition, as well as other data classification/recognition tasks (see Ling, [0003]).
As per claim 2, HWANG, Moreno, SHI and Ling teach the invention according to claim 1 above. Moreno further teaches wherein the model storage area is configured to store a network structure of the pre-trained neural network models and parameters of the pre-trained neural network models (Moreno, Fig. 2, 208; Fig. 3, 304 memory, machine learning models; [0034] lines 8-9, a model having architecture or structure associated with the machine learning algorithm; [0025] lines 1-3, the generated models may operate with different parameters, and thus may produce different outcomes; [0048] lines 8-9, The trained models may be loaded on the memory 304; (as including the network structure and parameters of the neural network model)).
As per claim 3, HWANG, Moreno, SHI and Ling teach the invention according to claim 1 above. HWANG further teaches wherein the base address is an initial storage address of a neural network model in the memory (HWANG, [0111] lines 1-11, the remote system 140 can have the infrastructure and capabilities, such as storage and/or compute capabilities, to compute and/or maintain a large number of rendering engine models. For example, the remote system 140 can be a server or cloud service containing a large repository of rendering engine models and having the compute capabilities to generate and store rendering engine models having increased complexity, customization, size, etc. The remote system 140 can also train rendering engine models and make the trained rendering engine models available; [0112] lines 3-7, the address 604 to rendering engine model 606 on the remote system 140. The address 604 can include, for example, a uniform resource identifier (URI), a link, a path, a resource locator, or any other address such as a network, storage (as including initial storage address); [0142] The storage device may include any of a variety of distributed or locally accessed data storage media such as…flash memory, volatile or non-volatile memory).
As per claim 4, HWANG, Moreno, SHI and Ling teach the invention according to claim 3 above. Ling teaches computing the data based on the corresponding hardware resources (Ling, [0030] computation units are generated to support the CNN algorithms to be performed by the CNN accelerator. According to an embodiment of the present invention, the computation units include kernels that perform convolution and noise filtering. The computation units may be generated to maximum performance utilizing resources available on a target implementing the CNN accelerator. The resources available on the target that may be utilized may include DSP blocks, memory blocks, shift registers, and adders. In one embodiment, dot product computations performed by kernels are organized to leverage vector modes supported by reconfigurable DSP blocks on the target; [0049] FIG. 7 is a block diagram of an exemplary CNN accelerator 700 that may be used to implement a CNN according to an embodiment of the present invention. The CNN accelerator 700 allows for efficient computation of forward propagation of convolution and other layers. The CNN accelerator 700 accepts an input image and may apply multiple convolution and other layers in succession; [0050] Input image pixels are transmitted into the processing element (PE) arrays 701-704 which may perform independent dot-product operations in a convolution procedure).
In addition, Moreno teaches compute the data read in the common data storage area, (Moreno, Fig. 3, 304 memory, data units (as common data storage area in the memory); [0027] lines 1-4, A data unit 202 describes an input data embedded in a smart contract. There may be a plurality of data units 202. A data unit 202 may be received from a data provider 204 and stored in a memory device or storage device; [0029] lines 1-12, Machine learning algorithms 206 are used to generate machine learning models 208. There may be new or existing machine learning models. A machine learning algorithm, for example, trains a machine learning model, for example, based on a data set contained or referenced in a data unit 202. A machine learning model trained by running a machine learning algorithm or technique may include parameters or weights which are generated during the training phase. A machine learning model takes data units as input (e.g., input feature vectors) in order to produce an outcome such as a prediction, classification, and/or another outcome):
preprocessing the data and inputting the preprocessed data into the invoked neural network models for computation (Moreno, Fig. 1, input to each invoked Model 116-122 (as invoked neural network for computation); [0020] lines 1-15, the data platform determines valuable data units for a type of machine learning algorithm by feeding different combinations of data units into separate machine learning instances. For instance, each combination of data units produces a different training set. (as preprocessing the data); A machine learning instance refers to a machine learning algorithm or architecture set up to build a machine learning model based on a training data set. The data platform in some embodiments evaluates the various training sets against an objective function (such as an expected shape), verifies the correspondence between input data units and outcomes of each machine learning model, and infers which data units provide the best value, for example, to a user of that machine learning model, for example, executed on the data platform).
As per claim 6, HWANG, Moreno, SHI and Ling teach the invention according to claim 1 above. Moreno further teaches wherein training performed to the pre-trained neural network models comprises: constructing a neural network (Moreno, Fig. 1, input to each invoked Model 116-122; [0002] lines 7-8, builds and/or uses machine learning models; [0048] The dataset held or referenced in the data units may be loaded onto the memory 304 for use in training the machine learning models. The trained models may be loaded on the memory 304), selecting a training data set (Moreno, [0034] lines 7-20, each machine learning algorithm may train a model having architecture or structure associated with the machine learning algorithm based on a set of training data. Responsive to a machine learning algorithm being selected, the data platform 210 may select a set of data providers 204. For instance, the data platform 210 may select data providers that provide the type of data used by the selected machine learning algorithm. For example, the data platform 210 may determine or select a data provider, for example, by retrieving the information in the catalog or registry, which specifies data providers, associated type of data offered by the data providers, and machine learning algorithms or types of machine learning algorithms that can use the data offered by the data providers), and training the constructed neural network using the selected training data set, and verifying the trained neural network (Moreno, [0020] lines 1-15, the data platform determines valuable data units for a type of machine learning algorithm by feeding different combinations of data units into separate machine learning instances. For instance, each combination of data units produces a different training set; A machine learning instance refers to a machine learning algorithm or architecture set up to build a machine learning model based on a training data set. The data platform in some embodiments evaluates the various training sets against an objective function (such as an expected shape), verifies the correspondence between input data units and outcomes of each machine learning model, and infers which data units provide the best value, for example, to a user of that machine learning model, for example, executed on the data platform).
As per claim 9, it is a computer device claim of claim 1 above. Therefore, it is rejected for the same reason as claim 1 above. In addition, HWANG further teaches A computer device, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor, when executing the computer program, implements (HWANG, Fig. 9, 915 memory, 910 processor; [0156] lines 1-10, Methods according to the above-described examples can be implemented using computer-executable instructions that are stored or otherwise available from computer readable media. Such instructions can include, for example, instructions and data which cause or otherwise configure a general purpose computer, special purpose computer, or a processing device to perform a certain function or group of functions).
As per claim 10, it is a non-transitory computer-readable storage medium claim of claim 1 above. Therefore, it is rejected for the same reason as claim 1 above.
As per claims 11-13 and 15, they are computer device claims of claims 2-4 and 6 respectively above. Therefore, they are rejected for the same reasons as claims 2-4 and 6 respectively above.
As per claim 16-18 and 20, they are storage medium claims of claims 2-4 and 6 respectively above. Therefore, they are rejected for the same reasons as claims 2-4 and 6 respectively above.
Response to Arguments
Applicant’s arguments with respect to claims 1-4, 6, 9-13, 15-18 and 20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ZUJIA XU whose telephone number is (571)272-0954. The examiner can normally be reached M-F 9:30-5:30 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Aimee J Li can be reached at (571) 272-4169. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ZUJIA XU/Primary Examiner, Art Unit 2195
