DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments with respect to the objections to the drawings and specification have been fully considered and are persuasive. Therefore, the objections to the drawings and specification have been withdrawn.
Applicant’s arguments with respect to the claims have been fully considered and are partially persuasive. Therefore, some of the objections to the claims discussed in the previous Office Action have been withdrawn. However, some objections discussed in the previous Office Action remain as discussed below.
In view of amended claims and Applicant’s arguments with respect to the 35 U.S.C. 101 rejections have been fully considered and are persuasive. Therefore, the 35 U.S.C. 101 rejections to claims 1-20 have been withdrawn.
Applicant’s arguments with respect to the 35 U.S.C. 102 rejections have been fully considered and are persuasive. Therefore, the rejections under 35 U.S.C. 102 have been withdrawn. However, new grounds of rejection under 35 U.S.C. 103 are made to claims 1-20 in light of the filed claim amendments (discussed below).
Claim Objections
Claims 13-20 are objected to because of the following informalities:
Claim 13 still includes reference numbers which should be removed for clarity of the claims; see the limitation in claim 13 “to obtain a plurality of extracted features of the input frame (320)”. Claims 14-20 are objected for their dependency on claim 13.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-3, 5-8, 13-14, 17, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Yang et al. (US 20170076195 A1 (Published 2017); hereinafter Yang) in view of Se et al. (US 20210227126 A1 (Published 2021); hereinafter Se).
Regarding Claim 1,
Yang teaches a method comprising: acquiring, by an input device, an input frame (Yang Specification [0052] “camera module 401 may attain an image or video of a scene and camera module 401 may generate image data 411); discloses a camera (corresponds to an input device) which attains a video of a scene to generate image data (corresponds to acquiring , by an input device, an input frame)); processing, on an embedded processor of the input device, the input frame with a model feature extractor to obtain a plurality of extracted features, the model feature extractor being a quantized version of a model feature extractor of a machine learning model(Yang Specification [0052] “FIG. 4 illustrates an example camera 400 for implementing at least a portion of a neural network … As shown in FIG. 4, camera 400 may include a camera module 401, a hardware (HW) accelerator 402 having a lower level layer (LLL) module 421, a sparse projection module 422, a compression module 423, and a transmitter 403”; [0054] “image data 411 may be provided to hardware accelerator 402, which may generate sub-sampled feature maps (SSFMs) 412. As shown, in some embodiments, hardware accelerator 402 which may generate sub-sampled feature maps 412. However, hardware accelerator 402 may generate any feature maps discussed herein such as convolutional neural network feature maps … hardware accelerator 402 may be a graphics processor”; [0060] “the model stored at camera 400 (or gateways 202, 302) to implement lower level layers of the distributed neural network may be stored in a 16-bit fixed point, 8-bit fixed point, or quantized representation”; discloses a camera system including a hardware accelerator which includes a graphics processor (corresponds to an embedded processor of the input device) which uses lower level layers to generate sub-sampled feature maps (corresponds to extracted features) from image data (corresponds to processing, on an embedded processor of the input device, the input frame with a model feature extractor to obtain a plurality of extracted features). Further, the model stored on the disclosed camera is stored in a quantized representation (corresponds to the model feature extractor being a quantized version of a model feature extractor of a machine learning model)); transmitting, using the input device, the plurality of extracted features to a processing device (Yang Specification [0058] “sub-sampled feature maps 412 may be provided to transmitter 403, which may transmit sub-sampled feature maps 413 to another device (e.g., a gateway or cloud computing device or the like)”; discloses a transmitter used to transmit sub-sampled feature maps (corresponds to extracted features) from the camera system (see Specification [0052] for another device as a processing device (corresponds to transmitting, using the input device, the plurality of extracted features to a processing device)); providing, using the processing device, the plurality of extracted features to a model feature aggregator of the machine learning model to obtain a model result (Yang Specification [0064] “system 500 may receive feature maps, sub-sampled feature maps (FMs) … from remote devices 551, 552 via communications interface 501 … remote devices 551, 552 may include cameras as discussed herein that may provide sub-sampled feature maps and/or image data”; [0065] “processor 502 may, via fully connected portion modules 522, apply one or more fully connected portions of neural networks to generate one or more object labels 512. For example, each of fully connected portion modules 522 may apply a specific object detection model to generate specific object detection output labels”; discloses a system (corresponds to the processing device) which receives sub-sampled feature maps from remote devices and uses fully connected portion modules of a neural network to generate object labels (corresponds to providing, using the processing device, the plurality of extracted features to a model feature aggregator of the machine learning model)). The resulting label is effectively the model result (corresponds to obtain a model result)).
Yang appears to not disclose explicitly transmitting, using the processing device, the model result to the input device.
However, Se teaches transmitting, using the processing device, the model result to the input device (Se Specification [0014] “a system includes a stereo imaging device comprising two or more image capture components configured to capture a pair of images of a scene, a vision processing unit configured to process the image pair through a first trained inference network to determine a first inference result”; [0006] “a first inference result (e.g., image classification, an object detection, a region of interest, an anomaly detection and/or a confidence score)”; [0038] “The inference camera may then send classification/detection inference results to peripheral devices (e.g., via GPIO)”; [00061] “the inference camera system outputs the inspection results via GPIO pins connected to a controller to convey an associated action to be taken (e.g., activate the display to show the result) … The inspection results may include streaming the image to a host system, storing the image on the imaging device, and communicating information (e.g., the inference result, confidence, location of the results) to a peripheral device via GPIO, and/or executing a second inference network”; discloses a system including an imaging device and vision processing unit (corresponds to the processing device) which processes an image to generate an inference result (corresponds to the model result). The inference result is sent to peripheral devices via GPIO which constitutes transmission of model results to the image device or its operative components (corresponds to transmitting, using the processing device, the model result to the input device)).
Yang and Se are considered to be analogous to the claimed invention because they are in the same field of utilizing machine learning models. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yang to incorporate the teachings of Se. Doing so could improve processing speed, efficiency, and security by performing inference on images disclosed by Yang on a local edge device, as suggested by Se (Se Specification [0035] “the inference camera is implemented as an edge device connected to a larger networked system. By enabling inference on the “edge” of a vision system, the inference cameras of the present disclosure deliver improvements in system speed, reliability, power efficiency, and security”).
Regarding Claim 2,
Yang in view of Se teaches the method of claim 1.
Yang further teaches wherein the model feature extractor executes a first neural network layer of a machine learning model on the input frame (Yang Specification [0054] “image data 411 may be provided to hardware accelerator 402 … Hardware accelerator 402 may generate sub-sampled feature maps 412 or the like using any suitable technique or techniques such as via implementation of one or more interleaved convolutional layers and sub-sampling layers”; discloses a hardware accelerator using interleaved convolutional layers and sub-sampling layers (corresponds to a first neural network layer of a machine learning model) to generate sub-sampled feature maps (corresponds to the model feature extractor) from image data (corresponds to the input frame)), and wherein the model feature aggregator executes a second neural network layer of the machine learning model on the plurality of extracted features (Yang Specification [0066] “processor 502 receives feature maps or sub-sampled feature maps … processing may continue via fully connected portion modules 522, which may apply one or more fully connected portions of neural networks to generate one or more object labels 512“; [0032] “fully connected portion 105 may include fully connected layers (FCLs) 116 and fully connected portion 105 may generate an object label … Fully connected portion 105 may include any number of fully connected layers 116 such as two layers, three layers, or more”; discloses a processor generating object labels (corresponds to the model feature aggregator) by utilizing fully connected layers (corresponds to executes a second neural network layer of the machine learning model) to process feature maps (corresponds to the plurality of extracted features)).
Regarding Claim 3,
Yang in view of Se teaches the method of claim 1.
Yang further teaches executing a plurality of model feature aggregators of a plurality of machine learning models on the plurality of extracted features (Yang Fig.3 (reproduced below);
PNG
media_image1.png
495
711
media_image1.png
Greyscale
Specification [0042] “FIG. 3 illustrates an example distributed neural network framework 300 … cloud computing resources (clouds) 303 (e.g., any number cloud computing resources including cloud 303-n) each or some having one or more lower level layer module 321 and/or one or more fully connected portion (FCP) modules 332“; [0047] “Cloud computing resources 303 may receive sets of sub-sampled feature maps 351 and/or sets of sub-sampled feature maps 352. Cloud computing resources 303 may generate object detection labels 353 based on the received sets of sub-sampled feature maps”; discloses several cloud computing resources (clouds) being used to process feature maps (corresponds to the plurality of extracted features) where each cloud includes a fully connected portion module (corresponds to a plurality of machine learning models) which is used to generate object labels (corresponds to executing a plurality of model feature aggregators)), wherein: the model feature extractor is a common model feature extractor for the plurality of machine learning models (Yang Specification [0046] “In embodiments including gateway 302, one or more of sets of sub-sampled feature maps 351 may be received at gateway 302 and gateway 302, via one or more of lower level layer modules 321 may generate sets of sub-sampled feature maps (SSFMs) 352 … sub-sampled feature maps 352 may be provided to one or more of cloud computing resources 303”; [0047] “Cloud computing resources 303 may receive sets of sub-sampled feature maps 351 and/or sets of sub-sampled feature maps 352. Cloud computing resources 303 may generate object detection labels 353 based on the received sets of sub-sampled feature maps”; discloses a gateway being used to generate sub-sampled feature maps (corresponds to the model feature extractor is a common model feature extractor) and provides the sub-sampled feature maps to the cloud computing resources which each include fully connected layers of a machine learning model (corresponds to the plurality of machine learning models)), and the model feature aggregator is one of the plurality of model feature aggregators (As discussed above, Yang Fig.3 discloses several cloud computing resources which each include a fully connected portion module which performs feature aggregation by producing labels from sub-sampled feature maps provided by a gateway (corresponds to the model feature aggregator is one of the plurality of model feature aggregators; for more please see Yang Specification [0046])).
Regarding Claim 5,
Yang in view of Se teaches the method of claim 1.
Yang further teaches training, on a computing system, a machine learning model using training data, the machine learning model comprising the model feature extractor and the model feature aggregator (Yang Specification [0061] “the pre-training of the distributed neural network may be performed using a generic model for generic object (e.g., based on a training dataset) to extract the interleaved convolutional layers and sub-sampling (e.g. max-pooling) layers. Such interleaved convolutional layers and sub-sampling layers may be implemented via camera 400 … To train specialized object detection and/or to upgrade or update such specialized object detection, the lower level parameters may be fixed while performing training to higher levels including subsequent lower level interleaved convolutional layers and sub-sampling layers, if any, and fully connected portions of the neural network”; [0112] “Various embodiments may be implemented using hardware elements, software elements, or a combination of both. Examples of hardware elements may include processors”; discloses performing pre-training based on a training dataset a distributed neural network to be implemented via camera 400 (corresponds to training, on a computing system, a machine learning model using training data). The distributed neural network uses convolutional layers and sub-sampling to generate feature maps (corresponds to the machine learning model comprising the model feature extractor; for more on feature extraction, please see Yang Specification [0054]). Higher level layers including fully connected portions of the neural network are trained as well which are used to generate object labels from feature maps (corresponds to the model feature aggregator; for more on feature aggregation, please see Yang [0065])); and deploying the model feature extractor to the input device and the model feature aggregator to the processing device (Yang Specification [0061] “the pre-training of the distributed neural network may be performed using a generic model for generic object (e.g., based on a training dataset) to extract the interleaved convolutional layers and sub-sampling (e.g. max-pooling) layers. Such interleaved convolutional layers and sub-sampling layers may be implemented via camera 400 … To train specialized object detection and/or to upgrade or update such specialized object detection, the lower level parameters may be fixed while performing training to higher levels including subsequent lower level interleaved convolutional layers and sub-sampling layers, if any, and fully connected portions of the neural network”; [0065] “processor 502 may, via fully connected portion modules 522, apply one or more fully connected portions of neural networks to generate one or more object labels 512”; discloses the pre-trained neural network being implemented via camera 400 (corresponds to deploying (605) the model feature extractor to the input device) and the trained high level fully connection portions are implemented by processor 502 (corresponds to deploying … the model feature aggregator to the processing device)).
Regarding Claim 6,
Yang in view of Se teaches the method of claim 5.
Yang further teaches executing a quantization process on the model feature extractor prior to deploying the model feature extractor (Yang Specification [0060] “the model stored at camera 400 (or gateways 202, 302) to implement lower level layers of the distributed neural network may be stored in a 16-bit fixed point, 8-bit fixed point, or quantized representation. Such representations of the model may provide substantial memory storage requirement reductions with similar accuracy”; discloses quantizing the lower level layers of the distributed network to then store in camera 400 in order to reduce memory storage requirements (corresponds to executing a quantization process on the model feature extractor prior to deploying the model feature extractor)).
Regarding Claim 7,
Yang in view of Se teaches the method of claim 1.
Yang further teaches wherein the model result includes at least one of:
a detection of a face in the input frame; or an attention status of an individual in the input frame (Yang Specification [0031] “Fully connected portion 105 may include any suitable feature classifier such as a multilayer perceptron (MLP) classifier … multiple fully connected portions may be implemented based on sub-sampled feature maps 115 with each fully connected portion performing a particular object detection such as face detection”; discloses a fully connected portion of a neural network classifier with a portion performing face detection (corresponds to the model result includes at least one of: a detection of a face in the input frame)).
Yang appears to not disclose explicitly the remaining limitations of claim 7.
However, Se teaches wherein at least one of, based on the input device receiving the model result: modifying an audio stream or a video stream acquired by the input device, based on the input device receiving the model result; or appending metadata to an input stream of an audio stream or a video stream acquired by the input device (Se Specification [0039] “The inference camera may also be configured to stream low latency images to a host system (e.g., using USB3/GigE vision) while on-camera inference augments the images with rich, descriptive metadata”; discloses appending augmenting inferenced images with rich, descriptive metadata (corresponds to at least one of, based on the input device receiving the model result: appending metadata to an input stream of … a video stream acquired by the input device)).
Yang and Se are considered to be analogous to the claimed invention because they are in the same field of utilizing machine learning models. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yang to incorporate the teachings of Se. Doing so could expand capabilities for existing vision systems to give users and developers flexibility to take advantage of deep learning networks on their own devices, as suggested by Se (Se Specification [0034] “an inference camera can tag images which are then passed to a host which carries out traditional rules-based image processing. In this way users can quickly expand the capabilities of their existing vision systems … the inference cameras disclosed herein may be implemented on an open platform, giving users the flexibility to take advantage of the rapid pace of advancement of deep learning networks and the associated toolchain for their training and optimization”).
Regarding Claim 8,
Yang in view of Se teaches the method of claim 7, wherein
Yang further discloses executing the model feature extractor comprises executing a first subset of neural network layers of a convolutional neural network on the video frame (Yang Specification [0052] “camera 400 may include a camera module 401, a hardware (HW) accelerator 402 having a lower level layer (LLL) module 421”; [0054] “image data 411 may be provided to hardware accelerator 402, which may generate sub-sampled feature maps (SSFMs) 412 … Hardware accelerator 402 may generate sub-sampled feature maps 412 or the like using any suitable technique or techniques such as via implementation of one or more interleaved convolutional layers and sub-sampling layers”; discloses implementing one or more interleaved convolutional layers and sub-sampling layers (corresponds to executing a first subset of neural network layers of a convolutional neural network) on a hardware accelerator to generate feature maps (corresponds to executing the model feature extractor) from image data (corresponds to the video frame)), and processing the model feature aggregator comprises executing a second subset of the neural network layers of a convolutional neural network (CNN) on the plurality of extracted features (Yang Fig. 1 (reproduced below);
PNG
media_image2.png
362
799
media_image2.png
Greyscale
Specification [0025] “FIG. 1 illustrates an example neural network 100, arranged in accordance with at least some implementations of the present disclosure. As shown in FIG. 1, neural network 100 may include lower level layers (LLLs) 121, which may include a convolutional layer (CL) … and a fully connected portion (FCP) 105”; [0031] “Sub-sampled features maps 115 may be provided to fully connected portion 105 of neural network 100. Fully connected portion 105 may include any suitable feature classifier … fully connected portion 105 may include fully connected layers (FCLs) 116 and fully connected portion 105 may generate an object label (OL) 117”; discloses the layers of a neural network including convolutional layers (corresponds to a convolutional neural network (CNN)) and a layer denoted by the fully connected portion (FCP 105; corresponds to a second subset of the neural network layers of the CNN) which is provided sub-sampled feature maps (corresponds to the plurality of extracted features) to generate object labels (corresponds to processing the model feature aggregator)).
Regarding Claim 13,
Yang teaches a system comprising: an input device comprising: an input stream sensor configured to capture an input stream comprising an input frame (Yang Specification [0052] “FIG. 4 illustrates an example camera 400 for implementing at least a portion of a neural network … camera module 401 may attain an image or video of a scene and camera module 401 may generate image data 411); discloses a camera 400 (corresponds to an input device) which attains a video of a scene to generate image data (corresponds to capture an input stream comprising an input frame) by using a camera module 401 (corresponds to an input stream sensor); an embedded processor configured to execute a model feature extractor on the input frame to obtain a plurality of extracted features of the input frame (320), the model feature extractor being a quantized version of a model feature extractor of a machine learning model (Yang Specification [0052] “FIG. 4 illustrates an example camera 400 for implementing at least a portion of a neural network … As shown in FIG. 4, camera 400 may include a camera module 401, a hardware (HW) accelerator 402 having a lower level layer (LLL) module 421, a sparse projection module 422, a compression module 423, and a transmitter 403”; [0054] “image data 411 may be provided to hardware accelerator 402, which may generate sub-sampled feature maps (SSFMs) 412. As shown, in some embodiments, hardware accelerator 402 which may generate sub-sampled feature maps 412. However, hardware accelerator 402 may generate any feature maps discussed herein such as convolutional neural network feature maps … hardware accelerator 402 may be a graphics processor”; [0060] “the model stored at camera 400 (or gateways 202, 302) to implement lower level layers of the distributed neural network may be stored in a 16-bit fixed point, 8-bit fixed point, or quantized representation”; discloses a camera (400) including a hardware accelerator which includes a graphics processor (corresponds to an embedded processor) which uses lower level layers to generate sub-sampled feature maps (corresponds to execute a model feature extractor … to obtain a plurality of extracted features) from image data (corresponds to the input frame). Further, the model stored on the disclosed camera is stored in a quantized representation (corresponds to the model feature extractor being a quantized version of a model feature extractor of a machine learning model)); an input device port configured to transmit the plurality of extracted features from the input device to a processing device (Yang Fig. 4 (reproduced below);
PNG
media_image3.png
210
395
media_image3.png
Greyscale
Specification [0058] “sub-sampled feature maps 412 may be provided to transmitter 403, which may transmit sub-sampled feature maps 413 to another device (e.g., a gateway or cloud computing device or the like) using any suitable communications channel (e.g., wired or wireless communication) and/or any suitable communications protocol”; discloses a camera (400) which includes a transmitter (corresponds to an input device port) to transmit sub-sampled features maps to another device for further processing (corresponds to transmit the plurality of extracted features from the input device to a processing device)); and
wherein the processing device is configured to execute a model feature aggregator of the machine learning model on the plurality of extracted features to obtain a model result (Yang Specification [0064] “system 500 may receive feature maps, sub-sampled feature maps (FMs) … from remote devices 551, 552 via communications interface 501 … remote devices 551, 552 may include cameras as discussed herein that may provide sub-sampled feature maps and/or image data”; [0065] “processor 502 may, via fully connected portion modules 522, apply one or more fully connected portions of neural networks to generate one or more object labels 512. For example, each of fully connected portion modules 522 may apply a specific object detection model to generate specific object detection output labels”; discloses a system including a processor (corresponds to the processing device) which receives sub-sampled feature maps from remote devices (corresponds to the plurality of extracted features (328)) and uses fully connected portion modules of a neural network to process the feature maps and generate object labels as a model result (corresponds to the processing device is configured to execute a model feature aggregator of the machine learning model on the plurality of extracted features to obtain a model result)).
Yang appears to not disclose explicitly the input device is configured to receive the model result from the processing device.
However, Se teaches wherein the input device is configured to receive the model result from the processing device (Se Specification [0014] “a system includes a stereo imaging device comprising two or more image capture components configured to capture a pair of images of a scene, a vision processing unit configured to process the image pair through a first trained inference network to determine a first inference result”; [0006] “a first inference result (e.g., image classification, an object detection, a region of interest, an anomaly detection and/or a confidence score)”; [0038] “The inference camera may then send classification/detection inference results to peripheral devices (e.g., via GPIO)”; [00061] “the inference camera system outputs the inspection results via GPIO pins connected to a controller to convey an associated action to be taken (e.g., activate the display to show the result) … The inspection results may include streaming the image to a host system, storing the image on the imaging device, and communicating information (e.g., the inference result, confidence, location of the results) to a peripheral device via GPIO, and/or executing a second inference network”; discloses a system including an imaging device and vision processing unit (corresponds to the processing device) which processes an image to generate an inference result (corresponds to the model result). The inference result is sent to peripheral devices via GPIO which constitutes transmission of model results to the image device or its operative components (corresponds to the input device is configured to receive the model result from the processing device).
Yang and Se are considered to be analogous to the claimed invention because they are in the same field of utilizing machine learning models. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yang to incorporate the teachings of Se. Doing so could improve processing speed, efficiency, and security by performing inference on images disclosed by Yang on a local edge device, as suggested by Se (Se Specification [0035] “the inference camera is implemented as an edge device connected to a larger networked system. By enabling inference on the “edge” of a vision system, the inference cameras of the present disclosure deliver improvements in system speed, reliability, power efficiency, and security”). .
Regarding Claim 14,
Yang in view of Se teaches the system of claim 13.
Yang further discloses memory storing the model feature aggregator; and
a hardware processor configured to execute the model feature aggregator stored in the memory (Yang Specification [0065] “processor 502 may receive feature maps … processor 502 may, via fully connected portion modules 522, apply one or more fully connected portions of neural networks to generate one or more object labels 512”; [0071] “FIG. 7 is an illustrative diagram of an example system 700 for implementing at least a portion of a neural network, arranged in accordance with at least some implementations of the present disclosure … system 700 may include a central processor 701, a graphics processor 702, a memory 703, a communications interface 501, and/or a transmitter 503. Also as shown, central processor 701 may include or implement lower level layer modules 521 and fully connected portion modules 522. In the example of system 700, memory 703 may store sensor data, image data, video data, or related content such as input layer data, feature maps, sub-sampled feature maps, neural network parameters or models, object labels, and/or any other data”; discloses a system 700 which performs the same processing as example system 500. The system 700 includes a memory storing the neural network models used for executing fully connected portion modules to generate object labels (corresponds to memory storing the model feature aggregator). System 700 also includes a central processor (701) to implement the fully connected portion modules to generate object labels (corresponds to a hardware processor configured to execute the model feature aggregator stored in the memory)).
Regarding Claim 17,
Yang in view of Se teaches the system of claim 13.
Yang further discloses the input stream sensor comprises a camera configured to capture a video stream comprising the input frame (Yang Specification [0052] “camera module 401 may attain an image or video of a scene and camera module 401 may generate image data 411. Image data 411 may include any suitable image or video frame data”; discloses a camera module (corresponds to the input stream sensor comprises a camera) which attains a video of a scene which includes frames (corresponds to capture a video stream comprising the input frame)), wherein the input frame is a video frame in the video stream (Yang Specification [0052] “camera module 401 may attain an image or video of a scene and camera module 401 may generate image data 411. Image data 411 may include any suitable image or video frame data”; discloses the camera module attaining a video of a scene (corresponds to the video stream) to generate image data which includes video frame data (corresponds to the input frame is a video frame in the video stream)), the model feature extractor comprises a first subset of neural network layers of a convolutional neural network (CNN) (Yang Specification [0052] “camera 400 may include a camera module 401, a hardware (HW) accelerator 402 having a lower level layer (LLL) module 421”; [0054] “image data 411 may be provided to hardware accelerator 402, which may generate sub-sampled feature maps (SSFMs) 412 … Hardware accelerator 402 may generate sub-sampled feature maps 412 or the like using any suitable technique or techniques such as via implementation of one or more interleaved convolutional layers and sub-sampling layers”; discloses implementing one or more interleaved convolutional layers and sub-sampling layers (corresponds to a first subset of neural network layers of a convolutional neural network (CNN)) on a hardware accelerator to generate feature maps from image data (corresponds to the model feature extractor)), and
the model feature aggregator comprises a second subset of neural network layers of the CNN (Yang Fig. 1 (reproduced below);
PNG
media_image2.png
362
799
media_image2.png
Greyscale
Specification [0025] “FIG. 1 illustrates an example neural network 100, arranged in accordance with at least some implementations of the present disclosure. As shown in FIG. 1, neural network 100 may include lower level layers (LLLs) 121, which may include a convolutional layer (CL) … and a fully connected portion (FCP) 105”; [0031] “Sub-sampled features maps 115 may be provided to fully connected portion 105 of neural network 100. Fully connected portion 105 may include any suitable feature classifier … fully connected portion 105 may include fully connected layers (FCLs) 116 and fully connected portion 105 may generate an object label (OL) 117”; discloses the layers of a neural network including convolutional layers (corresponds to the CNN) and a layer denoted by the fully connected portion (see Yang Fig.1, FCP 105; corresponds to a second subset of the neural network layers of the CNN) which is provided sub-sampled feature maps to generate object labels (corresponds to the model feature aggregator)).
Regarding Claim 19,
Yang in view of Se teaches the system of claim 13.
Yang further teaches an embedded processor executed model comprising a second model feature aggregator that executes on the plurality of extracted features (Yang Fig.3 (reproduced below);
PNG
media_image1.png
495
711
media_image1.png
Greyscale
Specification [0042] “FIG. 3 illustrates an example distributed neural network framework 300 … cloud computing resources (clouds) 303 (e.g., any number cloud computing resources including cloud 303-n) each or some having one or more lower level layer module 321 and/or one or more fully connected portion (FCP) modules 332“; [0047] “Cloud computing resources 303 may receive sets of sub-sampled feature maps 351 and/or sets of sub-sampled feature maps 352. Cloud computing resources 303 may generate object detection labels 353 based on the received sets of sub-sampled feature maps”; [0078] “The fully connected portion of a neural network may be implemented using any suitable technique or techniques. In some embodiments, the fully connected portion may be implemented via fully connected portion module 232 of cloud computing resource 203, any of fully connected portion modules 332 of any of cloud computing resources 303, any of fully connected portion modules 521 as implemented via processor 502 of system 500 or as implemented via central processor 701 of system 700”; discloses fully connected portion modules implemented via a processor (corresponds to an embedded processor executed model). Fig.3 further discloses cloud computing resources (clouds) which each included a fully connected portion module for generated object labels from feature maps (corresponds to a second model feature aggregator that executes on the plurality of extracted features)), wherein: the model feature aggregator is a first model feature aggregator and is an offloaded model (Yang Specification [0030] “lower level layers 121 of neural network 100 may include, in some embodiments, interleaved convolutional layers 101, 103 and sub-sampling layers 102, 104”; [0031] “Sub-sampled features maps 115 may be provided to fully connected portion 105 of neural network 100. Fully connected portion 105 may include any suitable feature classifier”; [0039] “cloud computing resource 203 may, via fully connected portion module 232 implement a fully connected portion (e.g., fully connected portion 105 or the like) of distributed neural network framework 200 to generate object label 253”; discloses a portion of a neural network including a fully connected portion which is used to generate object labels (corresponds to the model feature aggregator is a first model feature aggregator). The entire neural network includes several layers, e.g., convolutional layers, sub-sampling layers, fully connected layers, and the portion including fully connected layers is effectively offloaded to the cloud computing resources while another device executes the remaining layers (corresponds to an offloaded model)), and the model feature extractor is a common model feature extractor for the second model feature aggregator and the first model feature aggregator (Yang Specification [0046] “In embodiments including gateway 302, one or more of sets of sub-sampled feature maps 351 may be received at gateway 302 and gateway 302, via one or more of lower level layer modules 321 may generate sets of sub-sampled feature maps (SSFMs) 352 … sub-sampled feature maps 352 may be provided to one or more of cloud computing resources 303”; [0047] “Cloud computing resources 303 may receive sets of sub-sampled feature maps 351 and/or sets of sub-sampled feature maps 352. Cloud computing resources 303 may generate object detection labels 353 based on the received sets of sub-sampled feature maps”; discloses a gateway being used to generate sub-sampled feature maps (corresponds to the model feature extractor is a common model feature extractor) and provides the sub-sampled feature maps to the cloud computing resources which each include fully connected layers of a machine learning model (corresponds to the second model feature aggregator and the first model feature aggregator)).
Regarding Claim 20,
Yang in view of Se teaches the system of claim 13.
Yang further teaches a plurality of model feature aggregators configured to individually execute the plurality of extracted features to obtain a plurality of model results (Yang Fig.3 (reproduced below);
PNG
media_image1.png
495
711
media_image1.png
Greyscale
Specification [0042] “FIG. 3 illustrates an example distributed neural network framework 300 … cloud computing resources (clouds) 303 (e.g., any number cloud computing resources including cloud 303-n) each or some having one or more lower level layer module 321 and/or one or more fully connected portion (FCP) modules 332“; [0047] “Cloud computing resources 303 may receive sets of sub-sampled feature maps 351 and/or sets of sub-sampled feature maps 352. Cloud computing resources 303 may generate object detection labels 353 based on the received sets of sub-sampled feature maps”; [0078] “The fully connected portion of a neural network may be implemented using any suitable technique or techniques. In some embodiments, the fully connected portion may be implemented via fully connected portion module 232 of cloud computing resource 203, any of fully connected portion modules 332 of any of cloud computing resources 303, any of fully connected portion modules 521 as implemented via processor 502 of system 500 or as implemented via central processor 701 of system 700”; discloses several cloud computing resources (clouds) being used to process feature maps (corresponds to the plurality of extracted features) where each cloud includes a fully connected portion module to generate object labels (corresponds to a plurality of model feature aggregators configured to individually execute the plurality of extracted features) where object labels are effectively a model result from each cloud (corresponds to obtain a plurality of model results), wherein: the model feature aggregator is one of the plurality of model feature aggregators (As discussed above, Yang Fig.3 discloses several cloud computing resources which each include a fully connected portion module which performs feature aggregation by producing labels from sub-sampled feature maps provided by a gateway (corresponds to the model feature aggregator is one of the plurality of model feature aggregators; for more please see Yang Specification [0046])), the model feature extractor is a common model feature extractor for the plurality of model feature aggregators (Yang Specification [0046] “In embodiments including gateway 302, one or more of sets of sub-sampled feature maps 351 may be received at gateway 302 and gateway 302, via one or more of lower level layer modules 321 may generate sets of sub-sampled feature maps (SSFMs) 352 … sub-sampled feature maps 352 may be provided to one or more of cloud computing resources 303”; [0047] “Cloud computing resources 303 may receive sets of sub-sampled feature maps 351 and/or sets of sub-sampled feature maps 352. Cloud computing resources 303 may generate object detection labels 353 based on the received sets of sub-sampled feature maps”; discloses a gateway being used to generate sub-sampled feature maps (corresponds to the model feature extractor is a common model feature extractor) and provides the sub-sampled feature maps to the cloud computing resources which each include fully connected layers of a machine learning model (corresponds to the plurality of model feature aggregators)) and a first model feature aggregator, and the plurality of model results comprises the model result (As discussed above, Yang Fig.3 discloses several cloud computing resources (corresponds to the first model feature aggregator) which each include a fully connected portion module which performs feature aggregation by producing labels from sub-sampled feature maps provided by a gateway (corresponds to the plurality of model results; for more please see Yang Specification [0046]). The results from each model constituting a feature aggregator from each cloud is thus the model result (corresponds to the model result)).
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Yang et al. (US 20170076195 A1 (Published 2017); hereinafter Yang) in view of Se et al. (US 20210227126 A1 (Published 2021); hereinafter Se) in view of Stahl et al. (“DeeperThings: Fully Distributed CNN Inference on Resource-Constrained Edge Devices” (Published 2021); hereinafter Stahl).
Regarding Claim 4,
Yang in view of Se teaches the method of claim 1, wherein the model feature aggregator is a first model feature aggregator, and the model result is a first model result, but appears to not disclose explicitly the remaining limitations of claim 4.
However, Stahl teaches executing, on the input device, a second model feature aggregator on the plurality of extracted features to obtain a second model result (Stahl P.618, Sec.6.2, Para.1 “we validated our complete DeeperThings approach on a physical edge cluster consisting of six Raspberry Pi 4 devices. They have a quad-core ARM CortexA72 processor, 2 GB of RAM and a gigabit Ethernet interface”; P.619 Para.1 “Edge devices fulfill two different roles during the inference process. Either they provide the input data as the source device, or they assist with the inference as worker devices”; P.609, Sec.4.2.1, Para.1 “The output subset can then be finalized by applying the activation function f. The final partitioned output values must then only be concatenated (Concat/L1C) to obtain the full output vector bl, thus synchronizing the layer’s output across all devices”; P.620, Para.1 “The run time is measured from start of inference until the final inference result has been calculated”; discloses a cluster of edge devices (corresponds to the input device) used for machine learning inference including concatenating partitioned output values to obtain a full output vector from feature maps (corresponds to executing a second model feature aggregator on the plurality of extracted features). The run time of the model is measure from start of inference until the final result is produced effectively acting as the result of the model’s inference (corresponds to a second model result); and processing the second model result (Stahl P.620, Para.1 “The run time is measured from start of inference until the final inference result has been calculated … For each configuration, ten measurements were taken and the results were averaged to take run time variability into account”; discloses using several measurements of model runtimes on the edge device cluster (corresponds to the second model result) to then take an average of run time (corresponds to processing the second model result)).
Yang, Se and Stahl are considered to be analogous to the claimed invention because they are in the same field of utilizing machine learning models on edge devices. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yang and Se to incorporate the teachings of Stahl. Doing so could reduce communication overhead and latency in convolutional neural network inference by offloading more model computation to the input edge device disclosed by Yang, as suggested by Stahl (Stahl P.601, Sec.1, Para.3, “we describe DeeperThings, a comprehensive approach for the distributed execution of complete CNNs considering all layer-types while simultaneously optimizing for computation, memory and communication demands. The computation and memory footprint of processing and storing feature and weight data is evenly distributed over all devices, such that the CNN inference task can be scaled down for resource-constrained IoT edge devices”).
Claims 9-12 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Yang et al. (US 20170076195 A1 (Published 2017); hereinafter Yang) in view of Se et al. (US 20210227126 A1 (Published 2021); hereinafter Se) in view of Pishehvar et al. (US 20220366927 A1 (Filed 2021); hereinafter Pishehvar).
Regarding Claim 9,
Yang in view of Se teaches the method of claim 1 and the input device, but appears to not disclose explicitly the remaining limitations of claim 9.
However, Pishehvar teaches capturing, by a microphone in the input device, an audio stream (Pishehvar Specification [0031] “The microphones 102, 103, and 104 may form a compact microphone array to capture speech signals from the user 110. As an example, the user 110 may utter a query keyword such as ‘Hey Siri’ followed by the query ‘What time is it?’ to request the current time from a smart assistant application”; discloses using microphones in a microphone array (corresponds to a microphone in the input device) in order to capture speech signals from a user (corresponds to capturing, …, an audio stream)); and
extracting the input frame from the audio stream, wherein the input frame is a sample of audio in the audio stream (Pishehvar Specification [0031] “The smartphone 101 may divide the speech signals captured by the microphones into frames and may transmit the audio data frames to a multi-task machine learning model running on the smartphone 101 or on a remote server”; discloses dividing speech signals of a user into frames (corresponds to extracting the input frame from the audio stream) where the divided frames are effectively samples of audio (corresponds to the input frame is a sample of audio in the audio stream)).
Yang, Se and Pishehvar are considered to be analogous to the claimed invention because they are in the same field of utilizing machine learning models to process image data. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yang and Se to incorporate the teachings of Pishehvar. Doing so could improve the robustness of models disclosed by Yang by utilizing multi-model signals, e.g., audio and video, to perform machine learning classification, as suggested by Pishehvar (Pishehvar Specification [0009] “the multi-task DNN model may process visual signals captured by one or more cameras or other multi-modal signals to further improve the robustness of the enhanced target speech signal”).
Regarding Claim 10,
Yang teaches a method comprising: acquiring, by an input device, an input frame … (Yang Specification [0052] “camera module 401 may attain an image or video of a scene and camera module 401 may generate image data 411); discloses a camera (corresponds to an input device) which attains a video of a scene to generate image data (corresponds to acquiring (701), …., an input frame)); executing, by an embedded processor of the input device, a model feature extractor of a machine learning model on the input frame to obtain a plurality of extracted features of the input frame (Yang Specification [0052] “FIG. 4 illustrates an example camera 400 for implementing at least a portion of a neural network … As shown in FIG. 4, camera 400 may include a camera module 401, a hardware (HW) accelerator 402 having a lower level layer (LLL) module 421, a sparse projection module 422, a compression module 423, and a transmitter 403”; [0054] “image data 411 may be provided to hardware accelerator 402, which may generate sub-sampled feature maps (SSFMs) 412. As shown, in some embodiments, hardware accelerator 402 which may generate sub-sampled feature maps 412. However, hardware accelerator 402 may generate any feature maps discussed herein such as convolutional neural network feature maps … hardware accelerator 402 may be a graphics processor”; discloses a camera system including a hardware accelerator which includes a graphics processor (corresponds to an embedded processor of the input device) which uses lower level layers to generate sub-sampled feature maps (corresponds to extracted features) from image data (corresponds to executing, …, a model feature extractor of a machine learning model on the input frame to obtain a plurality of extracted features of the input frame)) transmitting the plurality of extracted features from the input device to a processing device (Yang Specification [0058] “sub-sampled feature maps 412 may be provided to transmitter 403, which may transmit sub-sampled feature maps 413 to another device (e.g., a gateway or cloud computing device or the like)”; discloses a transmitter used to transmit sub-sampled feature maps (corresponds to extracted features) from the camera system (see Specification [0052] to another device (corresponds to transmitting the plurality of extracted features from the input device to a processing device)); providing, using the processing device, the plurality of extracted features to a model feature aggregator to obtain a model result (Yang Specification [0064] “system 500 may receive feature maps, sub-sampled feature maps (FMs) … from remote devices 551, 552 via communications interface 501 … remote devices 551, 552 may include cameras as discussed herein that may provide sub-sampled feature maps and/or image data”; [0065] “processor 502 may, via fully connected portion modules 522, apply one or more fully connected portions of neural networks to generate one or more object labels 512. For example, each of fully connected portion modules 522 may apply a specific object detection model to generate specific object detection output labels”; discloses a system (corresponds to the processing device) which receives sub-sampled feature maps from remote devices and uses fully connected portion modules of a neural network to generate object labels (corresponds to providing, using the processing device, the plurality of extracted features to a model feature aggregator)). The resulting label is effectively the model result (corresponds to obtain a model result)); and
Yang appears to not disclose explicitly an input frame that includes an audio frame and transmitting, using the processing device, the model result to the input device.
However, Se teaches transmitting, using the processing device, the model result to the input device (Se Specification [0014] “a system includes a stereo imaging device comprising two or more image capture components configured to capture a pair of images of a scene, a vision processing unit configured to process the image pair through a first trained inference network to determine a first inference result”; [0006] “a first inference result (e.g., image classification, an object detection, a region of interest, an anomaly detection and/or a confidence score)”; [0038] “The inference camera may then send classification/detection inference results to peripheral devices (e.g., via GPIO)”; [00061] “the inference camera system outputs the inspection results via GPIO pins connected to a controller to convey an associated action to be taken (e.g., activate the display to show the result) … The inspection results may include streaming the image to a host system, storing the image on the imaging device, and communicating information (e.g., the inference result, confidence, location of the results) to a peripheral device via GPIO, and/or executing a second inference network”; discloses a system including an imaging device and vision processing unit (corresponds to the processing device) which processes an image to generate an inference result (corresponds to the model result). The inference result is sent to peripheral devices via GPIO which constitutes transmission of model results to the image device or its operative components (corresponds to transmitting, using the processing device, the model result to the input device)).
Yang and Se are considered to be analogous to the claimed invention because they are in the same field of utilizing machine learning models. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yang to incorporate the teachings of Se. Doing so could improve processing speed, efficiency, and security by performing inference on images disclosed by Yang on a local edge device, as suggested by Se (Se Specification [0035] “the inference camera is implemented as an edge device connected to a larger networked system. By enabling inference on the “edge” of a vision system, the inference cameras of the present disclosure deliver improvements in system speed, reliability, power efficiency, and security”).
Yang in view of Se does not appear to disclose explicitly an input frame that includes an audio frame.
However, Pishehvar teaches an input frame that includes an audio frame (Pishehvar Specification [0031] “FIG. 1 depicts a scenario of a user uttering speech during a telephony or video conferencing call or issuing a voice command to a smartphone … The smartphone 101 may divide the speech signals captured by the microphones into frames and may transmit the audio data frames”; discloses capturing and transmitting audio data frames (corresponds to an input frame that includes an audio frame)).
Yang, Se, and Pishehvar are considered to be analogous to the claimed invention because they are in the same field of utilizing machine learning models. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yang and Se to incorporate the teachings of Pishehvar. Doing so could improve the robustness of models disclosed by Yang by utilizing multi-model signals, e.g., audio and video, to perform machine learning classification, as suggested by Pishehvar (Pishehvar Specification [0009] “the multi-task DNN model may process visual signals captured by one or more cameras or other multi-modal signals to further improve the robustness of the enhanced target speech signal”).
Regarding Claim 11,
Yang in view of Se in view of Pishehvar teaches the method of claim 10.
Yang appears to not disclose explicitly the limitations of claim 10.
However, Se teaches processing, using the input device, the model result to determine an action identifier (Se Specification [0008] “The image device may further include a processing component configured to control the operation of the imaging device, including processing the first inference result and/or determining an associate action to take for the image. In some embodiments, the associated action may include streaming the image to a host system, storing the image on the imaging device, and/or executing a second inference network”; discloses an image device to process inference results (corresponds to processing, using the input device, the model result) in order to determine an associated action to take for an image (corresponds to determine an action identifier)); and wherein the action identifier causes the input device to perform a corresponding action as triggered by the model result; or
wherein the action identifier includes at least one of: displaying information in a graphical user interface according to the model result; transmitting an alert according to the model result;
transforming an input stream acquired by the input device according to the model result;
modifying an audio stream or a video stream acquired by the input device by changing the audio stream or video stream, according to the model result; or appending metadata to an input stream, transmitting an alert, or performing another action as triggered by the model result (Se Specification [0008] “The image device may further include a processing component configured to control the operation of the imaging device, including processing the first inference result and/or determining an associate action to take for the image. In some embodiments, the associated action may include streaming the image to a host system, storing the image on the imaging device, and/or executing a second inference network”; [0061] “the inference camera system outputs the inspection results via GPIO pins connected to a controller to convey an associated action to be taken (e.g., activate the display to show the result)”; discloses determining an action for an inference system based on an inference result (corresponds to the action identifier causes the input device to perform a corresponding action as triggered by the model result). Actions to be taken include streaming the image to a host system, storing the image, executing a second inference network (corresponds to the action identifier includes at least one of: … performing another action as triggered by the model result), and/or activating a display to show the inference result (corresponds to the action identifier includes at least one of: … displaying information in a graphical user interface according to the model result)).
Yang, Pishehvar and Se are considered to be analogous to the claimed invention because they are in the same field of utilizing machine learning models. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yang and Pishehvar to incorporate the teachings of Se. Doing so could improve data transmission for edge devices performing image inference by transmitting descriptive data with respect to images such as actions instead of the entire images themselves, as suggested by Se (Se Specification [0035] “Rather than transmitting whole images to a remote server, the inference camera can transmit descriptive data as needed, which may greatly reduce the amount of data which a system must transmit, minimizing network bandwidth and system latency”).
Regarding Claim 12,
Yang in view of Se in view of Pishehvar teaches the method of claim 10.
Yang further discloses executing a plurality of model feature aggregators of a plurality of machine learning models on the plurality of extracted features (Yang Fig.3 (reproduced below);
PNG
media_image1.png
495
711
media_image1.png
Greyscale
Specification [0042] “FIG. 3 illustrates an example distributed neural network framework 300 … cloud computing resources (clouds) 303 (e.g., any number cloud computing resources including cloud 303-n) each or some having one or more lower level layer module 321 and/or one or more fully connected portion (FCP) modules 332“; [0047] “Cloud computing resources 303 may receive sets of sub-sampled feature maps 351 and/or sets of sub-sampled feature maps 352. Cloud computing resources 303 may generate object detection labels 353 based on the received sets of sub-sampled feature maps”; discloses several cloud computing resources (clouds) being used to process feature maps (corresponds to the plurality of extracted features) where each cloud includes a fully connected portion module (corresponds to a plurality of machine learning models) which is used to generate object labels (corresponds to executing a plurality of model feature aggregators)), wherein: the model feature extractor is a common model feature extractor for the plurality of machine learning models (Yang Specification [0046] “In embodiments including gateway 302, one or more of sets of sub-sampled feature maps 351 may be received at gateway 302 and gateway 302, via one or more of lower level layer modules 321 may generate sets of sub-sampled feature maps (SSFMs) 352 … sub-sampled feature maps 352 may be provided to one or more of cloud computing resources 303”; [0047] “Cloud computing resources 303 may receive sets of sub-sampled feature maps 351 and/or sets of sub-sampled feature maps 352. Cloud computing resources 303 may generate object detection labels 353 based on the received sets of sub-sampled feature maps”; discloses a gateway being used to generate sub-sampled feature maps (corresponds to the model feature extractor is a common model feature extractor) and provide the sub-sampled feature maps to the cloud computing resources which include fully connected layers of a machine learning model (corresponds to the plurality of machine learning models)), and the model feature aggregator is one of the plurality of model feature aggregators (As discussed above, Yang Fig.3 discloses several cloud computing resources which each include a fully connected portion module which performs feature aggregation by producing labels from sub-sampled feature maps provided by a gateway (corresponds to the model feature aggregator is one of the plurality of model feature aggregators; for more please see Yang Specification [0046])).
Regarding Claim 18,
Yang in view of Se teaches the system of claim 13 and the input stream sensor (see Yang Specification [0052]; camera module 401).
Yang further discloses the model feature extractor comprises a first subset of neural network layers of a recurrent neural network (RNN) (Yang Specification [0052] “camera 400 may include a camera module 401, a hardware (HW) accelerator 402 having a lower level layer (LLL) module 421”; [0054] “image data 411 may be provided to hardware accelerator 402, which may generate sub-sampled feature maps (SSFMs) 412 … Hardware accelerator 402 may generate sub-sampled feature maps 412 or the like using any suitable technique or techniques such as via implementation of one or more interleaved convolutional layers and sub-sampling layers”; [0020] “the distributed neural network may include any suitable neural network such as a convolutional neural network (CNN), a deep neural network (DNN), recurrent convolutional neural network (RCNN)”; discloses implementing one or more interleaved convolutional layers and sub-sampling layers (corresponds to a first subset of neural network layers) on a hardware accelerator to generate feature maps (corresponds to the model feature extractor) and further discloses the distributed network to be a recurrent convolutional neural network (corresponds to a recurrent neural network (RNN)), and
the model feature aggregator comprises a second subset of neural network layers of the RNN (Yang Fig. 1 (reproduced below);
PNG
media_image2.png
362
799
media_image2.png
Greyscale
Specification [0025] “FIG. 1 illustrates an example neural network 100, arranged in accordance with at least some implementations of the present disclosure. As shown in FIG. 1, neural network 100 may include lower level layers (LLLs) 121, which may include a convolutional layer (CL) … and a fully connected portion (FCP) 105”; [0031] “Sub-sampled features maps 115 may be provided to fully connected portion 105 of neural network 100. Fully connected portion 105 may include any suitable feature classifier … fully connected portion 105 may include fully connected layers (FCLs) 116 and fully connected portion 105 may generate an object label (OL) 117”; [0020] “the distributed neural network may include any suitable neural network such as a convolutional neural network (CNN), a deep neural network (DNN), recurrent convolutional neural network (RCNN)”; discloses the layers of a neural network including convolutional layers (corresponds to the CNN) and a layer denoted by the fully connected portion (FCP 105; corresponds to a second subset of the neural network layers of the CNN) which is provided sub-sampled feature maps (corresponds to the plurality of extracted features) to generate object labels (corresponds to processing the model feature aggregator) and further discloses the distributed network to be a recurrent convolutional neural network (corresponds to the RNN).
Yang appears to not disclose explicitly the input stream sensor comprises a microphone configured to capture an audio stream comprising the input frame.
However, Pishehvar teaches the input stream sensor comprises a microphone configured to capture an audio stream comprising the input frame (Pishehvar Specification [0031] “The microphones 102, 103, and 104 may form a compact microphone array to capture speech signals from the user 110. As an example, the user 110 may utter a query keyword such as ‘Hey Siri’ followed by the query ‘What time is it?’ to request the current time from a smart assistant application”; [0031] The smartphone 101 may divide the speech signals captured by the microphones into frames and may transmit the audio data frames to a multi-task machine learning model running on the smartphone 101 or on a remote server”; discloses using microphones in a microphone array (corresponds to the input stream sensor (322) comprises a microphone (308)) in order to capture speech signals from a user which are divided into frames (corresponds to capture an audio stream comprising the input frame)), wherein the input frame is a sample of audio in the audio stream (Pishehvar Specification [0031] “The smartphone 101 may divide the speech signals captured by the microphones into frames and may transmit the audio data frames to a multi-task machine learning model running on the smartphone 101 or on a remote server”; discloses dividing speech signals of a user into frames (corresponds to the input frame) where the divided frames are effectively samples of audio (corresponds to the input frame is a sample of audio in the audio stream)).
Yang, Se and Pishehvar are considered to be analogous to the claimed invention because they are in the same field of utilizing machine learning models to process image data. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yang and Se to incorporate the teachings of Pishehvar. Doing so could improve the robustness of models disclosed by Yang by utilizing multi-model signals, e.g., audio and video, to perform machine learning classification, as suggested by Pishehvar (Pishehvar Specification [0009] “the multi-task DNN model may process visual signals captured by one or more cameras or other multi-modal signals to further improve the robustness of the enhanced target speech signal”).
Claims 15-16 are rejected under 35 U.S.C. 103 as being unpatentable over Yang et al. (US 20170076195 A1 (Published 2017); hereinafter Yang) in view of Se et al. (US 20210227126 A1 (Published 2021); hereinafter Se) in view of Lai et al. (“Deep Convolutional Neural Network Inference with Floating-point Weights and Fixed-point Activations” (Published 2017); hereinafter Lai).
Regarding Claim 15,
Yang in view of Se teaches the system of claim 13.
Yang further discloses the model feature extractor on the input device is a fixed-point version (Yang Specification [0060] “the model stored at camera 400 (or gateways 202, 302) to implement lower level layers of the distributed neural network may be stored in a 16-bit fixed point, 8-bit fixed point, or quantized representation. Such representations of the model may provide substantial memory storage requirement reductions with similar accuracy”; discloses storing the lower level layers in camera 400 (corresponds to the input device (302) which generates feature maps (corresponds to the model feature extractor (216); for more, please see Yang Specification [0054]) is stored in a 16-bit or 8-bit fixed point representation (corresponds to a fixed-point version (212)), and the model feature aggregator on the processing device is a floating-point version (Yang Specification [0060] “the models stored at the gateway and/or the cloud computing resources for … the fully connected portion (e.g., at the cloud computing resources) may be stored as 32-bit floating point representations of the models”; discloses storing the fully connected portion which generate object labels (corresponds to the model feature aggregator (218); for more please see Yang Specification [0065]) on the cloud computing resources and a system including a processor (corresponds to the processing device (304); for more on the system including a processor, please see Yang Specification [0065]) as a 32-bit floating point representation (corresponds to an floating-point version (214)).
Yang in view of Se appears to not disclose explicitly a computing system comprising a hardware processor executing a model training system to train a floating-point version of the model feature extractor and the model feature aggregator.
However, -----Lai teaches a computing system comprising a hardware processor executing a model training system to train a floating-point version of the model feature extractor and the model feature aggregator (Lai P.1, Col.2, Para.2 “Typically, CNNs are trained on high performance CPU/GPU with 32-bit floating-point data”; P.3, Col.2, Sec.3.3, Para.1 “CNNs typically consist of multiple convolution layers interspersed by pooling, ReLU and normalization layers followed by fully-connected layers”; discloses convolutional neural networks being trained by a CPU or GPU (corresponds to a computing system comprising a hardware processor executing a model training system (200) using 32-bit floating point data (corresponds to train a floating-point version of the model). Convolutional neural networks perform feature extraction (corresponds to the model feature extractor (208)), and the CNNs discussed by Lai include pooling layers which perform feature aggregation (corresponds to the model feature aggregator (210)).
Yang, Se and Lai are considered to be analogous to the claimed invention because they are in the same field of utilizing neural network models. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yang and Se to incorporate the teachings of Lai. Doing so could improve the efficiency of storage and power consumption of models trained in Yang, as suggested by Lai (Lai P.1, Col.2, Para.2 “Fixed-point representation with shorter bit-width for CNN weights and activations has been widely explored … which significantly reduces the storage requirements, memory bandwidth and power consumption without sacrificing accuracy”).
Regarding Claim 16,
Yang in view of Se in view of Lai teaches the system of claim 15 and the hardware processor.
Lai further teaches execut[ing] a quantization process to reduce the floating-point version of the model feature extractor to the fixed-point version of the model feature extractor (Lai P.2, Col.1, Para.1 “we propose using floating-point numbers for CNN weights and fixed-point numbers for activations”; P.4, Col.1, Sec.5.1, Para.2 “We apply the weight quantization on four popular CNN networks … We evaluate the network accuracy by doing quantization for all convolutional and fully-connected layer weights. The activation is quantized to 16-bit fixed-point.”; discloses performing quantization on convolutional neural networks (corresponds to execut[ing] a quantization process [204]) which reduces a floating-point representation to a 16-bit fixed point representation for activations (corresponds to reduce the floating-point version (206) of the model … to the fixed-point version (212) of the model). The quantization is performed on the convolutional layers which, in a convolutional neural network, perform feature extraction (corresponds to the model feature extractor (208) and the model feature extractor (216))).
Yang, Se and Lai are considered to be analogous to the claimed invention because they are in the same field of utilizing neural network models. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yang to incorporate the teachings of Lai. Doing so could assist in reducing the size of models on camera devices disclosed by Yang, as suggested by Lai (Lai P.9, Col.1, Para.1 “The proposed scheme can reduce the weight storage by up to 36% and the multiplier power by up to 50%”).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KAMRAN AFSHAR whose telephone number is 571-272-7796. The examiner can normally be reached Monday-Thursday: 8:00AM-4:00PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KAMRAN AFSHAR/Supervisory Patent Examiner, Art Unit 2125
Prosecution Insights