DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 05/05/2026 has been entered.
Status of Claims
The present application is being examined under the claims filed 05/05/2026.
Claims 1-20 are pending.
Allowable Subject Matter
Claims 8-9 and 18-19 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. Examiner notes that the claims must also be amended to overcome the rejections under 35 U.S.C. 112 prior to allowance. Reasons for allowance appear in a previous correspondence and are thus not included herein.
Response to Amendment
This Office Action is in response to Applicant’s communication filed 05/05/2026 in response to office action mailed 12/08/2025. The Applicant’s remarks and any amendments to the claims or specification have been considered with the results that follow.
Response to Arguments
Regarding objections and 35 U.S.C. 112(b) rejections
In Remarks page 13, Argument 1
(Examiner summarizes Applicant’s arguments) Applicant argues that claims 9-10 and 14 have been amended thus obviating the objections and 112(b) rejections.
Examiner’s response to Argument 1
Each and every objection and rejection under 35 U.S.C. 112 appears to have been overcome.
Regarding 35 U.S.C. 103 rejections
In Remarks pages 13-17, Argument 2
(Examiner summarizes Applicant’s arguments) Applicant argues that claim 1 has been amended to require that the perturbation is applied by modifying data values in the input array. Applicant argues that Xu does not teach applying perturbations by modifying data values but instead teaches matching backgrounds of multiple images taken by a camera, meanwhile Fong teaches that the entire input data array must be processed to obtain a result.
Examiner’s response to Argument 2
Applicant’s arguments ignore the merits of the combination of the two references and instead attacks the references in isolation of one another. 35 U.S.C. 103 rejections are not based on any one reference alone, but the combination of references. A person having ordinary skill in the art would recognize that a perturbed image and an unperturbed image (as taught by Fong) have substantial overlapping regions that are exactly the same. For example, see figure 1 of Fong
PNG
media_image1.png
177
375
media_image1.png
Greyscale
Meanwhile, Xu teaches identifying patches of an image where the pixels are the same/similar and reusing cached outputs for those patches, while not reusing portions that are dissimilar. Xu specifically states that this method was found to substantially speed up the processing of a similar image.
PNG
media_image2.png
153
324
media_image2.png
Greyscale
A person having ordinary skill in the art would recognize that since the perturbed vs. unperturbed images taught by Fong are mostly the same with some differences, one could use the method of Xu to obtain substantial processing speedups when using a perturbed image in a neural network. While the originally intended usage of Xu varies slightly from the instant application, it is obvious that the techniques it discloses would be applicable to any images that are substantially similar.
Furthermore, Xu considers that there is some overhead in the process of identifying regions of images that are similar. This is because, for the images that Xu is using, it is not already known a priori which portions of the images are the same or different. However, in Fong it already is known which blocks would contain perturbations and which ones do not. Even further speedups could be obtained by simply skipping the pixel matching step and instead reusing the portions that are already known to be the same while processing the portions that are already known to be different. A person having ordinary skill in the art would recognize that the benefits of Xu would be even more applicable to Fong than in the original use case that Xu proposes.
Thus, while indeed no reference on its own teaches the entire claim (i.e. Fong does not teach reusing neural network processing and Xu does not teach performing modifications to pixels of an image), their combination teaches the claim as a whole and there is overwhelming motivation to apply Xu to Fong to obtain significant speed improvements.
To further support the rejection, examiner points to MPEP 2143 I D. Applying a known technique (applying a neural network to only part of a data array to significantly improve speed) to a known method (the perturbations and neural network processing taught by Fong) ready for improvement (Fong does not teach some of the neural network techniques claimed, thus Fong’s neural network presumably operate at a normal speed ready for improvement) to yield predictable results (a person having ordinary skill in the art would recognize the proven speed improvements stated explicitly by Xu and that the neural network operates under similar conditions as Fong and thus would predictably improve the speed of the neural network models of Fong as well). A similar rationale applies to claim 10 rejected by Xu in view of Gould as well.
In Remarks pages 17-18, Argument 3
(Examiner summarizes Applicant’s arguments) Applicant argues that the references used to teach the dependent claims do not add anything to Fong to teach on the independent claim.
Examiner’s response to Argument 3
The rejection of the independent claims are maintained for the reasons provided above. The rejections of the dependent claims are also maintained.
In Remarks pages 18-21, Argument 4
(Examiner summarizes Applicant’s arguments) Applicant argues that claim 10 has been amended and thus the rejection under Fong in view of Xu and Jiao no longer applies to the claims as currently amended.
Examiner’s response to Argument 4
An updated search revealed new art that is pertinent to the claim as amended and thus a new rejection is issued for claim 10 under Fong in view of new reference Gould. Applicant’s arguments pertaining to the rejection of claim 10 are thus rendered moot. The rejection can be found below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 11-13, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Fong et al. “Interpretable Explanations of Black Boxes by Meaningful Perturbation” herein referred to as Fong in view of Xu et al “Accelerating Convolutional Neural Networks for Continuous Mobile Vision via Cache Reuse”, herein referred to as Xu.
Regarding Claim 1
Fong teaches:
A method of performing neural network processing in a data processing system, […], the method comprising: for an input data array comprising a set of data elements to be processed by a neural network, subjecting the input data array to neural network processing to generate a result of the neural network processing for the input data array
(page 3429 column 1 “introduction” paragraph 1) “Given the powerful but often opaque nature of modern black box predictors such as deep neural networks [4,5], there is a considerable interest in explaining and understanding predictors a-posteriori, after they have been learned.”; (page 3430 column 1 section 3 paragraph 1) “A black box is a map f : X → Y[*Examiner notes: neural network processing] from an input space X to an output space Y, typically obtained from an opaque learning process”; (page 3432 column 2 section 4.1 paragraph 1) “The aim of saliency is to identify which regions of an image x0[*Examiner notes: input data array comprising set of data elements] are used by the black box to produce the output value f(x0)[*Examiner notes: result of neural network processing]”
and applying a perturbation to a part but not all of the input data array by modifying data values of data elements of the part of the input data array to generate a perturbed version of the input data array, the perturbed version of the input data array thereby being made up of a perturbed part that differs from the input data array, and a non-perturbed part that is the same as the input data array
(page 3432 column 2 section 4.1 paragraph 1) “We can do so by observing how the value of f(x) changes as x is obtained “deleting” different regions R of x0[*Examiner notes: applying perturbation to part but not all of the input data array].”; [*Examiner notes: x0 is the same as x except for the deleted (perturbed) portion. Fong also teaches using other kinds of perturbations]
and performing the neural network processing using the perturbed version of the input data array comprising the data elements with data values modified by the perturbation to generate a result of the neural network processing for the perturbed version of the input data array
(page 3430 column 1 section 3 paragraph 1) “A black box is a map f : X → Y[*Examiner notes: neural network processing] from an input space X to an output space Y, typically obtained from an opaque learning process”; (page 3432 column 2 section 4.1 paragraph 1) “We can do so by observing how the value of f(x) changes[*Examiner notes: performing the neural network processing using the perturbed version] as x is obtained “deleting” different regions R of x0.”
and comparing the result of the neural network processing of the perturbed version of the input data array with the result of the neural network processing of the input data array without the perturbation, to determine whether the perturbation of the input data array has an effect on the result of the neural network processing of the perturbed version of the input data array relative to the result of the neural networking processing of the input data array without the perturbation.
(page 3432 column 2 section 4.1 paragraph 1) “We can do so by observing how the value of f(x)[*Examiner notes: processing of perturbed version] changes as x is obtained “deleting” different regions R of x0. For example, if f(x0) = +1 denotes a robin image, we expect that f(x) = +1[*Examiner notes: determine whether perturbation affects result] as well unless the choice of R deletes the robin from the image. Given that x is a perturbation of x0, this is a local explanation (sec. 3.2) and we expect the explanation to characterize the relationship between f and x0.”
Fong does not teach:
the data processing system comprising a processor operable to execute a neural network, and operable to store data relating to the neural network processing being performed by the processor to memory
wherein performing the neural network processing for the perturbed version of the input data array comprises: subjecting only some but not all of the perturbed version of the input data array comprising the data elements with data values modified by the perturbation to neural network processing when performing the neural network processing for the perturbed version of the input data array, based on the part of the input data array to which the perturbation has been applied
However, Xu teaches:
the data processing system comprising a processor operable to execute a neural network, and operable to store data relating to the neural network processing being performed by the processor to memory
(figure 4)
PNG
media_image3.png
160
307
media_image3.png
Greyscale
wherein performing the neural network processing for the perturbed version of the input data array comprises: subjecting only some but not all of the perturbed version of the input data array comprising the data elements with data values modified by the perturbation to neural network processing when performing the neural network processing for the perturbed version of the input data array, based on the part of the input data array to which the perturbation has been applied
[*Examiner notes: Xu discloses accelerating neural network computations when two similar images are passed through a neural network, which could be readily applied to the problem of a perturbed image (i.e. data values modified by perturbation) and a non-perturbed images as taught by Fong in combination]; (page 6 column 1 paragraph 1) “Figure 6 shows an output example of applying our matching algorithm on two consecutively captured images[*Examiner notes: corresponds to perturbed version of input data array]. As observed, the second frame image is different from the first one in two aspects. First, the camera is moving, so the overall background also moves in certain direction. This movement is captured in Step 3 by looking into the movement of each small block and combining them together. Second, the objects in sight are also moving. Those moved objects (regions) should be detected and marked as non-reusable. This detection is achieved in Step 4.”; [*Examiner notes: The second image is perturbed from the first because it was taken immediately after. The second image is similar, but not exactly the same as the first.] (page 1 abstract) “To cache and reuse the computations of the similar image regions which are consecutively captured by mobile devices[*Examiner notes: subjecting some but not all of the perturbed version to neural network processing based on the part of the input to which perturbation has been applied], CNNCache leverages two novel techniques: an image matching algorithm that quickly identifies similar image regions between images, and a cache-aware CNN inference engine that propagates the reusable regions through varied layers and reuses the computation results at layer granularity.”
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the neural networks and perturbations taught by Fong with the subjecting of only some but not all of the perturbed input to neural network processing taught by Xu because (Xu page 1 abstract) “The results show that CNNCache can accelerate the execution of CNN models by 20.2% on average and up to 47.1% under certain scenarios”
Regarding Claim 2
Fong in view of Xu teaches:
The method of claim 1
(see rejection of claim 1)
And Xu further teaches
comprising storing some or all of an output of neural network processing for a layer or layers of the neural network processing when processing the input data array, and reusing the output of the neural network processing for the layer or layers of the neural network processing stored from the processing of the input data array when performing the neural network processing for the perturbed version of the input data array
(page 1 abstract) “To cache and reuse the computations of the similar image regions which are consecutively captured by mobile devices, CNNCache leverages two novel techniques: an image matching algorithm that quickly identifies similar image regions between images, and a cache-aware CNN inference engine that propagates the reusable regions through varied layers and reuses the computation results at layer granularity.”
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to combine Fong with Xu for the same reasons given in claim 1 above.
Regarding Claim 3
Fong in view of Xu teaches:
The method of claim 2,
(see rejection of claim 2)
And Xu further teaches:
comprising reusing the output of the neural network processing for the layer or layers of the neural network processing stored from the processing of the input data array as part of an input for a fully connected layer or layers when performing neural network processing for the perturbed version of the input data array
(page 3 column 1 section III B paragraph 1) “CNNCache is based on a key observation that consecutively captured images often have substantial overlapped (similar) regions. The reason is that mobile devices, e.g., smartphones and head-mounted devices, are in slow motion or even held still especially when users are using these devices for vision tasks such as augmenting reality [42], [20]. Thus, CNNCache tries to cache the intermediate computation results of previous frames[*Examiner notes: output stored from the processing of input data array], and reuse the results of unchanged regions to accelerate the processing of current frame.”; (page 4 column 1 paragraph 1) “In other words, there are still plenty of room (88.5%) to be improved via reusing the cache of layers before fully-connected layer[*Examiner notes: fully connected layer]”
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to combine Fong with Xu for the same reasons given in claim 1 above.
Regarding Claim 11
Fong teaches:
subject an input data array comprising a set of data elements to neural network processing to generate a result of the neural network processing for the input data array
(page 3429 column 1 “introduction” paragraph 1) “Given the powerful but often opaque nature of modern black box predictors such as deep neural networks [4,5], there is a considerable interest in explaining and understanding predictors a-posteriori, after they have been learned.”; (page 3430 column 1 section 3 paragraph 1) “A black box is a map f : X → Y[*Examiner notes: neural network processing] from an input space X to an output space Y, typically obtained from an opaque learning process”; (page 3432 column 2 section 4.1 paragraph 1) “The aim of saliency is to identify which regions of an image x0[*Examiner notes: input data array] are used by the black box to produce the output value f(x0)[*Examiner notes: result of neural network processing]”
and to subject a perturbed version of the input data array to the neural network processing to generate a result of the neural network processing for the perturbed version of the input data array
(page 3430 column 1 section 3 paragraph 1) “A black box is a map f : X → Y[*Examiner notes: neural network processing] from an input space X to an output space Y, typically obtained from an opaque learning process”; (page 3432 column 2 section 4.1 paragraph 1) “We can do so by observing how the value of f(x) changes[*Examiner notes: performing the neural network processing using the so-perturbed version] as x is obtained “deleting” different regions R of x0.”
the perturbed version of the input data array comprising a version of the input data array in which a perturbation has been applied to a part but not all of the input data array by modifying data values of data elements of the part of the input data array, the perturbed version of the input data array thereby being made up of a perturbed part that differs from the input data array, and anon-perturbed part that is the same as the input data array;;
(page 3432 column 2 section 4.1 paragraph 1) “We can do so by observing how the value of f(x) changes as x is obtained “deleting” different regions R of x0[*Examiner notes: applying perturbation to part but not all of the input data array].”; [*Examiner notes: x0 is the same as x except for the deleted (perturbed) portion. Fong also teaches using other kinds of perturbations]
the data processing system further comprising: a processing circuit configured to compare the result of the neural network processing of the perturbed version of the input data array with the result of the neural network processing of the input data array without the perturbation, to determine whether the perturbation of the input data array has an effect on the result of the neural network processing of the perturbed version of the input data array relative to the result of the neural networking processing of the input data array without the perturbation..
(page 3432 column 2 section 4.1 paragraph 1) “We can do so by observing how the value of f(x)[*Examiner notes: processing of perturbed version] changes as x is obtained “deleting” different regions R of x0. For example, if f(x0) = +1 denotes a robin image, we expect that f(x) = +1[*Examiner notes: determine whether perturbation affects result] as well unless the choice of R deletes the robin from the image. Given that x is a perturbation of x0, this is a local explanation (sec. 3.2) and we expect the explanation to characterize the relationship between f and x0.”
Fong does not explicitly teach:
A data processing system, the data processing system comprising: a processor operable to execute a neural network and operable to store data relating to the neural network processing being performed by the processor to memory; the data processing system further comprising a processing circuit configured to cause the processor to:
wherein performing the neural network processing for the perturbed version of the input data array comprises: subjecting only some but not all of the perturbed version of the input data array comprising the data elements with data values modified by the perturbation to neural network processing when performing the neural network processing for the perturbed version of the input data array, based on the part of the input data array to which the perturbation has been applied;
However, Xu teaches:
A data processing system, the data processing system comprising: a processor operable to execute a neural network and operable to store data relating to the neural network processing being performed by the processor to memory; the data processing system further comprising a processing circuit configured to cause the processor to:
(figure 4)
PNG
media_image3.png
160
307
media_image3.png
Greyscale
wherein performing the neural network processing for the perturbed version of the input data array comprises: subjecting only some but not all of the perturbed version of the input data array comprising the data elements with data values modified by the perturbation to neural network processing when performing the neural network processing for the perturbed version of the input data array, based on the part of the input data array to which the perturbation has been applied
[*Examiner notes: Xu discloses accelerating neural network computations when two similar images are passed through a neural network, which could be readily applied to the problem of a perturbed image (i.e. data values modified by perturbation) and a non-perturbed images as taught by Fong in combination]; (page 6 column 1 paragraph 1) “Figure 6 shows an output example of applying our matching algorithm on two consecutively captured images[*Examiner notes: corresponds to perturbed version of input data array]. As observed, the second frame image is different from the first one in two aspects. First, the camera is moving, so the overall background also moves in certain direction. This movement is captured in Step 3 by looking into the movement of each small block and combining them together. Second, the objects in sight are also moving. Those moved objects (regions) should be detected and marked as non-reusable. This detection is achieved in Step 4.”; [*Examiner notes: The second image is perturbed from the first because it was taken immediately after. The second image is similar, but not exactly the same as the first.] (page 1 abstract) “To cache and reuse the computations of the similar image regions which are consecutively captured by mobile devices[*Examiner notes: subjecting some but not all of the perturbed version to neural network processing based on the part of the input to which perturbation has been applied], CNNCache leverages two novel techniques: an image matching algorithm that quickly identifies similar image regions between images, and a cache-aware CNN inference engine that propagates the reusable regions through varied layers and reuses the computation results at layer granularity.”
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the neural networks and perturbations taught by Fong with the subjecting of only some but not all of the perturbed input to neural network processing taught by Xu because (Xu page 1 abstract) “The results show that CNNCache can accelerate the execution of CNN models by 20.2% on average and up to 47.1% under certain scenarios”
Regarding Claim 12
Fong in view of Xu teaches:
The system of claim 11
(see claim 11)
And Xu further teaches:
wherein the processing circuit is configured to store some or all of an output of neural network processing for a layer or layers of the neural network processing when processing the input data array, and reuse the output of the neural network processing for the layer or layers of the neural network processing stored from the processing of the input data array when performing the neural network processing for the perturbed version of the input data array
(page 1 abstract) “To cache and reuse the computations of the similar image regions which are consecutively captured by mobile devices, CNNCache leverages two novel techniques: an image matching algorithm that quickly identifies similar image regions between images, and a cache-aware CNN inference engine that propagates the reusable regions through varied layers and reuses the computation results at layer granularity.”
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to combine Fong with Xu for the same reasons given in claim 11 above.
Regarding Claim 13
Fong in view of Xu teaches:
The system of claim 12
(see rejection of claim 12)
And Xu further teaches:
wherein the processing circuit is configured to reuse the output of the neural network processing for the layer or layers of the neural network processing stored from the processing of the input data array as part of an input for a fully connected layer or layers when performing neural network processing for the perturbed version of the input data array
(page 3 column 1 section III B paragraph 1) “CNNCache is based on a key observation that consecutively captured images often have substantial overlapped (similar) regions. The reason is that mobile devices, e.g., smartphones and head-mounted devices, are in slow motion or even held still especially when users are using these devices for vision tasks such as augmenting reality [42], [20]. Thus, CNNCache tries to cache the intermediate computation results of previous frames[*Examiner notes: output stored from the processing of input data array], and reuse the results of unchanged regions to accelerate the processing of current frame.”; (page 4 column 1 paragraph 1) “In other words, there are still plenty of room (88.5%) to be improved via reusing the cache of layers before fully-connected layer[*Examiner notes: fully connected layer]”
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to combine Fong with Xu for the same reasons given in claim 11 above.
Regarding Claim 20
Fong teaches:
the method comprising: for an input data array comprising a set of data elements to be processed by a neural network, subjecting the input data array to neural network processing to generate a result of the neural network processing for the input data array
(page 3429 column 1 “introduction” paragraph 1) “Given the powerful but often opaque nature of modern black box predictors such as deep neural networks [4,5], there is a considerable interest in explaining and understanding predictors a-posteriori, after they have been learned.”; (page 3430 column 1 section 3 paragraph 1) “A black box is a map f : X → Y[*Examiner notes: neural network processing] from an input space X to an output space Y, typically obtained from an opaque learning process”; (page 3432 column 2 section 4.1 paragraph 1) “The aim of saliency is to identify which regions of an image x0[*Examiner notes: input data array] are used by the black box to produce the output value f(x0)[*Examiner notes: result of neural network processing]”
and applying a perturbation to a part but not all of the input data array by modifying data values of data elements of the part of the input data array to generate a perturbed version of the input data array, the perturbed version of the input data array thereby being made up of a perturbed part that differs from the input data array, and anon-perturbed part that is the same as the input data array,
(page 3432 column 2 section 4.1 paragraph 1) “We can do so by observing how the value of f(x) changes as x is obtained “deleting” different regions R of x0[*Examiner notes: applying perturbation to part but not all of the input data array].”; [*Examiner notes: x0 is the same as x except for the deleted (perturbed) portion. Fong also teaches using other kinds of perturbations]
and performing the neural network processing using the perturbed version of the input data array comprising the data elements with data values modified by the perturbation to generate a result of the neural network processing for the perturbed version of the input data array
(page 3430 column 1 section 3 paragraph 1) “A black box is a map f : X → Y[*Examiner notes: neural network processing] from an input space X to an output space Y, typically obtained from an opaque learning process”; (page 3432 column 2 section 4.1 paragraph 1) “We can do so by observing how the value of f(x) changes[*Examiner notes: performing the neural network processing using the so-perturbed version] as x is obtained “deleting” different regions R of x0.”
and comparing the result of the neural network processing of the perturbed version of the input data array with the result of the neural network processing of the input data array without the perturbation, to determine whether the perturbation of the input data array has an effect on the result of the neural network processing of the perturbed version of the input data array relative to the result of the neural networking processing of the input data array without the perturbation.
(page 3432 column 2 section 4.1 paragraph 1) “We can do so by observing how the value of f(x)[*Examiner notes: processing of perturbed version] changes as x is obtained “deleting” different regions R of x0. For example, if f(x0) = +1 denotes a robin image, we expect that f(x) = +1[*Examiner notes: determine whether perturbation affects result] as well unless the choice of R deletes the robin from the image. Given that x is a perturbation of x0, this is a local explanation (sec. 3.2) and we expect the explanation to characterize the relationship between f and x0.”
Fong does not teach:
A non-transitory computer readable storage medium storing computer software code which when executing on at least one processor performs a method of performing neural network processing in a data processing system, the data processing system comprising a processor operable to execute a neural network, and operable to store data relating to the neural network processing being performed by the processor to memory
wherein performing the neural network processing for the perturbed version of the input data array comprises: subjecting only some but not all of the perturbed version of the input data array comprising the data elements with data values modified by the perturbation to neural network processing when performing the neural network processing for the perturbed version of the input data array, based on the part of the input data array to which the perturbation has been applied
However, Xu teaches:
A non-transitory computer readable storage medium storing computer software code which when executing on at least one processor performs a method of performing neural network processing in a data processing system, the data processing system comprising a processor operable to execute a neural network, and operable to store data relating to the neural network processing being performed by the processor to memory
(figure 4)
PNG
media_image3.png
160
307
media_image3.png
Greyscale
wherein performing the neural network processing for the perturbed version of the input data array comprises: subjecting only some but not all of the perturbed version of the input data array comprising the data elements with data values modified by the perturbation to neural network processing when performing the neural network processing for the perturbed version of the input data array, based on the part of the input data array to which the perturbation has been applied
[*Examiner notes: Xu discloses accelerating neural network computations when two similar images are passed through a neural network, which could be readily applied to the problem of a perturbed image (i.e. data values modified by perturbation) and a non-perturbed images as taught by Fong in combination]; (page 6 column 1 paragraph 1) “Figure 6 shows an output example of applying our matching algorithm on two consecutively captured images[*Examiner notes: corresponds to perturbed version of input data array]. As observed, the second frame image is different from the first one in two aspects. First, the camera is moving, so the overall background also moves in certain direction. This movement is captured in Step 3 by looking into the movement of each small block and combining them together. Second, the objects in sight are also moving. Those moved objects (regions) should be detected and marked as non-reusable. This detection is achieved in Step 4.”; [*Examiner notes: The second image is perturbed from the first because it was taken immediately after. The second image is similar, but not exactly the same as the first.] (page 1 abstract) “To cache and reuse the computations of the similar image regions which are consecutively captured by mobile devices[*Examiner notes: subjecting some but not all of the perturbed version to neural network processing based on the part of the input to which perturbation has been applied], CNNCache leverages two novel techniques: an image matching algorithm that quickly identifies similar image regions between images, and a cache-aware CNN inference engine that propagates the reusable regions through varied layers and reuses the computation results at layer granularity.”
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the neural networks and perturbations taught by Fong with the subjecting of only some but not all of the perturbed input to neural network processing taught by Xu because (Xu page 1 abstract) “The results show that CNNCache can accelerate the execution of CNN models by 20.2% on average and up to 47.1% under certain scenarios”
Claims 4 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Fong in view of Xu and further in view of NPL reference Riera Villanueva “Low-Power Accelerators for Cognitive Computing” herein referred to as Riera.
Regarding Claim 4
Fong in view of Xu teaches:
The method of claim 1
(see rejection of claim 1)
And Xu further teaches:
comprising: storing an output of neural network processing for a layer or layers of the neural network processing when processing the input data array
(page 1 abstract) “To cache and reuse the computations of the similar image regions which are consecutively captured by mobile devices, CNNCache leverages two novel techniques: an image matching algorithm that quickly identifies similar image regions between images, and a cache-aware CNN inference engine that propagates the reusable regions through varied layers and reuses the computation results at layer granularity.”
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to combine Fong with Xu for the same reasons given in claim 1 above.
Xu does not explicitly teach:
comparing an output of the neural network processing for a layer of the neural network processing when processing the perturbed version of the input data array to the stored output of the neural network processing of that layer of the neural network processing when processing the input data array;
and determining whether to continue the neural network processing for a part or parts of the perturbed version of the input data array on the basis of the comparison.
However, Riera teaches:
comparing an output of the neural network processing for a layer of the neural network processing when processing the perturbed version of the input data array to the stored output of the neural network processing of that layer of the neural network processing when processing the input data array;
and determining whether to continue the neural network processing for a part or parts of the perturbed version of the input data array on the basis of the comparison.
(page 64 section 4.2.4) “Therefore, RNNs only require extra storage for the inputs/outputs of one layer, whereas MLPs and CNNs require extra storage for all the layers where the computation reuse technique is applied. In other words, temporal locality of the redundant computations is higher in RNNs. Second, the four gates (four FC layers) in one LSTM cell share the same inputs. Hence, we only compare the inputs once with the previous values and, in case an input remains unmodified, computations and memory accesses are avoided in the four gates.”; [*Examiner notes: Comparing inputs with previous values is the same as comparing the outputs of layers because an input is an output of an input layer]
Fong, Xu, Riera, and the instant application are analogous because they are all directed to machine learning.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the neural networks and perturbations of Fong in view of Xu with the determining whether to continue the neural network processing taught by Riera because (Riera page 64 section 4.2.4) “Hence, we only compare the inputs once with the previous values and, in case an input remains unmodified, computations and memory accesses are avoided in all the gates”. That is, if processing is halted then less computations would be performed, leading to better efficiency of the neural network.
Regarding Claim 14
Fong in view of Xu teaches:
The system of claim 11
(see rejection of claim 11)
And Xu further teaches:
wherein the processing circuit is configured to: store an output of neural network processing for a layer or layers of the neural network processing when processing the input data array;
(page 1 abstract) “To cache and reuse the computations of the similar image regions which are consecutively captured by mobile devices, CNNCache leverages two novel techniques: an image matching algorithm that quickly identifies similar image regions between images, and a cache-aware CNN inference engine that propagates the reusable regions through varied layers and reuses the computation results at layer granularity.”
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to combine Fong with Xu for the same reasons given in claim 11 above.
Fong in view of Xu does not teach:
compare an output of the neural network processing for a layer of the neural network processing when processing the perturbed version of the input data array to the stored output of the neural network processing of that layer when processing the input data array;
and determine whether to continue the neural network processing for a part or parts of the perturbed version of the input data array on the basis of the comparison.
However, Riera teaches:
compare an output of the neural network processing for a layer of the neural network processing when processing the perturbed version of the input data array to the stored output of the neural network processing of that layer when processing the input data array;
and determine whether to continue the neural network processing for a part or parts of the perturbed version of the input data array on the basis of the comparison.
(page 64 section 4.2.4) “Therefore, RNNs only require extra storage for the inputs/outputs of one layer, whereas MLPs and CNNs require extra storage for all the layers where the computation reuse technique is applied. In other words, temporal locality of the redundant computations is higher in RNNs. Second, the four gates (four FC layers) in one LSTM cell share the same inputs. Hence, we only compare the inputs once with the previous values and, in case an input remains unmodified, computations and memory accesses are avoided in the four gates.”; [*Examiner notes: Comparing inputs with previous values is the same as comparing the outputs of layers because an input is an output of an input layer]
Fong, Xu, Riera, and the instant application are analogous because they are all directed to machine learning.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the neural networks and perturbations of Fong in view of Xu with the determining whether to continue the neural network processing taught by Riera because (Riera page 64 section 4.2.4) “Hence, we only compare the inputs once with the previous values and, in case an input remains unmodified, computations and memory accesses are avoided in all the gates”. That is, if processing is halted then less computations would be performed, leading to better efficiency of the neural network.
Claims 5 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Fong in view of Xu, Riera, and further in view of Foreign patent reference Velic et al. (EP 3744410 A1) herein referred to as Velic.
Regarding Claim 5
Fong in view of Xu in view of Riera teaches:
The method of claim 4
(see rejection of claim 4)
Fong in view of Xu and Rierra does not teach:
wherein the layer or layers of the neural network processing for which an output is stored when the processing the input data array comprises a pooling layer, and the output of the neural network processing for that pooling layer when processing the perturbed version of the input data array is compared to the stored output of the neural network processing for that pooling layer when processing the input data array
However, Velic teaches:
wherein the layer or layers of the neural network processing for which an output is stored when the processing the input data array comprises a pooling layer, and the output of the neural network processing for that pooling layer when processing the perturbed version of the input data array is compared to the stored output of the neural network processing for that pooling layer when processing the input data array
(column 39 line 16 “embodiment 19”) “wherein the recognition system is configured to estimate the one or more additional attributes of the real-world toy object depicted in the captured image by comparing an output of the convolutional stage of the trained convolutional classification model produced by the trained convolutional classification model based on the captured image with one or more of the stored reference representations associated with the predicted object identifier.”; (column 40 line 23 “embodiment 25”) “wherein the convolutional classification model comprises one or more pooling layers”
Fong, Xu, Rierra, Velic, and the instant application are analogous because they are all directed to machine learning.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the neural networks and perturbations of Fong in view of Xu and Rierra with the pooling layer of Velic because (Velic paragraph [0035]) “After obtaining said feature maps with the convolution operation, the resulting images may be subsampled (or "pooled") to reduce the computational requirement for further processing.”
Regarding Claim 15
Fong in view of Xu and Riera teaches:
The system of claim 14
(see rejection of claim 14)
Fong in view of Xu and Riera does not teach:
wherein the layer or layers of the neural network processing for which an output is stored when the processing the input data array comprises a pooling layer, and the output of the neural network processing for that pooling layer when processing the perturbed version of the input data array is compared to the stored output of the neural network processing of that pooling layer when processing the input data array
However, Velic teaches:
wherein the layer or layers of the neural network processing for which an output is stored when the processing the input data array comprises a pooling layer, and the output of the neural network processing for that pooling layer when processing the perturbed version of the input data array is compared to the stored output of the neural network processing of that pooling layer when processing the input data array
(column 39 line 16 “embodiment 19”) “wherein the recognition system is configured to estimate the one or more additional attributes of the real-world toy object depicted in the captured image by comparing an output of the convolutional stage of the trained convolutional classification model produced by the trained convolutional classification model based on the captured image with one or more of the stored reference representations associated with the predicted object identifier.”; (column 40 line 23 “embodiment 25”) “wherein the convolutional classification model comprises one or more pooling layers”
Fong, Xu, Rierra, Velic, and the instant application are analogous because they are all directed to machine learning.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the neural networks and perturbations of Fong in view of Xu and Rierra with the pooling layer of Velic because (Velic paragraph [0035]) “After obtaining said feature maps with the convolution operation, the resulting images may be subsampled (or "pooled") to reduce the computational requirement for further processing.”
Claims 6 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Fong in view of Xu and Shrivastava et al. (US Patent no. US 10984272 B1) herein referred to as Shrivastava.
Regarding Claim 6
Fong in view of Xu teaches:
The method of claim 1
(see rejection of claim 1)
Fong in view of Xu does not teach:
wherein neural network processing for the input data array is performed on a block-by-block basis, such that the input data array is divided into and processed as one or more blocks, and the perturbation is applied to a region of the input data array; such that: the perturbed region is confined to a single block of said one or more blocks or comprises an integer number of whole blocks; and/or at least one boundary of the perturbed region aligns with at least one boundary between said one or more blocks.
However, Shrivastava teaches:
wherein neural network processing for the input data array is performed on a block-by-block basis, such that the input data array is divided into and processed as one or more blocks, and the perturbation is applied to a region of the input data array; such that: the perturbed region is confined to a single block of said one or more blocks or comprises an integer number of whole blocks; and/or at least one boundary of the perturbed region aligns with at least one boundary between said one or more blocks.
(column 9 line 43) “At 402, an input image is divided into source patches[*Examiner notes: input array divided into and processed as one or more blocks]. The source patches may be of the same or different sizes. Some or all of the source patches may be overlapping. Alternatively, the source patches may be non-overlapping. At least one of the source patches within the input image includes a noise-based perturbation[*Examiner notes: integer number of whole blocks]”; (page 10 line 26) “At 408, the denoised image is output to a neural network for classification.”; Figure 4
Fong, Xu, Shrivastava, and the instant application are analogous because they are all directed to machine learning.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the neural networks and perturbations as taught by Fong in view of Xu with the block-by-block processing as taught by Shrivasta because (column 3 line 33) “The systems and techniques of this disclosure improve the performance of a neural network. In particular, the neural network defenses presented in the implementations of this disclosure are non-differentiable, thereby making it non-trivial for an adversary to find gradient-based attacks. In addition, the implementations of this disclosure do not require a neural network to be fine-tuned using adversarial examples, thereby increasing robustness relative to unknown attacks. The neural network defenses presented in the implementations of this disclosure have been shown to yield benefits in black box, grey-box, and white-box settings.”
Regarding Claim 16
Fong in view of Xu teaches:
The system of claim 11
(see rejection of claim 11)
Fong in view of Xu does not teach:
wherein the processing circuit is configured to cause neural network processing for the input data array to be performed on a block-by-block basis, such that the input data array is divided into and processed as one or more blocks, and the perturbation has been applied to a region of the input data array; such that: the perturbed region is confined to a single block of said one or more blocks or comprises an integer number of whole blocks; and/or at least one boundary of the perturbed region aligns with at least one boundary between said one or more blocks
Accelerating Convolutional Neural Networks for
Continuous Mobile Vision via Cache Reuse
However, Shrivastava teaches:
wherein the processing circuit is configured to cause neural network processing for the input data array to be performed on a block-by-block basis, such that the input data array is divided into and processed as one or more blocks, and the perturbation has been applied to a region of the input data array; such that: the perturbed region is confined to a single block of said one or more blocks or comprises an integer number of whole blocks; and/or at least one boundary of the perturbed region aligns with at least one boundary between said one or more blocks.
(column 9 line 43) “At 402, an input image is divided into source patches[*Examiner notes: input array divided into and processed as one or more blocks]. The source patches may be of the same or different sizes. Some or all of the source patches may be overlapping. Alternatively, the source patches may be non-overlapping. At least one of the source patches within the input image includes a noise-based perturbation[*Examiner notes: integer number of whole blocks]”; (page 10 line 26) “At 408, the denoised image is output to a neural network for classification.”; Figure 4
Fong, Xu, Shrivastava, and the instant application are analogous because they are all directed to machine learning.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the neural networks and perturbations as taught by Fong in view of Xu with the block-by-block processing as taught by Shrivasta because (column 3 line 33) “The systems and techniques of this disclosure improve the performance of a neural network. In particular, the neural network defenses presented in the implementations of this disclosure are non-differentiable, thereby making it non-trivial for an adversary to find gradient-based attacks. In addition, the implementations of this disclosure do not require a neural network to be fine-tuned using adversarial examples, thereby increasing robustness relative to unknown attacks. The neural network defenses presented in the implementations of this disclosure have been shown to yield benefits in black box, grey-box, and white-box settings.”
Claims 7 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Fong in view of Xu, and further in view of NPL reference Li et al. “Block Convolution: Towards Memory-Efficient Inference of Large-Scale CNNs on FPGA” herein referred to as Li.
Regarding Claim 7
Fong in view of Xu teaches
The method of claim 1
(see rejection of claim 1)
Fong in view of Xu does not teach:
wherein the perturbation is applied to a region of the input data array, and the size of the perturbed region is based on a memory transaction size of the data processing system
However, Li teaches:
wherein the perturbation is applied to a region of the input data array, and the size of the perturbed region is based on a memory transaction size of the data processing system
(page 8 column 2 second to last paragraph) “Suppose a single channel feature map of size 128×128 , when square blocking is utilized, it can be partitioned into the size of 128×128,64×64,32×32,16×16 , etc. If the on-chip memory capacity is 128×100 , the largest block that can fit on chip, in this scenario, is the one of size 64×64 ; thus, the memory utilization is only 40.96%. However, if we use rectangular blocking, such as 128×64 , the memory utilization can be easily doubled.”
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the neural networks and perturbations of Fong in view of Xu with the size of the perturbed region based on memory transaction size because (Li page 1 abstract) “The fundamental idea of block convolution is to eliminate the dependency of feature map tiles in the spatial dimension when spatial tiling is used, which is realized by splitting a feature map into independent blocks so that convolution can be performed separately on individual blocks. […] We also showcase two CNN accelerators via algorithm/hardware co-design based on block convolution on memory-limited FPGAs, and evaluation shows that both accelerators substantially outperform the baseline without off-chip transfer of intermediate feature maps.”
Regarding Claim 17
Fong in view of Xu teaches:
The system of claim 11
(see rejection of claim 11)
Fong in view of Xu does not teach:
wherein the perturbation has been applied to a region of the input data array, and the size of the perturbed region is based on a memory transaction size of the data processing system.
However, Li teaches:
wherein the perturbation has been applied to a region of the input data array, and the size of the perturbed region is based on a memory transaction size of the data processing system.
(page 8 column 2 second to last paragraph) “Suppose a single channel feature map of size 128×128 , when square blocking is utilized, it can be partitioned into the size of 128×128,64×64,32×32,16×16 , etc. If the on-chip memory capacity is 128×100 , the largest block that can fit on chip, in this scenario, is the one of size 64×64 ; thus, the memory utilization is only 40.96%. However, if we use rectangular blocking, such as 128×64 , the memory utilization can be easily doubled.”
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the neural networks and perturbations of Fong in view of Xu with the size of the perturbed region based on memory transaction size because (page 1 abstract) “The fundamental idea of block convolution is to eliminate the dependency of feature map tiles in the spatial dimension when spatial tiling is used, which is realized by splitting a feature map into independent blocks so that convolution can be performed separately on individual blocks. […] We also showcase two CNN accelerators via algorithm/hardware co-design based on block convolution on memory-limited FPGAs, and evaluation shows that both accelerators substantially outperform the baseline without off-chip transfer of intermediate feature maps.”
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Fong in view of Gould et al. (Patent no. US10460175B1) herein referred to as Gould.
Regarding Claim 10
Fong teaches:
A method of performing neural network processing in a data processing system, […], the method comprising: for an input data array to be processed by a neural network, performing neural network processing using the input data array to generate a result of the neural network processing for the input data array
(page 3429 column 1 “introduction” paragraph 1) “Given the powerful but often opaque nature of modern black box predictors such as deep neural networks [4,5], there is a considerable interest in explaining and understanding predictors a-posteriori, after they have been learned.”; (page 3430 column 1 section 3 paragraph 1) “A black box is a map f : X → Y[*Examiner notes: neural network processing] from an input space X to an output space Y, typically obtained from an opaque learning process”; (page 3432 column 2 section 4.1 paragraph 1) “The aim of saliency is to identify which regions of an image x0[*Examiner notes: input data array] are used by the black box to produce the output value f(x0)[*Examiner notes: result of neural network processing]”
and applying a perturbation to a part but not all of the input data array to generate a perturbed version of the input data array, the perturbed version of the input data array thereby being made up of a perturbed part that differs from the input data array, and anon-perturbed part that is the same as the input data array
(page 3432 column 2 section 4.1 paragraph 1) “We can do so by observing how the value of f(x) changes as x is obtained “deleting” different regions R of x0[*Examiner notes: applying perturbation to part but not all of the input data array].”; [*Examiner notes: x0 is the same as x except for the deleted (perturbed) portion. Fong also teaches using other kinds of perturbations]
and performing neural network processing using the so-perturbed version of the input data array to generate a result of the neural network processing for the perturbed version of the input data array;
(page 3430 column 1 section 3 paragraph 1) “A black box is a map f : X → Y[*Examiner notes: neural network processing] from an input space X to an output space Y, typically obtained from an opaque learning process”; (page 3432 column 2 section 4.1 paragraph 1) “We can do so by observing how the value of f(x) changes[*Examiner notes: performing the neural network processing using the so-perturbed version] as x is obtained “deleting” different regions R of x0.”
Fong does not teach:
the data processing system comprising a processor operable to execute a neural network, and operable to store data relating to the neural network processing being performed by the processor to memory
the performing neural network processing using the input data array comprising storing an output of the neural network processing for an intermediate layer or layers of the neural network processing when processing the input data array;
wherein performing the neural network processing for the perturbed version of the input data array comprises: comparing an output of neural network processing for an intermediate layer of the neural network processing when processing the perturbed version of the input data array to the output of the neural network processing of that intermediate layer when processing the input data array without the perturbation that was stored
and determining whether or not to continue the neural network processing for a part or parts of the perturbed version of the input data array through remaining layers of the neural network on the basis of the comparison
including: determining to continue the neural network processing for the part or parts of the perturbed version of the input data array through the remaining layers of the neural network, when the output of neural network processing for the intermediate layer of the neural network processing for the perturbed version of the input data array is determined to not match or not be sufficiently similar to the output of the neural network processing of that intermediate layer when processing the input data array without the perturbation
and determining to terminate the neural network processing for the part or parts of the perturbed version of the input data array without processing the remaining layers of the neural network, when the output of neural network processing for the intermediate layer of the neural network processing for the perturbed version of the input data array is determined to match or be sufficiently similar to the output of the neural network processing of that intermediate layer when processing the input data array without the perturbation.
However, Gould teaches:
the data processing system comprising a processor operable to execute a neural network, and operable to store data relating to the neural network processing being performed by the processor to memory
(column 4 line 31) “In one embodiment, the neural network includes a detection module 150, a processing device 138 and a memory 136 configured to execute and store instructions associated with the functionality of the various components, services, and modules of the neural network 120, as described in greater detail below in connection with FIGS. 2-8.”
the performing neural network processing using the input data array comprising storing an output of the neural network processing for an intermediate layer or layers of the neural network processing when processing the input data array;
(column 4 line 63) “In one embodiment, the computation set generator 124 processes a frame of the video 110 (e.g., Frame N) using the multiple layers of the neural network. In one embodiment, the output of each layer of processing of the frame is a matrix (or computation set) that is stored in the computation set storage 130.”
wherein performing the neural network processing for the perturbed version of the input data array comprises: comparing an output of neural network processing for an intermediate layer of the neural network processing when processing the perturbed version of the input data array to the output of the neural network processing of that intermediate layer when processing the input data array without the perturbation that was stored
(column 8 line 50) “In block 510, a neural network applies a first layer of processing to a first frame of a video to generate and store a first matrix, as detailed above. In block 520, the same layer of the neural network is applied to a second frame of the video to generate and store a second matrix. In block 530, the two matrices (e.g., the first matrix and the second matrix) are compared.”
and determining whether or not to continue the neural network processing for a part or parts of the perturbed version of the input data array through remaining layers of the neural network on the basis of the comparison including: determining to continue the neural network processing for the part or parts of the perturbed version of the input data array through the remaining layers of the neural network, when the output of neural network processing for the intermediate layer of the neural network processing for the perturbed version of the input data array is determined to not match or not be sufficiently similar to the output of the neural network processing of that intermediate layer when processing the input data array without the perturbation
(column 10 line 1) “In one embodiment, for example, a first portion of the matrices corresponding to the results of the processing by a layer (e.g., L2) of contiguous frames (N and N+1) may match, while a second portion of the matrices does not match. In one embodiment, processing continues with respect to the portion that is different (i.e., the non-matching portion), while the processing is terminated with respect to the portion that is the same (i.e., the matching portion) because it is known how the matching portion is going to map onto the next layer (e.g., Layer 3) of the network.”; (Figure 5)
PNG
media_image4.png
528
416
media_image4.png
Greyscale
and determining to terminate the neural network processing for the part or parts of the perturbed version of the input data array without processing the remaining layers of the neural network, when the output of neural network processing for the intermediate layer of the neural network processing for the perturbed version of the input data array is determined to match or be sufficiently similar to the output of the neural network processing of that intermediate layer when processing the input data array without the perturbation.
[*Examiner notes: When applying the method for consecutive images (i.e. 2 similar images) to the perturbed and unperturbed images of Fong, neural network processing is terminated when the processing of the perturbed version matches the processing of the unperturbed version and continue processing (for at least a part of the unperturbed version) when there is no match or only a partial match. This is consistent for e.g. paragraphs 121-123 of the Specification]; () “In one embodiment, the computation set comparison module 126 may retrieve the results of the same layer processing of a previous frame (e.g., Frame N) and compare to those results (e.g., 1,000 labels corresponding to L2 (224×224×64) and Frame N (e.g., M1 (Frame N, L2)) to M2. In one embodiment, if the difference metric is less than a threshold value and it is determined that the entire Frame N and N+1 are the same or substantially the same, then the processing of Frame N+1 can “skip” ahead and apply a final result (e.g., a feature map) of Frame N to Frame N+1. In one embodiment, if the matrices match, the feature map for Frame N is associated with Frame N+1 (e.g., a final classification represented by a feature map for Frame N is also applied to Frame N+1). For example, if the processing of Frame N generates a feature map indicating that Frame N includes a “dog”, then the same final classification is applied to Frame N+1 (e.g., Frame N+1 also has a same feature map indicating a classification of a “dog”).”; (Fig. 4)
PNG
media_image5.png
541
365
media_image5.png
Greyscale
Fong, Gould, and the instant application are analogous because they are all directed to machine learning.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the neural networks and perturbations as taught by Fong by re-using calculations at layers from similar image frames because (Gould column 1 line 24) “As such, there are instances when a large amount of processing is performed with regard to two or more contiguous frames of video (i.e., processing each frame through the entire neural network), even though only a small amount of the data has changed from one frame to the next. Accordingly, processing of the multiple frames of a video in this manner results in computational inefficiencies” and (Gould column 9 line 12) “In one embodiment, even though this portion of the two frames may not meet the threshold to be considered a “matching” portion, the non-matching portion may be sufficiently similar to enable one or more layers of the neural network to be skipped without sacrificing the accuracy of the final classification. In one embodiment, by fast forwarding to a subsequent layer (e.g., skipping from processing layer 2 to processing layer 4, layer 5, or layer 6), the computations associated with the intervening layers (e.g., the layers that are skipped) are not performed, thereby resulting in a reduced computational expense”. That is, processing two frames of highly similar data (for example, 2 consecutive frames of a video or a modified vs. unmodified image) from scratch is inefficient because the computations are largely the same. It is more efficient to terminate processing and reuse prior stored processing where possible to reduce unnecessary processing.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ezra J Baker whose telephone number is (703)756-1087. The examiner can normally be reached Monday - Friday 10:00 am - 8:00 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached at (571) 270-7519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/E.J.B./Examiner, Art Unit 2126
/DAVID YI/Supervisory Patent Examiner, Art Unit 2126
Prosecution Insights