Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 5-18 and 20-21 are pending. This Office Action is responsive to the amendments filed on 08/18/2026, which have been entered into the above identified application.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 5-6, 12-18, and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Jiang et al. (US 11150327 B1, filed 07/24/2018), hereinafter Jiang, in view of Chung et al. (“Hierarchical Multiscale Recurrent Neural Networks”, published 03/09/2017), hereinafter Chung; in further view of Imam et al. (US 20180174023 A1, filed 12/20/2016), hereinafter Imam. Jiang and Imam were cited in the previous Office Action.
Regarding Claim 5, Jiang teaches An artificial neural network system (Jiang: “A spiking neuromorphic network is inspired by the human brain in that it process signals in the spiking domain, where all of the signals are represented by spiking sequences.” [Col. 1, Lines 42-44]; Plain meaning of “artificial neural network” is a model that mimics the structure and function of the biological human brain in an "artificial", or computer-based, environment) comprising:
means for generating a data environment, wherein the means for generating a data environment is configured to select data for input into a recurrent neural network (Jiang: “a feature extractor configured to extract a plurality of features from the synthetic aperture radar signal” [Abstract]);
means for encoding the data selected by the means for generating the data environment for input into an artificial recurrent neural network (Jiang: “an input spiking neural network configured to encode the features as a first plurality of spiking signals” [Abstract]);
the artificial recurrent neural network coupled to receive the encoded data from the means for encoding (Jiang: “a multi-layer recurrent neural network configured to compute a second plurality of spiking signals based on the first plurality of spiking signals” [Abstract]),
wherein the artificial recurrent neural network models a degree of the structure of a biological brain (Jiang: “A spiking neuromorphic network is inspired by the human brain in that it process signals in the spiking domain, where all of the signals are represented by spiking sequences.” [Col. 1, Lines 42-44]); and
an output encoder coupled and configured to identify decisions made by the artificial recurrent neural network and compile those decisions into an output (Jiang: “a readout neural layer configured to compute a signal identifier based on the second plurality of spiking signals; and an output configured to output the signal identifier, the signal identifier identifying the target.” [Abstract]).
However, Jiang fails expressly disclose in reaction to the receipt of the encoded data, the artificial recurrent neural network configured to generate decisions at different hierarchical levels wherein dependencies that emerge between decisions at a lower level in a hierarchy of decisions during processing of the encoded data define a state of entanglement that constructs a decision at a higher level in the hierarchy of decisions; an output encoder coupled and configured to identify the decisions at the different hierarchical levels made by the artificial recurrent neural network and compile those decisions at the different hierarchical levels as respective symbols of an output code; and means for translating the output code into actions.
In the same field of endeavor, Chung teaches in reaction to the receipt of the encoded data, the artificial recurrent neural network configured to generate decisions at different hierarchical levels wherein dependencies that emerge between decisions at a lower level in a hierarchy of decisions during processing of the encoded data define a state of entanglement that constructs a decision at a higher level in the hierarchy of decisions (Chung: “we propose a novel multiscale approach, called the hierarchical multiscale recurrent neural network, that can capture the latent hierarchical structure in the sequence by encoding the temporal dependencies with different timescales using a novel update mechanism.” [Abstract]; “From this simple example, we can see the advantages of having a hierarchical multiscale structure: (1) as the W2P-RNN is updated at a much slower update rate than the C2W-RNN, a considerable amount of computation can be saved, (2) gradients are backpropagated through a much smaller number of time steps, and (3) layer-wise capacity control becomes possible (e.g., use a smaller number of hidden units in the first layer which models short-term dependencies but whose updates are invoked much more often).” [Section 3.1 Motivation]; “A key element of our model is the introduction of a parametrized boundary detector, which outputs a binary value, in each layer of a stacked RNN, and learns when a segment should end in such a way to optimize the overall target objective. Whenever the boundary detector is turned on at a time step of layer
l
(i.e., when the boundary state is 1), the model considers this to be the end of a segment corresponding to the latent abstraction level of that layer (e.g., word or phrase) and feeds the summarized representation of the detected segment into the upper layer (
l
+1). Using the boundary states, at each time step, each layer selects one of the following operations: UPDATE, COPY or FLUSH. The selection is determined by (1) the boundary state of the current time step in the layer below
z
t
l
-
1
and(2) the boundary state of the previous time step in the same layer
z
t
+
1
l
.” [Section 3.2 The Proposed Model]; In light of [Page 16, Lines 4-8] of the specification, which states “Elementary decisions are entangled to capture the range of the complexity of the computations performed by the neurosynaptic computer system. For example, elementary decisions are entangled to construct unitary decisions. Unitary decisions are entangled to construct successively higher levels in a hierarchy of decisions and arbitrarily complex cognitive algorithms”, BRI of “entanglement” would encompass cascading decision making/computations in which the higher level decisions are dependent on combinations of lower level decisions).
an output encoder coupled and configured to identify the decisions at the different hierarchical levels made by the artificial recurrent neural network and compile those decisions at the different hierarchical levels as respective symbols (Chung: “In this paper, we propose a novel multiscale approach, called the hierarchical multiscale recurrent neural network, that can capture the latent hierarchical structure in the sequence by encoding the temporal dependencies with different timescales using a novel update mechanism.” [Abstract]; “To do this, we introduce a binary boundary detector at each layer. The boundary detector is turned on only at the time steps where a segment of the corresponding abstraction level is completely processed. Otherwise, i.e., during the within segment processing, it stays turned off. Using the hierarchical boundary states, we implement three operations, UPDATE, COPY and FLUSH, and choose one of them at each time step.” [Section 1. Introduction]; “A key element of our model is the introduction of a parametrized boundary detector, which outputs a binary value, in each layer of a stacked RNN, and learns when a segment should end in such a way to optimize the overall target objective. Whenever the boundary detector is turned on at a time step of layer
l
(i.e., when the boundary state is 1), the model considers this to be the end of a segment corresponding to the latent abstraction level of that layer (e.g., word or phrase) and feeds the summarized representation of the detected segment into the upper layer (
l
+1). Using the boundary states, at each time step, each layer selects one of the following operations: UPDATE, COPY or FLUSH.” [Section 3.2 The Proposed Model]; “Character-level language modelling is a representative example of discrete sequence modelling, where the discrete symbols form a distinct hierarchical multiscale structure. The performance on real-valued sequences is tested on the handwriting sequence generation in which a relatively clear hierarchical multiscale structure exists compared to other data such as speech signals.” [Section 3. Experiments]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated in reaction to the receipt of the encoded data, the artificial recurrent neural network configured to generate decisions at different hierarchical levels wherein dependencies that emerge between decisions at a lower level in a hierarchy of decisions during processing of the encoded data define a state of entanglement that constructs a decision at a higher level in the hierarchy of decisions; an output encoder coupled and configured to identify the decisions at the different hierarchical levels made by the artificial recurrent neural network and compile those decisions at the different hierarchical levels as respective symbols, as taught by Chung to the system of Jiang because both of these systems are directed towards employing a multi-layer deep recurrent neural network architecture for hierarchical decision making and classification. In making this combination and employing a hierarchical multiscale structure for decision entanglement, it would allow the system of Jiang to save on computation through less frequent updates at higher levels of abstraction and achieve layer-wise capacity control (Chung: [Section 3.1 Motivation]).
Jiang and Chung still fail to expressly disclose an output encoder coupled to compile decisions into an output code; and means for translating the output code into actions.
In the same field of endeavor, Imam teaches an output encoder coupled to compile decisions into an output code (Imam: “path planning may involve making a navigation decision at a particular starting point (e.g., 705) in a physical environment, where the goal of the navigation is to find an optimized path to a desired destination, or reward location (e.g., 710). In such examples, a SNN may be implemented using a neuromorphic computing device, with each neuron in the SNN assigned to correspond to a point within the physical environment.” [0069]; “neurons (e.g., 651-656) of the SNN used to model a path between points in a network (through a spike chain generated through activation of a source neuron (e.g., 652) on synapses (e.g., 657-661) strengthened through previous training of the SNN)) may include further synapses to send output spikes (e.g., 665) to motor control logic 650 to direct an autonomous passenger vehicle, drone, robot, etc. to navigate the physical space corresponding to the SNN. For example, the spike chain may cause a receiving neuron (e.g., 653) to not only propagate the spike chain to the next neuron (e.g., 654) in the path (e.g., on synapse 659), but may also cause an output spike 665 to the motor control 650” [0068]); and
means for translating the output code into actions (Imam: “may also cause an output spike 665 to the motor control 650, which the motor control may interpret to indicate a direction in which the motor control is to direct a device (e.g., drone, robot, etc.) along the corresponding path, among other examples.” [0068]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated an output encoder coupled to compile those decisions into an output code; and means for translating the output code into actions, as taught by Imam to the system of Jiang and Chung because both of these systems are directed to using a spiking neural network modeled after the human brain to make decisions. In making this combination and compiling and deploying the decisions made by the neural network as output code, as taught by Imam, it would allow the system of Jiang and Chung to “advantageously interface with and interoperate with other devices”, thereby allowing for the system to execute the decisions that are made (Imam: [0031]).
Regarding Claim 6, Jiang, Chung, and Imam teach the artificial neural network system of Claim 5, further comprising: means for learning configured to vary parameters in the artificial neural network system to achieve a desired result (Jiang: “a training process in which the parameters of the spiking input neural network 130 and the multi-layer recurrent neural network 150 are computed and the classifier weights of the readout neural layer 170 are computed” [Col. 19, Lines 2-5]).
Regarding Claim 12, Jiang, Chung, and Imam teach the artificial neural network system of Claim 5, wherein the means for encoding the data comprises one or more of: a timing encoder configured to encode the selected data in a pulse position modulation signal for input into neurons and/or synapses of the artificial recurrent neural network; or a statistical encoder configured to encode the selected data in statistical probabilities of activation of neurons and/or synapses in the artificial recurrent neural network; or a byte amplitude encoder configured to encode the selected data in proportional perturbations of neurons and/or synapses in the artificial recurrent neural network; or a frequency encoder configured to encode the selected data in frequencies of activation of neurons and/or synapses in the artificial recurrent neural network; or a noise encoder configured to encode the selected data in a proportional perturbation of a noise level of stochastic processes in the neurons and/or synapses in the artificial recurrent neural network; or a byte synapse spontaneous event encoder configured to encode the selected data in a either a set frequency or probability of spontaneous events in the neurons and/or synapses in the artificial recurrent neural network (Jiang: “In one embodiment, in operation 230, each value of a real-valued feature vector is converted into a corresponding spiking sequence generated by a corresponding input neuron of the input spiking neural network 130. As such, according to one embodiment, the size of the input spiking neural network 130 (e.g., the number of neurons in the input spiking neural network 130) is equal to the number of features in the input 2D feature vector.” [Col. 8, Lines 55-62]; “Each spiking sequence generated by each input neuron is made up of spikes that are randomly (or pseudo-randomly) generated in accordance with a Poisson distribution or exponential distribution having a mean spiking rate that corresponds to the value of its corresponding feature (e.g., a value in the normalized range of [0, π]). The exponential distribution may be used as a computationally simpler approximation of the Poisson distribution. In particular, the real values of the feature vector are treated as average (e.g., mean) values of the distribution, and a random number generator is used to produce spikes such that the randomly generated spiking sequences have mean values corresponding to the values of the feature vector. For example, if each feature vector FSAR, included 2M×N different features, those features would be converted into 2M×N different spiking sequences, each spiking sequence having a spiking rate distribution in accordance with the real value of the corresponding feature.” [Col. 8, Line 63 - Col. 9, Line 13]).
Regarding Claim 13, Jiang, Chung, and Imam teach the artificial neural network system of Claim 5, wherein the means for encoding is configured to map a sequence of bits in a byte to a sequential time point in a time series of events where ON bits produce a positive activation of neurons and/or synapses in the artificial recurrent neural network and OFF bits do not produce an activation of neurons and/or synapses in the artificial recurrent neural network, wherein the positive activation of neurons and/or synapses increases a frequency or probability of events in the neurons and/or synapses (Imam: “As spikes are received, the neuron's activation (tracked using the soma 285 (and local memory 260)) may increase.” [0034]; Jiang: “A binary spiking neural network may be computationally more efficient (e.g., energy efficient) than a real-valued neural network because processing spiking signals (binary sequences) consumes much less power than processing real-valued signals.” [Col. 5, Lines 46-51]; “spiking neuromorphic networks consume very little energy because energy is only consumed when there is a spike and because, most of the time, there is no spike.” [Col. 1, Lines 45-47]).
Regarding Claim 14, Jiang, Chung, and Imam teach the artificial neural network system of Claim 5, wherein the means for encoding is configured to map a sequence of bits in a byte to a sequential time point in a time series of events where ON bits produce a positive activation of neurons and/or synapses and OFF bits produce a negative activation of neurons and/or synapses in the artificial recurrent neural network, wherein the positive activation of neurons and/or synapses increases a frequency or probability of events in the neurons and/or synapses and the negative activation of neurons and/or synapses decreases the frequency or probability of events in the neurons and/or synapses (Imam: “As spikes are received, the neuron's activation (tracked using the soma 285 (and local memory 260)) may increase.” [0034]; Jiang: “A binary spiking neural network may be computationally more efficient (e.g., energy efficient) than a real-valued neural network because processing spiking signals (binary sequences) consumes much less power than processing real-valued signals.” [Col. 5, Lines 46-51]; “In a leaky integrate-and-fire model, the neuron firing (e.g., emission of a spike) is described by the dynamic equation of a neuron membrane potential and current, as given by Equation 9, below: [Equation 9] where u(t) is the membrane potential (e.g., the voltage across the capacitor C), I(t) is the membrane current, and the constants τm and R are the membrane time constant and resistance of the neuron, respectively. When the membrane potential u(t) is greater than or equal to a firing threshold Vth at time tf, the neuron outputs a spike which is scaled by the connection weight wδ(t−tf) (in the case of an inhibitory neuron, the spike has the opposite sign from the excitatory neuron, e.g., −wδ(t−tf)), and, after firing, the capacitor C is reset (e.g., set to zero volts by shorting the capacitor to ground).” [Col. 9, Line 57 – Col. 10, Line 9]).
Regarding Claim 15, Jiang, Chung, and Imam teach the artificial neural network system of Claim 5, wherein the means for encoding is configured to map a sequence of bits in a byte to a sequential time point in a time series of events where ON bits activate excitatory neurons and/or synapses and OFF bits activate inhibitory neurons and/or synapses in the artificial recurrent neural network (Imam: “As spikes are received, the neuron's activation (tracked using the soma 285 (and local memory 260)) may increase.” [0034]; Jiang: “A binary spiking neural network may be computationally more efficient (e.g., energy efficient) than a real-valued neural network because processing spiking signals (binary sequences) consumes much less power than processing real-valued signals.” [Col. 5, Lines 46-51]; “In a leaky integrate-and-fire model, the neuron firing (e.g., emission of a spike) is described by the dynamic equation of a neuron membrane potential and current, as given by Equation 9, below: [Equation 9] where u(t) is the membrane potential (e.g., the voltage across the capacitor C), I(t) is the membrane current, and the constants τm and R are the membrane time constant and resistance of the neuron, respectively. When the membrane potential u(t) is greater than or equal to a firing threshold Vth at time tf, the neuron outputs a spike which is scaled by the connection weight wδ(t−tf) (in the case of an inhibitory neuron, the spike has the opposite sign from the excitatory neuron, e.g., −wδ(t−tf)), and, after firing, the capacitor C is reset (e.g., set to zero volts by shorting the capacitor to ground).” [Col. 9, Line 57 – Col. 10, Line 9]).
Regarding Claim 16, Jiang, Chung, and Imam teach the artificial neural network system of Claim 5, wherein the means for encoding comprises a target generator configured to determine which neurons and/or synapses in the artificial recurrent neural network are to receive at least some of the selected data (Jiang: “The input spiking neural network may include a plurality of input neurons arranged in a grid, wherein the first excitatory neuron layer may include a plurality of first excitatory neurons arranged in a grid, the plurality of first excitatory neurons including a plurality of critical neurons uniformly distributed in the grid, and wherein the plurality of input neurons may be connected to the critical neurons to maintain spatial relationships between the input neurons in corresponding ones of the critical neurons.” [Col. 2, Lines 29-37]; “FIG. 6 is a schematic diagram of neurons in a portion of the input neural layer 130 and their connections to neurons in a corresponding portion of the first excitatory neuron layer E1 152 of the multi-layer recurrent neural network 150 in accordance with one embodiment of the present invention. In some embodiments, the first excitatory neuron layer E1 152 is larger than the input layer 130, and the connections 132 between the input layer 130 and the first excitatory neuron layer E1 152 are on a one-to-one basis (e.g., one neuron of the input layer 130 to one corresponding “critical” neuron of the first excitatory neuron layer E1 152) with fixed synaptic weights (e.g., in some embodiments, there is no need to learn or modify these weights because the function of connections 132 is to pass the spikes generated by the input layer 130 to the multi-layer recurrent neural network 150). As shown in FIG. 6, each neuron of the input layer 130 and the first excitatory layer E1 152 is associated with a position (e.g., in a square grid) within its respective excitatory layer. In one embodiment, the critical neurons on the first excitatory neuron layer E1 152 are uniformly distributed (e.g., they are located on a uniform sub-sampling grid on the first excitatory neuron layer E1 152).” [Col. 10, Lines 29-50]).
Regarding Claim 17, Jiang, Chung, and Imam teach the artificial neural network system of Claim 5, wherein the target generator determines which neurons and/or synapses are to receive the selected data based on one or more of: a region of the artificial recurrent neural network; or a layer or cluster within a region of the artificial recurrent neural network; or a specific voxel location of the neurons and/or synapses within a region of the artificial recurrent neural network; or a type of the neurons and/or synapses within the artificial recurrent neural network (Jiang: “FIG. 6 is a schematic diagram of neurons in a portion of the input neural layer 130 and their connections to neurons in a corresponding portion of the first excitatory neuron layer E1 152 of the multi-layer recurrent neural network 150 in accordance with one embodiment of the present invention. In some embodiments, the first excitatory neuron layer E1 152 is larger than the input layer 130, and the connections 132 between the input layer 130 and the first excitatory neuron layer E1 152 are on a one-to-one basis (e.g., one neuron of the input layer 130 to one corresponding “critical” neuron of the first excitatory neuron layer E1 152) with fixed synaptic weights (e.g., in some embodiments, there is no need to learn or modify these weights because the function of connections 132 is to pass the spikes generated by the input layer 130 to the multi-layer recurrent neural network 150). As shown in FIG. 6, each neuron of the input layer 130 and the first excitatory layer E1 152 is associated with a position (e.g., in a square grid) within its respective excitatory layer. In one embodiment, the critical neurons on the first excitatory neuron layer E1 152 are uniformly distributed (e.g., they are located on a uniform sub-sampling grid on the first excitatory neuron layer E1 152).” [Col. 10, Lines 29-50]).
Regarding Claim 18, Jiang, Chung, and Imam teach the artificial neural network system of Claim 5, wherein the artificial recurrent neural network is a spiking recurrent neural network (Jiang: “Aspects of embodiments of the present invention relate to a target recognition system that is configured to recognize different targets from raw SAR data (e.g., phase history data) using a multi-layer recurrent spiking neural network for capturing different spatial-temporal patterns in the raw SAR data.” [Col. 5, Lines 56-61]).
Regarding Claim 21, Jiang, Chung, and Imam teach the artificial neural network system of Claim 5, wherein the means for translating the output code into actions comprises a plurality of machine learning models that have each been trained to process the symbols of a relevant hierarchical level of the output code (Chung: “This model, called a hierarchical multiscale recurrent neural network (HM-RNN), does not assign fixed update rates, but adaptively determines proper update times corresponding to different abstraction levels of the layers. We find that this model tends to learn fine timescales for low-level layers and coarse timescales for high-level layers. To do this, we introduce a binary boundary detector at each layer. The boundary detector is turned on only at the time steps where a segment of the corresponding abstraction level is completely processed.” [Section 1. Introduction]; “Akey element of our model is the introduction of a parametrized boundary detector, which outputs a binary value, in each layer of a stacked RNN, and learns when a segment should end in such a way to optimize the overall target objective.” [Section 3.2 The Proposed Model]).
Claims 7-9 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Jiang in view of Chung and Imam, as applied to Claim 5 above, in further view of Bai et al. (“Automatic Data Augmentation from Massive Web Images for Deep Visual Recognition”, published 07/24/2018), hereinafter Bai. Bai was cited in the previous Office Action.
Regarding Claim 7, Jiang, Chung, and Imam teach the artificial neural network system of Claim 5, a data selection manager configured to select a subset of data (Jiang: “According to one embodiment, in operation 210, the feature extractor 110 receives synthetic aperture radar (SAR) raw data (e.g., from a radar antenna), and computes features from the SAR raw data. The 2D complex phase history raw SAR data does not have an intuitive or natural mapping to spiking signals. In some embodiments of the present invention, four different features are computed from raw SAR data: amplitude, off-amplitude, positive phase, and negative phase.” [Col. 6, Lines 43-51]); and
a data preprocessor configured to preprocess the selected subset of data (Jiang: “According to some embodiments of the present invention, the feature extractor 110 divides the 2D array of raw SAR feature data into a sequence of subsets (or time windows) of the 2D array. Using a sequence of 2D subsets (or time windows) to represent the 2D SAR array enables embodiments of the present invention to: capture some spatial relationships and/or patterns presented in raw SAR data; utilize the memory property of recurrent spiking neural networks to capture the temporal correlations presented in the sequence of 2D subsets; process the raw SAR data in real-time or substantially real-time because, in practice, raw SAR data is generally obtained as a sequence of 2D RF signals.” [Col. 6, Line 58 - Col. 7, Line 3]).
However, Jiang, Chung, and Imam fail to expressly disclose a search engine configured to search one or more databases and output search results.
In the same field of endeavor, Bai teaches a search engine configured to search one or more databases and output search results (Bai: “we attempt to automatically augment an existing dataset from the web” [Section 1. Introduction]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated a search engine configured to search one or more databases and output search results, as taught by Bai to the system Jiang, Chung, and Imam because both of these systems are directed to using feature extraction for training an objection recognition model. In making this combination and gathering data through a search engine, as taught by Bai, it would allow the system of Jiang, Chung, and Imam access to the web, which “hosts massive images with rich contextual information and the volume keeps growing fast”, “is also the basic source of many datasets, which are scraped from search engines without further human labeling” and “often comes with rich contextual information edited by web authors” (Bai: [Section I. Introduction]).
Regarding Claim 8, Jiang, Chung, and Imam teach the artificial neural network system of Claim 7. However, they fail to expressly disclose wherein the data preprocessor is configured to: adjust a size or dimensions of the selected subset of the results output from the search engine; or create a hierarchy of resolution versions of the selected subset of the results output from the search engine; or filter the selected subset of the results output from the search engine; create statistical variants of the selected subset of the results output from the search engine.
In the same field of endeavor, Bai teaches wherein the data preprocessor is configured to: adjust a size or dimensions of the selected subset of the results output from the search engine; or create a hierarchy of resolution versions of the selected subset of the results output from the search engine; or filter the selected subset of the results output from the search engine; create statistical variants of the selected subset of the results output from the search engine (Bai: “we automatically augment ILSVRC-2012 with an additional 12.5 million images from the web” [Abstract]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated wherein the data preprocessor is configured to: adjust a size or dimensions of the selected subset of the results output from the search engine; or create a hierarchy of resolution versions of the selected subset of the results output from the search engine; or filter the selected subset of the results output from the search engine; create statistical variants of the selected subset of the results output from the search engine, as taught by Bai to the system Jiang, Chung, and Imam because both of these systems are directed to using feature extraction for training an objection recognition model. In making this combination and creating statistical variants of the data (i.e. data augmentation), it would allow the system of Jiang, Chung, and Imam to minimize the gap between expected error and empirical error by increasing sample size, without increasing the labeling cost associated with increasing the size of an existing dataset (Bai: [Section 1. Introduction]).
Regarding Claim 9, Jiang, Chung, Imam and Bai teach the artificial neural network system of Claim 8, wherein the data preprocessor is configured to create statistical variants of the selected subset of the results by introducing statistical noise, changing orientation of an image, cropping an image, or applying a clip mask to an image (Bai: “The traditional data augmentation methods such as mirror transformation, random cropping are equipped during training for all of the image recognition models in this article. For ResNet-50 training, we also used color shifting and random image resizing (the short side in the range of [256, 480]) for data augmentation.” [Section 4.1 Image Classification]).
Claims 10-11 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Jiang in view Chung, Imam, and Bai, as applied to Claim 7 above, in further view of Piao et al. (“Depth-induced Multi-scale Recurrent Attention Network for Saliency Detection”, published 10/27/2019), hereinafter Piao. Piao was cited in the previous Office Action.
Regarding Claim 10, Jiang, Chung, Imam, and Bai teach the artificial neural network system of Claim 7. However, they fail to expressly disclose wherein: the data preprocessor is configured to apply a plurality of different filter functions to an image to generate a plurality of differently-filtered images; and the data preprocessor is configured to present the differently-filtered images simultaneously such that the artificial recurrent neural network receives the differently-filtered images at a same time.
In the same field of endeavor, Piao teaches wherein: the data preprocessor is configured to apply a plurality of different filter functions to an image to generate a plurality of differently-filtered images (Piao: “To be specific, in order to capture multi-scale context features, we impose a global pooling layer and several parallel convolutional layers with different kernel sizes and different dilation rates on the input feature Ffuse. In this way, six multiscale features Fm ( m = 1, 2, . . . , 6) with the same resolution but different contexts are generated.” [Section 3.3. Depth-induced Multi-scale Weighting Module]); and
the data preprocessor is configured to present the differently-filtered images simultaneously such that the artificial recurrent neural network receives the differently-filtered images at a same time (Piao: “Finally, all features fi in multiple layers are summated as Ffuse = PNi=1 fi in an element-wise manner, where N=5 denotes the total number of convolutional blocks. In this way, discriminative multi-level RGB and depth features are effectively learned and fused. This fusion strategy enables our model to produce more accurate saliency results because of the comprehensive combination of both local spatial details and global semantic information.” [Section 3.2. Multi-level Fusion Module]; “Those results also demonstrate that the combination of depth information and multi-scale features can improve the detection accuracy.” [Section 4.4. Ablation analysis]; “In this work, our proposed ‘DMRANet’ enhances the performance of saliency detection from three aspects: 1) fully extracts and fuses multi-level paired complementary features by using a simple yet effective DRB; 2) innovatively combines depth cues with multi-scale information to accurately locate and identify salient objects; 3) progressively generates more accurate saliency results through a novel recurrent attention model.” [Section 5. Conclusion]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated wherein: the data preprocessor is configured to apply a plurality of different filter functions to an image to generate a plurality of differently-filtered images; and the artificial recurrent neural network coupled to receive the differently-filtered images at a same time, as taught by Piao to the system of Jiang, Chung, Imam, and Bai because both of these systems are directed to using a recurrent artificial neural network for object recognition. In making this combination and applying filter functions to data, as taught by Piao, it would allow the system of Jiang, Chung, Imam, and Bai to “enhance [the] model’s robustness especially in challenging scenes” (Piao: [Section 1. Introduction]).
Regarding Claim 11, Jiang, Chung, Imam, and Bai teach the artificial neural network system of Claim 7. However, they fail to expressly disclose wherein: the data preprocessor is configured to contextually filter an image by processing a background of an image through a machine learning model to form a contextually-filtered image; the data preprocessor is configured to perceptually filter the image by segmenting the image to obtain features of object and form a perceptually-filtered image; the data preprocessor is configured to attention filter the image to identify salient information in the image and form an attention-filtered image; and the artificial recurrent neural network coupled to receive the contextually-filtered image, the perceptually-filtered image, and attention-filtered image at a same time.
In the same field of endeavor, Piao teaches wherein: the data preprocessor is configured to contextually filter an image by processing a background of an image through a machine learning model to form a contextually-filtered image (Piao: “Depth information containing various depth cues such as spatial structure and 3D layout has been demonstrated to alleviate those issues in SOD (salient objection detection). In this paper, we mainly focus on effectively using RGB-D data to enhance model’s robustness especially in challenging scenes. As exemplified in Fig. 1, RGB-D methods are superior to RGB methods in terms of a complex scene, in which the salient object shares similar appearance with its surroundings.” [Section 1. Introduction]; “we also create a large scale RGB-D dataset with 1200 paired images containing more complex scenarios, such as multiple or transparent objects, similar foreground and background, complex background, low-intensity environment.” [Section 1. Introduction]);
the data preprocessor is configured to perceptually filter the image by segmenting the image to obtain features of object and form a perceptually-filtered image (Piao: “depth cues with abundant spatial information are innovatively combined with multi-scale context features for accurately locating salient objects.” [Abstract]; “Salient object detection (SOD) aims to identify regions in a scene that visually attract human attention most. Recently, this fundamental task plays an important role in various computer vision applications, e.g., visual tracking, image segmentation and object recognition.” [Section 1. Introduction]);
the data preprocessor is configured to attention filter the image to identify salient information in the image and form an attention-filtered image (Piao: “Multiple objects in a scene have large variations in both depth and scale. Exploring the relationship between depth cues and objects with different scales can further provide vital guidance cues for accurately locating salient regions.” [Section 1. Introduction]); and
the artificial recurrent neural network coupled to receive the contextually-filtered image, the perceptually-filtered image, and attention-filtered image at a same time (Piao: “Finally, all features fi in multiple layers are summated as Ffuse = PNi=1 fi in an element-wise manner, where N=5 denotes the total number of convolutional blocks. In this way, discriminative multi-level RGB and depth features are effectively learned and fused. This fusion strategy enables our model to produce more accurate saliency results because of the comprehensive combination of both local spatial details and global semantic information.” [Section 3.2. Multi-level Fusion Module]; “Those results also demonstrate that the combination of depth information and multi-scale features can improve the detection accuracy.” [Section 4.4. Ablation analysis]; “In this work, our proposed ‘DMRANet’ enhances the performance of saliency detection from three aspects: 1) fully extracts and fuses multi-level paired complementary features by using a simple yet effective DRB; 2) innovatively combines depth cues with multi-scale information to accurately locate and identify salient objects; 3) progressively generates more accurate saliency results through a novel recurrent attention model.” [Section 5. Conclusion]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated wherein: the data preprocessor is configured to contextually filter an image by processing a background of an image through a machine learning model to form a contextually-filtered image; the data preprocessor is configured to perceptually filter the image by segmenting the image to obtain features of object and form a perceptually-filtered image; the data preprocessor is configured to attention filter the image to identify salient information in the image and form an attention-filtered image; and the artificial recurrent neural network coupled to receive the contextually-filtered image, the perceptually-filtered image, and attention-filtered image at a same time, as taught by Piao to the system of Jiang, Chung, Imam, and Bai because both of these systems are directed to using a recurrent artificial neural network for object recognition. In making this combination and applying contextual, perceptual, and attention filter functions to the input data, as taught by Piao, it would allow the system of Jiang, Chung, Imam, and Bai to “enhance [the] model’s robustness especially in challenging scenes”, “such as multiple or transparent objects, similar foreground and background, complex background, low-intensity environment.” (Piao: [Section 1. Introduction]).
Regarding Claim 20, Jiang, Chung, Imam and Bai teach the artificial neural network system of Claim 8. However, they fail to expressly disclose wherein the data preprocessor is configured to create a hierarchy of resolution versions by either down sampling the selected subset of the results or dimensional reduction of the selected subset of the results.
In the same field of endeavor, Piao teaches wherein the data preprocessor is configured to create a hierarchy of resolution versions by either down sampling the selected subset of the results or dimensional reduction of the selected subset of the results (Piao: “As illustrated in Fig. 3, the inputs fRGB i i represent the side-out features from the RGB and depth streams in the i-th level respectively. We feed fdepth i into a series of weight layers Ψ(·) containing two convolutional layers and two PReLU activation functions [19] to learn a depth residual ∆depthi = Ψ(fdepth i ). Then, the depth residual is added to the RGB feature by residual connection to learn a fused feature ffuse i = fRGB i +∆depthi. In this way, complementary clues in the i-th level are fused effectively. Then, we reshape (i.e., up-sample with bilinear interpolation or down sample with max-pooling operation) ffuse i to the same resolution.” [Section 3.2. Multi-level Fusion Module]; “As shown in Fig. 4, depth cues with abundant spatial in formation are further processed to learn a depth vector to guide the weight allocation of multi-scale features. To be specific, in order to capture multi-scale context features, we impose a global pooling layer and several parallel convolutional layers with different kernel sizes and different dilation rates on the input feature Ffuse. In this way, six multi scale features Fm ( m=1,2, ..., 6) with the same resolution but different contexts are generated. Detailed parameters are shown in Fig. 4.” [Section 3.3. Depth-induced Multi-scale Weighting Module]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated wherein the data preprocessor is configured to create a hierarchy of resolution versions by either down sampling the selected subset of the results or dimensional reduction of the selected subset of the results, as taught by Piao to the system of Jiang, Chung, Imam, and Bai because both of these systems are directed to using a recurrent artificial neural network for object recognition. In making this combination and creating a hierarchy of resolution versions through down sampling, it would allow the system of Jiang, Chung, Imam, and Bai to “produce more accurate saliency results because of the comprehensive combination of both local spatial details and global semantic information” (Piao: [Section 3.2. Multi-level Fusion Module]).
Response to Arguments
Examiner acknowledges the Applicant’s amendments to Claims 5, 8, and 17.
Applicant’s arguments, filed 08/18/2026, regarding the rejection of Claims 5-18 and 20-21 under 35 U.S.C. § 103 have been fully considered and are found moot in light of the new ground(s) of rejection under 35 U.S.C. § 103 (see rejection above).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Hwang et al. (“Character-level language modeling with hierarchical recurrent neural networks”) proposes a hierarchical RNN based language model consisting of low-level and high-level RNNs, the former of which employs character-level input and output and provides short-term embeddings to the latter, which operates at the word-level.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MEGAN E HWANG whose telephone number is (703)756-1377. The examiner can normally be reached Monday-Thursday 10:00AM-7:30PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Welch can be reached at (571) 272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/M.E.H./Examiner, Art Unit 2143
/JENNIFER N WELCH/Supervisory Patent Examiner, Art Unit 2143
Prosecution Insights