DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitations use a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitations are: "an image capturing device" in claims 14 and 15.
Because these claim limitations are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, they are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have these limitations interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitations to avoid them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function; or (2) present a sufficient showing that the claim limitations recites sufficient structure to perform the claimed function so as to avoid them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-5, 9, and 12-15 are rejected under 35 U.S.C. 103 as being unpatentable over
Sudhakar et al. (Sudhakar S, Sze V, Karaman S. Uncertainty from motion for DNN monocular depth estimation. In2022 International Conference on Robotics and Automation (ICRA) 2022 May 23 (pp. 8673-8679). IEEE. Hereafter referred to as Sudhakar.)
in view of Lakshminarayanan et. al (Lakshminarayanan B, Pritzel A, Blundell C. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems. 2017;30. Hereafter referred to as Lakshminarayanan)
and in view of Srinivasan et al. (US 11386325 B1 hereafter referred to as Srinivasa----n).
Regarding claim 1:
Sudhakar teaches:
a method using a plurality of neural networks (On page 2 of Sudhakar, “let
θ
1
:
M
be M functional mappings parameterized by weights (e.g., a DNN ensemble of size M or M sampled networks from a BNN).”)
trained to generate a perception output based on an input image (On page 2 of Sudhakar, “let
θ
m
X
n
→
Z
m
,
n
.
σ
m
,
n
2
be the m’th functional mapping parameterized by weights (DNN) that takes in the n’th image
X
n
[an input image] in video sequence 𝒩 = {1 ,…n, …N} and outputs per-pixel depth prediction Zm,n and per-pixel aleatoric variance prediction σ2m,n” [a perception output])
the computer implemented method comprising: for a time instance of a plurality of consecutive time instances: (On page 2 of Sudhakar, “let
θ
m
X
n
→
Z
m
,
n
.
σ
m
,
n
2
be the m’th functional mapping parameterized by weights (DNN) that takes in the n’th image
X
n
[a time instance] in video sequence 𝒩 = {1 ,…n, …N} [a plurality of consecutive time instances] and outputs per-pixel depth prediction Zm,n and per-pixel aleatoric variance prediction σ2m,n”)
processing the image associated with the time instance using a subset of neural networks of the plurality of neural networks, thereby obtaining a network output for the time instance; (On pages 2 and 3 of Sudhakar, “let
θ
1
:
M
be M functional mappings parameterized by weights (e.g., a DNN ensemble of size M or Mk sampled networks from a BNN) [the plurality of neural networks] […] We cycle through the M networks [the plurality of neural networks] over the N images such that we run θm [a subset of neural networks] on the n’th input image Xn [the image associated with the time instance] where m = n modulo M to obtain the depth prediction Zm,n and aleatoric variance prediction σm,n [obtaining a network output]”.
determining an aggregated network output by combining the obtained network output for the time instance with network outputs obtained for a number of preceding time instances, (On page 3 of Sudhakar, “Let u, v, index all pixels in the image. For each Zm,n and σm,n [obtained network output for the time instance] at pixel u, v, we project Zm,n and σm,n to 3D space […] If mask D indexed at pixel u, v, is true, than pixel u, v, represents K’th view of a point we have seen before (K > 1), and we update µ1:K-1 and ∑1:K-1 [network outputs obtained for a number of preceding time instances] with the new K’th Gaussian with mean µK and covariance ∑K [combining] […] The total uncertainty σ2n for the pixel u, v, is the z-variance component of ∑K. We repeat this process for all the pixels in the predicted depth map, and obtain the total predictive uncertainty σ2n for all pixels in the n’th image [aggregated network output]” )
wherein the obtained network output for the time instance and the network outputs obtained for the number of preceding time instances are obtained from different subsets of neural networks of the plurality of neural networks. (On page 3 of Sudhakar, “We cycle through the M networks [the plurality of neural networks] over the N images [the number of preceding time instances] such that we run θm on the n’th input image Xn [the time instance]where m = n modulo M to obtain the depth prediction Zm, n and aleatoric variance prediction σm, n [the obtained network output].” Each time instance n is assigned a neural network m which is selected by calculating m = n modulo M. The list of M neural networks is cycled through over time, so that each image is paired with a different neural network.)
Sudhakar does not explicitly teach:
that at least two neural networks of the plurality of neural networks are different from each other,
that the perception task performed in the method is of an electronic device or vehicle, and
obtaining an image depicting at least a portion of the surrounding environment of the electronic device or vehicle at the time instance.
Lakshminarayanan teaches
that at least two neural networks of the plurality of neural networks are different from each other. (On page 4 of Lakshminarayanan, “We found that random initialization of the NN parameters [one difference], along with random shuffling of the data points [another difference], was sufficient to obtain good performance in practice.” [0045] of the specification for the instant application states that “At least two neural networks of the plurality of the neural networks are different from each other. This may be to ensure that the method takes advantage of an ensemble of neural networks which can give different outputs. By having different neural networks, the results of the perception task can be improved. It may be further advantageous to have different neural networks specialized at different scenarios. In some embodiments, each neural network of the plurality of neural networks is different from each other. The neural networks of the plurality of neural networks may differ from each other in one or more different ways. As an example, the neural networks may have different network weights. As another example, the neural networks of the plurality of neural networks may be trained differently from each other. The neural networks may e.g. be trained using different training data sets, using different pre-processing steps on the training data (e.g. data augmentation, such as rotation, cropping and/or re-scaling of the training data), using different hyper parameters (e.g. loss function, weight initialization, learning rate, batch size) during the training process, or by using different stochastic depths.” Based on the examples given of neural networks “using different hyperparameters” and “using different pre-processing steps on the training data”, random initialization of neural network parameters and random shuffling of input data is sufficient to meet the broadest reasonable interpretation of “different from each other”.
Lakshminarayanan and Sudhakar are both related to the same field of endeavor (i.e. performing perception tasks using ensembles of neural networks). In view of the teachings of Lakshminarayanan it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to apply the teachings of Lakshminarayanan to Sudhakar based on the fact that the method of Sudhakar is tested directly on a version of Lakshminarayanan’s system (On page 4 of Sudhakar, “we test two use cases of UfM: Ensemble-UfM where θ1:M is an ensemble with M ensemble members as in Lakshminarayanan et al. and MC-Dropout-UfM where θ1:M is an aleatoric network that is sampled M times with different dropouts as in Kendall et al. ”).
Srinivasan teaches:
--performing a perception task of an electronic device or vehicle and
obtaining an image depicting at least a portion of a surrounding environment of the electronic device or vehicle at the time instance (In col. 7 lines 58-63 of Srinivasan, “In some embodiments, the dash cam (which is referred to more generally as a “vehicle device”) is configured to execute one or more neural networks (and/or other artificial intelligence and program logic), such as based on input from one or more of the cameras and other sensors associated with the dash cam [obtaining an image depicting at least a portion of the surrounding environment of the electronic device or vehicle at the time instance], to intelligently detect safety events [performing a perception task of an electronic device or vehicle]”).
Srinivasan and Sudhakar are both related to the same field of endeavor (i.e. performing perception tasks using ensembles of neural networks). In view of the teachings of Srinivasan, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to apply the teachings of Srinivasan to Sudhakar in order to detect possible vehicle crashes at a reduced computational cost (see Srinivasan col. 1 lines 19-22, “Processing sensor data from a vehicle that is usable to detect, in real-time, events that are indicative of a crash or increased risk of a crash, requires significant data storage and processing power”).
Regarding claim 2:
Sudhakar in view of Lakshminarayanan and Srinivasan teaches the method according to claim 1. Lakshminarayanan additionally teaches wherein the neural networks are trained differently from each other (On Lakshminarayanan page 4, “We found that random initialization of the NN parameters [one difference], along with random shuffling of the data points [another difference], was sufficient to obtain good performance in practice”). [0045] of the specification states that “As another example, the neural networks may be trained differently from each other. The neural networks may e.g. be trained using different training data sets, using different pre-processing steps on the training data (e.g. data augmentation, such as rotation, cropping and/or re-scaling of the training data), using different hyper parameters (e.g. loss function, weight initialization, learning rate, batch size) during the training process, or by using different stochastic depths.” Based on the examples given of neural networks “using different hyperparameters” and “using different pre-processing steps on the training data”, random initialization of neural network parameters and random shuffling of input data is sufficient to meet the broadest reasonable interpretation of “trained differently”.
Sudhakar, Lakshminarayanan, and Srinivasan are combinable for the same rationale as set forth above with respect to claim 1.
Regarding claim 3:
Sudhakar in view of Lakshminarayanan and Srinivasan teaches the method according to claim 1. Lakshminarayanan additionally teaches wherein the neural networks of the plurality of neural networks have a same network architecture (On Lakshminarayanan page 5, “For both classification and regression, we evaluate the negative log likelihood (NLL) which depends on the predictive uncertainty. NLL is a proper scoring rule and a popular metric for evaluating predictive uncertainty. For classification we additionally measure classification accuracy and the Brier score, defined as BS = K-1 ΣKk=1 (t*k – p(y = k|x*))2 where t*k=1 if k = y*, and 0 otherwise. For regression problems, we additionally measured the root mean squared error (RMSE). Unless otherwise specified, we used batch size of 100 and Adam optimizer with fixed learning rate of 0.1 in our experiments. We use the same technique for generating adversarial training examples for regression problems. Goodfellow et al. used a fixed ϵ for all dimensions; this is unsatisfying if the input dimensions have different ranges. Hence, in all of our experiments, we set ϵ to 0.01 times the range of the training data along that particular dimension. We used the default weight initialization in Torch). [0045] of the specification for the instant application states that “The neural networks of the plurality of neural networks may have a same network architecture. In such case, the neural networks of the plurality of neural networks may differ in some other way as explained above. It should be noted that by having the same network architecture should be interpreted as having generally the same network architecture. The neural networks of the plurality of neural networks may have different network architectures. In case the neural networks differ in network architecture to any extent, they may advantageously have a same execution time or similar (i.e. within a certain tolerance) execution time. The neural networks may for instance have different network architectures, but a same or similar number and/or types of operations to achieve a same or similar runtime latency.” Based on the clarification that “same network architecture should be interpreted as having generally the same network architecture” and the example given that “the neural networks may for instance have different network architectures but a same or similar number and/or types of operations to achieve a same or similar runtime latency”, each network in the ensemble using the NLL scoring rule, adversarial training examples, and default weight initialization is sufficient to meet the broadest reasonable interpretation of having “same network architecture”.
Sudhakar, Lakshminarayanan, and Srinivasan are combinable for the same rationale as set forth above with respect to claim 1.
Regarding claim 4:
Sudhakar in view of Lakshminarayanan and Srinivasan teaches the method according to claim 1. Srinivasan additionally teaches wherein the perception task is any one of an object detection task, an object classification task, an image classification task, an object recognition task, a free-space estimation task, an object-tracking task and an image segmentation task (In col. 31 lines 19-28, “The modular neural network may include a plurality of machine learning models. Each of the plurality of machine learning models may be independently tunable and trainable to identify corresponding features. The plurality of the machine learning models may include face-hand detector (e.g., face and hand detection model) [an object recognition task], a hand action classifier (e.g., a hand action classification model) [an image classification task], and/or a gaze detection classifier (e.g., a gaze classification model) [an image classification task]or a gaze detection model) [an object recognition task]”).
Sudhakar, Lakshminarayanan, and Srinivasan are combinable for the same rationale as set forth above with respect to claim 1.
Regarding claim 5:
Sudhakar in view of Lakshminarayanan and Srinivasan teaches the method according to claim 1. Sudhakar additionally teaches wherein the subset of neural networks of the plurality of neural networks consists of one neural network of the plurality of neural networks (On page 3 of Sudhakar, “Although we have access to all DNNs θ1:M, the efficiency advantage of UfM comes from only running one inference using a single DNN per cycle [the subset of neural networks consists of one neural network]. We cycle through the M networks [the plurality of neural networks] over the N images such that we run θm on the n’th input image Xn where m = n modulo M to obtain the depth prediction Zm, n and aleatoric variance prediction σm, n”).
Sudhakar, Lakshminarayanan, and Srinivasan are combinable for the same rationale as set forth above with respect to claim 1.
Regarding claim 9:
Sudhakar in view of Lakshminarayanan and Srinivasan teaches the method according to claim 1. Sudhakar in view of Lakshminarayanan and Srinivasan additionally teaches wherein the aggregated network output is an average of the obtained network output for the time instance and the network outputs maintained for the number of preceding time instances (On page 3 of Sudhakar, “Let u, v, index all pixels in the image. For each Zm,n and σm,n [obtained network output for the time instance] at pixel u, v, we project Zm,n and σm,n to 3D space […] If mask D indexed at pixel u, v, is true, than pixel u, v, represents K’th view of a point we have seen before (K > 1), and we update µ1:K-1 and ∑1:K-1 [network outputs obtained for a number of preceding time instances] with the new K’th Gaussian with mean [an average of] µK and covariance ∑K […] The total uncertainty σ2n for the pixel u, v, is the z-variance component of ∑K [aggregated network output]. We repeat this process for all the pixels in the predicted depth map, and obtain the total predictive uncertainty σ2n for all pixels in the n’th image)
Sudhakar, Lakshminarayanan, and Srinivasan are combinable for the same rationale as set forth above with respect to claim 1.
Regarding claim 12:
Sudhakar in view of Lakshminarayanan and Srinivasan teaches the method according to claim 1. Sudhakar in view of Lakshminarayanan and Srinivasan additionally teaches a non-transitory computer-readable storage medium having a computer program stored thereon, the computer program including computer readable instructions which, when executed by a computing device, causes the computing device to carry out the method according to claim 1 (On page 5 of Sudhakar, “We run all the experiments in PyTorch on a Nvidia RTX 2080 Ti GPU”).
Sudhakar, Lakshminarayanan, and Srinivasan are combinable for the same rationale as set forth above with respect to claim 1.
Regarding claim 13:
The claim is similar to claim 1 and is rejected under the same rationale as claim 1’s rejection.
Regarding claim 14:
Sudhakar in view of Lakshminarayanan and Srinivasan teaches an electronic device comprising an apparatus according to claim 13.
Srinivasan additionally teaches that the electronic device comprises an image capturing device configured to capture an image depicting at least a portion of the surrounding environment of the electronic device. (In col. 7 lines 58-63 of Srinivasan, “In some embodiments, the dash cam (which is referred to more generally as a “vehicle device”) is configured to execute one or more neural networks (and/or other artificial intelligence and program logic), such as based on input from one or more of the cameras and other sensors associated with the dash cam).
Regarding claim 15:
The claim is similar to claims 1 and 14. However, claim 15 has one additional limitation which is addressed by Srinivasan.
Srinivasan teaches a vehicle (In col 7 lines 50-53 of Srinivasan, “The dash cam is installable onto existing vehicles and provides real-time alerts based on processing of video data from one or more cameras of the dash cam)” comprising an image capturing device and apparatus for performing a perception task .
Sudhakar, Lakshminarayanan, and Srinivasan are combinable for the same rationale as set forth above with respect to claim 1.
Claim(s) 6, 7, 10, and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Sudhakar in view of Lakshminarayanan and Srinivasan as applied to claim 1 above, and further in view of Zhou (Zhou ZH. Machine learning. Springer nature; 2021 Aug 20. Hereafter referred to as Zhou).
Regarding claim 6:
Sudhakar in view of Lakshminarayanan and Srinivasan teaches the method according to claim 1.
Sudhakar in view of Lakshminarayanan and Srinivasan does not teach wherein the subset of neural network(s) of the plurality of neural networks comprises two or more neural networks of the plurality of neural networks.
Zhou teaches wherein the subset of neural network(s) of the plurality of neural networks comprises two or more neural networks of the plurality of neural networks (On page 182 of Zhou, “Ensemble learning, also known as multiple classifier system and committee-based learning, trains and combines multiple learners [two or more neural networks of the plurality of neural networks] to solve a learning problem”).
Sudhakar and Zhou are both related to the same field of endeavor (i.e. machine learning). In view of the teachings of Zhou it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to apply the teachings of Zhou to Sudhakar et al. in order to improve the generalization ability and therefore the practical applicability of each subset of neural networks, without incurring the computational costs of running the entire plurality of neural networks on each time instance (On page 182 of Zhou, “By combining multiple learners, the generalization ability of an ensemble is often much stronger than that of an individual learner”).
Regarding claim 7:
Sudhakar in view of Lakshminarayanan and Srinivasan and further in view of Zhou teaches the method according to claim 6.
Zhou additionally teaches wherein the network output of the subset of neural network(s) is an aggregated sub-network output of the two or more neural network(s) of the set of neural network(s) (On page 182 of Zhou, “Ensemble learning, also known as multiple classifier system and committee-based learning, trains and combines [an aggregated sub-network output] multiple learners [two or more neural networks of the plurality of neural networks] to solve a learning problem”).
Sudhakar and Zhou are combinable for the same rationale as set forth above with respect for claim 7.
Regarding claim 10:
Sudhakar in view of Lakshminarayanan and Srinivasan teaches the method according to claim 1.
Sudhakar in view of Lakshminarayanan and Srinivasan does not teach wherein the aggregated network output is a weighted average of the obtained network output for the time instance and the network outputs obtained for the number of preceding time instances.
Zhou teaches wherein the aggregated network output is a weighted average of the obtained network output for the time instance and the network outputs obtained for the number of preceding time instances (On page 194 of Zhou, “Weighted averaging has been widely used since the 1950s and it was first used in ensemble learning in Perrone and Cooper. Weighted averaging plays an important role in ensemble learning since other combination methods can all be viewed as its special cases or variants. Indeed, weighted averaging can be regarded as a fundamental motivation of ensemble learning studies. Given a set of base learners, different ensemble methods can be viewed as different ways of assigning the weights”).
Sudhakar and Zhou are both related to the same field of endeavor (i.e., machine learning). It would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to apply the teachings of Zhou to Sudhakar in order to make heterogenous ensembles more effective (On pages 194-195 of Zhou, “Generally speaking, the weighted averaging method is a better choice when individual learners have considerably different performance, while the simple averaging method is preferred when individual learners share similar performance”).
Regarding claim 11:
Sudhakar in view of Lakshminarayanan and Srinivasan teaches the method according to claim 1.
Sudhakar in view of Lakshminarayanan and Srinivasan does not teach wherein combining the obtained network output for the time instance with the network outputs obtained for the number of preceding time instances comprises:
feeding the obtained network output for the time instance and the network outputs obtained from the number of preceding time instances into a machine learning model configured to output the aggregated network output.
Zhou does teach wherein combining the obtained network output for the time instance with the network outputs obtained for the number of preceding time instances comprises:
feeding the obtained network output for the time instance and the network outputs obtained from the number of preceding time instances into a machine learning model configured to output the aggregated network output (On page 196 of Zhou, “Stacking starts by training the first level learners using the original training set and then “generating” a new data set for training the second level learner [feeding obtained network output and the network outputs obtained from the number of preceding time instances into a machine learning model configured to output the aggregated network output]. In the new data set the outputs of the first-level learners are used as the input features [obtained network output and the network outputs obtained from the number of preceding time instances], while the labels from the original training set remain unchanged”).
Sudhakar and Zhou are both related to the same field of endeavor (i.e., machine learning). It would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to apply the teachings of Zhou to Sudhakar in order to robustly combine networks in situations with noisy data (On page 198 of Zhou, “In practice, however, the correct data generating model is by no way guaranteed in the models under consideration, and sometimes even hard to be approximated by the models under consideration. Hence, Stacking usually performs better than BMA in practice since it is more robust than BMA”).
Claim(s) 8 is rejected under 35 U.S.C. 103 as being unpatentable over Sudhakar in view of Lakshminarayanan and Srinivasan as applied to claim 1 above, and further in view of Kim et al. (Kim YW, Byun YC, Krishna AV, Krishnan B. Selfie segmentation in video using n-frames ensemble. IEEE Access. 2021 Dec 6;9:163348-62. Hereafter referred to as Kim).
Sudhakar in view of Lakshminarayanan and Srinivasan teaches the method according to claim 1.
Sudhakar in view of Lakshminarayanan and Srinivasan does not teach wherein the number of preceding time instances is based on a number of neural networks of the plurality of neural networks or a number of subsets of neural network(s) of the plurality of neural networks.
Kim does teach wherein the number of preceding time instances is based on a number of neural networks of the plurality of neural networks or a number of subsets of neural network(s) of the plurality of neural networks (Figure 2. on page 5 of Kim depicts a method for performing a segmentation task using an ensemble of neural networks. At each frame, one mask is generated by an ensemble member and combined with a number of masks output by other models in the ensemble which are based off of previous frames. Each output mask is combined with the output masks for the N – 1 preceding frames, so that outputs from a total of N frames are combined together, where N is the total number of models in the ensemble. Page 4 of Kim describes this in more detail: “In detail, at a given time t, a video has a frame t, and the frame t is fed to a segmentation model n to generate a segmented nth output mask. The NF ensemble combines the output mask with other output masks generated from previous segmentation model n - 1, … model 1 to generate the final mask t. It rotates each segmentation model using a round-robin way. For example, if there are total n models for ensemble [the plurality of neural networks] and model n was used for the frame t at a given time. The NF ensemble uses model 1 for the frame t + 1 at a given time t + 1. It combines the segmented output mask of model 1 with previous output masks generated from segmentation model n, …, model 2 [the number of preceding time instances] to create the final output mask t + 1.” On page 6 of Kim, “This work uses four selfie DSMs for the ensemble.” indicates that the models being used are neural networks).
Sudhakar and Kim are both related to the same field of endeavor (i.e. performing perception tasks using ensembles of neural networks). It would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to apply the teachings of Kim to Sudhakar in order to aggregate a plurality of neural networks using a soft voting method (On page 4 of Kim, “In this paper, a soft voting method is used for the ensemble. It treats the individual classifiers equally and averages the outputs of the individual.” In order to achieve the goal of treating individual classifiers equally, it is necessary that each one introduce the same number of outputs into the aggregate; otherwise, the aggregate generated here would be more influenced by some classifiers than by others).
Conclusion
Any inquiry concerning this communication or earlier communication from the examiner should be directed to Alma Thompson whose telephone number is +1 (571) 270-1810. The examiner can normally be reached Monday-Thursday, 8:30 am – 6:00 pm EST. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael Huntley can be reached at +1 (303) 297-4307.
Should the Applicant desire to communicate via email or schedule an Applicant-initiated
interview, the Applicant may use the Automated Interview Request (AIR) form online at
<<http://www.uspto.gov/patent/uspto-automated-interview-request-air-form>> In particular, the AIR form allows the Applicant to include authorization to communicate via email with the Examiner for
interview-related communications.
Information regarding the status of an application may be obtained from the Patent Application
Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pairdirect.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at (866) 217-9197 (toll-free). If you would like assistance from a USPTO Customer Service
Representative or access to the automated information system, call (800) 786-9199 (IN USA OR
CANADA) or (571) 272-1000.
/AT/Examiner, Art Unit 2129
/MICHAEL J HUNTLEY/Supervisory Patent Examiner, Art Unit 2129
Prosecution Insights