Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
The Office Action is in response to the amendment filed 12/02/2025. Claims 1-10 and 12-21 are presently pending and are presented for examination.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 10/06/2025 and 10/14/2025 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Response to Arguments
Applicant's arguments filed 12/02/2025 regarding the rejection of claims 1-10 and 12-21 under 35 U.S.C. 103 have been fully considered but they are not persuasive. Applicant argues that the amendments to the claims should distinguish the claim language from the prior art, particularly regarding claim 1 in view of Commons US 9875440 B1 (“Commons”). Specifically, applicant argues that the language “executing, by the at least one processor, a first neural network to the sensor data to cause the first neural network to generate an intermediate result at an intermediate layer of the first neural network independent from outputs generated by trigger classifiers configured to receive the intermediate result as input, the first neural network configured to process a set of features of the surroundings of the vehicle to generate a final output that is not based on outputs generated by trigger classifiers”. However, there are two problems with this argument. First, as disclosed in further detail below, the amendments to the claims are now rejected under 35 U.S.C. 112(b) for indefiniteness. Second, the element “the first neural network independent from outputs generated by trigger classifiers configured to receive the intermediate result as input” is already covered by Commons in that the first layer of the neural network has not received input from other neural networks and is working with the raw sensory input, with the stacked neural network in FIG. 2 being a prime example. Third, as best can be understood, the element “the first neural network configured to process a set of features of the surroundings of the vehicle to generate a final output that is not based on outputs generated by trigger classifiers” is also taught in the stack shown in FIG. 2 of Commons, which doesn’t rely on feedback to lower stages. For these reasons, the rejection of the claims under 35 U.S.C. 103 is maintained.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-10 and 12-21 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites the limitation “to generate a final output that is not based on outputs generated by trigger classifiers” in lines 7-8. The word “final” indicates that it is the last result of the first neural network and that no additional outputs will be generated. However, the rest of the claim language indicates that further outputs are generated from the system, which contradicts the common definition of “final”. Further, the specification in paragraph 69 defines “final output” as “the output of the final layer of the model”, which does not clarify the ambiguity regarding the word “final”. This makes the claim indefinite, as it is unclear what is meant by “final output”. Likewise, claims 2-10, 11-18, and 21 which depend from claim 1, are also indefinite by virtue of their dependency.
Claim 19 recites the limitation “to generate a final output that is not based on outputs generated by trigger classifiers” in lines 9-10. For the same reasons as explained with claim 1, claim 19 is indefinite under 35 U.S.C. 112(b). Likewise, claim 20, which also recites “to generate a final output that is not based on outputs generated by trigger classifiers” in lines”, is also indefinite.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-10 and 12-21 are rejected under 35 U.S.C. 103 as being unpatentable over Commons US 9875440 B1 (“Commons”) in view of Martinson et al. US 9542626 B2 (“Martinson”).
Regarding Claim 1. Commons teaches a method, comprising:
receiving, by the at least one processor, sensor data representing surroundings of a vehicle;
executing, by the at least one processor, a first neural network to the sensor data to cause the first neural network to generate an intermediate result at an intermediate layer of the first neural network (FIG. 1, neural network 20, labeled Nm, which is receiving data from the sensory input at 60 [Column 32, lines 22-43]), the first neural network configured to process a set of features of the surroundings of the vehicle independent from outputs generated by trigger classifiers configured to receive the intermediate result as input (Column 32, lines 22-43, and FIG. 2, where the first layer of the neural network is shown to have not received input from other neural networks and is working with the raw sensory input);
determining, by the at least one processor, that the intermediate result meets a first trigger condition of one or more trigger conditions indicating that the first neural network cannot make a predetermined classification (“Tasks that fall within the paradigm of supervised learning are pattern recognition (also known as classification) and regression (also known as function approximation)” [Column 4, lines 11-31]. One way to supplement training of the neural network is to provide a generic set of pattern recognizers (not necessarily neural network implemented) which trigger upon detection of a pattern, but a pattern that is not recognized. That is, a recognized pattern normally produces a highly correlated response within the network, while an unrecognized pattern will produce a broad, but subthreshold response from many neurons. The pattern recognizers may be statistically based, rule based, or the like, and extract the "object" having an unrecognized pattern from the input space of the ANN system [Column 21, lines 58-67, Column 22, lines 1-4]. Note that just because one version of this doesn’t necessarily implement the neural network for pattern recognition doesn’t mean there cannot also be an embodiment that does implement the neural network. This reads on determining that the intermediate result meets a first trigger condition indicating that the first neural network cannot make a predetermined classification, wherein the trigger condition is an unrecognized pattern that the first neural network could not make a predetermined classification);
in response to determining that the intermediate result meets the first trigger condition, executing, by the at least one processor, a second neural network comprising a trigger classifier trained using previous intermediate results for a target use case to process the intermediate results as an input from the first neural network (FIG. 1 shows a hierarchical stacked neural network composed of a plurality of neural networks at 20, 22, 24, 26, etc., wherein Nm reads on a first neural network sending an output to a second neural network at Nm+1. The second neural network has an output at 40 that is feedback to the lower stage neural network Nm [Column 32, lines 22-43]) to determine a classifier score for the sensor data associated with the intermediate result, the classifier score indicating a real-world object, a scenario, or a feature in the surroundings of the vehicle (The computer system at 400 of FIG. 4 may be used to implement the techniques described herein. According to one embodiment, those techniques are performed by computer system 400 in response to processor 404 executing one or more sequences of one or more instructions contained in main memory 406 [Column 53, lines 58-62]. FIG. 1 shows a hierarchical stacked neural network. A sensory input 60 to stacked neural network 10 enters lowest stage/order neural network 20. The output of each of neural networks 20, 22, 24, 26, etc., is the input for the next neural network in the stack [Column 32, lines 28-30]. “Neural network 2114 is a feed-forward heteroassociative neural network that performs processing actions at stage/order 3, the Sensory-Motor stage/order, of the model described in Table 1. At this stage an intelligent system can recognize objects and place them in classes. Using Sensory-Motor tasks, neural network 2114 analyzes patterns output by neural network 2112 and determines whether a pattern is “a road sign”, “a traffic control device”, “another vehicle”, “a pedestrian”, etc. (Other data that is relevant to driving will be apparent to persons skilled in the art.)”, meaning that the neural network at 2114 can act as a trigger classifier that recognizes objects and places them in classes, while the weights applied by the network read on a classifier score based on the network’s successes and failures in identifying specified objects. For example, a deep learning system for autonomous driving may have difficulty analyzing and identifying a tunnel exit. A training data set is created with positive and negative examples of a tunnel exit. In some embodiments, the trigger classifier is trained on an initial training data set using an intermediate output of a layer of the existing machine learning model. In some embodiments, the layer is the intermediate layer. For example, data from the training set is fed to an existing machine learning model and the output of the second to last layer of the model is used as input to train a trigger classifier. In some embodiments, the trigger classifier is a support vector machine that is trained offline from the deployed deep learning application [Column 20, lines 48-67]);
determining, by the at least one processor, to transmit via a computer network at least a portion of the sensor data associated with the intermediate result in response to determining the classifier score for the sensor data (Commons teaches a “Perceptron” which is a linear classifier for classifying data [Column 1, lines 43-55]. Multi-Layered Perceptrons can be used to train neural networks, which indicates that they had to be distinct [Column 4, lines11-31]. Temporal perceptual learning relies on finding temporal relationships in sensory signal streams. In an environment, statistically salient temporal correlations can be found by monitoring the arrival times of sensory signals. This is done by the perceptual network, and can include gesture and pattern recognition [Column 7, lines 5-23]. So the perceptron is capable of classifying sensory signals (sensor data) and retraining the neural network with that data, wherein the pattern and gesture recognition implies a classifier score for comparing different sensor input); and
transmitting, by the at least one processor, the at least a portion of the sensor data associated with the intermediate result for use as training data for retraining the first neural network or a different neural network (A sensory input to stacked neural network enters the lowest stage/order neural network. The output of each of the neural networks is the input for the next network in the stack. The neural networks in the hierarchical stack can send a request for sensory input to feed more information to neural network. A neural network can send this request when its input does not provide enough information for it to determine an output [Column 32, lines 22-43]. In solving a task, information moves through each network in ascending order by stage. Training is done at each stage [Column 15, lines 19-32], which means that sensor data is transmitted by at least one processor to be used as training data).
Commons does not teach:
wherein the determining whether to transmit includes comparing the classifier score with a threshold value.
However, Martinson teaches:
wherein the determining whether to transmit includes comparing the classifier score with a threshold value (Column 4, lines 5-41).
It would have been obvious to one of ordinary skill in the art at the time the invention was filed to modify the invention of Commons with wherein the determining whether to transmit includes comparing the classifier score with a threshold value as taught by Martinson so that the classifier score will have a baseline metric to compare with the intermediate result of the classifier score to eliminate false positives, as discussed in Martinson in paragraph 18.
Regarding Claim 2. Commons in combination with Martinson teaches the method of claim 1.
Commons also teaches:
wherein the intermediate result is an output of an intermediate layer of the first neural network (Neural network 2114 (which is the trigger classifier of claim 1) is an intermediate network of the system as shown in FIG. 11. Additionally, Commons teaches a three-layer neural network having an intermediary, second layer that provides an output [Column 22, lines 19-20, Column 28, lines 38-41]).
Regarding Claim 3. Commons in combination with Martinson teaches the method of claim 2.
Commons also teaches:
wherein the intermediate result is an output of a second to last layer of the first neural network (Neural network 2114 (which is the trigger classifier of claim 1) is an intermediate network of the system as shown in FIG. 11. Additionally, Commons teaches a three-layer neural network having an intermediary, second layer that provides an output, which means the second to last layer in the three layer neural network [Column 22, lines 19-20, Column 28, lines 38-41]).
Regarding Claim 4. Commons in combination with Martinson teaches the method of claim 1.
Commons does not teach:
wherein the first neural network is a convolutional neural network.
However, Martinson teaches:
wherein the first neural network is a convolutional neural network (A method of augmenting layer-based object detection with deep Convolutional Neural Networks (CNN), wherein the convolutional neural network is a classifier [paragraph 41 of the PGPUB]).
It would have been obvious to one of ordinary skill in the art at the time the invention was filed to modify the invention of Commons with wherein the first neural network is a convolutional neural network as taught by Martinson in order to improve upon the neural networks’ ability to identify and classify objects as taught by Martinson [paragraph 6 of the PGPUB], in order to enhance safety by accurately determining environmental conditions and objects around the vehicle.
Regarding Claim 5. Commons in combination with Martinson teaches the method of claim 1.
Commons also teaches:
wherein the trigger classifier is trained using a training data set at least partially analyzed by a third neural network using a machine learning model based on the neural network used to determine the classifier score (A neural network in the hierarchical stack can train other neural networks that operate at the same order/stage of hierarchical complexity. In this training, the information from the neural network is transferred to an architecturally distinct unit that is analogous in structure to the original neural network. This type of training constitutes a transfer of learning from one neural network to another; the new neural network does not have to be independently trained, thereby saving time and resources [Column 34, lines 26-31, Column 47, lines 9-14]. As shown in FIG. 1, this can be done through a chain of stacked neural networks, and Nm+2 is a third neural network).
Regarding Claim 6. Commons in combination with Martinson teaches the method of claim 5.
Commons also teaches:
wherein the trigger classifier is trained using an input vector, wherein the input vector is an output of a layer of the third neural network (Neural network 2114 is a feed-forward heteroassociative neural network that performs processing actions at stage/order 3, the Sensory-Motor stage/order, of the model described in Table 1. At this stage an intelligent system can recognize objects and place them in classes. Using Sensory-Motor tasks, neural network 2114 analyzes patterns output by neural network 2112 and determines whether a pattern is “a road sign”, “a traffic control device”, “another vehicle”, “a pedestrian”, etc. (Other data that is relevant to driving will be apparent to persons skilled in the art.) Patterns identified by neural network 2112 need not be identical to stored patterns to activate an output signal. Stored patterns that identify “a pedestrian” are based on major human features such as a head, a face, arms, and hands. The patterns associated with “another vehicle”, “a road sign”, and “a traffic control device” use a system of weights that weight some pattern components more heavily than others [Column 47, lines 58-61, Column 48, lines 3-5]).
Regarding Claim 7. Commons in combination with Martinson teaches the method of claim 6.
Commons also teaches:
wherein the layer of the third neural network is dynamically selected (The number of hidden layers within a network and the interconnections between layers depend on the nature of the tasks that the neural network at that particular stage/order is performing [Column 33, lines 56-59], which reads on the layer being dynamically selected).
Regarding Claim 8. Commons in combination with Martinson teaches the method of claim 6.
Commons also teaches:
wherein the trigger classifier is transmitted wirelessly to a vehicle applying the second neural network (FIG. 8 illustrates a car 800 driven by a robot. The robot is wirelessly connected with three 360 degree cameras located at the back 810, roof 820, and front 830 of the car. The robot is also connected with a microphone 840 located on the roof of the car. As noted above, the wireless connection may be Bluetooth, WiFi, microwave, or any other known wireless connection means. In yet another embodiment, the robot may connect to the car over a cellular or Internet network, and the robot may be located remotely from the car [Column 39, lines 12-21]. This robot can be a robotic system that drives the car using the layered neural networks described above [Column 25, lines 9-12]. The neural network can be received/placed in the car’s computer system [Column 54, lines 61-66]).
Regarding Claim 9. Commons in combination with Martinson teaches the method of claim 1.
Commons also teaches:
wherein the trigger classifier has been generated based on an identified improvement need for the first neural network (Neural network 2114 is trained by inputting patterns of “a road sign,” a “traffic control device,” “another vehicle,” “a pedestrian,” etc. A backward-propagation algorithm 2132 adjusts neural network 2114's weights based on the network's successes and failures in identifying “a road sign,” a “traffic control device,” “another vehicle,” “a pedestrian,” etc., which reads on identifying a need for improvement [Column 27, lines 18-22]).
Regarding Claim 10. Commons in combination with Martinson teaches the method of claim 1.
Commons also teaches:
wherein the trigger classifier is used to identify one or more of the following: a tunnel entrance, a tunnel exit, a fork in a road, an obstacle in a road, road lane lines, or drivable space (In other embodiments of the invention, the system and method for the gas station optimizer could be modified to select an optimal speed to travel, optimal locations for rest stops during a road trip, a system and method for avoiding potholes, etc. In one embodiment, each of these involves creating a cognitive model of the unique needs of a user [Column 51, lines 45-46]).
Regarding Claim 12. Commons in combination with Martinson teaches the method of claim 1.
Commons also teaches:
further comprising determining whether to apply the trigger classifier based on one or more required conditions (An output response to neural network 2112 is triggered when threshold levels of excitation in contiguous neurons constitute a large enough area of excitation to make it highly probable that a new motion vector has been generated or a new object has entered the scene covered by a camera. Vectors containing the excitation area's centroid, dimensions and coordinates are output to neural network 2112 [Column 47, lines 9-16]).
Regarding Claim 13. Commons in combination with Martinson teaches the method of claim 12.
Commons also teaches:
wherein the one or more required conditions are based on one or more of the following: a length of time driving, a minimum time since a last retained sensor data of the trigger classifier, a disengagement event associated with an autonomous driving feature, a vehicle type, a steering angle threshold, or a road type requirement (In one embodiment, the driver can “co-pilot” the automatic vehicle and be able to override any of the actions of the automatic vehicle (a disengagement associated with an autonomous driving feature), wherein a human user is able to modify or override an auto-pilot setting [Column 50, lines 21-23, Column 51, lines 38-41]).
Regarding Claim 14. Commons in combination with Martinson teaches the method of claim 1.
Commons also teaches:
wherein the trigger classifier specifies a particular layer of the second neural network from which to receive the intermediate result (The algorithms a particular neural network in a hierarchical stack uses to assign connection weights between neurons also vary, depending on the nature of the problem that the neural network is solving and the input and internal stimuli that the neural network processes. Specific algorithms are not associated with particular stages/orders of neural networks in the hierarchical stack. For example, a type of algorithm that assigns connection weights in a neural network at stage/order m that names geometric objects may differ from an algorithm that assigns connection weights in a neural network at stage/order m that names people [Column 34, lines 1-5]. The stacked neural network can begin at any stage/order and end at any stage/order, but information must be processed by each stage/order in sequence and ascending order (specifying a particular layer) [Column 34, lines 1-5, Column 34, lines 57-61]).
Regarding Claim 15. Commons in combination with Martinson teaches the method of claim 1.
Commons also teaches:
further comprising transmitting at least the portion of the sensor data and metadata identifying one or more of the following: a classifier score, a location, a timestamp, a road type, a length of time since a previously transmitted sensor data, or a vehicle type (Neural network 2116 outputs to neural network 2118 an array pattern for the “motion vector that may lead to a collision” and the history of store coordinates of the “another vehicle” array at successive times [Column 48, lines 34-38]).
Regarding Claim 16. Commons in combination with Martinson teaches the method of claim 1.
Commons also teaches:
further comprising transmitting at least the portion of the sensor data and operating conditions of a vehicle identifying one or more of the following: a vehicle speed, a vehicle acceleration, a vehicle braking, or a vehicle steering angle (Neural network 2118 is trained using patterns of movement that a vehicle would follow while it is driving. In one embodiment, a driving simulation video game or computer model is used to train the neural network. In another embodiment, in the training stage, the neural network is placed in a car with an experienced driving instructor, who provides feedback to the neural network in order to train it. In this embodiment, there is little chance that the neural network will cause an accident because the driving instructor has access to an emergency brake and can take control of the car away from the neural network, or override any of the neural network's decisions, if necessary [Column 49, lines 12-23]).
Regarding Claim 17. Commons in combination with Martinson teaches the method of claim 1.
Commons also teaches:
further comprising receiving via the computer network the trigger classifier represented by a vector of weights (Neural network 2112 (the second neural network) performs processing actions at stage/order 2, the Circular Sensory-Motor stage/order, in the model described in Table 1. At this stage/order an intelligent system distinguishes objects and tracks them. Using Circular Sensory-Motor stage/order tasks, neural network 2112 maps input excitation patterns from neural network 2110 to clusters. Cluster weights are adjusted each time a new excitation pattern enters neural network 2112 from neural network 2110. A second hidden layer of neurons tracks excitation patterns through the scene and links their movement centroids to the appropriate clusters [Column 47, lines 31-41]. Neural network 2114 (the trigger classifier) is a feed-forward heteroassociative neural network that performs processing actions at stage/order 3, the Sensory-Motor stage/order, of the model described in Table 1. At this stage an intelligent system can recognize objects and place them in classes [Column 47, lines 53-57]).
Regarding Claim 18. Commons in combination with Martinson teaches the method of claim 17.
Commons also teaches:
wherein the trigger classifier is represented by the vector of weights and a bias (Neural network 2114 (the trigger classifier) is trained by inputting patterns of “a road sign,” a “traffic control device,” “another vehicle,” “a pedestrian,” etc. A backward-propagation algorithm 2132 adjusts neural network 2114's weights based on the network's successes and failures in identifying “a road sign,” a “traffic control device,” “another vehicle,” “a pedestrian,” etc. When neural network 2114 associates a pattern with “a road sign,” a “traffic control device,” “another vehicle,” “a pedestrian,” etc. the network outputs to neural network 2116 the pattern's classification as “a road sign,” a “traffic control device,” “another vehicle,” “a pedestrian,” etc., as well as the pattern's centroid, dimensions, store coordinates, and history of centroid positions at successive times [Column 48, lines 3-15]).
Regarding Claim 19. Commons teaches a computer program product, the computer program product being embodied in a non- transitory computer readable storage medium and comprising computer instructions for:
executing, a first neural network to the sensor data to cause the first neural network to generate an intermediate result at an intermediate layer of the first neural network (FIG. 1, neural network 20, labeled Nm, which is receiving data from the sensory input at 60 [Column 32, lines 22-43]), the first neural network configured to process a set of features of the surroundings of the vehicle independent from outputs generated by trigger classifiers configured to receive the intermediate result as input (Column 32, lines 22-43, and FIG. 2, where the first layer of the neural network is shown to have not received input from other neural networks and is working with the raw sensory input);
determining, by the at least one processor, that the intermediate result meets a first trigger condition of one or more trigger conditions indicating that the first neural network cannot make a predetermined classification (“Tasks that fall within the paradigm of supervised learning are pattern recognition (also known as classification) and regression (also known as function approximation)” [Column 4, lines 11-31]. One way to supplement training of the neural network is to provide a generic set of pattern recognizers (not necessarily neural network implemented) which trigger upon detection of a pattern, but a pattern that is not recognized. That is, a recognized pattern normally produces a highly correlated response within the network, while an unrecognized pattern will produce a broad, but subthreshold response from many neurons. The pattern recognizers may be statistically based, rule based, or the like, and extract the "object" having an unrecognized pattern from the input space of the ANN system [Column 21, lines 58-67, Column 22, lines 1-4]. Note that just because one version of this doesn’t necessarily implement the neural network for pattern recognition doesn’t mean there cannot also be an embodiment that does implement the neural network. This reads on determining that the intermediate result meets a first trigger condition indicating that the first neural network cannot make a predetermined classification, wherein the trigger condition is an unrecognized pattern that the first neural network could not make a predetermined classification);
in response to determining that the intermediate result meets the first trigger condition, executing, a second neural network comprising a trigger classifier independently trained using previous intermediate results for a target use case to process the intermediate result as an input from the first neural network (FIG. 1 shows a hierarchical stacked neural network composed of a plurality of neural networks at 20, 22, 24, 26, etc., wherein Nm reads on a first neural network sending an output to a second neural network at Nm+1. The second neural network has an output at 40 that is feedback to the lower stage neural network Nm [Column 32, lines 22-43]) to cause the trigger classifier to determine a classifier score for the first sensor data, the classifier score indicating a real-world object, a scene associated with the intermediate result, or a feature in the surroundings of the vehicle (The computer system at 400 of FIG. 4 may be used to implement the techniques described herein. According to one embodiment, those techniques are performed by computer system 400 in response to processor 404 executing one or more sequences of one or more instructions contained in main memory 406 [Column 53, lines 58-62]. FIG. 1 shows a hierarchical stacked neural network. A sensory input 60 to stacked neural network 10 enters lowest stage/order neural network 20. The output of each of neural networks 20, 22, 24, 26, etc., is the input for the next neural network in the stack [Column 32, lines 28-30]. “Neural network 2114 is a feed-forward heteroassociative neural network that performs processing actions at stage/order 3, the Sensory-Motor stage/order, of the model described in Table 1. At this stage an intelligent system can recognize objects and place them in classes. Using Sensory-Motor tasks, neural network 2114 analyzes patterns output by neural network 2112 and determines whether a pattern is “a road sign”, “a traffic control device”, “another vehicle”, “a pedestrian”, etc. (Other data that is relevant to driving will be apparent to persons skilled in the art.)”, meaning that the neural network at 2114 can act as a trigger classifier that recognizes objects and places them in classes, while the weights applied by the network read on a classifier score based on the network’s successes and failures in identifying specified objects. For example, a deep learning system for autonomous driving may have difficulty analyzing and identifying a tunnel exit. A training data set is created with positive and negative examples of a tunnel exit. In some embodiments, the trigger classifier is trained on an initial training data set using an intermediate output of a layer of the existing machine learning model. In some embodiments, the layer is the intermediate layer. For example, data from the training set is fed to an existing machine learning model and the output of the second to last layer of the model is used as input to train a trigger classifier. In some embodiments, the trigger classifier is a support vector machine that is trained offline from the deployed deep learning application [Column 20, lines 48-67]);
determining to transmit via a computer network at least a portion of the sensor data associated with the intermediate result in response to determining the classifier score (Commons teaches a “Perceptron” which is a linear classifier for classifying data [Column 1, lines 43-55]. Multi-Layered Perceptrons can be used to train neural networks, which indicates that they had to be distinct [Column 4, lines11-31]. Temporal perceptual learning relies on finding temporal relationships in sensory signal streams. In an environment, statistically salient temporal correlations can be found by monitoring the arrival times of sensory signals. This is done by the perceptual network, and can include gesture and pattern recognition [Column 7, lines 5-23]. So the perceptron is capable of classifying sensory signals (sensor data) and retraining the neural network with that data, wherein the pattern and gesture recognition implies a classifier score for comparing different sensor input); and
transmitting the at least a portion of the sensor data associated with the intermediate result for use as training data for retraining the first neural network or a different neural network (A sensory input to stacked neural network enters the lowest stage/order neural network. The output of each of the neural networks is the input for the next network in the stack. The neural networks in the hierarchical stack can send a request for sensory input to feed more information to neural network. A neural network can send this request when its input does not provide enough information for it to determine an output [Column 32, lines 22-43]. In solving a task, information moves through each network in ascending order by stage. Training is done at each stage [Column 15, lines 19-32], which means that sensor data is transmitted by at least one processor to be used as training data).
Commons does not teach:
wherein the determining to transmit includes comparing the classifier score with a threshold value.
However, Martinson teaches:
wherein the determining to transmit includes comparing the classifier score with a threshold value (Column 4, lines 5-41).
It would have been obvious to one of ordinary skill in the art at the time the invention was filed to modify the invention of Commons with wherein the determining to transmit includes comparing the classifier score with a threshold value as taught by Martinson, in part because it would be an obvious combination of known elements to produce a predictable result, and so that the classifier score will have a baseline metric to compare the intermediate result to.
Regarding Claim 20. Commons teaches a system, comprising:
a sensor on a vehicle (A global positioning system unit in the car will tell the automatic driver where the car is located [Column 47, lines 65-67]);
an artificial intelligence processor (A trained artificial neural network based processor [abstract, Claim 18]);
a vehicle control module (Vehicle control logic [Column 49, lines 6-9]);
an image signal processor configured to:
receive an image captured using the sensor;
process the image to generate a processed image (Hierarchical stacked neural network having processing facilities, and which is trained to search the camera images for information relevant to driving [Column 41, lines 25-28, Column 46, lines 47-50]); and
provide the processed image to a first neural network (the hierarchical stacked neural network can be trained to search the camera images for information relevant to driving and provide the result to another neural network [Column 47, lines 35-39]);
a memory coupled with the artificial intelligence processor, wherein the memory is configured to provide the artificial intelligence processor (execution of the sequences of instructions contained in main memory causes the processor to perform the process steps [Column 54, lines 25-27]) with instructions which when executed cause the artificial intelligence processor to:
receive the processed image;
perform an inference using the first neural network on the processed image by executing the neural network to using the processed image to cause the first neural network to generate an intermediate result at an intermediate layer of the first neural network and an inference result that is not generated based on the intermediate result (Column 32, lines 22-43, and FIG. 2, where the first layer of the neural network is shown to have not received input from other neural networks and is working with the raw sensory input), the first neural network configured to process a set of features of the surroundings of the vehicle (The computer system at 400 of FIG. 4 may be used to implement the techniques described herein. According to one embodiment, those techniques are performed by computer system 400 in response to processor 404 executing one or more sequences of one or more instructions contained in main memory 406 [Column 53, lines 58-62]. FIG. 1 shows a hierarchical stacked neural network. A sensory input 60 to stacked neural network 10 enters lowest stage/order neural network 20. The output of each of neural networks 20, 22, 24, 26, etc., is the input for the next neural network in the stack [Column 32, lines 28-30]. “Neural network 2114 is a feed-forward heteroassociative neural network that performs processing actions at stage/order 3, the Sensory-Motor stage/order, of the model described in Table 1. At this stage an intelligent system can recognize objects and place them in classes. Using Sensory-Motor tasks, neural network 2114 analyzes patterns output by neural network 2112 and determines whether a pattern is “a road sign”, “a traffic control device”, “another vehicle”, “a pedestrian”, etc. (Other data that is relevant to driving will be apparent to persons skilled in the art.)”, meaning that the neural network at 2114 can act as a trigger classifier that recognizes objects and places them in classes, while the weights applied by the network read on a classifier score based on the network’s successes and failures in identifying specified objects. For example, a deep learning system for autonomous driving may have difficulty analyzing and identifying a tunnel exit. A training data set is created with positive and negative examples of a tunnel exit. In some embodiments, the trigger classifier is trained on an initial training data set using an intermediate output of a layer of the existing machine learning model. In some embodiments, the layer is the intermediate layer. For example, data from the training set is fed to an existing machine learning model and the output of the second to last layer of the model is used as input to train a trigger classifier. In some embodiments, the trigger classifier is a support vector machine that is trained offline from the deployed deep learning application [Column 20, lines 48-67]);
determine that the intermediate result of the first neural network meets a first trigger condition of one or more trigger conditions (“Tasks that fall within the paradigm of supervised learning are pattern recognition (also known as classification) and regression (also known as function approximation)” [Column 4, lines 11-31]. One way to supplement training of the neural network is to provide a generic set of pattern recognizers (not necessarily neural network implemented) which trigger upon detection of a pattern, but a pattern that is not recognized. That is, a recognized pattern normally produces a highly correlated response within the network, while an unrecognized pattern will produce a broad, but subthreshold response from many neurons. The pattern recognizers may be statistically based, rule based, or the like, and extract the "object" having an unrecognized pattern from the input space of the ANN system [Column 21, lines 58-67, Column 22, lines 1-4]. Note that just because one version of this doesn’t necessarily implement the neural network for pattern recognition doesn’t mean there cannot also be an embodiment that does implement the neural network. This reads on determining that the intermediate result meets a first trigger condition indicating that the first neural network cannot make a predetermined classification, wherein the trigger condition is an unrecognized pattern that the first neural network could not make a predetermined classification);
in response to determining that the intermediate result of the neural network meets the first trigger condition, provide the intermediate result of the first neural network to a trigger classifier trained using previous intermediate results for a target use case to process the intermediate results as an input from the first neural network (FIG. 1 shows a hierarchical stacked neural network composed of a plurality of neural networks at 20, 22, 24, 26, etc., wherein Nm reads on a first neural network sending an output to a second neural network at Nm+1. The second neural network has an output at 40 that is feedback to the lower stage neural network Nm [Column 32, lines 22-43]), wherein the trigger classifier is used to determine a classifier score corresponding to the image associated with the intermediate result, the classifier score indicating a real-world object, a scenario, or a feature in the surroundings of the vehicle (The computer system at 400 of FIG. 4 may be used to implement the techniques described herein. According to one embodiment, those techniques are performed by computer system 400 in response to processor 404 executing one or more sequences of one or more instructions contained in main memory 406 [Column 53, lines 58-62]. FIG. 1 shows a hierarchical stacked neural network. A sensory input 60 to stacked neural network 10 enters lowest stage/order neural network 20. The output of each of neural networks 20, 22, 24, 26, etc., is the input for the next neural network in the stack [Column 32, lines 28-30]. “Neural network 2114 is a feed-forward heteroassociative neural network that performs processing actions at stage/order 3, the Sensory-Motor stage/order, of the model described in Table 1. At this stage an intelligent system can recognize objects and place them in classes. Using Sensory-Motor tasks, neural network 2114 analyzes patterns output by neural network 2112 and determines whether a pattern is “a road sign”, “a traffic control device”, “another vehicle”, “a pedestrian”, etc. (Other data that is relevant to driving will be apparent to persons skilled in the art.)”, meaning that the neural network at 2114 can act as a trigger classifier that recognizes objects and places them in classes, while the weights applied by the network read on a classifier score based on the network’s successes and failures in identifying specified objects. For example, a deep learning system for autonomous driving may have difficulty analyzing and identifying a tunnel exit. A training data set is created with positive and negative examples of a tunnel exit. In some embodiments, the trigger classifier is trained on an initial training data set using an intermediate output of a layer of the existing machine learning model. In some embodiments, the layer is the intermediate layer. For example, data from the training set is fed to an existing machine learning model and the output of the second to last layer of the model is used as input to train a trigger classifier. In some embodiments, the trigger classifier is a support vector machine that is trained offline from the deployed deep learning application [Column 20, lines 48-67]); and
provide an inference result of the first neural network to the vehicle control module to at least in part autonomously operate the vehicle (an alert to the vehicle control logic is generated when the pattern of the neuron excitation indicates that the “another vehicle” having a “motion vector that may lead to a collision” is sufficiently close to the vehicle being operated or is approaching the vehicle being operated at a high speed so that the self-driving vehicle might respond appropriately [Column 38, lines 59-60, Column 49, lines 6-11)); and a network interface configured to:
transmit at least a portion of the image associated with the intermediate result based at least in part on the classifier score for retraining the first neural network or a different neural network (A sensory input to stacked neural network enters the lowest stage/order neural network. The output of each of the neural networks is the input for the next network in the stack. The neural networks in the hierarchical stack can send a request for sensory input to feed more information to neural network. A neural network can send this request when its input does not provide enough information for it to determine an output [Column 32, lines 22-43]. In solving a task, information moves through each network in ascending order by stage. Training is done at each stage [Column 15, lines 19-32], which means that sensor data is transmitted by at least one processor to be used as training data).
Commons does not teach:
wherein the determining whether to transmit includes comparing the classifier score with a threshold value.
However, Martinson teaches:
wherein the determining whether to transmit includes comparing the classifier score with a threshold value (Column 4, lines 5-41).
It would have been obvious to one of ordinary skill in the art at the time the invention was filed to modify the invention of Commons with wherein the determining whether to transmit includes comparing the classifier score with a threshold value as taught by Martinson, in part because it would be an obvious combination of known elements to produce a predictable result, and so that the classifier score will have a baseline metric to compare the intermediate result to.
Regarding Claim 21. Commons in combination with Martinson teaches the method of claim 1.
Commons also teaches:
wherein the one or more trigger conditions correspond to one or more of whether a length of time since a last capture has exceeded a minimum amount of time, whether a minimum length of time driving has elapsed, or whether a time of day is within a certain range (In reinforcement learning, data are usually not given, but generated by an agent’s interactions with the environment. At each point in time t, the agent performs an action and the environment generates an observation Xt and an instantaneous cost Ct. This is common in the art [Column 4, lines 51-58]. At each time t, the agent perceives its state and the set of possible actions. It chooses an action and receives from the environment the new state and a reward [Column 5, lines 34-41]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AARON G CAIN whose telephone number is (571)272-7009. The examiner can normally be reached Monday: 7:30am - 4:30pm EST to Friday 7:30pm - 4:30am.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Wade Miles can be reached at (571) 270-7777. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AARON G CAIN/Examiner, Art Unit 3656