DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement(s) (IDS)(s) submitted on 03/27/2025 has/have been received, considered, and is/are in compliance with the provisions of 37 CFR 1.97. Accordingly, the IDS(s) has/have been considered by the Examiner.
Specification
Applicant is reminded of the proper content of an abstract of the disclosure.
A patent abstract is a concise statement of the technical disclosure of the patent and should include that which is new in the art to which the invention pertains. The abstract should not refer to purported merits or speculative applications of the invention and should not compare the invention with the prior art.
If the patent is of a basic nature, the entire technical disclosure may be new in the art, and the abstract should be directed to the entire disclosure. If the patent is in the nature of an improvement in an old apparatus, process, product, or composition, the abstract should include the technical disclosure of the improvement. The abstract should also mention by way of example any preferred modifications or alternatives.
Where applicable, the abstract should include the following: (1) if a machine or apparatus, its organization and operation; (2) if an article, its method of making; (3) if a chemical compound, its identity and use; (4) if a mixture, its ingredients; (5) if a process, the steps.
Extensive mechanical and design details of an apparatus should not be included in the abstract. The abstract should be in narrative form and generally limited to a single paragraph within the range of 50 to 150 words in length.
See MPEP § 608.01(b) for guidelines for the preparation of patent abstracts.
Applicant is reminded of the proper language and format for an abstract of the disclosure.
The abstract should be in narrative form and generally limited to a single paragraph on a separate sheet within the range of 50 to 150 words in length. The abstract should describe the disclosure sufficiently to assist readers in deciding whether there is a need for consulting the full patent text for details.
The language should be clear and concise and should not repeat information given in the title. It should avoid using phrases which can be implied, such as, “The disclosure concerns,” “The disclosure defined by this invention,” “The disclosure describes,” etc. In addition, the form and legal phraseology often used in patent claims, such as “means” and “said,” should be avoided.
The abstract of the disclosure is objected to because: "The abstract should not refer to purported merits or speculative applications of the invention and should not compare the invention with the prior art." and "It should avoid using phrases which can be implied" such as the recited limitations “can”, “if”, etc.”. Correction is required. See MPEP § 608.01(b).
Claim Rejections – 35 U.S.C. § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more.
On January 7, 2019, the USPTO released new examination guidelines for determining whether a claim is directed to non-statutory subject matter. According to the guidelines, a claim is directed to non-statutory subject matter if: (a) it does not fall within one of the four statutory categories of invention or (b) or meets a three-prong test for determining that: (1) the claim recites a judicial exception, e.g. an abstract idea, (2) without integration into a practical application and (3) does not recite additional elements that provide significantly more than the recited judicial exception.
Claim(s) 1,13 and 18 is/are directed to a method of controlling a vehicle (i.e., a process). Therefore, claim(s) 1, 13 and 18 is/are within at least one of the four statutory categories of an invention. However, the claim(s) clearly do/does not meet the test for patentability.
With regard to Prong I of the Step 2A analysis in the 2019 PEG, the claims are to be analyzed to determine whether they recite subject matter that falls within one of the following groups of abstract ideas:
(a) Mathematical concepts - mathematical relationships, mathematical formulas or equations, mathematical calculations;
(b) Certain methods of organizing human activity - fundamental economic principles or practices (including hedging, insurance, mitigating risk); commercial or legal interactions (including agreements in the form of contracts; legal obligations; advertising, marketing or sales activities or behaviors; business relations); managing personal behavior or relationships or interactions between people (including social activities, teaching, and following rules or instructions); and
(c) Mental processes - concepts performed in the human mind (including an observation, evaluation, judgment, opinion).
Independent claim 1 includes limitations that recite an abstract idea (emphasized below) and will be used as a representative claim for the remainder of the 101 rejection. Claim 1 recites:A method comprising:
obtaining a training data set for object motion prediction, the training data set including movement data associated with an object, the movement data indicating historical movement of the object prior to a point in time and ground truth movement of the object subsequent to the point in time;
training a machine learning model, wherein training the machine learning model comprises:
conducting a forward temporal prediction training pass to predict, from the historical movement for the object, a predicted future motion of the object subsequent to the point in time;
conducting a backward temporal prediction training pass to predict, from the predicted future motion, a predicted historical motion of the object prior to the point in time; and
modifying weights of the machine learning model based on a comparison between the predicted historical motion and the historical movement indicated in the movement data for the object; and
providing the trained machine learning model to an autonomous vehicle, wherein the autonomous vehicle is configured to utilize the trained machine learning model to forecast future object motion from observed object motion.
The Examiner submits that the foregoing bolded limitation(s) constitute a “mental process” because under its broadest reasonable interpretation, the claim covers performance of the limitation in the human mind. For example, “conducting a…prediction…to predict” in the context of this claim encompasses a person (operator) looking at data collected and forming a simple judgement. Accordingly, the claim recites at least one abstract idea.
Regarding Prong II of the Step 2A analysis in the 2019 PEG, the claims are to be analyzed to determine whether the claim, as a whole, integrates the abstract idea into a practical application. As noted in the 2019 PEG, it must be determined whether any additional elements in the claim beyond the abstract idea integrate(s) the exception into practical application in a manner that imposes a meaningful limit on the judicial exception. The courts have indicated that additional elements merely using a computer to implement an abstract idea, adding insignificant extra solution activity, or generally linking the use of a judicial exception to a particular technological environment or field of use do not integrate a judicial exception into a “practical application.”
In the present case, the additional limitations beyond the above-noted abstract idea are as follows (where the underlined portions are the “additional limitations” while the bolded portions continue to represent the “abstract idea”):
A method comprising:
obtaining a training data set for object motion prediction, the training data set including movement data associated with an object, the movement data indicating historical movement of the object prior to a point in time and ground truth movement of the object subsequent to the point in time;
training a machine learning model, wherein training the machine learning model comprises:
conducting a forward temporal prediction training pass to predict, from the historical movement for the object, a predicted future motion of the object subsequent to the point in time;
conducting a backward temporal prediction training pass to predict, from the predicted future motion, a predicted historical motion of the object prior to the point in time; and
modifying weights of the machine learning model based on a comparison between the predicted historical motion and the historical movement indicated in the movement data for the object; and
providing the trained machine learning model to an autonomous vehicle, wherein the autonomous vehicle is configured to utilize the trained machine learning model to forecast future object motion from observed object motion.
For the following reason(s), the Examiner submits that the above identified additional limitations do not integrate the above-noted abstract idea into a practical application.
Regarding the additional limitations of “obtaining a training data set for object motion prediction”, “training a machine learning model”, “conducting a…prediction training pass to predict…a predicted future motion of the object,” “conducting a…prediction training pass to predict…a predicted historical motion of the object” and “modifying weights of the machine learning model”, the Examiner submits that these limitations are insignificant extra-solution activities that merely use a computer to perform the process. In particular, the obtaining training data step for object motion prediction, the machine learning model training step, the conducting steps for the prediction training pass are recited at a high level of generality (i.e. as a general means of gathering vehicle and road condition data for use in the modifying weights of the machine learning model step), and amounts to mere data gathering, which is a form of insignificant extra-solution activity. The “providing the trained machine learning model to an automotive vehicle…to forecast future object motion” is also recited at a high level of generality (i.e. as a general means of providing the trained machine learning model from the conducted prediction training pass(es)), and amounts to mere post solution result(s), which is/are a form of insignificant extra-solution activity. Lastly, there is no controller, computer or processor to describe how to “apply” the otherwise mental judgement(s) in a generic or general purpose controller/training data environment. The method is recited at a high level of generality and merely automates the obtaining, training, conducting, modifying and providing steps of a trained machine learning model.
Thus, taken alone, the additional elements do not integrate the abstract idea into a practical application. Further, looking at the additional limitation(s) as an ordered combination or as a whole, the limitation(s) add nothing that is not already present when looking at the elements taken individually. For instance, there is no indication that the additional elements, when considered as a whole, reflect an improvement in the functioning of a computer or an improvement to another technology or technical field, apply or use the above-noted judicial exception to effect a particular treatment or prophylaxis for a disease or medical condition, implement/use the above-noted judicial exception with a particular machine or manufacture that is integral to the claim, effect a transformation or reduction of a particular article to a different state or thing, or apply or use the judicial exception to a particular technological environment, such that the claim as a whole is not more than a drafting effort designed to monopolize the exception (MPEP § 2106.05). Accordingly, the additional limitation(s) do/does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
Regarding Step 2B of the 2019 PEG, representative independent claim 1 does not include any additional elements (considered both individually and as an ordered combination) that are sufficient to amount to significantly more than the judicial exception for the same reasons to those discussed above with respect to determining that the claim does not integrate the abstract idea into a practical application. As discussed above with respect to integration of the abstract idea into a practical application, the additional element of using a controller, computer or processor to perform the obtaining, training, conducting or modifying or providing…amounts to nothing more than applying the exception using a generic computer component. Generally applying an exception using a generic computer component cannot provide an inventive concept. Therefore, the Examiner submits that these limitations are insignificant extra-solution activities.
Further, a conclusion that an additional element is insignificant extra-solution activity in Step 2A should be re-evaluated in Step 2B to determine if they are more than what is well-understood, routine, conventional activity in the field. The additional limitations of “movement data…, historical movement of the object prior to a point in time…and ground truth movement…,” “a comparison between the predicted historical motion and the historical movement indicated in the movement data” and “configured to utilize the trained machine learning mode,” are well-understood, routine, and conventional activities because the background recites that object motion forecasting generally describes mechanisms for predicting future motion of an object based on past observed motion of the object or other objects and object motion forecasting may be applied in an autonomous vehicle to enable the vehicle to predict how other objects in or near a roadway will move, and the specification does not provide any indication that the computer(s), processors and/or controller(s) is/are anything other than a conventional computer within a vehicle. MPEP 2106.05(d)(II), and the cases cited therein, including Intellectual Ventures I, LLC v. Symantec Corp., 838 F.3d 1307, 1321 (Fed. Cir. 2016), TLI Communications LLC V. AV Auto. LLC, 823 F.3d 607, 610 (Fed. Cir. 2016), and OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1259, 1363 (Fed. Cir. 2015), indicating that the mere collection or receipt of data over a network is a well-understood, routine, and conventional function when it is claimed in a merely generic manner. The additional limitation of “displaying…,” is a well-understood, routine and conventional activity because the Federal Circuit in Trading Techs. Int’l v. IBG LLC, 921 F.3d 1084, 1093 (Fed. Cir. 2019), and Intellectual Ventures I LLC v. Erie Indemnity Co., 850 F.3d 1315, 1331 (Fed. Cir. 2017), for example, indicated that the mere displaying of data is a well understood, routine, and conventional function. Hence, the claim is not patent eligible.
Dependent claim(s) 2-12, 14-17 and 19-20 do not recite any further limitations that cause the claim(s) to be patent eligible. Rather, the limitations of dependent claims are directed toward additional aspects of the judicial exception and/or well-understood, routine and conventional additional elements that do not integrate the judicial exception into a practical application because they are nothing more than well-understood, routine and conventional activities and/or functions. Therefore, dependent claims 2-12, 14-17 and 19-20 are not patent eligible under the same rationale as provided for in the rejection of independent claim(s) 1, 13 and 18.
Therefore, claims 1-20 is/are ineligible under 35 U.S.C. 101.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 3-7 and 10-20 are rejected under 35 U.S.C. 103 as being unpatentable over H. Sun et al.: “Reciprocal Twin Networks for Pedestrian Motion Learning and Future Path Prediction”, IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 3. pp. 1483-1497, 27 April 2021 (2021-04-27) (Sun) in view of U.S. 20220187841 A1 to Ebrahimi Afrouzi et al. (Ebrahimi).
Regarding claim 1, Sun discloses a method comprising:
obtaining a training data set for object motion prediction, the training data set including movement data associated with an object, the movement data indicating historical movement of the object prior to a point in time and ground truth movement of the object subsequent to the point in time (Sun discloses obtaining training data for object motion prediction including movement data with an object such as a person and in relation to ground truth movement at a point in time tn (Section III-A: let X=[X1,X2,⋯,XN] be the trajectories of all human in the scene…our task is to predict the future trajectories of all human Y^=[Y^1,Y^2,⋯,Y^N] simultaneously…the input trajectory of human n is given by Xn=(xtn,ytn) for time steps t=1,2,⋯,T0 …the ground truth of future trajectory is given by Yn=(xtn, ytn) for time step t=To+1,⋯,Tp)));
training a machine learning model (Sun discloses a training machine learning model regarding coupling networks and reciprocal learning (see Fig. 1; we are learning two coupling networks, the forward prediction network Fθ which predicts the future trajectories Y=Fθ(X) from the past data X , and the backward prediction network Gϕ which predicts the past trajectories X=Gϕ(Y) from the future data Y). This set of neural networks is considered as a machine learning model)), wherein training the machine learning model comprises:
conducting a forward temporal prediction training pass to predict, from the historical movement for the object, a predicted future motion of the object subsequent to the point in time (Sun discloses conducting a forward temporal prediction training pass to predict from the historical movement for the object and a predicted future motion of the object at a point in time (Section III-B: “the forward prediction network Fθ which predicts the future trajectories Y=Fθ(X) from the past data X”)));
conducting a backward temporal prediction training pass to predict, from the predicted future motion, a predicted historical motion of the object prior to the point in time (Sun discloses conducting a backward temporal prediction training pass to predict from the predicted future motion of the object, a predicted historical motion historical movement for the object and a predicted future motion of the object prior to a point in time (Section III-B: (the backward prediction network Gϕ which predicts the past trajectories X=Gϕ(Y) from the future data Y)); (Section III-C: (use this network to map the prediction result of Fθ, Y^=Fθ(X), back to the past trajectory, which is given by X^=Gϕ(Y^)=Gϕ(Fθ(X)))); and
modifying weights of the machine learning model based on a comparison between the predicted historical motion and the historical movement indicated in the movement data for the object (Sun discloses modifying weights of the machine learning model based on comparing predicted historical motion and historical movement from the object movement data to determine error gradients between the motion data (Section III-B (if the backward prediction network Gϕ is trained, we can use the reciprocal constraint(1) to double check the accuracy of the forward prediction network Fθ and improve its performance during training); Section III-C (The past trajectory loss is then given by L−=||X−X^||2); Section IV-D (The only difference is that network training modifies the network weights based on error gradients…it propagates the error all the way to the input layer to modify the original input image to minimize the loss))); and
providing the trained machine learning model to an Sun discloses providing the trained machine learning model to a vehicle, wherein the vehicle is configured to utilize the trained machine learning model to forecast future object motion from observed object motion via a vehicle on-board camera (Section IV: (predicting the human future trajectory from an egocentric view of a moving vehicle with the on-board camera))).
However, Sun does not appear to further expressly disclose:
an autonomous vehicle.
Ebrahimi, in the same field of endeavor, further discloses:
an autonomous vehicle (Ebrahimi discloses autonomous car 14901 operating on a path based on sensor data ([0996] (FIG. 149 illustrates a factory robot 14900 and an autonomous car 14901…car 14901 may approach the robot in a controlled way and ends up where it is supposed to be given the fixed location of the factory robot 14900))).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the method of Sun to incorporate the method of Ebrahimi to include an autonomous vehicle for which the vehicle-mounted camera captures the observed object motion which is then utilized to forecast future object motion, with predictable results, with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to combine Sun and Ebrahimi for the express benefit of including a camera mounted to an autonomous vehicle that conducts backwards and forwards temporal prediction training and uses historical motion data to forecast future object motion obstacle detector to determine whether an obstacle is in the vicinity of and/or on a collision path with the vehicle and output a collision risk warning via a sound output, as explained in Ebrahimi [0996].
Regarding claim 3, Sun discloses the method of claim 1,
wherein the training data set includes movement data for one or more additional objects indicating a position of the one or more additional objects relative to the object, historical movement of the one or more additional objects prior to the point in time, and ground truth movement of the one or more additional objects subsequent to the point in time (Sun discloses training set(s) including movement data of object indicating a position of other objects relative to the vehicle, historical movement and ground truth movement of the objects at a subsequent point in time regarding the trajectory (Section III-A: (The input trajectory of human n is given by Xn=(xtn,ytn) for time steps t=1,2,⋯,T…the ground truth of future trajectory is given by Yn=(xtn,ytn) for time step t=To+1,⋯,Tp))).
Regarding claim 4, Sun discloses the method of claim 1,
wherein conducting a backward temporal prediction training pass to predict, from the predicted future motion, a predicted historical motion of the object prior to the point in time comprises inverting the predicted future motion (Sun discloses comparing the predicted historical motion and historical movement data of the object between an inversion of the historical movement and predicted historical motion of the object (Section III-C: (In reciprocal training, we first pre-train the backward prediction network Gϕ using the training data with all trajectories reversed in time…we then use this network to map the prediction result of Fθ , Y^=Fθ(X), back to the past trajectory)), which implies that the backward prediction network Gϕ takes as input inverted predicted future trajectories during the joint training of the forward and backward prediction networks))).
Regarding claim 5, Sun discloses the method of claim 1,
wherein the comparison between the predicted historical motion and the historical movement indicated in the movement data for the object is a comparison between an inversion of the predicted historical motion and the historical movement for the object indicated in the movement data for the object ((in claim 4, e.g. Sun); in Sun, the past trajectory loss is then given by L−=||X−X^||2 implies that X and X^ are in the same time direction. Whether the backward prediction network outputs a predicted past trajectory in the same or opposite time direction than X appears as a mere design choice for the skilled person, without any technical effort).
Regarding claim 6, Sun discloses the method of claim 1,
wherein the backward temporal prediction training pass further predicts the predicted historical motion of the object based on the ground truth movement of the object (in claim(s) 1 & 3-4, e.g. Sun).
Regarding claim 7, Sun discloses the method of claim 1, wherein the training data set further includes movement data for a second object indicating historical movement of the second object prior to a second point in time and ground truth movement of the second object subsequent to the second point in time (Sun discloses that the training is implicitly performed for all humans in a scene (Section III-A and V-A). During the pre-training of the backward prediction network, ground-truth trajectories are used as input and during the joint training predicted future trajectories are used as input of the backward prediction network), and wherein training the machine learning model further comprises:
conducting a second forward temporal prediction training pass to predict, from the historical movement for the second object, a predicted future motion of the second object subsequent to the second point in time (in claim 1, e.g. Sun);
selecting at least one of the predicted future motion of the second object or the ground truth movement of the second object to use as an input for a subsequent backward temporal prediction training pass (in claim 1, e.g. Sun);
conducting the subsequent backward temporal prediction training pass to predict, from the input, a predicted historical motion of the second object prior to the second point in time (in claim 1, e.g. Sun); and
modifying the weights of the machine learning model based on a comparison between the predicted historical motion of the second object and the historical movement of the second object indicated in the movement data for the second object (in claim 1, e.g. Sun).
Regarding claim 10, Sun discloses the method of claim 1, wherein training the machine learning model further comprises
modifying weights of the machine learning model based on a comparison between the predicted future motion and the ground truth movement (Sun discloses modifying weights of machine learning model based on comparing predicted future motion and ground truth movement to get the loss function for the forward prediction network including a future trajectory loss (Section III-C (The loss function for the forward prediction network includes a future trajectory loss given by L+=||X−X^||2 implies that X and X^ are in the same time direction))).
Regarding claim 11, Sun discloses the method of claim 1,
wherein the comparison between the predicted historical motion and the historical movement indicated in the movement data for the object comprises calculating a result of a cycle consistency loss function (Sun discloses wherein comparing the predicted historical motion and historical movement data for the object comprises calculating cycle consistency restraints such as loss function in order to be more in line with the ground truth (Section II-D: (our work is the first to employ cycle-consistency in human trajectory prediction domain); Section III-C: (The past trajectory loss is then given by L−=||X−X^||2); Section IV-F: (With the reciprocal consistence constraints, during training, our model forces the backward predicted trajectory to be consistent with the observed past trajectory, thus the predicted future trajectory which is the input of the backward network will be forced to be closer to the ground truth))).
Regarding claim 12, Sun discloses the method of claim 11, wherein the cycle consistency loss function quantifies how closely the predicted historical motion of the object matches the historical movement for the object (in claim 11, e.g. Sun).
Regarding claim 13, Sun discloses a system, comprising:
a processor configured to execute computer-executable instructions (in claim 1, e.g. Sun); and
a data store storing computer-executable instructions that, when executed by the processor, cause the system to:
obtain a training data set for object motion prediction, the training data set including movement data associated with an object, the movement data indicating historical movement of the object prior to a point in time and ground truth movement of the object subsequent to the point in time (in claim 1, e.g. Sun);
train a machine learning model, wherein training the machine learning model comprises:
conducting a forward temporal prediction training pass to predict, from the historical movement for the object, a predicted future motion of the object subsequent to the point in time(in claim 1, e.g. Sun);
conducting a backward temporal prediction training pass to predict, from the predicted future motion, a predicted historical motion of the object prior to the point in time (in claim 1, e.g. Sun); and
modifying weights of the machine learning model based on a comparison between the predicted historical motion and the historical movement indicated in the movement data for the object (in claim 1, e.g. Sun); and
provide the trained machine learning model to an autonomous vehicle, wherein the autonomous vehicle is configured to utilize the trained machine learning model to forecast future object motion from observed object motion (in claim 1, e.g. Sun).
Regarding claim 14, Sun discloses the system of claim 13,
wherein the backward temporal prediction training pass further predicts the predicted historical motion of the object based on the ground truth movement of the object (in claim(s) 1, 3-4 and 6, e.g. Sun).
Regarding claim 15, Sun discloses the system of claim 13, wherein the training data set further includes movement data for a second object indicating historical movement of the second object prior to a second point in time and ground truth movement of the second object subsequent to the second point in time (in claim 7, e.g. Sun), and wherein training the machine learning model further comprises:
conducting a second forward temporal prediction training pass to predict, from the historical movement for the second object, a predicted future motion of the second object subsequent to the second point in time (in claim(s) 1 & 7, e.g. Sun);
selecting at least one of the predicted future motion of the second object or the ground truth movement of the second object to use as an input for a subsequent backward temporal prediction training pass (in claim(s) 1 & 7, e.g. Sun);
conducting the subsequent backward temporal prediction training pass to predict, from the input, a predicted historical motion of the second object prior to the second point in time (in claim(s) 1 & 7, e.g. Sun); and
modifying the weights of the machine learning model based on a comparison between the predicted historical motion of the second object and the historical movement of the second object indicated in the movement data for the second object (in claim(s) 1 & 7, e.g. Sun).
Regarding claim 16, Sun discloses the system of claim 13, wherein training the machine learning model further comprises modifying weights of the machine learning model based on a comparison between the predicted future motion and the ground truth movement (in claim 10, e.g. Sun).
Regarding claim 17, Sun discloses the system of claim 13,
wherein the comparison between the predicted historical motion and the historical movement indicated in the movement data for the object comprises calculating a result of a cycle consistency loss function (in claim(s) 11 & 12, e.g. Sun).
Regarding claim 18, Sun discloses one or more non-transitory computer-readable media comprising computer- executable instructions that, when executed by a computing system comprising a processor, cause the computing system to:
obtain a training data set for object motion prediction, the training data set including movement data associated with an object, the movement data indicating historical movement of the object prior to a point in time and ground truth movement of the object subsequent to the point in time (in claim(s) 1 & 13, e.g. Sun);
train a machine learning model, wherein training the machine learning model comprises:
conducting a forward temporal prediction training pass to predict, from the historical movement for the object, a predicted future motion of the object subsequent to the point in time (in claim(s) 1 & 13, e.g. Sun);
conducting a backward temporal prediction training pass to predict, from the predicted future motion, a predicted historical motion of the object prior to the point in time (in claim(s) 1 & 13, e.g. Sun); and
modifying weights of the machine learning model based on a comparison between the predicted historical motion and the historical movement indicated in the movement data for the object (in claim(s) 1 & 13, e.g. Sun); and
provide the trained machine learning model to an autonomous vehicle, wherein the autonomous vehicle is configured to utilize the trained machine learning model to forecast future object motion from observed object motion (in claim(s) 1 & 13, e.g. Sun).
Regarding claim 19, Sun discloses the one or more non-transitory computer-readable media of claim 18,
wherein the backward temporal prediction training pass further predicts the predicted historical motion of the object based on the ground truth movement of the object (in claim(s) 4, 6 and 14, e.g. Sun).
Regarding claim 20, Sun discloses the one or more non-transitory computer-readable media of claim 18,
wherein the comparison between the predicted historical motion and the historical movement indicated in the movement data for the object comprises calculating a result of a cycle consistency loss function (in claim(s) 11-12 & 17, i.e. Sun).
Claim(s) 2 and 8-9 are rejected under 35 U.S.C. 103 as being unpatentable over Sun, in further view of U.S. 20220187841 A1 to Ebrahimi as applied to the claims above, in further view of US. 20100253601 A to Seder et al. (Seder).
Regarding claim 2, the combination of Sun and Ebrahimi discloses the method of claim 1.
However, the combination of Sun and Ebrahimi does not appear to further expressly disclose:
wherein the object is a target entity moving with respect to a lane graph.
Seder, in the same field of endeavor, further discloses:
wherein the object is a target entity moving with respect to a lane graph (Seder discloses a target object such as a target vehicle moving with respect to a lane graph (see Fig. 19; [0110] (Known vehicle systems utilize sensors, inputs from various devices, and on-board or remote processing to establish information regarding the environment surrounding the vehicle…adaptive cruise control systems utilize sensors such as radar devices to track objects such as a target vehicle in front of the host vehicle and adjust vehicle speed in accordance with a range and a change in range sensed with respect to the target vehicle…lane keeping systems utilize available sensor and data to maintain a vehicle within lane markings); [0130] (The shaded bars are the radar tracks overlaid in the image of a forward-looking camera…the position and image extraction module extracts the image patches enclosing the range sensor tracks…the feature extraction module computes the features of the image patches using following transforms…in FIG. 19, the boxes A and B are identified as vehicles while the unlabelled box is identified as a road-side object…the prediction process module utilizes an object's historical information…and predicts the current values...the data association links the current measurements with the predicted objects, or determines the source of a measurement))).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of the combination of Sun and Ebrahimi to incorporate the windshield HUD system of Seder to include a windshield heads-up display (HUD) in order to determine target objects moving with respect to a lane graph of lanes on the road in order to better navigate the travel route and avoid detected objects along the way, with predictable results, with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to combine Sun, Ebrahimi and Seder for the express benefit of including a windshield HUD system to assist in detecting target objects moving within the vicinity of the vehicle with respect to a lane graph(s), as explained in Seder [0110], [0130].
Regarding claim 8, the combination of Sun, Ebrahimi and Seder discloses the method of claim 1 in for example the obviousness to combine in the rejection of corresponding parts of claim 1 & 2 above incorporated herein by reference,
wherein data in the training data set is generated based in whole or in part on operation of sensors within one or more autonomous vehicles (in claim 2, e.g. Sun & Seder).
It would have been obvious to combine for the reasons set forth in the rejection of corresponding parts of claim(s) 1 & 2 above incorporated herein by reference.
Regarding claim 9, the combination of Sun, Ebrahimi and Seder discloses the method of claim 1 in for example the obviousness to combine in the rejection of corresponding parts of claim(s) 1-2 and 8 above incorporated herein by reference, the method of claim 1,
wherein the training data set includes data representing object movement in multiple physical locations (in claim(s) 1-2 & 8, e.g. Sun & Seder).
It would have been obvious to combine for the reasons set forth in the rejection of corresponding parts of claim(s) 1-2 & 8 above incorporated herein by reference.
Conclusion
The prior art made of record and not relied upon is considered pertinent to Applicant’s disclosure as teaching the state of the art of motion forecasting in autonomous vehicles using a machine learning model trained with cycle consistency loss, at the time of filing. For example:
US 20100253542 A1 to Seder; Thomas teaches, inter alia POINT OF INTEREST LOCATION MARKING ON FULL WINDSHIELD HEAD-UP DISPLAY in for example the ABSTRACT, Figures and/or Paragraphs below:
“A substantially transparent windscreen head up display includes one of light emitting particles or microstructures over a predefined region of the windscreen permitting luminescent display while permitting vision through the windscreen. A method to display a graphic identifying a point-of-interest upon the substantially transparent windscreen head up display of a vehicle includes monitoring the point-of-interest, monitoring a GPS digital map device, determining a graphic describing the location of the point-of-interest based upon data from the GPS digital map device for display upon the substantially transparent windscreen head up display, and displaying the graphic upon the substantially transparent windscreen head up display.”
PNG
media_image1.png
442
641
media_image1.png
Greyscale
PNG
media_image2.png
426
676
media_image2.png
Greyscale
US 20220026920 A1 to Ebrahimi; Ali teaches, inter alia LIGHT WEIGHT AND REAL TIME SLAM FOR ROBOTS in for example the ABSTRACT, Figures and/or Paragraphs below:
“Some aspects include a method for operating a cleaning robot, including: capturing LIDAR data; generating a first iteration of a map of the environment in real time; capturing sensor data from different positions within the environment; capturing movement data indicative of movement of the cleaning robot; aligning and integrating newly captured LIDAR data with previously captured LIDAR data at overlapping points; generating additional iterations of the map based on the newly captured LIDAR data and at least some of the newly captured sensor data; localizing the cleaning robot; planning a path of the cleaning robot; and actuating the cleaning robot to drive along a trajectory that follows along the planned path by providing pulses to one or more electric motors of wheels of the cleaning robot.”
PNG
media_image3.png
654
494
media_image3.png
Greyscale
PNG
media_image4.png
562
526
media_image4.png
Greyscale
US 20100253526 A1 to Szczerba; Joseph teaches, inter alia DRIVER DROWSY ALERT ON FULL-WINDSHIELD HEAD-UP DISPLAY in for example the ABSTRACT, Figures and/or Paragraphs below:
“A substantially transparent windscreen head up display of a vehicle includes a display having one of light emitting particles or microstructures over a predefined region of the windscreen permitting luminescent display while permitting vision through the windscreen. A method to alert a drowsy driver condition upon a substantially transparent windscreen head up display includes monitoring a sensor configured to observe an operator of the vehicle, determining the drowsy driver condition based upon the monitoring the sensor, and alerting the drowsy driver condition upon the substantially transparent windscreen head up display based upon the determining the drowsy driver condition. Alerting the drowsy driver condition includes displaying a graphic selected to attract attention of the operator and the substantially transparent windscreen head up display.”
PNG
media_image5.png
686
446
media_image5.png
Greyscale
PNG
media_image6.png
686
442
media_image6.png
Greyscale
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROBERT L PINKERTON whose telephone number is (571)272-9820. The examiner can normally be reached M-TH 9:00-4:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hunter Lonsberry can be reached on 571-272-7298. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ROBERT L PINKERTON/Examiner, Art Unit 3665
/DANIEL L GREENE/Primary Examiner, Art Unit 3665
Prosecution Insights