Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 8 is objected to because of the following informalities: “one or more images sensors…” Appropriate correction is required.
Claims 15-20 objected to because of the following informalities: “one or more processor”. The claimed “processor” needs to be processors. Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea of a mathematical concept without significantly more. The claims recite the mathematical concept of: generating an image embedding; generating a text embedding; determining cosine similarity between the embeddings; rewarding a model based on the similarity; and optimizing the model based on the reward. This judicial exception is not integrated into a practical application because the additional limitations of image sensors, an autonomous vehicle, processors and memory, and receiving a text merely link the mathematical concept to the field of autonomous vehicles. MPEP 2106.04(d)(1). The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements merely link the mathematical concept to the field of autonomous vehicles. MPEP 2106.05(h).
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
Claims 1-20 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, because the specification, while being enabling for a CLIP model (spec. 47), does not reasonably provide enablement for “determining a similarity score representing a similarity between the image-based embeddings of the current state and the text-based embeddings of the goal…” (Claim 1). The specification does not enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make the invention commensurate in scope with these claims. The claims are directed to a broader scope of models that don’t share an embedding space and are not jointly trained. Applicant has not described that invention in the specification, but it is claimed. To overcome the rejection, stick to the CLIP model in spec. 44-47.
Claims 5, 12 and 19 also don’t enable the full scope of the claim. Equation 1 in spec. 44 is the only inverse similarity disclosed. Applicant has not enabled all the different types of inversion – reciprocal, additive inverse etc. Therefore the full scope of claims 5, 12 and 19 are not enabled.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 4, 6-9, 11, 13-16, 18 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over VISION-LANGUAGE MODELS ARE ZERO-SHOT REWARD MODELS FOR REINFORCEMENT LEARNING by Rocamonde et al, US20200387161A1 to Qizwini et al and Talk2Car: Taking Control of Your Self-Driving Car by Deruyttere et al (Talk2Car).
Claims 3, 5, 10, 12, 17 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over VISION-LANGUAGE MODELS ARE ZERO-SHOT REWARD MODELS FOR REINFORCEMENT LEARNING by Rocamonde et al, US20200387161A1 to Qizwini et al, Talk2Car: Taking Control of Your Self-Driving Car by Deruyttere et al (Talk2Car) and Discrete Factorial Representations as an Abstraction for Goal Conditioned RL by Islam et al.
Rocamonde teaches claims 1, 8 and 15. A method for optimizing an action policy of a machine learning model of an autonomous (Rocamonde abs “VLM-RMs based on CLIP to train a MuJoCo humanoid to learn complex tasks without a manually specified reward function, such as kneeling, doing the splits, and sitting in a lotus position.”)
passing the (Rocamonde sec. 2 “image encoder CLIPI... mapping [images] into latent space V…” The mapped images are the embeddings.)
passing the text prompt through a text encoder to generate text-based embeddings of the goal; (Rocamondes sec. 3.1 uses “natural language task description” as the text prompt. Rocamonde’s text description is turned into an embedding using “CLIPL” from Rocamonde sec. 2.)
determining a similarity score representing a similarity between the image-based embeddings of the current state and the text-based embeddings of the goal; (Rocamonde equation 2 p. 3 below, right side of equation is a similarity score.)
PNG
media_image1.png
42
254
media_image1.png
Greyscale
executing a reinforcement learning model for a closed-loop autonomous driving task, wherein the similarity score is utilized as a reward in the reinforcement learning model; and (Rocamonde’s RCLIP is the reward, see below.)
PNG
media_image1.png
42
254
media_image1.png
Greyscale
optimizing an action policy of the reinforcement learning model based on the similarity score utilized as the reward, wherein the action policy is associated with a control command of the autonomous (Rocamonde sec. 3.3 “We can then compute the reward function as cosine similarity between the state embeddings and the task embedding which we only need to compute once. Once we have computed the reward for a batch of interactions, we can use them to perform the standard RL algorithm updates.”)
Rocamonde doesn’t teach the vehicle.
However, Qizwini teaches A method for optimizing an action policy of a machine learning model of an autonomous vehicle, the method comprising: (Quizwini abs “sequence of images with a deep reinforcement learning agent associated with a control feature of the autonomous vehicle to obtain an optimized set of control policies…”)
generating an image of an environment about an autonomous vehicle based on vehicle sensor data representing a current state of the autonomous vehicle; (Qizwini abs “real world data including a sequence of images of a road environment, the sequence of images generated based on a vehicle traversing the road environment…” Qizwini para 51 “The vision sequence module 102 receives real world environment data 110 which includes image data captured of the environment by one or more sensors of the sensor system 28 (e.g., camera, lidar, etc.).”)
passing the generated image through an image encoder… (Qizwini para 52 “The simulation module 104 processes the vision sequence 112…”)
Rocamonde, Qizwini and the claims all process images. It would have been obvious to a person having ordinary skill in the art, at the time of filing, to apply Rocamonde to a autonomous vehicle because “it is desirable to provide improved systems and methods for training an autonomous vehicle without the need for labeled images.” Qizwini para 4.
Rocamonde doesn’t teach receiving a text prompt.
However, Talk2Car teaches receiving a text prompt representing a goal of the autonomous vehicle; (Talk2Car sec. 1 “Talk2Car, that comes with commands formulated in textual natural language for self-driving cars.”)
Rocamonde, Talk2Car and the claims all process language. It would have been obvious to a person having ordinary skill in the art, at the time of filing, to receive text prompts in Rocamonde to solve “the problem in an autonomous driving set ting, where a passenger can control the actions of an Autonomous Vehicle (AV) by giving natural language commands.” Talk2Car p.1.
Rocamonde teaches claims 2, 9 and 16. The method of claim 1, further comprising:
executing a foundation model to perform the determining of the similarity score.
(Rocamonde equation 2 p. 3 below.)
PNG
media_image1.png
42
254
media_image1.png
Greyscale
Rocamonde teaches claims 3, 10 and 17. The method of claim 2, wherein the similarity score is determined as follows:
PNG
media_image2.png
70
590
media_image2.png
Greyscale
wherein r represents the reward utilized in the reinforcement learning model, FMstate represents the image-based embeddings of the current state of the autonomous vehicle, and FMgoal represents the text-based embeddings of the goal. (Rocamonde equation 2 p. 3 below.)
PNG
media_image1.png
42
254
media_image1.png
Greyscale
Rocamonde doesn’t inverse the similarity.
However, Islam teaches inverse similarity. (Islam p. 8 “For the baseline goal-DQN agent, we provide an additional reward bonus based on the cosine distance between continuous embeddings of the state observation and goal…” cosine distance is 1 – cosine similarity.)
Rocamonde, Islam and the claims all calculate the cosine similarity. It would have been obvious to a person having ordinary skill in the art, at the time of filing, to use cosine distance because Rocamonde sec. 2 states that “[c]ommonly CLIP encoders are trained to minimize the cosine distance between embeddings…” and using cosine distance would predictable provide a reward that increases as the embeddings stray from the goal embeddings.
Talk2Car teaches claims 4, 11 and 18. The method of claim 1, wherein the text prompt is a human-crafted text prompt not generated by a machine learning model. (Talk2Car sec. 1 “Talk2Car, that comes with commands formulated in textual natural language for self-driving cars.”)
Rocamonde teaches claims 5, 12 and 19. The method of claim 1, wherein the determining of the similarity score includes deriving an (Rocamonde equation 2 p. 3 below.)
PNG
media_image1.png
42
254
media_image1.png
Greyscale
Rocamonde doesn’t inverse the similarity.
However, Islam teaches inverse similarity. (Islam p. 8 “For the baseline goal-DQN agent, we provide an additional reward bonus based on the cosine distance between continuous embeddings of the state observation and goal…” cosine distance is 1 – cosine similarity.)
Rocamonde, Islam and the claims all calculate the cosine similarity. It would have been obvious to a person having ordinary skill in the art, at the time of filing, to use cosine distance because Rocamonde sec. 2 states that “[c]ommonly CLIP encoders are trained to minimize the cosine distance between embeddings…” and using cosine distance would predictable provide a reward that increases as the embeddings stray from the goal embeddings.
Rocamonde teaches claims 6, 13 and 20. The method of claim 1, wherein the image encoder is part of a vision-language model (VLM) configured to generate a vector representing the generated image in a learned embedding space. (Rocamonde sec. 2 “CLIPI”)
Rocamonde teaches claims 7 and 14. The method of claim 6, wherein the text encoder is part of a large language model (LLM) configured to generate a vector representing the goal in a learned embedding space. (Rocamonde sec. 2 “CLIPL”)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Austin Hicks whose telephone number is (571)270-3377. The examiner can normally be reached Monday - Thursday 8-4 PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mariela Reyes can be reached at (571) 270-1006. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AUSTIN HICKS/Primary Examiner, Art Unit 2142
Prosecution Insights