DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Pending
1-20
35 U.S.C. 112
18, 20
35 U.S.C. 101
1-20
35 U.S.C. 103
1-20
Priority
Applicant’s indication of Domestic Benefit information based on provisional application 63/681,363 filed 08/09/2024 is acknowledged.
Information Disclosure Statement
The information disclosure statement(s) (IDS(s)) submitted on 12/16/2024 is/are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement(s) is/are being considered by the examiner.
Claim Objections
Claims 18 and 20 are objected to because of the following informalities. Both claims recite “content creation for 3D assets”, “conversational AI operations”, and “generative AI operations” without previously defining the abbreviations “3D” and “AI”. Appropriate correction is required.
Claim Rejections - 35 USC § 112
Claims 18 and 20 are rejected on the basis that it contains an improper Markush grouping of alternatives. See In re Harnisch, 631 F.2d 716, 721-22 (CCPA 1980) and Ex parte Hozumi, 3 USPQ2d 1059, 1060 (Bd. Pat. App. & Int. 1984). A Markush grouping is proper if the alternatives defined by the Markush group (i.e., alternatives from which a selection is to be made in the context of a combination or process, or alternative chemical compounds as a whole) share a “single structural similarity” and a common use. A Markush grouping meets these requirements in two situations.
First, a Markush grouping is proper if the alternatives are all members of the same recognized physical or chemical class or the same art-recognized class, and are disclosed in the specification or known in the art to be functionally equivalent and have a common use. Second, where a Markush grouping describes alternative chemical compounds, whether by words or chemical formulas, and the alternatives do not belong to a recognized class as set forth above, the members of the Markush grouping may be considered to share a “single structural similarity” and common use where the alternatives share both a substantial structural feature and a common use that flows from the substantial structural feature. See MPEP § 2117.
The Markush grouping of claim 18 is improper because the alternatives defined by the Markush grouping do not share both a single structural similarity and a common use.
Claim 18 recites:
“The at least one processor of claim 11, wherein the at least one processor is comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system for generating or presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system implemented using a robot; a system for performing one or more conversational AI operations; a system for performing one or more generative AI operations; a system implementing one or more large language models (LLMs); a system implementing one or more vision language models (VLMs); a system implementing one or more multi-modal language models; a system for generating synthetic data; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.”
A system using a robot, for example, does not share structural similarities or a common use with a system for performing light transport simulation. A perception system for an autonomous or semi-autonomous machine, for example, does not share structural similarities or a common use with a system for performing one or more conversational AI operations. Generally, each grouping of claim 18 does not relate to the structure or use of the previous grouping or following grouping of the claim.
The Markush grouping of claim 20 is improper because the alternatives defined by the Markush grouping do not share both a single structural similarity and a common use.
Claim 20 recites:
“The robot of claim 19, wherein the robot is comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system for generating or presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system implemented using a robot; a system for performing one or more conversational AI operations; a system for performing one or more generative AI operations; a system implementing one or more large language models (LLMs); a system implementing one or more vision language models (VLMs); a system implementing one or more multi-modal language models; a system for generating synthetic data; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.”
A system implemented using a robot, for example, does not share structural similarities or a common use with a system for performing light transport simulation. A perception system for an autonomous or semi-autonomous machine, for example, does not share structural similarities or a common use with a system for performing one or more conversational AI operations. Generally, each grouping of claim 18 does not relate to the structure or use of the previous grouping or following grouping of the claim.
To overcome this rejection, Applicant may set forth each alternative (or grouping of patentably indistinct alternatives) within an improper Markush grouping in a series of independent or dependent claims and/or present convincing arguments that the group members recited in the alternative within a single claim in fact share a single structural similarity as well as a common use.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C 101 because the claimed invention is directed to an abstract idea without significantly more.
Claim(s) 1, 11, and 19 is/are rejected under 35 U.S.C 101 because the claimed invention is directed to an abstract idea without significantly more. The claim(s) recite(s):
1.
A method comprising:
converting one or more sensory inputs obtained using one or more sensors of a machine into a plurality of segments;
for each segment included in the plurality of segments:
generating, via execution of a machine learning model, a caption for the segment; and
storing, in a data store, a representation of the caption in association with the segment; and
performing, by the machine, one or more actions based at least on one or more queries of the data store.
11.
At least one processor comprising:
processing circuitry to cause performance of operations comprising:
for individual segments of a plurality of segments of sensor data:
generating, via execution of a machine learning model, a descriptive caption for the individual segments; and
storing, in a data store, a representation of the descriptive caption along with time and location information;
receiving one or more requests;
generating, based at least on querying the data store, one or more responses to the one or more requests; and
causing, using one or more output devices of a robot, visual or audible presentation of the one or more responses.
19.
A robot comprising:
one or more graphics processing units (GPUs);
one or more central processing units (CPUs);
one or more hardware accelerators;
one or more sensors; and
a data store,
wherein the robot is to perform one or more operations based at least on one or more descriptive memories stored in the data store,
wherein the one or more descriptive memories are determined using sensor data obtained using the one or more sensors over one or more time intervals, and
wherein the one or more descriptive memories are stored to include at least a caption, a time, and a location associated therewith.
These limitations, as drafted, are simple processes that, under their broadest reasonable interpretation, cover performance of the mind, but for the recitation of the underlined and italicized limitations above. That is, other than reciting the underlined and italicized limitations, nothing in the claim elements preclude the steps from being performed in the mind. For example, a human can, in their mind, perform the bolded limitations recited above.
This judicial exception is not integrated into a practical application. The claim recites the additional elements underlined and italicized above. The italicized elements is/are recited at a high level of generality and merely link(s) the use of the abstract idea to a particular technological environment (see MPEP 2106.05(h)). The underlined elements is/are recited at a high level of generality and amounts to mere data gathering, manipulation, and transmission, which is a form of insignificant extra-solution activity (see MPEP 2106.05(g)). Accordingly, even in combination, the additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The additional italicized elements is/are no more than mere generic linking of the abstract idea to a technological environment, which cannot provide an inventive concept. The additional underlined elements is/are mere data gathering, manipulation, and transmission, and is a well-understood, routine, and conventional function (see MPEP 2106.05(d) and see Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93), and thus is/are no more than insignificant extra-solution activity (see MPEP 2106.05(g) and see OIP Techs., 788 F.3d at 1362-63, 115 USPQ2d at 1092-93). Thus, the limitations do not provide an inventive concept, and the claim contains ineligible subject matter.
Claim(s) 2-10, 12-18, and 20 recite(s) limitations that are no more that the abstract idea recited in claim(s) 1, 11, and 19. See the table below for a detailed bolding/italicizing/underlining of the limitations of the dependent claims.
Bolded limitations can reasonably be performed in the human mind.
Italicized elements are recited at a high level of generality to generically link the use of the abstract idea in a particular technological environment.
Underlined elements is/are mere data gathering, manipulation, and transmission, and is/are a well-understood, routine, and conventional function, and thus is/are no more than insignificant extra-solution activity. See MPEP 2106.05(g).
Thus, the claim(s) contain(s) ineligible subject matter.
2.
storing the representation of the caption in association with the segment comprises: converting, via execution of a second machine learning model, the caption into an embedding corresponding to the representation of the caption; and storing the embedding and the segment in a vector database corresponding to the data store.
3.
wherein the performing the one or more actions comprises: matching a first query included in the one or more queries to one or more segments in the data store; generating a second query included in the one or more queries based at least on the one or more segments; and determining the one or more actions based at least on one or more additional segments in the data store that are matched to the second query.
4.
wherein the generating the second query comprises inputting, into a second machine learning model, (i) a context that includes information from the one or more segments and (ii) a prompt to generate the second query based at least on the context.
5.
further comprising generating, via execution of a second machine learning model, the one or more queries based at least on a question from a user.
6.
wherein the one or more actions comprise at least one of outputting an answer to the one or more queries or navigating to a location associated with the one or more queries.
7.
wherein the one or more queries comprise at least one of a position, a time, or a description.
8.
wherein each segment included in the plurality of segments spans a time interval.
9.
wherein the machine learning model comprises a vision language model (VLM).
10.
wherein the segment comprises at least one of: one or more positions of the machine, one or more images captured by one or more cameras included in the one or more sensors, or one or more timestamps.
12.
wherein the storing the representation of the descriptive caption comprises: converting, via execution of a second machine learning model, the descriptive caption into an embedding corresponding to the representation of the descriptive caption; and storing the embedding in a vector database corresponding to the data store.
13.
wherein the at least one processor is included in the robot, on an on-premises computing system communicatively coupled to the robot, or in a remotely located data center communicatively coupled to the robot.
14.
wherein the generating the one or more responses comprises: determining a context associated with the one or more requests using the data store; and determining one of a time or a location related to the context, wherein the one or more responses are generated based at least on the context, the time, or the location.
15.
wherein the one or more responses include at least one of textual information, a position, a time, a duration, or a binary answer.
16.
wherein the descriptive caption describes perceived information corresponding to static and dynamic aspects of a scene associated with a sequence of video frames included in the individual segments.
17.
wherein the machine learning model is a vision language model (VLM) or a multi-modal language model (MMLM), and one of: the one or more responses are generated using a second machine learning model different from the machine learning model; or the one or more responses are generated using the machine learning model.
18.
wherein the at least one processor is comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system for generating or presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system implemented using a robot; a system for performing one or more conversational AI operations; a system for performing one or more generative AI operations; a system implementing one or more large language models (LLMs); a system implementing one or more vision language models (VLMs); a system implementing one or more multi-modal language models; a system for generating synthetic data; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.
20.
wherein the robot is comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system for generating or presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system implemented using a robot; a system for performing one or more conversational AI operations; a system for performing one or more generative AI operations; a system implementing one or more large language models (LLMs); a system implementing one or more vision language models (VLMs); a system implementing one or more multi-modal language models; a system for generating synthetic data; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.
Examiner Note - Prior Art
Examiner has cited particular paragraphs/columns and line numbers or figures in the references as applied to the claims below for the convenience of the applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested from the applicant, in preparing the responses, to fully consider the references in their entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the examiner. See MPEP 2141.02 [R-01.2024] VI. A prior art reference must be considered in its entirety, i.e., as a whole, including portions that would lead away from the claimed invention. W.L. Gore & Assoc., Inc. v. Garlock, Inc., 721 F.2d 1540, 220 USPQ 303 (Fed. Cir. 1983), cert. denied, 469 U.S. 851 (1984) . See also MPEP §2123. Applicant is reminded that the Examiner is entitled to give the broadest reasonable interpretation to the language of the claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1-7, 9, 11-15, 17-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Salama et al. (US 2025/0258861 A1, “Salama”) and further in view of Najmark et al. (US 2024/0248458 A1, “Najmark”).
Regarding claim 1: Salama teaches: A method comprising: ([0132] method)
converting one or more [. . .] inputs [. . .] into a plurality of segments; ([0003] input NL query and image are processed to generate sub-images (“tiles”) of input image that are relevant to NL query. [0041] input text query refer to input image. input text query refer to objects in input image. query refers to both input image and object in image. [0057] prompt engine generates VLM prompt based on input natural language query, image tiles and image facts. VLM prompt input into VLMs, which process prompt to generate response)
for each segment included in the plurality of segments: ([0003] input NL query and image are processed to generate sub-images (“tiles”) of input image that are relevant to NL query. tiles are processed by image analysis models, to generate image facts that relate to tiles, contents of tile, identities of objects/people in tile)
generating, via execution of a machine learning model, a caption for the segment; and ([0003] input NL query and image are processed to generate sub-images (“tiles”) of input image that are relevant to NL query. tiles are processed by image analysis models, to generate image facts that relate to tiles, contents of tile, identities of objects/people in tile. [0034] VQA models can process image input (image tiles) and input text query to determine/generate image facts relating to input text query. [0106] system receives plurality of image facts from image models. Examples include: objects/person identities of objects/people in tile; text extracted from tile; data extracted from graphs/charts in tile; equations, identities of equations in tile; classifications of tile; natural language descriptions of tile)
storing, in a data store, a representation of the caption in association with the segment; and ([0055]. [0124] Computing device, processor. storage, memory, user interface output and input devices. [0132] method implemented by processor(s) provided and includes receiving input query associated with client device. input query includes image and text query. generating, from input image and query, image tiles. Each image tile sub-image of input image. providing, to image analysis models, plurality of image tiles. receiving, from image analysis models, plurality of image facts. Each image fact corresponds to respective of image tiles. generating, using vision and language model, response to input query based on input query, image tiles, and image facts. causing response to input query to be rendered at client device)
performing [. . .] actions based at least on one or more queries of the data store ([0035] search engines can perform searches, text image searches, based on image facts generated from input image, input query. [0039] search request engine generates natural language search requests from input natural language query and image facts. search requests are transmitted to search engines, which perform searches based on search requests, and generate search responses. search responses are provided to prompt preparation engine, which utilizes search responses when generating prompts for VLMs, prompts for VLMs are further based on search responses).
However, Salama does not explicitly teach: sensory inputs obtained using one or more sensors of a machine; performing by the machine, one or more actions.
Najmark teaches: sensory inputs obtained using one or more sensors of a machine; ([0072]-[0074] Robotic system include sensors arranged to sense aspects of robotic system. [0040] applying set of sensor data to machine learning model (object detection model, visual language model with a query (“did anything drop on the floor”) to generate both output representation of the set of sensor data (indication of location, pose, size, identity about object(s) in sensor data) and estimate of confidence of generated output)
performing by the machine, one or more actions ([0008] operating robot to generate sensor data about environment, to navigate within environment, and to perform tasks in environment. [0055] model trained/updated based on specific environment or task to generate a ‘specific’ model that is especially adapted to specific environment or task and that can be transmitted to robots in specific environment or performing specific task).
Salama and Najmark are analogous art to the claimed invention since they are from the similar field of data processing and machine learning in a robot environment. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify the invention of Salama with the aspects of Najmark to create, with a reasonable expectation for success, a method that uses sensory inputs obtained using one or more sensors of a machine, and that performs, by the machine, one or more actions based on data store queries. The motivation for modification would have been to improve methods for selecting sensor data to be annotated and/or used to update machine learning models for a robot. These improved selection methods allow fewer, higher-quality sensor data sets to be used to train the robotic machine learning models, thereby reducing the costs of such training data (financial, computational, bandwidth, power costs) (Najmark, [0006]). This motivation for modification applies similarly to claims dependent upon this claim.
Regarding claim 2: Salama-Najmark further teach: The method of claim 1, wherein storing the representation of the caption in association with the segment comprises: converting, via execution of a second machine learning model, the caption into an embedding corresponding to the representation of the caption; and (Salama: [0106] system receives plurality of image facts from image models. Examples include: objects/person identities of objects/people in tile; text extracted from tile; data extracted from graphs/charts in tile; equations, identities of equations in tile; classifications of tile; natural language descriptions of tile. Najmark: [0057]-[0060] embedding vectors and vector database. [0061] such embedding vectors could be used to ‘mine’ stored sets of previously-obtained sensor data. For example, embedding vectors could be determined for sets of sensor data that are stored in a database (e.g., that were previously obtained by one or more robots and then stored for future use). The determined embedding vectors could then be compared to the embedding vector(s) of previously-selected sets of sensor data (e.g., by determine L1 or L2 distances therebetween). Sets of sensor data in the database whose embedding vectors are sufficiently similar (e.g., more similar than a threshold level) to one or more of the bedding vector(s) of previously-selected sets of sensor data could themselves be selected for human annotation and/or use in training and/or updating machine learning models. The machine learning models generated based on such selected database-stored sets of sensor data could then be transmitted to one or more robots to improve their operation)
storing the embedding and the segment in a vector database corresponding to the data store (Najmark: [0057]-[0060] embedding vectors and vector database. [0061] such embedding vectors could be used to ‘mine’ stored sets of previously-obtained sensor data. For example, embedding vectors could be determined for sets of sensor data that are stored in a database (e.g., that were previously obtained by one or more robots and then stored for future use). The determined embedding vectors could then be compared to the embedding vector(s) of previously-selected sets of sensor data (e.g., by determine L1 or L2 distances therebetween). Sets of sensor data in the database whose embedding vectors are sufficiently similar (e.g., more similar than a threshold level) to one or more of the bedding vector(s) of previously-selected sets of sensor data could themselves be selected for human annotation and/or use in training and/or updating machine learning models. The machine learning models generated based on such selected database-stored sets of sensor data could then be transmitted to one or more robots to improve their operation).
The motivation for modification would have been to improve methods for selecting sensor data to be annotated and/or used to update machine learning models for a robot. These improved selection methods allow fewer, higher-quality sensor data sets to be used to train the robotic machine learning models, thereby reducing the costs of such training data (financial, computational, bandwidth, power costs) (Najmark, [0006]).
Regarding claim 3: Salama-Najmark further teach: The method of claim 1, wherein the performing the one or more actions comprises: matching a first query included in the one or more queries to one or more segments in the data store; generating a second query included in the one or more queries based at least on the one or more segments; and determining the one or more actions based at least on one or more additional segments in the data store that are matched to the second query (Salama: [0059]-[0068] example of classroom, VLM prompt generated: “Please reply in polite and helpful manner. Query: what topic is teacher teaching?; [Image tile 1- blackboard]; Image fact: [Equation] Group axioms; [Image tile 2]; Image fact: [Text] Elementary Group Theory; Image fact: [Image search] Undergraduate mathematics textbook. VLM(s) generates corresponding response by processing input prompt. VLM(s) generates response “The teacher is teaching undergraduate group theory using textbook “Elementary Group Theory”. response is then rendered at user application, as text in text-based dialogue/chat application, converted to speech using text-to-speech engine, or like. See also [0075]-[0084]).
Regarding claim 4: Salama-Najmark further teach: The method of claim 3, wherein the generating the second query comprises inputting, into a second machine learning model, (i) a context that includes information from the one or more segments and (ii) a prompt to generate the second query based at least on the context (Salama: [0026] tiling engine can utilize one or more VLMs to generate plurality of tiles. one or more VLMs can be trained/fine-tuned to generate relevant image tiles from input image and text query. [0041] explicit text query is “what are the contents of the toolbox in this image?”. refers to both input image and object in image explicitly. [0055] determine a tile classification, and determine one or more of the image analysis models to send the tile to for analysis. pretrained image classification model can be used to classify an image tile into one of a plurality of classes. Each class is associated with one or more image analysis models).
Regarding claim 5: Salama-Najmark further teach: The method of claim 1, further comprising generating, via execution of a second machine learning model, the one or more queries based at least on a question from a user (Salama: [0026] tiling engine can utilize one or more VLMs to generate plurality of tiles. one or more VLMs can be trained/fine-tuned to generate relevant image tiles from input image and text query. [0041] explicit text query is “what are the contents of the toolbox in this image?”. refers to both input image and object in image explicitly).
Regarding claim 6: Salama-Najmark further teach: The method of claim 1, wherein the one or more actions comprise at least one of outputting an answer to the one or more queries or navigating to a location associated with the one or more queries (Salama: [0092] prompt “Which part of the image is related to this text extract: a white Tesla” along with the image. VLM outputs data indicating location of bounding box that contains white Tesla, and provides data to client application. client application renders bounding box on input image, as an overlay. [0108] VLM processes prompt based on input query (input natural language query and input image), plurality of image tiles and plurality of image facts to generate a natural language (text) response, and outputs response).
Regarding claim 7: Salama-Najmark further teach: The method of claim 1, wherein the one or more queries comprise at least one of a position, a time, or a description (Salama: [0041] input text query refer to objects in input image. explicit text query “what are contents of toolbox in this image?”. query refers to both input image (“this image”) and object (“the toolbox”) in image. implicit text query “what in it?”. query refers to image and object in image (“it”) implicitly, by virtue of being submitted with input image. [0075]-[0082] input query may contain picture of street containing number of cars, and natural language query “what make is the red car?” tiling engine generates plurality of image tiles based on natural language query, where plurality of image tiles contain respective car in image).
Regarding claim 9: Salama-Najmark further teach: The method of claim 1, wherein the machine learning model comprises a vision language model (VLM) (Salama: [0003] responding to input query, comprising natural language (NL) query and image, using vision and language model (VLM)).
Regarding claim 11: Salama teaches: At least one processor comprising: ([0124] Computing device, processor)
processing circuitry to cause performance of operations comprising: ([0094], [0111] computing device, processors, memory)
for individual segments of a plurality of segments of [. . .] data: ([0003] input NL query and image are processed to generate sub-images (“tiles”) of input image that are relevant to NL query. [0041] input text query refer to input image. input text query refer to objects in input image. query refers to both input image and object in image. [0057] prompt engine generates VLM prompt based on input natural language query, image tiles and image facts. VLM prompt input into VLMs, which process prompt to generate response)
generating, via execution of a machine learning model, a descriptive caption for the individual segments; and ([0003] input NL query and image are processed to generate sub-images (“tiles”) of input image that are relevant to NL query. tiles are processed by image analysis models, to generate image facts that relate to tiles, contents of tile, identities of objects/people in tile. [0034] VQA models can process image input (image tiles) and input text query to determine/generate image facts relating to input text query. [0106] system receives plurality of image facts from image models. Examples include: objects/person identities of objects/people in tile; text extracted from tile; data extracted from graphs/charts in tile; equations, identities of equations in tile; classifications of tile; natural language descriptions of tile)
storing, in a data store, a representation of the descriptive caption [. . .]; ([0055]. [0124] Computing device, processor. storage, memory, user interface output and input devices. [0132] method implemented by processor(s) provided and includes receiving input query associated with client device. input query includes image and text query. generating, from input image and query, image tiles. Each image tile sub-image of input image. providing, to image analysis models, plurality of image tiles. receiving, from image analysis models, plurality of image facts. Each image fact corresponds to respective of image tiles. generating, using vision and language model, response to input query based on input query, image tiles, and image facts. causing response to input query to be rendered at client device)
receiving one or more requests; ([0035] image facts generated from input image, input query. [0039] natural language search requests from input natural language query and image facts. perform searches based on search requests, and generate search responses, which utilizes search responses when generating prompts for VLMs)
generating, based at least on querying the data store, one or more responses to the one or more requests; and ([0035] search engines can perform searches, text image searches, based on image facts generated from input image, input query. [0039] search request engine generates natural language search requests from input natural language query and image facts. search requests are transmitted to search engines, which perform searches based on search requests, and generate search responses. search responses are provided to prompt preparation engine, which utilizes search responses when generating prompts for VLMs, prompts for VLMs are further based on search responses)
causing, [. . .], visual or audible presentation of the one or more responses ([0035] search engines can perform searches, text image searches, based on image facts generated from input image, input query. [0039] search request engine generates natural language search requests from input natural language query and image facts. search requests are transmitted to search engines, which perform searches based on search requests, and generate search responses. search responses are provided to prompt preparation engine, which utilizes search responses when generating prompts for VLMs, prompts for VLMs are further based on search responses. [0110] system causes response to input prompt to be rendered at the client device. response rendered graphically in interface of application of client device via which query was submitted. response audibly rendered via speakers of client device via which query was submitted).
However, Salama does not explicitly teach: sensor data; descriptive caption along with time and location information; causing, using one or more output devices of a robot, responses.
Najmark teaches: sensor data; ([0072]-[0074] Robotic system include sensors arranged to sense aspects of robotic system. [0040] applying set of sensor data to machine learning model (object detection model, visual language model with a query (“did anything drop on the floor”) to generate both output representation of the set of sensor data (indication of location, pose, size, identity about object(s) in sensor data) and estimate of confidence of generated output)
descriptive caption along with time and location information; ([0039] sensor data sets taken proximate in time to the failure of the tasks, sensor data sets taken proximate in time to beginning the tasks, sensor data sets taken proximate in time to a time when the robot determined that the tasks should be attempted, sensor data sets taken proximate in time to a time when the robot began interacting with a target of the tasks (when the robot began interacting with a target object using an end-of-arm-system), or some other time relevant to the beginning, progression, or end of the failed tasks. [0046] if ‘bottles’ of FIG. 7A are such type of novel object, then classifier outputs corresponding to whether image or a portion thereof (box indicating location and extent of detected discrete object within image, tile or other regularly spaced segment of image) represents ‘bottle’ could be compared to threshold. [0028] receive sensor data and output information about the structure (presence and location of obstacles, pattern of navigable areas, location of doors or egress points) or contents (location, pose, size, geometry, identity of objects) of the environment)
causing, using one or more output devices of a robot, responses ([0008] operating robot to generate sensor data about environment, to navigate within environment, and to perform tasks in environment. [0055] model trained/updated based on specific environment or task to generate a ‘specific’ model that is especially adapted to specific environment or task and that can be transmitted to robots in specific environment or performing specific task).
Salama and Najmark are analogous art to the claimed invention since they are from the similar field of data processing and machine learning in a robot environment. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify the invention of Salama with the aspects of Najmark to create, with a reasonable expectation for success, a method that uses sensory inputs obtained using one or more sensors of a machine including time and location information, and that performs, by the machine, one or more actions based on data store queries. The motivation for modification would have been to improve methods for selecting sensor data to be annotated and/or used to update machine learning models for a robot. These improved selection methods allow fewer, higher-quality sensor data sets to be used to train the robotic machine learning models, thereby reducing the costs of such training data (financial, computational, bandwidth, power costs) (Najmark, [0006]). This motivation for modification applies similarly to claims dependent upon this claim.
Regarding claim 12: Salama-Najmark further teach: The at least one processor of claim 11, wherein the storing the representation of the descriptive caption comprises: converting, via execution of a second machine learning model, the descriptive caption into an embedding corresponding to the representation of the descriptive caption; and (Salama: [0106] system receives plurality of image facts from image models. Examples include: objects/person identities of objects/people in tile; text extracted from tile; data extracted from graphs/charts in tile; equations, identities of equations in tile; classifications of tile; natural language descriptions of tile. Najmark: [0057]-[0060] embedding vectors and vector database. [0061] such embedding vectors could be used to ‘mine’ stored sets of previously-obtained sensor data. For example, embedding vectors could be determined for sets of sensor data that are stored in a database (e.g., that were previously obtained by one or more robots and then stored for future use). The determined embedding vectors could then be compared to the embedding vector(s) of previously-selected sets of sensor data (e.g., by determine L1 or L2 distances therebetween). Sets of sensor data in the database whose embedding vectors are sufficiently similar (e.g., more similar than a threshold level) to one or more of the bedding vector(s) of previously-selected sets of sensor data could themselves be selected for human annotation and/or use in training and/or updating machine learning models. The machine learning models generated based on such selected database-stored sets of sensor data could then be transmitted to one or more robots to improve their operation)
storing the embedding in a vector database corresponding to the data store (Najmark: [0057]-[0060] embedding vectors and vector database. [0061] such embedding vectors could be used to ‘mine’ stored sets of previously-obtained sensor data. For example, embedding vectors could be determined for sets of sensor data that are stored in a database (e.g., that were previously obtained by one or more robots and then stored for future use). The determined embedding vectors could then be compared to the embedding vector(s) of previously-selected sets of sensor data (e.g., by determine L1 or L2 distances therebetween). Sets of sensor data in the database whose embedding vectors are sufficiently similar (e.g., more similar than a threshold level) to one or more of the bedding vector(s) of previously-selected sets of sensor data could themselves be selected for human annotation and/or use in training and/or updating machine learning models. The machine learning models generated based on such selected database-stored sets of sensor data could then be transmitted to one or more robots to improve their operation).
The motivation for modification would have been to improve methods for selecting sensor data to be annotated and/or used to update machine learning models for a robot. These improved selection methods allow fewer, higher-quality sensor data sets to be used to train the robotic machine learning models, thereby reducing the costs of such training data (financial, computational, bandwidth, power costs) (Najmark, [0006]).
Regarding claim 13: Salama-Najmark further teach: The at least one processor of claim 11, wherein the at least one processor is included in the robot, on an on-premises computing system communicatively coupled to the robot, or in a remotely located data center communicatively coupled to the robot (Salama: [0016] all or aspects of NL based response system implemented locally at client device. all or some aspects of NL based response system can be implemented remotely from client device as depicted in FIG. 1 (at remote server(s)). client device and NL based response system can be communicatively coupled with each other via one or more networks, such as one or more wired or wireless local area networks. [0023] other implementations one or more of software applications can be hosted remotely (by one or more servers) and can be accessible by the client device over one or more of the networks. Najmark: [0034] methods herein could be applied to a set of sensor data obtained by robot (applied by on-board controller of robot) and set of sensor data could be transmitted to remote system).
Regarding claim 14: Salama-Najmark further teach: The at least one processor of claim 11, wherein the generating the one or more responses comprises: determining a context associated with the one or more requests using the data store; and (Salama: [0106] system receives plurality of image facts from image models. Examples include: objects/person identities of objects/people in tile; text extracted from tile; data extracted from graphs/charts in tile; equations, identities of equations in tile; classifications of tile; natural language descriptions of tile. [0041] input text query refer to objects in input image. explicit text query “what are contents of toolbox in this image?”. query refers to both input image (“this image”) and object (“the toolbox”) in image. implicit text query “what in it?”. query refers to image and object in image (“it”) implicitly, by virtue of being submitted with input image. [0075]-[0082] input query may contain picture of street containing number of cars, and natural language query “what make is the red car?” tiling engine generates plurality of image tiles based on natural language query, where plurality of image tiles contain respective car in image. prompt preparation engine prepares VLM prompt based on image and natural language query, and image facts generated from image tiles. preparation engine prepares prompt: “Please reply in polite and helpful manner. Query: what make red car?; [Image tile - red car]; Image fact: [Object] Red BMW car; [Image tile - green car]; Image fact: [Object] Green VW car; [Image tile - white car]; Image fact: [Object] White Tesla).
determining one of a time or a location related to the context, wherein the one or more responses are generated based at least on the context, the time, or the location (Salama: [0041] input text query refer to objects in input image. explicit text query “what are contents of toolbox in this image?”. query refers to both input image (“this image”) and object (“the toolbox”) in image. implicit text query “what in it?”. query refers to image and object in image (“it”) implicitly, by virtue of being submitted with input image. [0075]-[0082] input query may contain picture of street containing number of cars, and natural language query “what make is the red car?” tiling engine generates plurality of image tiles based on natural language query, where plurality of image tiles contain respective car in image. prompt preparation engine prepares VLM prompt based on image and natural language query, and image facts generated from image tiles. preparation engine prepares prompt: “Please reply in polite and helpful manner. Query: what make red car?; [Image tile - red car]; Image fact: [Object] Red BMW car; [Image tile - green car]; Image fact: [Object] Green VW car; [Image tile - white car]; Image fact: [Object] White Tesla).
Regarding claim 15: Salama-Najmark further teach: The at least one processor of claim 11, wherein the one or more responses include at least one of textual information, a position, a time, a duration, or a binary answer (Salama: [0106] system receives plurality of image facts from image models. Examples include: objects/person identities of objects/people in tile; text extracted from tile; data extracted from graphs/charts in tile; equations, identities of equations in tile; classifications of tile; natural language descriptions of tile. [0041] input text query refer to objects in input image. explicit text query “what are contents of toolbox in this image?”. query refers to both input image (“this image”) and object (“the toolbox”) in image. implicit text query “what in it?”. query refers to image and object in image (“it”) implicitly, by virtue of being submitted with input image. See also [0075]-[0082]).
Regarding claim 17: Salama-Najmark further teach: The at least one processor of claim 11, wherein the machine learning model is a vision language model (VLM) or a multi-modal language model (MMLM), and one of: (Salama: [0003] responding to input query, comprising natural language (NL) query and image, using vision and language model (VLM))
the one or more responses are generated using a second machine learning model different from the machine learning model; or the one or more responses are generated using the machine learning model (Salama: [0026] tiling engine can utilize one or more VLMs to generate plurality of tiles. one or more VLMs can be trained/fine-tuned to generate relevant image tiles from input image and text query. [0041] explicit text query is “what are the contents of the toolbox in this image?”. refers to both input image and object in image explicitly).
Regarding claim 18: Salama-Najmark further teach: The at least one processor of claim 11, wherein the at least one processor is comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system for generating or presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system implemented using a robot; a system for performing one or more conversational AI operations; a system for performing one or more generative AI operations; a system implementing one or more large language models (LLMs); a system implementing one or more vision language models (VLMs); a system implementing one or more multi-modal language models; a system for generating synthetic data; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources (Salama: [0017] client device can be: computing device of a vehicle, virtual or augmented reality computing device. Najmark: [0062] Robotic system configured to operate autonomously, semi-autonomously). The motivation for modification would have been to improve methods for selecting sensor data to be annotated and/or used to update machine learning models for a robot. These improved selection methods allow fewer, higher-quality sensor data sets to be used to train the robotic machine learning models, thereby reducing the costs of such training data (financial, computational, bandwidth, power costs) (Najmark, [0006]).
Claim(s) 8, 10, 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Salama et al. (US 2025/0258861 A1, “Salama”) in view of Najmark et al. (US 2024/0248458 A1, “Najmark”) and further in view of Naphade et al. (US 20220165304 A1, “Naphade”).
Regarding claim 8: Salama-Najmark further teach: The method of claim 1. However, Salama-Najmark do not explicitly teach: wherein each segment included in the plurality of segments spans a time interval.
Naphade teaches: wherein each segment included in the plurality of segments spans a time interval ([0023] processes continuous data stream acquired by sensors to produce detection data corresponding to at least one time segment in data stream. [0024] frame of video data using timestamp corresponding to time segment. detection data for frame of video data with specific timestamp and may synchronize frame and detection data by associating detection data with that timestamp. timestamp associated with detection data used to extract clip from rolling buffer. [0032] extraction trigger defines portion of continuous data stream stored in rolling buffer to be extracted to generate clip. portion is associated with time segment and specified by time stamp, frame number).
Salama-Najmark and Naphade are analogous art to the claimed invention since they are from the similar field of data processing and machine learning in a robot environment. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify the invention of Salama-Najmark with the aspects of Naphade to create, with a reasonable expectation for success, a method that uses sensory segments wherein each segment included in the plurality of segments spans a time interval. The motivation for modification would have been to improve the overall efficiency of the system by streamlining data storage and organization (Naphade, [0016]).
Regarding claim 10: Salama-Najmark further teach: The method of claim 1. However, Salama-Najmark do not explicitly teach: wherein the segment comprises at least one of: one or more positions of the machine, one or more images captured by one or more cameras included in the one or more sensors, or one or more timestamps.
Naphade teaches: wherein the segment comprises at least one of: one or more positions of the machine, one or more images captured by one or more cameras included in the one or more sensors, or one or more timestamps ([0018] camera feeds in real-time. [0023] processes continuous data stream acquired by sensors to produce detection data corresponding to at least one time segment in data stream. indicate presence of object, occurrence, or instance in data stream during time segment. include classification, object segmentation, and position data. indicate time segments that correspond to objects in video stream, sounds in audio stream. location and attributes (color, shape) of identified objects. [0024] annotations (bounding boxes, labels, or coordinates) included in detection data could represent location(s) or dimensions of object or other attributes of identified object in video. frame of video data using timestamp corresponding to time segment. detection data for frame of video data with specific timestamp and may synchronize frame and detection data by associating detection data with that timestamp. timestamp associated with detection data used to extract clip from rolling buffer. [0032] extraction trigger defines portion of continuous data stream stored in rolling buffer to be extracted to generate clip. portion is associated with time segment and specified by time stamp, frame number).
Salama-Najmark and Naphade are analogous art to the claimed invention since they are from the similar field of data processing and machine learning in a robot environment. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify the invention of Salama-Najmark with the aspects of Naphade to create, with a reasonable expectation for success, a method that uses sensory segments wherein the segment comprises at least one of: one or more positions of the machine, one or more images captured by one or more cameras included in the one or more sensors, or one or more timestamps. The motivation for modification would have been to improve the overall efficiency of the system by streamlining data storage and organization (Naphade, [0016]).
Regarding claim 16: Salama-Najmark further teach: The at least one processor of claim 11. However, Salama-Najmark do not explicitly teach: wherein the descriptive caption describes perceived information corresponding to static and dynamic aspects of a scene associated with a sequence of video frames included in the individual segments.
Naphade teaches: wherein the descriptive caption describes perceived information corresponding to static and dynamic aspects of a scene associated with a sequence of video frames included in the individual segments ([0023] processes continuous data stream acquired by sensors to produce detection data corresponding to at least one time segment in data stream. indicate presence of object, occurrence, or instance in data stream during time segment. include classification, object segmentation, and position data. indicate time segments that correspond to objects in video stream, sounds in audio stream. location and attributes (color, shape) of identified objects. [0024] annotations (bounding boxes, labels, or coordinates) included in detection data could represent location(s) or dimensions of object or other attributes of identified object in video. frame of video data using timestamp corresponding to time segment. detection data for frame of video data with specific timestamp and may synchronize frame and detection data by associating detection data with that timestamp. timestamp associated with detection data used to extract clip from rolling buffer. [0032] extraction trigger defines portion of continuous data stream stored in rolling buffer to be extracted to generate clip. portion is associated with time segment and specified by time stamp, frame number).
Salama-Najmark and Naphade are analogous art to the claimed invention since they are from the similar field of data processing and machine learning in a robot environment. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify the invention of Salama-Najmark with the aspects of Naphade to create, with a reasonable expectation for success, a method that uses sensory data and descriptive captions wherein the descriptive caption describes perceived information corresponding to static and dynamic aspects of a scene associated with a sequence of video frames included in the individual segments. The motivation for modification would have been to improve the overall efficiency of the system by streamlining data storage and organization (Naphade, [0016]).
Claim(s) 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Salama et al. (US 2025/0258861 A1, “Salama”) and further in view of Najmark et al. (US 2024/0248458 A1, “Najmark”) and Naphade et al. (US 20220165304 A1, “Naphade”).
Regarding claim 19: Salama teaches: A [device] comprising: ([0124] Computing device, processor)
one or more graphics processing units (GPUs); one or more central processing units (CPUs); one or more hardware accelerators; ([0150] Some implementations include an apparatus comprising one or more processors (e.g., CPU(s), GPU(s), and/or TPU(s)) and a memory) [. . .]
a data store, ([0055]. [0124] Computing device, processor. storage, memory, user interface output and input devices)
[perform operations] based at least on one or more descriptive memories stored in the data store, ([0035] search engines can perform searches, text image searches, based on image facts generated from input image, input query. [0039] search request engine generates natural language search requests from input natural language query and image facts. search requests are transmitted to search engines, which perform searches based on search requests, and generate search responses. search responses are provided to prompt preparation engine, which utilizes search responses when generating prompts for VLMs, prompts for VLMs are further based on search responses) [. . .]
wherein the one or more descriptive memories are stored to include at least a caption [. . .] associated therewith ([0003] input NL query and image are processed to generate sub-images (“tiles”) of input image that are relevant to NL query. tiles are processed by image analysis models, to generate image facts that relate to tiles, contents of tile, identities of objects/people in tile. [0034] VQA models can process image input (image tiles) and input text query to determine/generate image facts relating to input text query. [0106] system receives plurality of image facts from image models. Examples include: objects/person identities of objects/people in tile; text extracted from tile; data extracted from graphs/charts in tile; equations, identities of equations in tile; classifications of tile; natural language descriptions of tile).
However, Salama does not explicitly teach: a robot comprising; one or more sensors; and wherein the robot is to perform one or more operations; wherein the one or more descriptive memories are determined using sensor data obtained using the one or more sensors over one or more time intervals, and wherein the one or more descriptive memories are stored to include at least [. . .] a time, and a location associated therewith.
Najmark teaches: a robot comprising; one or more sensors; and ([0072]-[0074] Robotic system include sensors arranged to sense aspects of robotic system. [0040] applying set of sensor data to machine learning model (object detection model, visual language model with a query (“did anything drop on the floor”) to generate both output representation of the set of sensor data (indication of location, pose, size, identity about object(s) in sensor data) and estimate of confidence of generated output)
wherein the robot is to perform one or more operations; ([0008] operating robot to generate sensor data about environment, to navigate within environment, and to perform tasks in environment. [0055] model trained/updated based on specific environment or task to generate a ‘specific’ model that is especially adapted to specific environment or task and that can be transmitted to robots in specific environment or performing specific task) [. . .]
wherein the one or more descriptive memories are stored to include at least a time, and a location associated therewith ([0039] sensor data sets taken proximate in time to the failure of the tasks, sensor data sets taken proximate in time to beginning the tasks, sensor data sets taken proximate in time to a time when the robot determined that the tasks should be attempted, sensor data sets taken proximate in time to a time when the robot began interacting with a target of the tasks (when the robot began interacting with a target object using an end-of-arm-system), or some other time relevant to the beginning, progression, or end of the failed tasks. [0046] if ‘bottles’ of FIG. 7A are such type of novel object, then classifier outputs corresponding to whether image or a portion thereof (box indicating location and extent of detected discrete object within image, tile or other regularly spaced segment of image) represents ‘bottle’ could be compared to threshold. [0028] receive sensor data and output information about the structure (presence and location of obstacles, pattern of navigable areas, location of doors or egress points) or contents (location, pose, size, geometry, identity of objects) of the environment)
Salama and Najmark are analogous art to the claimed invention since they are from the similar field of data processing and machine learning in a robot environment. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify the invention of Salama with the aspects of Najmark to create, with a reasonable expectation for success, a robot that uses sensory inputs obtained using one or more sensors of a machine including time and location information, and that performs, by the robot, one or more actions based on data store queries. The motivation for modification would have been to improve methods for selecting sensor data to be annotated and/or used to update machine learning models for a robot. These improved selection methods allow fewer, higher-quality sensor data sets to be used to train the robotic machine learning models, thereby reducing the costs of such training data (financial, computational, bandwidth, power costs) (Najmark, [0006]). This motivation for modification applies similarly to claims dependent upon this claim.
However, Salama-Najmark does not explicitly teach: wherein the one or more descriptive memories are determined using sensor data obtained using the one or more sensors over one or more time intervals.
Naphade teaches: wherein the one or more descriptive memories are determined using sensor data obtained using the one or more sensors over one or more time intervals ([0018] camera feeds in real-time. [0023] processes continuous data stream acquired by sensors to produce detection data corresponding to at least one time segment in data stream. indicate presence of object, occurrence, or instance in data stream during time segment. include classification, object segmentation, and position data. indicate time segments that correspond to objects in video stream, sounds in audio stream. location and attributes (color, shape) of identified objects. [0024] annotations (bounding boxes, labels, or coordinates) included in detection data could represent location(s) or dimensions of object or other attributes of identified object in video. frame of video data using timestamp corresponding to time segment. detection data for frame of video data with specific timestamp and may synchronize frame and detection data by associating detection data with that timestamp. timestamp associated with detection data used to extract clip from rolling buffer. [0032] extraction trigger defines portion of continuous data stream stored in rolling buffer to be extracted to generate clip. portion is associated with time segment and specified by time stamp, frame number).
Salama-Najmark and Naphade are analogous art to the claimed invention since they are from the similar field of data processing and machine learning in a robot environment. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify the invention of Salama-Najmark with the aspects of Naphade to create, with a reasonable expectation for success, a robot with sensor data and descriptive memories wherein the one or more descriptive memories are determined using sensor data obtained using the one or more sensors over one or more time intervals. The motivation for modification would have been to improve the overall efficiency of the system by streamlining data storage and organization (Naphade, [0016]).
Regarding claim 20: Salama-Najmark-Naphade further teach: The robot of claim 19, wherein the robot is comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system for generating or presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system implemented using a robot; a system for performing one or more conversational AI operations; a system for performing one or more generative AI operations; a system implementing one or more large language models (LLMs); a system implementing one or more vision language models (VLMs); a system implementing one or more multi-modal language models; a system for generating synthetic data; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources (Salama: [0017] client device can be: computing device of a vehicle, virtual or augmented reality computing device. Najmark: [0062] Robotic system configured to operate autonomously, semi-autonomously). The motivation for modification would have been to improve methods for selecting sensor data to be annotated and/or used to update machine learning models for a robot. These improved selection methods allow fewer, higher-quality sensor data sets to be used to train the robotic machine learning models, thereby reducing the costs of such training data (financial, computational, bandwidth, power costs) (Najmark, [0006]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MADISON B EMMETT whose telephone number is (303)297-4231. The examiner can normally be reached Monday - Friday 9:00 - 5:00 ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tommy Worden can be reached at (571)272-4876. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MADISON B EMMETT/Examiner, Art Unit 3658
Prosecution Insights