DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Toms U.S. PAP 2025/0139845 A1 in view of Wu U.S. PAP 20250200283 A1.
Regarding claim 1 Toms teaches a method performed in a computing system for generating structured data for use in mixed reality applications (systems and methods of learning object relationships and properties from multivariate content for improved context-based AR, see par. [0015]), the method comprising: receiving unstructured data including instructional content (Content may be accessed from various content sources to learn augmented unification (AU) data, see par. [0015]; content may be structured and/or unstructured, see par. [0052]); receiving schema input specifying a first schema to which the unstructured data is to conform (data acquisition subsystem 120 may use a topology/relationship schema specifically to house the learned properties and relationships. The schema establishes a structured framework for capturing and representing the learned properties of, and relationships between, objects analyzed from the acquired content, see par. [0057]); establishing a system task prompt based on the unstructured instructional content and the schema input (object learning subsystem 130 may access content, such as content acquired by the data acquisition subsystem 120, and learn properties of and relationships among data objects in the content. A data object is a representation of an object. The representation can include text such as the word “bell” that represents a bell, an image such as an image of the bell, audio such as a sound of a bell, and/or other types of representations that can be included in the content. The object learning subsystem 130 may identify the data object from the content. , see par. [0058]); submitting the system task prompt to a large language model (LLM) (Learned properties and relationships of objects may also enable generative AI systems that use Large Language Models (LLMs) to use programmatically configured inputs, such as prompts, for relevant content generation, see par. [0048]); receiving proposed structured data from the LLM (object learning subsystem 130 may use topic modeling to identify and extract relevant topics or themes from the content. Topic modeling is a statistical technique that may identify topics in the content, see par. [0060]); and transforming the structured data into a procedure usable in a mixed reality application ( named entity is a word or phrase that refers to specific object. This allows for a more precise recognition and correlation of objects within the augmented reality environment, see par. [0061]).
However Toms does not teach parsing the proposed structured data into structured data according to the first schema.
In the same field of endeavor Wu teaches a graphical user interface 610 and content manager 612 for use in providing, synthesizing, rendering, compositing, modifying, or using content for presentation (or other purposes) on or by the client device 602. The content application 604 can also include a language module 614 that can perform various generating tasks, such as to update or augment a text-based representation, see par. [0138]. An LLM 516 trained with an RTL (or other DSL) corpus built from a database 512 of map ground truth data can be queried to correct features output from a machine learning (ML) automation pipeline. The output of an LLM 516—such as by using a writer and/or parser component or module 520—can be mapped back to the extracted features, see par. [0118]. An RTL API can provide a public interface that lets developers generate structured language documents, and also allows users to query a trained model. As described herein, a schema can be provided to tokenize the edge sequence path, see par. [0131].
It would have been obvious to one of ordinary skill in the art to combine the Toms invention with the teachings of Wu in order to perform various generating tasks, such as to update or augment a text-based representation, see par. [0138].
Regarding claim 2 Toms teaches the method of claim 1 wherein creating the system task prompt comprises: receiving preliminary input specifying a command to be followed by the LLM in generating the proposed structured data (Large Language Models (LLMs) to use programmatically configured inputs, such as prompts, for relevant content generation, see par. [0048]); and creating the system task prompt based on the preliminary input (the system may be used to generate informative context-based prompts based on semantic and other relationships between various objects, see par. [0048]).
Regarding claim 3 Wu teaches the method of claim 1, further comprising: receiving supplemental input specifying a command to be followed by the LLM in revising the structured data ( The language model may also take other inputs as well, such as prior maps or context information , see par. [0038]); creating a supplemental prompt based on the supplemental input (he input data can be represented by embeddings, feature vectors, or points in a latent space, which allows for relatively simple searching for similar environments, see par. [0038]); submitting the supplemental prompt to the LLM (The generative AI model might take the sensor data directly as input or might receive input that is generated from the sensor data in one or more stages, see par. [0038]); and receiving revised structured data from the LLM ( correct or update noisy or partial environment graphs or maps, see par. [0038]).
Regarding claim 4 Toms teaches the method of claim 1, further comprising: presenting to a viewer a mixed reality experience based on the procedure (generating an AR display in which one or more virtual objects are to be overlaid onto a real world (RW) environment, see par. [0105]); receiving from the viewer a query regarding the mixed reality experience (o cause the interaction, the processor is programmed to do so in response to an input from a user, see claim 3); creating a viewer prompt based on the query and the procedure (defining a behavior of the virtual object with respect to the physical environment based on the AU object, see par. [0108]); submitting the viewer prompt to the LLM (receiving a virtual object to augment the electronic display, see par. [0109]); receiving, from the LLM, a response based on the viewer prompt (updating the electronic display to include the virtual object, see par. [0110]); and presenting to the viewer, by the mixed reality experience, a virtual artifact based on the response (causing an interaction between the physical object and the virtual object based on the one or more permissible actions to be displayed in the electronic display, see par. [0111]).
Regarding claim 5 Toms teaches the method of claim 1, further comprising: presenting to a viewer a mixed reality experience based on the procedure (causing an interaction between the physical object and the virtual object based on the one or more permissible actions to be displayed in the electronic display, see par. [0111]).
Regarding claim 6 Toms teaches the method of claim 1, further comprising: receiving input specifying one or more steps in the structured data to include in the procedure (Structured content is data that is organized in a predefined format that can be used to search and extract data, see par. [0052]); and including the specified one or more steps in the procedure (context-based AU system 110 may acquire content from the data sources 101, learn relationships between data objects based on the content, and store and transmit the relationships via an API 152 and/or one or more end user applications , see par. [0053]).
Regarding claim 7 Wu teaches the method of claim 1, further comprising: detecting an error in the parsing (Generative AI can be trained in such a way as to be able to fill in gaps or correct errors in the sensor data based on a semantic understanding of the objects , see par. [0038]); receiving correction input specifying a correction to the system task prompt (fill in gaps or correct errors in the sensor data based on a semantic understanding of the objects , see par. [0038]); creating a corrected prompt based on the system task prompt and the correction input (correct or update noisy or partial environment graphs or maps, see par. [0038]); submitting the corrected prompt to the LLM (The aligned map data and at least a subset of the set of observations can then be provided 456 as input to a trained language model, see par. [0105]); and receiving, from the LLM, a response based on the corrected prompt (tokenized description can then be received 458 as output of the trained language model(tokenized description can then be received 458 as output of the trained language model, see par. [0105]). Regarding claim 8 Toms teaches a system for generating structured data for use in mixed reality applications (systems and methods of learning object relationships and properties from multivariate content for improved context-based AR, see par. [0015]), the system comprising: one or more memories configured to collectively store computer instructions (system memory, see par. [0115]); and one or more processors configured to collectively execute the stored computer (processor, see par. [0115]) instructions to: .
receive unstructured data including instructional content (Content may be accessed from various content sources to learn augmented unification (AU) data, see par. [0015]; content may be structured and/or unstructured, see par. [0052]); establish a schema to which the unstructured data is to conform (data acquisition subsystem 120 may use a topology/relationship schema specifically to house the learned properties and relationships. The schema establishes a structured framework for capturing and representing the learned properties of, and relationships between, objects analyzed from the acquired content, see par. [0057]); establish a system task prompt based on the unstructured instructional content and the schema (object learning subsystem 130 may access content, such as content acquired by the data acquisition subsystem 120, and learn properties of and relationships among data objects in the content. A data object is a representation of an object. The representation can include text such as the word “bell” that represents a bell, an image such as an image of the bell, audio such as a sound of a bell, and/or other types of representations that can be included in the content. The object learning subsystem 130 may identify the data object from the content. , see par. [0058]); submit the system task prompt to a large language model (LLM) (Learned properties and relationships of objects may also enable generative AI systems that use Large Language Models (LLMs) to use programmatically configured inputs, such as prompts, for relevant content generation, see par. [0048]); receive proposed structured data from the LLM (object learning subsystem 130 may use topic modeling to identify and extract relevant topics or themes from the content. Topic modeling is a statistical technique that may identify topics in the content, see par. [0060]); and construct, using the structured data, a procedure usable in a mixed reality application ( named entity is a word or phrase that refers to specific object. This allows for a more precise recognition and correlation of objects within the augmented reality environment, see par. [0061]).
However Toms does not teach parsing the proposed structured data into structured data according to the first schema.
In the same field of endeavor Wu teaches a graphical user interface 610 and content manager 612 for use in providing, synthesizing, rendering, compositing, modifying, or using content for presentation (or other purposes) on or by the client device 602. The content application 604 can also include a language module 614 that can perform various generating tasks, such as to update or augment a text-based representation, see par. [0138]. An LLM 516 trained with an RTL (or other DSL) corpus built from a database 512 of map ground truth data can be queried to correct features output from a machine learning (ML) automation pipeline. The output of an LLM 516—such as by using a writer and/or parser component or module 520—can be mapped back to the extracted features, see par. [0118]. An RTL API can provide a public interface that lets developers generate structured language documents, and also allows users to query a trained model. As described herein, a schema can be provided to tokenize the edge sequence path, see par. [0131].
It would have been obvious to one of ordinary skill in the art to combine the Toms invention with the teachings of Wu in order to perform various generating tasks, such as to update or augment a text-based representation, see par. [0138].
Regarding claim 9 Toms teaches the system of claim 8, wherein the one or more processors are further configured to: receive preliminary input specifying a command to be followed by the LLM in generating the proposed structured data (Large Language Models (LLMs) to use programmatically configured inputs, such as prompts, for relevant content generation, see par. [0048]); and create the system task prompt based on the preliminary input (the system may be used to generate informative context-based prompts based on semantic and other relationships between various objects, see par. [0048]). Regarding claim 10 Wu teaches the system of claim 8, wherein the one or more processors are further configured to: receive supplemental input specifying a command to be followed by the LLM in revising the structured data ( The language model may also take other inputs as well, such as prior maps or context information , see par. [0038]); create a supplemental prompt based on the supplemental input (he input data can be represented by embeddings, feature vectors, or points in a latent space, which allows for relatively simple searching for similar environments, see par. [0038]); submit the supplemental prompt to the LLM (The generative AI model might take the sensor data directly as input or might receive input that is generated from the sensor data in one or more stages, see par. [0038]); and receive revised structured data from the LLM ( correct or update noisy or partial environment graphs or maps, see par. [0038]). Regarding claim 11 Toms teaches the system of claim 8, wherein the one or more processors are further configured to: present to a viewer a mixed reality experience based on the procedure (causing an interaction between the physical object and the virtual object based on the one or more permissible actions to be displayed in the electronic display, see par. [0111]);
present to a viewer a mixed reality experience based on the procedure (generating an AR display in which one or more virtual objects are to be overlaid onto a real world (RW) environment, see par. [0105]); receive from the viewer a query regarding the mixed reality experience (o cause the interaction, the processor is programmed to do so in response to an input from a user, see claim 3); create a viewer prompt based on the query and the procedure (defining a behavior of the virtual object with respect to the physical environment based on the AU object, see par. [0108]); submit the viewer prompt to the LLM (receiving a virtual object to augment the electronic display, see par. [0109]); receive, from the LLM, a response based on the viewer prompt (updating the electronic display to include the virtual object, see par. [0110]); and present to the viewer, by the mixed reality experience, a virtual artifact based on the response (causing an interaction between the physical object and the virtual object based on the one or more permissible actions to be displayed in the electronic display, see par. [0111]). Regarding claim 12 Toms teaches the system of claim 8, wherein the one or more processors are further configured to: present to a viewer a mixed reality experience based on the procedure (causing an interaction between the physical object and the virtual object based on the one or more permissible actions to be displayed in the electronic display, see par. [0111]).
Regarding claim 13 Toms teaches the system of claim 8, wherein the one or more processors are further configured to: receive input specifying one or more steps in the structured data to include in the procedure (Structured content is data that is organized in a predefined format that can be used to search and extract data, see par. [0052]); and include the specified one or more steps in the procedure (context-based AU system 110 may acquire content from the data sources 101, learn relationships between data objects based on the content, and store and transmit the relationships via an API 152 and/or one or more end user applications , see par. [0053]).
Regarding claim 14 Toms teaches the system of claim 8, wherein the one or more processors are further configured to: present to a viewer a mixed reality experience based on the procedure (causing an interaction between the physical object and the virtual object based on the one or more permissible actions to be displayed in the electronic display, see par. [0111]); receive contextual data corresponding to a feature of a physical environment while the one or more processors present the mixed reality experience (learning contextual data between the two data objects based at least on the content from which the two data objects were identified, see par. [0101]); create a contextual prompt based on the contextual data and the procedure (may include generating a linked data record comprising an identification for each of the two data objects and the learned contextual data so that identification of at least one of the data objects is sufficient to identify the linked data record, see par. [0102]); submit the contextual prompt to the LLM (Learned properties and relationships of objects may also enable generative AI systems that use Large Language Models (LLMs) to use programmatically configured inputs, such as prompts, for relevant content generation, see par. [0048]); receive, from the LLM, a response based on the contextual prompt (receiving a virtual object to augment the electronic display and one or more permissible actions that can be used based on contextual data, see par. [0109]); and present to the viewer, by the mixed reality experience, a virtual artifact based on the response (updating the electronic display to include the virtual object, see par. [0110]).
Regarding claim 15 Toms teaches one or more memories collectively storing instructions that, when executed by one or more processors in a computing system, cause the one or more processors to perform a method (processor 912 and system memory 918, see par. [0115]), the method comprising: receiving unstructured data including instructional content (Content may be accessed from various content sources to learn augmented unification (AU) data, see par. [0015]; content may be structured and/or unstructured, see par. [0052]); receiving schema input specifying a first schema to which the unstructured data is to conform (data acquisition subsystem 120 may use a topology/relationship schema specifically to house the learned properties and relationships. The schema establishes a structured framework for capturing and representing the learned properties of, and relationships between, objects analyzed from the acquired content, see par. [0057]); establishing a system task prompt based on the unstructured instructional content and the schema input (object learning subsystem 130 may access content, such as content acquired by the data acquisition subsystem 120, and learn properties of and relationships among data objects in the content. A data object is a representation of an object. The representation can include text such as the word “bell” that represents a bell, an image such as an image of the bell, audio such as a sound of a bell, and/or other types of representations that can be included in the content. The object learning subsystem 130 may identify the data object from the content. , see par. [0058]); submitting the system task prompt to a large language model (LLM) (Learned properties and relationships of objects may also enable generative AI systems that use Large Language Models (LLMs) to use programmatically configured inputs, such as prompts, for relevant content generation, see par. [0048]); receiving proposed structured data from the LLM (object learning subsystem 130 may use topic modeling to identify and extract relevant topics or themes from the content. Topic modeling is a statistical technique that may identify topics in the content, see par. [0060]); and transforming the structured data into a procedure usable in a mixed reality application ( named entity is a word or phrase that refers to specific object. This allows for a more precise recognition and correlation of objects within the augmented reality environment, see par. [0061]).
However Toms does not teach parsing the proposed structured data into structured data according to the first schema.
In the same field of endeavor Wu teaches a graphical user interface 610 and content manager 612 for use in providing, synthesizing, rendering, compositing, modifying, or using content for presentation (or other purposes) on or by the client device 602. The content application 604 can also include a language module 614 that can perform various generating tasks, such as to update or augment a text-based representation, see par. [0138]. An LLM 516 trained with an RTL (or other DSL) corpus built from a database 512 of map ground truth data can be queried to correct features output from a machine learning (ML) automation pipeline. The output of an LLM 516—such as by using a writer and/or parser component or module 520—can be mapped back to the extracted features, see par. [0118]. An RTL API can provide a public interface that lets developers generate structured language documents, and also allows users to query a trained model. As described herein, a schema can be provided to tokenize the edge sequence path, see par. [0131].
It would have been obvious to one of ordinary skill in the art to combine the Toms invention with the teachings of Wu in order to perform various generating tasks, such as to update or augment a text-based representation, see par. [0138].
Regarding claim 16 Toms teaches the one or more memories of claim 15, the method further comprising: receiving preliminary input specifying a command to be followed by the LLM in generating the proposed structured data (Large Language Models (LLMs) to use programmatically configured inputs, such as prompts, for relevant content generation, see par. [0048]); and creating the system task prompt based on the preliminary input (the system may be used to generate informative context-based prompts based on semantic and other relationships between various objects, see par. [0048]).
Regarding claim 17 Wu teaches the one or more memories of claim 15, the method further comprising:
receiving supplemental input specifying a command to be followed by the LLM in revising the structured data ( The language model may also take other inputs as well, such as prior maps or context information , see par. [0038]); creating a supplemental prompt based on the supplemental input (he input data can be represented by embeddings, feature vectors, or points in a latent space, which allows for relatively simple searching for similar environments, see par. [0038]); submitting the supplemental prompt to the LLM (The generative AI model might take the sensor data directly as input or might receive input that is generated from the sensor data in one or more stages, see par. [0038]); and receiving revised structured data from the LLM ( correct or update noisy or partial environment graphs or maps, see par. [0038]).
Regarding claim 18 Toms teaches the one or more memories of claim 15, the method further comprising: presenting to a viewer a mixed reality experience based on the procedure (generating an AR display in which one or more virtual objects are to be overlaid onto a real world (RW) environment, see par. [0105]); receiving from the viewer a query regarding the mixed reality experience (o cause the interaction, the processor is programmed to do so in response to an input from a user, see claim 3); creating a viewer prompt based on the query and the procedure (defining a behavior of the virtual object with respect to the physical environment based on the AU object, see par. [0108]); submitting the viewer prompt to the LLM (receiving a virtual object to augment the electronic display, see par. [0109]); receiving, from the LLM, a response based on the viewer prompt (updating the electronic display to include the virtual object, see par. [0110]); and presenting to the viewer, by the mixed reality experience, a virtual artifact based on the response (causing an interaction between the physical object and the virtual object based on the one or more permissible actions to be displayed in the electronic display, see par. [0111]).
Regarding claim 19 Toms teaches the one or more memories of claim 15, the method further comprising: presenting a step in a mixed reality experience based on a step in the procedure (causing an interaction between the physical object and the virtual object based on the one or more permissible actions to be displayed in the electronic display, see par. [0111]); selecting content in a portion of the unstructured data corresponding to the step in the procedure (accessing content comprising text and/or an image in a structured or unstructured format, see par. [0099]); and presenting to the viewer, by the mixed reality experience, a virtual artifact based on the selected content (include generating a linked data record comprising an identification for each of the two data objects and the learned contextual data so that identification of at least one of the data objects is sufficient to identify the linked data record, see par. [0102]).
Regarding claim 20 Toms teaches the one or more memories of claim 15, the method further comprising: presenting to a viewer a mixed reality experience based on the procedure (causing an interaction between the physical object and the virtual object based on the one or more permissible actions to be displayed in the electronic display, see par. [0111]); receiving contextual data corresponding to an action taken by the viewer while the one or more processors are presenting the mixed reality experience (learning contextual data between the two data objects based at least on the content from which the two data objects were identified, see par. [0101]); creating a contextual prompt based on the contextual data and the procedure (may include generating a linked data record comprising an identification for each of the two data objects and the learned contextual data so that identification of at least one of the data objects is sufficient to identify the linked data record, see par. [0102]); submitting the contextual prompt to the generative artificial intelligence model (Learned properties and relationships of objects may also enable generative AI systems that use Large Language Models (LLMs) to use programmatically configured inputs, such as prompts, for relevant content generation, see par. [0048]); receiving, from the generative artificial intelligence model, a response based on the contextual prompt (receiving a virtual object to augment the electronic display and one or more permissible actions that can be used based on contextual data, see par. [0109]); and presenting to the viewer, by the mixed reality experience, a virtual artifact based on the response (updating the electronic display to include the virtual object, see par. [0110]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Pertinent prior art available on form 892.
Neerukonda ‘644 teaches a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content, see abstract.
Gusarov ‘816 teaches a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content, see par. [0001].
Kreis ‘612 teaches a system that includes one or more language models, such as large language models (LLMs); a system for generating or presenting virtual reality (VR) content, augmented reality (AR) content, and/or mixed reality (MR) content, see par. [0016].
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Michael Ortiz-Sanchez whose telephone number is (571)270-3711. The examiner can normally be reached Monday- Friday 9AM-6PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Bhavesh Mehta can be reached at 571-272-7453. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL ORTIZ-SANCHEZ/Primary Examiner, Art Unit 2656