DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is responsive to the original application filed on 10/5/2023. Acknowledgment is made with respect to a claim of priority to Provisional Application 63/460,190 filed on 4/18/2023.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C 101 because the claimed invention is directed to an abstract idea without significantly more. The analysis of the claims will follow the 2019 Revised Patent Subject Matter Eligibility Guidance, 84 Fed. Reg. 50 (“2019 PEG”).
When considering subject matter eligibility under 35 U.S.C. 101, it must be determined whether the claim is directed to one of the four statutory categories of invention, i.e., process, machine, manufacture, or composition of matter (Step 1). If the claim does fall within one of the statutory categories, the second step in the analysis is to determine whether the claim is directed to a judicial exception (Step 2A). The Step 2A analysis is broken into two prongs. In the first prong (Step 2A, Prong 1), it is determined whether or not the claims recite a judicial exception (e.g., mathematical concepts, mental processes, certain methods of organizing human activity). If it is determined in Step 2A, Prong 1 that the claims recite a judicial exception, the analysis proceeds to the second prong (Step 2A, Prong 2), where it is determined whether or not the claims integrate the judicial exception into a practical application. If it is determined at step 2A, Prong 2 that the claims do not integrate the judicial exception into a practical application, the analysis proceeds to determining whether the claim is a patent-eligible application of the exception (Step 2B). If an abstract idea is present in the claim, any element or combination of elements in the claim must be sufficient to ensure that the claim integrates the judicial exception into a practical application, or else amounts to significantly more than the abstract idea itself.
Claim 1
Step 1: The claim recites a method; therefore, it is directed to the statutory category of a process.
Step 2A Prong 1: The claim recites, inter alia:
generating a prompt for a generative machine learning model using the query: Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of generating a model prompt using a query, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. For example, one can practically and mentally generate a prompt based on a question or query.
generate one or more data objects that match the one or more attributes defined by the query: Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of generating data objects that match an attribute of a query, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. For example, one can practically and mentally generate information about an attribute of a question or query.
generating, … one or more XR objects based on the one or more data objects generated by the generative machine learning model responsive to the prompt: Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of generating XR or data objects based on other data objects, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. For example, one can practically and mentally generate data objects based on particular information.
Step 2A Prong 2: The claim does not recite any additional limitations which integrate the abstract idea into a practical application. Specifically, the additional elements consist of “accessing, by an interaction application, an extended reality (XR) application;
receiving, by the interaction application, a query that defines one or more attributes of the XR application”, “processing the prompt using the generative machine learning model to”, and “using the XR application”.
The additional elements of “processing the prompt using the generative machine learning model to” and “using the XR application” amount to reciting only the idea of a solution or outcome i.e., the claim fails to recite details of how a solution to a problem is accomplished because it is not clear how the generic generative model is broadly used to process a prompt to generate a data object or how the generic XR application is broadly used to generate XR objects. Thus, the additional elements amount to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
The additional elements “accessing, by an interaction application, an extended reality (XR) application” and “receiving, by the interaction application, a query that defines one or more attributes of the XR application” are insignificant extra-solution activities required for any uses of the abstract ideas (see MPEP § 2106.05(g)).
Thus, even when viewed individually and as an ordered combination, these additional elements do not integrate the abstract idea into a practical application and the claim is thus directed to the abstract idea.
Step 2B: Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea.
The additional elements of “processing the prompt using the generative machine learning model to” and “using the XR application” amount to reciting only the idea of a solution or outcome i.e., the claim fails to recite details of how a solution to a problem is accomplished because it is not clear how the generic generative model is broadly used to process a prompt to generate a data object or how the generic XR application is broadly used to generate XR objects. Thus, the additional elements amount to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
The additional elements “accessing, by an interaction application, an extended reality (XR) application” and “receiving, by the interaction application, a query that defines one or more attributes of the XR application” are insignificant extra-solution activities required for any uses of the abstract ideas (see MPEP § 2106.05(g)), and are well-understood, routine, conventional activities (see MPEP § 2106.05(d)(II)(i); “Receiving or transmitting data over a network”).
Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible.
Claim 2
Step 1: A process, as above.
Step 2A Prong 1: The claim recites, inter alia:
adding to the prompt the one or more attributes of the query and the data definitions: Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of adding information to a prompt, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper.
Step 2A Prong 2, Step 2B: The additional element “obtaining data definitions from the XR application” is insignificant extra-solution activity required for any uses of the mental processes (see MPEP § 2106.05(g)), and is a well-understood, routine, conventional activity (see MPEP § 2106.05(d)(II)(i); “Receiving or transmitting data over a network”). Taken alone or in combination, the additional elements of the claim do not provide an inventive concept, integrate the abstract ideas into a practical application, or provide significantly more than the abstract ideas of the claim and thus the claim is subject-matter ineligible.
Claim 3
Step 1: A process, as above.
Step 2A Prong 1: The claim recites the abstract ideas of the preceding claims from which it depends.
Step 2A Prong 2, Step 2B: The additional element “wherein the data definitions comprise a subject, a graphical element, and a visual attribute of the graphical element” amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use (see MPEP § 2106.05(h). Taken alone or in combination, the additional elements of the claim do not provide an inventive concept, integrate the abstract ideas into a practical application, or provide significantly more than the abstract ideas of the claim and thus the claim is subject-matter ineligible.
Claim 4
Step 1: A process, as above.
Step 2A Prong 1: The claim recites the abstract ideas of the preceding claims from which it depends.
Step 2A Prong 2, Step 2B: The additional elements “receiving a first data object from the generative machine learning model, the first data object comprising a first subject matching the subject matter definition, a first graphical element associated with the first subject and having a first visual attribute; and receiving a second data object from the generative machine learning model, the second data object comprising a second subject matching the subject matter definition, a second graphical element associated with the second subject and having a second visual attribute” are insignificant extra-solution activities required for any uses of the mental processes (see MPEP § 2106.05(g)), and are well-understood, routine, conventional activities (see MPEP § 2106.05(d)(II)(i); “Receiving or transmitting data over a network”). The additional element “wherein the one or more attributes comprise a subject matter definition” amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use (see MPEP § 2106.05(h). Taken alone or in combination, the additional elements of the claim do not provide an inventive concept, integrate the abstract ideas into a practical application, or provide significantly more than the abstract ideas of the claim and thus the claim is subject-matter ineligible.
Claim 5
Step 1: A process, as above.
Step 2A Prong 1: The claim recites, inter alia:
generating a first XR object comprising the first data object and a second XR object comprising the second data object: Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of generating data objects based on other data objects, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper.
overlaying the first XR object on a real-world object depicted in an image: Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of overlaying objects on an image, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper.
Step 2A Prong 2, Step 2B: The claim does not recite any additional elements that are sufficient to integrate the judicial exceptions into a practical application or amount to significantly more than the judicial exception. As such, the claim is ineligible.
Claim 6
Step 1: A process, as above.
Step 2A Prong 1: The claim recites, inter alia:
after a threshold period of time, overlaying the second XR object on the real-world object depicted in the image instead of the first XR object.: Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of overlaying objects on an image, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper.
Step 2A Prong 2, Step 2B: The claim does not recite any additional elements that are sufficient to integrate the judicial exceptions into a practical application or amount to significantly more than the judicial exception. As such, the claim is ineligible.
Claim 7
Step 1: A process, as above.
Step 2A Prong 1: The claim recites, inter alia:
randomly selecting between presenting the first XR object and presenting the second XR object in response to a user request to launch the XR application: Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of selecting to present an object, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper.
Step 2A Prong 2, Step 2B: The claim does not recite any additional elements that are sufficient to integrate the judicial exceptions into a practical application or amount to significantly more than the judicial exception. As such, the claim is ineligible.
Claim 8
Step 1: A process, as above.
Step 2A Prong 1: The claim recites, inter alia:
populating a data template associated with the XR application using the one or more data objects generated by the generative machine learning model, the data template being used by the XR application to present the one or more XR objects: Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of populating a template, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper.
Step 2A Prong 2, Step 2B: The claim does not recite any additional elements that are sufficient to integrate the judicial exceptions into a practical application or amount to significantly more than the judicial exception. As such, the claim is ineligible.
Claim 9
Step 1: A process, as above.
Step 2A Prong 1: The claim recites the abstract ideas of the preceding claims from which it depends.
Step 2A Prong 2, Step 2B: The additional elements “storing the populated data template in association with the XR application; sending the XR application with the populated data template to a second user system; and presenting the one or more XR objects on the second user system based on the populated data template” are insignificant extra-solution activities required for any uses of the mental processes (see MPEP § 2106.05(g)), and are well-understood, routine, conventional activities (see MPEP § 2106.05(d)(II)(i); “Receiving or transmitting data over a network”, “Storing and retrieving information in memory”). The additional element “the query being received from a first user system and the prompt being generated by the first user system” amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use (see MPEP § 2106.05(h). Taken alone or in combination, the additional elements of the claim do not provide an inventive concept, integrate the abstract ideas into a practical application, or provide significantly more than the abstract ideas of the claim and thus the claim is subject-matter ineligible.
Claim 10
Step 1: A process, as above.
Step 2A Prong 1: The claim recites, inter alia:
generating an additional prompt for the generative machine learning model using the additional query: Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of generating a prompt, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper.
generate a second set of data objects that match the second set of attributes: Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of generating data objects that match attributes, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper.
populating the data template using the second set of data objects to generate a second populated data template: Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of populating a template, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper.
generating, …, a second set of XR objects based on the second populated data template: Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of generating data objects based on a template, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper.
Step 2A Prong 2, Step 2B: The additional elements of “processing the additional prompt using the generative machine learning model to” and “using the XR application using the XR application” amount to reciting only the idea of a solution or outcome i.e., the claim fails to recite details of how a solution to a problem is accomplished because it is not clear how the prompt is broadly processed using the generic generative ML model or how the generic XR application is broadly used to generate XR objects. Thus, the additional elements amount to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)). The additional elements “receiving, by the second user system, an additional query that defines a second set of attributes of the XR application” are insignificant extra-solution activities required for any uses of the mental processes (see MPEP § 2106.05(g)), and are well-understood, routine, conventional activities (see MPEP § 2106.05(d)(II)(i); “Receiving or transmitting data over a network”, “Storing and retrieving information in memory”). The additional element “wherein the one or more attributes comprise a first set of attributes, and wherein the populated data template is a first populated data template” amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use (see MPEP § 2106.05(h). Taken alone or in combination, the additional elements of the claim do not provide an inventive concept, integrate the abstract ideas into a practical application, or provide significantly more than the abstract ideas of the claim and thus the claim is subject-matter ineligible.
Claim 11
Step 1: A process, as above.
Step 2A Prong 1: The claim recites the abstract ideas of the preceding claims from which it depends.
Step 2A Prong 2, Step 2B: The additional elements “presenting a first icon in association with the XR application for launching the XR application using the first populated data template; and presenting a second icon in association with the XR application for launching the XR application using the second populated data template” are insignificant extra-solution activities required for any uses of the mental processes (see MPEP § 2106.05(g)), and are well-understood, routine, conventional activities (see MPEP § 2106.05(d)(II)(i); “Receiving or transmitting data over a network”, “Storing and retrieving information in memory”, “Presenting offers and gathering statistics”). Taken alone or in combination, the additional elements of the claim do not provide an inventive concept, integrate the abstract ideas into a practical application, or provide significantly more than the abstract ideas of the claim and thus the claim is subject-matter ineligible.
Claim 12
Step 1: A process, as above.
Step 2A Prong 1: The claim recites, inter alia:
selecting, by the XR application, a segmentation machine learning model from a plurality of segmentation machine learning models based on the description of the body part: Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of selecting a model, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper.
segmenting a portion of a real-world object depicted in an image based on the segmentation machine learning model: Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of segmenting an object in an image, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper.
applying the one or more XR objects to the portion of the real-world object in real time to modify the portion of the real-world object to match the attribute of the body part: Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of modifying an object to match an attribute, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper.
Step 2A Prong 2, Step 2B: The additional element “wherein the one or more attributes comprise a body part of a person, wherein the one or more data objects comprise a description of the body part and an attribute of the body part” amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use (see MPEP § 2106.05(h). Taken alone or in combination, the additional elements of the claim do not provide an inventive concept, integrate the abstract ideas into a practical application, or provide significantly more than the abstract ideas of the claim and thus the claim is subject-matter ineligible.
Claim 13
Step 1: A process, as above.
Step 2A Prong 1: The claim recites the abstract ideas of the preceding claims from which it depends.
Step 2A Prong 2, Step 2B: The additional element “wherein the description of the body part comprises hair on a head of the person, wherein the segmentation machine learning model outputs a segmentation of the hair on the head of the real-world object depicted in the image, and wherein the attribute of the body part comprises a hair color” amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use (see MPEP § 2106.05(h). Taken alone or in combination, the additional elements of the claim do not provide an inventive concept, integrate the abstract ideas into a practical application, or provide significantly more than the abstract ideas of the claim and thus the claim is subject-matter ineligible.
Claim 14
Step 1: A process, as above.
Step 2A Prong 1: The claim recites, inter alia:
processing the types of objects associated with the scene and types of characters matching the one or more characters: Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of processing data objects, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper.
generating the one or more XR objects using the visual object definitions: Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of generating data objects using definitions, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper.
Step 2A Prong 2, Step 2B: The additional elements of “using a visual object model generator” amount to reciting only the idea of a solution or outcome i.e., the claim fails to recite details of how a solution to a problem is accomplished because it is not clear how the generic visual object model generator is used to process objects. Thus, the additional elements amount to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)). The additional elements “receiving visual object definitions corresponding to the types of objects associated with the scene and the types of characters from the visual object model generator” are insignificant extra-solution activities required for any uses of the mental processes (see MPEP § 2106.05(g)), and are well-understood, routine, conventional activities (see MPEP § 2106.05(d)(II)(i); “Receiving or transmitting data over a network”, “Storing and retrieving information in memory”). The additional element “wherein the one or more attributes comprise a description of a scene and one or more characters in the scene, wherein the one or more data objects comprise types of objects associated with the scene and types of characters matching the one or more characters” amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use (see MPEP § 2106.05(h). Taken alone or in combination, the additional elements of the claim do not provide an inventive concept, integrate the abstract ideas into a practical application, or provide significantly more than the abstract ideas of the claim and thus the claim is subject-matter ineligible.
Claim 15
Step 1: A process, as above.
Step 2A Prong 1: The claim recites the abstract ideas of the preceding claims from which it depends.
Step 2A Prong 2, Step 2B: The additional element “wherein the visual object model generator comprises a three-dimensional (3D) model generator, and wherein the visual object definitions comprise 3D models of the types of objects associated with the scene and the types of characters” amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use (see MPEP § 2106.05(h). Taken alone or in combination, the additional elements of the claim do not provide an inventive concept, integrate the abstract ideas into a practical application, or provide significantly more than the abstract ideas of the claim and thus the claim is subject-matter ineligible.
Claim 16
Step 1: A process, as above.
Step 2A Prong 1: The claim recites the abstract ideas of the preceding claims from which it depends.
Step 2A Prong 2, Step 2B: The additional elements of “animating the one or more XR objects based on the visual object definitions” amount to reciting only the idea of a solution or outcome i.e., the claim fails to recite details of how a solution to a problem is accomplished because it is not clear how the objects are broadly animated based on object definitions. Thus, the additional elements amount to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)). Taken alone or in combination, the additional elements of the claim do not provide an inventive concept, integrate the abstract ideas into a practical application, or provide significantly more than the abstract ideas of the claim and thus the claim is subject-matter ineligible.
Claim 17
Step 1: A process, as above.
Step 2A Prong 1: The claim recites the abstract ideas of the preceding claims from which it depends.
Step 2A Prong 2, Step 2B: The additional elements “receiving input defining one or more conditions that trigger the animating of the one or more XR objects based on the visual object definitions and defining locations within a real-world scene in which to place the one or more XR objects” are insignificant extra-solution activities required for any uses of the mental processes (see MPEP § 2106.05(g)), and are well-understood, routine, conventional activities (see MPEP § 2106.05(d)(II)(i); “Receiving or transmitting data over a network”, “Storing and retrieving information in memory”). Taken alone or in combination, the additional elements of the claim do not provide an inventive concept, integrate the abstract ideas into a practical application, or provide significantly more than the abstract ideas of the claim and thus the claim is subject-matter ineligible.
Claim 15
Step 1: A process, as above.
Step 2A Prong 1: The claim recites the abstract ideas of the preceding claims from which it depends.
Step 2A Prong 2, Step 2B: The additional element “wherein the one or more conditions comprise at least one of a virtual distance between a user system and the location of a given one of the XR objects or time” amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use (see MPEP § 2106.05(h). Taken alone or in combination, the additional elements of the claim do not provide an inventive concept, integrate the abstract ideas into a practical application, or provide significantly more than the abstract ideas of the claim and thus the claim is subject-matter ineligible.
Claim 19
Claim 19 recites a system (step 1: a machine) using a processor and memory to perform the steps of claim 1, which by MPEP 2106.05(f) (“apply it”) cannot integrate an abstract idea into a practical application or provide significantly more than the abstract idea by itself, and is thus rejected for the same reasons set forth in the rejection of claim 1.
Claim 20
Claim 20 recites a non-transitory computer-readable storage medium (step 1: a manufacture) using a processor to perform the steps of claim 1, which by MPEP 2106.05(f) (“apply it”) cannot integrate an abstract idea into a practical application or provide significantly more than the abstract idea by itself, and is thus rejected for the same reasons set forth in the rejection of claim 1.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-5, 8, 14-17, 19, and 20 are rejected under 35 USC § 102(a)(1) as being anticipated by Shekhar et al. (US10665030 B1, hereinafter “Shekhar”).
Regarding claim 1, Shekhar discloses [a] method comprising: (Column 4, Lines 18-21; “FIG. 2 is a process flow diagram of an example method 200 for processor-based conversion of textual input into an augmented reality experience, in accordance with an embodiment of the present disclosure”)
accessing, by an interaction application, an extended reality (XR) application; (Column 3, Lines 31-39; “The system 100 includes a computing device 110 having a processor 120, a Text-to-AR Scene Conversion Application 130, and a graphical user interface (GUI) 150. … The processor 120 of the computing device 110 is configured to execute Text Input Module 140, a Natural Language Processing (NLP) Back End Module 142, and an Augmented Reality (AR) Front End Module 144”, which discloses an interaction application or text input module 140 that accesses and interfaces with the XR application or AR Front End Module 144)
receiving, by the interaction application, a query that defines one or more attributes of the XR application (Column 5, Lines 14-20; “The user interface 300 of FIG. 3 includes one or more fields 302 for inputting or selecting a sentence that describes, in natural language (such as English, French, Spanish, etc.), one or more objects and people in the scene, and relationships between those objects and people. One example of a natural language description of an AR scene is: “A man is sitting on a bench.””; and Column 4, Lines 64-67; “The text input front end 202 provides a user interface, such as the example user interface 300 of FIG. 3, for entering the natural language description(s) of the AR scene to be rendered”, which discloses that the natural language description entered in the text input front end is a “query that defines one or more attributes” of the AR or XR scene to be rendered and the attributes include a subject such as a man, graphical elements like a bench, and their spatial or visual relationships)
generating a prompt for a generative machine learning model using the query; (Column 5, Lines 46-49, 65-67 to Column 6, Line 1; “the text input 160 is parsed and converted into a scene graph using any suitable natural language processing techniques as implemented in the natural language processing back end 204 … The natural language input 502 is converted to a scene graph by scene graph generation 504 using one or more language-to-visual datasets that describe a wide variety of entities and common relations between those entities”, which discloses a “prompt” or scene graph that is constructed from the user’s natural language input and is subsequently used by the generative model. The prompt is any model input derived from a query; and Column 2, Lines 66-67 to Column 3, Lines 1-3; “The disclosed techniques use deep learning (also known as deep structured learning or hierarchical learning in the context of machine learning methods) to predict the relative sizes and the relative positions of the objects in the scene with respect to each other”)
processing the prompt using the generative machine learning model to generate one or more data objects that match the one or more attributes defined by the query; and (Column 6, Lines 1-9; “The scene graph represents each of the entities in the described scene and the relations between those entities. Entities can include virtual representations of inanimate objects and living things, such as humans or animals … Next, to enhance the AR experience for the user, the scene can be augmented 506 with information that is not explicitly stated in the natural language input 502 but can be implicitly reasoned from it”; and Column 8, Lines 21-24 and 51-53; “A three-layered neural network is trained on text embeddings of the object and human and the position and size of the first object, to predict the position and size of the second object … Given a subject-relation-object, the position and the size of the subject, the model predicts the position and size of the object”, which discloses ML models that process the scene graph prompt and generate structured entity data objects with predicted attributes like size and position that match attributes defined by the input query)
generating, using the XR application, one or more XR objects based on the one or more data objects generated by the generative machine learning model responsive to the prompt (Column 2, Lines 38-42; “The AR scene is then rendered using the augmented reality display device according to its natural language description using 3D models of objects and humanoid characters with associated animations of those characters, as well as from extensive language-to-visual datasets”; and Column 4, Lines 50-50-58; “The AR Front End Module 144 is configured to render the AR scene produced by a natural language processing back end 204 using one or more 3D models 210, thereby providing the AR scene rendition 162. The AR Front End Module 144 is configured to render the AR scene on various surfaces of objects in the real-world environment (for example, rendering an image of a computer-generated lamp such that it appears on the top of a physical table in the real-world environment of the user)”, which discloses an AR Front End Module or XR application that generates 3D AR or XR objects based on data objects or scene graph entities with visual attributes that are output by the ML or NLP back end).
Regarding claim 19, it is a system claim corresponding to the steps of claim 1, and is rejected for the same reasons as claim 1.
Regarding claim 20, it is a non-transitory computer-readable storage medium claim corresponding to the steps of claim 1, and is rejected for the same reasons as claim 1.
Regarding claim 2, the rejection of claim 1 is incorporated and Shekhar further discloses obtaining data definitions from the XR application; and adding to the prompt the one or more attributes of the query and the data definitions (Column 4, Lines 37-49; “Various object-to-object relation datasets 208 and object-to-human relation datasets are used to predict the relative size and the relative position of objects in the scene. For example, the datasets can include any dataset having a large set of images with tagged object boxes and defined relationships between the images, such as the Stanford Visual Genome dataset, or a dataset of 3D scenes and text describing the relations between objects in the scenes, such as the Stanford Text2Scene dataset. Other datasets or object models can be used to augment the scene with additional entities as well as to infer animations and backgrounds that are appropriate for the entities explicitly or implicitly described by the text input”).
Regarding claim 3, the rejection of claims 1 and 2 are incorporated and Shekhar further discloses wherein the data definitions comprise a subject, a graphical element, and a visual attribute of the graphical element (Column 6, Lines 35-41 and 56-59; “A scene graph is a graphical representation of natural language text, where each node in the graph corresponds to objects or other entities referenced in the text. The scene graph includes two types of edges: attribute edges, which describe some aspect of the objects/entities, and relation edges, which describe spatial and size relationships between objects/entities … The extended scene graph 600, as produced by the scene graph generation 504, includes nodes 608 and 610 that represent the objects described by the natural language input 502 that are to be rendered in the AR space”).
Regarding claim 4, the rejection of claim 1 is incorporated and Shekhar further discloses wherein the one or more attributes comprise a subject matter definition, further comprising: receiving a first data object from the generative machine learning model, the first data object comprising a first subject matching the subject matter definition, a first graphical element associated with the first subject and having a first visual attribute; and receiving a second data object from the generative machine learning model, the second data object comprising a second subject matching the subject matter definition, a second graphical element associated with the second subject and having a second visual attribute (Column 6, Lines 1-9; “The scene graph represents each of the entities in the described scene and the relations between those entities. Entities can include virtual representations of inanimate objects and living things, such as humans or animals; and Column 5, Lines 40-44; “For example, FIG. 4 shows AR scene renditions of “Jack is walking near a beach with a city view,” “Jack and Jull are at a party with a birthday cake on the table,” and “Jill is sitting in a playground.”).
Regarding claim 5, the rejection of claims 1 and 4 are incorporated and Shekhar further discloses generating a first XR object comprising the first data object and a second XR object comprising the second data object; and overlaying the first XR object on a real-world object depicted in an image (Abstract; “The user can also select a physical real-world surface on which the AR scene is to be rendered”; and Column 4, Lines 53-58; “The AR Front End Module 144 is configured to render the AR scene on various surfaces of objects in the real-world environment (for example, rendering an image of a computer-generated lamp such that it appears on the top of a physical table in the real-world environment of the user)”).
Regarding claim 8, the rejection of claim 1 is incorporated and Shekhar further discloses populating a data template associated with the XR application using the one or more data objects generated by the generative machine learning model, the data template being used by the XR application to present the one or more XR objects (Column 6, Lines 30-33; “The data resulting from the natural language processing back end 204 is then provided as back end output 512 for use by the AR front end 206, where the AR scene is rendered”; and Column 2, Lines 20-23; “the conversion process may include positioning objects in an AR scene according to pre-defined relations between the objects and templates that generalize the text-to-visual scene conversion”; and Column 3, Lines 52-56; “the Text-to-AR Scene Conversion Application 130 is generally configured to perform one or more of the following functions: scene graph generation, scene augmentation, size and position prediction, animation and background inference, and AR scene rendition”).
Regarding claim 14, the rejection of claim 1 is incorporated and Shekhar further discloses wherein the one or more attributes comprise a description of a scene and one or more characters in the scene, wherein the one or more data objects comprise types of objects associated with the scene and types of characters matching the one or more characters, further comprising: processing the types of objects associated with the scene and types of characters matching the one or more characters using a visual object model generator; receiving visual object definitions corresponding to the types of objects associated with the scene and the types of characters from the visual object model generator; and generating the one or more XR objects using the visual object definitions (Column 2, Lines 38-42; “The AR scene is then rendered using the augmented reality display device according to its natural language description using 3D models of objects and humanoid characters with associated animations of those characters, as well as from extensive language-to-visual datasets”; and Column 4, Lines 29-34; “Still referring to FIG. 2, the NLP Back End Module 142 is configured to convert the text input received by the text input front end 202 into a scene graph representing objects and object relations in the scene described by the text input, to augment the scene with additional information not explicitly described by the text input”; and Column 3, Lines 46-50; “The AR scene rendition 162 depends on several things, including one or more object-to-object relation models and one or more object-to-human relation models, such as variously described in this disclosure”; and Column 2, Lines 56-61; “the disclosed techniques employ a learning framework that maps natural language text to AR scenes using 3D object models, such that easily understood, plain language inputs are recognized and converted into corresponding virtual elements rendered in AR”).
Regarding claim 15, the rejection of claims 1 and 14 are incorporated and Shekhar further discloses wherein the visual object model generator comprises a three-dimensional (3D) model generator, and wherein the visual object definitions comprise 3D models of the types of objects associated with the scene and the types of characters (Column 4, Lines 50-53; “The AR Front End Module 144 is configured to render the AR scene produced by a natural language processing back end 204 using one or more 3D models 210, thereby providing the AR scene rendition 162”; and Column 2, Lines 56-61; “the disclosed techniques employ a learning framework that maps natural language text to AR scenes using 3D object models, such that easily understood, plain language inputs are recognized and converted into corresponding virtual elements rendered in AR”).
Regarding claim 16, the rejection of claims 1 and 14 are incorporated and Shekhar further discloses animating the one or more XR objects based on the visual object definitions (Column 2, Lines 38-42; “The AR scene is then rendered using the augmented reality display device according to its natural language description using 3D models of objects and humanoid characters with associated animations of those characters, as well as from extensive language-to-visual datasets”; and Column 6, Lines 27-29; “Next, entities such as humanoids are animated 510, wherein animation is inferred from the natural language scene description”).
Regarding claim 17, the rejection of claims 1 and 14 and 16 are incorporated and Shekhar further discloses receiving input defining one or more conditions that trigger the animating of the one or more XR objects based on the visual object definitions and defining locations within a real-world scene in which to place the one or more XR objects (Column 5, Lines 14-18; “The user interface 300 of FIG. 3 includes one or more fields 302 for inputting or selecting a sentence that describes, in natural language (such as English, French, Spanish, etc.), one or more objects and people in the scene, and relationships between those objects and people”; and Column 2, Lines 61-66; “The natural language input can describe static and dynamic (animated) scenes both explicitly, where the spatial and size relationships between objects are defined by the textual description, and implicitly, where objects that are not explicitly described can instead be gleaned from models and machine learning”).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 6, 7, 12, 13, and 18 are rejected under 35 USC § 103 as being obvious over Shekhar in view of Rykhliuk et al. (US 20210390745 A1, hereinafter “Rykh”).
Regarding claim 6, the rejection of claims 1, 4, and 5 are incorporated and Shekhar fails to explicitly disclose but Rykh discloses after a threshold period of time, overlaying the second XR object on the real-world object depicted in the image instead of the first XR object ([0034]; “The ephemeral timer system 202 is responsible for enforcing the temporary or time-limited access to content by the messaging client 104 and the messaging server 114. The ephemeral timer system 202 incorporates a number of timers that, based on duration and display parameters associated with a message, or collection of messages (e.g., a story), selectively enable access (e.g., for presentation and display) to messages and associated content via the messaging client 104”, which discloses selectively enabling or restricting AR content based on elapsed time).
Shekhar and Rykh are analogous art because both are concerned with augmented reality systems. Before the effective filing date of the claimed invention, it would have been obvious to one skilled in augmented reality technology to combine the object overlays based on time of Rykh and the method and XR objects of Shekhar to yield to the predictable result of after a threshold period of time, overlaying the second XR object on the real-world object depicted in the image instead of the first XR object. The motivation for doing so would be to use machine learning to provide decisions in augmented reality content items (Rykh; [0002]).
Regarding claim 7, the rejection of claims 1, 4, and 5 are incorporated and Shekhar fails to explicitly disclose but Rykh discloses selecting between presenting the first XR object and presenting the second XR object in response to a user request to launch the XR application ([0017]; “the machine learning model is accessed from a resource library”; and [0019]; “The custom augmentation system accesses an augmentation reality content item”).
The motivation to combine Shekhar and Rykh is the same as discussed above with respect to claim 6.
Regarding claim 12, the rejection of claim 1 is incorporated and Shekhar fails to explicitly disclose but Rykh discloses wherein the one or more attributes comprise a body part of a person, wherein the one or more data objects comprise a description of the body part and an attribute of the body part, further comprising: selecting, by the XR application, a segmentation machine learning model from a plurality of segmentation machine learning models based on the description of the body part; segmenting a portion of a real-world object depicted in an image based on the segmentation machine learning model; and applying the one or more XR objects to the portion of the real-world object in real time to modify the portion of the real-world object to match the attribute of the body part ([0017]; “The machine learning model may be a segmentation model, a classification model, an object detection model, or a saliency model. A segmentation model is a type of machine learning model that filters a portion of an image based on certain criteria”; and [0110]; and [0031]).
The motivation to combine Shekhar and Rykh is the same as discussed above with respect to claim 6.
Regarding claim 13, the rejection of claim 1 and 12 are incorporated and Shekhar fails to explicitly disclose but Rykh discloses wherein the description of the body part comprises hair on a head of the person, wherein the segmentation machine learning model outputs a segmentation of the hair on the head of the real-world object depicted in the image, and wherein the attribute of the body part comprises a hair color ([0012-0013]; and Figures 8-9).
The motivation to combine Shekhar and Rykh is the same as discussed above with respect to claim 6.
Regarding claim 18, the rejection of claims 1 and 14 and 16 and 17 are incorporated and Shekhar fails to explicitly disclose but Rykh discloses wherein the one or more conditions comprise at least one of a virtual distance between a user system and the location of a given one of the XR objects or time ([0034]; “The ephemeral timer system 202 is responsible for enforcing the temporary or time-limited access to content by the messaging client 104 and the messaging server 114. The ephemeral timer system 202 incorporates a number of timers that, based on duration and display parameters associated with a message”).
The motivation to combine Shekhar and Rykh is the same as discussed above with respect to claim 6.
Claims 9 and 10 are rejected under 35 USC § 103 as being obvious over Shekhar in view of Daniels et al. (US 20160133230 A1, hereinafter “Daniels”).
Regarding claim 9, the rejection of claims 1 and 8 are incorporated and Shekhar fails to explicitly disclose but Daniels discloses storing the populated data template in association with the XR application; sending the XR application with the populated data template to a second user system; and presenting the one or more XR objects on the second user system based on the populated data template (Abstract; “A system is provided for enabling a shared augmented reality experience … The on-site devices synchronize the content used to create the augmented reality experience with the off-site devices in real time such that the augmented reality representations and the virtual augmented reality representations are consistent with each other” and Figure 2A)
Shekhar and Daniels are analogous art because both are concerned with augmented reality systems. Before the effective filing date of the claimed invention, it would have been obvious to one skilled in augmented reality technology to combine the storing, sending, and presenting of Daniels and the method and XR objects of Shekhar to yield to the predictable result of storing the populated data template in association with the XR application; sending the XR application with the populated data template to a second user system; and presenting the one or more XR objects on the second user system based on the populated data template. The motivation for doing so would be to use machine learning to position, locate, interact and/or share augmented reality content and other location-based information between people by the use of digital device (Daniels; [0003]).
Regarding claim 10, the rejection of claims 1 and 8 and 9 are incorporated and Shekhar discloses wherein the one or more attributes comprise a first set of attributes, and wherein the populated data template is a first populated data template, further comprising: receiving, by the second user system, an additional query that defines a second set of attributes of the XR application; generating an additional prompt for the generative machine learning model using the additional query; processing the additional prompt using the generative machine learning model to generate a second set of data objects that match the second set of attributes; populating the data template using the second set of data objects to generate a second populated data template; and generating, using the XR application, a second set of XR objects based on the second populated data template (Column 5, Lines 8-11; “The GUI 300 is configured to permit a user to enter a natural language scene description of the AR scene or to select from a pre-defined list of natural language scene descriptions”).
The motivation to combine Shekhar and Daniels is the same as discussed above with respect to claim 9.
Claim 11 is rejected under 35 USC § 103 as being obvious over Shekhar in view of Daniels and Rykh.
Regarding claim 11, the rejection of claims 1 and 8-10 are incorporated and Shekhar fails to explicitly disclose but Rykh discloses presenting a first icon in association with the XR application for launching the XR application using the first populated data template; and presenting a second icon in association with the XR application for launching the XR application using the second populated data template ([0041]; “The messaging client 104 also supports both the voice and text messaging (e.g., chats) within the context of gameplay, provides a leaderboard for the games, and also supports the provision of in-game rewards (e.g., coins and items)”; and [0019]; “The custom augmentation system accesses an augmentation reality content item. The augmentation reality content item is configured to modify image content of the received image”).
Shekhar, Daniels, and Rykh are analogous art because all are concerned with augmented reality systems. Before the effective filing date of the claimed invention, it would have been obvious to one skilled in augmented reality technology to combine the storing, sending, and presenting of Daniels and the method and XR objects of Shekhar to yield to the predictable result of presenting a first icon in association with the XR application for launching the XR application using the first populated data template; and presenting a second icon in association with the XR application for launching the XR application using the second populated data template. The motivation for doing so would be to use machine learning to provide decisions in augmented reality content items (Rykh; [0002]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Brent Hoover whose telephone number is (303)297-4403. The examiner can normally be reached Monday - Friday 9-5 MST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abdullah Kawsar can be reached at 571-270-3169. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BRENT JOHNSTON HOOVER/Primary Examiner, Art Unit 2127
Prosecution Insights