DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The Amendment filed on 01/20/2026 has been entered. Claims 1-9 and 15-20 remain pending in the application.
Claim Objections
Claim 15 is objected to because of the following informalities:
Claim 15 recites ‘determining, by an automated assistant application provided by a first entity, that a user has selected a virtual feature of a virtual environment in furtherance of controlling an operation a separate application’; however, it should recite ‘determining, by an automated assistant application provided by a first entity, that a user has selected a virtual feature of a virtual environment in furtherance of controlling an operation of a separate application”.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claim 1 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention.
Claim 1 recites the limitation " the virtual environment application". There is insufficient antecedent basis for this limitation in the claim.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-9 and 15-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claims 1-9 and 15-20 are directed to a method. Therefore, the claims are eligible under Step 1 for being directed to a process.
Step 2A Prong 1:
Independent claim 1 recites:
determining that a user is viewing a portion of a virtual environment that includes a virtual object - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating data and selecting data based on judgement, which is observing, evaluating and judging that is practically capable of being performed in the human mind with the assistance of pen and paper.
determining, based on the virtual object, that the virtual object is associated with one or more operations - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating data and selecting data based on judgement, which is observing, evaluating and judging that is practically capable of being performed in the human mind with the assistance of pen and paper.
generating, based on the one or more operations associated with the virtual object, virtual feature data that characterizes a virtual feature for controlling the one or more operations when the user interacts with the virtual feature via the virtual environment - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating data and generating virtual feature data based on selected data based on judgement, which is observing, evaluating and judging that is practically capable of being performed in the human mind with the assistance of pen and paper.
Dependent claim 3 recites:
wherein generating the virtual feature data that characterizes the virtual feature includes: determining textual content to include with the virtual feature, wherein the textual content indicates the one or more operations that can be controlled by interacting with the virtual feature - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating data and generating virtual feature data based on judgement, which is observing, evaluating and judging that is practically capable of being performed in the human mind with the assistance of pen and paper.
Dependent claim 5 recites:
wherein generating the virtual feature data that characterizes the virtual feature includes: determining textual content to include with the virtual feature,
wherein the textual content indicates the separate application that can be controlled by interacting with the virtual feature - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating data and generating virtual feature data based on judgement, which is observing, evaluating and judging that is practically capable of being performed in the human mind with the assistance of pen and paper.
Independent claim 15 recites:
determining that a user has selected a virtual feature of a virtual environment in furtherance of controlling an operation a separate application - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating data and selecting data based on judgement, which is observing, evaluating and judging that is practically capable of being performed in the human mind with the assistance of pen and paper.
determining that the user has selected one or more parameters for the operation - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating data and selecting data based on judgement, which is observing, evaluating and judging that is practically capable of being performed in the human mind with the assistance of pen and paper.
determining, based on the user selecting the one or more parameters, that the user has previously selected the one or more parameters for the operation during a prior interaction between the user and the automated assistant application - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating data and selecting data based on judgement, which is observing, evaluating and judging that is practically capable of being performed in the human mind with the assistance of pen and paper.
generating, based on the user having previously selected the one or more parameters during the prior interaction between the user and the automated assistant application, virtual feature data - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating data and generating virtual feature data based on selected data based on judgement, which is observing, evaluating and judging that is practically capable of being performed in the human mind with the assistance of pen and paper.
Dependent claim 16 recites:
wherein determining that the user has selected the one or more parameters for the operation includes: determining that the user has directed a spoken utterance to the automated assistant application, and that the spoken utterance includes a warm word characterizing the one or more parameters, wherein the spoken utterance is independent of any invocation phrase for the automated assistant application - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating data and selecting data based on judgement, which is observing, evaluating and judging that is practically capable of being performed in the human mind with the assistance of pen and paper.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because they recite the additional elements:
Independent claim 1:
A method implemented by one or more processors, an automated assistant application provided by a first entity; by the automated assistant application accessing the virtual environment application; provided by a second entity different from the first entity; wherein the user views the virtual environment via a display interface of a computing device; one or more operations that are capable of being controlled via a separate application - These limitations amount to components of a general purpose computer that applies a judicial exception, by use of conventional computer functions (see MPEP § 2106.05(b)).
causing, based on the virtual feature data and by the automated assistant application, rendering of the virtual feature in the virtual environment, wherein the virtual feature is rendered at a location within the virtual environment that is associated with the virtual object - the steps recited at a high level of generality, and amounts to mere data outputting, which is a form of insignificant extra-solution activity (see MPEP § 2106.05(g)).
in response to determining that the user interacted with the virtual feature via the virtual environment: causing the automated assistant to control the separate application according to an interaction between the user and the virtual feature - the steps recited at a high level of generality, and amounts to more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Dependent claim 2:
wherein causing rendering of the virtual feature in the virtual environment causes the virtual feature to be rendered at a separate display interface of a separate computing device of another user who is viewing the location within the virtual environment - the steps recited at a high level of generality, and amounts to mere data outputting, which is a form of insignificant extra-solution activity (see MPEP § 2106.05(g)).
Dependent claim 4:
wherein the interaction with the virtual feature includes the user providing a spoken utterance to the automated assistant application while the user is viewing the virtual feature via the display interface of the computing device, and wherein the spoken utterance describes the textual content without providing any invocation phrase for the automated assistant application - the steps recited at a high level of generality, and amounts to mere data gathering, which is a form of insignificant extra-solution activity (see MPEP § 2106.05(g)).
Dependent claim 6:
causing, based on the virtual feature data, the automated assistant application to initialize warm word detection for one or more particular spoken words, wherein the warm word detection causes the automated assistant application to monitor for occurrence of speaking of the one or more particular spoken words, independent of detecting an invocation phrase, and to control the one or more operations in response to detecting, during monitoring for the occurrence, of speaking of the one or more particular spoken words - the steps recited at a high level of generality, and amounts to more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Dependent claim 7:
wherein causing the automated assistant application to initialize the warm word detection for the one or more particular spoken words is performed further based on the virtual object being rendered within a field of view of the user while the user is accessing the virtual environment - the steps recited at a high level of generality, and amounts to more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Dependent claim 8:
wherein the virtual environment includes an avatar that represents a position of the user within the virtual environment - the steps recited at a high level of generality, and amounts to mere data outputting, which is a form of insignificant extra-solution activity (see MPEP § 2106.05(g)), and
wherein causing the automated assistant to initialize the warm word detection for the one or more particular spoken words is performed further based on the virtual object being within a threshold distance of the avatar for the user - the steps recited at a high level of generality, and amounts to more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Dependent claim 9:
wherein the virtual environment includes an avatar that represents a position of the user within the virtual environment - the steps recited at a high level of generality, and amounts to mere data outputting, which is a form of insignificant extra-solution activity (see MPEP § 2106.05(g)), and
wherein causing the automated assistant to initialize warm word detection for the one or more particular spoken words is performed further based on the virtual object being within a threshold distance of the avatar for the user, and/or within a field of view of the user, during a most recent threshold duration of time - the steps recited at a high level of generality, and amounts to more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Independent claim 15:
A method implemented by one or more processors, an automated assistant application, by the automated assistant application provided by a first entity; wherein the separate application is provided by a second entity different from the first entity; wherein the user accesses the virtual environment by viewing a display interface of a computing device - These limitations amount to components of a general purpose computer that applies a judicial exception, by use of conventional computer functions (see MPEP § 2106.05(b)).
wherein the user selects the one or more parameters by further interacting with the automated assistant and/or further interacting with the virtual environment - the steps recited at a high level of generality, and amounts to mere data inputting, which is a form of insignificant extra-solution activity (see MPEP § 2106.05(g)).
wherein the virtual feature data characterizes an additional virtual feature that can be rendered in the virtual environment with, or to replace, the virtual feature - the steps recited at a high level of generality, and amounts to mere data outputting, which is a form of insignificant extra-solution activity (see MPEP § 2106.05(g)).
causing, by the automated assistant application and based on the virtual feature data, rendering of the additional virtual feature in the virtual environment,
wherein a selection of the additional virtual feature causes the automated assistant application to initialize performance of the operation in accordance with the one or more parameters previously selected by the user - the steps recited at a high level of generality, and amounts to mere data outputting, which is a form of insignificant extra-solution activity (see MPEP § 2106.05(g)).
Dependent claim 17:
wherein the additional virtual feature includes virtual content that characterizes the one or more parameters, and the virtual feature does not include the virtual content - the step recited at a high level of generality, and amounts to mere data description, which is a form of insignificant extra-solution activity (see MPEP § 2106.05(g)).
Dependent claim 18:
wherein causing rendering of the additional virtual feature in the virtual environment includes: causing the additional virtual feature to be rendered on a virtual surface of a virtual object that is provided by the separate application, and the operation is performed by the separate application, wherein an entity that provides the separate application is different from the first entity that provides the automated assistant and also different from a third entity that provides the virtual environment - the steps recited at a high level of generality, and amounts to mere data outputting, which is a form of insignificant extra-solution activity (see MPEP § 2106.05(g)).
Dependent claim 19:
wherein performance of the operation includes causing an output interface of a separate computing device in a physical environment of the user to render audio and/or visual output - the steps recited at a high level of generality, and amounts to mere data outputting, which is a form of insignificant extra-solution activity (see MPEP § 2106.05(g)).
Dependent claim 20:
causing, in response to a separate user selecting the additional virtual feature, a separate instance of the automated assistant application to initialize execution of the operation for the separate user - the steps recited at a high level of generality, and amounts to mere data outputting, which is a form of insignificant extra-solution activity (see MPEP § 2106.05(g)).
Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claims are thus directed to the abstract idea.
Step 2B: The claims do not include additional elements that amount to significantly more than the judicial exception.
The additional elements:
Independent claim 1:
A method implemented by one or more processors, an automated assistant application provided by a first entity; by the automated assistant application accessing the virtual environment application; provided by a second entity different from the first entity; wherein the user views the virtual environment via a display interface of a computing device; one or more operations that are capable of being controlled via a separate application - These limitations amount to components of a general purpose computer that applies a judicial exception, by use of conventional computer functions (see MPEP § 2106.05(b)).
causing, based on the virtual feature data and by the automated assistant application, rendering of the virtual feature in the virtual environment, wherein the virtual feature is rendered at a location within the virtual environment that is associated with the virtual object - which is a well-understood, routine, conventional activity similar to presenting offers and gathering statistics described in MPEP 2106.05(d)(II).
in response to determining that the user interacted with the virtual feature via the virtual environment: causing the automated assistant to control the one or more operations the separate application according to an interaction between the user and the virtual feature - the steps recited at a high level of generality, and amounts to more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Dependent claim 2:
wherein causing rendering of the virtual feature in the virtual environment causes the virtual feature to be rendered at a separate display interface of a separate computing device of another user who is viewing the location within the virtual environment - which is a well-understood, routine, conventional activity similar to presenting offers and gathering statistics described in MPEP 2106.05(d)(II).
Dependent claim 4:
wherein the interaction with the virtual feature includes the user providing a spoken utterance to the automated assistant application while the user is viewing the virtual feature via the display interface of the computing device, and wherein the spoken utterance describes the textual content without providing any invocation phrase for the automated assistant application - which is a well-understood, routine, conventional activity similar to receiving or transmitting data over a network described in MPEP 2106.05(d)(II).
Dependent claim 6:
causing, based on the virtual feature data, the automated assistant application to initialize warm word detection for one or more particular spoken words, wherein the warm word detection causes the automated assistant application to monitor for occurrence of speaking of the one or more particular spoken words, independent of detecting an invocation phrase, and to control the one or more operations in response to detecting, during monitoring for the occurrence, of speaking of the one or more particular spoken words - the steps recited at a high level of generality, and amounts to more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Dependent claim 7:
wherein causing the automated assistant application to initialize the warm word detection for the one or more particular spoken words is performed further based on the virtual object being rendered within a field of view of the user while the user is accessing the virtual environment - the steps recited at a high level of generality, and amounts to more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Dependent claim 8:
wherein the virtual environment includes an avatar that represents a position of the user within the virtual environment - which is a well-understood, routine, conventional activity similar to presenting offers and gathering statistics described in MPEP 2106.05(d)(II).
wherein causing the automated assistant to initialize the warm word detection for the one or more particular spoken words is performed further based on the virtual object being within a threshold distance of the avatar for the user - the steps recited at a high level of generality, and amounts to more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Dependent claim 9:
wherein the virtual environment includes an avatar that represents a position of the user within the virtual environment - which is a well-understood, routine, conventional activity similar to presenting offers and gathering statistics described in MPEP 2106.05(d)(II).
wherein causing the automated assistant to initialize warm word detection for the one or more particular spoken words is performed further based on the virtual object being within a threshold distance of the avatar for the user, and/or within a field of view of the user, during a most recent threshold duration of time - the steps recited at a high level of generality, and amounts to more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Independent claim 15:
A method implemented by one or more processors, an automated assistant application; by the automated assistant application provided by a first entity; wherein the separate application is provided by a second entity different from the first entity; wherein the user accesses the virtual environment by viewing a display interface of a computing device - These limitations amount to components of a general purpose computer that applies a judicial exception, by use of conventional computer functions (see MPEP § 2106.05(b)).
wherein the user selects the one or more parameters by further interacting with the automated assistant and/or further interacting with the virtual environment - which is a well-understood, routine, conventional activity similar to receiving or transmitting data over a network described in MPEP 2106.05(d)(II).
wherein the virtual feature data characterizes an additional virtual feature that can be rendered in the virtual environment with, or to replace, the virtual feature - which is a well-understood, routine, conventional activity similar to receiving or transmitting data over a network described in MPEP 2106.05(d)(II).
causing, by the automated assistant application and based on the virtual feature data, rendering of the additional virtual feature in the virtual environment,
wherein a selection of the additional virtual feature causes the automated assistant application to initialize performance of the operation in accordance with the one or more parameters previously selected by the user - which is a well-understood, routine, conventional activity similar to receiving or transmitting data over a network described in MPEP 2106.05(d)(II).
Dependent claim 17:
wherein the additional virtual feature includes virtual content that characterizes the one or more parameters, and the virtual feature does not include the virtual content - viewed individually or in combination, describes selecting a particular data source or type of data to be manipulated similar to selecting information, based on types of information and availability of information in a power-grid environment, for collection, analysis and display described in MPEP § 2106.05(g).
Dependent claim 18:
wherein causing rendering of the additional virtual feature in the virtual environment includes: causing the additional virtual feature to be rendered on a virtual surface of a virtual object that is provided by the separate application, and the operation is performed by the separate application, wherein an entity that provides the separate application is different from the first entity that provides the automated assistant and also different from a third entity that provides the virtual environment - which is a well-understood, routine, conventional activity similar to receiving or transmitting data over a network described in MPEP 2106.05(d)(II).
Dependent claim 19:
wherein performance of the operation includes causing an output interface of a separate computing device in a physical environment of the user to render audio and/or visual output - which is a well-understood, routine, conventional activity similar to presenting offers and gathering statistics described in MPEP 2106.05(d)(II).
Dependent claim 20:
causing, in response to a separate user selecting the additional virtual feature, a separate instance of the automated assistant application to initialize execution of the operation for the separate user which is a well-understood, routine, conventional activity similar to presenting offers and gathering statistics described in MPEP 2106.05(d)(II).
Accordingly, these additional elements do not amount to significantly more than the judicial exception. As such, the claims are ineligible.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1 and 3-7 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Poulos et al. (hereinafter Poulos), US 20140237366 A1.
Regarding independent claim 1, Poulos teaches a method implemented by one or more processors ([0070]), the method comprising:
determining, by an automated assistant application provided by a first entity, that a user is viewing a portion of a virtual environment that includes a virtual object ([0055] FIG. 7 shows an example method 700 for operating a user interface on an augmented reality computing device (i.e. a first entity) comprising a see-through display system and a sensor system including one or more sensors, e.g. including an eye tracking sensor; [0056] At 702, method 700 includes displaying one or more virtual images such that a field of view of a user through the see-through display system comprises a set of augmented reality features. The set of augmented reality features may be based upon a type, content of, and/or property of the virtual images and/or real objects. For example, the augmented reality features may include real or virtual objects selectable by a user. Further, the features also may include sounds or other augmented reality environment aspects; [0057] At 704, method 700 includes receiving a user input selecting an object. Examples of user inputs may include gaze inputs, voice inputs, gesture inputs, posture inputs, body motion inputs, eye tracking inputs, and/or any other suitable inputs. As one more specific non-limiting example, an object may be selected in response to determining that the user has gazed at the selected object for a duration greater than a first time threshold. This is shown at 706, where method 700 may include determining that a user has gazed at the selected object),
wherein the user views the virtual environment via a display interface of a computing device ([0014] FIG. 1 shows an example embodiment of a use environment for an embodiment of a head-mounted display device 104, wherein environment 100 takes the form of a living room. A user 106 is viewing the living room through an augmented reality computing device in the form of the see-through head-mounted display device 104, and may interact with the augmented environment via a user interface of display device 104. FIG. 1 also depicts a user field of view 102 comprising a portion of the environment viewable through the display device 104, and thus the portion of the environment that may be augmented with images displayed via the display device 104);
determining, by the automated assistant application accessing the virtual environment application and based on the virtual object, that the virtual object is associated with one or more operations that are capable of being controlled via a separate application provided by a second entity different from the first entity ([0058] At 708, method 700 includes determining a first group of commands that are operable based on one or more of an identification of the selected object and a state of the object; [0029] FIG. 4 shows example embodiments of interactions at 402, 404, 406, and 408 of a user 106 wearing a see-through display device 104 with objects in a field of view 102 of the see-through display device 104. By way of example, FIG. 4 shows a first object 410 of a first type and a second object 414 of a second type in an augmented reality environment. First object 410 and second object 414 may be any suitable objects within an augmented reality environment, and may be virtual objects or real objects. For example, first object 410 may be a virtual television, such as virtual television device 111 shown in FIG. 1, overlaid onto a background scene whereas second object 414 may be a media device such as device 115 of FIG. 1, or a telephony device such as device 113 of FIG. 1. As described in more detail below with regard to FIG. 6, each object in an augmented reality environment may have different groups of commands associated with it, depending on the type of object and state of the object; [0020] FIG. 1 shows example gaze lines directed at various objects in the augmented reality environment, wherein the gaze lines represent the user focusing on the object at which the gaze lines intersect to select the object. For example, user focus 130 may indicate that user 106 desires to select music or media device 115, user focus 132 may indicate that user 106 desires to select telephony device 113, and user focus 134 may indicate that user 106 desires to select virtual television device 111. In some examples, a user may only select a single object in the augmented reality environment in order to perform an action on the object. In other examples, a user may select a plurality of objects in the augmented reality environment in order to perform actions on each object in the plurality of objects. For example, user 106 may select both media device 115 and virtual television device 111 in order to perform a command to reduce a volume of both devices. As another example, a global action may be performed on multiple objects in the environment when no objects are selected by a user. For example, if a user is looking out towards no particular object and says "raise volume," the application, operating system or other global software recognizes that no objects have focus and may raise a global setting for volume on all objects that recognize a "raise volume" command in a current state; Fig. 5; [0036] As another example, as shown in FIG. 5, presentation of commands associated with the selected object may be activated by user/object proximity; [0037] For example, as shown in FIG. 5 at 502, when user 106 is a first distance 512 away from an object 508 (i.e. a second entity), user 106 may select object 508 by gazing at or focusing on object 508 with a focus 510. However, at first distance 512, user 106 may be too far away from object 508 for some commands associated with the object to be applicable. Thus, even though user 106 selects object 508 at 502, no commands may be presented to the user. Next, as the user moves closer to object 508, a first group of commands available for the object may be displayed to the user. This is shown at 504, where, as user 106 moves toward object 508 to a second distance 514 from object 508 that is less than first distance 512, a first group of available commands 518 is presented to user 106. For example, if object 508 is a television device, then at distance 514, user 106 may be close enough to the device to perform certain actions, such as play or pause, but may still be too far away to perform other actions on the device, such as select interactive objects within an image displayed on the television object. As the user moves even closer to object 508, even more commands may become available and may be presented to the user. This is shown at 506, where, as user 106 moves to a third distance 516 less than the second distance 514 from the object 508, a second group of available commands 520 is presented to user 106. The second group of commands 520 may includes a greater number of commands than the first group of commands 518, a lesser number of commands, or a same number of different commands, or a same number of same commands. For example, if object 508 is a television device then the second group of commands 520 may include the first group of commands 518, e.g., play and pause, plus additional available commands, e.g., that regard interaction with an in-picture object);
generating, based on the one or more operations associated with the virtual object, virtual feature data that characterizes a virtual feature for controlling the one or more operations when the user interacts with the virtual feature via the virtual environment ([0048] FIG. 6 shows example context-aware speech recognition key phrase groups for a first object type 602 and a second object type 604 during different states. For example, first object type 602 may have a first state 606 and a first group of commands 614 associated with the first state 606. The first group of commands 614 may include a plurality of commands, e.g., command 1 through command N, which are recognizable by a selected object with the first object type in the first state. Each command in the first group of commands 614 may have a plurality of key phrases associated with the command. For example, command 1 has key phrase 1 through key phrase N associated with it; [0049] The first object type 602 may also have a second state 608 and a second group of commands 616 associated the second state 608. The second group of commands 616 may include a plurality of commands, e.g., command 1 through command N, which are recognizable by a selected object with the first object type in the second state. Each command in the second group of commands 616 may have a plurality of key phrases associated with the command. For example, command 1 has key phrase 1 through key phrase N associated with it; [0060] various triggering conditions may be used to determine whether to present the first group of commands to a user);
causing, based on the virtual feature data and by the automated assistant application, rendering of the virtual feature in the virtual environment, wherein the virtual feature is rendered at a location within the virtual environment that is associated with the virtual object ([0059] At 712, method 700 includes presenting the first group of commands to the user. For example, presenting the first group of commands to the user may include displaying representations of commands in the first group of commands, potentially at a location adjacent to the selected object, via the see-through display system); and
in response to determining that the user interacted with the virtual feature via the virtual environment: causing the automated assistant to control the separate application according to an interaction between the user and the virtual feature ([0062] At 722, method 700 includes receiving a command from the first group of commands. For example, at 724, method 700 may include receiving speech input of a key phrase associated with the command. In response, method 700 comprises, at 726, performing the command).
Regarding dependent claim 3, Poulos teaches all the limitations as set forth in the rejection of claim 1 that is incorporated. Poulos further teaches wherein generating the virtual feature data that characterizes the virtual feature includes: determining textual content to include with the virtual feature, wherein the textual content indicates the one or more operations that can be controlled by interacting with the virtual feature ([0017] A user, e.g., user 106, of an augmented reality user interface may be able to perform any suitable actions on real and virtual objects in an augmented reality environment. However, as mentioned above, the interactions available to the user may depend on the object type and/or object state. For example, if the selected object is a virtual television device 111, then commands available for virtual television device 111 may depend on a state of television device 111. As a more specific example, if virtual television device 111 is playing a movie, then commands such as "pause," "forward," and "rewind" may be available for the user to select in that state, whereas commands such as "play" may not be available. Likewise, if virtual television device 111 is not playing a movie when selected, then commands such as "play," "forward," and "rewind" may be available for the user to select whereas other commands, such as "pause," may not be available for selection; [0039] The representations of commands displayed to user 106 may take any suitable form. For example, the representations may include icons, floating three-dimensional signs, lists of available command options, and other indications of available commands. As a more specific example, a text menu of available actions may be wrapped onto the surfaces of real-world walls, e.g., as discovered with surface reconstruction technology).
Regarding dependent claim 4, Poulos teaches all the limitations as set forth in the rejection of claim 1 that is incorporated. Poulos further teaches
wherein the interaction with the virtual feature includes the user providing a spoken utterance to the automated assistant application while the user is viewing the virtual feature via the display interface of the computing device, and wherein the spoken utterance describes the textual content without providing any invocation phrase for the automated assistant application ([0048] FIG. 6 shows example context-aware speech recognition key phrase groups for a first object type 602 and a second object type 604 during different states; [0062] At 722, method 700 includes receiving a command from the first group of commands. For example, at 724, method 700 may include receiving speech input of a key phrase associated with the command; [0052] In some examples, these key phrases may be handled by a speech recognition system at an application level for an application running on an augmented reality computing device or at an operating system level of an augmented reality computing device. For example, a user may provide speech input of one of the key phrases associated with a command in the first group of commands and the application, the operating system, or the speech recognition system may interpret the speech input of a key phrase into the associated command and send the associated command to the user selected objects).
Regarding dependent claim 5, Poulos teaches all the limitations as set forth in the rejection of claim 1 that is incorporated. Poulos further teaches wherein generating the virtual feature data that characterizes the virtual feature includes: determining textual content to include with the virtual feature, wherein the textual content indicates the separate application that can be controlled by interacting with the virtual feature ([0017] A user, e.g., user 106, of an augmented reality user interface may be able to perform any suitable actions on real and virtual objects in an augmented reality environment. However, as mentioned above, the interactions available to the user may depend on the object type and/or object state. For example, if the selected object is a virtual television device 111, then commands available for virtual television device 111 may depend on a state of television device 111. As a more specific example, if virtual television device 111 is playing a movie, then commands such as "pause," "forward," and "rewind" may be available for the user to select in that state, whereas commands such as "play" may not be available. Likewise, if virtual television device 111 is not playing a movie when selected, then commands such as "play," "forward," and "rewind" may be available for the user to select whereas other commands, such as "pause," may not be available for selection; [0018] Since the list of actions may vary between objects and also over time as an object changes state, it may be difficult for a user to know which actions are available in a current context of a selected object. Thus, according to embodiments disclosed herein, augmented reality images displayed through display device 104, potentially in combination with information regarding use environment 100, may be used to present representations of commands associated with user-selected objects on display device 104. For example, upon selection by user 106 of one or more real or virtual objects in the augmented environment, display device 104 may present one or more augmented reality features that indicate possible actions which may be performed on the selected objects).
Regarding dependent claim 6, Poulos teaches all the limitations as set forth in the rejection of claim 1 that is incorporated. Poulos further teaches further comprising:
causing, based on the virtual feature data, the automated assistant application to initialize warm word detection for one or more particular spoken words, wherein the warm word detection causes the automated assistant application to monitor for occurrence of speaking of the one or more particular spoken words, independent of detecting an invocation phrase, and to control the one or more operations in response to detecting, during monitoring for the occurrence, of speaking of the one or more particular spoken words ([0052] In some examples, these key phrases may be handled by a speech recognition system at an application level for an application running on an augmented reality computing device or at an operating system level of an augmented reality computing device. For example, a user may provide speech input of one of the key phrases associated with a command in the first group of commands and the application, the operating system, or the speech recognition system may interpret the speech input of a key phrase into the associated command and send the associated command to the user selected objects; [0053] As a more specific example, in an environment containing a floating holographic television object selected by a user, the speech recognition system may associate the key phrases "load video," "play movie," and "start" with a "play" command recognizable by the television object. For example, the television object may not recognize the key phrases "load video," "play movie," and "start" but may only recognize the command "play." In this example, the speech recognition system may translate the key phrases "load video," "play movie," and "start" into the command "play" sent to the television object to perform the play command; [0054] Further, key phrases associated with commands may be selectively activated or deactivated in response to a change in context, changes in user selections of objects, and changes in states of the selected objects. For example, individual objects may enable or disable groups of key phrases as they go through context transitions).
Regarding dependent claim 7, Poulos teaches all the limitations as set forth in the rejection of claim 6 that is incorporated. Poulos further teaches wherein causing the automated assistant application to initialize the warm word detection for the one or more particular spoken words is performed further based on the virtual object being rendered within a field of view of the user while the user is accessing the virtual environment ([0061] Further, as mentioned above, distance also may be used as a trigger to display commands. Thus, at 720, method 700 may include increasing a number of commands presented to the user in response to a decrease in a user distance from the selected object. For example, a number of commands in the first group of commands presented to the user may be increased in response to detecting a decrease in distance between the user and the selected object).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Poulos as applied in claim 1, in view of FAULKNER, US 20200184217 A1.
Regarding dependent claim 2, Poulos teaches all the limitations as set forth in the rejection of claim 1 that is incorporated.
Poulos does not explicitly teach wherein causing rendering of the virtual feature in the virtual environment causes the virtual feature to be rendered at a separate display interface of a separate computing device of another user who is viewing the location within the virtual environment.
However, in the same field of endeavor, FAULKNER teaches wherein causing rendering of the virtual feature in the virtual environment causes the virtual feature to be rendered at a separate display interface of a separate computing device of another user who is viewing the location within the virtual environment ([0024] FIG. 1 illustrates an example user interface 100 displaying aspects of a collaborative environment that may be utilized to facilitate aspects of the present disclosure. In this example, an image sensor 105, e.g., a camera, of the computing device 101 is used to capture an image of the real-world object 103 within a real-world environment 112. In this illustrative example, the real-world object 103 is an engine. The computing device 101 can share an image of the real-world object 103 with a number of individual users 102A-102D (“users 102”) that are communicating within the collaborative environment. The computing device 101 can also coordinate with other computing devices to generate a shared display of virtual objects 104 that are rendered with a view of a real-world object 103; [0025] The users 102 can create and edit the virtual objects 104 by the use of voice commands, movement gestures, or other types of interactions with a wide range of input devices; [0046] FIG. 6 illustrates a number of example recommendations. For instance, a recommendation can be in the form of a new virtual object 601 that is displayed as a virtual partial overlay over the image of the real-world object. In this example, the new virtual object 601 is in the form of another engine, e.g., Acme Model 1, that meets the performance requirements indicated in the data records 203. Such a recommendation may be automatically rendered as a three-dimensional or two-dimensional virtual object that is provided as an overlay over real-world objects or other virtual objects. By providing an overlay over real-world objects, users can readily visualize a proposal without requiring users to manually manipulate content).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of providing intelligent agents for managing data associated with objects that are displayed within mixed-reality and virtual-reality collaboration environments as suggested in FAULKNER into Poulos’s system because both of these systems are addressing display of virtual objects that are rendered with a view of a real-world object. This modification would have been motivated by the desire to improve the efficiency of a system by providing intelligent agents for managing data associated with real-world objects and virtual objects that are displayed within collaborative environments (FAULKNER, [0004]).
Claims 8-9 are rejected under 35 U.S.C. 103 as being unpatentable over Poulos as applied in claim 6, in view of Amidon, US 7840903 B1.
Regarding dependent claim 8, Poulos teaches all the limitations as set forth in the rejection of claim 6 that is incorporated.
Poulos further teaches wherein causing the automated assistant to initialize the warm word detection for the one or more particular spoken words is performed further based on the virtual object being within a threshold distance of ([0019] User 106 may select an object for interaction in any suitable manner detectable by display device 104. For example, user 106 may select an object by gazing at the object, wherein a gaze direction may be detected by one or more eye tracking sensors, potentially for a threshold amount of time. In another example, user 106 may select an object by moving his or her head, hand, entire body, etc., to or towards the object, as detected by inertial motion sensors and or image sensors. A user also may issue voice commands detected by one or more microphones; [0034] This presentation of available commands for a selected object may be triggered in a variety of ways. For example, the presentation of commands may be triggered in response to user 106 gazing or focusing on the selected object for a threshold time duration, e.g., at least one second. As another example, a menu of available commands may be activated and presented to the user in response to a degree of user stability as detected via eye tracking sensors and/or motion sensors. For example, if user 106 stands still and looks at first object 410 with minimal changes to body position and view direction for a threshold time duration, then menu visibility may be enabled. As another example, a command menu may be activated in response to user 106 gazing or focusing on a secondary embedded focus target, e.g., target region 412. For example, user 106 may look at an icon or glowing particle cluster overlaid on the object, enabling menu visibility. As another example, presentation of commands associated with a selected object may be activated by speech input received from user 106. For example, user 106 may speak a key phrase, such as "help me" to activate a display of commands associated with the selected object).
Poulos does not explicitly teaches wherein the virtual environment includes an avatar that represents a position of the user within the virtual environment, and
However, in the same field of endeavor, Amidon teaches wherein the virtual environment includes an avatar that represents a position of the user within the virtual environment (Col 2, lines 56-57 FIG. 1A illustrates a view of an exemplary virtual environment provided to a user represented by an avatar 10).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of representing the users by avatars in a virtual environment and the group content representation in a view of the virtual environment provided to the interested user as suggested in Amidon into Poulos’s system because both of these systems are addressing generating content representation presented in a view of the virtual environment. This modification would have been motivated by the desire for a system and method for representing content shared or otherwise made available in such environments in a manner that enables users to intuitively identify other users having content of interest (Amidon, Col 1, lines 18-29).
Regarding dependent claim 9, Poulos teaches all the limitations as set forth in the rejection of claim 6 that is incorporated. Poulos further teaches wherein causing the automated assistant to initialize warm word detection for the one or more particular spoken words is performed further based on the virtual object being within a threshold distance of ([0019] User 106 may select an object for interaction in any suitable manner detectable by display device 104. For example, user 106 may select an object by gazing at the object, wherein a gaze direction may be detected by one or more eye tracking sensors, potentially for a threshold amount of time. In another example, user 106 may select an object by moving his or her head, hand, entire body, etc., to or towards the object, as detected by inertial motion sensors and or image sensors. A user also may issue voice commands detected by one or more microphones; [0034] This presentation of available commands for a selected object may be triggered in a variety of ways. For example, the presentation of commands may be triggered in response to user 106 gazing or focusing on the selected object for a threshold time duration, e.g., at least one second. As another example, a menu of available commands may be activated and presented to the user in response to a degree of user stability as detected via eye tracking sensors and/or motion sensors. For example, if user 106 stands still and looks at first object 410 with minimal changes to body position and view direction for a threshold time duration, then menu visibility may be enabled. As another example, a command menu may be activated in response to user 106 gazing or focusing on a secondary embedded focus target, e.g., target region 412. For example, user 106 may look at an icon or glowing particle cluster overlaid on the object, enabling menu visibility. As another example, presentation of commands associated with a selected object may be activated by speech input received from user 106. For example, user 106 may speak a key phrase, such as "help me" to activate a display of commands associated with the selected object).
Poulos does not explicitly teaches wherein the virtual environment includes an avatar that represents a position of the user within the virtual environment.
However, in the same field of endeavor, Amidon teaches wherein the virtual environment includes an avatar that represents a position of the user within the virtual environment (Col 2, lines 56-57 FIG. 1A illustrates a view of an exemplary virtual environment provided to a user represented by an avatar 10).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of representing the users by avatars in a virtual environment and the group content representation in a view of the virtual environment provided to the interested user as suggested in Amidon into Poulos’s system because both of these systems are addressing generating content representation presented in a view of the virtual environment. This modification would have been motivated by the desire for a system and method for representing content shared or otherwise made available in such environments in a manner that enables users to intuitively identify other users having content of interest (Amidon, Col 1, lines 18-29).
Claims 15-20 are rejected under 35 U.S.C. 103 as being unpatentable over Poulos, in view of Wenzel, US 20220200979 A1.
Regarding independent claim 15, Poulos teaches
a method implemented by one or more processors ([0070]), the method comprising:
determining, by an automated assistant application provided by a first entity, that a user has selected a virtual feature of a virtual environment in furtherance of controlling an operation a separate application ([0055] FIG. 7 shows an example method 700 for operating a user interface on an augmented reality computing device (i.e. a first entity) comprising a see-through display system and a sensor system including one or more sensors, e.g. including an eye tracking sensor; [0057] At 704, method 700 includes receiving a user input selecting an object. Examples of user inputs may include gaze inputs, voice inputs, gesture inputs, posture inputs, body motion inputs, eye tracking inputs, and/or any other suitable inputs. As one more specific non-limiting example, an object may be selected in response to determining that the user has gazed at the selected object for a duration greater than a first time threshold. This is shown at 706, where method 700 may include determining that a user has gazed at the selected object),
wherein the separate application is provided by a second entity different from the first entity ([0020] FIG. 1 shows example gaze lines directed at various objects in the augmented reality environment, wherein the gaze lines represent the user focusing on the object at which the gaze lines intersect to select the object. For example, user focus 130 may indicate that user 106 desires to select music or media device 115, user focus 132 may indicate that user 106 desires to select telephony device 113, and user focus 134 may indicate that user 106 desires to select virtual television device 111. In some examples, a user may only select a single object in the augmented reality environment in order to perform an action on the object. In other examples, a user may select a plurality of objects in the augmented reality environment in order to perform actions on each object in the plurality of objects. For example, user 106 may select both media device 115 and virtual television device 111 in order to perform a command to reduce a volume of both devices. As another example, a global action may be performed on multiple objects in the environment when no objects are selected by a user. For example, if a user is looking out towards no particular object and says "raise volume," the application, operating system or other global software recognizes that no objects have focus and may raise a global setting for volume on all objects that recognize a "raise volume" command in a current state; Fig. 5; [0036] As another example, as shown in FIG. 5, presentation of commands associated with the selected object may be activated by user/object proximity; [0037] For example, as shown in FIG. 5 at 502, when user 106 is a first distance 512 away from an object 508 (i.e. a second entity), user 106 may select object 508 by gazing at or focusing on object 508 with a focus 510. However, at first distance 512, user 106 may be too far away from object 508 for some commands associated with the object to be applicable. Thus, even though user 106 selects object 508 at 502, no commands may be presented to the user. Next, as the user moves closer to object 508, a first group of commands available for the object may be displayed to the user. This is shown at 504, where, as user 106 moves toward object 508 to a second distance 514 from object 508 that is less than first distance 512, a first group of available commands 518 is presented to user 106. For example, if object 508 is a television device, then at distance 514, user 106 may be close enough to the device to perform certain actions, such as play or pause, but may still be too far away to perform other actions on the device, such as select interactive objects within an image displayed on the television object. As the user moves even closer to object 508, even more commands may become available and may be presented to the user. This is shown at 506, where, as user 106 moves to a third distance 516 less than the second distance 514 from the object 508, a second group of available commands 520 is presented to user 106. The second group of commands 520 may includes a greater number of commands than the first group of commands 518, a lesser number of commands, or a same number of different commands, or a same number of same commands. For example, if object 508 is a television device then the second group of commands 520 may include the first group of commands 518, e.g., play and pause, plus additional available commands, e.g., that regard interaction with an in-picture object)),
wherein the user accesses the virtual environment by viewing a display interface of a computing device ([0056] At 702, method 700 includes displaying one or more virtual images such that a field of view of a user through the see-through display system comprises a set of augmented reality features. The set of augmented reality features may be based upon a type, content of, and/or property of the virtual images and/or real objects. For example, the augmented reality features may include real or virtual objects selectable by a user. Further, the features also may include sounds or other augmented reality environment aspects);
determining, by the automated assistant application, that the user has selected one or more parameters for the operation, wherein the user selects the one or more parameters by further interacting with the automated assistant and/or further interacting with the virtual environment ([0034] presentation of commands associated with a selected object may be activated by speech input received from user 106. For example, user 106 may speak a key phrase, such as "help me" to activate a display of commands associated with the selected object).
Poulos does not explicitly teach
determining, based on the user selecting the one or more parameters, that the user has previously selected the one or more parameters for the operation during a prior interaction between the user and the automated assistant application;
generating, based on the user having previously selected the one or more parameters during the prior interaction between the user and the automated assistant application, virtual feature data,
wherein the virtual feature data characterizes an additional virtual feature that can be rendered in the virtual environment with, or to replace, the virtual feature; and
causing, by the automated assistant application and based on the virtual feature data, rendering of the additional virtual feature in the virtual environment,
wherein a selection of the additional virtual feature causes the automated assistant application to initialize performance of the operation in accordance with the one or more parameters previously selected by the user.
However, in the same field of endeavor, Wenzel teaches
determining, based on the user selecting the one or more parameters, that the user has previously selected the one or more parameters for the operation during a prior interaction between the user and the automated assistant application ([0073] if a user collaborates regularly with another co-worker who is also a user of the virtual space, then the option to visit such a user may be provided to the user; [0074] Returning to operation 401, on accessing the virtual space the user may choose to search for a user. For example, a user may wish to have a conversation with a specific employee. A search function provides the user with the ability to locate the user with which they wish to speak rapidly);
generating, based on the user having previously selected the one or more parameters during the prior interaction between the user and the automated assistant application, virtual feature data, wherein the virtual feature data characterizes an additional virtual feature that can be rendered in the virtual environment with, or to replace, the virtual feature ([0073] On accessing the virtual space at operation 401 a user may choose to view a recently visited user at operation 405. For example, if a user collaborates regularly with another co-worker who is also a user of the virtual space, then the option to visit such a user may be provided to the user. A list of users to select may be provided to the user. The list may be generated on the basis of most recently visited users, most frequently visited users, a priority list generated by the user, or any suitable preference of the company and/or the user for convenience. The list may include a list of sub-spaces together with their current users. The list may additionally include the status of the users); and
causing, by the automated assistant application and based on the virtual feature data, rendering of the additional virtual feature in the virtual environment ([0073] if a user collaborates regularly with another co-worker who is also a user of the virtual space, then the option to visit such a user may be provided to the user. A list of users to select may be provided to the user. The list may be generated on the basis of most recently visited users, most frequently visited users, a priority list generated by the user, or any suitable preference of the company and/or the user for convenience),
wherein a selection of the additional virtual feature causes the automated assistant application to initialize performance of the operation in accordance with the one or more parameters previously selected by the user ([0076] On selection of a user's sub-space at operations 405 and 406, more detailed information of the user may be provided. For example, the detailed information may comprise profile information associated with the user's sub-space, such as job title, department, etc.; [0077] After either of operations 405 or 406, the user may select a viewed a searched user and request to visit that user at operation 407).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of providing a list of users to select wherein the list generated on the basis of most recently visited users, most frequently visited users, a priority list generated by the user as suggested in Wenzel into Poulos’s system because both of these systems are addressing generating a virtual space comprising one or more virtual subspaces. This modification would have been motivated by the desire in providing a virtual space such as a virtual office environment in which users may interact in a convenient and intuitive manner (Wenzel, [0028]).
Regarding dependent claim 16, the combination of Poulos and Wenzel teaches all the limitations as set forth in the rejection of claim 15 that is incorporated. Poulos further teaches wherein determining that the user has selected the one or more parameters for the operation includes:
determining that the user has directed a spoken utterance to the automated assistant application, and that the spoken utterance includes a warm word characterizing the one or more parameters, wherein the spoken utterance is independent of any invocation phrase for the automated assistant application ([0034] presentation of commands associated with a selected object may be activated by speech input received from user 106. For example, user 106 may speak a key phrase, such as "help me" to activate a display of commands associated with the selected object).
Regarding dependent claim 17, the combination of Poulos and Wenzel teaches all the limitations as set forth in the rejection of claim 15 that is incorporated. Wenzel further teaches wherein the additional virtual feature includes virtual content that characterizes the one or more parameters, and the virtual feature does not include the virtual content ([0073] if a user collaborates regularly with another co-worker who is also a user of the virtual space, then the option to visit such a user may be provided to the user. A list of users to select may be provided to the user. The list may be generated on the basis of most recently visited users, most frequently visited users, a priority list generated by the user, or any suitable preference of the company and/or the user for convenience. The list may include a list of sub-spaces together with their current users. The list may additionally include the status of the users).
Regarding dependent claim 18, the combination of Poulos and Wenzel teaches all the limitations as set forth in the rejection of claim 15 that is incorporated. Poulos further teaches wherein causing rendering of the additional virtual feature in the virtual environment includes:
causing the additional virtual feature to be rendered on a virtual surface of a virtual object that is provided by the separate application, and the operation is performed by the separate application ([0033] After the first object 410 is selected by the user and/or after the target region 412 is gazed upon by the user, visibility of representations of commands in a first group of commands representing currently available commands may be triggered to display to the user a current set of available actions associated with the selected object. An example of such representations is shown at 404 in FIG. 4. In this specific example, representations 422 of groups of commands are displayed to the user at a location adjacent to the first object via the see-through display device 104. As another example, representations of groups of commands may be may be overlaid over the background scene adjacent to the selected object, as shown at 424. For example, the represented group of commands could be displayed on a virtual object that is mapped to align with the surface of a real world object, e.g., a wall),
wherein an entity that provides the separate application is different from the first entity that provides the automated assistant and also different from a third entity that provides the virtual environment ([0031] First object 410 may be a television device, such a virtual television device 111 shown in FIG. 1. If user 106 selects the television device, e.g. by focusing on the device, while the television device is playing a movie, then currently available commands available for the television device may include commands such as "pause" and "adjust volume" whereas commands not available for the device in its current state may include "play." Likewise, if user 106 selects the television device while a movie is paused on the device, then available commands for the television device in its current state may include a "play" command, but not a "pause" command or an "adjust volume" command; [0032] As another example, first object 410 may represent a telephone device, e.g., telephony device 113 shown in FIG. 1. If a user selects the telephone while the telephone is ringing, e.g., to answer or decline a call, then available commands for the device in its current state may include "answer" or "ignore" whereas commands such as "call a contact" may not be available. However, if user 106 selects the telephone object during a condition when an incoming call is not occurring then commands such as "answer" or "ignore" may not be available as commands for the telephone whereas commands available for the device in its current state may include "call a contact.").
Regarding dependent claim 19, the combination of Poulos and Wenzel teaches all the limitations as set forth in the rejection of claim 15 that is incorporated. Poulos further teaches wherein performance of the operation includes causing an output interface of a separate computing device in a physical environment of the user to render audio and/or visual output ([0014] FIG. 1 also depicts a user field of view 102 comprising a portion of the environment viewable through the display device 104, and thus the portion of the environment that may be augmented with images displayed via the display device 104. The augmented environment may include a plurality of real and virtual objects, e.g., media devices, with which the user may interact. For example, the augmented environment may include a virtual television device 111, a real telephone device 113, and a real media player device 115. Each of these objects in the augmented environment may be selectable by user 106 so that the user may perform actions on the selected object; [0032] As another example, first object 410 may represent a telephone device, e.g., telephony device 113 shown in FIG. 1. If a user selects the telephone while the telephone is ringing, e.g., to answer or decline a call, then available commands for the device in its current state may include "answer" or "ignore" whereas commands such as "call a contact" may not be available. However, if user 106 selects the telephone object during a condition when an incoming call is not occurring then commands such as "answer" or "ignore" may not be available as commands for the telephone whereas commands available for the device in its current state may include "call a contact.").
Regarding dependent claim 20, the combination of Poulos and Wenzel teaches all the limitations as set forth in the rejection of claim 15 that is incorporated. Wenzel further teaches further comprising:
causing, in response to a separate user selecting the additional virtual feature, a separate instance of the automated assistant application to initialize execution of the operation for the separate user ([0014] The method may further comprise: receiving a visit request from the first user device to visit a virtual sub-space occupied by the second user at the second user device; and in response to receiving a visit approval response from the second user device, providing access to the virtual sub-space occupied by the second user to the first user such that a communication link is established between the first user device and the second user device).
Response to Arguments
Applicant's arguments filed 01/20/2026 have been fully considered. Each of applicant’s remarks is set forth, followed by examiner’s response.
(1) Regarding 35 U.S.C. 101 Rejection(s), Applicant states in an effort to advance prosecution, and without conceding the propriety of the Office Action's 101 rejections, claims 1 and 15 are amended herein. The Applicant's attorney respectfully requests reconsideration of the Office Action's 101 rejection at least in view of the amendments included herein.
As to point (1), newly amended limitations “determining, by an automated assistant application provided by a first entity”, “determining, by the automated assistant application accessing the virtual environment application” and “a separate application provided by a second entity different from the first entity” provide nothing more than mere instructions to implement an abstract idea on a generic computer. See MPEP 2106.05(f). MPEP 2106.05(f) provides the following considerations for determining whether a claim simply recites a judicial exception with the words “apply it” (or an equivalent), such as mere instructions to implement an abstract idea on a computer: (1) whether the claim recites only the idea of a solution or outcome i.e., the claim fails to recite details of how a solution to a problem is accomplished; (2) whether the claim invokes computers or other machinery merely as a tool to perform an existing process; and (3) the particularity or generality of the application of the judicial exception. The claim does not provide any details about how the automated assistant application provided by a first entity operates or how the detection is made and how the separate application provided by a second entity different from the first entity control the operations. The additional elements do not integrate the recited judicial exception into a practical application and these additional elements do not amount to significantly more than the judicial exception. As such, the claims are ineligible.
(2) The 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph rejections to Claim 15 is respectfully withdrawn in response to Applicant's amendment to the claim.
(3) Regarding the 35 U.S.C. 102 rejection, Applicant alleges that the Office Action cites to para. [0029] and FIG. 4. of Poulos. Office Action, p. 20. For example, the Office Action appears to equate "first object 410 may be a virtual television, such as virtual television device..., overlaid onto a background scene whereas second object 414 may be a media device ... or a telephony device" from Poulos with "the virtual object is associated with one or more operations that are capable of being controlled via the automated assistant application and or via a separate application" of original claim 1. However, the cited aspects of Poulos fail to disclose that the "automated assistant application" is "provided by a first entity" and the "separate application [is] provided by a second entity different from the first entity" as set forth in amended claim 1. Furthermore, the Office Action appears to equate Poulos'"[a]t 708, method 700 includes determining a first group of commands that are operable based on one or more of an identification of the selected object and a state of the object" with original claim l's"determining, based on the virtual object, that the virtual object is associated with one or more operations that are capable of being controlled via the automated assistant application and or via a separate application." However, this cited aspect of Poulos fails to disclose "determining, by the automated assistant application accessing the virtual environment application" as set forth in amended claim 1.
As to point (3), Applicant’s arguments filed 01/20/2026 have been fully considered but are moot due to new grounds of rejection necessitated by amendment.
Although a new ground of rejection has been used to address additional limitations that have been added to claims, a response is considered necessary for several of the Applicant’s arguments since the references on record, Poulos, will continue to be used to meet several of the claimed limitations. Fig. 1 of Poulos depicts a user 106 is viewing the living room through an augmented reality computing device in the form of the see-through head-mounted display device 104, and may interact with the augmented environment via a user interface of display device 104. FIG. 1 also depicts a user field of view 102 comprising a portion of the environment viewable through the display device 104, and thus the portion of the environment that may be augmented with images displayed via the display device 104. The augmented environment may include a plurality of real and virtual objects, e.g., media devices, with which the user may interact ([0014]). Thus, Poulos’s augmented reality computing device is considered to teach “an automated assistant application provided by a first entity” as recited in claim 1. Poulos further teaches the augmented reality computing device is able to determine a group of available commands to control the objects like a television device, i.e. a second entity described in Fig. 5; [0037]. Thus, Poulos is considered to teach the "separate application [is] provided by a second entity different from the first entity". Poulos further teaches a user interface on an augmented reality computing device comprising a see-through display system and a sensor system including one or more sensors, e.g. including an eye tracking sensor. The user interface receives a user input selecting an object, determines a first group of commands that are operable based on one or more of an identification of the selected object and a state of the object ([0055]-[0058]) and Fig. 4 depicts example embodiments of interactions at 402, 404, 406, and 408 of a user 106 wearing a see-through display device 104 with objects in a field of view 102 of the see-through display device 104 ([0029]). Thus, Poulos is considered to teach "determining, by the automated assistant application accessing the virtual environment application" as recited in claim 1.
(4) Regarding independent claim 15, similar arguments have been presented for claim 15 and thus, Applicant’s arguments are not persuasive for the same reasons.
(5) Regarding independent claim 15, Applicant further argues that neither Poulos nor Wenzel teaches or suggests determining that a user has "previously selected the one or more parameters... during a prior interaction" and dynamically generating "an additional virtual feature that, upon selection, causes the assistant to initialize performance of the operation in accordance with the one or more parameters previously selected by the user" as set forth in claim 15. In rejecting this limitation of claim 15, the Office Action on pg. 36 relies on para. [0073] of Wenzel. More particularly, the Office Action relies on Wenzel's disclosure of "if a user collaborates regularly with another co-worker who is also a user of the virtual space, then the option to visit such a user may be provided to the user." However, the cited portion of Wenzel fails to teach the "additional virtual feature [causing] the assistant to initialize performance of the operation".
As to point (5), as presented in the previous office action, Examiner cited [0076]-[0077] of Wenzel which teaches the limitation of “wherein a selection of the additional feature causes the automated assistant application to initialize performance of the operation in accordance with the one or more parameters previously selected by the user” as recited in claim 15.
Claims 1-9 and 15-20 remain rejected as set forth above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Applicant is required under 37 C.F.R. § 1.111(c) to consider these references fully when responding to this action.
Ni et al. (US 20190361575 A1) providing composite graphical assistant interfaces through which to control multiple different devices.
It is noted that any citation to specific pages, columns, lines, or figures in the prior art references and any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331, 1332-33, 216 U.S.P.Q. 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 U.S.P.Q. 275, 277 (C.C.P.A. 1968)).
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AMY P HOANG whose telephone number is (469)295-9134. The examiner can normally be reached M-TH 8:30-5:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, JENNIFER WELCH can be reached at 571-272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AMY P HOANG/ Examiner, Art Unit 2143
/JENNIFER N WELCH/ Supervisory Patent Examiner, Art Unit 2143