DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments with respect to claim(s) 21-40 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 21-22, 25-26, 28-29, 33-34, 36-38, and 40 is/are rejected under 35 U.S.C. 103 as being unpatentable over Assa et al. (US 2023/0067305 A1), hereinafter referred to as Assa, in view of Solomon et al. (US 2018/0233141 A1), hereinafter referred to as Solomon.
Regarding claim 21, Assa teaches:
A method comprising:
instantiating a virtual interaction environment including one or more objects (Fig. 7A, para [0129], [0132], where a GUI of an AR system is displayed, including objects in the schema);
causing presentation of the virtual interaction environment on a display (Fig. 7A, para [0129], [0132], where a GUI of an AR system is displayed);
during the presentation, receiving, from a user, a voice input corresponding to one or more desired changes to the one or more objects (Fig. 7A, para [0129], [0132], where a speech input from a user is processed for changing an object);
processing, using one or more language models, first data corresponding to the voice input and second data corresponding to the virtual interaction environment, the second data providing context for the voice input (Fig. 7A, para [0108], [0125], where the conversation-based AR system uses a trained speech understanding model to process the raw speech input and the schema, which provides the context for the input);
based at least on the processing, determining that an intent of the voice input is for one or more changes to the one or more objects within the virtual interaction environment (Fig. 7A, para [0129], [0132], where an intent is determined for changing at least one object);
updating the virtual interaction environment to reflect the one or more changes to the one or more objects (Fig. 7A, para [0129], [0132], where the change is reflected in the updated GUI shown to the user); and
causing presentation of the updated virtual interaction environment, including the one or more changes to the one or more objects, on the display (Fig. 7A, para [0129], [0132], where the change is reflected in the updated GUI shown to the user).
Assa does not teach:
based at least on the determined intent for the one or more changes to the one or more objects, iteratively determining whether (1) to generate a follow on question to obtain, from the voice input, additional information as to the one or more changes to the one or more objects, or (2) sufficient information has been obtained with respect to the one or more changes to the one or more objects;
Solomon teaches:
based at least on the determined intent for the one or more changes to the one or more objects, iteratively determining (Claim 1, where actions are performed in a loop to determine information for slots in an intent template) whether (1) to generate a follow on question to obtain, from the voice input, additional information as to the one or more changes to the one or more objects (Claim 1, where a query is presented to the user to fill or resolve a selected slot), or (2) sufficient information has been obtained with respect to the one or more changes to the one or more objects (Claim 1, where the loop is exited upon a determination that all slots required by the intent template are filled and resolved);
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Assa by using the intent template of Solomon (Solomon Claim 1) for the query of Assa (Assa Fig. 7A, para [0129], [0132]), by using the loop of Solomon to fill the slots of Assa, in order to resolve ambiguous information and fill in missing information (Solomon para [0274]).
Regarding claim 22, Assa in view of Solomon teaches:
The method of claim 21, wherein the voice input is received by a conversational artificial intelligence (AI) associated with the virtual interaction environment (Assa para [0017], [0059-60], where the conversation-based AR system uses a speech understanding model, such as a neural network).
Regarding claim 25, Assa in view of Solomon teaches:
The method of claim 21, further comprising:
selecting, from a predetermined list, one or more actions supported by the virtual interaction environment determined to correspond to the one or more changes to the one or more objects (Assa para [0058], [0063], where changes to property values of the configurable entity are considered the actions, the configurable entities including lists of objects and corresponding properties); and
providing the one or more actions to the virtual interaction environment to cause the updating (Assa para [0063], where the updates with the changes are sent to the client device).
Regarding claim 26, Assa in view of Solomon teaches:
The method of claim 21, wherein the intent is determined at least in part by a trained neural network (Assa para [0017], [0059-60], where the conversation-based AR system uses a speech understanding model, such as a neural network).
Regarding claim 28, Assa teaches:
At least one processor, comprising:
processing circuitry (para [0135], where a processor is used) to:
instantiate a virtual interaction environment including one or more objects (Fig. 7A, para [0129], [0132], where a GUI of an AR system is displayed, including objects in the schema);
present the virtual interaction environment on a display (Fig. 7A, para [0129], [0132], where a GUI of an AR system is displayed);
during the presenting, receive, from a user, a voice input corresponding to one or more desired changes to the one or more objects (Fig. 7A, para [0129], [0132], where a speech input from a user is processed for changing an object);
process, using one or more language models, first data corresponding to the voice input and second data corresponding to the virtual interaction environment, the second data providing context for the voice input (Fig. 7A, para [0108], [0125], where the conversation-based AR system uses a trained speech understanding model to process the raw speech input and the schema, which provides the context for the input);
based at least on the processing, determine that an intent of the voice input is for one or more changes to the one or more objects within the virtual interaction environment (Fig. 7A, para [0129], [0132], where an intent is determined for changing at least one object);
update the virtual interaction environment to reflect the one or more changes to the one or more objects (Fig. 7A, para [0129], [0132], where the change is reflected in the updated GUI shown to the user); and
present the updated virtual interaction environment, including the one or more changes to the one or more objects, on the display (Fig. 7A, para [0129], [0132], where the change is reflected in the updated GUI shown to the user).
Assa does not teach:
based at least on the determined intent for the one or more changes, iteratively determining whether (1) to generate a follow on question to obtain, from the voice input, additional information as to the one or more changes to the one or more objects, or (2) sufficient information has been obtained with respect to the one or more changes to the one or more objects;
Solomon teaches:
based at least on the determined intent for the one or more changes, iteratively determining (Claim 1, where actions are performed in a loop to determine information for slots in an intent template) whether (1) to generate a follow on question to obtain, from the voice input, additional information as to the one or more changes to the one or more objects (Claim 1, where a query is presented to the user to fill or resolve a selected slot), or (2) sufficient information has been obtained with respect to the one or more changes to the one or more objects (Claim 1, where the loop is exited upon a determination that all slots required by the intent template are filled and resolved);
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Assa by using the intent template of Solomon (Solomon Claim 1) for the query of Assa (Assa Fig. 7A, para [0129], [0132]), by using the loop of Solomon to fill the slots of Assa, in order to resolve ambiguous information and fill in missing information (Solomon para [0274]).
Regarding claim 29, Assa in view of Solomon teaches:
The at least one processor of claim 28, wherein the processing circuitry is further to execute a trained entailment neural network, wherein the processing circuitry is further to determine the intent of the voice input using the trained entailment neural network (Assa para [0038-39], where a speech understanding model uses a neural network to generate intents from the raw speech input).
Regarding claim 33, Assa in view of Solomon teaches:
The at least one processor of claim 28, wherein the processing circuitry is further to select, from a predetermined list, one or more actions supported by the virtual interaction environment determined to correspond to the one or more changes to the one or more objects, wherein the processing circuitry is further to provide the one or more actions to the virtual interaction environment to cause the updating (Assa para [0058], [0063], where changes to property values of the configurable entity are considered the actions, the configurable entities including lists of objects and corresponding properties, and where the updates with the changes are sent to the client device).
Regarding claim 34, Assa in view of Solomon teaches:
The at least one processor of claim 28, wherein the intent is determined at least in part by a trained neural network (Assa para [0017], [0059-60], where the conversation-based AR system uses a speech understanding model, such as a neural network).
Regarding claim 36, Assa teaches:
A system, comprising one or more processors (para [0135], where a processor is used) to cause performance of operations comprising:
displaying an instance of a virtual interaction environment including one or more content elements (Fig. 7A, para [0129], [0132], where a GUI of an AR system is displayed, including objects in the schema);
receiving, from a user, an utterance associated with one or more specified modifications to be made to the content elements as displayed in the instance (Fig. 7A, para [0129], [0132], where a speech input from a user is processed for changing an object);
generating a representation of an intent of the utterance by using natural language understanding to process at least part of the utterance and information associated with the virtual interaction environment (Fig. 7A, para [0108], [0125], where the conversation-based AR system uses a trained speech understanding model to process the raw speech input and the schema, which provides the context for the input, and para [0129], [0132], where an intent is determined for changing at least one object);
determining the generated representation of the intent is related to one or more modifications, supported by the virtual interaction environment, to the content elements (Fig. 7A, para [0129], [0132], where an intent is determined for changing at least one object);
causing the one or more modifications to the content elements to be made within the virtual interaction environment to update the instance (Fig. 7A, para [0129], [0132], where the change is reflected in the updated GUI shown to the user); and
displaying the updated instance of the virtual interaction environment including the one or more modifications to the content elements (Fig. 7A, para [0129], [0132], where the change is reflected in the updated GUI shown to the user).
Assa does not teach:
based at least on the determined representation of the intent related to the one or more modifications, determining whether (1) to generate an operation to obtain, from the utterance, additional information as to the one or more modifications, or (2) sufficient information has been obtained with respect to the one or more modifications;
Solomon teaches:
based at least on the determined representation of the intent related to the one or more modifications, determining (Claim 1, where actions are performed in a loop to determine information for slots in an intent template) whether (1) to generate an operation to obtain, from the utterance, additional information as to the one or more modifications (Claim 1, where a query is presented to the user to fill or resolve a selected slot), or (2) sufficient information has been obtained with respect to the one or more modifications (Claim 1, where the loop is exited upon a determination that all slots required by the intent template are filled and resolved);
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Assa by using the intent template of Solomon (Solomon Claim 1) for the query of Assa (Assa Fig. 7A, para [0129], [0132]), by using the loop of Solomon to fill the slots of Assa, in order to resolve ambiguous information and fill in missing information (Solomon para [0274]).
Regarding claim 37, Assa in view of Solomon teaches:
The system of claim 36, wherein the determining is based, at least in part, on a pre-determined list of entities associated with capabilities of the virtual interaction environment (Assa para [0108], where lists of properties or slot values that the slots can receive are used).
Regarding claim 38, Assa in view of Solomon teaches:
The system of claim 36, further comprising:
extracting, from the utterance, one or more portions associated with the intent to provide additional information to answer the intent (Assa Fig. 7A, para [0125], where both intents and slots are determined from the input); and
identifying, using the one or more extracted portions, the one or more modifications to the content elements (Assa Fig. 7A, para [0125], where the slot values are the additional information corresponding to the intent).
Regarding claim 40, Assa in view of Solomon teaches:
The system of claim 36, further comprising:
selecting the intent from a list of intents, individual intents of the list of intents corresponding to a respective intent label (Assa para [0108], where lists of intents are used and selected from, each intent corresponding to a label such as put_mascara).
Claim(s) 23-24, 27, and 31-32 is/are rejected under 35 U.S.C. 103 as being unpatentable over Assa, in view of Solomon, and further in view of Duong et al. (US 2023/0376696 A1), hereinafter referred to as Duong.
Regarding claim 23, Assa in view of Solomon teaches:
The method of claim 21, wherein the one or more language models generate a formulation of the intent for the processing (Assa Fig. 7A, para [0125], where the intent is determined).
Assa in view of Solomon does not teach the intent being in natural language
Duong teaches:
wherein the one or more language models generate a natural language formulation of the intent for the processing (para [0037-38], where the natural language formulation of the intent corresponds to the user utterance of "order pizza").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Assa in view of Solomon by using the natural language intent of Duong (Duong para [0037-38]) in the intent determination of Assa in view of Solomon (Assa para [0125]), in order to engage with a user in conversation, such as for ordering a pizza (Duong para [0037]).
Regarding claim 24, Assa in view of Solomon and Duong teaches:
The method of claim 23, further comprising:
determining the natural language formulation requires additional information from the voice input (Assa Fig. 7A, para [0125], where both intents and slots are determined from the input); and
identifying, from the voice input, the additional information required by natural language formulation (Assa Fig. 7A, para [0125], where the slot values are the additional information corresponding to the intent).
Regarding claim 27, Assa in view of Solomon teaches:
The method of claim 21, further comprising:
determining a plurality of pre-determined intent labels based at least in part on the voice input (Fig. 6B element 632, para [0058], where a list of intents is stored);
Assa in view of Solomon does not teach:
determining a probability, for individual labels of the plurality of pre-determined intent labels, of corresponding to the voice input; and
selecting one or more of the individual labels, having the probability exceeding a threshold, to be used as the intent.
Duong teaches:
determining a probability, for individual labels of the plurality of pre-determined intent labels, of corresponding to the voice input (para [0101], where a confidence score is used for identifying an intent); and
selecting one or more of the individual labels, having the probability exceeding a threshold, to be used as the intent (para [0101], where a threshold confidence score must be met for selection).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Assa in view of Solomon by using the confidence scores of Duong (Duong para [0101]) in the intent determination of Assa in view of Solomon (Assa para [0058]), in order to determine a particular skill bot to be invoked to handle the utterance (Duong para [0101]).
Regarding claim 31, Assa in view of Solomon teaches:
The at least one processor of claim 28, wherein the one or more language models generate a formulation of the intent for the processing (Assa Fig. 7A, para [0125], where the intent is determined).
Assa in view of Solomon does not teach the intent being in natural language.
Duong teaches:
wherein the one or more language models generate a natural language formulation of the intent for the processing para [0037-38], where the natural language formulation of the intent corresponds to the user utterance of "order pizza").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Assa in view of Solomon by using the natural language intent of Duong (Duong para [0037-38]) in the intent determination of Assa in view of Solomon (Assa para [0125]), in order to engage with a user in conversation, such as for ordering a pizza (Duong para [0037]).
Regarding claim 32, Assa in view of Solomon and Duong teaches:
The at least one processor of claim 31, further comprising:
determine the natural language formulation requires additional information from the voice input (Assa Fig. 7A, para [0125], where both intents and slots are determined from the input); and
identify, from the voice input, the additional information required by natural language formulation (Assa Fig. 7A, para [0125], where the slot values are the additional information corresponding to the intent).
Claim(s) 30 and 35 is/are rejected under 35 U.S.C. 103 as being unpatentable over Assa, in view of Solomon, and further in view of Avijeet (US 2022/0130378 A1).
Regarding claim 30, Assa in view of Solomon teaches:
The at least one processor of claim 28,
Assa in view of Solomon does not teach:
wherein the processing circuitry is further to execute a trained extractive question and answer neural network model, wherein the processing circuitry is further to determine, at least in part, the one or more changes to the one or more objects using the trained extractive question and answer neural network model.
Avijeet teaches:
wherein the processing circuitry is further to execute a trained extractive question and answer neural network model, wherein the processing circuitry is further to determine, at least in part, the one or more changes to the one or more objects using the trained extractive question and answer neural network model (para [0072], where a neural network performs question answering by fetching data from external sources).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Assa in view of Solomon by using the question answering of Avijeet (Avijeet para [0072]) in the neural networks of Assa in view of Solomon (Assa para [0038-39]), in order to determine a proper response to user queries (Avijeet para [0072]).
Regarding claim 35, Assa in view of Solomon teaches:
The at least one processor of claim 28, wherein the processing circuitry is further to:
Assa in view of Solomon does not teach:
receive a second voice input from the user;
determine a second intent of the second voice input cannot be identified; and
provide a response to the user that includes a request for additional intent information.
Avijeet teaches:
receive a second voice input from the user (para [0072], where a user queries about the weather);
determine a second intent of the second voice input cannot be identified (para [0072], where the system determines a query to resolve ambiguities); and
provide a response to the user that includes a request for additional intent information (para [0072], where the system outputs the query to the user).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Assa in view of Solomon by using the question answering of Avijeet (Avijeet para [0072]) in the neural networks of Assa in view of Solomon (Assa para [0038-39]), in order to determine a proper response to user queries (Avijeet para [0072]).
Claim(s) 39 is/are rejected under 35 U.S.C. 103 as being unpatentable over Assa, in view of Solomon, and further in view of Zhao (US 2020/0251091 A1).
Regarding claim 39, Assa in view of Solomon teaches:
The system of claim 36, further comprising:
Assa in view of Solomon does not teach:
determining the intent based, at least in part, on one or more machine learning systems using a zero-shot approach.
Zhao teaches:
determining the intent based, at least in part, on one or more machine learning systems using a zero-shot approach (para [0037-38], [0070], where the Zero-shot intent recognition model is trained using machine learning tools to recognize intents).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Assa in view of Solomon by using the dynamic intent list of Zhao (Zhao para [0038], [0076]) in the intent determination of Assa in view of Solomon (Assa para [0125]), to achieve rich semantic parsing results including matching results with all the intents, without the need for retraining (Zhao para [0038]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. US 2021/0375404 A1 para [0137-138] teaches filling semantic slots using inquiry questions until all the slots are filled.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BRYAN S BLANKENAGEL whose telephone number is (571)270-0685. The examiner can normally be reached 8:00am-5:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Richemond Dorvil can be reached at 571-272-7602. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BRYAN S BLANKENAGEL/Primary Examiner, Art Unit 2658
Prosecution Insights