DETAILED ACTION
Notice of Pre-AIA or AIA Status
In response to Applicant’s claims filed on February 10, 2026, claims 1-22 are now pending for examination in the application.
Response to Arguments
This office action is in response to amendment filed 02/10/2026. In this action 1, Claim(s) 1-22 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ekron (US Pub. No. 20220366131) and Bansal (US Pub. No. 20210291046) in further view of Person et al. (US Pub. No. 20210337284). The Ekron reference has been added to address the amendment of receive, from a user application providing a user interface via the input and output devices, object metadata descriptive of the content of the user interface, the object metadata defining aspects of user interface elements displayed to the user interface including available actions that can be performed to the user interface elements.
Applicant’s arguments:
In regards to claim 1 on Page(s) 9, applicant argues “Thus, the recited subject matter improves the way a computer system enables users, such as users who cannot effectively use conventional visual or manual controls, to operate software through an alternate pathway. This is a concrete technological technique for user interface control and accessibility and is therefore not a mental process that can be performed in the human mind. Moreover, regarding Step 2A, the claimed approach embodies this as a practical application to control software operation and improve how users interact with software. Also, regarding Step 2B, the ordered combination as claimed is not a well-understood, routine or conventional approach, as discussed in detail below.”
Examiner’s Reply:
The examiner respectfully disagrees and would like to point out that human mind using a computer as a tool is fully capable of augmenting an interface for the visually-impaired. A human would be able to follow these steps along with any needed additional elements (eg using receiving streaming data). The examiner notes that the computer (being used as a generic tool) as recited in the claims is being used for presenting audio. The use of interface elements does not improve the functioning of a computer. Therefore, the abstract idea recited in the claims is generally linking it to a computer environment, and does not integrate the abstract idea into a practical application.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-22 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-patentable subject matter. The claims are directed to an abstract idea without significantly more.
Claim 1-22 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The judicial exception is not integrated into a practical application. The claims do not include additional elements that are sufficient to amount to significantly more than judicial exception. The eligibility analysis in support of these findings is provided below, on Claim Rejections - 35 USC 101 accordance with the "2019 Revised Patent Subject Matter Eligibility Guidance" (published on 1/7/2019 in Fed, Register, Vol. 84, No. 4 at pgs. 50-57, hereinafter referred to as the "2019 PEG").
Step 1. in accordance with Step 1 of the eligibility inquiry (as explained in MPEP 2106), it is first noted the claim system (claims 1-12), method(s) (claims 13-21), and medium 22 is/are directed to one of the eligible categories of subject matter and therefore satisfies Step 1.
Step 2A. In accordance with Step 2A, prong one of the 2019 PEG, it is noted that the independent claims recite an abstract idea falling within the Mental Processes enumerated groupings of abstract ideas set forth in the 2019 PEG. Examiner is of the position that independent claims 1 and 13 are directed towards the Mental Process Grouping of Abstract Ideas.
Independent claims 1 and 13 recites the following limitations directed towards a Mental Processes:
generate an augmented description of the user interface, the augmented description including a description of surroundings in the user interface and a listing of actions to be performed to the user interface, based on the available actions (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by generating a description),
instructing the user application to perform the identified available action (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by instructing the user application), and
update the augmented description based on the user input (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by updating a description).
Step 2A. In accordance with Step 2A, prong two of the 2019 PEG, the judicial exception is not integrated into a practical application because of the recitation in claim(s) 1 and 11:
receive, from a user application providing a user interface via the input and output devices, object metadata descriptive of the content of the user interface, the object metadata defining aspects of user interface elements displayed to the user interface including available actions that can be performed to the user interface elements (recites insignificant extra solution activity of mere data gathering),
present the augmented description using the output devices (recites insignificant extra solution activity of presenting data),
process user input requesting one of the actions by identifying, from the user input, one of the available actions defined by the object metadata (recites insignificant extra solution activity of processing data).
Step 2B. Similar to the analysis under 2A Prong Two, the claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception. Because the additional elements of the independent claims amount to insignificant extra solution activity and/or mere instructions, the additional elements do not add significantly more to the judicial exception such that the independent claims as a whole would be patent eligible.
Independent claims 22 recites the following limitations directed towards a Mental Processes:
filter the object metadata using properties of the object metadata (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by filtering metadata),
to determine relevant objects in the object metadata (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by determining an object),
generate an augmented description of the user interface using the relevant objects, the augmented description including a description of surroundings in the user interface and a listing of actions to be performed to the user interface based on the available actions as filtered (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by generating a description),
instructing the user application to perform the identified available action (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by instructing the user application);
update the augmented description based on the user input (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by updating a description).
Step 2A. In accordance with Step 2A, prong two of the 2019 PEG, the judicial exception is not integrated into a practical application because of the recitation in claim(s) 22:
receive, from a user application providing a user interface via input and output devices of the computing device, object metadata descriptive of the content of the user interface, including to utilize an API of the narrative engine to receive the object metadata from the user application, the API including extensions for each type of the user interface to be supported to allow access to the object metadata of that specific user interface type, the object metadata defining aspects of user interface elements displayed to the user interface including available actions that can be performed to the user interface, the reception (recites insignificant extra solution activity of mere data gathering),
present the augmented description using the output devices, as one or more of an overlay superimposed on the user interface or audibly as computer-generated speech (recites insignificant extra solution activity of presenting data),
process user input requesting one of the actions (recites insignificant extra solution activity of processing data),
present the updated augmented description using the output devices (recites insignificant extra solution activity of presenting data).
Step 2B. Similar to the analysis under 2A Prong Two, the claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception. Because the additional elements of the independent claims amount to insignificant extra solution activity and/or mere instructions, the additional elements do not add significantly more to the judicial exception such that the independent claims as a whole would be patent eligible.
Therefore, independent claims 1, 13, 22 are rejected under 35 U.S.C. 101.
With respect to claim(s) 2 and 14:
Step 2A, prong one of the 2019 PEG:
Examiner is of the position the dependent claim is directed toward additional elements.
Step 2A Prong Two Analysis:
utilize an application programming interface (API) to receive the object metadata from the user application, the API including extensions for each type of the user interface to be supported (recites insignificant extra solution activity of mere data gathering), to allow the narrative engine to access the object metadata of that specific user interface type (recites insignificant extra solution activity of accessing data).
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 3:
Step 2A, prong one of the 2019 PEG:
Examiner is of the position the dependent claim is directed toward additional elements.
Step 2A Prong Two Analysis:
wherein the user interface is a 3D user interface is rendered by a 3D engine, elements of the 3D user interface are rendered according to the object metadata (recites insignificant extra solution activity of rendering data), and the object metadata is accessed by the narrative engine via the API (recites insignificant extra solution activity of accesing data).
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 4:
Step 2A, prong one of the 2019 PEG:
Examiner is of the position the dependent claim is directed toward additional elements.
Step 2A Prong Two Analysis:
wherein the user interface is a web user interface rendered by a web browser, elements of the web user interface are rendered according to the object metadata included in hypertext transfer protocol (HTTP) markup (recites insignificant extra solution activity of rendering data), and
the object metadata is accessed from the HTTP markup by the narrative engine via the API (recites insignificant extra solution activity of accessing data).
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 5:
Step 2A, prong one of the 2019 PEG:
Examiner is of the position the dependent claim is directed toward additional elements.
Step 2A Prong Two Analysis:
wherein the user interface is a console application user interface, and the object metadata is accessed from a console text buffer of the console application by the narrative engine via the API (recites insignificant extra solution activity of accessing data).
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 6 and 15:
Step 2A, prong one of the 2019 PEG:
Examiner is of the position the dependent claim is directed toward additional elements.
Step 2A Prong Two Analysis:
wherein the augmented description is presented as an overlay superimposed on the user interface (recites insignificant extra solution activity of presenting data).
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 7 and 16:
Step 2A, prong one of the 2019 PEG:
Examiner is of the position the dependent claim is directed toward additional elements.
Step 2A Prong Two Analysis:
wherein the augmented description is presented audibly as computer-generated speech (recites insignificant extra solution activity of presenting data).
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 8 and 17:
Step 2A, prong one of the 2019 PEG:
wherein the narrative engine includes user settings that define how to present the augmented description based on a level or type of disability of a user of the narrative engine (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by defining a presentation).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no
additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 9 and 18:
Step 2A, prong one of the 2019 PEG:
filter, by an attention filter, the object metadata using properties of the object metadata to determine relevant objects in the object metadata (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by filtering metadata), including one or more of to:
limit the object metadata to elements of the user interface within a predefined distance from an avatar of a user (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by limiting metadata),
limit the object metadata to the elements of the user interface within a field of view of the user (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by limiting metadata), or
limit the object metadata to the elements of the user interface that are within a predefined 2D distance from a mouse cursor, limit the object metadata to the elements of the user interface that are enabled (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by limiting metadata).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no
additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 10 and 19:
Step 2A, prong one of the 2019 PEG:
construct an interface model descriptive of the properties and available actions of the relevant objects (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by constructing a model);
use a description creator to generate the description of the surroundings based on the properties of the relevant objects (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by generating the description),
and to generate the listing of actions based on the available actions of the relevant objects (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by generating a listing).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no
additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 11 and 20:
Step 2A, prong one of the 2019 PEG:
wherein the description creator is configured to generate the augmented description using templates that include natural language text and placeholders for values of the properties or the available actions of the relevant objects to be described (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by generating a description).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no
additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 12 and 21:
Step 2A, prong one of the 2019 PEG:
utilize a speech-to-text engine to convert the user input into recognized text (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by converting speech to text);
scan the recognized text for names of the available actions in the interface model (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by scanning names); and
instruct the user application to perform the named available action that was spoken (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by performing an action).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no
additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-22 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ekron (US Pub. No. 20220366131) and Bansal (US Pub. No. 20210291046) in further view of Person et al. (US Pub. No. 20210337284).
With respect to claim 1, Ekron discloses a system, comprising:
a computing device including input and output devices, the computing device being programmed to execute a narrative engine (Paragraph 80 discloses a user using an input device, such as a keyboard, a mouse, a touch pad, a touch screen, a joystick, a microphone, an image sensor, and/or any other device connectable to the computing device) to
receive, from a user application providing a user interface via the input and output devices, object metadata descriptive of the content of the user interface, the object metadata defining aspects of user interface elements displayed to the user interface including available actions that can be performed to the user interface elements (Paragraph 517 discloses identifying in the website code a plurality of elements corresponding to multiple objects displayed on the website. For example, a parser may scan the website code (e.g., including metadata, and/or comments embedded therein) of the website, or may scan website code of pages and/or website navigable from the website for words, terms or phrases that may identify elements corresponding to objects and Paragraph 87 discloses screen reader text may be automatically added to help the screen reader describe certain actions and areas based on the context of a website),
generate an augmented description of the user interface, the augmented description including a description of surroundings in the user interface and a listing of actions to be performed to the user interface based on the available actions (Paragraph 82 discloses to address the particular visual disabilities of website users, the accessibility GUI may enable website users to change the presentation of web content into forms that are more usable for their particular needs. Changes to presentation may include, for example, enlarging or reducing text size and images, customizing settings for fonts, colors, and spacing, text-to-speech synthesis of the content, providing audio descriptions of video in multimedia, reading text using refreshable Braille, and/or any other visual, audible, or tactile alteration),
present the augmented description using the output devices (Paragraph 82 discloses to address the particular visual disabilities of website users, the accessibility GUI may enable website users to change the presentation of web content into forms that are more usable for their particular needs. Changes to presentation may include, for example, enlarging or reducing text size and images, customizing settings for fonts, colors, and spacing, text-to-speech synthesis of the content, providing audio descriptions of video in multimedia, reading text using refreshable Braille, and/or any other visual, audible, or tactile alteration). Ekron does not disclose process user input requesting one of the actions by identifying, from the user input, one of the available actions defined by the object metadata
process user input requesting one of the actions by identifying, from the user input, one of the available actions defined by the object metadata (Paragraph 32 discloses facilitating playing of the video game by users with visual impairments. When these feature(s) and/or mode(s) are enabled, the online connected game may be configured to request the provision of speech audio based on text currently being displayed to or selected by a user 110 by a respective instance of the online connected game 122. For example, speech audio based on text included in a menu item currently being selected by a user 110 may be requested, or speech audio based on text at or proximate to a position of a pointer controlled by a user 110 may be requested and Paragraph 32 discloses request may identify the type of video game content to be provided, e.g. where the content provision server 130 is usable to provide multiple types of video game content. The type of video game content identified could be, but is not limited to, the types of video game content described above, e.g. speech audio, music, non-player character behaviour, character animations, video game terrain, locations for entities in a video game environment), instructing the user application to perform the identified available action (Paragraphs 24-25 discloses request the provision of speech audio based on text currently being displayed to or selected by a user 110 by a respective instance of the online connected game 122. For example, speech audio based on text included in a menu item currently being selected by a user 110 may be requested, or speech audio based on text at or proximate to a position of a pointer controlled by a user 110 may be requested. The online connected game 122 may additionally or alternatively be configured to request video game content from the content provision server 130 to be utilized in the display of an in-game menu. For example, the online connected game 122 may be configured to request recommendations for player character choices and/or game modes from the content provision server 130 and highlight the recommended player character choices and/or game modes in the in-game menu. The recommendation request may include properties using which a recommendation may be made, e.g., the amount of time that the user has played various game modes for; the previous player character choices that the user has made; and/or the playstyle of the user, such as the actions taken by the user previously).
Therefore, it would have been obvious at the time the invention was made to a person having ordinary skill in the art to modify Ekron with Bansal et al. to include process user input requesting one of the actions. This would have facilitated interactions with people of different abilities. See Person et al. Paragraphs 2-4.
Ekron as modified by Bansal et al. does not explicitly disclose update the augmented description based on the user input.
However, Person et al. teaches update the augmented description based on the user input (Paragraph 58 discloses may output speech or display text in various languages, and/or the like and Paragraph 7 discloses a machine learning model to identify one or more characteristics of the interaction information, determine, using the machine learning model, one or more modifications to the immersive environment based on the identified one or more characteristics, and outputting the determined one or more modifications to the multimedia output system).
Therefore, it would have been obvious at the time the invention was made to a person having ordinary skill in the art to modify Ekron and Bansal et al. with Person et al. to include update the augmented description based on the user input. This would have facilitated interactions with people of different abilities. See Person et al. Paragraphs 4-23.
The Ekron reference as modified by Bansal et al. and Person et al. teaches all the limitations of claim 1. Regarding claim 4, Ekron discloses the system of claim 1, wherein the computing device is further programmed to:
utilize an application programming interface (API) to receive the object metadata from the user application, the API including extensions for each type of the user interface to be supported, to allow the narrative engine to access the object metadata of that specific user interface type (Paragraph 4 discloses an application programming interface).
The Ekron reference as modified by Bansal et al. and Person et al. teaches all the limitations of claim 2. Regarding claim 3, Bansal et al. discloses the system of claim 2, wherein the user interface is a 3D user interface is rendered by a 3D engine, elements of the 3D user interface are rendered according to the object metadata, and the object metadata is accessed by the narrative engine via the API (Paragraph 93 discloses generating test outputs using the respective derived machine learning model. These generated test outputs may be video game content or outputs from which video game content may be derived, e.g. phonemes and/or spectrogram frames for speech audio, a terrain heightfield for use in generating a 3D mesh for an in-game terrain, or latent embeddings of the video game content). The motivation to combine statement previously provided in the rejection of independent claim 2 provided above, combining the Ekron reference and the Bansal et al. reference is applicable to dependent claim 3.
The Ekron reference as modified by Bansal et al. and Person et al. teaches all the limitations of claim 2. Regarding claim 4, Ekron discloses the system of claim 2, wherein the user interface is a web user interface rendered by a web browser, elements of the web user interface are rendered according to the object metadata included in hypertext transfer protocol (HTTP) markup, and the object metadata is accessed from the HTTP markup by the narrative engine via the API (Paragraph 6 discloses displaying an accessibility graphical user interface (GUI) for a website, wherein the accessibility GUI includes a plurality of web accessibility profiles associated with differing disabilitie).
The Ekron reference as modified by Bansal et al. and Person et al. teaches all the limitations of claim 2. Regarding claim 5, Bansal et al. discloses the system of claim 2, wherein the user interface is a console application user interface, and the object metadata is accessed from a console text buffer of the console application by the narrative engine via the API (Paragraph 3 discloses a distributed computing system for providing video game content to an online connected game). The motivation to combine statement previously provided in the rejection of independent claim 2 provided above, combining the Ekron reference and the Bansal et al. reference is applicable to dependent claim 5.
The Ekron reference as modified by Bansal et al. and Person et al. teaches all the limitations of claim 1. Regarding claim 6, Ekron discloses the system of claim 1, wherein the augmented description is presented as an overlay superimposed on the user interface (Paragraph 143 discloses select from a graphical user interface and/or display a graphical element corresponding to the second web accessibility profile).
The Ekron reference as modified by Bansal et al. and Person et al. teaches all the limitations of claim 1. Regarding claim 7, Bansal et al. discloses the system of claim 1, wherein the augmented description is presented audibly as computer-generated speech (Paragraph 24 discloses When these feature(s) and/or mode(s) are enabled, the online connected game may be configured to request the provision of speech audio based on text currently being displayed to or selected by a user 110 by a respective instance of the online connected game 122). The motivation to combine statement previously provided in the rejection of independent claim 2 provided above, combining the Ekron reference and the Bansal et al. reference is applicable to dependent claim 7.
The Ekron reference as modified by Bansal et al. and Person et al. teaches all the limitations of claim 1. Regarding claim 8, Ekron discloses the system of claim 1, wherein the narrative engine includes user settings that define how to present the augmented description based on a level or type of disability of a user of the narrative engine (Paragraph 24 discloses the online connected game 122 may include one or more accessibility feature(s) and/or mode(s) for facilitating playing of the video game by users with visual impairments. When these feature(s) and/or mode(s) are enabled, the online connected game may be configured to request the provision of speech audio based on text currently being displayed to or selected by a user 110 by a respective instance of the online connected game 122).
The Ekron reference as modified by Bansal et al. and Person et al. teaches all the limitations of claim 1. Regarding claim 9, Person et al. discloses the system of claim 1, wherein the narrative engine is further programmed to:
filter, by an attention filter, the object metadata using properties of the object metadata to determine relevant objects in the object metadata, including one or more of to:
limit the object metadata to elements of the user interface within a predefined distance from an avatar of a user,
limit the object metadata to the elements of the user interface within a field of view of the user, or
limit the object metadata to the elements of the user interface that are within a predefined 2D distance from a mouse cursor, limit the object metadata to the elements of the user interface that are enabled (Paragraph 4 discloses tracking system may track the one or more users interaction with the one or more user input devices within the interactive media structure. The multimedia output system may modify output of the one or more virtual objects based on the interaction and Paragraph 8 discloses the immersive environment may further comprise avatars of the one or more users. In a variant, the tracking system may comprise one or more light distance and ranging devices or one or more motion capture devices). The motivation to combine statement previously provided in the rejection of independent claim 1 provided above, combining the Ekron reference and the Person et al. reference is applicable to dependent claim 9.
The Ekron reference as modified by Bansal et al. and Person et al. teaches all the limitations of claim 9. Regarding claim 10, Person et al. discloses the system of claim 9, wherein the narrative engine is further programmed to:
construct an interface model descriptive of the properties and available actions of the relevant objects (Paragraph 7 discloses Paragraph 7 discloses a machine learning model to identify one or more characteristics of the interaction information, determine, using the machine learning model, one or more modifications to the immersive environment based on the identified one or more characteristics, and outputting the determined one or more modifications to the multimedia output system); and
use a description creator to generate the description of the surroundings based on the properties of the relevant objects, and to generate the listing of actions based on the available actions of the relevant objects (Paragraph 7 discloses Paragraph 7 discloses a machine learning model to identify one or more characteristics of the interaction information, determine, using the machine learning model, one or more modifications to the immersive environment based on the identified one or more characteristics, and outputting the determined one or more modifications to the multimedia output system). The motivation to combine statement previously provided in the rejection of independent claim 9 provided above, combining the Ekron reference and the Person et al. reference is applicable to dependent claim 10.
The Ekron reference as modified by Bansal et al. and Person et al. teaches all the limitations of claim 10. Regarding claim 11, Person et al. discloses the system of claim 10, wherein the description creator is configured to generate the augmented description using templates that include natural language text and placeholders for values of the properties or the available actions of the relevant objects to be described (Paragraph 50 discloses the users may communicate using input commands by interacting with user interface elements of the immersive environment (e.g., to send a pre-defined text or sound message or an emoji to another interactive multimedia structure 100), by speaking into a microphone (e.g., to provide voice or speech-to-text communications), using a chat or texting function (e.g., WeChat™, short message service (SMS) messaging, or the like), by capturing and sending an image or video, or the like). The motivation to combine statement previously provided in the rejection of independent claim 10 provided above, combining the Ekron reference and the Person et al. reference is applicable to dependent claim 11.
The Ekron reference as modified by Bansal et al. and Person et al. teaches all the limitations of claim 10. Regarding claim 12, Person et al. discloses the system of claim 10, wherein the narrative engine is further programmed to:
utilize a speech-to-text engine to convert the user input into recognized text (Paragraph 50 discloses the users may communicate using input commands by interacting with user interface elements of the immersive environment (e.g., to send a pre-defined text or sound message or an emoji to another interactive multimedia structure 100), by speaking into a microphone (e.g., to provide voice or speech-to-text communications), using a chat or texting function (e.g., WeChat™, short message service (SMS) messaging, or the like), by capturing and sending an image or video, or the like);
scan the recognized text for names of the available actions in the interface model (Paragraph 50 discloses the users may communicate using input commands by interacting with user interface elements of the immersive environment (e.g., to send a pre-defined text or sound message or an emoji to another interactive multimedia structure 100), by speaking into a microphone (e.g., to provide voice or speech-to-text communications), using a chat or texting function (e.g., WeChat™, short message service (SMS) messaging, or the like), by capturing and sending an image or video, or the like); and
instruct the user application to perform the named available action that was spoken (Paragraph 50 discloses the users may communicate using input commands by interacting with user interface elements of the immersive environment (e.g., to send a pre-defined text or sound message or an emoji to another interactive multimedia structure 100), by speaking into a microphone (e.g., to provide voice or speech-to-text communications), using a chat or texting function (e.g., WeChat™, short message service (SMS) messaging, or the like), by capturing and sending an image or video, or the like). The motivation to combine statement previously provided in the rejection of independent claim 10 provided above, combining the Ekron reference and the Person et al. reference is applicable to dependent claim 12.
With respect to claim 13, Arana et al. discloses a method, comprising:
receive, from a user application providing a user interface via the input and output devices, object metadata descriptive of the content of the user interface, the object metadata defining aspects of user interface elements displayed to the user interface including available actions that can be performed to the user interface elements (Paragraph 517 discloses identifying in the website code a plurality of elements corresponding to multiple objects displayed on the website. For example, a parser may scan the website code (e.g., including metadata, and/or comments embedded therein) of the website, or may scan website code of pages and/or website navigable from the website for words, terms or phrases that may identify elements corresponding to objects and Paragraph 87 discloses screen reader text may be automatically added to help the screen reader describe certain actions and areas based on the context of a website),
generate an augmented description of the user interface, the augmented description including a description of surroundings in the user interface and a listing of actions to be performed to the user interface based on the available actions (Paragraph 82 discloses to address the particular visual disabilities of website users, the accessibility GUI may enable website users to change the presentation of web content into forms that are more usable for their particular needs. Changes to presentation may include, for example, enlarging or reducing text size and images, customizing settings for fonts, colors, and spacing, text-to-speech synthesis of the content, providing audio descriptions of video in multimedia, reading text using refreshable Braille, and/or any other visual, audible, or tactile alteration),
present the augmented description using the output devices (Paragraph 82 discloses to address the particular visual disabilities of website users, the accessibility GUI may enable website users to change the presentation of web content into forms that are more usable for their particular needs. Changes to presentation may include, for example, enlarging or reducing text size and images, customizing settings for fonts, colors, and spacing, text-to-speech synthesis of the content, providing audio descriptions of video in multimedia, reading text using refreshable Braille, and/or any other visual, audible, or tactile alteration). Ekron does not disclose process user input requesting one of the actions by identifying, from the user input, one of the available actions defined by the object metadata
process user input requesting one of the actions by identifying, from the user input, one of the available actions defined by the object metadata (Paragraph 32 discloses facilitating playing of the video game by users with visual impairments. When these feature(s) and/or mode(s) are enabled, the online connected game may be configured to request the provision of speech audio based on text currently being displayed to or selected by a user 110 by a respective instance of the online connected game 122. For example, speech audio based on text included in a menu item currently being selected by a user 110 may be requested, or speech audio based on text at or proximate to a position of a pointer controlled by a user 110 may be requested and Paragraph 32 discloses request may identify the type of video game content to be provided, e.g. where the content provision server 130 is usable to provide multiple types of video game content. The type of video game content identified could be, but is not limited to, the types of video game content described above, e.g. speech audio, music, non-player character behaviour, character animations, video game terrain, locations for entities in a video game environment), instructing the user application to perform the identified available action (Paragraphs 24-25 discloses request the provision of speech audio based on text currently being displayed to or selected by a user 110 by a respective instance of the online connected game 122. For example, speech audio based on text included in a menu item currently being selected by a user 110 may be requested, or speech audio based on text at or proximate to a position of a pointer controlled by a user 110 may be requested. The online connected game 122 may additionally or alternatively be configured to request video game content from the content provision server 130 to be utilized in the display of an in-game menu. For example, the online connected game 122 may be configured to request recommendations for player character choices and/or game modes from the content provision server 130 and highlight the recommended player character choices and/or game modes in the in-game menu. The recommendation request may include properties using which a recommendation may be made, e.g., the amount of time that the user has played various game modes for; the previous player character choices that the user has made; and/or the playstyle of the user, such as the actions taken by the user previously).
Therefore, it would have been obvious at the time the invention was made to a person having ordinary skill in the art to modify Ekron with Bansal et al. to include process user input requesting one of the actions. This would have facilitated interactions with people of different abilities. See Person et al. Paragraphs 2-4.
Ekron as modified by Bansal et al. does not explicitly disclose update the augmented description based on the user input.
However, Person et al. teaches update the augmented description based on the user input (Paragraph 58 discloses may output speech or display text in various languages, and/or the like and Paragraph 7 discloses a machine learning model to identify one or more characteristics of the interaction information, determine, using the machine learning model, one or more modifications to the immersive environment based on the identified one or more characteristics, and outputting the determined one or more modifications to the multimedia output system).
Therefore, it would have been obvious at the time the invention was made to a person having ordinary skill in the art to modify Ekron and Bansal et al. with Person et al. to include update the augmented description based on the user input. This would have facilitated interactions with people of different abilities. See Person et al. Paragraphs 4-23.
With respect to claim 14, it is rejected on grounds corresponding to above rejected claim 2, because claim 14 is substantially equivalent to claim 2.
With respect to claim 15, it is rejected on grounds corresponding to above rejected claim 6, because claim 15 is substantially equivalent to claim 6.
With respect to claim 16, it is rejected on grounds corresponding to above rejected claim 7, because claim 16 is substantially equivalent to claim 7.
With respect to claim 17, it is rejected on grounds corresponding to above rejected claim 8, because claim 17 is substantially equivalent to claim 8.
With respect to claim 18, it is rejected on grounds corresponding to above rejected claim 9, because claim 18 is substantially equivalent to claim 9.
With respect to claim 19, it is rejected on grounds corresponding to above rejected claim 10, because claim 19 is substantially equivalent to claim 10.
With respect to claim 20, it is rejected on grounds corresponding to above rejected claim 11, because claim 20 is substantially equivalent to claim 11.
With respect to claim 21, it is rejected on grounds corresponding to above rejected claim 12, because claim 21 is substantially equivalent to claim 12.
With respect to claim 22, Arana et al. discloses a non-transitory computer-readable medium comprising instructions of a narrative engine that, when executed by one or more processors of a computing device, cause the computing device to perform operations including to:
receive, from a user application providing a user interface via the input and output devices, object metadata descriptive of the content of the user interface, the object metadata defining aspects of user interface elements displayed to the user interface including available actions that can be performed to the user interface elements (Paragraph 517 discloses identifying in the website code a plurality of elements corresponding to multiple objects displayed on the website. For example, a parser may scan the website code (e.g., including metadata, and/or comments embedded therein) of the website, or may scan website code of pages and/or website navigable from the website for words, terms or phrases that may identify elements corresponding to objects and Paragraph 87 discloses screen reader text may be automatically added to help the screen reader describe certain actions and areas based on the context of a website),
generate an augmented description of the user interface, the augmented description including a description of surroundings in the user interface and a listing of actions to be performed to the user interface based on the available actions (Paragraph 82 discloses to address the particular visual disabilities of website users, the accessibility GUI may enable website users to change the presentation of web content into forms that are more usable for their particular needs. Changes to presentation may include, for example, enlarging or reducing text size and images, customizing settings for fonts, colors, and spacing, text-to-speech synthesis of the content, providing audio descriptions of video in multimedia, reading text using refreshable Braille, and/or any other visual, audible, or tactile alteration),
present the augmented description using the output devices (Paragraph 82 discloses to address the particular visual disabilities of website users, the accessibility GUI may enable website users to change the presentation of web content into forms that are more usable for their particular needs. Changes to presentation may include, for example, enlarging or reducing text size and images, customizing settings for fonts, colors, and spacing, text-to-speech synthesis of the content, providing audio descriptions of video in multimedia, reading text using refreshable Braille, and/or any other visual, audible, or tactile alteration). Ekron does not disclose process user input requesting one of the actions by identifying, from the user input, one of the available actions defined by the object metadata
process user input requesting one of the actions by identifying, from the user input, one of the available actions defined by the object metadata (Paragraph 32 discloses facilitating playing of the video game by users with visual impairments. When these feature(s) and/or mode(s) are enabled, the online connected game may be configured to request the provision of speech audio based on text currently being displayed to or selected by a user 110 by a respective instance of the online connected game 122. For example, speech audio based on text included in a menu item currently being selected by a user 110 may be requested, or speech audio based on text at or proximate to a position of a pointer controlled by a user 110 may be requested and Paragraph 32 discloses request may identify the type of video game content to be provided, e.g. where the content provision server 130 is usable to provide multiple types of video game content. The type of video game content identified could be, but is not limited to, the types of video game content described above, e.g. speech audio, music, non-player character behaviour, character animations, video game terrain, locations for entities in a video game environment), instructing the user application to perform the identified available action (Paragraphs 24-25 discloses request the provision of speech audio based on text currently being displayed to or selected by a user 110 by a respective instance of the online connected game 122. For example, speech audio based on text included in a menu item currently being selected by a user 110 may be requested, or speech audio based on text at or proximate to a position of a pointer controlled by a user 110 may be requested. The online connected game 122 may additionally or alternatively be configured to request video game content from the content provision server 130 to be utilized in the display of an in-game menu. For example, the online connected game 122 may be configured to request recommendations for player character choices and/or game modes from the content provision server 130 and highlight the recommended player character choices and/or game modes in the in-game menu. The recommendation request may include properties using which a recommendation may be made, e.g., the amount of time that the user has played various game modes for; the previous player character choices that the user has made; and/or the playstyle of the user, such as the actions taken by the user previously).
Therefore, it would have been obvious at the time the invention was made to a person having ordinary skill in the art to modify Ekron with Bansal et al. to include process user input requesting one of the actions. This would have facilitated interactions with people of different abilities. See Person et al. Paragraphs 2-4.
Ekron as modified by Bansal et al. does not explicitly disclose update the augmented description based on the user input.
However, Person et al. teaches update the augmented description based on the user input (Paragraph 58 discloses may output speech or display text in various languages, and/or the like and Paragraph 7 discloses a machine learning model to identify one or more characteristics of the interaction information, determine, using the machine learning model, one or more modifications to the immersive environment based on the identified one or more characteristics, and outputting the determined one or more modifications to the multimedia output system).
Therefore, it would have been obvious at the time the invention was made to a person having ordinary skill in the art to modify Ekron and Bansal et al. with Person et al. to include update the augmented description based on the user input. This would have facilitated interactions with people of different abilities. See Person et al. Paragraphs 4-23.
Relevant Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US PG-Pub. No. 20220277036 is directed to AUTOMATIC ENABLING OF ACCESSIBILITY FEATURES BASED ON PROFILING AND DETECTION OF DISABILITY: [0073] as the need for inclusivity increases within the media industry, accessibility features are typically present as a feature of media systems. Accessibility features may include, e.g., embedded or integrated described video narrative (DV). DV is a narrated description of a program's non-verbal elements that may include setting, action, costumes, and body language, for example. Description may be added during pauses in dialogue and sound effects and can enable people to form a mental understanding and picture of what is happing in the media asset and can help visually impaired users to have a more fulfilling viewing experience.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NICHOLAS E ALLEN whose telephone number is (571)270-3562. The examiner can normally be reached Monday through Thursday 830-630.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Boris Gorney can be reached at (571) 270-5626. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/N.E.A/Examiner, Art Unit 2154
/BORIS GORNEY/Supervisory Patent Examiner, Art Unit 2154
Prosecution Insights