Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Priority
No foreign or domestic priority is claimed. The effective filing date of U.S. Application No. 18/960,642 is 11/26/2024.
Status of Claims
Claims 81–100 are pending in the application. Claims 81-86, 89-96, 99-100 are rejected.
Claims 87, 88, 97, 98 are objected to.
Allowable Subject Matter
Claims 87, 88, 97, 98 are objected to as being dependent upon a rejected base claim(s), but would be allowable if rewritten in independent form including all of the limitations of the base claim(s) and any intervening claim(s).
Overview of Grounds of Rejection
Ground of Rejection
Claim(s)
Statute(s)
Reference(s)
Ground of Rejection 1
81, 82, 84, 86, 90, 91, 92, 94, 96, 100
§ 103
Liang et al. (NPL), Wen et al. (NPL), Bennett (US20130241942A1)
Ground of Rejection 2
83, 85, 93, 95
§ 103
Liang et al. (NPL), Wen et al. (NPL), Bennett (US20130241942A1), Hickman et al. (US20140340398A1)
Ground of Rejection 3
89, 99
§ 103
Liang et al. (NPL), Wen et al. (NPL), Bennett (US20130241942A1), Xia et al. (NPL)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. § 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
(Please see the cited paragraphs, sections, pages, or surrounding text in the references for the paraphrased content.)
Ground of Rejection 1
Claims 81, 82, 84, 86, 90, 91, 92, 94, 96, 100 are rejected under 35 U.S.C. § 103 as being unpatentable over Liang et al. (NPL) in view of Wen et al. (NPL) and further in view of Bennett (US20130241942A1).
As per Claim 81, Liang et al. (NPL) teach the following portion of Claim 81, which recites:
“A method comprising: receiving a first application and a first set of attributes associated with the first application;”
Liang et al. (NPL) teaches TaskMatrix.AI, in which a foundation model generates executable action codes using APIs to accomplish tasks. Liang states that the MCFM is responsible for “generating executable codes based on APIs to accomplish specific tasks,” and that the API Executor executes generated action codes by “calling the relevant APIs.” Liang et al. (NPL), §2.1, Fig. 1, p. 2.
Liang teaches receiving/storing APIs and their associated documentation attributes. The API Platform stores “millions of APIs with different kinds of functions,” and each API document includes “API Name,” “Parameter List,” “API Description,” “Usage Example,” and “Composition Instructions.” The API corresponds to the claimed first application, and the API-document fields correspond to the claimed first set of attributes. Liang et al. (NPL), §2.1, §2.3, pp. 2-4.
Liang teaches the following portion of Claim 81, which recites:
“storing the first set of attributes:”
Liang teaches that the API Platform provides storage for different APIs and their documentation schema, including the above API attributes. Liang et al. (NPL), §2.3, p. 4.
Liang alone does not explicitly teach all the limitation(s) of the claim. However, when combined with Wen et al. (NPL), they collectively teach all of the limitation(s).
Liang and Wen teach the following portion of Claim 81, which recites:
“detecting a first input;”
Liang teaches receiving a user instruction as input. Liang states that the MCFM takes as inputs the API platform, “the user instruction, denoted as I,” and conversational context, and generates action codes to accomplish the user instruction. Liang et al. (NPL), §2.1, p. 2.
Wen teaches that, given “a natural language description of a desired task,” DroidBot-GPT can generate and execute actions. Wen also states that it translates “app GUI state information and the available actions on the smartphone screen to natural language prompts” and asks the LLM “to make a choice of actions.” Wen et al. (NPL), Abstract, p. 1.
Liang and Wen teach the following portion of Claim 81, which recites:
“identifying the first application based, at least in part, on the first input;”
Liang teaches that the API Selector recommends APIs based on the user command. Liang states that the API Selector “can recommend related APIs based on MCFM’s comprehension of the user command,” and identifies the “most suitable APIs from API platform that fit the task requirement and solution outline.” Liang et al. (NPL), §2.1, §2.4, pp. 2, 4.
Wen teaches Android application automation based on a task description. Wen states that “[g]iven an Android application and a task described by the user,” DroidBot-GPT fetches the app state, combines state information, action history, and the task into a prompt, and sends it to ChatGPT. Wen et al. (NPL), §2, p. 3.
Liang and Wen teach the following portion of Claim 81, which recites:
“selecting a first attribute of the first set of attributes associated with the first application;”
Liang teaches selecting and using API documentation attributes, including parameters, for action-code generation. Liang states that the “Parameter List” includes input parameters, return value, parameter name, description, data type, and default value, which assist MCFM in filling parameters in the corresponding positions. Liang et al. (NPL), §2.3, p. 4.
Wen in Figure 3 teaches GUI element attributes including “editable,” “clicked,” and “selected.” Wen et al. (NPL), Fig. 3, p. 3.
Liang teaches the following portion of Claim 81, which recites:
“identifying a first entry of a plurality of entries, wherein:”
Liang teaches API documents/entries in an API Platform containing millions of APIs. The selected API document is the claimed first entry of a plurality of entries. Liang et al. (NPL), §2.1, §2.3, pp. 2-4.
Liang teaches the following portion of Claim 81, which recites:
“each entry of the plurality of entries associates one or more attributes with one or more applications; and”
Liang teaches that each API document associates API attributes, including API name, parameters, description, usage example, and composition instructions, with a corresponding API/application. Liang et al. (NPL), §2.3, p. 4.
Liang teaches the following portion of Claim 81, which recites:
“the first entry associates the first attribute with the first application and one or more additional applications;”
Liang teaches API packages and composition instructions for combining multiple APIs. Liang states that developers offering a package of APIs may provide “Composition Instructions” as guidance on “how to combine multiple APIs to accomplish complex user instructions.” Thus, an API entry/package associates an attribute, such as a parameter or composition rule, with a first API/application and additional APIs/applications. Liang et al. (NPL), §2.3, p. 4.
Liang teaches the following portion of Claim 81, which recites:
“identifying a second application of the one or more additional applications based, at least in part, on identifying the first entry of the plurality of entries, wherein the second application is associated with a plurality of actions;”
Liang teaches selecting related APIs and executing actions using those APIs. The API Selector selects suitable APIs, and the Action Executor runs APIs “ranging from simple HTTP requests to complex algorithms or AI models.” Liang also gives PowerPoint APIs such as “create_slide, select_title, select_content, insert_text, move_to_slide, resize_picture, move_picture,” which correspond to a plurality of actions. Liang et al. (NPL), §2.4-2.5, §3.3, pp. 4, 11.
Liang and Wen teach the following portion of Claim 81, which recites:
“identifying a first action of the plurality of actions based, at least in part, on the first attribute; and”
Liang teaches generating action codes using selected APIs and their attributes. For example, the PowerPoint API documentation includes select_title() and insert_text(text:str), and explains that the title/content box should be selected before inserting text. The selected parameter/text attribute is used to identify the API action to perform. Liang et al. (NPL), §4.2, Fig. 11, pp. 18-19.
Wen teaches available actions including “click,” “long click,” “check,” “edit,” “scroll up,” and “scroll down,” and further teaches that when ChatGPT returns a numbered choice, DroidBot-GPT executes the corresponding action, e.g., “click the button ‘Sort by’.” Wen et al. (NPL), §2.2, Fig. 3, p. 3.
Liang and Wen alone do not explicitly teach all the limitation(s) of the claim. However, when combined with Bennett, they collectively teach all of the limitation(s).
Bennett teaches the following portion of Claim 81, which recites:
“causing a first notification to be displayed, wherein the first notification corresponds to the first action.”
Bennett teaches displaying a prompt corresponding to an action option. Bennett states that “the user may be prompted... to select between continuing at the first performance level... or to reduce graphics performance to a second performance level,” and that the user input may cause adaptive graphics processing circuitry to adapt a graphics feature. The displayed prompt corresponds to the claimed first notification, and the performance-reduction option corresponds to the claimed first action. Bennett, ¶ [0026].
Before the effective filing date of the claimed invention, a person of ordinary skill in the art would have been motivated to combine Liang et al. (NPL)’s API platform, API selector, and action executor with Wen et al. (NPL)’s mobile GUI automation and Bennett’s displayed user prompt/notification. Liang provides structured entries for APIs/applications and action-code execution; Wen confirms that mobile application GUI state and available actions can be translated into prompts for choosing and executing actions on a smartphone; and Bennett teaches displaying a user-facing prompt corresponding to a selectable action. The combination would have predictably improved application/action selection and user notification by using stored attribute-to-API entries to select relevant actions and present corresponding notifications, producing expected results in automated task-completion systems.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
As per Claim 82, Liang alone does not explicitly teach all the limitation(s) of the claim. However, when combined with Bennett, they collectively teach all of the limitation(s).
Bennett teaches the following portion of Claim 82, which recites:
“The method of claim 81, further comprising: receiving a selection of the first notification; and”
Bennett teaches displaying a user prompt and receiving the user’s selection. Bennett states that “the user may be prompted ... to select between continuing at the first performance level ... or to reduce graphics performance to a second performance level.” Bennett, ¶ [0026].
Bennett teaches the following portion of Claim 82, which recites:
“causing a device to perform the first action based, at least in part, on the selection of the first notification.”
Bennett further teaches that the user’s selection causes the device to perform the selected action. Bennett states that “The user input may cause adaptive graphics processing circuitry 134 to adapt one or more graphics feature.” Bennett, ¶ [0026].
The rationale and motivation to combine the references as set forth for claim 81 are incorporated herein by reference for the present claim.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
As per Claim 84, Liang alone does not explicitly teach all the limitation(s) of the claim. However, when combined with Bennett, they collectively teach all of the limitation(s).
Bennett teaches the following portion of Claim 84, which recites:
“The method of claim 81, wherein the first set of attributes and the first application are stored in a memory of a device.”
Bennett teaches a device having storage/memory that stores applications and associated attribute/settings data. Bennett states that “example device 200 includes storage 202,” and that “Storage 202 includes... thermally and/or power adaptive client-side graphics applications 236, thermally and/or power adaptive stand-alone graphics applications 238, thermally and/or power adaptive APIs 240, and thermally and/or power adaptive graphics drivers 242.”
Bennett further teaches stored/current graphics feature settings, stating that “GPU 602 also includes data 606, which may include graphics feature settings.” Bennett, ¶¶ [0040], [0042], [0068].
Thus, Bennett teaches the claimed first application stored in device storage/memory and the claimed first set of attributes stored as graphics feature/settings data.
The rationale and motivation to combine the references as set forth for claim 81 are incorporated herein by reference for the present claim.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
As per Claim 86, Liang alone does not explicitly teach all the limitation(s) of the claim. However, when combined with Wen et al. (NPL), they collectively teach all of the limitation(s).
Wen teaches Claim 86, which recites:
“The method of claim 81, further comprising: transmitting a first prompt associated with the first attribute to the second application; and receiving, from the second application, a second prompt associated with the first action.”
Wen teaches transmitting a prompt containing GUI/application state, attributes, action choices, task information, and action history to a second application, i.e., ChatGPT/LLM. Wen states that DroidBot-GPT “combines the state information, the action history, and the task into a prompt, and sends it to ChatGPT,” and that “ChatGPT generates the sends back the proper action.” Wen et al. (NPL), §2, p. 3.
Wen further teaches that GUI element attributes and available actions are included in the prompt. For example, GUI element attributes include “editable,” “clicked,” and “selected,” and available actions include “click,” “long click,” “check,” “edit,” “scroll up,” and “scroll down.” Wen et al. (NPL), Fig. 3, p. 3.
Wen also teaches receiving an action-associated response from ChatGPT. For example, where a prompt contains “a view ‘Sort by’ that can click (0),” ChatGPT returns “0,” and DroidBot-GPT executes “click the button ‘Sort by’.” Wen et al. (NPL), §2.2, p. 3.
Thus, Wen teaches transmitting the claimed first prompt associated with the selected attribute/action information to the second application and receiving, from that second application, returned action text/selection associated with the claimed first action.
The rationale and motivation to combine the references as set forth for claim 81 are incorporated herein by reference for the present claim.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
As per Claim 90, Liang alone does not explicitly teach all the limitation(s) of the claim. However, when combined with Wen et al. (NPL), they collectively teach all of the limitation(s).
Wen teaches Claim 90, which recites:
“The method of claim 81, wherein the first input is a tap on a touchscreen.”
Wen et al. teaches Android smartphone UI automation in which available actions on the smartphone screen are translated into prompts and selected for execution. Wen states that DroidBot-GPT translates “the app GUI state information and the available actions on the smartphone screen” into prompts and asks the LLM “to make a choice of actions.” Wen further teaches mobile-device actions including “click,” and gives the example “click the button ‘Sort by’.” Wen et al. (NPL), Abstract, §2.2, Fig. 3, pp. 1, 3.
Thus, Wen’s smartphone-screen click action reasonably teaches or suggests the claimed tap on a touchscreen.
The rationale and motivation to combine the references as set forth for claim 81 are incorporated herein by reference for the present claim.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
Claim 91 does not include any additional limitations that would significantly distinguish it from claim 81. Therefore, it is likewise rejected under 35 U.S.C. § 103 in view of the same references and for the same reasons set forth above.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
Claim 92 does not include any additional limitations that would significantly distinguish it from claim 82. Therefore, it is likewise rejected under 35 U.S.C. § 103 in view of the same references and for the same reasons set forth above.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
Claim 94 does not include any additional limitations that would significantly distinguish it from claim 84. Therefore, it is likewise rejected under 35 U.S.C. § 103 in view of the same references and for the same reasons set forth above.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
Claim 96 does not include any additional limitations that would significantly distinguish it from claim 86. Therefore, it is likewise rejected under 35 U.S.C. § 103 in view of the same references and for the same reasons set forth above.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
Claim 100 does not include any additional limitations that would significantly distinguish it from claim 90. Therefore, it is likewise rejected under 35 U.S.C. § 103 in view of the same references and for the same reasons set forth above.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
Ground of Rejection 2
Claims 83, 85, 93, 95 are rejected under 35 U.S.C. § 103 as being unpatentable over Liang et al. (NPL) in view of Wen et al. (NPL), further in view of Bennett (US20130241942A1), and further in view of Hickman et al. (US20140340398A1).
As per Claim 83, Liang alone does not explicitly teach all the limitation(s) of the claim. However, when combined with Bennett and Hickman, they collectively teach all of the limitation(s).
Bennett and Hickman teach Claim 83, which recites:
“The method of claim 81, wherein the first notification is displayed on a wearable extended reality (XR) device.”
Bennett teaches displaying a user prompt/notification corresponding to an action, stating that “the user may be prompted” to select between continuing at a first performance level or reducing graphics performance to a second performance level. Bennett, ¶ [0026].
Hickman teaches wearable/headset-type display devices for displaying 3D content. Hickman states that a 3D object data model “can be rendered or displayed as a two-dimensional image via 3D rendering or displayed as a three-dimensional image,” and further states that the computing device may be “a wireless web-watch device” or “a personal headset device.” Hickman et al., ¶¶ [0003], [0073].
Before the effective filing date of the claimed invention, a person of ordinary skill in the art would have been motivated to display Bennett’s user prompt/notification on Hickman et al.’s wearable/headset-type display device to provide the user with the same action-related prompt in the user’s active display environment. Bennett teaches prompting the user to select an action option, and Hickman teaches that computing devices may include a “wireless web-watch device” or “personal headset device” capable of displaying 3D content. The combination would have predictably improved usability by allowing action notifications to be viewed and selected directly on a wearable/headset display, with expected results.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
As per Claim 85, Liang alone does not explicitly teach all the limitation(s) of the claim. However, when combined with Bennett and Hickman, they collectively teach all of the limitation(s).
Bennett and Hickman teach Claim 85, which recites:
“The method of claim 84, wherein: the device is a wearable extended reality (XR) device; and the device displays the first notification.”
Bennett teaches the device displaying the claimed notification because Bennett states that “the user may be prompted” to select between continuing at a first performance level or reducing graphics performance to a second performance level. Bennett, ¶ [0026].
Hickman teaches a wearable/headset-type device capable of 3D display, stating that a 3D object data model may be “displayed as a three-dimensional image,” and that the computing device may be implemented as a “wireless web-watch device” or “personal headset device.” Hickman et al., ¶¶ [0003], [0073].
Thus, it would have been obvious to display Bennett’s user prompt/notification on Hickman’s wearable headset-type 3D display device, thereby teaching the claimed device as a wearable XR-type device that displays the first notification.
The rationale and motivation to combine the references as set forth for claim 83 are incorporated herein by reference for the present claim.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
Claim 93 does not include any additional limitations that would significantly distinguish it from claim 83. Therefore, it is likewise rejected under 35 U.S.C. § 103 in view of the same references and for the same reasons set forth above.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
Claim 95 does not include any additional limitations that would significantly distinguish it from claim 85. Therefore, it is likewise rejected under 35 U.S.C. § 103 in view of the same references and for the same reasons set forth above.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
Ground of Rejection 3
Claims 89, 99 are rejected under 35 U.S.C. § 103 as being unpatentable over Liang et al. (NPL) in view of Wen et al. (NPL), further in view of Bennett (US20130241942A1), and further in view of Xia et al. (NPL).
As per Claim 89, Liang alone does not explicitly teach all the limitation(s) of the claim. However, when combined with Xia, they collectively teach all of the limitation(s).
Xia teaches Claim 89, which recites:
“The method of claim 81, wherein the first input is a gaze of a user.”
Xia et al. teaches adaptive real-time rendering based on a user/viewer’s viewing direction, stating that the simplifications are dependent on “viewing direction,” and that adaptive levels of detail may be based on vector attributes such as “the direction of vertex normals.” Xia et al. (NPL), Abstract, §I, pp. 171-172.
Thus, Xia’s viewing direction provides the best available teaching of the claimed gaze of a user as an input for view-dependent processing.
Before the effective filing date of the claimed invention, a person of ordinary skill in the art would have been motivated to use Xia et al.’s view-dependent input, including “viewing direction,” as the user input in the Liang/Wen automated application-action system because gaze/viewing direction was a known input for adapting displayed content in interactive 3D environments. The combination would have predictably allowed the system to identify or surface actions based on where the user is looking, improving hands-free interaction with expected results.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
Claim 99 does not include any additional limitations that would significantly distinguish it from claim 89. Therefore, it is likewise rejected under 35 U.S.C. § 103 in view of the same references and for the same reasons set forth above.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
Conclusion
The prior art made of record and relied upon in this action is as follows:
Patent Literature:
Hickman et al. (US20140340398A1) - “Encoding and Compressing Three-Dimensional (3D) Object Data Models.”
Bennett (US20130241942A1) - “Thermal and Power Aware Graphics Processing.”
Non-Patent Literature (NPL):
Liang et al. - “TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs,” 2023-03-29. Available at: [https://arxiv.org/pdf/2303.16434]
Wen et al. - “DroidBot-GPT: GPT-powered UI Automation for Android,” 2024-01-07. Available at: [https://arxiv.org/pdf/2304.07061]
Xia et al. - “Adaptive Real-Time Level-of-detail-based Rendering for Polygonal Models,” 1997-06. Available at: [https://www.cs.umd.edu/~varshney/papers/av_vd.pdf]
Note: A PDF copy of each NPL reference is attached with this Office Action. URLs are included for applicant convenience. If a link becomes unavailable in the future, the citation information may be used to locate the reference or access archived versions via the Wayback Machine.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure and is listed as follows:
Patent Literature:
Hoppe (US20050116950A1) - “Regional progressive meshes.”
Maetz et al. (US20140115345A1) - “Methods and devices for optimising rendering of an encrypted 3d graphical object.”
Non-Patent Literature (NPL):
Mahdaoui et al. - “3D Point Cloud Simplification Based on k-Nearest Neighbor and Clustering,” 2020-07-15. Available at: [https://onlinelibrary.wiley.com/doi/epdf/10.1155/2020/8825205]
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ADEEL BASHIR whose telephone number is (571) 270-0440. The examiner can normally be reached Monday-Thursday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Hajnik can be reached on (571) 276-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ADEEL BASHIR/
Examiner, Art Unit 2616
/DANIEL F HAJNIK/Supervisory Patent Examiner, Art Unit 2616
Prosecution Insights