DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This Office action is responsive to the reply filed on 08/09/2026. The claims 6, 17- 21, & 25- 26 were cancelled by Applicant. Accordingly, the claim(s) 1-5, 7-16, & 22- 24 is/are pending herein, of which the claim(s) 1 & 14 is/are in independent form.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
In light of the amendments made to the claim 3, the outstanding claim objection thereto has been rendered moot and therefore is withdrawn.
Response to Arguments
I) As to outstanding prior art rejections, applicant’s arguments with respect to claim(s) 1 & 14 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Specifically, applicant’s arguments (see Remarks, pages 7-11) directed to prior cited Gollnick and the motivation (see Remarks, page 12) applied to combine Tian (US 20200117898 A1) with Gollnick are not applicable. A new ground of art rejection is introduced herein as necessitated by amending of the independent claims.
Note: due to applicant’s amending of the independent claims to recite the new subject matter of “the autonomous action considers but is not identical to the input” and “determine at least a portion of the autonomous action independently of the input” necessitated new interpretation of the prior cited Tian reference. Accordingly, the current office action has been made final.
II) As to claims 1 & 14, applicant argues that not only Gollnick but the Tian (US 20200117898 A1) also fails to teach limitations of the prior dependent claim 6 (which is incorporated in claim 1 and similarly in claim 14).
Applicant argues that Tian “describes user-initiated, discretionary intervention, not a system generated request triggered by an inability of the machine learning algorithm.” See Remarks, page 12. While examiner agrees that Tian does not expressly teach “requesting the user” portion to provide the input, nevertheless, it renders the limitation of previous dependent claim 6 obvious to PHOSITA. See MPEP 2144.01. This is so because Tian’s goal is to-- continue autonomous actions as long as possible and request user to provide input via haptic feedback when needed. Thus, the previous claim 6 was rendered obvious over Tian since “haptic feedback” to the operator is generated for requesting user to provide input which can be due to robot’s failure of executing the automation action itself. Specifically, the goal of Tian is being described as:
[006] technique provides the user more freedom and information during the interaction with the physical environment via the robot, such as switching viewpoints in real-time, inspecting selected portions of the environment,
[005] techniques for supporting indirect and/or intermittent intervention in a robot's operation.
[033] haptic-enabled mixed-initiative remote robotic control utilizes a human user to provide high-level guidance in real-time and to structure and guide the low-level navigation and operation capabilities of the robot, such that the robot operates with sufficient guidelines and ample margin for machine errors.
That is, the haptic feedback of the “haptic feedback provided to the user via a haptic-enabled input device” can be understood or can suggest/request the user to provide the input as claimed.
Note: Nevertheless, due to discovery of new prior art US 20230048635 A1 to Kravitz that explicitly and directly teaches argued limitation, a new ground of the 103 rejection is introduced herein. That is, Kravitz clearly teaches “a system generated
request triggered by an inability of the machine learning algorithm” as required by the amended claims 1 & 14.
Kravitz discloses:
[0012] (i) The portable computerized device, wherein in case the one or more machine learning models fail to identify the item, the processing circuitry is further configured to generate and prompt instructions to a user requesting the user to scan the identification tag with the scanner.
[0013] (ii) The portable computerized device, wherein, in case the one or more machine learning models fail to identify the item, the processing circuitry is configured to:
[0014] identify the item based on scanning output data generated by the scanner in response to scanning of a respective identification tag associated with the item.
49. The method of claim 48, further comprising in case the one or more machine learning models fail to identify the item, generating and prompting instructions to a user requesting the user to scan with the scanner an identification tag that is associated with the item.
Claim Rejections - 35 USC § 112
Claims 1- 3, 7- 15, & 22- 24 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 1, the claim recites “the autonomous action considers but is not identical to the input” in line 13. However, there is insufficient antecedent basis in the limitation for phrase “the autonomous action” thereby rendering the scope of the claim indefinite.
Examiner notes, the claim also recites “the autonomous robotic device is configured to perform an autonomous action based, at least m part, on the input transmitted by the user interface” in lines 15-16. However, the lines 15-16 are recited after the line 13 that recites “the autonomous action considers but is not identical to the input”. Therefore, the limitation of lines 15- 16 fails to provide clear antecedent basis for “an autonomous action”.
For the examination, purpose, the limitation of line 13 is interpreted as being after the phrase “user interface” in limitation of lines 15-16.
Regarding claim 4, the claim recites “a machine learning algorithm”, however, the claim 1 is amended to recite “use a machine learning algorithm” in line 20. Therefore, the claim 4 fails to clarify the relationship for the claim element “machine learning algorithm” in claim 1 and claim 4 thereby rendering the scope of the claim 4 indefinite. Specifically, it is not clear whether “a machine learning algorithm” recited in claim 4 corresponds to same machine learning model of the claim 1 or a different machine learning model.
For the examination purpose, this is interpreted as same machine learning model as that of the claim 1.
Regarding claims 2- 5 & 7- 13, they are also rejected because of their dependency with rejected claim 1.
Regarding claim 16, the claim recites “the machine learning algorithm has
been trained using data points representative of the physical environment, data points indicative of a success score of the autonomous action.”
However, the claim fails to clarify whether the 2nd “data points” corresponds to first “data points” of the claim 16 or corresponds to different data points thereby rendering the scope of the claim indefinite.
For the examination purpose, 2nd data points is interpreted as “the data points”.
Claim Rejections - 35 USC § 103
Claim(s) 1- 3, 7- 15, & 22- 24 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tian et al. (US 20200117898 A1, reference of record) in view of Kravitz (US 20230048635 A1).
Regarding claim 1, Tian teaches an extended reality (XR) system [“mixed-initiative robotic control environment 100”, the mixed-initiative robotic control is a subset of extended reality] comprising: ([042, 0124], Figs. 1, 8, 10);
[a] an autonomous robotic device [“human operator controlling the robot”, e.g., robot 102 of fig. 1] located in a physical environment [local site 104] at a device location [a location where sensor(s) like 3D camera 106 to monitor robots are present] ([006, 043], Fig. 1, Fig. 8);
[b] a user interface [interface that user (“haptic-enabled mixed reality system that provides haptic interfaces”, “displaying a control user interface via a display generation component coupled to the computing device” like item 108) can use in remote site like in site 105 of fig. 1 using a “computing device is communicably coupled to a robot”] configured to: ([007-008, 045]);
- construct an 1XR environment [“facilitates remote mixed-initiative robotic control”] corresponding to at least a portion of the physical environment ([005-006, 045-046, 0125]);
- receive an input [“a mixed reality environment is utilized for the user to perceive the 3D scene and provide guidance inputs.” “operator 108 uses the haptic-enabled input device to insert one or more virtual objects”, “While displaying the virtual pointer object in the virtualized environment, the computing device detects (806) first user input via a haptic-enabled input device. The first user input includes a first movement input and one or more marking inputs at one or more locations along a movement path”] from a user [Fig. 1, “a remotely located human operator 108”] at a user location [remote site 105] interacting with the XR environment, the input based on the user's perception in the XR environment ([004-006, 049, 061, 077, 0103]); and
- transmit [“the marking and movement of the virtual object is processed in real-time in the virtualized scene and followed by the robot in real-time in the physical scene”, “robot is configured to navigate within a physical environment in accordance with the planned path received from the computing device” wherein “the computing device is a computer that is collocated with a human operator that facilitates remote interaction between the user and the robot”. The user entered planned path is received at the robot 102 means the path selection is being transmitted] the input to the autonomous robotic device ([006, 0049, 061, 0101]);
wherein:
the user location [location where the user 108 is located is different than the location of the robot 102] is remote [“the human operator is remotely located from the robot and the factory server” as clearly shown in fig. 1] from the device location (Fig. 1, [045, 051]);
the autonomous action [the performed low level tasks that are determined by sufficient programming and training at the robot itself are not same as provided input by the user; this disclosure of Tian is similar concept disclosed in applicant’s spec in para. 2043] considers but is not identical to the input ([008, 031, 048]); and
the autonomous robotic device is configured to:
perform an autonomous action [“human user to tailor the amount and types of control and intervention provided to the un-customized robot, such that the robot
performs customized tasks in its actual customized environment”, “robot has onboard processing capabilities to perform low level functions such as moving, grabbing, translating and rotating a grabbed object, and optionally obstacle avoidance during movement… perform intermedia level tasks, such as folding, sweeping, wiping, pressing”] based, at least in part, on the input [“a human user to tailor the amount and types of control”] transmitted by the user interface ([010, 032, 043, 0124]);
determine at least a portion [“relies on the robot's existing programming for low level tasks that are well trained and programmed”, “locally-stored task-execution instructions”, “robot is programmed to perform various low level or intermediate level tasks in accordance with environment input collected via on-board sensors”] of the autonomous action independently of the input ([029-031, 0111]); and
request [“The haptic feedback provided to the user via a haptic-enabled input device”, “Haptic-feedback provided to the human operator through the haptic-enabled input device… and more accurately provides guidance to the robot by changing”] the user to provide [“mixed-initiative remote robotic control utilizes a human user to provide high-level guidance”] the input, when the autonomous robotic device is unable to use a machine learning algorithm [“such that the un-customized robot may utilize the history to self-training through machine learning”] to perform the autonomous action without the user's input ([029-033]).
Since Tian does not expressly state that the providing of the haptic feedback is after determining autonomous robot is unable to use a machine learning, it can be argued that Tian’s generating of the haptic feedback to the human operator to provide/enter human’s high-level commands/tasks may or may not be “when the autonomous robotic device is unable to use a machine learning algorithm” as claimed. That is, Tian may or may not teach:
“when the autonomous robotic device is unable to use a machine learning algorithm”.
While examiner disagrees with such argument, nevertheless by giving the benefits of doubts, examiner further relies on the disclosure of Kravitz that explicitly discloses this condition as well.
Kravitz teaches a system comprising an autonomous device [device 300 attached with a shopping cart 200 and configured with sensors like cameras] to perform one or more autonomous actions [“detect and identify items being inserted or removed from the shopping container”] independent of user input using one or more machine learning algorithms and located in a physical environment [the retailer] (Figs. 1- 2, [086, 093, 098, 0103]). Specifically, Kravitz teaches the autonomous robotic device is configured to: request the user to provide [“in case the one or more machine learning models fail to identify the item, the processing circuitry is further configured to generate and prompt instructions to a user requesting the user to scan the identification tag with the scanner”] the input when [“in case the one or more machine learning models fail to identify the item, generating and prompting instructions”] the autonomous robotic device is unable to use a machine learning algorithm to perform the autonomous action without the user's input ([012-016, 098], Fig. 5, No in Step 513, Claim 49).
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to have (1) combined Kravitz and Tian because they both related to a device performing pluralities of autonomous actions using decisions both from machine learning models and user inputs and (2) modified Tian’s autonomous robotic device to have requested the user to provide the input when the autonomous robotic device is unable to use a machine learning algorithm to perform the autonomous action without the user's input as in Kravitz. Doing so even if the autonomous actions cannot be performed by the learning models of the robot 102, human user 108 will be notified/prompted of such fact and can resolve the problem by providing high level inputs (Kravitz [098]). Furthermore, Kravitz teaches a solution to the Tian’s system about how to perform tasks (requesting to user to provide input to execute high level tasks) that are not fully trained or programmed to be performed by Tian’s robotic device (Kravitz, [098] & Tian [006]).
Accordingly, Tian in view of Kravitz teaches each element of the claim and renders invention thereof obvious to PHOSITA.
Regarding claim 2, Tian in view of Kravitz further teaches/suggests the XR system of claim 1, wherein the user location is physically remote from the device location [the local site 104 is different from the location of the remote site 105] (Tian, Fig. 1 [045]).
Regarding claim 3, Tian in view of Kravitz further teaches/suggests the XR system of claim 2, wherein the input is transmitted to the autonomous robotic device [item 102] via a network interface; and the physical remoteness between the device location and the user location [location of item 108] is such that the user's perception of the physical environment is only through the XR environment (Fig. 1 & associated texts, [0124]).
Regarding claim 7, Tain in view of Kravitz further teaches/suggests the XR system of claim 3, wherein the user interface is configured to receive the input from the user via the network interface (Tian, [045-046], Fig.1 & associated texts).
Regarding claim 8, Tain in view of Kravitz further teaches/suggests the XR system of claim 3, wherein the user interface is further configured to enable [“display generation component 112, such as a display, a projector, a heads up display or the like, is used to display a virtualized version of the physical environment”; this display can be observed by many users like other uses like 108s at the same time in the screen 112 or in the screen of the server 110] one or more additional users to interact with the XR environment to monitor the input provided by the user (Tian Fig. 1 [045, 0112]).
Regarding claim 9, Tain in view of Kravitz further teaches/suggests the XR system of claim 3 further comprising:
one or more sensors [sensors 106] at the device location configured to monitor at least one discrete data value [data captured by sensors 106] in the physical environment; wherein the user interface is further configured to display the XR environment based, at least in part, on at least one discrete data value (Tian [043, 0102]).
Regarding claim 10, Tain in view of Kravitz further teaches/suggests the XR system of claim 3 further comprising: user equipment [“the haptic-enabled input device (e.g., a haptic-enabled 3D mouse 114)” like point device/mouse] configured to allow the user to interact with the XR environment (Tian [046, 0103]).
Regarding claim 11, Tain in view of Kravitz further teaches/suggests the XR system of claim 10, wherein the user interface is further configured to display the constructed XR environment; and the user equipment comprises a head mounted display (HMD) [“display generation component (e.g., a display, a heads-up display”, the display 112 can be mounted in user’s head to provide better vision and avoid being distracted with surrounding environment to increase user’s focus on monitoring the activities of the robot—see MPEP 2144.01] configured to display the XR environment to the user (Tian [045, 102], Fig. 1).
Regarding claim 12, Tain in view of Kravitz further teaches/suggests the XR The XR system of claim 10, wherein the user equipment comprises a controller [e.g., a means that allows user to provide input, e.g., “provided by a mouse, joystick, touch-pad, haptic-enabled pointing device, etc.”] configured to allow the user to provide the input (Tian [0102-0103]).
Regarding claim 13, Tain in view of Kravitz further teaches/suggests the XR system of claim 10, wherein:
the user interface is further configured to display the constructed XR environment; the user equipment comprises: a HMD [“a display generation component 112, such as a display, a projector, a heads up display”] configured to display the XR environment to the user (Fig. 1 [045]); and
a controller [“haptic-enabled input device 114 (e.g., a three-dimensional haptic-enabled pointing device, a haptic-enabled glove, etc.)”] configured to allow the user to provide the input; and the user interface is further configured to monitor movement of the controller by the user and alter [changing the display based on the user’s selections/inputs and corresponding actions performed by the robot 102 being displayed at the display screen 112] the display of the XR environment based on one or more monitored movements (Tian, Fig. 1, [046, 092]).
Regarding claim 14, Tian in view of Kravitz teaches/suggests invention of this method claim for the similar reasons set forth above in system claim 1.
Regarding claim 15, Tian in view of Kravitz teaches/suggests the method of claim 14, wherein the performing is further based, at least in part, by using a machine learning [“self-training through machine learning”] algorithm (Tian [031-032] & Kravitz [009]).
Regarding claim 22, Tian in view of Kravitz teaches/suggests the method of claim 14 further comprising: displaying the constructed XR environment (Tian [056]); and
monitoring, with one or more sensors [items 106] at the device location, at least one discrete data value in the physical environment; wherein the displayed XR environment is based [“RGBD data 302 received from the sensors (e.g., camera 106) collocated with the robot (e.g., robot 102) is streamed over a network and provided to a graphical processing unit 302”], at least in part, on the at least one discrete data value (Tian Fig. 1, [063]).
Regarding claim 23, Tian in view of Kravitz teaches/suggests the method of claim 22 further comprising: interacting, by the user using user equipment [item 114], with the XR environment (Tian Fig. 1).
Regarding claim 24, Tian in view of Kravitz teaches/suggests the method of claim 14 further comprising: displaying the XR environment to the user on a head mounted display (HMD) [“virtualized scene) via a display generation component (e.g., a display, a heads-up display”]; generating, by the user, with a controller [controller/CPU/receiving circuit of the “input device 114”], the input; monitoring movement of the controller by the user; and altering the display of the XR environment based on the monitored movement (Tian [047, 0102, 0125]).
Claim(s) 4- 5 & 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tian in view of Kravitz, and further in view of Bramble (US 20220382620 A1).
Regarding claim 4, Tian in view of Kravitz further teaches/suggests the XR system of claim 3, wherein:
the autonomous robotic device is further configured to use a machine learning algorithm [“history to self-training through machine learning”, “utilize the history to self-training through machine learning and other artificial intelligence training methods”] to perform the autonomous action and the machine learning algorithm is trained using data points representative of the physical environment
Tian in view of Kravitz fails to teach the data points indicative of a success score of the autonomous action as shown above with strikethrough emphasis but are cured by Bramble.
Bramble relates to monitoring failure and successful events occurred in data pipelines 104 in computing systems using machine learning to determine which actions work better and to identify patterns related to the input data (Fig. 1, [016, 018]). Specifically, Bramble teaches a computing device to:
use a machine learning algorithm to perform the autonomous action, the machine learning algorithm is trained using data points [“a model may be trained with historical failure events and remedial actions performed that were successful and not successful”] indicative of a success score [successful and not-successful action(s) indicates success score] of the autonomous action ([020, 041, 064-065])
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to have (1) combined Bramble and Tian in view of Kravitz because they both related to training machine learning model with historical automation actions logs and (2) also train the machine learning models of Tian in view of Kravitz with data points indicative of success score of the automation actions as in Bramble. Doing so would allow to identify which autonomous actions to be performed by the autonomous robotic device of Tian in view of Kravitz may cause anomalies ahead of time and further increase accuracy of machine learning models (Bramble [0065-066]).
Regarding claim 5, Tian in view of Kravitz further teaches the XR system of claim 3, wherein, the autonomous robotic device is further configured to determine one more predicted values [“generates predicted positions and commands”] for the automation actions and request [“Haptic-feedback provided to the human operator”/ “prompting him to scan the barcode of the item just inserted” of Kravitz] the user to provide the input (Tian [034, 053] & Kravitz [098]).
However, Tian in view of Kravitz fails to teach the autonomous robotic device is further configured to
determine a predicted success score for the autonomous action; and request the user provide the input based on the predicted success score.
Bramble relates to monitoring failure and successful events occurred in data pipelines in computing systems using machine learning to select which failure remedial actions work better and to identify patterns related to the input data ([016, 018]). Specifically, Bramble teaches a computing device to:
determine a predicted success score [“a probability of success weighted with the cost”, “the probability of success”] for the autonomous action; and request the user to provide the input [input for another remedial action of “performs each of the additional remedial actions until the failure event is resolved”] based on the predicted success score ([020, 041, 060], Fig. 4,).
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to have (1) combined Bramble and Tian in view of Kravitz because they both related to a computing device to accept pluralities of user inputs for the automation actions and (2) modified the autonomous robotic device of Tian in view of Kravitz to determine a predicted success score for the autonomous action and request the user to provide the input based on the predicted success score (probability of success) as suggested by Bramble. Doing so would allow users of Tian in view of Kravitz’s system to select best input out of pluralities of available input options after receiving the feedback/request for the operator input (Tian [0103] & Bramble [060]).
Regarding claim 16, Tian in view of Kravitz and Bramble teaches/suggests invention of this method claim for the similar reasons set forth above in system claim 4.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Contacts
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SANTOSH R. POUDEL whose telephone number is (571)272-2347. The examiner can normally be reached Monday - Friday (8:30 am - 5:00 pm).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kamini Shah can be reached at (571) 272-2279. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SANTOSH R POUDEL/ Primary Examiner, Art Unit 2115
1 PHOSITA knows that the extended reality (XR) is a umbrella/genus term for the mixed/virtual reality, see the NPL cited in office action mailed on 08/19/2026 in footnote 1 of page 5
2 “does not require the user's input to control each facet of the operation to be performed… user's input could designate a point where the autonomous robotic… determine where to grasp the raw natural material, but the robot autonomously makes additional decisions”.
Prosecution Insights