DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the response to this Office Action, the Examiner respectfully requests that support be shown for language added to any original claims on amendment and any new claims. That is, indicate support for newly added claim language by specifically pointing to page(s) and line numbers in the specification and/or drawing figure(s). This will assist the Examiner in prosecuting this application.
Election/Restrictions
Applicant's election without traverse of Invention I: Figure 6 in the reply filed on 12/18/2025 is acknowledged.
Claims 4-6, 9-11, 15-17, and 20-21 recite limitations that are embodied by non-elected Invention II and are therefore withdrawn from consideration.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the claims at issue are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); and In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on a nonstatutory double patenting ground provided the reference application or patent either is shown to be commonly owned with this application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The USPTO internet Web site contains terminal disclaimer forms which may be used. Please visit http://www.uspto.gov/forms/. The filing date of the application will determine what form should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to http://www.uspto.gov/patents/process/file/efs/guidance/eTD-info-I.jsp.
Claims 2-3, 7-8, 12-14, and 18-19 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1, 5-7, 11, 14, 16, and 18-19 of U.S. Patent 12,056,275 B2. Although the claims at issue are not identical, they are not patentably distinct from each other.
The following is an example for comparing claim 2 of this application and claim 11 of U.S. Patent 12,056,275 B2:
Claim 2 of this application
Claim 11 of U.S. Patent 12,056,275 B2
2. A non-transitory, computer-readable storage medium including executable instructions that, when executed by one or more processors, cause the one or more processors to:
in response to a command received from a user of a head-wearable device:
obtain an image of a physical environment surrounding the user wearing the head-wearable device;
determine a region of interest in the image based at least on a gaze of an eye of the user, the region of interest including one or more available physical devices;
determine, by a machine-learning model, an intent of the user to interact with a particular physical device of the one or more available physical devices; and
based on the intent of the user to interact with a particular physical device, send a representation of the command to the particular physical device.
11. A computer-readable non-transitory storage medium including instructions that, when executed by one or more processors of a computer system, are configured to cause the one or more processors to:
capture an image of a physical environment surrounding a user wearing the AR device;
access, at a computing device, a three-dimensional (3D) map corresponding to the physical environment;
determine a pose of the AR device relative to the three-dimensional map based on first features of physical objects captured in the image and second features of object representations in the three-dimensional map;
determine a historic state relative to a present time and a current state, the current state including a location of the user at the present time;
receive an instruction from the user;
determine, using an eye tracker, a gaze of an eye of the user;
compute, based on the gaze and the pose, a region of interest in the three-dimensional map;
identify one or more representations of physical devices in the region of interest of the three-dimensional map;
assign a confidence distribution score associated with a physical device corresponding to one of the one or more representations of the physical device, based on at least one of the historic state and the current state;
determine, by a machine learning model, an intent of the user to interact with the physical device based on the instruction, the confidence distribution score associated with the physical device, and one or more contextual signals; and
issue a command to the physical device based on the determined intent of the user.
The limitations of claim 2 of current application are broader and are therefore anticipated by those found in claim 11 of U.S. Patent 12,056,275 B2.
Claims 3, 7-8, 12-14, and 18-19 are similarly rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1, 5-7, 11, 14, 16, and 18-19 of U.S. Patent 12,056,275 B2.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2-3, 7-8, 12-14, and 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication 2017/0123491 A1 to Hansen et al. (hereinafter "Hansen") in view of U.S. Patent Application Publication to 2017/0061694 A1 to Giraldi et al. (hereinafter "Giraldi”), and further in view of U.S. Patent Application Publication to 2022/0083198 A1 to Kaehler et al. (hereinafter "Kaehler").
Regarding Claims 2, 13, and 18, Hansen teaches a head-wearable device, a method for interacting with physical devices via a head-wearable device including: a camera; one or more processors; and a computer-readable non-transitory storage medium in communication with the one or more processors and comprising instructions, that, when executed by the one or more processors, are configured to cause the head-wearable device to: in response to a command received from a user of the head-wearable device (Fig. 1; Claims 1, 8-10; Para. 35, 74-78 of Hansen; event may be any detectable event indicating that the person wants to interact with the object, e.g. that the person's gaze has dwelled on the object for a predefined period of time, that the person has given a spoken command and/or a gesture command, or that a button is pressed… wearable computing device comprises a display 103 of the see-through type, an eye-tracker 102, a scene camera 107… scene camera 107 captures an image of the scene and thereby the lamp in front of the person's head. The scene camera outputs the image to a processor 113 that processes the image and identifies the gazed object): obtain, from the camera, an image of a physical environment surrounding a user wearing the head-wearable device (Fig. 1; Claims 1, 8-10; Para. 74-78 of Hansen; wearable computing device comprises a display 103 of the see-through type, an eye-tracker 102, a scene camera 107… scene camera 107 captures an image of the scene and thereby the lamp in front of the person's head. The scene camera outputs the image to a processor 113 that processes the image and identifies the gazed object); determine a region of interest in the image based at least on a gaze of an eye of the user, the region of interest including one or more available physical devices (Figs. 1-3; Para. 28-35,74-105 of Hansen; gaze vector intersects with the display 103 in a point-of-regard 106… eye-tracker is configured to compute an estimate of the person's gaze e.g. in the form of a gaze vector which may indicate a gaze direction relative to a predefined direction, e.g. the direction of the eye-tracker's camera or a vector normal to a region of the display 103 of the wearable device 108. A frame 205 depicts the location of a point-of-regard 206 on a display... eye-tracker may therefore detect that at least one of the eyes with iris 203 and pupil 202 has moved to the right in the eye socket 204. The position of the point-of-regard can be updated as shown in frame 205, where it is shown that the point-of-regard has moved to the right in the frame… component 406 uses a different eye-tracking setup and estimates the user's gaze as a 3D vector relative to the head. Having the position and orientation of the user's head (measured by sensors), the gaze vector can be estimated in the 3D space 412. The unit 413 finds the intersection of the gaze vector with the objects in the environment and recognizes the object that intersects the gaze… object identification is based on a 3D model, wherein objects' positions are represented in a 3D model space. The gaze is transformed to a 3D gaze point position, and the 3D model is examined to reveal the identity of one or more objects, if any, coinciding or being in proximity of the 3D gaze point position. The 3D model space may be stored in a local or remote database… When, and if, a gazed object is recognised, the processor 113 then monitors whether the gaze dwells on the recognised object also when one image or multiple further images is/are captured by the scene camera 107, i.e. whether the gaze dwells on the recognised object for a predefined first period of time. In the affirmative event thereof, the processor 113 displays a user interface element 104, with a spatial expanse, on the display 103 in a region adjacent to the point-of-regard 106. The spatial expanse is illustrated by the expanse extent of a line, but in embodiments the user interface element 104 is an expanse defined in a 2D or 3D space); determine an intent of the user to interact with a particular physical device of the one or more available physical devices (Figs. 1-3; Para. 28-35,74-105 of Hansen; gaze vector intersects with the display 103 in a point-of-regard 106… eye-tracker is configured to compute an estimate of the person's gaze e.g. in the form of a gaze vector which may indicate a gaze direction relative to a predefined direction, e.g. the direction of the eye-tracker's camera or a vector normal to a region of the display 103 of the wearable device 108. A frame 205 depicts the location of a point-of-regard 206 on a display... A frame, 207, shows an object of interest, 209… A box 208 is shown as a so-called bounding box and it represents the location of the object of interest 209… As shown in frame 210, the point-of-regard 206 coincides with the object of interest 209 and the second user interface element 212. This event is detected and in correspondence with the icon shown on the user interface element (a downwardly point arrow), a message is communicated to the object of interest to dim the light).
Hansen does not explicitly disclose determining, by a machine-learning model, an intent of the user to interact with a particular physical device of the one or more available physical devices; based on the intent of the user to interact with a particular physical device, sending a representation of the command to the particular physical device.
However, Giraldi teaches based on intent of a user to interact with a particular physical device, sending a command to the particular physical device (Figs. 4-5; Para. 47-70 of Giraldi; wearer input performed by the wearer of the HMD, such as a gesture (e.g., a pinching of fingers, closing of a fist, etc.), that indicates a virtual interaction with a user interface visually presented via a display of a computing device in the physical space… communication interface 512 may be configured to send control signals to a computing device to adjust operation of the computing device in order to facilitate a virtual interaction of a wearer of the HMD with the computing device).
Therefore, at the time when the invention was filed, it would have been obvious to a person of ordinary skill in the art to include based on the intent of the user to interact with a particular physical device, sending a representation of the command to the particular physical device using the teachings of Giraldi in order to modify the device taught by Hansen. The motivation to combine these analogous arts would have been to provide an approach for virtually interacting with a computing device by providing user input to the computing device via an augmented-reality device (Para. 11 of Giraldi).
The combination of Hansen and Giraldi does not explicitly disclose determining, by a machine-learning model, an intent of the user to interact with a particular physical device of the one or more available physical devices.
However, Kaehler teaches determining, by a machine-learning model, an intent of a user to interact with a particular physical device of one or more available physical devices (Figs. 8-11; Para. 81-101 of Kaehler; object recognitions can additionally or alternatively be performed by a variety of machine learning algorithm… wearable system may receive input from the user and other users regarding the environment of the user. This may be achieved through various input devices, and knowledge already possessed in the map database. The user's FOV camera, sensors, GPS, eye tracking, etc., convey information to the system at block 810… The object recognizers 708 a-708 n may crawl through these collected points and recognize one or more objects using a map database at block 830… Based at least partly on the detected gesture, eye pose, head pose, or input through the totem, the wearable system detects a position, orientation, or movement of the totem (or the user's eyes or head or gestures) with respect to a reference frame… The reference frame may be a set of map points based on which the wearable system translates the movement of the totem (or the user) to an action or command. At block 1030, the user's interaction with the totem is mapped. Based on the mapping of the user interaction with respect to the reference frame 1020, the system determines the user input at block 104).
Therefore, at the time when the invention was filed, it would have been obvious to a person of ordinary skill in the art to include determining, by a machine-learning model, an intent of the user to interact with a particular physical device of the one or more available physical devices using the teachings of Kaehler in order to modify the device taught by the combination of Hansen and Giraldi. The motivation to combine these analogous arts would have been for rendering virtual user input controls and detecting interactions with the virtual user input controls in virtual reality and augmented reality imaging and visualization systems (Para. 2-4 of Giraldi).
Regarding Claims 3, 14, and 19, the combination of Hansen, Giraldi, and Kaehler teaches that each of the one or more available physical devices is associated with a respective bounding box; and the determination of the intent of the user to interact with the particular physical device is based, in part, on a respective bounding box associated with the particular physical device (Figs. 1-3; Para. 74-94 of Hansen; gaze vector intersects with the display 103 in a point-of-regard 106… eye-tracker is configured to compute an estimate of the person's gaze e.g. in the form of a gaze vector which may indicate a gaze direction relative to a predefined direction, e.g. the direction of the eye-tracker's camera or a vector normal to a region of the display 103 of the wearable device 108. A frame 205 depicts the location of a point-of-regard 206 on a display... A frame, 207, shows an object of interest, 209… A box 208 is shown as a so-called bounding box and it represents the location of the object of interest 209).
Regarding Claim 7, the combination of Hansen, Giraldi, and Kaehler teaches that the one or more available physical devices include at least one of smart devices, smart units, and any internet-of-things (IoT) devices (Figs. 1-3; Para. 74-94 of Hansen; user deliberately desires to communicate with the object of interest 101. In the affirmative event, the event is processed by issuing an action e.g. comprising communicating a message to a remote system 115 via a communications unit 112. Communication may take place wirelessly via antennas 114 and 116. Communication may take place in various ways e.g. by means of a wireless network e.g. via a so-called Wi-Fi network or via a Bluetooth connection… remote system 115 is in communication with the object of interest by a wired and/or wireless connection. The remote system 115 may also be integrated with the lamp in which case such an object as the lamp is often denoted a network enabled device. Network enabled devices may comprise lamps or other home appliances such as refrigerators, automatic doors, vending machines etc.).
Regarding Claim 8, the combination of Hansen, Giraldi, and Kaehler teaches that the command received from the user of the head-wearable device is at least one of a head gesture, a hand gesture, a voice command, a finger tap, a drag and drop movement, a rotational movement, a button press, and a gaze gesture (Figs. 4-5; Para. 47-70 of Giraldi; wearer input performed by the wearer of the HMD, such as a gesture (e.g., a pinching of fingers, closing of a fist, etc.), that indicates a virtual interaction with a user interface visually presented via a display of a computing device in the physical space).
Regarding Claim 12, the combination of Hansen, Giraldi, and Kaehler teaches that the gaze of the eye of the user is based on gaze data received from a gaze-tracking camera of the head-wearable device (Figs. 1-3; Para. 74-94 of Hansen; gaze vector intersects with the display 103 in a point-of-regard 106… eye-tracker is configured to compute an estimate of the person's gaze e.g. in the form of a gaze vector which may indicate a gaze direction relative to a predefined direction, e.g. the direction of the eye-tracker's camera or a vector normal to a region of the display 103 of the wearable device 108. A frame 205 depicts the location of a point-of-regard 206 on a display... A frame, 207, shows an object of interest, 209… A box 208 is shown as a so-called bounding box and it represents the location of the object of interest 209).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ABHISHEK SARMA whose telephone number is (571)272-9887. The examiner can normally be reached on Mon - Fri 8:00-5:00.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amr Awad can be reached on 571-272-7764. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ABHISHEK SARMA/
Primary Examiner, Art Unit 2621