DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-7, 9-15 and 17-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Furtwangler (US Patent Application Publication No. 2024/0192766) in view of Pinchon (US Patent Application Publication No. 2022/0198755).
Regarding claim 1, Furtwangler discloses a head-mounted display device, comprising [see abstract; a head-wearable device worn by a user, a representation of a user's position within an artificial-reality environment, in response to receiving an indication that a positional-control activation hand gesture has been performed]:
a camera unit, configured to capture a plurality of streaming images [see para. 0262; a user can initiate a video stream using the AR device and back-end tasks associated with performing the video stream (e.g., video rendering) can be offloaded to the HIPD, which the HIPD performs and provides corresponding data to the AR device to perform remaining front-end tasks associated with the video stream (e.g., presenting the rendered video data via a display of the AR device); which corresponds to capture image data with HMD];
and a processor, coupled to the camera unit, and the processor is configured
to: track a hand gesture according to the streaming images [see para. 0309; the positional-control input hand gesture is maintained and the positional-control action is based on movement a user's hand while the positional-control input hand gesture is maintained. The user's hand can be tracked and, while the gesture is maintained, the type of positional change is also adjusted. For example, the user can go from a walk to a sprint; from running/walking left to running/walking right; from looking left to looking right. Additionally, the maintained gesture can be tracked to perform multiple actions; which corresponds based on image data obtained];
monitor whether the hand gesture matches with a preparation pattern [see para. 0318, 0319; the positional-control activation hand gesture is a maintained gesture (e.g., a pinch) and the positional-control input hand gesture is release of the maintained gesture (e.g., a releasing pinch) and he method includes displaying at least on additional positional-control UI element adjacent to the positional-control UI, the additional positional-control UI element configured to perform an additional positional-control action; which corresponds to position control activate pinching gestures];
in response to the hand gesture matching with the preparation pattern at a first time point, activate a tracking of a hand movement during a sensing period started from the first time point until a second time point [see para. 0087, 0088, 0323; 0325; At a first point in time, a positional-control activation hand gesture (e.g., a pinch gesture) is performed, which causes the head-wearable device to display the positional-control UI and the navigation UI element the positional-control activation hand gesture is performed by a first hand, the positional-control UI is a first positional-control UI, the positional-control UI element is a first positional-control UI element, the positional-control action is a first positional-control action, and the positional-control input hand gesture is a first positional-control input hand gesture. The method includes, while displaying the first positional-control UI, in response to receiving an indication that at a second point in time, the user, while holding the pinch gesture, moves their hand slightly. The slight hand movement of the user does not significantly change its position within the positional-control UI; which corresponds to the system activates the positional control user interface mode tracking active gesture is held and released];
and in response to the hand movement matching with the command pattern, execute an operation corresponding to the command pattern [see para. 0135, 0141; the positional-control input hand gesture is maintained and the positional-control action is based on movement a user's hand while the positional-control input hand gesture is maintained. In other words, the user 115's hand can be tracked such that a positional-control input hand gesture performs a different position-control action based on the positional-control input hand gesture's location relative to the positional-control UI. For example, based on the positional-control input hand gesture performed by the user, the user's representation within the AR environment transition from walking to running, looking left to looking right; etc. The user can maintain the positional-control input hand gesture (which is tracked) to perform multiple positional-control actions. For example, the user can perform positional-control input hand gestures to perform a running jump, a diagonal walk or run (e.g., movement in at least two directions), an interaction with an object while moving, a non-positional-control action in conjunction with a positional-control action (e.g., aim and shoot a rifle or throw an object while moving); which corresponds to execute based on the detected movement]; however, Furtwangler fails to explicitly teach during the sensing period, monitor whether the hand movement matches with a command pattern corresponding to the preparation pattern.
Pinchon discloses during the sensing period, monitor whether the hand movement matches with a command pattern corresponding to the preparation pattern [see para. 0094; the user's hand position in relation to the origin point in environment. The system is making a comparison to determine a length of vector in the X/Y plane. The location and viewpoint of the user will change in the VR environment, where the speed of this movement proportional to the determined length of vector. The movement directly forward from the user's current view or angled according to a direction of the vector. The system can continue to update the viewpoint of the user (e.g., every threshold amount of time, such as every 0.1, 0.5, or 1 second), until the user makes the movement end gesture (e.g., releasing the pinched together figures on their hand). For example, the user can make the movement start gesture (e.g., a pinch gesture) at origin point and moves (e.g., drags) her hand to the hand position, causing the viewpoint to change, moving her forward in the artificial reality environment by an amount corresponding to the distance between the origin point and the new hand position and process continues to monitor the user's hand posture, the user performs the movement start gesture and movement end gesture with a single hand. The user performs the movement start gesture and the movement end gesture with two hands, the user separately controls forward/backward movement and the direction orientation the user is facing; which corresponds to compute a vector from the gesture to detect the precise command pattern and direction].
It would have been obvious to one of an ordinary skill in the art, having the teachings of Furtwangler and Pinchon before the affective filing date of the claimed invention to modify, an artificial reality application using hand gestures of Furtwangler to incorporate the origion point vector tracking and monitoring, as taught by Pinchon. One would have been motivated to make such a combination in order to provide precise control without physical controllers and improving accuracy.
Regarding claim 2 and 13, Furtwangler discloses wherein the processor is configured to track the hand gesture by: performing a computer vision algorithm to locate knuckle positions of a hand in the streaming images; and tracking the hand gesture according to the knuckle positions [see para. 0035; while the head-wearable device (which in this example is a virtual-reality headset) displays the representation of the user's position within the AR environment (e.g., the user's field of view), the head-wearable device receive an indication (e.g., from a wrist-wearable device that can be configured to detect in-air hand gestures or from an intermediary device facilitating interactions between the wrist-wearable device and the head-wearable device, and/or the head-wearable device can detect and process gestures, such as in-air hand gestures, on its own or in conjunction with the wrist-wearable device) that a positional-control activation hand gesture has been performed (e.g., an in-air hand gesture, such as a pinch gesture in which a user's thumb makes contact with a distal portion of the user's index finger). The head-wearable device, in response to receiving the indication that the positional-control activation hand gesture has been performed, displays a positional-control user interface (UI) overlaid on a portion of the AR environment; which corresponds to finger joint detection gesture].
Regarding claims 3 and 14, Furtwangler further comprising: transceiver, configured to communicate with a wearable device, wherein the wearable device comprise an inertial measurement unit for generating inertial measurement data [see para. 0240, 0241; a head-wearable device can include an IMU, and the intermediary processing device (a neckband and/or an HIPD) compute all inertial and spatial calculations from the IMUs located on the head-wearable device. Additional examples of processing performed by a communicatively coupled device].
Regarding claims 4 and 15, Furtwangler discloses wherein in response to the hand gesture matching with the preparation pattern, the processor is configured to transmit a triggering signal to the wearable device for activating an inertial measurement unit of the wearable device [see para. 0246; one or more devices configured to be part of the computing system, many of which have been defined above and/or described with respect to wrist-wearable devices. For example, the peripherals interface can include one or more sensors. Some example sensors include one or more coupling sensors, one or more acoustic sensors, one or more imaging sensors, one or more EMG sensors, one or more capacitive sensors, and/or one or more IMUs].
Regarding claim 5, Furtwangler discloses wherein in response to the hand gesture matching with the preparation pattern, the processor is configured to track the hand movement according to inertial measurement data received from the wearable device [see para. 0162; The user use any of the wrist-wearable device, the AR device, and/or the HIPD to provide user inputs. For example, the user perform one or more hand gestures that are detected by the wrist-wearable device (e.g., using one or more EMG sensors and/or IMUs) and/or AR device (e.g., using one or more image sensors or cameras) to provide a user input. Alternatively, the user provide a user input via one or more touch surfaces of the wrist-wearable device, the AR device, and/or the HIPD, and/or voice commands captured by a microphone of the wrist-wearable device].
Regarding claim 6, Furtwangler discloses wherein the wearable device comprises at least one smart ring wearable on at least one finger of a user [see para. 0240, 0265; The controller of the intermediary processing device processes information generated by the sensors on the intermediary processing device and/or the head-wearable devices. The intermediary processing device, such as an HIPD, process information generated by one or more of its sensors and/or information provided by other communicatively coupled devices. For example, a head-wearable device can include an IMU, and the intermediary processing device (a neckband and/or an HIPD) compute all inertial and spatial calculations from the IMUs located on the head-wearable device and the HIPD includes one or more sensors for sensing data used in the performance of one or more operations and/or functions. For example, the HIPD include an IMU that is used in conjunction with cameras for 3-dimensional object manipulation (e.g., enlarging, moving, destroying, etc. an object) in an AR or VR environment].
Regarding claim 7, Pinchon discloses wherein the processor is configured to track the hand movement by: performing a computer vision algorithm to locate knuckle positions of a
hand in the streaming images; and tracking the hand movement according to the knuckle positions [see para. 0068; a hand posture can comprise one or both of where the hand is in an artificial reality environment and/or the shape the hand is making. Certain hand postures can be previously identified as “gestures,” where the gesture is identified when the hand posture matches to a threshold amount. The processes described herein can monitor hand postures in different manners. In some cases, hand postures can be identified using input from external facing cameras that capture depictions of user hands. In other cases, hand postures can be based on input from a wearable device such as a glove or wristband that tracks aspects of the user's hands, input can be interpreted as postures mapped as certain gestures by applying the input to a machine learning model trained to identify hand postures and/or gestures based on such input. implementations In some, heuristics or rules can be used to analyze the input to identify hand postures and/or gestures].
Regarding claims 9 and 17, Furtwangler discloses wherein the processor deactivates the tracking of the hand movement in response to the hand gesture failing to match with the preparation pattern or in response to the sensing period being expired [see para 0318-0320; the positional-control activation hand gesture is a maintained gesture (e.g., a pinch) and the positional-control input hand gesture is release of the maintained gesture (e.g., a releasing pinch). The method includes displaying at least on additional positional-control UI element adjacent to the positional-control UI, the additional positional-control UI element configured to perform an additional positional-control action, additional positional-control UI elements can be displayed next to the positional-control UI. The method further includes, while displaying the positional-control UI, receiving an indication that a non-positional control input hand gesture is performed, preforming a non-positional-control action associated with the non-positional control input hand gesture, and continuing to display the positional-control UI. In other words, the user is able to continue to perform other non-position-based actions while the positional-control UI is displayed. Alternatively, in some embodiments, the non-positional control input hand gesture automatically causes the positional-control UI to no longer be displayed; which corresponds to the gesture is release no longer detected].
Regarding claims 10 and 18; Pinchon discloses wherein the preparation pattern comprises a clicking preparation pattern with at least one finger hovering diagonally upward in front of the head-mounted display device, the command pattern comprises a clicking command pattern with the at least one finger pressing or moving downward [see para. 0095, 0096; the user performs the movement start gesture and movement end gesture with a single hand. In other cases, the user performs the movement start gesture and the movement end gesture with two hands, the user separately controls forward/backward movement and the direction orientation the user is facing. The user controls rotation (e.g., turn-in-place by incremental degrees) with one hand in environment and controls forward/backward movement with their other hand in environment. The user can pinch and drag to specify a direction of movement can pinch and drag to specify orientation. The user controls forward/backward movement and the direction orientation the user is facing with the same hand, depending on which gesture is being held. The user controls forward/backward movement with a first gesture (e.g., pinch with index finger) in environment and controls rotation with a second gesture (e.g., pinch with middle finger) in environment].
Regarding claims 11 and 19, Pinchon discloses wherein the preparation pattern comprises a pinching preparation pattern with two fingers hovering with a space between the two fingers, the command pattern comprises a pinching command pattern with the two fingers moving toward each other [see para. 0094; The system can continue to update the viewpoint of the user (e.g., every threshold amount of time, such as every 0.1, 0.5, or 1 second), until the user makes the movement end gesture (e.g., releasing the pinched together figures on their hand). For example, the user can make the movement start gesture (e.g., a pinch gesture) at origin point and moves (e.g., drags) her hand to the hand position, causing the viewpoint to change, moving her forward in the artificial reality environment by an amount corresponding to the distance between the origin point and the new hand position. At hand position, the user makes the movement end gesture, and no further movement is made].
Regarding claim 12, Furtwangler discloses a command sensing method, comprising [see abstract; a head-wearable device worn by a user, a representation of a user's position within an artificial-reality environment, in response to receiving an indication that a positional-control activation hand gesture has been performed]:
capturing a plurality of streaming images [see para. 0262; a user can initiate a video stream using the AR device and back-end tasks associated with performing the video stream (e.g., video rendering) can be offloaded to the HIPD, which the HIPD performs and provides corresponding data to the AR device to perform remaining front-end tasks associated with the video stream (e.g., presenting the rendered video data via a display of the AR device); which corresponds to capture image data with HMD];
tracking a hand gesture according to the streaming images [see para. 0309; the positional-control input hand gesture is maintained and the positional-control action is based on movement a user's hand while the positional-control input hand gesture is maintained. The user's hand can be tracked and, while the gesture is maintained, the type of positional change is also adjusted. For example, the user can go from a walk to a sprint; from running/walking left to running/walking right; from looking left to looking right. Additionally, the maintained gesture can be tracked to perform multiple actions; which corresponds based on image data obtained];
monitoring whether the hand gesture matches with a preparation pattern [see para. 0318, 0319; the positional-control activation hand gesture is a maintained gesture (e.g., a pinch) and the positional-control input hand gesture is release of the maintained gesture (e.g., a releasing pinch) and he method includes displaying at least on additional positional-control UI element adjacent to the positional-control UI, the additional positional-control UI element configured to perform an additional positional-control action; which corresponds to position control activate pinching gestures];
in response to the hand gesture matching with the preparation pattern at a first time point, activate a tracking of a hand movement during a sensing period started from the first time point until a second time point [see para. 0087, 0088, 0323; 0325; At a first point in time, a positional-control activation hand gesture (e.g., a pinch gesture) is performed, which causes the head-wearable device to display the positional-control UI and the navigation UI element the positional-control activation hand gesture is performed by a first hand, the positional-control UI is a first positional-control UI, the positional-control UI element is a first positional-control UI element, the positional-control action is a first positional-control action, and the positional-control input hand gesture is a first positional-control input hand gesture. The method includes, while displaying the first positional-control UI, in response to receiving an indication that at a second point in time, the user, while holding the pinch gesture, moves their hand slightly. The slight hand movement of the user does not significantly change its position within the positional-control UI; which corresponds to the system activates the positional control user interface mode tracking active gesture is held and released];
and in response to the hand movement matching with the command pattern, execute an operation corresponding to the command pattern [see para. 0135, 0141; the positional-control input hand gesture is maintained and the positional-control action is based on movement a user's hand while the positional-control input hand gesture is maintained. In other words, the user 115's hand can be tracked such that a positional-control input hand gesture performs a different position-control action based on the positional-control input hand gesture's location relative to the positional-control UI. For example, based on the positional-control input hand gesture performed by the user, the user's representation within the AR environment transition from walking to running, looking left to looking right; etc. The user can maintain the positional-control input hand gesture (which is tracked) to perform multiple positional-control actions. For example, the user can perform positional-control input hand gestures to perform a running jump, a diagonal walk or run (e.g., movement in at least two directions), an interaction with an object while moving, a non-positional-control action in conjunction with a positional-control action (e.g., aim and shoot a rifle or throw an object while moving); which corresponds to execute based on the detected movement]; however, Furtwangler fails to explicitly teach during the sensing period, monitor whether the hand movement matches with a command pattern corresponding to the preparation pattern.
Pinchon discloses during the sensing period, monitor whether the hand movement matches with a command pattern corresponding to the preparation pattern [see para. 0094; the user's hand position in relation to the origin point in environment. The system is making a comparison to determine a length of vector in the X/Y plane. The location and viewpoint of the user will change in the VR environment, where the speed of this movement proportional to the determined length of vector. The movement directly forward from the user's current view or angled according to a direction of the vector. The system can continue to update the viewpoint of the user (e.g., every threshold amount of time, such as every 0.1, 0.5, or 1 second), until the user makes the movement end gesture (e.g., releasing the pinched together figures on their hand). For example, the user can make the movement start gesture (e.g., a pinch gesture) at origin point and moves (e.g., drags) her hand to the hand position, causing the viewpoint to change, moving her forward in the artificial reality environment by an amount corresponding to the distance between the origin point and the new hand position and process continues to monitor the user's hand posture, the user performs the movement start gesture and movement end gesture with a single hand. The user performs the movement start gesture and the movement end gesture with two hands, the user separately controls forward/backward movement and the direction orientation the user is facing; which corresponds to compute a vector from the gesture to detect the precise command pattern and direction].
It would have been obvious to one of an ordinary skill in the art, having the teachings of Furtwangler and Pinchon before the affective filing date of the claimed invention to modify, an artificial reality application using hand gestures of Furtwangler to incorporate the origin point vector tracking and monitoring, as taught by Pinchon. One would have been motivated to make such a combination in order to provide precise control without physical controllers and improving accuracy.
Regarding claims 13-15 and 17-19, directly or indirectly dependent on claim 12, essentially correspond to those of claims 2-5 and 9-11 respectively. Accordingly, the same reasoning as in claims 2-5 and 9-11 applies to claims 13-15 and 17-19.
Regarding claim 20 is an independent claim and relates to a non-transitory computer readable storage medium with a computer program to execute a command sensing method, wherein the command sensing method comprises: Since the features of claim 20 are substantially the same as those of claim 1 except for the category of invention, the same reasoning as in claim 1 applies to claim 20.
Allowable Subject Matter
Claims 8 and 16 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure (See PTO-892).
Stafford et al. (US 10,156,908) discloses a system for performing a pinch and hold gesture is described. The system includes a head-mounted display (HMD) and a glove, which is worn by a hand of the user. Each finger segment of the glove includes a sensor for detecting positions of the finger segment when moved by the hand. The system includes a computing device interfaced with the HMD and the glove. The computing device analyzes data from the sensors of the finger segments to determine that a pinch and hold gesture is performed by at least two of the finger segments. Moreover, the computing device generates image data that is communicated to the HMD, such that a scene rendered on the HMD is modified to render a visual cue indicative of a location in the scene at which the pinch and hold gesture is associated.
A reference to specific paragraphs, columns, pages, or figures in a cited prior art reference is not limited to preferred embodiments or any specific examples. It is well settled that a prior art reference, in its entirety, must be considered for all that it expressly teaches and fairly suggests to one having ordinary skill in the art. Stated differently, a prior art disclosure reading on a limitation of Applicant's claim cannot be ignored on the ground that other embodiments disclosed were instead cited. Therefore, the Examiner's citation to a specific portion of a single prior art reference is not intended to exclusively dictate, but rather, to demonstrate an exemplary disclosure commensurate with the specific limitations being addressed. In re Heck, 699 F.2d 1331, 1332-33,216 USPQ 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006,1009, 158 USPQ 275, 277 (CCPA 1968)). In re: Upsher-Smith Labs. v. Pamlab, LLC, 412 F.3d 1319, 1323, 75 USPQ2d 1213, 1215 (Fed. Cir. 2005); In re Fritch, 972 F.2d 1260, 1264, 23 USPQ2d 1780, 1782 (Fed. Cir. 1992); Merck & Co. v. Biocraft Labs., Inc., 874 F.2d 804, 807, 10 USPQ2d 1843, 1846 (Fed. Cir. 1989); In re Fracalossi, 681 F.2d 792,794 n.1,215 USPQ 569, 570 n.1 (CCPA 1982); In re Lamberti, 545 F.2d 747, 750, 192 USPQ 278, 280 (CCPA 1976); In re Bozek, 416 F.2d 1385, 1390, 163 USPQ 545, 549 (CCPA 1969).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CAO H NGUYEN whose telephone number is (571)272-4053. The examiner can normally be reached on Mon-Fri 9am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kieu Vu can be reached on 571-272-4057. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CAO H NGUYEN/Primary Examiner, Art Unit 2171
Prosecution Insights