DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
The Office acknowledges the amendment dated 28 January 2026, in which:
Claims 1-20 are currently pending.
Claims 1, 13, 15 and 20 are amended.
Response to Arguments/Amendments/Remarks
Applicant’s arguments with respect to claims 1-20 have been considered but are moot because the arguments do not apply to the combination of references used in the current rejection.
Regarding the 35 U.S.C. § 102 Rejection:
Applicant argues that Forutanpour fails to teach receiving images via a first protocol and receiving interaction data including movement information via a second, different protocol, noting that Forutanpour utilizes a single communication transceiver without protocol differentiation. The Examiner agrees that Forutanpour does not explicitly disclose the dual-protocol network architecture as newly claimed. Therefore, the 35 U.S.C. § 102 rejection has been withdrawn. However, because routing media and interaction data over different network protocols is a well-known standard in AR networking to manage latency, a new ground of rejection under 35 U.S.C. § 103 is necessitated and set forth below, citing Xiao (US 2019/0206133) as a secondary reference.
Regarding the 35 U.S.C. § 103 Rejection (Avatar Mirroring):
Applicant argues that the secondary reference, Andel (referred to by Applicant as "Van", U.S. 2018/0315247), only teaches a virtual guide limb that demonstrates target movements (e.g., an avatar the user attempts to follow), which is the inverse of an avatar limb that mirrors the user's actual movement in real-time.
The Examiner respectfully disagrees with this narrow characterization of the reference. While Andel does disclose a virtual guide limb for a user to follow, the specification explicitly discloses an avatar that mirrors the user's real-time physical movements as well. Specifically, Andel teaches:
"The user 605 may control movement of the avatar 615... The tracked movement may be used by the sports training AR system 600 to display the movement using a representation of the movement via the avatar 615 performing the movement virtually." (Andel, Para. [0052]).
Furthermore, Andel teaches generating representations based directly on the captured video of the movements the user actually performed:
"The display screen 205 may display a real-time representation 210 of the user 225, for example using the real time captured video of the user 225." (Andel, Para. [0036]), and
"The display 415 may show the captured video of the movements the user performed... as a virtual representation 430 of the user 405." (Andel, Para. [0044])
Therefore, the reference clearly teaches an avatar animating in response to and in correspondence with the user's actual limb movement in real-time. Accordingly, he rejection is maintained.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 14, 15 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Forutanpour et al. (US 2022/0405959, hereinafter “Forutanpour”) in view of Xiao (US 2019/0206133, hereinafter “Xiao”).
With respect to Claim 1 (Currently Amended), Forutanpour teaches a method comprising:
establishing, by one or more processors of an augmented reality (AR) device, a communication session with an external device (Forutanpour: Para. [0057], [0060], “imaging device… can make use of one or more data streams from one or more external devices… based on one or more transmissions… over a wireless network”);
receiving a communication from the external device indicating that a predetermined portion of a body of a user has been detected within view of a camera of the external device (Forutanpour: Para. [0060], the external device… can perform preliminary processing… hand detection and/or hand tracking… send data corresponding to preliminary processing to the imaging device); and
in response to receiving the communication, presenting an AR notification to inform the user that the AR device is ready to control a first AR object (Forutanpour: Para. [0007], [0060], generating virtual content based on the location of the object determined from the external device’s data meets the broadest reasonable interpretation (BRI) of this limitation as the continuous display or virtual content would necessarily inform the user how that virtual object is controlled).
Forutanpour does not explicitly disclose that the communication session includes a first communication channel using a first protocol for images/videos and a second communication channel using a second protocol different from the first protocol for interaction data.
However, Xiao teaches an AR system architecture that communicates with external devices using a first protocol (e.g., HTTP) to transmit display/media information and a second protocol different from the first protocol. (e.g., TCP/IP) to transmit position/interaction data (Xiao: Para. [0098] – [0100]).
Therefore, it would have been obvious to one of ordinary skill in the art to modify the communication session of Forutanpour to route the image/video data via a first protocol and the interaction/movement data via a second different protocol, as taught by Xiao, in order to achieve reasonable utilization of resources because interaction data requires strong timeliness (low latency) to ensure the AR objects react accurately to user movement, it is assigned a fast, reliable protocol, whereas standard display media is assigned a separate protocol suitable for bulk data transfer (Xiao: Para. [0098] – [0100]).
Regarding Claims 15 and 20, these claims recite a system and a non-transitory machine-readable storage medium comprising instructions for performing the method of Claim 1. They are rejected for the same reasons of obviousness as Claim 1 (Forutanpour: Para. [0152], [0170]).
With respect to Claim 2, the combination of Forutanpour as modified by Xiao teaches the method of claim 1, comprising:
causing, by the AR device, overlay of the first AR object on a real-world environment being viewed using the AR device (Forutanpour: Para. [0002], [0059]);
receiving, by the AR device, interaction data from the external device representing movement of the user determined by the external device (Forutanpour: Para. [0060]); and
in response to receiving the interaction data from the external device, modifying the first AR object by the AR device (Forutanpour: Para. [0006], [0059], “position a virtual object held by the user’s hand accurately”).
With respect to Claim 14, the combination of Forutanpour as modified by Xiao teaches the method of claim 1, wherein the external device is placed on a surface, and wherein the camera of the external device points towards the user who is using the AR device to capture a whole body of the user, wherein the external device generates interaction data based on one or more images captured by the camera of the external device (Forutanpour: Para. [0063], the external device can be a television or desktop computer monitor).
Claims 3-13 and 19 rejected under 35 U.S.C. 103 as being unpatentable over Forutanpour and Xiao, as applied to claims 1, 2, 14, 15 and 20 above, and further in view of Andel (US 2018/0315247).
With respect to Claim 3, the combination of Forutanpour as modified by Xiao teaches the method of claim 1, wherein the external device comprises an external vision system (Forutanpour: Para. [0060]).
Forutanpour fails to expressly disclose:
the method further comprising:
presenting by the external device one or more images depicting the user to enable the user to position the user in a way that allows the camera of the external device to capture images of an entire body of the user; and
presenting by the external device an indication on a screen in response to detecting by the external device the entire body of the user, the external device transmitting the communication in response to detecting by the external device the entire body of the user.
However, Andel discloses:
the method further comprising:
presenting by the external device one or more images depicting the user to enable the user to position the user in a way that allows the camera of the external device to capture images of an entire body of the user (Andel: Fig. 1A, 2, 4B); and
presenting by the external device an indication on a screen in response to detecting by the external device the entire body of the user, the external device transmitting the communication in response to detecting by the external device the entire body of the user (Andel: Fig. 1A, 2, 4B, Para. [0036], presenting captured video of the user on a display screen to allow the user to match movements or ensure proper form. This enables the user to position themselves for capture).
Therefore, it would be obvious to one of ordinary skill in the art to modify the method, as taught by Forutanpour and Xiao, to incorporate biomechanical feedback and avatar control, as taught by Andel, in order to provide visual and haptic feedback to help guide the user through prescribed movements with the AR system (Andel: Para. [0027]).
With respect to Claim 4, the combination of Forutanpour as modified by Xiao teaches the method of claim 1.
Forutanpour fails to expressly disclose wherein modifying the first AR object comprises animating an avatar.
However, Andel discloses:
wherein modifying the first AR object comprises animating an avatar (Andel: Para. [0030], displaying an avatar 120; Para. [0037], a virtual guide limb that demonstrates movements or mirrors the user).
Therefore, it would be obvious to one of ordinary skill in the art to modify the method, as taught by Forutanpour, to incorporate biomechanical feedback and avatar control, as taught by Andel, in order to provide visual and haptic feedback to help guide the user through prescribed movements with the AR system (Andel: Para. [0027]).
With respect to Claim 5, the combination of Forutanpour as modified by Andel teaches the method of claim 4, wherein animating the avatar comprises:
determining that interaction data comprises body movement information (Andel: Para. [0017], capturing three-dimensional movement of a target body part); and
generating a whole body avatar as the avatar in response to determining that the interaction data includes body movement information (Andel: Fig. 4B, Para. [0044], virtual representation 430 of the user 405).
With respect to Claim 6, the combination of Forutanpour as modified by Andel teaches the method of claim 5, further comprising:
updating one or more body parts of the avatar based on the body movement information received from the external device (Andel: Para. [0048], capturing movement of specific body parts and using this to update the virtual representation or animation).
With respect to Claim 7, the combination of Forutanpour as modified by Xiao teaches the method of claim 1.
the combination of Forutanpour as modified by Xiao teaches the external device has an external vision system, at Para. [0060]. Forutanpour teaches that the external device can be a mobile handset (Para. [0040], [0063]) coupled to the AR device (Para. [0058] – “when plugged into the imaging device”). While, Forutanpour does not restrict the software to a “messaging application,” it is well known in the art that mobile handsets utilize messaging applications that access the camera.
Andel teaches using the system for remote diagnosis and providing assessments via “electronic communications such as an email, text message” (Para. [0029]).
It would have been obvious to one of ordinary skill in the art to implement the external vision system with in a messaging application in order to facilitate the remote communications, as taught by Andel (Para. [0018] – “remote diagnosis or feedback”).
With respect to Claim 8, the combination of Forutanpour as modified by Andel teaches the method of claim 7, wherein the messaging application is configured to apply one or more machine learning models to one or more images depicting the user that have been captured by the mobile device to generate interaction data, wherein the user is using the AR device (Andel: Para. [0017], [0019], applying machine learning models or artificial intelligence to the capture images to generate data. Combining Forutanpour’s hardware architecture with Andel’s ML-based analysis renders this limitation obvious).
With respect to Claim 9, the combination of Forutanpour as modified by Xiao teaches the method of claim 1.
Forutanpour fails to expressly disclose:
wherein the first AR object comprises an avatar representing the user, further comprising:
determining that movement of the user comprises a fitness activity; and
animating the avatar to represent the fitness activity.
However, Andel discloses:
wherein the first AR object comprises an avatar representing the user (Andel: Fig. 1A, 2, 4A-B), further comprising:
determining that movement of the user comprises a fitness activity (Andel: Para. [0003], [0051]); and
animating the avatar to represent the fitness activity (Andel: Para. [0030], [0052]).
Therefore, it would be obvious to one of ordinary skill in the art to modify the method, as taught by Forutanpour, to incorporate biomechanical feedback and avatar control, as taught by Andel, in order to provide visual and haptic feedback to help guide the user through prescribed movements with the AR system (Andel: Para. [0027]).
With respect to Claim 10, the combination of Forutanpour as modified by Andel teaches the method of claim 9, further comprising:
determining, based on interaction data (Andel: Para. [0017], capturing movement of a limb), that a limb of a user is being moved in a determinable direction (Andel: Para. [0035], [0048], determining the direction/path of movement); and
in response to determining that the limb of the user is being moved in the determinable direction, animating a limb of the avatar to move along the determinable direction, the limb of the avatar corresponding to the limb of the user (Andel: Para. [0037], animating a limb of the avatar to correspond to the user’s limb).
With respect to Claim 11, the combination of Forutanpour as modified by Xiao teaches the method of claim 1. Forutanpour fails to expressly disclose:
determining, based on interaction data, that a specified limb of the user has been moved in a particular direction until reaching a certain threshold and held in that position for a threshold period of time, the certain threshold representing completion of a single repetition of a fitness activity;
updating the first AR object to mimic movement of the specified limb of the user;
computing a quantity of times that the specified limb has repeated been moved in the particular direction until reaching the certain threshold and held in that position for the threshold period of time; and
updating a counter that is presented on the AR device to represent the quantity of times.
However, Andel discloses:
determining, based on interaction data, that a specified limb of the user has been moved in a particular direction until reaching a certain threshold and held in that position for a threshold period of time, the certain threshold representing completion of a single repetition of a fitness activity (Andel: Para. [0040], [0049], providing haptic feedback when the target body part 510 has reached the predetermined position);
updating the first AR object to mimic movement of the specified limb of the user (Andel: Para. [0040], [0046]);
computing a quantity of times that the specified limb has repeated been moved in the particular direction until reaching the certain threshold and held in that position for the threshold period of time (Andel: Para. [0046]); and
updating a counter that is presented on the AR device to represent the quantity of times (Andel: Para. [0040], [0046], updating a counter presented to the user).
Therefore, it would be obvious to one of ordinary skill in the art to modify the method, as taught by Forutanpour, to incorporate biomechanical feedback and avatar control, as taught by Andel, in order to provide visual and haptic feedback to help guide the user through prescribed movements with the AR system (Andel: Para. [0027]).
With respect to Claim 12, the combination of Forutanpour as modified by Xiao and Andel teaches the method of claim 10, further comprising:
determining, by the AR device, that movement of the limb of the avatar corresponds to a completed iteration of the fitness activity (Andel: Para. [0048]); and
in response to determining by the AR device that movement of the limb of the avatar corresponds to the completed iteration of the fitness activity, updating a counter representing a quantity of completed iterations of the fitness activity (Andel: Para. [0046]).
With respect to Claim 13 (Currently Amended), the combination of Forutanpour as modified by Xiao teaches the method of claim 1, further comprising displaying, by the AR device, a representation of the counter together with the avatar (Andel: Para. [0036], [0046] – [0047]);
comprising:
determining that fitness activity associated with the interaction data comprises weightlifting in which a user holds a weight (Andel: Para. [0040]);
animating an arm of an avatar representing the user to move in correspondence with movement of an arm of the user (Andel: Para. [0044], [0052]); and
rendering, within lenses of the AR device, a virtual weight held by the avatar, the virtual weight corresponding to the weight held by the user (Andel: Para. [0040]).
Therefore, it would be obvious to one of ordinary skill in the art to modify the method, as taught by Forutanpour and Xiao, to incorporate biomechanical feedback and avatar control, as taught by Andel, in order to provide visual and haptic feedback to help guide the user through prescribed movements with the AR system (Andel: Para. [0027]).
With respect to Claim 19, the combination of Forutanpour as modified by Xiao teaches the system of claim 15.
Forutanpour fails to expressly disclose:
the operations comprising:
determining, based on interaction data, that a specified limb of the user has been moved in a particular direction until reaching a certain threshold and held in that position for a threshold period of time, the certain threshold representing completion of a single repetition of a fitness activity;
updating the first AR object to mimic movement of the specified limb of the user;
computing a quantity of times that the specified limb has repeated been moved in the particular direction until reaching the certain threshold and held in that position for the threshold period of time; and
updating a counter that is presented on the AR device to represent the quantity of times.
However, Andel discloses:
the operations comprising:
determining, based on interaction data, that a specified limb of the user has been moved in a particular direction until reaching a certain threshold and held in that position for a threshold period of time, the certain threshold representing completion of a single repetition of a fitness activity (Andel: Para. [0040], [0049], providing haptic feedback when the target body part 510 has reached the predetermined position);
updating the first AR object to mimic movement of the specified limb of the user (Andel: Para. [0040], [0046]);
computing a quantity of times that the specified limb has repeated been moved in the particular direction until reaching the certain threshold and held in that position for the threshold period of time (Andel: Para. [0046]); and
updating a counter that is presented on the AR device to represent the quantity of times (Andel: Para. [0040], [0046], updating a counter presented to the user).
Therefore, it would be obvious to one of ordinary skill in the art to modify the system, as taught by Forutanpour, to incorporate biomechanical feedback and avatar control, as taught by Andel, in order to provide visual and haptic feedback to help guide the user through prescribed movements with the AR system (Andel: Para. [0027]).
Claims 16-18 are rejected under 35 U.S.C. 103 as being unpatentable over Forutanpour and Xiao, as applied to claims 1, 2, 14, 15 and 20 above, and further in view of Perez (US 2010/0306716).
With respect to Claim 16, the combination of Forutanpour as modified by Xiao teaches the system of claim 15.
Forutanpour fails the specific game mechanics including the operations comprising:
presenting, as the first AR object, an avatar comprising a skier going downhill on a virtual mountain;
presenting additional virtual objects animated as approaching the avatar; and
moving a body of the avatar based on interaction data to avoid collision between the avatar and the additional virtual objects.
However, Perez discloses:
the operations comprising:
presenting, as the first AR object, an avatar comprising a skier going downhill on a virtual mountain (Perez: Fig. 6, Para. [0113]);
presenting additional virtual objects animated as approaching the avatar (Perez: Para. [0114]); and
moving a body of the avatar based on interaction data to avoid collision between the avatar and the additional virtual objects (Perez: Fig. 6, Para. [0113], [0117]).
Therefore, it would be obvious to one of ordinary skill in the art to modify the system, as taught by Forutanpour, to incorporate remappable gestures, as taught by Perez, in order to adapt the gesture detection to suit a particular user (Perez: Para. [0121] – [0124]).
With respect to Claim 17, the combination of Forutanpour as modified by Xiao and Perez teaches the system of claim 16, the operations comprising:
determining that the interaction data indicates that the user has leaned towards a direction by a certain amount of degrees (Perez: Fig. 7B, Para. [0113], [0117]); and
updating an orientation of the avatar to lean towards the direction by the certain amount of degrees (Perez: Fig. 6).
With respect to Claim 18, the combination of Forutanpour as modified by Xiao and Perez teaches the system of claim 16, the operations comprising updating a score by the AR device based on a quantity of the additional virtual objects with which the avatar has successfully avoided collision (Perez: Fig. 6, Para. [0032], Perez teaches the core mechanic: a skier avatar controlled by user leaning to avoid collision on a virtual mountain. Modifying Perez to count avoided objects instead of or in addition to distance or time traveled would be a simple substitution of one known scoring variable for another, yielding the predictable result of rewarding accuracy.).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BRYAN EARLES whose telephone number is (571)272-4628. The examiner can normally be reached on Monday - Thursday at 7:30am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, William Boddie can be reached on 571-272-0666. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BRYAN EARLES/Primary Examiner, Art Unit 2625
Prosecution Insights