DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment/ Arguments
Applicant’s amendments to the independent claims have been considered and, in response, the amendments do not respectively overcome the prior art of record. That is to say, the added claim features regarding rendering an avatar of the user (based on semantic intent directives and head pose) by the device of another user is taught by Bradski. See the paragraphs immediately following the subtitle entitled “Avatars in the Passable World”, paras. 598-605. In these paragraphs, Bradski teaches rendering avatars of users to mimic said users, and rendering these avatars for other users, as they all interact in virtual/augmented reality. Head pose of a user is one of the inputs that is collected and used to render these avatars (para. 599). Portions of these paragraphs are reproduced, with emphasis, in this office action.
Because of this teaching, the examiner disagrees with Applicant’s conclusory statement that Bradski does not teach the newly added claim features by amendment. The examiner also disagrees that Bradski does not teach or suggest “semantic intent directive”. Applicant’s argument (reproduced below):
PNG
media_image1.png
214
828
media_image1.png
Greyscale
Is, with respect, not accurate. The visual interest in an object is determined such to share with other users, in a multi-user environment. The intent to want to interact with an object is transmitted so that users can interact in a multi-user environment. This is multi-user interaction in a shared environment.
Otherwise, for Applicant’s arguments to be correct, Applicant is basically stating that, even though Bradski determines a user’s visual interest to interact with an object, Bradski does nothing with that determination, in multi-user interactive environment. Therefore, there actually isn’t any interaction, per Applicant’s arguments. This is respectfully factually incorrect. Bradski has many, many, many examples of determining visual interest/semantic intent, and actually using that determined intent to allow a user to interact with objects in a multi-user, interactive environment. See e.g. Fig. 143 for a flowchart, the section “Avatars in the Passable World”, as referenced above, and/or paras. 169, 175-78, 182-86.
The rejections are maintained.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be- negated by the manner in which the invention was made.
Claim(s) 19-38 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bradski (U.S. Patent App. Pub. No. 2019/0094981 A1).
Regarding claim 19:
Bradski teaches: a computer-implemented method for determining user intent (para. 17, methods for facilitating virtual and augmented reality interaction, interaction teaches “user intent”), comprising:
tracking head pose of a user, wherein tracking head pose of a user records movement data (para., 897, head pose processor to calculate near or real-time head pose, to do this, movement data needs to be recorded; additional teaching: para. 944-45, which teach averages of head position);
extracting, using the movement data and as extracted features, features in an environment of the user (e.g. para. 945, “current measures of head position…allows consistent rendering of objects on/around a user’s body.” The “extracted features” can features in the environment that allow for rendering of objects on/around a user’s body) (another example: para. 1159, head pose information can be used to determine object and listener pose – the object and listener pose are extracted features in the user environment);
performing eye tracking (para. 157, the AR device can have eye tracking cameras) to obtain a fixation point of eyes of the user in a field-of-view of the user (para. 815, eye tracking is used to track where a user is looking (fixation point), which is also the vergence of the eyes) in a head frame (The examiner is interpreting “in a head frame” two different ways, both of which are taught by Bradski:
Paragraphs 1004-1011 describe correlating the eye coordinates with world coordinates of a world camera, the “world camera” being cameras placed on a user’s head (head cameras). This result of this correlation teaches performing eye tracking “in a head frame”; OR
Paragraphs 1004-1011, the coordinates of the eye tracking are “in a head frame” by themselves, before correlation with world coordinates.
*Claim interpretation: either interpretation a broad, reasonable interpretation of what is meant by “in a head frame”, consistent with Applicant’s specification as filed, where “head frame” is described in paragraph [0202], as: “the head frame is associated with a coordinate system local to Alice”. The eye coordinate system of Bradski is a local coordinate system, so too is the eye coordinates correlated with world coordinates)
calculating, based on a combination of the fixation point of eyes of the user and the head pose of the user, an eye gaze target point (see para. 1006, which teaches that there are “three main components to gaze tracking: an eye tracking module (pupil detection and center of cornea detection), a head tracking module, and a correlation module that correlates the eye tracking module with the head tracking module.” The functions of the correlation module teach calculating an eye gaze target point (e.g. para. 1015,1029, a target point or position) based on a combination of fixation point and head pose, as claimed);
determining what virtual objects in the environment of the user intersect with a gaze fixation point in local space, wherein the gaze fixation point is based on the calculation of the eye gaze target point (see e.g. paras. 1015, 1018, which teaches a ‘gaze line’ whereby a gaze fixation point can be any point along a determined gaze line, for example. See also the section entitled “Gaze Tracking Hardware” beginning at para. 1019 and Figs 119-122);
determining that the gaze fixation point intersects with a virtual object of the virtual objects (e.g. para. 1002-04, which teaches tracking whether a user’s eye is directed at one or more virtual objects. Applying this to gaze fixation point, related to eye tracking, would have been obvious and predictable to one of ordinary skill);
extracting semantic intent directives associated with the virtual object (para. 1003, virtual object(-s) a user is looking at “may further allow the system to understand a user's interest in a particular virtual or real object”. Indications of user interest teaches “semantic intent directives” associated with the virtual object. Another example of extracting semantic intent directives is by treating the gaze as user interaction. See para. 1001, 1172, 1391); and
communicating the semantic intent directives associated with the virtual object and the head pose of the user to a wearable computing device of another user for rendering an avatar of the user by the wearable computing device of another user based on the semantic input directives (in the paragraphs immediately following the subtitle entitled “Avatars in the Passable World”, Bradski teaches the above communicating step. See e.g. paras. 598-605, portions of said paras. reproduced below for convenience (emphasis added):
Avatars in the Passable World
[0598] As an extension of the passable world model, not only objects are recognized, but other users/people of the real world may be recognized and may be rendered as virtual objects. For example, as discussed above, a friend may be rendered as an avatar at the AR system of the first user.
[0599] In some implementations, in order to render an avatar that properly mimics the user, the user may train the AR system, for example by moving through a desired or prescribed set of movements…. In one or more embodiments, the AR system knows where the pose of the user's head, eyes, and/or hands based on data captured by various sensors of his/her individual AR system.
[0601 ] In one or more embodiments, the passable world may also contain information about various avatars inhabiting a space. It should be appreciated that every user may be rendered as an avatar in one embodiment. Or, a user operating an individual AR system from a remote location can create an avatar and digitally occupy a particular space as well. In either case, since the passable world is not a static data structure, but rather constantly receives information, avatar rendering and remote presence of users into a space may be based on the user's interaction with the user's individual AR system.
[0602] More particularly, the user's individual AR system contains information about the user's head pose and orientation in a space, information about hand movement etc. of the user, information about the user's eyes and eye gaze, information about any totems that are being used by the user. Thus, the user's individual AR system already holds a lot of information about the user's interaction within a particular space that is transmitted to the passable world model. This information may then be reliably used to create avatars for the user and help the avatar communicate with other avatars or users of that space.
See also, for more teaching: e.g. paras. 175-83, which describe systems and methods in AR and VR whereby one or more users can control, manipulate or alter virtual objects (para. 178) with simultaneous user interaction (paras. 180-86). As noted, in the above example paragraphs, “first user” and “friend” of the first user, or really any combination of the multiple users that Bradski teaches can be interacting via avatars that mimic each user, including head pose, correspond to Applicant’s claimed user and another user, and vice versa.
It would have been obvious for one of ordinary skill in the art to have further modified the applied reference(-s), in view of same, to have obtained the above, and the results of the modification would have been obvious and predictable to one of ordinary skill in the art as of the effective filing date of the claimed invention. See MPEP §2143(A).
The prior art included each element recited in claim 19, although not necessarily in a single embodiment, with the only difference being between the claimed element and the prior art being the lack of actual combination of certain elements in a single prior art embodiment, as described above.
One of ordinary skill in the art could have combined the elements as claimed by known methods, and in that combination, each element merely performs the same function as it does separately. One of ordinary skill in the art would have also recognized that the results of the combination were predictable as of the effective filing date of the claimed invention.
Regarding claim 20:
Bradski teaches: the computer-implemented method of claim 19, wherein tracking head pose of a user is performed using a wearable computing device of the user (Fig. 3: 30, HMD wearable device) comprising a combination of an inertial measurement unit (IMU) (claim 1, the HMD can have an IMU, used to detect movements of the HMD, including initiating data capture) and a camera (para. 1013, head camera (world cameras)) to record rotational and translational motion of the user (paras. 1011-1015, similar to eye tracking, rotational and translational head movement can be obtained in relation to the world/head camera;
also, even without the rotational and translational vectors, rotational and translational motion (movement that includes and does not include changes in orientation, is recorded as a result of head motion from the world camera).
It would have been obvious for one of ordinary skill in the art, as of the effective filing date of Applicant’s claims, to have further modified the applied reference(-s) in view of same to have obtained the above, motivated to facilitate user movement tracking to discern interaction with a system.
Regarding claim 21:
Bradski teaches: the computer-implemented method of claim 19, comprising, estimating, using the extracted features, the head pose of the user with respect to a world frame associated with the environment of the user (e.g. paras. 1004-1011, head tracking or head pose is estimated with respect to “world coordinates” (para. 1006), or a world frame as claimed).
It would have been obvious for one of ordinary skill in the art, as of the effective filing date of Applicant’s claims, to have further modified the applied reference(-s) in view of same to have obtained the above, motivated to facilitate user movement tracking to discern interaction with a system.
Regarding claim 22:
Bradski teaches: the computer-implemented method of claim 19, wherein the head frame is associated with a coordinate system local to the user (see mapping to claim 1 and e.g. paras. 1004-11, head frame is associated with the world coordinate system and/or eye coordinate system, both local to a user).
It would have been obvious for one of ordinary skill in the art, as of the effective filing date of Applicant’s claims, to have further modified the applied reference(-s) in view of same to have obtained the above, motivated to facilitate user movement tracking to discern interaction with a system.
Regarding claim 23:
Bradski teaches: the computer-implemented method of claim 19, wherein determining what virtual objects in the environment of the user intersect with the eye gaze target point in local space includes use of static virtual scene models (e.g. para. 190-92, 203, static aspects of a digital world, or para. 931, a static 3D background) in a world frame associated with the environment of the user (para. 190, 203, a digital world is a world frame associated with the environment of the user) and dynamic virtual objects in the environment of the user (paras. 185, 190, 191, all of which teach dynamic virtual objects) (the examiner’s interpretation of “static virtual scene models” is a broad, reasonable interpretation consistent with Applicant’s specification as filed; see specification, para. 204, which states that “ static virtual scene models in the world frame 2140b (which may describe static virtual objects in the environment A)“) .
It would have been obvious for one of ordinary skill in the art, as of the effective filing date of Applicant’s claims, to have further modified the applied reference(-s) in view of same to have obtained the above, motivated to facilitate virtual scene rendering that accounts for various scene changes and/or interactivity.
Regarding claim 24:
Bradski teaches: the computer-implemented method of claim 19, comprising calculating the gaze fixation point with respect to a world frame associated with the environment of the user and a ray direction in the world frame (e.g. paras. 1015, 1018, which teaches a ‘gaze line’ whereby a gaze fixation point can be any point along a determined gaze line, for example. See also the section entitled “Gaze Tracking Hardware” beginning at para. 1019 and Figs 119-122. The gaze line teaches a ray direction in the world frame).
It would have been obvious for one of ordinary skill in the art, as of the effective filing date of Applicant’s claims, to have further modified the applied reference(-s) in view of same to have obtained the above, motivated to facilitate user movement tracking to discern interaction with a system.
Regarding claim 25:
Bradski teaches: the computer-implemented method of claim 19, wherein the virtual object intersected by the gaze fixation point is classified as an object of interest (e.g. paras. 10, 1033, 1722, objects a user is looking at can be considered an “object of interest”. Applying this to gaze fixation point, as mapped in claim 19, would have been obvious and predictable to one of ordinary skill in the art as of the effective filing date of the claimed invention. See MPEP §2143(A)).
One of ordinary skill in the art could have combined the elements as claimed by known methods, and in that combination, each element merely performs the same function as it does separately. One of ordinary skill in the art would have also recognized that the results of the combination were predictable as of the effective filing date of the claimed invention.
Regarding claim 26: see also claim 19.
Bradski teaches: a non-transitory, computer-readable medium storing one or more instructions executable by a computer system to perform one or more operations (para. 194, memory storing executable code that is local to a computing device, such as Fig. 1: 12), comprising:
The operations correspond to the method of claim 19; the same rationale for rejection applies.
Regarding claim 27: see also claim 20.
These claims are similar; the same rationale for rejection applies.
Regarding claim 28: see also claim 21.
These claims are similar; the same rationale for rejection applies.
Regarding claim 29: see also claim 22.
These claims are similar; the same rationale for rejection applies.
Regarding claim 30: see also claim 23.
These claims are similar; the same rationale for rejection applies.
Regarding claim 31: see also claim 24.
These claims are similar; the same rationale for rejection applies.
Regarding claim 32: see also claim 25.
These claims are similar; the same rationale for rejection applies.
Regarding claim 33: see also claim 19.
Bradski teaches: a computer-implemented system (Fig. 1: 10 system), comprising: one or more computers (Fig. 1: 12 devices ); and one or more computer memory devices interoperably coupled with the one or more computers (para. 185, 189-190, the devices can be interoperaby coupled with have memory local to Fig. 1: 12, 14 or 11)) and having tangible, non-transitory, machine-readable media storing one or more instructions that, when executed by the one or more computers (para. 194, the devices can have executable code stored in memory on the device), perform one or more operations, comprising:
The operations correspond to the method of claim 19; the same rationale for rejection applies.
Regarding claim 34: see also claim 20.
These claims are similar; the same rationale for rejection applies.
.
Regarding claim 35: see also claim 21.
These claims are similar; the same rationale for rejection applies.
Regarding claim 36: see also claim 22.
These claims are similar; the same rationale for rejection applies.
Regarding claim 37: see also claim 23.
These claims are similar; the same rationale for rejection applies.
Regarding claim 38: see also claim 24.
These claims are similar; the same rationale for rejection applies.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
U.S. 20160328874A1 Avatar facial expression animations with head rotation
In embodiments, an apparatus may include an avatar animation engine configured to receive a plurality of facial motion parameters and a plurality of head gestures parameters, respectively associated with a face and a head of a user.
The plurality of facial motion parameters may depict facial action movements of the face, and the plurality of head gesture parameters may depict head pose gestures of the head.
Further, the avatar animation engine may be configured to drive an avatar model with facial and skeleton animations to animate an avatar, using the facial motion parameters and the head gestures parameters, to replicate a facial expression of the user on the avatar that includes impact of head post rotation of the user.
* * * * *
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
* * * * *
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Sarah Lhymn whose telephone number is (571)270-0632. The examiner can normally be reached M-F, 9:00 AM to 6:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached at 571-272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
Sarah Lhymn
Primary Examiner
Art Unit 2613
/Sarah Lhymn/Primary Examiner, Art Unit 2613
Prosecution Insights