DETAILED ACTION
This Office Action is in response to the Applicants' communication filed on September 17, 2026, which amends the independent claim 1, amends the dependent claims 2-4, add a new independent claim 5, and presents arguments, is hereby acknowledged. Claims 1-5 are currently pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicant’s arguments filed on September 17, 2026, have been fully considered.
Applicant argues that by this response, claim 1 has been amended to overcome the 112 (f) interpretation.
Examiner replies that the 112 (f) interpretation for claims 1-4 has been removed.
Applicant argues that by this response, the independent claim 1 is hereby amended to add a new limitation “wherein the acquisition unit processing circuitry is configured to be able to selectively receive, from a first user terminal of a first user, first video data in which a first portion of a body of the first user is shown within video data obtained by photographing the first user in a plurality of different directions without receiving, from the first user terminal, second video data in which a second portion different from the first portion of the body of the first user is shown during a first period” in order to overcome the 35 U.S.C. §103 rejection.
Examiner replies that the amended claims with new limitations may overcome the cited portions of the prior arts. However, with more explicit mapping to the amended element, a newly found art, Garrido, etc. (US 20180089880 A1) teaches that wherein the acquisition unit processing circuitry is configured to be able to selectively receive, from a first user terminal of a first user, first video data in which a first portion of a body of the first user is shown within video data obtained by photographing the first user in a plurality of different directions without receiving, from the first user terminal, second video data in which a second portion different from the first portion of the body of the first user is shown during a first period (See Garrido: Figs. 1-2, and [0015], “This disclosure pertains to systems, methods, and computer readable media for avatar based video communication between multiple online users. In general, the source device transmits avatar data describing the characteristics of the sending user to the receiving device in real-time. The avatar data may then be provisioned as an avatar at the receiving end to mimic the sending user's characteristics. The avatar data may be transmitted in addition to video feed, or alternatively, the communication of the avatar data may replace the transmission of the video feed”; [0024], “The data transmission management (DTM) unit 107 manages transmission of data from the source device 110 to the receiving device 130. In an embodiment, the DTM unit 107 may transmit the avatar data 122 in addition to the video stream 121 and the audio stream 123. In alternative embodiments, the communication of the avatar data 122 may replace the communication of the video stream 121. In an embodiment, the DTM unit 107 may categorize the received avatar data based on their importance and priority. Subsequently, the DTM unit 107 can decide which data to send to the receiving device based on its priority and factors such as the availability of resources (e.g. bandwidth) and user configurations”; [0033], “In an embodiment, the sending user 211 may select a choice of avatar that the captured video is rendered on. As a result, the source device receives a selection of a choice of avatar from the sending user 211. In other embodiments, the sending user 211 may select to keep a portion of the original video but modify the other parts. For example, the sending user 211 may make a selection of a virtual clothes to be displayed instead of his/her original clothing. Other selections may include, but not limited to, hairstyle, skin tones, beard/moustache, glasses, hats, and etc. In yet another embodiment, the choice of avatar may be communicated by the receiving user. In an embodiment, the avatar may be selected from categories such as animal avatars, cartoon characters, or other virtual creatures”; [0043], “In an embodiment, at block 302, the avatar data is categorized based on their importance. In context of modeling information, as explained previously, different features of the sending user captured by the video may be grouped together based on the features' importance in generating a descriptive model. Initially, when no model has been formed on the receiving end, some of the modeling information may be prioritized over some of the tracking information. As the user's model is developed on the receiving end, the prioritization may change to allow more of the tracking information be transmitted to the receiving device. In an embodiment, some of the avatar data transmitted to the receiving device could be used both to create a model and provide indications as to movements or expressions of the sending user”. Note that avatar data is categorized into groups with the first group having more unique higher priority data, and the source devices selectively transmits at least the first group based on available bandwidth, the full video stream may be replaced by the selective avatar data and only necessary information is transmitted, these features are mapped to the current cited limitations). The remaining arguments are mooted in view of the newly found art.
Examiner respectfully further replies that the Applicant's arguments have been fully considered and a new ground of rejections have been made. Accordingly, new grounds of rejection are set forth below. Since the new grounds of rejection are necessitated by Applicant's amendments to the claims, the present action is made final.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5 are rejected under 35 U.S.C. 103 as being unpatentable over Quinn, etc. (US 20170039752 A1) in view of Leahy, etc. (US 20090177980 A1), further in view of Garrido, etc. (US 20180089880 A1).
Regarding claim 1, Quinn teaches that a virtual space presenting device for presenting a three-dimensional virtual space shared by a plurality of users to each user, the virtual space presenting device being located at a separate location from user terminals of the plurality of users, the virtual space presenting device (See Quinn: Fig. 2, and [0031], "FIG. 2 illustrates an embodiment of a system 10 including hardware and software components for automatically generating a facial avatar resembling a user in a defined art style. An example embodiment of the capture device 20 is configured to capture video having a depth image that may include depth values via any suitable technique including, for example, time-of-flight, structured light, or the like. According to one embodiment, the capture device 20 may organize the calculated depth information into "Z layers," or layers that may be perpendicular to a Z axis extending from the depth camera along its line of sight. X and Y axes may be defined as being perpendicular to the Z axis. The Y axis may be vertical and the X axis may be horizontal. Together, the X, Y and Z axes define the 3-D real world space captured by capture device 20"; and [0022], "Technology is described for automatically generating a facial avatar in a defined art style which resembles a user based on image data captured of the user and a predetermined set of transferable head (including facial) features associated with the defined art style. In embodiments discussed below, human facial features can be represented in one or more shape units associated with a head model. As described further below, depth and image data of a user's head features including facial features can be captured. Typically, a point cloud of points having 3D Cartesian or polar coordinates with respect to an origin of a coordinate system can be used to represent the head and facial features by facial modelling software. A mesh, e.g. a triangular or polygon mesh, may be used to connect or fit the points of the point cloud") comprising:
processing circuitry configured to:
acquire video data obtained by photographing users (See Quinn: Figs. 1-3, and [0032], "As shown in FIG. 2, this exemplary capture device 20 may include an image and depth camera component 22 which captures a depth image by a pixelated sensor array 26. A depth value may be associated with each captured pixel. Some examples of a depth value are a length or distance in, for example, centimeters, millimeters, or the like of an object in the captured scene from the camera. Sensor array pixel 28 is a representative example of a pixel with subpixel sensors sensitive to RGB visible light plus an IR sensor for determining a depth value for pixel 28. Other arrangements of depth sensitive and visible light sensors may be used. An infrared (IR) illumination component 24 emits an infrared light onto the scene, and the IR sensors detect the backscattered light from the surface of one or more targets and objects in the scene in the field of view of the sensor array 26 from which a depth map of the scene can be created. In some examples, time-of-flight analysis based on intensity or phase of IR light received at the sensors may be used for making depth determinations"; and [0047], "The 3D capture device 20 is continuously capturing depth image data of the user, for example at a rate of 30 frames a second. From this data, facial expression changes of the user can be tracked. The 3D facial recognition and reconstruction engine 116 can determine changes in measurements of a set of facial features periodically in accordance with user 3D facial expression metrics 146 for tracking facial changes. Facial action criteria for determined whether the 3D action measurements indicate a change in a user facial feature may be stored in the user 3D facial expression metrics 146. These changes in measurements, in some examples, may be stored as action unit coefficients for action unit meshes or sub-meshes which may be blended into one or more shape units for representing expression changes of a use r's face. For transferable expressions, the action unit coefficients for the sub-meshes are mapped to corresponding action units for sub-meshes of corresponding shape units of an avatar character model. As described below, in some examples, an avatar in a defined art style may have rules, like avatar character(s) facial expression definitions and rules 154, which indicate a set of expressions an avatar can make and for which of those expressions, are user expression data transferable. For example, the 3D facial recognition and reconstruction engine 116 can include pattern recognition software which recognizes facial expressions such as a smile or a sigh based on audiovisual capture of the user as well as structures like teeth and a mouth. Details of how a user smiles may include closed lip smile or open mouth smile and may be represented in data as action unit coefficients. In some embodiments, rules for an avatar defined art style like avatar character(s) facial expression definitions and rules 154 may identify facial features of an avatar rather than expressions, which facial features move like the facial features for resembling the user's expressions more faithfully. As part of an initialization process, a user may be prompted for expressions an avatar in a defined art style can make to capture reference image data". Note that the images are continuously captured, such as 30 fps video, which is mapped to acquire video);
generate avatars to be arranged in the virtual space in correspondence with the users on the basis of the video data of the users (See Quinn: Figs. 1A-B, and [0030], "Before proceeding to discuss in more detail hardware and software components for a system for generating a facial avatar resembling a user in a defined art style, FIG. 1B illustrates a flowchart of an overview of a method embodiment for automatically generating a facial avatar resembling a user in a defined art style. In step 101, a 3D image capture device (e.g. 20) captures 3D image data of head features including facial features of a user, and in step 102, one or more software controlled processors of one or more computer systems communicatively coupled to the capture device generate a user 3D head model for the user based on the 3D image data. In step 103, the one or more processors identify a set of transferable user head features from the user 3D head model which can be represented in a facial avatar of a defined art style based on transferable head feature rules for the defined art style, and in step 104, the one or more processors identify a set of avatar character head features to re present the set of user head features based on rules for the defined art style. In step 105, the one or more processors generate a facial avatar in the defined art style including the set of avatar character head features and cause a communicatively coupled display to display the facial avatar in step 106. Additionally, a facial expression of the facial avatar may be updated and displayed responsive to changes detected in user facial features"); and
generate and present videos corresponding to (See Quinn: Figs. 1A-B, and [0030], "In step 105, the one or more processors generate a facial avatar in the defined art style including the set of avatar character head features and cause a communicatively coupled display to display the facial avatar in step 106. Additionally, a facial expression of the facial avatar may be updated and displayed responsive to changes detected in user facial features"; and [0050], "Accessible avatar character facial expression definitions and rules 154 may also be accessed for identifying expressions which the avatar character can make in general and those in which the avatar mimicks from a user, e.g. user transferable expressions. Data, for example action unit coefficients, may be stored for representing the expression by one or more shape units in an avatar head model. There may be rules governing some expressions an avatar does not make such as smiling or how long an expression lasts. Some examples of expressions which may be represented by action unit coefficients include puckered lips, puckered lips with ballooned cheeks, left mount corner down, closed mouth smile, an open mouthed smile, wide open mouth, blinking of an eye, brow squeeze, raised left brow, and raised right brow") visual fields from virtual viewpoints of the users set in the virtual space with respect to the users,
wherein the processing circuitry is configured to be able to selectively receive, from a first user terminal of a first user, first video data in which a first portion of a body of the first user is shown within video data obtained by photographing the first user in a plurality of different directions without receiving, from the first user terminal, second video data in which a second portion different from the first portion of the body of the first user is shown during a first period (See Quinn: Figs. 1A-B, and [0033], "According to another embodiment, the capture device 20 may include two or more physically separated cameras that may view a scene from different angles, to obtain visual stereo data that may be resolved to generate depth information"; [0035], "In the example embodiment, the capture device 20 may include a processor 32 in communication with the image and depth camera component 22 and having access to a memory component 34 that may store instructions for execution by the processor 32 as well as images or frames of images captured and perhaps processed by the 3D camera. The memory component 34 may include random access memory (RAM), read only memory (ROM), cache, Flash memory, a hard disk, or any other suitable storage component. The processor 32 may also perform image processing, including some object recognition steps, and formatting of the captured image data"; and [0040], "An application programming interface (API) 125 provides an interface for one or more multimedia applications 128 to register avatar characters and data definitions and rules for them associated with the applications. Data sources 132 may store these avatar data definitions and rules. Besides user specific data like personal identifying information including personally identifying image data, user profile data 130 may also store data or data references to stored locations of a user 3D head model generated for the user and avatars, including their facial aspects, gene rated for the user". Note that the depth and stereo images are continuously captured from different angles with inherent partial visibility), and
wherein, when the processing circuitry receives the first video data without receiving the second video data during the first period, the processing circuitry generates the first portion of a first avatar corresponding to the first user for the first period on the basis of the first video data received during the first period (See Quinn: Figs. 3A-B and 4A-B, and [0040], "An application programming interface (API) 125 provides an interface for one or more multimedia applications 128 to register avatar characters and data definitions and rules for them associated with the applications. Data sources 132 may store these avatar data definitions and rules. Besides user specific data like personal identifying information including personally identifying image data, user profile data 130 may also store data or data references to stored locations of a user 3D head model generated for the user and avatars, including their facial aspects, generated for the user."; [0042], "FIGS. 4A and 4B illustrate some examples of shape units of a 3D head model. As mentioned above, a shape unit is a collection of 3D points which represents a head feature or an aspect of a head feature. In the illustrated examples, the collection of 3D points is a set of vertices or connection points in a 3D head mesh which may also include a three dimensional (3D) rig which models human bone structure and bone movement under a skin surface which skin surface is represented by the mesh"; and Fig. 7, and [0064], "] FIG. 7 illustrates a flowchart of a process example of updating a facial expression of the facial avatar responsive to changes detected in user facial features. In step 302, 3D action measurements of a set of facial features are periodically determined based on stored 3D facial expression metrics (e.g. 146) for tracking facial changes based on periodically captured 3D image data of user head features. In step 304, it is determined whether the 3D action measurements indicate a change in a user facial feature satisfying facial action criteria. If not, in step 306, control is returned for other processing until the next time for the periodic performance of step 302". Note that the current visible features and historical shape features for completing other portions of the avatar generation, dynamic updating blends real-time changes with stored based avatar model, and no second period video is needed, are mapped to "the generation unit generates the first portion of a first avatar corresponding to the first user for the first period on the basis of the first video data acquired during the first period", also the current real-time is mapped to the first period, and historical data is mapped to the second period), and
generates the second portion of the first avatar for the first period on the basis of the second video data received during a second period earlier than the first period (See Quinn: Figs. 1A-B, and [0023], "Differences between the points or vertices in the mesh and a reference human head model can be identified and stored as coefficients of 3D space vectors. In some examples, the coefficients values may be between O and 1, where a 1 means an exact correspondence between a vertex in the fitted mesh and the corresponding vertex in the human head reference model. A shape unit is a collection of these 3D vertices which represents a head feature or an aspect of a head feature. If not just the boundary points of a shape unit but also the points on the interior of the shape unit only differ in scale from the human head reference model, then a scalar coefficient value between O and 1 may be applied to all the points in the shape unit at the same time, simplifying transformation operations for generating a 3D head model of the user's head and face. The more varied and detailed the collection of human head reference models stored, the more likely scalar coefficient operations on a shape unit may be performed"; [0035], "In the example embodiment, the capture device 20 may include a processor 32 in communication with the image and depth camera component 22 and having access to a memory component 34 that may store instructions for execution by the processor 32 as well as images or frames of images captured and perhaps processed by the 3D camera. The memory component 34 may include random access memory (RAM), read only memory (ROM), cache, Flash memory, a hard disk, or any other suitable storage component. The processor 32 may also perform image processing, including some object recognition steps, and formatting of the captured image data"; and Figs. 3A-B, and [0048], "FIGS. 3B and 3C illustrate some examples of data which may be stored in an embodiment of user 3D head model data 144. FIG. 3B illustrates an example including 3D measurements for head features 147 relating head features by distance, angle and proportion and 3D measurements for facial expressions 148. FIG. 3C illustrates an example in which such 3D measurements 147 are embodied as coefficients for 3D vertices in each shape unit N 145, and the 0.3D measurements of changes in facial features for one or more expressions are embodied as 3D action unit coefficient measurements for facial expressions 149". Note that image and data stored and captured before the current time is mapped to the second period data).
However, Quinn fails to explicitly disclose that presenting a three-dimensional virtual space shared by a plurality of users to each user; the virtual space presenting device being located at a separate location from user terminals of the plurality of users; visual fields from virtual viewpoints of the users set in the virtual space with respect to the users; and wherein the processing circuitry is configured to be able to selectively receive, from a first user terminal of a first user.
However, Leahy teaches that presenting a three-dimensional virtual space shared by a plurality of users to each user (See Leahy: Fig. 1, and [0018], "Each user is free to move his or her avatar around in the virtual world. In order that each user see the correct location of each of the other avatars, each client machine sends its current location, or changes in its current location, to the server and receives updated position information of the other clients"; and [0019], "While FIG. 1 shows two avatars (and implies a third), typically many more avatars will be present. A typical virtual world will also be more complex than a single room. The virtual world view shown in FIG. 1 is part of a virtual world of several rooms and connecting hallways as indicated in a world map panel 19, and may include hundreds or users and their avatars. So that the virtual world is scalable to a large number of clients, the virtual world server must be much more discriminating as to what data is provided to each client. In the example of FIG. 1, although a status panel 17 indicates that six other avatars are present, many other avatars are in the room, but are filtered out for crowd control");
the virtual space presenting device being located at a separate location from user terminals of the plurality of users (See Leahy: Fig. 3, and [0023], "Referring now to FIG. 3, a block diagram is shown of a world system 54 in which a user A, at a first client system 60 (client A), interacts with a user B at a second client system 60 (client B) via a server 61. Client system 60 includes several databases, some of which are fixed and some of which are modifiable. Client system 60 also includes storage for program routines. Mechanisms for storing, reading and modifying data on computers such as client system 60 are well known in the art, as are methods and means for executing programs and displaying graphical results thereof. One such program executed by client system 60 is a graphical rendering engine which generates the user's view of the virtual world". Note that Client A and Client B and server 61 are separated at different location and may be mobile which dynamically changing locations).; and
visual fields from virtual viewpoints of the users set in the virtual space with respect to the users (See Leahy: Fig. 1, and [0017], "FIG. 1 is an illustration of a client screen display 10 seen by one user in the chat system. Screen display 10 is shown with several stationary objects (wall, floor, ceiling and clickable object 13) and two "avatars" 18. Each avatar 18 is a three-dimensional figure chosen by a user to represent the user in the virtual world. Each avatar 18 optionally includes a label chosen by the user. In this example, two users are shown: "Paula" and "Ken", who have chosen the "robot" avatar and the penguin avatar, respectively. Each user interacts with a client machine (not shown) which produces a display similar to screen display 10, but from the perspective of the avatar for that client/user. Screen display 10 is the view from the perspective of a third user, D, whose avatar is not shown since D's avatar is not within D's own view. Typically, a user cannot see his or her own avatar unless the chat system allows "our of body" viewing or the avatar's image is reflected in a mirrored object in the virtual world").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Quinn to have presenting a three-dimensional virtual space shared by a plurality of users to each user; the virtual space presenting device being located at a separate location from user terminals of the plurality of users; and visual fields from virtual viewpoints of the users set in the virtual space with respect to the users as taught by Leahy in order to provide graphical interaction between the users operating the client systems (See Leahy: Fig. 1, and [0007], "Therefore, what is needed is an efficient system for communication between many client systems over dedicated or open networks to provide graphical interaction between users operating the client systems"). Quinn teaches a method and system that may generate avatars from video in real-time; while Leahy teaches a system and method that may establish multi-user shared 3D virtual space architecture and per-user viewpoint rendering in order to provide interactions between users in the virtual space. Therefore, it is obvious to one of ordinary skill in the art to modify Quinn by Leahy to present the virtual space for sharing among multiple users. The motivation to modify Quinn by Leahy is "Use of known technique to improve similar devices (methods, or products) in the same way".
However, Quinn, modified by Leahy, fails to explicitly disclose that wherein the processing circuitry is configured to be able to selectively receive, from a first user terminal of a first user.
However, Garrido teaches that wherein the processing circuitry is configured to be able to selectively receive, from a first user terminal of a first user (See Garrido: Figs. 1-2, and [0015], “This disclosure pertains to systems, methods, and computer readable media for avatar based video communication between multiple online users. In general, the source device transmits avatar data describing the characteristics of the sending user to the receiving device in real-time. The avatar data may then be provisioned as an avatar at the receiving end to mimic the sending user's characteristics. The avatar data may be transmitted in addition to video feed, or alternatively, the communication of the avatar data may replace the transmission of the video feed”; [0024], “The data transmission management (DTM) unit 107 manages transmission of data from the source device 110 to the receiving device 130. In an embodiment, the DTM unit 107 may transmit the avatar data 122 in addition to the video stream 121 and the audio stream 123. In alternative embodiments, the communication of the avatar data 122 may replace the communication of the video stream 121. In an embodiment, the DTM unit 107 may categorize the received avatar data based on their importance and priority. Subsequently, the DTM unit 107 can decide which data to send to the receiving device based on its priority and factors such as the availability of resources (e.g. bandwidth) and user configurations”; [0033], “In an embodiment, the sending user 211 may select a choice of avatar that the captured video is rendered on. As a result, the source device receives a selection of a choice of avatar from the sending user 211. In other embodiments, the sending user 211 may select to keep a portion of the original video but modify the other parts. For example, the sending user 211 may make a selection of a virtual clothes to be displayed instead of his/her original clothing. Other selections may include, but not limited to, hairstyle, skin tones, beard/moustache, glasses, hats, and etc. In yet another embodiment, the choice of avatar may be communicated by the receiving user. In an embodiment, the avatar may be selected from categories such as animal avatars, cartoon characters, or other virtual creatures”; [0043], “In an embodiment, at block 302, the avatar data is categorized based on their importance. In context of modeling information, as explained previously, different features of the sending user captured by the video may be grouped together based on the features' importance in generating a descriptive model. Initially, when no model has been formed on the receiving end, some of the modeling information may be prioritized over some of the tracking information. As the user's model is developed on the receiving end, the prioritization may change to allow more of the tracking information be transmitted to the receiving device. In an embodiment, some of the avatar data transmitted to the receiving device could be used both to create a model and provide indications as to movements or expressions of the sending user”. Note that avatar data is categorized into groups with the first group having more unique higher priority data, and the source devices selectively transmits at least the first group based on available bandwidth, the full video stream may be replaced by the selective avatar data and only necessary information is transmitted, these features are mapped to the current cited limitations).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Quinn to have wherein the processing circuitry is configured to be able to selectively receive, from a first user terminal of a first user as taught by Garrido in order to allow audio-video synchronization on a receiving end (See Garrido: Fig. 1, and [0027], " ] In one embodiment, the video stream 121, the avatar data 122, and the audio stream 123 may be transmitted through the network 120 to the receiving device 130. In another embodiment, only the avatar data 122 and the audio stream 123 may be sent. In an embodiment where the avatar data is generated on the receiving end, only transmission of the video stream 121 and audio stream 123 may be required. In still another embodiment, the video communication may begin with displaying the video stream 121 but when the conditions are not good for the display of video stream, the system may switch to only sending the avatar data 122. For example, the network bandwidth may initially be sufficient for a video steam. However, the network bandwidth may be dropped during the communication, and therefore, the system stops transmitting the video stream 121 and instead it only sends the avatar data 122 and the audio streams 123. The avatar data 122 may include real-time features and movement of the sending user transmitted through the network 120 for rendering on the receiver device 130. The avatar stream is real-time data (RTP) which would allow for an audio-video synchronization on the receiving end "). Quinn teaches a method and system that may generate avatars from video in real-time; while Garrido teaches a system and method that may selectively transmit data according to the bandwidth. Therefore, it is obvious to one of ordinary skill in the art to modify Quinn by Garrido to selectively omitting data for bandwidth reasons. The motivation to modify Quinn by Garrido is "Use of known technique to improve similar devices (methods, or products) in the same way".
Regarding claim 2, Quinn, Leahy and Garrido teach all the features with respect to claim 1 as outlined above. Further, Leahy teaches that the virtual space presenting device according to claim 1, wherein the processing circuitry is configured to set a portion of the first avatar visible to a second user who is a user different from the first user among the plurality of users as the first portion on the basis of the virtual viewpoint of the second user and set a portion of the first avatar invisible to the second user as the second portion (See Leahy: Fig. 1, and [0017], "FIG. 1 is an illustration of a client screen display 10 seen by one user in the chat system. Screen display 10 is shown with several stationary objects (wall, floor, ceiling and clickable object 13) and two "avatars" 18. Each avatar 18 is a three dimensional figure chosen by a user to represent the user in the virtual world. Each avatar 18 optionally includes a label chosen by the user. In this example, two users are shown: "Paula" and "Ken", who have chosen the "robot" avatar and the penguin avatar, respectively. Each user interacts with a client machine (not shown) which produces a display similar to screen display 10, but from the perspective of the avatar for that client/user. Screen display 10 is the view from the perspective of a third user, D, whose avatar is not shown since D's avatar is not within D's own view. Typically, a user cannot see his or her own avatar unless the chat system allows "our of body" viewing or the avatar's image is reflected in a mirrored object in the virtual world"; Fig. 3, and [0030], "Generally, the limit set by server 61 of N avatars and the limit set by client 60 of N' avatars control how many avatars A sees. If server 61 sets a very high value for N, then the limit set by client 60 is the only controlling factor. In some cases, the definition of "neighboring" might be controlled by other factors besides proximity. For example, the virtual world might have a video telephone object where A can speak with and see a remote avatar. Also, where Nor more unfriendly avatars are in close proximity to A's avatar and they persist in following A's avatar, A will not be able to see or communicate with other, friendly avatars. To prevent this problem, user A might have a way to filter out avatars on other variables in addition to proximity, such as user ID"; [0037], "Rooms database 70 in a fixed virtual world is provided to the user with the software required to instantiate the client. Rooms database 70 specifies a list of rooms, including walls, doors and other connecting passageways. Client 60 uses the locations of walls and other objects to determine how A's avatar's position is constrained. Rooms database 70 also contains the texture maps used to texture the walls and other objects. Avatar database 71 specifies the bitmaps used to render various predefined avatars provided with the client system. Using rooms database 70 and the locations, tags and images of all the neighboring avatars, then a view of objects and other avatars in the virtual world can be rendered using the room primitives database and the avatar primitives database"; and [0079], "Two methods exist for updating the position of an actor (avatar). The LONGLOCCMD method uses full absolute position (X, Y, and Z) and orientation. The SHORTLOCCMD only updates the X and Y coordinates and the orientation. In addition, the short method limits the change in position to plus or minus 127 in the X and/or Y coordinates and/or +/-127 in the orientation. Client 60 sends a LONGLOCCMD to server 61 to update the client's position. Whenever possible, server 61 uses the combined SHORTLOCCMD to update all of the visible avatars at once. If an avatar has moved too great a distance, or has moved in the Z direction, server 61 then uses a LONGLOCCMD for that avatar". Note that the view is rendered based user's viewpoint with natural visibility handling, and updating video data including avatar visibility in real-time, but retaining the historical data for occluded /invisible portion).
Regarding claim 3, Quinn, Leahy and Garrido teach all the features with respect to claim 1 as outlined above. Further, Quinn teaches that the virtual space presenting device according to claim 1, wherein the processing circuitry is configured to acquire movement information about movement of a body of the first user (See Quinn: Fig. 1, and [0038], "In addition to one or more processors and accessible memory typically found in computer systems, computer system 12 comprises an operating system 110, a network interface 136 for communicating with other computer systems, a display interface 124 for communicating data, instructions or both, to a display like display 14 of display device 16, and a camera interface 134 for coordinating exchange of depth image data and instructions with 3D capture device 20. An image and audio processing engine 113 comprises natural user interface software 122 which may include software like gesture recognition and sound recognition software for identifying actions of a user's body or vocal cues which are commands or advance the action of a multimedia application. Additionally, 3D object recognition engine 114 detects boundaries using techniques such as edge detection and compares the boundaries with stored shape data for identifying types of objects. Color image data may also be used in object recognition. A type of object which can be identified is a human body including body parts like a human head. A scene mapping engine 118 tracks a location of one or more objects in the field of view of the 3D capture device. Additionally, object locations and movements may be tracked overtime with respect to a camera independent coordinate system"), set a portion of which predetermined movement or more has been detected within the body of the first user as the first portion on the basis of the movement information (See Quinn: Fig. 7, and [0065], "Responsive to the 3D action measurements indicating a change in a user facial feature satisfying facial action criteria, shape units of the user 3D head model effected by the facial changes are identified in step 308, and action unit coefficients are determined in step 310 for action unit sub-meshes for the effected shape units. In step 312, a facial expression of the facial avatar is updated to mimick the change in the user facial feature in accordance with the 3D action measurements. For example, in order to mimick the change, the 3D action coefficients for the action sub-meshes also effect an underlying 3D rig model for the user's facial avatar head mesh model"), and set a portion of which predetermined movement or more has not been detected within the body of the first user as the second portion (See Quinn: Fig. 7, and [0064], "FIG. 7 illustrates a flowchart of a process example of updating a facial expression of the facial avatar responsive to changes detected in user facial features. In step 302, 3D action measurements of a set of facial features are periodically determined based on stored 3D facial expression metrics (e.g. 146) for tracking facial changes based on periodically captured 3D image data of user head features. In step 304, it is determined whether the 3D action measurements indicate a change in a user facial feature satisfying facial action criteria. If not, in step 306, control is returned for other processing until the next time for the periodic performance of step 302").
Regarding claim 4, Quinn, Leahy and Garrido teach all the features with respect to claim 3 as outlined above. Further, Quinn teaches that the virtual space presenting device according to claim 3, wherein the processing circuitry sets a whole body of the first user as the first portion in an initial state (See Quinn: Fig. 1, and [0038], "In addition to one or more processors and accessible memory typically found in computer systems, computer system 12 comprises an operating system 110, a network interface 136 for communicating with other computer systems, a display interface 124 for communicating data, instructions or both, to a display like display 14 of display device 16, and a camera interface 134 for coordinating exchange of depth image data and instructions with 3D capture device 20. An image and audio processing engine 113 comprises natural user interface software 122 which may include software like gesture recognition and sound recognition software for identifying actions of a user's body or vocal cues which are commands or advance the action of a multimedia application. Additionally, 3D object recognition engine 114 detects boundaries using techniques such as edge detection and compares the boundaries with stored shape data for identifying types of objects. Color image data may also be used in object recognition. A type of object which can be identified is a human body including body parts like a human head. A scene mapping engine 118 tracks a location of one or more objects in the field of view of the 3D capture device. Additionally, object locations and movements may be tracked overtime with respect to a camera independent coordinate system"; and Figs. SA-B, and [0055], "FIG. SA illustrates a flowchart of an embodiment of a method for generating a user 3D head model based on captured 3D image data. In step 201, one or more processors executing software like the 3D object recognition software 114 generate a point cloud based on the captured 3D image data. Facial recognition software like 3D facial recognition and reconstruction engine 116 identifies head features including facial features in the point cloud in step 203, and in step 205 fits a 3D mesh to the head features including the facial features in the point cloud. In step 208, a set of measurements relating the head features including the facial features of the 3D mesh by a distance, an angle or a proportion are determined and stored. In step 210, the set of measurements and the 3D mesh are stored as a user 3D head model for example by the 3D facial and reconstruction engine 116". Note that initially the whole body data captured are used to generate the 3D head model),
changes a portion of which predetermined movement or more has not been detected continuously during a predetermined period within the first portion to the second portion (See Quinn: Fig. 1, and [0037], "The 3D capture device 20 is continuously capturing depth image data of the user, for example at a rate of 30 frames a second. From this data, facial expression changes of the user can be tracked. The 3D facial recognition and reconstruction engine 116 can determine changes in measurements of a set of facial features periodically in accordance with user 3D facial expression metrics 146 for tracking facial changes. Facial action criteria for determined whether the 3D action measurements indicate a change in a user facial feature may be stored in the user 3D facial expression metrics 146. These changes in measurements, in some examples, may be stored as action unit coefficients for action unit meshes or sub-meshes which may be blended into one or more shape units for representing expression changes of a use r's face. For transferable expressions, the action unit coefficients for the sub-meshes are mapped to corresponding action units for sub-meshes of corresponding shape units of an avatar character model. As described below, in some examples, an avatar in a defined art style may have rules, like avatar character(s) facial expression definitions and rules 154, which indicate a set of expressions an avatar can make and for which of those expressions, are user expression data transferable. For example, the 3D facial recognition and reconstruction engine 116 can include pattern recognition software which recognizes facial expressions such as a smile or a sigh based on audiovisual capture of the user as well as structures like teeth and a mouth. Details of how a user smiles may include closed lip smile or open mouth smile and may be represented in data as action unit coefficients. In some embodiments, rules for an avatar defined art style like avatar character(s) facial expression definitions and rules 154 may identify facial features of an avatar rather than expressions, which facial features move like the facial features for resembling the user's expressions more faithfully. As part of an initialization process, a user may be prompted for expressions an avatar in a defined art style can make to capture reference image data"; Figs. 3A-B, and [0041], "Facial features, for example as represented in shape units, can be determined from a tracking mask generated from image data of the user and fitted to a head reference model having a head shape similar to that of the user. In other embodiments, measurements of user head features, for example as shape units, determined from the tracking mask can be stored as data representing a user head model without using a head reference model, and retrieved later when generating the avatar. In some examples, a matching criteria may be used for identifying that user head model data measurements, e.g. measured shape units, qualify as accurately representing the tracking mask. The matching criteria may be defined in terms of an allowable offset in 3D between the vertices of a head reference model being morphed to result in the user's head model and vertices of the tracking mask. The matching criteria can be stored in the head features definitions 142 in some examples"; Figs. 4A-B, and [0042], "FIGS. 4A and 4B illustrate some examples of shape units of a 3D head model. As mentioned above, a shape unit is a collection of 3D points which represents a head feature or an aspect of a head feature. In the illustrated examples, the collection of 3D points is a set of vertices or connection points in a 3D head mesh which may also include a three dimensional (3D) rig which models human bone structure and bone movement under a skin surface which skin surface is represented by the mesh"; Figs. SA-B, and [0056], "In step 222, the 3D coefficients data for each point in each respective shape unit N which satisfied the matching criteria are stored as coefficient data of the user 3D head model"; and Fig. 7, and [0064], "FIG. 7 illustrates a flowchart of a process example of updating a facial expression of the facial avatar responsive to changes detected in user facial features. Instep 302, 3D action measurements of a set of facial features are periodically determined based on stored 3D facial expression metrics (e.g. 146) for tracking facial changes based on periodically captured 3D image data of user head features. In step 304, it is determined whether the 3D action measurements indicate a change in a user facial feature satisfying facial action criteria. If not, in step 306, control is returned for other processing until the next time for the periodic performance of step 302". Note that kinetic skeletal tracking distinguishes active joint motion from the static portions over time frames, and this is mapped to the current cited limitation), and
changes the second portion of which movement has been detected to the first portion when the predetermined movement or more has been detected in the second portion (See Quinn: Fig. 7, and [0065], "Responsive to the 3D action measurements indicating a change in a user facial feature satisfying facial action criteria, shape units of the user 3D head model effected by the facial changes are identified in step 308, and action unit coefficients are determined in step 310 for action unit sub-meshes for the effected shape units. In step 312, a facial expression of the facial avatar is updated to mimick the change in the user facial feature in accordance with the 3D action measurements. For example, in order to mimick the change, the 3D action coefficients for the action sub-meshes also effect an underlying 3D rig model for the user's facial avatar head mesh model"; Figs.3A-B, and [0041], "FIG. 3A illustrates an embodiment of a system from a software perspective for automatically generating a facial avatar resembling a user within a defined art style. The embodiment may be used to generate a model of the user's head and facial features, and then to generate a model of a facial avatar incorporating head features from the user's head mode I which head features are predetermined to be transferable to the avatar based on its defined art style"; Fig. 4A-B, and [0042], "FIGS. 4A and 4B illustrate some examples of shape units of a 3D head model. As mentioned above, a shape unit is a collection of 3D points which represents a head feature or an aspect of a head feature. In the illustrated examples, the collection of 3D points is a set of vertices or connection points in a 3D head mesh which may also include a three dimensional (3D) rig which models human bone structure and bone movement under a skin surface which skin surface is represented by the mesh";, and Fig. 7, and [0064], "FIG. 7 illustrates a flowchart of a process example of updating a facial expression of the facial avatar responsive to changes detected in user facial features. In step 302, 3D action measurements of a set of facial features are periodically determined based on stored 3D facial expression metrics (e.g. 146) for tracking facial changes based on periodically captured 3D image data of user head features. In step 304, it is determined whether the 3D action measurements indicate a change in a user facial feature satisfying facial action criteria. If not, in step 306, control is returned for other processing until the next time for the periodic performance of step 302". Note that the dynamic remapping of transferrable features from the new captured image data onto the model whenever changes are detected, and this is mapped to the current cited limitation).
Regarding claim 5, Quinn, Leahy and Garrido teach all the features with respect to claim 3 as outlined above. Further, Quinn, Leahy and Garrido teach that a method, implemented by processing circuitry of a virtual space presenting device for presenting a three-dimensional virtual space shared by a plurality of users to each user (See Leahy: Fig. 1, and [0018], "Each user is free to move his or her avatar around in the virtual world. In order that each user see the correct location of each of the other avatars, each client machine sends its current location, or changes in its current location, to the server and receives updated position information of the other clients"; and [0019], "While FIG. 1 shows two avatars (and implies a third), typically many more avatars will be present. A typical virtual world will also be more complex than a single room. The virtual world view shown in FIG. 1 is part of a virtual world of several rooms and connecting hallways as indicated in a world map panel 19, and may include hundreds or users and their avatars. So that the virtual world is scalable to a large number of clients, the virtual world server must be much more discriminating as to what data is provided to each client. In the example of FIG. 1, although a status panel 17 indicates that six other avatars are present, many other avatars are in the room, but are filtered out for crowd control"), the virtual space presenting device being located at a separate location from user terminals of the plurality of users (See Leahy: Fig. 3, and [0023], "Referring now to FIG. 3, a block diagram is shown of a world system 54 in which a user A, at a first client system 60 (client A), interacts with a user B at a second client system 60 (client B) via a server 61. Client system 60 includes several databases, some of which are fixed and some of which are modifiable. Client system 60 also includes storage for program routines. Mechanisms for storing, reading and modifying data on computers such as client system 60 are well known in the art, as are methods and means for executing programs and displaying graphical results thereof. One such program executed by client system 60 is a graphical rendering engine which generates the user's view of the virtual world". Note that Client A and Client B and server 61 are separated at different location and may be mobile which dynamically changing locations), the method See Quinn: Fig. 2, and [0031], "FIG. 2 illustrates an embodiment of a system 10 including hardware and software components for automatically generating a facial avatar resembling a user in a defined art style. An example embodiment of the capture device 20 is configured to capture video having a depth image that may include depth values via any suitable technique including, for example, time-of-flight, structured light, or the like. According to one embodiment, the capture device 20 may organize the calculated depth information into "Z layers," or layers that may be perpendicular to a Z axis extending from the depth camera along its line of sight. X and Y axes may be defined as being perpendicular to the Z axis. The Y axis may be vertical and the X axis may be horizontal. Together, the X, Y and Z axes define the 3-D real world space captured by capture device 20"; and [0022], "Technology is described for automatically generating a facial avatar in a defined art style which resembles a user based on image data captured of the user and a predetermined set of transferable head (including facial) features associated with the defined art style. In embodiments discussed below, human facial features can be represented in one or more shape units associated with a head model. As described further below, depth and image data of a user's head features including facial features can be captured. Typically, a point cloud of points having 3D Cartesian or polar coordinates with respect to an origin of a coordinate system can be used to represent the head and facial features by facial modelling software. A mesh, e.g. a triangular or polygon mesh, may be used to connect or fit the points of the point cloud") comprising:
acquiring video data obtained by photographing the users (See Quinn: Figs. 1-3, and [0032], "As shown in FIG. 2, this exemplary capture device 20 may include an image and depth camera component 22 which captures a depth image by a pixelated sensor array 26. A depth value may be associated with each captured pixel. Some examples of a depth value are a length or distance in, for example, centimeters, millimeters, or the like of an object in the captured scene from the camera. Sensor array pixel 28 is a representative example of a pixel with subpixel sensors sensitive to RGB visible light plus an IR sensor for determining a depth value for pixel 28. Other arrangements of depth sensitive and visible light sensors may be used. An infrared (IR) illumination component 24 emits an infrared light onto the scene, and the IR sensors detect the backscattered light from the surface of one or more targets and objects in the scene in the field of view of the sensor array 26 from which a depth map of the scene can be created. In some examples, time-of-flight analysis based on intensity or phase of IR light received at the sensors may be used for making depth determinations"; and [0047], "The 3D capture device 20 is continuously capturing depth image data of the user, for example at a rate of 30 frames a second. From this data, facial expression changes of the user can be tracked. The 3D facial recognition and reconstruction engine 116 can determine changes in measurements of a set of facial features periodically in accordance with user 3D facial expression metrics 146 for tracking facial changes. Facial action criteria for determined whether the 3D action measurements indicate a change in a user facial feature may be stored in the user 3D facial expression metrics 146. These changes in measurements, in some examples, may be stored as action unit coefficients for action unit meshes or sub-meshes which may be blended into one or more shape units for representing expression changes of a use r's face. For transferable expressions, the action unit coefficients for the sub-meshes are mapped to corresponding action units for sub-meshes of corresponding shape units of an avatar character model. As described below, in some examples, an avatar in a defined art style may have rules, like avatar character(s) facial expression definitions and rules 154, which indicate a set of expressions an avatar can make and for which of those expressions, are user expression data transferable. For example, the 3D facial recognition and reconstruction engine 116 can include pattern recognition software which recognizes facial expressions such as a smile or a sigh based on audiovisual capture of the user as well as structures like teeth and a mouth. Details of how a user smiles may include closed lip smile or open mouth smile and may be represented in data as action unit coefficients. In some embodiments, rules for an avatar defined art style like avatar character(s) facial expression definitions and rules 154 may identify facial features of an avatar rather than expressions, which facial features move like the facial features for resembling the user's expressions more faithfully. As part of an initialization process, a user may be prompted for expressions an avatar in a defined art style can make to capture reference image data". Note that the images are continuously captured, such as 30 fps video, which is mapped to acquire video);
generating avatars to be arranged in the virtual space in correspondence with the users on the basis of the video data of the users(See Quinn: Figs. 1A-B, and [0030], "Before proceeding to discuss in more detail hardware and software components for a system for generating a facial avatar resembling a user in a defined art style, FIG. 1B illustrates a flowchart of an overview of a method embodiment for automatically generating a facial avatar resembling a user in a defined art style. In step 101, a 3D image capture device (e.g. 20) captures 3D image data of head features including facial features of a user, and in step 102, one or more software controlled processors of one or more computer systems communicatively coupled to the capture device generate a user 3D head model for the user based on the 3D image data. In step 103, the one or more processors identify a set of transferable user head features from the user 3D head model which can be represented in a facial avatar of a defined art style based on transferable head feature rules for the defined art style, and in step 104, the one or more processors identify a set of avatar character head features to re present the set of user head features based on rules for the defined art style. In step 105, the one or more processors generate a facial avatar in the defined art style including the set of avatar character head features and cause a communicatively coupled display to display the facial avatar in step 106. Additionally, a facial expression of the facial avatar may be updated and displayed responsive to changes detected in user facial features"); and
generating and presenting videos corresponding (See Quinn: Figs. 1A-B, and [0030], "In step 105, the one or more processors generate a facial avatar in the defined art style including the set of avatar character head features and cause a communicatively coupled display to display the facial avatar in step 106. Additionally, a facial expression of the facial avatar may be updated and displayed responsive to changes detected in user facial features"; and [0050], "Accessible avatar character facial expression definitions and rules 154 may also be accessed for identifying expressions which the avatar character can make in general and those in which the avatar mimicks from a user, e.g. user transferable expressions. Data, for example action unit coefficients, may be stored for representing the expression by one or more shape units in an avatar head model. There may be rules governing some expressions an avatar does not make such as smiling or how long an expression lasts. Some examples of expressions which may be represented by action unit coefficients include puckered lips, puckered lips with ballooned cheeks, left mount corner down, closed mouth smile, an open mouthed smile, wide open mouth, blinking of an eye, brow squeeze, raised left brow, and raised right brow") to visual fields from virtual viewpoints of the users set in the virtual space with respect to the users (See Leahy: Fig. 1, and [0017], "FIG. 1 is an illustration of a client screen display 10 seen by one user in the chat system. Screen display 10 is shown with several stationary objects (wall, floor, ceiling and clickable object 13) and two "avatars" 18. Each avatar 18 is a three-dimensional figure chosen by a user to represent the user in the virtual world. Each avatar 18 optionally includes a label chosen by the user. In this example, two users are shown: "Paula" and "Ken", who have chosen the "robot" avatar and the penguin avatar, respectively. Each user interacts with a client machine (not shown) which produces a display similar to screen display 10, but from the perspective of the avatar for that client/user. Screen display 10 is the view from the perspective of a third user, D, whose avatar is not shown since D's avatar is not within D's own view. Typically, a user cannot see his or her own avatar unless the chat system allows "our of body" viewing or the avatar's image is reflected in a mirrored object in the virtual world").
wherein the method includes selectively receiving, from a first user terminal of a first user(See Garrido: Figs. 1-2, and [0015], “This disclosure pertains to systems, methods, and computer readable media for avatar based video communication between multiple online users. In general, the source device transmits avatar data describing the characteristics of the sending user to the receiving device in real-time. The avatar data may then be provisioned as an avatar at the receiving end to mimic the sending user's characteristics. The avatar data may be transmitted in addition to video feed, or alternatively, the communication of the avatar data may replace the transmission of the video feed”; [0024], “The data transmission management (DTM) unit 107 manages transmission of data from the source device 110 to the receiving device 130. In an embodiment, the DTM unit 107 may transmit the avatar data 122 in addition to the video stream 121 and the audio stream 123. In alternative embodiments, the communication of the avatar data 122 may replace the communication of the video stream 121. In an embodiment, the DTM unit 107 may categorize the received avatar data based on their importance and priority. Subsequently, the DTM unit 107 can decide which data to send to the receiving device based on its priority and factors such as the availability of resources (e.g. bandwidth) and user configurations”; [0033], “In an embodiment, the sending user 211 may select a choice of avatar that the captured video is rendered on. As a result, the source device receives a selection of a choice of avatar from the sending user 211. In other embodiments, the sending user 211 may select to keep a portion of the original video but modify the other parts. For example, the sending user 211 may make a selection of a virtual clothes to be displayed instead of his/her original clothing. Other selections may include, but not limited to, hairstyle, skin tones, beard/moustache, glasses, hats, and etc. In yet another embodiment, the choice of avatar may be communicated by the receiving user. In an embodiment, the avatar may be selected from categories such as animal avatars, cartoon characters, or other virtual creatures”; [0043], “In an embodiment, at block 302, the avatar data is categorized based on their importance. In context of modeling information, as explained previously, different features of the sending user captured by the video may be grouped together based on the features' importance in generating a descriptive model. Initially, when no model has been formed on the receiving end, some of the modeling information may be prioritized over some of the tracking information. As the user's model is developed on the receiving end, the prioritization may change to allow more of the tracking information be transmitted to the receiving device. In an embodiment, some of the avatar data transmitted to the receiving device could be used both to create a model and provide indications as to movements or expressions of the sending user”. Note that avatar data is categorized into groups with the first group having more unique higher priority data, and the source devices selectively transmits at least the first group based on available bandwidth, the full video stream may be replaced by the selective avatar data and only necessary information is transmitted, these features are mapped to the current cited limitations), first video data in which a first portion of a body of the first user is shown within video data obtained by photographing the first user in a plurality of different directions without receiving, from the first user terminal, second video data in which a second portion different from the first portion of the body of the first user is shown during a first period (See Quinn: Figs. 1A-B, and [0033], "According to another embodiment, the capture device 20 may include two or more physically separated cameras that may view a scene from different angles, to obtain visual stereo data that may be resolved to generate depth information"; [0035], "In the example embodiment, the capture device 20 may include a processor 32 in communication with the image and depth camera component 22 and having access to a memory component 34 that may store instructions for execution by the processor 32 as well as images or frames of images captured and perhaps processed by the 3D camera. The memory component 34 may include random access memory (RAM), read only memory (ROM), cache, Flash memory, a hard disk, or any other suitable storage component. The processor 32 may also perform image processing, including some object recognition steps, and formatting of the captured image data"; and [0040], "An application programming interface (API) 125 provides an interface for one or more multimedia applications 128 to register avatar characters and data definitions and rules for them associated with the applications. Data sources 132 may store these avatar data definitions and rules. Besides user specific data like personal identifying information including personally identifying image data, user profile data 130 may also store data or data references to stored locations of a user 3D head model generated for the user and avatars, including their facial aspects, gene rated for the user". Note that the depth and stereo images are continuously captured from different angles with inherent partial visibility), and
when the processing circuitry receives the first video data without receiving the second video data during the first period, generating the first portion of a first avatar corresponding to the first user for the first period on the basis of the first video data received during the first period (See Quinn: Figs. 3A-B and 4A-B, and [0040], "An application programming interface (API) 125 provides an interface for one or more multimedia applications 128 to register avatar characters and data definitions and rules for them associated with the applications. Data sources 132 may store these avatar data definitions and rules. Besides user specific data like personal identifying information including personally identifying image data, user profile data 130 may also store data or data references to stored locations of a user 3D head model generated for the user and avatars, including their facial aspects, generated for the user."; [0042], "FIGS. 4A and 4B illustrate some examples of shape units of a 3D head model. As mentioned above, a shape unit is a collection of 3D points which represents a head feature or an aspect of a head feature. In the illustrated examples, the collection of 3D points is a set of vertices or connection points in a 3D head mesh which may also include a three dimensional (3D) rig which models human bone structure and bone movement under a skin surface which skin surface is represented by the mesh"; and Fig. 7, and [0064], "] FIG. 7 illustrates a flowchart of a process example of updating a facial expression of the facial avatar responsive to changes detected in user facial features. In step 302, 3D action measurements of a set of facial features are periodically determined based on stored 3D facial expression metrics (e.g. 146) for tracking facial changes based on periodically captured 3D image data of user head features. In step 304, it is determined whether the 3D action measurements indicate a change in a user facial feature satisfying facial action criteria. If not, in step 306, control is returned for other processing until the next time for the periodic performance of step 302". Note that the current visible features and historical shape features for completing other portions of the avatar generation, dynamic updating blends real-time changes with stored based avatar model, and no second period video is needed, are mapped to "the generation unit generates the first portion of a first avatar corresponding to the first user for the first period on the basis of the first video data acquired during the first period", also the current real-time is mapped to the first period, and historical data is mapped to the second period), and generating the second portion of the first avatar for the first period on the basis of the second video data received during a second period earlier than the first period (See Quinn: Figs. 1A-B, and [0023], "Differences between the points or vertices in the mesh and a reference human head model can be identified and stored as coefficients of 3D space vectors. In some examples, the coefficients values may be between O and 1, where a 1 means an exact correspondence between a vertex in the fitted mesh and the corresponding vertex in the human head reference model. A shape unit is a collection of these 3D vertices which represents a head feature or an aspect of a head feature. If not just the boundary points of a shape unit but also the points on the interior of the shape unit only differ in scale from the human head reference model, then a scalar coefficient value between O and 1 may be applied to all the points in the shape unit at the same time, simplifying transformation operations for generating a 3D head model of the user's head and face. The more varied and detailed the collection of human head reference models stored, the more likely scalar coefficient operations on a shape unit may be performed"; [0035], "In the example embodiment, the capture device 20 may include a processor 32 in communication with the image and depth camera component 22 and having access to a memory component 34 that may store instructions for execution by the processor 32 as well as images or frames of images captured and perhaps processed by the 3D camera. The memory component 34 may include random access memory (RAM), read only memory (ROM), cache, Flash memory, a hard disk, or any other suitable storage component. The processor 32 may also perform image processing, including some object recognition steps, and formatting of the captured image data"; and Figs. 3A-B, and [0048], "FIGS. 3B and 3C illustrate some examples of data which may be stored in an embodiment of user 3D head model data 144. FIG. 3B illustrates an example including 3D measurements for head features 147 relating head features by distance, angle and proportion and 3D measurements for facial expressions 148. FIG. 3C illustrates an example in which such 3D measurements 147 are embodied as coefficients for 3D vertices in each shape unit N 145, and the 0.3D measurements of changes in facial features for one or more expressions are embodied as 3D action unit coefficient measurements for facial expressions 149". Note that image and data stored and captured before the current time is mapped to the second period data).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GORDON G LIU whose telephone number is (571)270-0382. The examiner can normally be reached Monday - Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona E Faulk can be reached at 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GORDON G LIU/Primary Examiner, Art Unit 2618
Prosecution Insights