DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
This application is a continuation of US Application No. 18/310,057, filed 005/01/2023, which claims the benefit of US Provisional Application No. 63/368,260.
Information Disclosure Statement
The information disclosure statement (IDS) submitted is considered by the examiner.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 7, 13-15 and 19-20 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention. Claims 7, 13 and 19 recite the limitation "wherein the first video signal data of the subject is received during a first period of time and the second video signal data of the subject is received during a second period of time that does not overlap with the first period of time". Paragraph [0016] of the specification stated that “body motion and facial expression data (computed based on captured video signal data of the subject) can be saved as collective body data and the digital asset can be mapped at a later time to the collective body data”. Also, paragraph [0030] disclosed that user viewing the display 114 of the device 100 can see a CGI Character with actions (e.g., facial and/or body movements) mapped to the actor, in some examples, while the actor is being filmed, or at a later time. By way of further example, the actor can be recorded by the image sensor 102 and body movements and/or facial expressions of the actor can be extracted from captured images by the image sensor 102 and stored as data for later digital asset mapping, and/or placement of the digital asset with animations based on the mapping into virtual or other recorded environments.” It is not found in the specification where the time period of the first and second period of times are disclosed; let alone, the time periods not overlapping. Support of the claimed limitation is respectfully requested. Claims 14-15 and 20 depend on claims 13 and 19 respectively.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xu et al. (US Publication Number 2012/0130717 A1, hereinafter “Xu”).
(1) regarding claim 1:
As explained in figs. 1-3, Xu disclosed a method for animating a digital asset during filming of a film (para. [0043], note that FIG. 3 is a flowchart showing an illustrative process of creating a personalized avatar comprising an animated representation of an individual 202) comprising:
receiving, by a processor, first video signal data capturing a facial feature of a subject for mapping onto a digital asset, the first video signal data being generated by a first device (para. [0054], note that FIG. 4, at 402, the avatar application 116 extracts features based on speech signals of the data from the facial markers what were placed in the face, see para. [0052]. The avatar application 116 extracts segmented speech phoneme and prosody features from the data);
receiving, by the processor, second video signal data capturing a body movement of the subject for the mapping onto the digital asset, the second video signal data being generated by a second device (para. [0055], note that at 404, the avatar application 116 transforms motion trajectories of the upper body parts to a new coordinate system based on motion signals of the data);
generating, by the processor, a digital asset animation of the digital asset based on the facial feature and body movement (para. [0056], note that at 406, the avatar application 116 trains the one or more animated models by using the extracted features from the speech 402, motion trajectories transformed from the motion data 404, and speech and motion data 400. The avatar application 116 trains the animated models using the extracted features, such as sentences, phrases, words, phonemes, and transformed motion trajectories on a new coordinate motion);
mapping, by the processor, the digital asset animation onto the digital asset (para. [0069], note that FIG. 7 is a flowchart showing an illustrative process 700 of mapping a 3D motion trajectories to a 2D cartoon face 208 (discussed at a high level) and providing real-time animation of personalized avatar 210).
Xu disclosed most of the subject matter as described as above except for specifically teaching causing, by the processor, augmented video data to be generated in real-time during the filming of the film based on the digital asset animation of the digital asset within a virtual environment, wherein the augmented video data is provided to one or more displays for viewing.
However, it would have been obvious for Xu to teach causing, by the processor, augmented video data to be generated in real-time during the filming of the film based on the digital asset animation of the digital asset within a virtual environment, wherein the augmented video data is provided to one or more displays for viewing (para. [0073], note that at 210, the avatar application 116 provides real-time animation of the personalized avatar. The animated sequence of upper body parts are combined with the personalized avatar in response to the real-time speech input. In particular, for 2D cartoon animations, the rendering process is a key frame illustration process).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach causing, by the processor, augmented video data to be generated in real-time during the filming of the film based on the digital asset animation of the digital asset within a virtual environment, wherein the augmented video data is provided to one or more displays for viewing. The suggestion/motivation for doing so would have been in order to for providing real-time animation for a personalized cartoon avatar and generate an animated sequence of motions of the one or more upper body parts by applying the one or more animated models in response to the real-time speech input (abs.). Therefore, it would have been obvious for Xu to obtain the invention as specified in claim 1.
(2) regarding claim 2:
Xu further disclosed the method of claim 1, further comprising recording, by an environmental sensor, a physical background that is mapped to the virtual environment in real-time (para. [0070], note that the avatar application 116 tracks or records movement i.e. using camera of about 60 points on a human face in 3D 702…This occurs by using face motion capture or performance capture, which makes use of facial expressions based on an actor acting out the scenes as if he or she was the character to be animated in real-time, see para. [0071]).
Claim(s) 3-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xu et al. (US Publication Number 2012/0130717 A1, hereinafter “Xu”) in view of Faulkner (US Publication Number 2019/0004639 A1).
(1) regarding claim 3:
Xu disclosed most of the subject as described as above except for specifically teaching wherein the environmental sensor scans the physical background to determine a position of the first and second device relative to the physical background.
However, Faulkner disclosed wherein the environmental sensor scans the physical background to determine a position of the first and second device relative to the physical background (para. [0027], note that the camera is positioned to capture live image data (for example, HD video) of the user 320 as he or she interacts with the first computing device 106. In some embodiments, the data capture device 310 is or includes other types of sensors for capturing movements or the location of a user (for example, infrared sensors, ultrasonic sensors, accelerometers, other types of motion sensors, and the like). Para. [0019], note that the virtual meeting manager 208 allows users of the first computing device 106 and the second computing device 108 (and optionally users using similar computing devices) to simultaneously conduct virtual meetings and, optionally, view and edit shared content).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach wherein the environmental sensor scans the physical background to determine a position of the first and second device relative to the physical background. The suggestion/motivation for doing so would have been in order to provide, the first user and a second user, an object within the virtual meeting. The object displays live visual data based on the live image data and the object moves with respect to the position of the cursor-control device associated with the first user (abs.). Therefore, it would have been obvious to combine Xu with Faulkner to obtain the invention as specified in claim 3.
(2) regarding claim 4:
Xu disclosed most of the subject as described as above except for specifically teaching further comprising positioning the digital asset at a location in the virtual environment associated with a physical position of the subject in the physical background.
However, Faulkner disclosed positioning the digital asset at a location in the virtual environment associated with a physical position of the subject in the physical background (para. [0029], note that the meeting server 102 provides an object within the virtual meeting (involving the first user and a second user) that displays the live data (or a representation thereof, such as an avatar), wherein the object is associated with and moves with the position of the cursor-control device 309 associated with the first computing device).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach positioning the digital asset at a location in the virtual environment associated with a physical position of the subject in the physical background. The suggestion/motivation for doing so would have been in order to provide, the first user and a second user, an object within the virtual meeting. The object displays live visual data based on the live image data and the object moves with respect to the position of the cursor-control device associated with the first user (abs.). Therefore, it would have been obvious to combine Xu with Faulkner to obtain the invention as specified in claim 4.
(3) regarding claim 5:
Xu disclosed most of the subject as described as above except for specifically teaching wherein the first and second device are positioned in a first perspective relative to the physical background and the augmented video data displays the digital asset in the virtual environment from the first perspective, the method further comprising: moving the first and second device from the first perspective to a second perspective relative the physical background, wherein the augmented video data is updated in real-time to display the digital asset from the second perspective in the virtual environment in response to moving the first and second device.
However, Faulkner disclosed wherein the first and second device are positioned in a first perspective relative to the physical background and the augmented video data displays the digital asset in the virtual environment from the first perspective (para. [0027], note that the camera is positioned to capture live image data of the user 320 as he or she interacts with the first computing device 106. In some embodiments, the data capture device 310 is or includes other types of sensors for capturing movements or the location of a user. Para. [0019], note that the virtual meeting manager 208 allows users of the first computing device 106 and the second computing device 108 (and optionally users using similar computing devices (not shown)) to simultaneously conduct virtual meetings and, optionally, view and edit shared content), the method further comprising: moving the first and second device from the first perspective to a second perspective relative the physical background, wherein the augmented video data is updated in real-time to display the digital asset from the second perspective in the virtual environment in response to moving the first and second device (para. [0029], note that the meeting server 102 provides an object within the virtual meeting (involving the first user and a second user) that displays the live data (or a representation thereof, such as an avatar), wherein the object is associated with and moves with the position of the cursor-control device 309 associated with the first computing device).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach wherein the first and second device are positioned in a first perspective relative to the physical background and the augmented video data displays the digital asset in the virtual environment from the first perspective, the method further comprising: moving the first and second device from the first perspective to a second perspective relative the physical background, wherein the augmented video data is updated in real-time to display the digital asset from the second perspective in the virtual environment in response to moving the first and second device. The suggestion/motivation for doing so would have been in order to provide, the first user and a second user, an object within the virtual meeting. The object displays live visual data based on the live image data and the object moves with respect to the position of the cursor-control device associated with the first user (abs.). Therefore, it would have been obvious to combine Xu with Faulkner to obtain the invention as specified in claim 5.
(4) regarding claim 6:
Xu disclosed most of the subject as described as above except for specifically teaching wherein the physical position of the subject is a first physical position, the method further comprising updating the location of the digital asset in the virtual environment in response the subject moving to a second physical position in real-time.
However, Faulkner disclosed wherein the physical position of the subject is a first physical position, the method further comprising updating the location of the digital asset in the virtual environment in response the subject moving to a second physical position in real-time (para. [0029], note that a user participating in the virtual environment views a cursor controlled by another user participating in the virtual environment, the user can more easily identify what user is moving the cursor and any other input provided by the user moving the cursor. As noted above, in some embodiments, such objects are referred to herein as living avatars as they associate a cursor with additional live input provided by a user associated with the cursor. Live input may be, for example, a cursor input, input from a device sensor, voice controls, typing input, keyboard commands, map coordinate positions, and the like).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach wherein the physical position of the subject is a first physical position, the method further comprising updating the location of the digital asset in the virtual environment in response the subject moving to a second physical position in real-time. The suggestion/motivation for doing so would have been in order to provide, the first user and a second user, an object within the virtual meeting. The object displays live visual data based on the live image data and the object moves with respect to the position of the cursor-control device associated with the first user (abs.). Therefore, it would have been obvious to combine Xu with Faulkner to obtain the invention as specified in claim 6.
(5) regarding claim 7:
Xu disclosed most of the subject as described as above except for specifically teaching wherein the first video signal data of the subject is received during a first period of time and the second video signal data of the subject is received during a second period of time that does not overlap with the first period of time (para. [0049], note that the speech and motion data may be collected using motion capture and/or performance capture, which records movement of the upper body parts and translates the movement onto the animated models. The upper body parts include but are not limited to one or more of overall face, a chin, a mouth, a tongue, a lip, a nose, eyes, eyebrows, a forehead, cheeks, a head, and a shoulder).
(6) regarding claim 8:
Xu further disclosed the method of claim 6, wherein the facial feature and/or expression of the subject and the body movement of the subject are captured simultaneously (para. [0049], note that the speech and motion data may be collected using motion capture and/or performance capture, which records movement of the upper body parts and translates the movement onto the animated models).
(7) regarding claim 9:
Xu disclosed most of the subject as described as above except for specifically teaching wherein the first device is a first mobile phone and the second device is a second mobile phone.
However, Faulkner disclosed wherein the first device is a first mobile phone and the second device is a second mobile phone (para. [0016], note that the system 100 includes a meeting server 102, a first computing device 106, and a second computing device 108. Also see para. [0022], that the devices are smart phone).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach wherein the first device is a first mobile phone and the second device is a second mobile phone. The suggestion/motivation for doing so would have been in order to provide, the first user and a second user, an object within the virtual meeting. The object displays live visual data based on the live image data and the object moves with respect to the position of the cursor-control device associated with the first user (abs.). Therefore, it would have been obvious to combine Xu with Faulkner to obtain the invention as specified in claim 9.
(8) regarding claim 10:
As explained in figs. 1-3, Xu disclosed a method for animating a digital asset during filming of a film (para. [0043], note that FIG. 3 is a flowchart showing an illustrative process of creating a personalized avatar comprising an animated representation of an individual 202) comprising:
receiving, by a processor, first video signal data capturing a facial feature of a subject for mapping onto a digital asset, the first video signal data being generated by a first device during filming (para. [0054], note that FIG. 4, at 402, the avatar application 116 extracts features based on speech signals of the data from the facial markers what were placed in the face, see para. [0052]. The avatar application 116 extracts segmented speech phoneme and prosody features from the data);
receiving, by the processor, second video signal data capturing a body movement of the subject for the mapping onto the digital asset, the second video signal data being generated by a second device during filming (para. [0055], note that at 404, the avatar application 116 transforms motion trajectories of the upper body parts to a new coordinate system based on motion signals of the data),
generating, by the processor, a digital asset animation of the digital asset based on the facial feature and body movement (para. [0056], note that at 406, the avatar application 116 trains the one or more animated models by using the extracted features from the speech 402, motion trajectories transformed from the motion data 404, and speech and motion data 400. The avatar application 116 trains the animated models using the extracted features, such as sentences, phrases, words, phonemes, and transformed motion trajectories on a new coordinate motion);
mapping, by the processor, the digital asset animation onto the digital asset (para. [0069], note that FIG. 7 is a flowchart showing an illustrative process 700 of mapping a 3D motion trajectories to a 2D cartoon face 208 (discussed at a high level) and providing real-time animation of personalized avatar 210);
recording, by an environmental sensor, a physical background that is mapped to a virtual environment in real-time (para. [0070], note that the avatar application 116 tracks or records movement i.e. using camera of about 60 points on a human face in 3D 702…This occurs by using face motion capture or performance capture, which makes use of facial expressions based on an actor acting out the scenes as if he or she was the character to be animated in real-time, see para. [0071]); and
causing, by the processor, augmented video data to be generated in real-time during the filming of the film based on the digital asset animation of the digital asset within the virtual environment, wherein the augmented video data is provided to one or more displays for viewing (para. [0073], note that at 210, the avatar application 116 provides real-time animation of the personalized avatar. The animated sequence of upper body parts are combined with the personalized avatar in response to the real-time speech input. In particular, for 2D cartoon animations, the rendering process is a key frame illustration process).
Xu disclosed most of the subject matter as described as above except for specifically teaching wherein the environmental sensor scans the physical background to determine a position of the first and second device relative to the physical background; and wherein the first and second video signal data do not include markers to track the facial feature or body movement.
However, Faulkner disclosed wherein the environmental sensor scans the physical background to determine a position of the first and second device relative to the physical background (para. [0027], note that the camera is positioned to capture live image data (for example, HD video) of the user 320 as he or she interacts with the first computing device 106. In some embodiments, the data capture device 310 is or includes other types of sensors for capturing movements or the location of a user (for example, infrared sensors, ultrasonic sensors, accelerometers, other types of motion sensors, and the like). Para. [0019], note that the virtual meeting manager 208 allows users of the first computing device 106 and the second computing device 108 (and optionally users using similar computing devices) to simultaneously conduct virtual meetings and, optionally, view and edit shared content); and wherein the first and second video signal data do not include markers to track the facial feature or body movement (para. [0039], note that the electronic processor 302 analyzes the received live data to capture facial or body movements of the user, and animates the avatar to closely mimic or duplicate the movements. In particular, when the user is speaking, the mouth of the avatar may move and, when the user shakes his or her head, the living avatar shakes its head similarly. In some embodiments, the animation focuses on the facial features of the avatar, while some embodiments also include hand and body movements).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach wherein the environmental sensor scans the physical background to determine a position of the first and second device relative to the physical background; and wherein the first and second video signal data do not include markers to track the facial feature or body movement. The suggestion/motivation for doing so would have been in order to provide, the first user and a second user, an object within the virtual meeting. The object displays live visual data based on the live image data and the object moves with respect to the position of the cursor-control device associated with the first user (abs.). Therefore, it would have been obvious to combine Xu with Faulkner to obtain the invention as specified in claim 10.
The proposed rejection of claims 3-10, as explained in the method claims render obvious the steps of the method of claims 11-15 and the non-transitory computer readable medium claims 16-20 because these steps occur in the operation of the proposed rejection as discussed above. Thus, the arguments similar to that presented above for claims 3-10 are equally applicable to claims 11-20.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Melo et al. (US Publication Number 2020/0098182 A1) disclosed an augmented reality system for an amusement ride includes a facial recognition sensor that detects a guest's face, a skeletal recognition sensor that detects a guest's body, a presence sensor that detects a guest's presence, and a controller.
Fang et al. (US Publication Number 2018/0300925 A1) disclosed a method for mapping facial expressions to an alternative avatar expression may include capturing a series of images of a face, and detecting a sequence of facial expressions of the face from the series of images. The method may include determining an alternative avatar expression mapped to the sequence of facial expressions, and animating an avatar using the alternative avatar expression.
Any inquiry concerning this communication or earlier communication from the examiner should be directed to Hilina K Demeter whose telephone number is (571) 270-1676.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Y. Poon could be reached at (571) 270- 0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about PAIR system, see http://pari-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HILINA K DEMETER/Primary Examiner, Art Unit 2617
Prosecution Insights