Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Response to Arguments
Regarding 35 USC § 102/103.
Applicant argues:
A feature of the claimed method is the structural personalization of a parametric model using actual biometric/morphological data (e.g., limb length, facial geometry) to create an avatar that is geometrically faithful to the user, for example.
While Saban generally discusses extracting measurements and using avatars, Saban fails to disclose the crucial link required by Claim 1: using the extracted morphological parameters to configure a parametric avatar model.
Saban's use of measurements: The Office Action points to Saban's paragraph [0150] as teaching the determination of "morphological parameters". However, paragraph [0150] and Figure 5 of Saban describe extracting body measurements (like the ratio between navel to height, or bust to navel) for an entirely different purpose: performing a "Body Mass Indicator (BMI) analysis" to determine health risks, weight, or "suggested outfit size". Saban does not teach feeding these anatomical proportions into a parametric avatar model.
Saban's use of the parametric model: To meet the limitation of applying parameters to a parametric avatar model, the Office Action cites Saban paragraph [0167], which states: "only a 3D sensor will generate depth measurement that will be used to control a parametric model of the user's avatar... To match the avatar to the user movement, a registration technique can be used". This is a fundamental misinterpretation of the reference. In Saban, the "parametric model" is controlled by depth measurements and registration pointers specifically to match the user's movement and pose, not to define the physical shape or morphology of the avatar itself.
In other words, Saban extracts static body measurements to suggest clothing sizes (Figure 5) and uses dynamic depth measurements to make a stock avatar mimic the user's live movements (paragraph [0167]). Nothing in Saban suggests combining these separate teachings, and making a further modification not disclosed by Saban, to use the static morphological parameters to alter the geometric mesh/structure of the parametric avatar model, which is the precise technical solution claimed in Claim 1.
Examiner replies that:
Applicants arguments are not found persuasive. Applicant argues Saban does not teach “using the extracted morphological parameters to configure a parametric avatar model.” Applicant further argues [0150] is used for BMI not the parametric avatar model.
Applicant argues [0150], but Saban is cited at both [0141] and [0150]. At [0141] the image of the user is used to create a model, which includes the users 3D body curves. This is processing said data in order to determine morphological parameters specific to the user, and Applicant has not argued or refuted the citation.
At [0150] additional morphological information is from the analyzed images/video. These provide additional examples of morphological information that can be obtained. While specifically Fig. 5 uses this information as part of detection for determining BMI, health risks, etc., the concept of the invention is generating “an avatar having body characteristics corresponding to body characteristics of a user appearing in the images” and all morphological information can be applied to the avatar since the avatar is a mirror of the user.
Applicant argues [0167] fails to teach the limitation, but Saban is cited at [0167], [0123], and [0169]. At [0167] the sensor is used to control a parametric model. Applicant includes posture as a morphological parameter, where Saban is obtaining body position with the 3D sensor. At [0123] specific morphological parameters of the avatar can be adjusted. At [0169] it specifically notes the images are used to generate the avatar to represent the user captured in the image. Saban is creating an avatar that matches both a users appearance and movement.
Applicant argues:
Impermissible Hindsight
The Office Action explicitly acknowledges that the elements relied upon to reject Claim 1 are found "in several embodiments" of Saban, and concludes that it would have been obvious to combine them "into a single arrangement". This is an impermissible ex post facto (hindsight) reconstruction.
There is no teaching, suggestion, or motivation in Saban to take the BMI and clothing- size measurement module (Figure 5) and integrate it into the real-time augmented reality motion- tracking module (paragraph [0167]) to physically shape a parametric avatar. Saban is primarily concerned with an augmented reality "mirror" that overlays virtual clothes or replaces colors on a live video feed of the user. Generating a standalone, morphologically personalized parametric avatar-as opposed to a dense volumetric capture or a stock "cartoon" avatar-runs contrary to Saban's focus on manipulating live video pixels.
For these reasons, Saban does not teach or suggest the method of Claim 1, and the rejection should be withdrawn.
Examiner replies that:
Applicants arguments are not found persuasive. Applicant has ignored citations, and taken citations out of context of the invention as a whole. Saban is creating an AR mirror, but creates “an avatar having body characteristics corresponding to body characteristics of a user appearing in the images” and all morphological information can be applied to the avatar.
[0082] “In this embodiment, the system obtains input image or video from, for example, the EyesMatch computerized method 201, or from any other image/video source, e.g., user smartphone, security camera, Google glass, mobile camera or stationary camera. Additional embodiments can include additional geometric information that will help to calculate proportion like user height, gaze and the like. If the user video or image is coming from the EyesMatch module (calibrated image/video), a more comprehensive model can be created that allows for body measurements, object pose, size, highly accurate orientation detection and the like. The additional information that can be calculated from the calibrated object or video can allow for object fitting, object replacement and insertion of new objects into the frame/video, since any distortion introduced by the location and field of view of the camera has been accounted for and corrected.”
[0167] “In some embodiments, only a 3D sensor will generate depth measurement that will be used to control a parametric model of the user's avatar. The user can be virtually dressed and explored in 360.degree.. In addition the system can be controlled to adapt position movement to the user for mirror application or video conferencing. To match the avatar to the user movement, a registration technique can be used where the registration pointers can be allocated in multiple ways, e.g., automatic noticeable element detection and RANSAM random sampling and matching technique. Once the system has the pointers, it is easier to move the avatar based on the actual user movements. The data from a depth sensor or 3D camera can pass a similar mapping transformation engine and the result can be the base of a more accurate 3D avatar of the user. The avatar can be used fully or partially as a model base to present virtual elements on the user.”
Claim 9 “operating the processor to generate an avatar having body characteristics corresponding to body characteristics of a user appearing in the images.”
Applicant argues:
D. The case of claim 8
It seems in general that the Office Action misinterprets the subject-matter of claim 1, and raises a further rejection of claim 8, which seems unsupported by the document.
The feature of claim 8 reads as follows: "wherein applying the morphological parameters comprises providing to the parametric avatar model at least shapes and dimensions of elements of the user's face, and dimensions of segments of the user's limbs."
Claim 8 explicitly forces the "morphological parameters" to be geometric values defining anatomical shapes and limb dimensions. Saban definitively does not teach feeding facial element dimensions or limb segment dimensions into a parametric model to define its shape. The Examiner should agree that this embodiment is not disclosed or suggested in any way, in Saban.
For at least the above reasons, the Applicant respectfully requests the rejections under 35 U.S.C. §103 be withdrawn.
Examiner replies that:
Applicants arguments are not found persuasive. Applicant argues “Claim 8 explicitly forces the "morphological parameters" to be geometric values defining anatomical shapes and limb dimensions.” The claim does not recite “geometric values”. The claim also recites “colors of at least one of the user’s skin or hair” which is not “geometric values defining anatomical shapes and limb dimensions.” Saban further teaches body measurements and size [0082] “a more comprehensive model can be created that allows for body measurements, object pose, size, highly accurate orientation detection and the like.” Saban obtains body measurements and sizes, skin and hair information and can measure to register hands and legs for matching movement. Further, the model is explicitly a parametric model of the user [0167], where parametric models inherently include defined measurements. Applicant argues “Saban definitively does not teach feeding facial element dimensions or limb segment dimensions into a parametric model to define its shape. The Examiner should agree that this embodiment is not disclosed or suggested in any way, in Saban.” however this is not clearly required by the claim. Obtaining the shapes/dimensions/colors from an image is still input. More clearly indicating the input parameters as separate geometric values to edit an avatar would likely overcome the prior.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-6, 8-13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Saban U.S. Patent/PG Publication 20140225978.
Regarding claim 1 (independent):
A method for creating an avatar of a user, the method being implemented by a computer device and comprising: (Saban Abstract The processor can also generate an avatar having body characteristics corresponding to body characteristics of a user appearing in the images. )(Saban [0014] In some embodiments, a non-transitory computer-readable storage medium for operating a monitor, a camera, and a processor, is provided and configured so as to display a mirror-mimicking image on the monitor, and comprising: on a device having the processor and a memory, storing a program for execution by the processor, the program including instructions for: capturing an image or video from the system; calibrating the image or video based on a user tracking system; detecting an edge of an object present in the captured image or video; measuring a characteristic of the object; and creating a measurement signature of the object.).
obtaining at least image data of the user (Saban [0042] In the disclosed embodiments the camera can be located anywhere. A best practice is to provide the camera above the screen facing the user. Additional locations can include the bottom of the screen, the sides of the screen or behind the screen if the screen is a bidirectional screen. In the case of 3D, two cameras can, for example, be provided at the top and side by side, which can provide superior results and ease the computerized method. Also, 3D can be manipulated from a single camera once the user is moving and/or rotating.)
and processing said data in order to determine morphological parameters specific to the user (Saban [0141] In case of video from a single camera, it is possible to obtain a 3D model even if the user is moving in front of the camera, e.g., turning around in front of the camera. In this case, it is also possible to obtain object measurement in multiple cuts to estimate the user's 3D body curves.)(Saban [0150] Modules 504 and 505 can provide additional information into the algorithm that can help with the image analysis, e.g., sex, age, race and the like of the user and statistical proportions associated with the information. This information can be used later on to focus the search of body parts. The height measurement can be used with complementary body proportion information to focus the search of specific body parts, for example, the bust, navel, waist, hips and the like.).
and applying, to a parametric avatar model, the morphological parameters specific to the user in order to obtain a personalized parametric avatar of the user (Saban [0167] In some embodiments, only a 3D sensor will generate depth measurement that will be used to control a parametric model of the user's avatar.)(Saban [0123] FIG. 3 depicts an augmented reality module that can replace a body part, color, orientation and texture of an item or object in the foreground or background of the image, e.g., the module can add hair to the user, change the user's eyes, skin and hair color, can change the pose of the eyes and the like.)(Saban [0169] Thus, as noted above, the real time images can be used to generate an avatar in real time, where the avatar faithfully represents the user captured in the images.).
Saban discloses the above elements in several embodiments. With the embodiments being disclosed in a single reference, one of ordinary skill in the art before the effective filing date of the invention being aware of one embodiment would also have been aware of the others, and it would have been obvious to one of ordinary skill in the art at the time of the filing of the invention to have combined these elements from two or more embodiments into a single arrangement for the benefit of enjoying the advantages of all the embodiments disclosed combined into a single arrangement.
Regarding claim 2:
The method of claim 1, has all of its limitations taught by Saban. Saban further teaches wherein the obtaining comprises acquiring the image data by a plurality of cameras in order to construct a 3D model of the user by stereoscopy (Saban [0163] In embodiments utilizing a 2D sensor, the following two examples can be implemented. Since the user will be moving in front of the mirror, the system can capture two images (e.g., consecutive images) that will be used to calculate depth or texture. There are multiple ways to do so and, as an example, the system can extract two or more images from the video and based on central mass detection can calculate the distance and use the distance for the stereoscopic calculation of the texture. That is, when taking a stereoscopic image using two cameras, the distance between the cameras is known, so that triangulation can be performed using the distance between the two cameras and their angle of optical axis. But if there's only one camera, then the system takes two images using the same camera, but separate in time.)(Saban [0169] In some embodiments, the system receives the stream of images (either from one camera, from two cameras, or from a 3D camera), and use the stream of images to generate virtual reality in real time.).
Regarding claim 3:
The method according to claim 1, has all of its limitations taught by Saban. Saban further teaches wherein the obtaining comprises acquiring the image data by at least one infrared camera in order to acquire at least image depth data of the user and construct a 3D model of the user (Saban [0066] FIG. 1 is a system block diagram for an augmented reality platform supporting a real-time or recorded video/image processing. The system can include one or a plurality (1:n) of input devices 101, including a video camera, a still camera, an IR camera, a 2D camera or a 3D camera. )(Saban [0052] Use of an infrared (IR) camera with three-dimensional (3D) depth measurement is one approach in the industry;)(Saban [0135] IR/3D depth camera,).
Regarding claim 4:
The method according to claim 1, has all of its limitations taught by Saban. Saban further teaches comprising:
determining, based on said image data, at least relative body measurements of the user by photogrammetry, and evaluating morphological parameters characteristic of the user (Saban [0141] In case of video from a single camera, it is possible to obtain a 3D model even if the user is moving in front of the camera, e.g., turning around in front of the camera. In this case, it is also possible to obtain object measurement in multiple cuts to estimate the user's 3D body curves.)(Saban [0150] Modules 504 and 505 can provide additional information into the algorithm that can help with the image analysis, e.g., sex, age, race and the like of the user and statistical proportions associated with the information. This information can be used later on to focus the search of body parts. The height measurement can be used with complementary body proportion information to focus the search of specific body parts, for example, the bust, navel, waist, hips and the like.).
Regarding claim 5:
The method according to claim 4, has all of its limitations taught by Saban. Sabanfurther teaches wherein the image data further comprises data from a test pattern of known dimension, in order to determine absolute measurements of the user (Saban [0074] The factory calibration module 109 can be adapted to define the mapping transformation between the camera and the user point of view in front of the screen. Also, factory calibration module 109 can be adapted to calibrate the image based on distance, a special location (e.g., markings on the floor), user height or any combination thereof.)(Saban [0195] In one embodiment, both stations are EyesMatched/calibrated to the same geometrics and dimensions;)(Saban [0082] Additional embodiments can include additional geometric information that will help to calculate proportion like user height, gaze and the like. If the user video or image is coming from the EyesMatch module (calibrated image/video), a more comprehensive model can be created that allows for body measurements, object pose, size, highly accurate orientation detection and the like. The additional information that can be calculated from the calibrated object or video can allow for object fitting, object replacement and insertion of new objects into the frame/video, since any distortion introduced by the location and field of view of the camera has been accounted for and corrected. These corrections enable highly accurate measurements of the user height, waist, etc., and fitting of the user's body to generally classified body types.).
Regarding claim 6:
The method according to claim 1, has all of its limitations taught by Saban. Saban further teaches comprising:
determining, by colorimetry and based on said image data, at least a skin color and/or hair color of the user (Saban [0169] Thus, as noted above, the real time images can be used to generate an avatar in real time, where the avatar faithfully represents the user captured in the images.)(Saban [0123] FIG. 3 depicts an augmented reality module that can replace a body part, color, orientation and texture of an item or object in the foreground or background of the image, e.g., the module can add hair to the user, change the user's eyes, skin and hair color, can change the pose of the eyes and the like.)(Saban [0040]For example, the image can be manipulated to change colors, to enable multi-level user identification and authentication, and to enable body measurements. The system includes a calibrated camera, such that the system is capable of providing augmented reality features, for example, color, texture, shape manipulation of body items or background in, e.g., a virtual mirror or video conference applications.) since an avatar with the users kin and hair color is created, and skin/hair colors can be later modified.
Regarding claim 8:
The method according to claim 1, has all of its limitations taught by Saban. Saban further teaches wherein an application for creating a parametric avatar is programmed to accept as input parameters at least the following:
shapes and dimensions of elements of the user's face, (Saban [0124] Modules 301, 302, 303 and 306 can work in a similar manner as compared to modules 201, 202, 203 and 206 of the previously described general augmented reality module. Module 304 can have the capability of calculating or obtaining additional information like head pose or body direction) colors of at least one of the user's skin or hair (Saban [0123] FIG. 3 depicts an augmented reality module that can replace a body part, color, orientation and texture of an item or object in the foreground or background of the image, e.g., the module can add hair to the user, change the user's eyes, skin and hair color, can change the pose of the eyes and the like.), dimensions of segments of the user's limbs (Saban [0168] Other technique to fit the registration mapping is to identify element in the image like head, hands, upper body legs and match the movement of the avatar to these elements motion.).
(Saban [0167] In some embodiments, only a 3D sensor will generate depth measurement that will be used to control a parametric model of the user's avatar. The user can be virtually dressed and explored in 360.degree.. In addition the system can be controlled to adapt position movement to the user for mirror application or video conferencing. To match the avatar to the user movement, a registration technique can be used where the registration pointers can be allocated in multiple ways, e.g., automatic noticeable element detection and RANSAM random sampling and matching technique. Once the system has the pointers, it is easier to move the avatar based on the actual user movements. The data from a depth sensor or 3D camera can pass a similar mapping transformation engine and the result can be the base of a more accurate 3D avatar of the user. The avatar can be used fully or partially as a model base to present virtual elements on the user. [0168] Other technique to fit the registration mapping is to identify element in the image like head, hands, upper body legs and match the movement of the avatar to these elements motion.)
Regarding claim 9:
The method according to claim 1, has all of its limitations taught by Saban. Saban further teaches further comprising:
modifying the personalized parametric avatar of the user, at the user's request (Saban [0057] In some embodiments, the main camera stream can be manipulated in real time to change the appearance of the user, or to completely make him disappear from the video or image.)(Saban [0061] The embodiments can be applied for video conference or gaming where the user would like to change his appearance (e.g., skin complexion, dressing, face, and the like). According to the present invention, the model can allow a smooth change into a desired appearance or even create an image whereby the user disappears from the video.).
Regarding claim 10:
The method according to claim 1, has all of its limitations taught by Saban. Saban further teaches further comprising, during execution of an application in which the user's avatar takes part:
modifying the personalized parametric avatar of the user, upon receiving a command from the application (Saban [0061] The embodiments can be applied for video conference or gaming where the user would like to change his appearance (e.g., skin complexion, dressing, face, and the like). According to the present invention, the model can allow a smooth change into a desired appearance or even create an image whereby the user disappears from the video.)(Saban [0082] FIG. 2 depicts an example of an augmented reality module, which can correspond with the augmented reality module 104 described above. Specifically, the augmented reality module can have a function of allowing a user to virtually dress themselves, change appearances, such as color, accessories, etc.)(Saban [0083] For example, the interface 206 may enable the user to select a garment to be changed, e.g., change color of fabric type. This election is sent to the election module 202, so that the proper item to be modified is extracted from the rest of the image.).
Regarding claim 11:
The method according to claim 1, has all of its limitations taught by Saban. Saban further teaches wherein, in addition to the image data, voice data of the user is acquired in order to accompany a physical appearance of the personalized avatar of the user, with a realistic vocal expression of the user (Saban [0191] FIG. 8a depicts a sharing mirror experience in a video/voice conference. As shown in FIG. 8a, the mirror experience can be shared live with user/users in the cloud or via another mirror station. In the first case, the user can see himself in a mirror mode though the video stream that goes to the cloud needs to pass an additional mirror flip back to correct the gaze orientation on the other side. So, when the user (in mirror mode) looks right or left, his eyes/gaze on the other side will move to the right direction as if he looked at the right direction.) since the users voice is passed through during a video/voice conference.
Regarding claim 12 (independent):
The claim is a/an parallel version of claim 1. As such it is rejected under the same teachings.
Regarding claim 13 (independent):
The claim is a/an parallel version of claim 1. As such it is rejected under the same teachings.
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Saban U.S. Patent/PG Publication 20140225978 in view of Sminchises U.S. Patent/PG Publication 20190371080.
Regarding claim 7:
The method according to claim 1, has all of its limitations taught by Saban. Saban further teaches comprising:
making use of (Saban [0167] In some embodiments, only a 3D sensor will generate depth measurement that will be used to control a parametric model of the user's avatar. The user can be virtually dressed and explored in 360.degree.. In addition the system can be controlled to adapt position movement to the user for mirror application or video conferencing. To match the avatar to the user movement, a registration technique can be used where the registration pointers can be allocated in multiple ways, e.g., automatic noticeable element detection and RANSAM random sampling and matching technique. Once the system has the pointers, it is easier to move the avatar based on the actual user movements. The data from a depth sensor or 3D camera can pass a similar mapping transformation engine and the result can be the base of a more accurate 3D avatar of the user. The avatar can be used fully or partially as a model base to present virtual elements on the user. [0168] Other technique to fit the registration mapping is to identify element in the image like head, hands, upper body legs and match the movement of the avatar to these elements motion.)
Saban does not teach AI segmentation. In a related field of endeavor, Sminchises teaches:
making use of artificial intelligence to identify, based on said image data, different parts of the user's body (Sminchises [0015] According to an embodiment, the first 3D human pose and body shape fitting model is determined or obtained by determining, with regard to the first person (in the first image) and by using a multitask deep neural network model corresponding 2D body joint locations, semantic body part segments and 3D pose and by refining the first 3D human pose and body shape fitting model by executing non-linear optimization. According to an embodiment, the second 3D human pose and body shape fitting module is determined or obtained by determining, with regard to the second person (in the second image) and by using the multitask deep neural network model, corresponding 2D body joint locations, semantic body part segments and 3D pose and by refining the second 3D human pose and body shape fitting model by executing non-linear optimization..)
Therefore, it would have been obvious before the effective filing date of the claimed invention to use AI for segmenting as taught by Sminchises. The motivation for doing so would have been that neural networks provide fast and accurate data classification for images. Therefore it would have been obvious to combine Sminchises with Saban to obtain the invention.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JASON PRINGLE-PARKER whose telephone number is (571) 272-5690 and e-mail is jason.pringle-parker@uspto.gov. The examiner can normally be reached on 8:30am-5:00pm est Monday-Friday. If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, King Poon can be reached on (571) 270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, seehttp://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JASON A PRINGLE-PARKER/
Primary Examiner, Art Unit 2617
Prosecution Insights