Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 2 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. The claim recites “construct a stereoscopic 3D model” however a 3D model is not itself stereoscopic, stereoscopic refers to generating a pair of images for viewing an object in 3D. The two concepts are related, but “a stereoscopic 3D model” combines the two concepts in a non-logical way. It is possible to, for example, a) construct stereoscopic images of the 3D model b) construct a 3D model for use with a stereoscopic display or c) construct a 3D model from stereoscopic images. Since claim 2 uses multiple cameras, option c) is being used as the interpretation.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-6, 8-13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Saban U.S. Patent/PG Publication 20140225978.
Regarding claim 1 (independent):
A method for creating an avatar of a user, the method being implemented by a computer device and comprising: (Saban Abstract The processor can also generate an avatar having body characteristics corresponding to body characteristics of a user appearing in the images. )(Saban [0014] In some embodiments, a non-transitory computer-readable storage medium for operating a monitor, a camera, and a processor, is provided and configured so as to display a mirror-mimicking image on the monitor, and comprising: on a device having the processor and a memory, storing a program for execution by the processor, the program including instructions for: capturing an image or video from the system; calibrating the image or video based on a user tracking system; detecting an edge of an object present in the captured image or video; measuring a characteristic of the object; and creating a measurement signature of the object.).
obtaining at least image data of the user (Saban [0042] In the disclosed embodiments the camera can be located anywhere. A best practice is to provide the camera above the screen facing the user. Additional locations can include the bottom of the screen, the sides of the screen or behind the screen if the screen is a bidirectional screen. In the case of 3D, two cameras can, for example, be provided at the top and side by side, which can provide superior results and ease the computerized method. Also, 3D can be manipulated from a single camera once the user is moving and/or rotating.)
and processing said data in order to determine morphological parameters specific to the user (Saban [0141] In case of video from a single camera, it is possible to obtain a 3D model even if the user is moving in front of the camera, e.g., turning around in front of the camera. In this case, it is also possible to obtain object measurement in multiple cuts to estimate the user's 3D body curves.)(Saban [0150] Modules 504 and 505 can provide additional information into the algorithm that can help with the image analysis, e.g., sex, age, race and the like of the user and statistical proportions associated with the information. This information can be used later on to focus the search of body parts. The height measurement can be used with complementary body proportion information to focus the search of specific body parts, for example, the bust, navel, waist, hips and the like.).
and applying, to a parametric avatar model, the morphological parameters specific to the user in order to obtain a personalized parametric avatar of the user (Saban [0167] In some embodiments, only a 3D sensor will generate depth measurement that will be used to control a parametric model of the user's avatar.)(Saban [0123] FIG. 3 depicts an augmented reality module that can replace a body part, color, orientation and texture of an item or object in the foreground or background of the image, e.g., the module can add hair to the user, change the user's eyes, skin and hair color, can change the pose of the eyes and the like.)(Saban [0169] Thus, as noted above, the real time images can be used to generate an avatar in real time, where the avatar faithfully represents the user captured in the images.).
Saban discloses the above elements in several embodiments. With the embodiments being disclosed in a single reference, one of ordinary skill in the art before the effective filing date of the invention being aware of one embodiment would also have been aware of the others, and it would have been obvious to one of ordinary skill in the art at the time of the filing of the invention to have combined these elements from two or more embodiments into a single arrangement for the benefit of enjoying the advantages of all the embodiments disclosed combined into a single arrangement.
Regarding claim 2:
The method of claim 1, has all of its limitations taught by Saban. Saban further teaches wherein the obtaining comprises acquiring the image data by a plurality of cameras in order to construct a stereoscopic 3D model of the user (Saban [0163] In embodiments utilizing a 2D sensor, the following two examples can be implemented. Since the user will be moving in front of the mirror, the system can capture two images (e.g., consecutive images) that will be used to calculate depth or texture. There are multiple ways to do so and, as an example, the system can extract two or more images from the video and based on central mass detection can calculate the distance and use the distance for the stereoscopic calculation of the texture. That is, when taking a stereoscopic image using two cameras, the distance between the cameras is known, so that triangulation can be performed using the distance between the two cameras and their angle of optical axis. But if there's only one camera, then the system takes two images using the same camera, but separate in time.)(Saban [0169] In some embodiments, the system receives the stream of images (either from one camera, from two cameras, or from a 3D camera), and use the stream of images to generate virtual reality in real time.).
Regarding claim 3:
The method according to claim 1, has all of its limitations taught by Saban. Saban further teaches wherein the obtaining comprises acquiring the image data by at least one infrared camera in order to acquire at least image depth data of the user and construct a 3D model of the user (Saban [0066] FIG. 1 is a system block diagram for an augmented reality platform supporting a real-time or recorded video/image processing. The system can include one or a plurality (1:n) of input devices 101, including a video camera, a still camera, an IR camera, a 2D camera or a 3D camera. )(Saban [0052] Use of an infrared (IR) camera with three-dimensional (3D) depth measurement is one approach in the industry;)(Saban [0135] IR/3D depth camera,).
Regarding claim 4:
The method according to claim 1, has all of its limitations taught by Saban. Saban further teaches comprising:
determining, based on said image data, at least relative body measurements of the user by photogrammetry, and evaluating morphological parameters characteristic of the user (Saban [0141] In case of video from a single camera, it is possible to obtain a 3D model even if the user is moving in front of the camera, e.g., turning around in front of the camera. In this case, it is also possible to obtain object measurement in multiple cuts to estimate the user's 3D body curves.)(Saban [0150] Modules 504 and 505 can provide additional information into the algorithm that can help with the image analysis, e.g., sex, age, race and the like of the user and statistical proportions associated with the information. This information can be used later on to focus the search of body parts. The height measurement can be used with complementary body proportion information to focus the search of specific body parts, for example, the bust, navel, waist, hips and the like.).
Regarding claim 5:
The method according to claim 4, has all of its limitations taught by Saban. Sabanfurther teaches wherein the image data further comprises data from a test pattern of known dimension, in order to determine absolute measurements of the user (Saban [0074] The factory calibration module 109 can be adapted to define the mapping transformation between the camera and the user point of view in front of the screen. Also, factory calibration module 109 can be adapted to calibrate the image based on distance, a special location (e.g., markings on the floor), user height or any combination thereof.)(Saban [0195] In one embodiment, both stations are EyesMatched/calibrated to the same geometrics and dimensions;)(Saban [0082] Additional embodiments can include additional geometric information that will help to calculate proportion like user height, gaze and the like. If the user video or image is coming from the EyesMatch module (calibrated image/video), a more comprehensive model can be created that allows for body measurements, object pose, size, highly accurate orientation detection and the like. The additional information that can be calculated from the calibrated object or video can allow for object fitting, object replacement and insertion of new objects into the frame/video, since any distortion introduced by the location and field of view of the camera has been accounted for and corrected. These corrections enable highly accurate measurements of the user height, waist, etc., and fitting of the user's body to generally classified body types.).
Regarding claim 6:
The method according to claim 1, has all of its limitations taught by Saban. Saban further teaches comprising:
determining, by colorimetry and based on said image data, at least a skin color and/or hair color of the user (Saban [0169] Thus, as noted above, the real time images can be used to generate an avatar in real time, where the avatar faithfully represents the user captured in the images.)(Saban [0123] FIG. 3 depicts an augmented reality module that can replace a body part, color, orientation and texture of an item or object in the foreground or background of the image, e.g., the module can add hair to the user, change the user's eyes, skin and hair color, can change the pose of the eyes and the like.)(Saban [0040]For example, the image can be manipulated to change colors, to enable multi-level user identification and authentication, and to enable body measurements. The system includes a calibrated camera, such that the system is capable of providing augmented reality features, for example, color, texture, shape manipulation of body items or background in, e.g., a virtual mirror or video conference applications.) since an avatar with the users kin and hair color is created, and skin/hair colors can be later modified.
Regarding claim 8:
The method according to claim 1, has all of its limitations taught by Saban. Saban further teaches wherein an application for creating a parametric avatar is programmed to accept as input parameters at least the following:
shapes and dimensions of elements of the user's face, (Saban [0124] Modules 301, 302, 303 and 306 can work in a similar manner as compared to modules 201, 202, 203 and 206 of the previously described general augmented reality module. Module 304 can have the capability of calculating or obtaining additional information like head pose or body direction) colors of at least one of the user's skin or hair (Saban [0123] FIG. 3 depicts an augmented reality module that can replace a body part, color, orientation and texture of an item or object in the foreground or background of the image, e.g., the module can add hair to the user, change the user's eyes, skin and hair color, can change the pose of the eyes and the like.), dimensions of segments of the user's limbs (Saban [0168] Other technique to fit the registration mapping is to identify element in the image like head, hands, upper body legs and match the movement of the avatar to these elements motion.).
(Saban [0167] In some embodiments, only a 3D sensor will generate depth measurement that will be used to control a parametric model of the user's avatar. The user can be virtually dressed and explored in 360.degree.. In addition the system can be controlled to adapt position movement to the user for mirror application or video conferencing. To match the avatar to the user movement, a registration technique can be used where the registration pointers can be allocated in multiple ways, e.g., automatic noticeable element detection and RANSAM random sampling and matching technique. Once the system has the pointers, it is easier to move the avatar based on the actual user movements. The data from a depth sensor or 3D camera can pass a similar mapping transformation engine and the result can be the base of a more accurate 3D avatar of the user. The avatar can be used fully or partially as a model base to present virtual elements on the user. [0168] Other technique to fit the registration mapping is to identify element in the image like head, hands, upper body legs and match the movement of the avatar to these elements motion.)
Regarding claim 9:
The method according to claim 1, has all of its limitations taught by Saban. Saban further teaches further comprising:
modifying the personalized parametric avatar of the user, at the user's request (Saban [0057] In some embodiments, the main camera stream can be manipulated in real time to change the appearance of the user, or to completely make him disappear from the video or image.)(Saban [0061] The embodiments can be applied for video conference or gaming where the user would like to change his appearance (e.g., skin complexion, dressing, face, and the like). According to the present invention, the model can allow a smooth change into a desired appearance or even create an image whereby the user disappears from the video.).
Regarding claim 10:
The method according to claim 1, has all of its limitations taught by Saban. Saban further teaches further comprising, during execution of an application in which the user's avatar takes part:
modifying the personalized parametric avatar of the user, upon receiving a command from the application (Saban [0061] The embodiments can be applied for video conference or gaming where the user would like to change his appearance (e.g., skin complexion, dressing, face, and the like). According to the present invention, the model can allow a smooth change into a desired appearance or even create an image whereby the user disappears from the video.)(Saban [0082] FIG. 2 depicts an example of an augmented reality module, which can correspond with the augmented reality module 104 described above. Specifically, the augmented reality module can have a function of allowing a user to virtually dress themselves, change appearances, such as color, accessories, etc.)(Saban [0083] For example, the interface 206 may enable the user to select a garment to be changed, e.g., change color of fabric type. This election is sent to the election module 202, so that the proper item to be modified is extracted from the rest of the image.).
Regarding claim 11:
The method according to claim 1, has all of its limitations taught by Saban. Saban further teaches wherein, in addition to the image data, voice data of the user is acquired in order to accompany a physical appearance of the personalized avatar of the user, with a realistic vocal expression of the user (Saban [0191] FIG. 8a depicts a sharing mirror experience in a video/voice conference. As shown in FIG. 8a, the mirror experience can be shared live with user/users in the cloud or via another mirror station. In the first case, the user can see himself in a mirror mode though the video stream that goes to the cloud needs to pass an additional mirror flip back to correct the gaze orientation on the other side. So, when the user (in mirror mode) looks right or left, his eyes/gaze on the other side will move to the right direction as if he looked at the right direction.) since the users voice is passed through during a video/voice conference.
Regarding claim 12 (independent):
The claim is a/an parallel version of claim 1. As such it is rejected under the same teachings.
Regarding claim 13 (independent):
The claim is a/an parallel version of claim 1. As such it is rejected under the same teachings.
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Saban U.S. Patent/PG Publication 20140225978 in view of Sminchises U.S. Patent/PG Publication 20190371080.
Regarding claim 7:
The method according to claim 1, has all of its limitations taught by Saban. Saban further teaches comprising:
making use of (Saban [0167] In some embodiments, only a 3D sensor will generate depth measurement that will be used to control a parametric model of the user's avatar. The user can be virtually dressed and explored in 360.degree.. In addition the system can be controlled to adapt position movement to the user for mirror application or video conferencing. To match the avatar to the user movement, a registration technique can be used where the registration pointers can be allocated in multiple ways, e.g., automatic noticeable element detection and RANSAM random sampling and matching technique. Once the system has the pointers, it is easier to move the avatar based on the actual user movements. The data from a depth sensor or 3D camera can pass a similar mapping transformation engine and the result can be the base of a more accurate 3D avatar of the user. The avatar can be used fully or partially as a model base to present virtual elements on the user. [0168] Other technique to fit the registration mapping is to identify element in the image like head, hands, upper body legs and match the movement of the avatar to these elements motion.)
Saban does not teach AI segmentation. In a related field of endeavor, Sminchises teaches:
making use of artificial intelligence to identify, based on said image data, different parts of the user's body (Sminchises [0015] According to an embodiment, the first 3D human pose and body shape fitting model is determined or obtained by determining, with regard to the first person (in the first image) and by using a multitask deep neural network model corresponding 2D body joint locations, semantic body part segments and 3D pose and by refining the first 3D human pose and body shape fitting model by executing non-linear optimization. According to an embodiment, the second 3D human pose and body shape fitting module is determined or obtained by determining, with regard to the second person (in the second image) and by using the multitask deep neural network model, corresponding 2D body joint locations, semantic body part segments and 3D pose and by refining the second 3D human pose and body shape fitting model by executing non-linear optimization..)
Therefore, it would have been obvious before the effective filing date of the claimed invention to use AI for segmenting as taught by Sminchises. The motivation for doing so would have been that neural networks provide fast and accurate data classification for images. Therefore it would have been obvious to combine Sminchises with Saban to obtain the invention.
Conclusion
For the prior art referenced and the prior art considered pertinent to Applicant’s disclosure but not relied upon, see PTO-892 “Notice of References Cited”.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JASON PRINGLE-PARKER whose telephone number is (571) 272-5690 and e-mail is jason.pringle-parker@uspto.gov. The examiner can normally be reached on 8:30am-5:00pm est Monday-Friday. If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, King Poon can be reached on (571) 270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, seehttp://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JASON A PRINGLE-PARKER/
Primary Examiner, Art Unit 2617