DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 5 and 18 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 5 recites the limitation "the motion of the object" in lines 2-3. There is insufficient antecedent basis for this limitation in the claim. For the purpose of prior art rejection, this limitation is interpreted as a movement or motion of the face.
Claim 18 recites the limitation "the motion of the object" in line 2. There is insufficient antecedent basis for this limitation in the claim. For the purpose of prior art rejection, this limitation is interpreted as a movement or motion of the face.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-5, 8, 10-11, and 14-20 are rejected under 35 U.S.C. 103 as being unpatentable over Dye et al. (US 2019/0342507, hereinafter Dye), and further in view of Tong et al. (US 2014/0198121, hereinafter Tong).
Regarding claim 1, Dye teaches an electronic device (portable multifunction device 100, fig. 1A and [0067]) comprising:
a camera (optical sensor 164, fig. 1A and [0067]; [0085]: In conjunction with imaging module 143 (also called a camera module), optical sensor 164 optionally captures still images or video. In some embodiments, an optical sensor is located on the back of device 100, opposite touch screen display 112 on the front of the device so that the touch screen display is enabled for use as a viewfinder for still and/or video image acquisition. In some embodiments, an optical sensor is located on the front of the device so that the user's image is, optionally, obtained for video conferencing while the user views the other video conference participants on the touch screen display);
a display (display system 112, fig. 1A and [0067]);
at least one processor (Processors 120, fig. 1A and [0067]); and
memory (memory 102, fig. 1A and [0067]) storing instructions that, when executed by the processor ([0072]: The one or more processors 120 run or execute various software programs and/or sets of instructions stored in memory 102 to perform various functions for device 100 and to process data), cause the electronic device to:
execute an application (launching a camera application; [0222]: In FIG. 6B, device 600 detects input 616 (e.g., a touch input on display 601) at a location corresponding to camera application affordance 609; [0223]: In FIG. 6C, in response to detecting input 616, device 600 launches a camera application associated with camera application affordance 609 and displays camera application user interface 615),
obtain a preview image through the camera ([0223]: Camera application user interface 615 includes image display region 620 which displays a representation of image data such as, for example, streamed image data (e.g., a live camera preview, live camera recording, or live video communications session) representing objects positioned within a field-of-view of a camera (e.g., a rear-facing camera or camera 602), or a media item such as, for example, a photograph or a video recording; [0229]: In FIG. 6E, in response to detecting input 623, device 600 activates camera 602 (e.g., switches from the rear-facing camera) and updates image display region 620 to display live camera preview 620-1 from camera 602),
display the preview image obtained through the camera on the display ([0223]: mage display region 620 which displays a representation of image data such as, for example, streamed image data (e.g., a live camera preview, live camera recording, or live video communications session) representing objects positioned within a field-of-view of a camera (e.g., a rear-facing camera or camera 602), or a media item such as, for example, a photograph or a video recording. In the embodiment illustrated in FIG. 6C, image display region 620 shows live camera preview 620-1′ from a rear-facing camera of device 600; [0229]: updates image display region 620 to display live camera preview 620-1 from camera 602, showing a representation of subject 632 positioned in the field-of-view of camera 602 and background 636 displayed behind subject 632),
receive an input for generating an avatar ([0232]: In FIG. 6F, device 600 detects input 626 (e.g., a tap gesture on display 601) on avatar effects affordance 624-1),
generate an avatar image which has features related to the face image ([0235]: a virtual avatar is a representation of the user that can be graphically depicted (e.g., a graphical representation of the user) … the avatar features correspond (e.g., are mapped) to one or more physical features of a user's face such that detected movement of the user's physical features (e.g., as determined based on a camera such as a depth sensing camera) affects the avatar feature (e.g., affects the feature's graphical representation); [0236]: In some examples, a user is able to manipulate characteristics or features of a virtual avatar using a camera sensor (e.g., camera module 143, optical sensor 164) and, optionally, a depth sensor (e.g., depth camera sensor 175). As a user's physical features (such as facial features) and position (such as head position, head rotation, or head tilt) changes, the electronic device detects the changes and modifies the displayed image of the virtual avatar to reflect the changes in the user's physical features and position. In some embodiments, the changes to the user's physical features and position are indicative of various expressions, emotions, context, tone, or other non-verbal communication. In some embodiments, the electronic device modifies the displayed image of the virtual avatar to represent these expressions, emotions, context, tone, or other non-verbal communication; [0237]: the virtual avatars are customizable avatars (e.g., customizable avatar 835). Customizable avatars are virtual avatars that can be selected and customized by a user, for example, to achieve a desired appearance (e.g., to look like the user). The customizable avatars generally have an appearance of a human character, rather than a non-human character such as an anthropomorphic construct of an animal or other nonhuman object. Additionally, features of the avatar can be created or changed, if desired, using an avatar editing user interface (e.g., such as the avatar editing user interface discussed below with respect to FIGS. 8AA-8AB). In some embodiments, customizable avatars can be created and configured to achieve a customized physical appearance, physical construct, or modeled behavior), and
display, through the display, the avatar image with the preview image such that the avatar image is located on the face image in the preview image (fig. 6G; [0239]: device 600 displays robot avatar 633 positioned on the face of subject 632 (e.g., the user) displayed in image display region 620. Device 600 displays the avatar so that it remains fixed on the subject's face as he moves within the field-of-view of camera 602. Device 600 also continually modifies the avatar based on any detected changes in the user's face, including pose of the face or changes in facial expressions, while maintaining the displayed subject's body and background 636 in the image shown in image display region 620).
Dye does not explicitly teach identifying a face image in the preview image in response to the input for generating the avatar, and generating an avatar image which has features related to the face image.
Tong teaches identifying a face image in the preview image (image analysis to detect a face/head in the live image/video) in response to the input for generating the avatar (fig. 7 steps 704-710; [0070]: Activation of the application may be followed by selection of an avatar 704. Selection of an avatar may include an interface being presented by the application to the user, the interface allowing the user to browse and select from predefined avatar files stored in an avatar database. The interface may further allow a user to select to have an avatar generated. Whether a user decides to have an avatar generated may be determined at operation 706. If it is determined that the user selects to have an avatar generated, as opposed to selecting a predefined avatar, camera in the device may then begin capturing images in operation 708. The images may be still images or live video (e.g., multiple images captured in sequence). In operation 710 image analysis may occur starting with detection/tracking of a face/head in the image. The detected face may then be analyzed in order to extract facial characteristics (e.g., facial landmarks, facial parameters, facial expression, etc.)), and generating an avatar image which has features related to the face image (fig. 7 step 712; [0070]: In operation 712, an avatar is generated based, at least in part, on the detected face/head position and/or facial characteristics). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Tong’s knowledge of generating an avatar that has features related to the identified face in the live image and modify the system of Dye because such a system enhance the user experience by providing quick animation response (in real-time or near real-time) and accurate and/or vivid representations of a user's face and facial expressions ([0013]).
Claims 14 and 20 are similar in scope to claim 1, and therefore the examiner provides similar rationale to reject these claims. Moreover, Dye teaches a non-transitory computer-readable medium ([0207]).
Regarding claim 2, the combination of Dye and Tong teaches the electronic device of claim 1, wherein the instructions, when executed by the processor, cause the electronic device to display an interface (avatar editing user interface 808, fig. 8AA) for changing at least one aspect of the avatar image (Dye – [0237]: features of the avatar can be created or changed, if desired, using an avatar editing user interface (e.g., such as the avatar editing user interface discussed below with respect to FIGS. 8AA-8AB). In some embodiments, customizable avatars can be created and configured to achieve a customized physical appearance, physical construct, or modeled behavior; Dye – [0240]: As shown in FIG. 6H, if device 600 detects that the user's head moves beyond the field-of-view of camera 602, device 600 displays, for example, prompt 638 containing representation 640 of the selected avatar (e.g., robot avatar) and message 642 instructing the user to reposition their head in the field-of-view of camera 602. In some embodiments, prompt 638 is displayed with a blurred background 644 (including blurring the user 632 and background 636). In some embodiments, displaying prompt 638 includes displaying an animation of the selected avatar returning to a center location of image display region 620, and showing a slowed movement of the avatar and its features as they appear to settle to a stop based on a physics model of the avatar features; Dye – [0330]: Edit affordance 804 can be selected (e.g., via input 807) to display avatar editing user interface 808 shown in FIG. 8AA. Avatar editing user interface 808 provides an interface for editing the avatar (e.g. customizable avatar 835) corresponding to the customizable avatar option (e.g., 830-5) associated with the selected edit affordance 804; Dye – [0332]: As shown in FIG. 8AA, avatar editing user interface 808 includes a representation of customizable avatar 835 and various avatar feature options 810 (e.g., face color options 810-1 and face shape options 810-2) that represent currently selected avatar feature options and available feature options that can be selected to modify avatar 835. The avatar feature options correspond to values for aspects of a particular avatar feature, specifically a currently selected avatar feature, such as an avatar head feature indicated by highlighted avatar head affordance 809-1. Avatar editing user interface 808 indicates a selected face color option 810-1a and a selected face shape option 810-2a, which are represented in avatar 835. The displayed avatar feature options can be changed by selecting a different avatar feature. For example, in FIG. 8 AB, device 600 updates avatar editing user interface 808 to display different avatar hair feature options (e.g., hair texture options 811-1 and hair color options 811-2) when the avatar hair affordance 809-2 is selected; Dye – [0333]: Device 600 modifies avatar 835 when different avatar feature options are selected. For example, in FIG. 8AB, device 600 detects selection of straight hair texture option 811-2a, and updates avatar 835 to have the selected straight hair texture option. In some embodiments, device 600 modifies the representation of avatar 835 shown in avatar editing user interface 808 in response to detected changes in a user's face).
Regarding claim 3, the combination of Dye and Tong teaches the electronic device of claim 1, wherein generating the avatar image includes selecting avatar features reflecting the features of the face image for inclusion in the avatar image, such that the generated avatar image resembles the face image ([0235]: a virtual avatar is a representation of the user that can be graphically depicted (e.g., a graphical representation of the user) … the avatar features correspond (e.g., are mapped) to one or more physical features of a user's face such that detected movement of the user's physical features (e.g., as determined based on a camera such as a depth sensing camera) affects the avatar feature (e.g., affects the feature's graphical representation); [0236]: In some examples, a user is able to manipulate characteristics or features of a virtual avatar using a camera sensor (e.g., camera module 143, optical sensor 164) and, optionally, a depth sensor (e.g., depth camera sensor 175). As a user's physical features (such as facial features) and position (such as head position, head rotation, or head tilt) changes, the electronic device detects the changes and modifies the displayed image of the virtual avatar to reflect the changes in the user's physical features and position. In some embodiments, the changes to the user's physical features and position are indicative of various expressions, emotions, context, tone, or other non-verbal communication. In some embodiments, the electronic device modifies the displayed image of the virtual avatar to represent these expressions, emotions, context, tone, or other non-verbal communication; [0237]: the virtual avatars are customizable avatars (e.g., customizable avatar 835). Customizable avatars are virtual avatars that can be selected and customized by a user, for example, to achieve a desired appearance (e.g., to look like the user). The customizable avatars generally have an appearance of a human character, rather than a non-human character such as an anthropomorphic construct of an animal or other nonhuman object. Additionally, features of the avatar can be created or changed, if desired, using an avatar editing user interface (e.g., such as the avatar editing user interface discussed below with respect to FIGS. 8AA-8AB). In some embodiments, customizable avatars can be created and configured to achieve a customized physical appearance, physical construct, or modeled behavior; Tong - [0016]: the system allows generation, rendering and animation of a two-dimensional (2-D) avatar of a user's face, wherein the 2-D avatar represents a user's basic face shape and key facial characteristics, including, but not limited to, position and shape of the eyes, nose, mouth, and face contour. The system is further configured to provide avatar animation based at least in part on the detected key facial characteristics of the user in real-time or near real-time during active communication and interaction; Tong – [0038]: The avatar control module 210 may further be configured to generate parameters for animating an avatar. Animation, as referred to herein, may be defined as altering the appearance of an image/model. A single animation may alter the appearance of a 2-D still image, or multiple animations may occur in sequence to simulate motion in the image (e.g., head turn, nodding, talking, frowning, smiling, laughing, etc.). A change in position of the detected face and/or facial characteristic 206 may be may converted into parameters that cause the avatar's features to resemble the features of the user's face).
Regarding claim 4, the combination of Dye and Tong teaches the electronic device of claim 2, wherein the instructions, when executed by the processor, cause the electronic device to: based on an input for selecting the avatar image located on the face image in the image (Dye – fig. 8Z; Dye – [0330]: a selection of a different customizable avatar option (e.g., customizable avatar option 830-5) and the corresponding displayed customizable avatar 835), display an interface for editing the avatar (Dye - fig. 8AA; Dye – [0330]: When customizable avatar option 830-5 is displayed in selection region 829, edit affordance 804 is displayed. Edit affordance 804 can be selected (e.g., via input 807) to display avatar editing user interface 808 shown in FIG. 8AA), the interface including a plurality of UI items for changing an attribute of the avatar (Dye – fig. 8AA; Dye – [0332]: As shown in FIG. 8AA, avatar editing user interface 808 includes a representation of customizable avatar 835 and various avatar feature options 810 (e.g., face color options 810-1 and face shape options 810-2) that represent currently selected avatar feature options and available feature options that can be selected to modify avatar 835), and based on an input for selecting a UI item included in the interface, modify a visual appearance of the avatar (fig. 8AC) by incorporating an attribute of a selected UI item (selecting straight hair texture option 811-2a, fig. 8AB) into the avatar (Dye – [0332]: As shown in FIG. 8AA, avatar editing user interface 808 includes a representation of customizable avatar 835 and various avatar feature options 810 (e.g., face color options 810-1 and face shape options 810-2) that represent currently selected avatar feature options and available feature options that can be selected to modify avatar 835. The avatar feature options correspond to values for aspects of a particular avatar feature, specifically a currently selected avatar feature, such as an avatar head feature indicated by highlighted avatar head affordance 809-1. Avatar editing user interface 808 indicates a selected face color option 810-1a and a selected face shape option 810-2a, which are represented in avatar 835. The displayed avatar feature options can be changed by selecting a different avatar feature. For example, in FIG. 8 AB, device 600 updates avatar editing user interface 808 to display different avatar hair feature options (e.g., hair texture options 811-1 and hair color options 811-2) when the avatar hair affordance 809-2 is selected; Dye – [0333]: Device 600 modifies avatar 835 when different avatar feature options are selected. For example, in FIG. 8AB, device 600 detects selection of straight hair texture option 811-2a, and updates avatar 835 to have the selected straight hair texture option. In some embodiments, device 600 modifies the representation of avatar 835 shown in avatar editing user interface 808 in response to detected changes in a user's face).
Regarding claim 5, the combination of Dye and Tong teaches the electronic device of claim 2, wherein the instructions, when executed by the processor, cause the electronic device to change the type of the avatar that simulates the motion of the object (movement of the user), based on an input selecting a UI item included in the interface (Dye – [0234]: Avatar options 630 correspond to a virtual avatar visual effect applied to a representation of the subject in image display region 620. Specifically, each avatar option 630 corresponds to a virtual avatar that, when selected, is transposed onto the face of the subject in the image display region, while other portions of the image in image display region (such as a background or other portions of the user, such as their body) remain displayed. A user (e.g., subject 632) positioned in the field-of-view of camera 602 can control visual aspects of the virtual avatar by changing the pose (e.g., rotation or orientation) of their face, including moving various facial features (e.g., winking, sticking out their tongue, smiling, etc.); Dye – [0235]: the avatar features correspond (e.g., are mapped) to one or more physical features of a user's face such that detected movement of the user's physical features (e.g., as determined based on a camera such as a depth sensing camera) affects the avatar feature (e.g., affects the feature's graphical representation); Dye – [0252]: In FIG. 6AD, device 600 detects input 668 (e.g. a tap gesture on display 601) on close icon 670 to close sticker options menu 656 and return to the camera application user interface 615 as shown in FIG. 6AE, which includes camera options region 625 showing capture affordance 621, visual effects option affordances 624, and highlighted effects affordance 622 indicating visual effects are enabled for display. In FIG. 6AE, device 600 shows the representation of the user in the image on image display region 620, but modified with visual effects that include robot avatar 633 (e.g., selected from avatar options menu 628) and helmet sticker 658-1 (e.g., selected from sticker options menu 656). The display of the visual effects is dynamic. For example, device 600 continuously modifies robot avatar 633 in response to detected changes in the user's face. In some embodiments, positioning of sticker 658-1 is also dynamic and changing based on the detected movements of the user; Dye - [0331]: device 600 modifies avatar 835 to mirror the real-time movements of the user; Dye – [0338]: n FIG. 8AL, device 600 shows that subject 832 is shifted back to the center of the screen with their head tilted. In response to detecting this movement of subject 832, device 600 displays helmet sticker 858-1 above customizable avatar 835 in FIG. 8AL and shifted laterally (from the position shown in FIG. 8AK) based on the movement of the subject's head to the center of the screen. Device 600 displays helmet sticker 858-1 maintaining the relative spacing with respect to avatar 835. However, helmet sticker 858-1 is not rotated (e.g., tilted) in response to the rotation (e.g., tilting) of the subject's head, whereas avatar 835 is rotated (e.g., tilted), as shown in FIG. 8AL).
Regarding claim 8, the combination of Dye and Tong teaches the electronic device of claim 1, wherein the instructions, when executed by the processor, cause the electronic device to: identify the face image in the preview image (Tong - [0029]: The device 102 may further include a face detection module 204 configured to identify and track a head, face and/or facial region within image(s) provided by camera 104 and to determine one or more facial characteristics of the user (i.e., facial characteristics 206). For example, the face detection module 204 may include custom, proprietary, known and/or after-developed face detection code (or instruction sets), hardware, and/or firmware that are generally well-defined and operable to receive a standard format image (e.g., but not limited to, a RGB color image) and identify, at least to a certain extent, a face in the image; Tong – [0030]: The face detection module 204 may also be configured to track the detected face through a series of images (e.g., video frames at 24 frames per second) and to determine a head position based on the detected face, as well as changes, such as, for example, movement, in facial characteristics of the user (e.g., facial characteristics 206); Tong – [0047]: The face detection module 204a may be configured to receive one or more images from the camera 104 via the camera and audio framework module 200 and identify, at least to a certain extent, a face (or optionally multiple faces) in the image; Tong – [0070]: In operation 710 image analysis may occur starting with detection/tracking of a face/head in the image), generate a synthesized preview image by synthesizing the avatar image with the preview image such that the avatar image is located on the face image in the preview image (Dye - [0234]: each avatar option 630 corresponds to a virtual avatar that, when selected, is transposed onto the face of the subject in the image display region, while other portions of the image in image display region (such as a background or other portions of the user, such as their body) remain displayed; Dye – [0234]: Because robot avatar 630-3 is selected in FIG. 6G, device 600 displays robot avatar 633 positioned on the face of subject 632 (e.g., the user) displayed in image display region 620. Device 600 displays the avatar so that it remains fixed on the subject's face as he moves within the field-of-view of camera 602).
Regarding claim 10, the combination of Dye and Tong teaches the electronic device of claim 1, wherein the instructions, when executed by the processor, cause the electronic device to: extract features of a user's face from the face image in the preview image (Tong - [0031]: identify, at least to a certain extent, one or more facial characteristics 206 in the image; Tong – [0032]: facial characteristics 206 may include features of the face, including, but not limited to, the location and/or shape of facial landmarks such as eyes, nose, mouth, facial contour, etc., as well as movement of such landmarks; Tong – [0047]: The facial characteristics 206 may include may include features of the face, including, but not limited to, the location and/or shape of facial landmarks such as eyes, nose, mouth, facial contour, eyebrows, etc. ; Tong – [0055]: the facial expression detection module 312 may determine size and/or position of the facial features (e.g., forehead, chin, eyes, nose, mouth, cheeks, facial contour, etc.); Tong - [0070]: The detected face may then be analyzed in order to extract facial characteristics (e.g., facial landmarks, facial parameters, facial expression, etc.)), and generate the avatar image having features similar to the face image based on the extracted features of the user's face (Dye – [0235]: the virtual avatar includes an avatar face having one or more avatar features (e.g., avatar facial features). In some embodiments, the avatar features correspond (e.g., are mapped) to one or more physical features of a user's face such that detected movement of the user's physical features (e.g., as determined based on a camera such as a depth sensing camera) affects the avatar feature (e.g., affects the feature's graphical representation); Dye - [0236]: In some examples, a user is able to manipulate characteristics or features of a virtual avatar using a camera sensor (e.g., camera module 143, optical sensor 164) and, optionally, a depth sensor (e.g., depth camera sensor 175). As a user's physical features (such as facial features) and position (such as head position, head rotation, or head tilt) changes, the electronic device detects the changes and modifies the displayed image of the virtual avatar to reflect the changes in the user's physical features and position. In some embodiments, the changes to the user's physical features and position are indicative of various expressions, emotions, context, tone, or other non-verbal communication. In some embodiments, the electronic device modifies the displayed image of the virtual avatar to represent these expressions, emotions, context, tone, or other non-verbal communication; Tong – [0017]: The application may be configured to allow a user to generate a 2-D avatar based on user's face and facial characteristics for display on a remote device, in a virtual space, etc. The camera may be configured to start capturing images and facial detection is then performed on the captured images, and facial characteristics are determined Avatar selection is then performed, wherein a user may select between a predefined 2-D avatar and generation of a 2-D avatar based on the facial characteristics of the user. Any detected face/head movements, including movement of one or more of the user's facial characteristics, including, but not limited to, eyes, nose and mouth and/or changes in facial features are then converted into parameters usable for animating the 2-D avatar on the at least one other device, within the virtual space, etc.; Tong – [0032]: avatar animation may be based on sensed facial actions (e.g., changes in facial characteristics 206); Tong – [0037]: generate a 2-D avatar based on the face/head position and/or facial characteristics 206 detected by face detection module 204; Tong – [0038]: A change in position of the detected face and/or facial characteristic 206 may be may converted into parameters that cause the avatar's features to resemble the features of the user's face; Tong – [0070]: In operation 712, an avatar is generated based, at least in part, on the detected face/head position and/or facial characteristics).
Regarding claim 11, the combination of Dye and Tong teaches the electronic device of claim 10, wherein the extracted features correspond to a face contour, a pupil, a shape or size of eyes, a nose, a mouth, and/or a skin color included in the face image (Tong – [0030]: skin tone analysis; Tong – [0047]: The facial characteristics 206 may include may include features of the face, including, but not limited to, the location and/or shape of facial landmarks such as eyes, nose, mouth, facial contour, eyebrows, etc.; Tong – [0050]: the landmark detection module 304 may be based on heuristic analysis and may be configured to identify and/or analyze the relative position, size, and/or shape of the forehead, eyes (and/or the corner of the eyes), nose (e.g., the tip of the nose), chin (e.g. tip of the chin), eyebrows, cheekbones, jaw, and facial contour. The eye-corners and mouth corners may also be detected using Viola-Jones based classifier; Tong - [0070]: The detected face may then be analyzed in order to extract facial characteristics (e.g., facial landmarks, facial parameters, facial expression, etc.); Tong – [0055]: the facial expression detection module 312 may determine size and/or position of the facial features (e.g., forehead, chin, eyes, nose, mouth, cheeks, facial contour, etc.)).
Claims 15, 16, 17, 18, and 19 are similar in scope to claims 2-3, 4, 10-11, 5, and 8, respectively, and therefore the examiner provides similar rationale to reject these claims.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Dye, in view of Tong, and further in view of Wilson et al. (US 2018/0091732, hereinafter Wilson).
Regarding claim 6, the combination of Dye and Tong teaches the electronic device of claim 2, wherein the instructions, when executed by the processor, cause the electronic device to change at least one of the shape or size of eyes (EYES, fig. 8AA and fig. 8AB, Dye), (LIPS, fig. 8AB, Dye), hair color (hair color options 811-2, fig. 8AA and [0332], Dye), hair style (hair texture options 811-1, fig. 8AA and [0332], Dye), face type (face shape options 810-2, fig. 8AA and [0332], Dye), a skin color (face color options 810-1, fig. 8AA, Dye), and costume of the avatar (adding a helmet to the avatar by selecting a helmet sticker as shown in fig. 6V-fig. 6AD, Dye), based on an input selecting a UI item included in the interface (Dye – [0332]: As shown in FIG. 8AA, avatar editing user interface 808 includes a representation of customizable avatar 835 and various avatar feature options 810 (e.g., face color options 810-1 and face shape options 810-2) that represent currently selected avatar feature options and available feature options that can be selected to modify avatar 835. The avatar feature options correspond to values for aspects of a particular avatar feature, specifically a currently selected avatar feature, such as an avatar head feature indicated by highlighted avatar head affordance 809-1. Avatar editing user interface 808 indicates a selected face color option 810-1a and a selected face shape option 810-2a, which are represented in avatar 835. The displayed avatar feature options can be changed by selecting a different avatar feature. For example, in FIG. 8 AB, device 600 updates avatar editing user interface 808 to display different avatar hair feature options (e.g., hair texture options 811-1 and hair color options 811-2) when the avatar hair affordance 809-2 is selected).
The combination of Dye and Tong does not explicitly teach to change the size or shape of the nose based on an input selecting a UI item included in the interface.
Wilson teaches to change the size or shape of the nose based on an input selecting a UI item included in the interface ([0275]: For example, when the first feature is hair style and the second feature is nose style, display of representation for different hair style options are replaced with display of affordances for different nose style options; [0276]: In accordance with a determination that the input (e.g., 923) corresponds to selection of the second option (e.g., 917) for the first avatar feature in the feature-option control region (1014), the electronic device updates (1016) the feature-option control region to indicate that the second option for the first avatar feature is currently selected (FIG. 9C); and updates (1018) the avatar to change the appearance of the first avatar feature in accordance with the second option for the first avatar feature (e.g., the hair of avatar 901 is updated from FIG. 9B to FIG. 9C) (e.g., changing the avatar based on characteristics of the feature option or if the feature selection is for nose styles, the shape of a nose changes to correspond to the selected nose style); [0355]: If pinch gesture 1526 in FIG. 15D is received when the nose is currently selected for editing (representation 1509 corresponds to the nose), device 600 changes the nose width in accordance with the pinch gesture (e.g., an expanding pinch widens the nose while a squeezing pinch narrows the nose). The results of expanding pinch 1526 of FIG. 15D on avatar 1501 are depicted in FIG. 15E. Specifically, in FIG. 15E, the nose on avatar 1501 has widened as compared to the nose of avatar 1501 in FIG. 15D. The effect (changing width) of pinch gesture 1526 (FIG. 15D) is different than the effect (changing scale) of pinch gesture 1513 (FIG. A) based on different avatar features being selected for editing). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Wilson’s knowledge of using editing user interface to change the nose of the avatar as taught and modify the system of Dye and Tong because such a system provides a faster and more efficient interfaces for avatar creation and editing, thereby increasing the effectiveness, efficiency, and user satisfaction with such devices ([0015]).
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Dye, in view of Tong, and further in view of Shuster (US 2009/0112906).
Regarding claim 7, the combination of Dye and Tong does not explicitly teach the electronic device of claim 2, wherein the instructions, when executed by the processor, cause the electronic device to determine a gender, an age and/or a race of a virtual character based on user inputs.
Shuster teaches to cause the electronic device to determine a gender, an age and/or a race of a virtual character based on user inputs ([0040]: An avatar manager 412 may be provided to manage configuration and control of avatar data and other data objects that are assigned stable properties or functions in response to user input. For example, input from remote clients may be used to determine gender and racial characteristics of each avatar, with each client specifying its own avatar characteristics). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Shuster’s knowledge of using user input to determine the gender or race of the avatar as taught and modify the system of Dye and Tong because such a system enhances a user’s experience by ensuring that the configurations of avatars and other client-controlled avatar data are in accordance with the configurations specified by user input ([0040]).
Claims 12-13 are rejected under 35 U.S.C. 103 as being unpatentable over Dye, in view of Tong, and further in view of Li et al. (US 2015/0325029, hereinafter Li).
Regarding claim 12, the combination of Dye and Tong does not explicitly teach the electronic device of claim 11, wherein the instructions, when executed by the processor, cause the electronic device to: determine a tracking level with respect to the face image, and extract the features of the user's face from the face image based on the determined tracking level.
Li teaches to: determine a tracking level with respect to the face image (tracking/detecting level is functionally analogous to the image sizes of the user image; [0016]: detection /tracking logic 203 including meshing and mapping module 205; computation engine 207 including reference value calculation logic 209, image size calculation logic 211, and avatar scale factor calculation logic 213; [0017]: The detecting and tracking of the user's face and its movements and expressions as performed by detection/tracking logic 203 may include detecting the user's face and determining various features of the face, such as positions of feature points, which may then be used to determine facial expression movements and head rigid movements; [0026]: Using detection/tracking logic 203, the avatar for the user's face is generated and its position in live video frames is located. This data is then provided to computation engine 207 where further calculations are performed, such as to derive the face size of the user in terms of face width and face height (e.g., face size=w*h, where w represents face width, and h represents face height); [0027]: image size calculation logic 211 may dynamically determine the changing image or face size of the current user and calculate, in real-time, the changing image or face size value. In one embodiment, these image size values along with the reference value are then used by avatar scale factor calculation logic 213 to determine the exact location of the user's face with respect to computing device 100, such as whether and how much the user is zooming-in or zooming-out; [0040]: At block 310, the user's image, such as the user's face, is detected and tracked while its position with respect to the computing device (such as getting closer or moving away, etc.) is located in live, real-time, video frame. At block 315, in one embodiment, image sizes, such as face sizes of the user's face, are determined and computed into an image size value, such as face size value, which is then divided by a pre-determined or pre-calculated reference value to derive or obtain an avatar scale factor; [0057]: detection/tracking logic to track changes in size of the user image, wherein tracking the changes includes locating one or more positions of the user image within each of the plurality of video frames), and extract the features of the user's face from the face image based on the determined tracking level (expressions are determined from the face image based on the changes in the size of the user image; [0029]: extraction and deformation logic 215 may access and preprocess the database 240 by extraction of the facial features for each frame of the database; more particularly, for each frame in the database, extraction and deformation logic 215 may compute multiple facial features; [0040]: At block 320, facial points are extracted by a 3D mesh; [0057]: detection/tracking logic to track changes in size of the user image, wherein tracking the changes includes locating one or more positions of the user image within each of the plurality of video frames; computation engine to compute, in real-time, user performances based on the changes in the size of the user image over the plurality of video frames; [0063]: user performances are dynamically computed based on the avatar scale factor, wherein the user performances comprise zooming-in of the user or zooming-out of the user with respect to the image source of the apparatus, the user performances further comprising one or more human expressions including one or more of laugh, anger, surprise, sad, fear, disgust, smile, frown, cry, happy, speak, silent, eat, drink, sing, yawn, and sneeze). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Li’s knowledge of using image sizes of the user image to extract facial expressions as taught and modify the system of Dye and Tong because such a system facilitates dynamic simulation of avatars corresponding to changing user performances as detected by computing devices ([0001]).
Regarding claim 13, the combination of Dye, Tong and Li teaches the electronic device of claim 12, wherein the tracking level is determined based on a size of an imaging area corresponding to the face image in the preview image (tracking/detecting level is functionally analogous to the image sizes of the user image; Li - [0016]: detection /tracking logic 203 including meshing and mapping module 205; computation engine 207 including reference value calculation logic 209, image size calculation logic 211, and avatar scale factor calculation logic 213; Li - [0017]: The detecting and tracking of the user's face and its movements and expressions as performed by detection/tracking logic 203 may include detecting the user's face and determining various features of the face, such as positions of feature points, which may then be used to determine facial expression movements and head rigid movements; Li - [0026]: Using detection/tracking logic 203, the avatar for the user's face is generated and its position in live video frames is located. This data is then provided to computation engine 207 where further calculations are performed, such as to derive the face size of the user in terms of face width and face height (e.g., face size=w*h, where w represents face width, and h represents face height); Li - [0027]: image size calculation logic 211 may dynamically determine the changing image or face size of the current user and calculate, in real-time, the changing image or face size value. In one embodiment, these image size values along with the reference value are then used by avatar scale factor calculation logic 213 to determine the exact location of the user's face with respect to computing device 100, such as whether and how much the user is zooming-in or zooming-out; Li - [0040]: At block 310, the user's image, such as the user's face, is detected and tracked while its position with respect to the computing device (such as getting closer or moving away, etc.) is located in live, real-time, video frame. At block 315, in one embodiment, image sizes, such as face sizes of the user's face, are determined and computed into an image size value, such as face size value, which is then divided by a pre-determined or pre-calculated reference value to derive or obtain an avatar scale factor; Li - [0057]: detection/tracking logic to track changes in size of the user image, wherein tracking the changes includes locating one or more positions of the user image within each of the plurality of video frames; computation engine to compute, in real-time, user performances based on the changes in the size of the user image over the plurality of video frames; Li - [0063]: user performances are dynamically computed based on the avatar scale factor, wherein the user performances comprise zooming-in of the user or zooming-out of the user with respect to the image source of the apparatus, the user performances further comprising one or more human expressions including one or more of laugh, anger, surprise, sad, fear, disgust, smile, frown, cry, happy, speak, silent, eat, drink, sing, yawn, and sneeze).
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Liu et al (US 2016/0179455) describes the avatar can be presented at a particular maximum size when process 400 determines that the user device is more than a threshold distance from the public display device (e.g., based on the location information received at 406). As the user device moves closer to a public display device presenting the avatar, process 400 can cause the size of the avatar to decrease, as the user can more easily perceive the avatar as they move closer.
Wu et al. (US 2017/0286897) describes sensing devices 112 may capture less-detailed data over longer distances as shown at 118 to at least detect trackable targets 118 moving into and out of the aisle. Sensing devices 114 may capture more-detailed data over shorter distances as illustrated at 122 for performing higher level detection such as gender determination and age estimation based on, for example, face detection, feature extraction from the detected face, etc.
Allowable Subject Matter
Claims 9 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding claim 9, Tong teaches to detect the face image within the preview image (fig. 7 steps 704-710; [0070]: Activation of the application may be followed by selection of an avatar 704. Selection of an avatar may include an interface being presented by the application to the user, the interface allowing the user to browse and select from predefined avatar files stored in an avatar database. The interface may further allow a user to select to have an avatar generated. Whether a user decides to have an avatar generated may be determined at operation 706. If it is determined that the user selects to have an avatar generated, as opposed to selecting a predefined avatar, camera in the device may then begin capturing images in operation 708. The images may be still images or live video (e.g., multiple images captured in sequence). In operation 710 image analysis may occur starting with detection/tracking of a face/head in the image. The detected face may then be analyzed in order to extract facial characteristics (e.g., facial landmarks, facial parameters, facial expression, etc.)).
Li teaches to detect face image within the preview image (fig. 3 step 310), and determine the size of the face image (fig. 3 step 315).
Amayeh et al. (US 10,976,549) describes to determine whether the face image is a predetermined size, based on a size of the face image (claim 16: analyze the pair of acquired images to estimate a size of the face of the user in an imaging plane associated with the pair of acquired images; and determine whether the size of the face of the user is less than a threshold size, wherein the threshold size is associated with the distance being outside of the certain range of distances).
However, none of the cited prior art references of record, teach, either individually or in combination, “determine a mask mode for replacing the face image in the preview image using the avatar image, when the face image is greater than or equal to the predetermined size”.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JWALANT B AMIN whose telephone number is (571)272-2455. The examiner can normally be reached Monday-Friday 10am - 630pm CST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Said Broome can be reached at 571-272-2931. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JWALANT AMIN/Primary Examiner, Art Unit 2612
Prosecution Insights