DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. CN202211687713.0, filed on 27th December, 2022.
Claim Objections
Claims 7, 10, 11, 16, 21, and 24 are objected to because of the following informalities:
Claim 7 recites the limitation(s): "the glasses wearing posture" on PG(s). 4, Line(s) 21; examiner suggests amending this to: "a glasses wearing posture";
Claim 7 recites the limitation(s): "the glasses type" on PG(s). 4, Line(s) 22; examiner suggests amending this to: "a glasses type";
Claim 10 recites the limitation(s): "a second brightness attenuation coefficient, which represents the change of the imaging brightness parameter of the virtual face model before and after the glasses are worn acquired by the virtual image acquisition device" on PG(s). 5, Line(s) 19; examiner suggests amending this to: "a second brightness attenuation coefficient, which represents the change of the imaging brightness parameter of the virtual face model before and after the glasses are worn, acquired by the virtual image acquisition device";
Claim 11 recites the limitation(s): "if the size difference between the glass light-transmitting area and the light transmission modified area is equal to or larger than a preset difference" on PG(s). 6, Line(s) 3-4; examiner suggests amending this to: "if a size difference between the glasses light-transmitting area and the light transmission modified area is equal to or larger than a preset difference"; additionally, the term "size difference" is unclear in the current claim language;
Claim 16 recites the limitation(s): "the computer instructions are used to cause the computer to perform" on PG(s). 6, Line(s) 19; examiner suggests amending this to: "the computer instructions are used to cause a computer to perform";
Claim 21 recites the limitation(s): "the brightness attenuation parameter further includes a second brightness attenuation coefficient, which represents the change of the imaging brightness parameter of the virtual face model before and after the glasses are worn acquired by the virtual image acquisition device" on PG(s). 8, Line(s) 11; examiner suggests amending this to: "the brightness attenuation parameter further includes a second brightness attenuation coefficient, which represents the change of the imaging brightness parameter of the virtual face model before and after the glasses are worn, acquired by the virtual image acquisition device"; and
Claim 24 recites the limitation(s): "the brightness attenuation parameter further includes a second brightness attenuation coefficient, which represents the change of the imaging brightness parameter of the virtual face model before and after the glasses are worn acquired by the virtual image acquisition device" on PG(s). 9, Line(s) 24; examiner suggests amending this to: "the brightness attenuation parameter further includes a second brightness attenuation coefficient, which represents the change of the imaging brightness parameter of the virtual face model before and after the glasses are worn, acquired by the virtual image acquisition device".Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 11 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 11 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being incomplete for omitting essential structural cooperative relationships of elements, such omission amounting to a gap between the necessary structural connections. See MPEP § 2172.01. The omitted structural cooperative relationships are: "perform image inpainting on the light transmission modified area based on the light transmission modified area". This step appears to be missing a step and/or element as claim language indicates that the "light transmission modified area" has been modified. Although the prior claim limitation states: "adjusting a position of the glasses light-transmitting area based on an image distortion parameter, to obtain a light transmission modified area", which indicates a modification step, the claim limitation of "perform image inpainting on the light transmission modified area based on the light transmission modified area" indicates that the "light transmission modified area" has another modification step that is not disclosed in both the claims and the specification. Furthermore, the specification does not explain how the "light transmission modified area" is based on itself. Therefore, Claim 11 is rendered indefinite.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-10, 12, 15-16, and 19-24 are rejected under 35 U.S.C. 103 as being unpatentable over Ye et al. (US 20160035133 A1), hereinafter referenced as Ye, in view of Kornilov et al. (US 20180374261 A1), hereinafter referenced as Kornilov.
Regarding Claim 1, Ye discloses an image synthesis method (Ye, [0019]: teaches a method for creating a real-time try-on experience <read on image synthesis> of a user wearing a pair of 3D eyeglasses), comprising:
[[acquiring a brightness parameter of a virtual face model by using a virtual image acquisition device with the assistance of a virtual light source;]]
determining a brightness attenuation parameter corresponding to glasses, which is used for image synthesis, based on a change of the brightness parameter of the virtual face model before and after wearing the glasses (Ye, [0047]: teaches changing the transparency <read on determined brightness attenuation parameter> of the lenses of a pair of 3D virtual eyeglasses based on the alpha channel value; FIG. 11b teaches a generated 3D face shape model <read on virtual face model> that is glasses-less <read on virtual face model before wearing glasses> from a facial tracking device, where the tinted glasses are applied at step S220 <read on virtual face model after wearing glasses>, which changes the brightness of the eye-area region of the generated 3D face shape model, thus synthesizing an image of a 3D face shape model wearing tinted glasses; Note: it is noted that the term "brightness attenuation" is interpreted as a decreased brightness value).
PNG
media_image1.png
538
354
media_image1.png
Greyscale
However, Ye does not expressly disclose
acquiring a brightness parameter of a virtual face model by using a virtual image acquisition device with the assistance of a virtual light source.
Kornilov discloses
acquiring a brightness parameter of a virtual face model by using a virtual image acquisition device with the assistance of a virtual light source (Kornilov, [0032]: teaches a client device 104 including a camera device <read on virtual image acquisition device> to track and capture a set of images of the user's face/head; [0079]: teaches determining a set of lighting parameters <read on brightness parameter> for each segment of the identified area of the user's face/head <read on virtual face model> in the image, where each segment includes "a direction of the light and a source of the light <read on virtual light source>").
Kornilov is analogous art with respect to Ye because they are from the same field of endeavor, namely providing a virtual try-on glasses experience. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to have the system determine a set of lighting parameters for the user's face as taught by Kornilov into the teaching of Ye. The suggestion for doing so would allow for the generation of a lighting model, which would provide more accurate casted shadows from the virtual eyeglasses, further making the virtual glasses appear more realistic. Therefore, it would have been obvious to combine Kornilov with Ye.
Regarding Claim 15, it recites the limitations that are similar in scope to Claim 1, but in an electronic device. As shown in the rejection, the combination of Ye and Kornilov discloses the limitations of Claim 1. Additionally, Ye discloses an electronic device (Ye, [0053]: teaches an augmented reality (AR) system 300 that includes electronic device 50 as shown in FIG. 13), comprising:
PNG
media_image2.png
592
357
media_image2.png
Greyscale
a processor (Ye, [0053]: teaches the electronic device 50 including processor 52); and,
a memory for storing a program (Ye, [0053]: teaches the electronic device 50 including memory 60 that stores eyeglass data for a 3D virtual eyeglasses try-on simulation program to access); wherein
the program comprises instructions which, when executed by the processor, cause the processor to perform (Ye, [0053]: teaches processor 52 of AR system 300 running the 3D virtual eyeglasses try-on simulation program 66, which performs tasks <read on instructions> that constructs a 3D face/head model wearing a pair of eyeglasses):…
Thus, Claim 15 is met by Ye according to the mapping presented in the rejection of Claim 1, given the image synthesis method corresponds to an electronic device.
Regarding Claim 16, it recites the limitations that are similar in scope to Claim 1, but in a non-transitory computer-readable storage medium. As shown in the rejection, the combination of Ye and Kornilov discloses the limitations of Claim 1. Additionally, Ye discloses a non-transitory computer-readable storage medium storing computer instructions (Ye, [0053]: teaches an augmented reality (AR) system 300 that includes electronic device 50, memory 60 <read on non-transitory computer-readable storage medium>, and a 3D virtual eyeglasses try-on simulation program to perform tasks <read on computer instructions>), wherein
the computer instructions are used to cause the computer to perform (Ye, [0053]: teaches processor 52 of electronic device 50 <read on computer> running the 3D virtual eyeglasses try-on simulation program 66, which performs tasks <read on computer instructions> that constructs a 3D face/head model wearing a pair of eyeglasses):…
Thus, Claim 16 is met by Ye according to the mapping presented in the rejection of Claim 1, given the image synthesis method corresponds to a non-transitory computer-readable storage medium.
Regarding Claim 2, the combination of Ye and Kornilov discloses the image synthesis method of Claim 1. Additionally, Ye further discloses wherein, the brightness parameter of the virtual face model includes
[[a first imaging brightness parameter of the virtual light source acquired by a virtual image acquisition device located on the virtual face model;]]
the brightness attenuation parameter at least includes a first brightness attenuation coefficient, which is determined based on the first imaging brightness parameters before and after the glasses are worn (Ye, [0047]: teaches changing the transparency <read on brightness attenuation parameter> of the lenses of a pair of 3D virtual eyeglasses based on the alpha channel value <read on first brightness attenuation coefficient>; FIG. 11b teaches a generated 3D face shape model that is glasses-less <read on before wearing glasses> from a facial tracking device, where the tinted glasses are applied at step S220 <read on after wearing glasses>, which changes the brightness of the eye-area region of the generated 3D face shape model).
However, Ye does not expressly disclose
a first imaging brightness parameter of the virtual light source acquired by a virtual image acquisition device located on the virtual face model.
Kornilov discloses
a first imaging brightness parameter of the virtual light source acquired by a virtual image acquisition device located on the virtual face model (Kornilov, [0032]: teaches a client device 104 including a camera device <read on virtual image acquisition device> to track and capture a set of images of the user's face/head; [0079]: teaches determining a set of lighting parameters <read on first imaging brightness parameter> that are associated with each predetermined set of segments of the identified area of the user's face <read on located on virtual face model>, where each segment includes "a direction of the light and a source of the light <read on virtual light source>"; Note: it is noted that a "set of lighting parameters" is interpreted to be more than one lighting parameter, where one of the lighting parameters is interpreted to be the first imaging brightness parameter).
Kornilov is analogous art with respect to Ye because they are from the same field of endeavor, namely providing a virtual try-on glasses experience. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to have the system determine a set of lighting parameters for the user's face as taught by Kornilov into the teaching of Ye. The suggestion for doing so would allow for the generation of a lighting model, which would provide more accurate casted shadows from the virtual eyeglasses, further making the virtual glasses appear more realistic. Therefore, it would have been obvious to combine Kornilov with Ye.
Regarding Claim 3, the combination of Ye and Kornilov discloses the image synthesis method of Claim 2. Ye does not expressly disclose the limitations of Claim 3; however, Kornilov discloses wherein, the acquiring a brightness parameter of a virtual face model by using a virtual image acquisition device with the assistance of a virtual light source, comprises:
binding the virtual image acquisition device at the position of the virtual face model (Kornilov, [0081]: teaches determining a lighting model based on the sets of lighting parameters determined <read on binding> from the plurality of images, which are images of the user's face/head <read on virtual face model> taken at differing positions via a camera <read on virtual image acquisition device>; Note: it is noted that "binding" is being interpreted broadly), and
acquiring the first imaging brightness parameter based on the virtual image acquisition device when the virtual face model is irradiated by the virtual light source (Kornilov, [0081]: teaches the average of the lighting parameters <read on first imaging brightness parameter>, such as the average source of the light <read on virtual light source> and the average intensity of the light <read on virtual face model being irradiated>, being used to determine the lighting model; Note: it is noted that "irradiated" is being interpreted as lighting up an area/region).
Kornilov is analogous art with respect to Ye because they are from the same field of endeavor, namely providing a virtual try-on glasses experience. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to have the system determine a set of lighting parameters for the user's face as taught by Kornilov into the teaching of Ye. The suggestion for doing so would allow for the generation of a lighting model, which would provide more accurate casted shadows from the virtual eyeglasses, further making the virtual glasses appear more realistic. Therefore, it would have been obvious to combine Kornilov with Ye.
Regarding Claim 4, the combination of Ye and Kornilov discloses the image synthesis method of Claim 2. Additionally, Ye further discloses wherein
[[the brightness parameter of the virtual face model includes a second imaging brightness parameter of the virtual light source located in the virtual face model at the virtual image acquisition device;]]
the brightness attenuation parameter further includes a second brightness attenuation coefficient, which is determined based on the second imaging brightness parameters before and after the glasses are worn (Ye, [0047]: teaches changing the transparency <read on brightness attenuation parameter> of the lenses of a pair of 3D virtual eyeglasses based on the alpha channel value <read on second brightness attenuation coefficient>; FIG. 11b teaches a generated 3D face shape model that is glasses-less <read on before wearing glasses> from a facial tracking device, where the tinted glasses are applied at step S220 <read on after wearing glasses>, which changes the brightness of the eye-area region of the generated 3D face shape model; Note: it is noted that the first and second brightness attenuation coefficients are being interpreted as the same coefficient).
However, Ye does not expressly disclose
the brightness parameter of the virtual face model includes a second imaging brightness parameter of the virtual light source located in the virtual face model at the virtual image acquisition device.
Kornilov discloses
the brightness parameter of the virtual face model includes a second imaging brightness parameter of the virtual light source located in the virtual face model at the virtual image acquisition device (Kornilov, [0032]: teaches a client device 104 including a camera device <read on virtual image acquisition device> to track and capture a set of images of the user's face/head; [0079]: teaches determining a set of lighting parameters <read on second imaging brightness parameter> that are associated with each predetermined set of segments of the identified area of the user's face <read on located on virtual face model>, where each segment includes "a direction of the light and a source of the light <read on virtual light source>"; Note: it is noted that a "set of lighting parameters" is interpreted to be more than one lighting parameter, where one of the lighting parameters is interpreted to be the second imaging brightness parameter).
Kornilov is analogous art with respect to Ye because they are from the same field of endeavor, namely providing a virtual try-on glasses experience. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to have the system determine a set of lighting parameters for the user's face as taught by Kornilov into the teaching of Ye. The suggestion for doing so would allow for the generation of a lighting model, which would provide more accurate casted shadows from the virtual eyeglasses, further making the virtual glasses appear more realistic. Therefore, it would have been obvious to combine Kornilov with Ye.
Regarding Claim 5, the combination of Ye and Kornilov discloses the image synthesis method of Claim 4. Ye does not expressly disclose the limitations of Claim 5; however, Kornilov discloses wherein, the acquiring a brightness parameter of a virtual face model by using a virtual image acquisition device with the assistance of a virtual light source, comprises:
binding the virtual light source at the virtual face model (Kornilov, [0081]: teaches determining a lighting model based on the sets of lighting parameters determined <read on binding> from the plurality of images, which are images of the user's face/head <read on virtual face model> taken at differing positions via a camera), and
acquiring the first imaging brightness parameter of the virtual light source based on the virtual image acquisition device (Kornilov, [0081]: teaches the average of the lighting parameters <read on first imaging brightness parameter>, such as the average source of the light <read on virtual light source> and the average intensity of the light, being used to determine the lighting model, where the lighting parameters are determined from differing positions of a camera <read on virtual image acquisition device>).
Kornilov is analogous art with respect to Ye because they are from the same field of endeavor, namely providing a virtual try-on glasses experience. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to have the system determine a set of lighting parameters for the user's face as taught by Kornilov into the teaching of Ye. The suggestion for doing so would allow for the generation of a lighting model, which would provide more accurate casted shadows from the virtual eyeglasses, further making the virtual glasses appear more realistic. Therefore, it would have been obvious to combine Kornilov with Ye.
Regarding Claim 6, the combination of Ye and Kornilov discloses the image synthesis method of Claim 1. Additionally, Ye further discloses wherein, the method further comprises:
[[determining two-dimensional sampling positions and three-dimensional sampling positions located on the virtual face model matching the two-dimensional sampling positions when the virtual face model does not wear glasses, based on a relative position between the virtual image acquisition device and the virtual face model;]]
[[obtaining the two-dimensional sampling positions when the glasses are worn by adjusting the two-dimensional sampling positions when the virtual face model does not wear glasses based on optical parameters of the glasses;]]
determining an image distortion parameter having a matching relationship with the brightness attenuation parameter, [[based on changes of two-dimensional sampling positions of the three-dimensional sampling positions located on the virtual face model before and after the glasses are worn]] (Ye, [0048]: teaches showing the modified lens images when applying a rotation angle <read on image distortion parameter> that affects the applied environmental mapping effect; [0050]: teaches head posture changes being affected by lighting; Note: it is noted that although a matching relationship between the modified lens images and the head posture changes is not expressly stated, it would be obvious for one skilled in the art to determine that the lens image would be affected by the posture changes, which would infer a type of linked/connected relationship <read on matching relationship>).
However, Ye does not expressly disclose
determining two-dimensional sampling positions and three-dimensional sampling positions located on the virtual face model matching the two-dimensional sampling positions when the virtual face model does not wear glasses, based on a relative position between the virtual image acquisition device and the virtual face model;
obtaining the two-dimensional sampling positions when the glasses are worn by adjusting the two-dimensional sampling positions when the virtual face model does not wear glasses based on optical parameters of the glasses;
determining an image distortion parameter having a matching relationship with the brightness attenuation parameter, based on changes of two-dimensional sampling positions of the three-dimensional sampling positions located on the virtual face model before and after the glasses are worn.
Kornilov discloses
determining two-dimensional sampling positions and three-dimensional sampling positions located on the virtual face model matching the two-dimensional sampling positions when the virtual face model does not wear glasses, based on a relative position between the virtual image acquisition device and the virtual face model (Kornilov, [0046]: teaches generating at least two 2D images <read on 2D sampling positions> from the modified 3D generic face model <read on virtual face model> that was morphed to match the 3D model of the user's face"; [0062]: teaches a "3D generic face model is morphed to correspond to the 3D model of the user's face by moving the each reference point located on the 3D generic face model to match the 3D coordinate of the corresponding reference point <read on 3D sampling positions> located on the 3D model of the user's face"; Note: it is noted that this claim limitation is unclear regarding if the 2D sampling positions located on the virtual face model are different from the other 2D sampling positions);
obtaining the two-dimensional sampling positions when the glasses are worn by adjusting the two-dimensional sampling positions when the virtual face model does not wear glasses based on optical parameters of the glasses (Kornilov, [0097]: teaches a 2D generic face image 1400 <read on 2D sampling positions> comprising "a projection of transformed 3D generic face model 1300"; [0098]: teaches fitting <read on wearing> a 3D glasses model 1502 to the transformed 3D generic face model 1300, where the orientation <read on adjusting> of the 3D glasses model 1502 matches the orientation of the transformed 3D generic face model 1300 <read on virtual face model not wearing glasses>; [0041]: teaches a glasses frame information of each 3D glasses model comprising "the dimensions of the glasses, attributes of the lenses <read on optical parameters> of the glasses (e.g., whether the lenses comprise a particular color), the manufacturer of the glasses, and the price of the glasses"; Note: it is noted that "optical parameters" is not explicitly defined in the specification; therefore, BRI is used; additionally, the 2D generic face image is being interpreted as being adjusted simultaneously with the 3D generic face image, where it contains 2D coordinate data);
determining an image distortion parameter having a matching relationship with the brightness attenuation parameter, based on changes of two-dimensional sampling positions of the three-dimensional sampling positions located on the virtual face model before and after the glasses are worn (Kornilov, [0098]: teaches fitting a 3D glasses model 1502 to the transformed 3D generic face model 1300 <read on virtual face model wearing glasses>, where the orientation <read on changes> of the 3D glasses model 1502 matches the orientation of the transformed 3D generic face model 1300 <read on virtual face model not wearing glasses>; [0097]: teaches a 2D generic face image 1400 <read on 2D sampling positions> comprising "a projection of transformed 3D generic face model 1300"; [0061]: teaches moving certain locations of the 3D generic face model to match to corresponding locations on the 3D model of the user's face).
Kornilov is analogous art with respect to Ye because they are from the same field of endeavor, namely providing a virtual try-on glasses experience. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to have the system determine a set of lighting parameters for the user's face as taught by Kornilov into the teaching of Ye. The suggestion for doing so would allow for the generation of a lighting model, which would provide more accurate casted shadows from the virtual eyeglasses, further making the virtual glasses appear more realistic. Therefore, it would have been obvious to combine Kornilov with Ye.
Regarding Claim 7, the combination of Ye and Kornilov discloses the image synthesis method of Claim 6. Additionally, Ye further discloses wherein
there is a matching relationship (Ye, [0050]: teaches "the width and height of shadow effect can be changed according to posture changes to mimic or simulate the real shadow effect affected <read on matching relationship> by lighting") among
the brightness attenuation parameter (Ye, [0051]: teaches adding lens transparency changes <read on brightness attenuation parameter> through an alpha effect),
the image distortion parameter (Ye, [0048]: teaches showing the modified lens images when applying a rotation angle <read on image distortion parameter> that affects the applied environmental mapping effect),
the glasses type (Ye, [0043]: teaches a product catalog of groups of 2D images that include different types of eyeglasses).
Regarding Claims 8, 20, and 23, the combination of Ye and Kornilov discloses the image synthesis method, the electronic device, and the non-transitory computer-readable storage medium of Claims 1, 15, and 16 respectively. Additionally, Ye further discloses
acquiring a naked-eye image (Ye, FIG. 11c teaches an image <read on naked-eye image> of a 3D face shape model that does not have glasses),
PNG
media_image3.png
565
358
media_image3.png
Greyscale
determining a glasses wearing parameter, a non-glasses area and a glasses light-transmitting area of a glasses image corresponding to the naked-eye image (Ye, [0043]: teaches differing eyeglass images being cataloged into different types <read on glasses wearing parameter>, which corresponds to the user's face; [0048]: teaches lens images <read on glasses light-transmitting area of glasses image> being modified as shown in FIGS. 8a to 8c; [0049]: teaches an environmental texture <read on non-glasses area> that is to be projected onto the lens image; Note: it is noted that the non-glasses area is interpreted to be both the face region and the external environment);
modifying the brightness of the glasses light-transmitting area and the brightness of the non-glasses area based on the brightness attenuation parameter corresponding to the glasses to obtain a modified naked-eye image (Ye, [0048]: teaches applying an environmental mapping effect to the pair of lens images <read on modifying brightness of non-glasses area> to configure and show a transient modified lens image in place of the 2D lens image in real-time, which adds more visual realism to the virtual 3D eyeglasses, where a rotation angle can be applied to obtain an alternate face/head posture <read on modified naked-eye image>; FIGS. 8a to 8c teaches applying an alpha percentage to the lenses of the virtual glasses <read on modifying brightness of glasses light-transmitting area>);
PNG
media_image4.png
211
384
media_image4.png
Greyscale
obtaining a glasses wearing image based on the glasses wearing parameter and the modified naked-eye image (Ye, FIG. 11c teaches applying glasses and a shadow texture on the 3D face model of the face image <read on naked-eye image> to obtain an image <read on glasses wearing image> of the user wearing virtual 3D eyeglasses).
Regarding Claim 9, the combination of Ye and Kornilov discloses the image synthesis method of Claim 8. Additionally, Ye further discloses wherein
the brightness attenuation parameter includes a first brightness attenuation coefficient, which represents the change of the imaging brightness parameter of the virtual face model irradiated by the virtual light source before and after the glasses are worn, that is acquired by the virtual image acquisition device (Ye, [0047]: teaches changing the transparency <read on brightness attenuation parameter> of the lenses of a pair of 3D virtual eyeglasses based on the alpha channel value <read on first brightness attenuation coefficient>; FIG. 11b teaches a generated 3D face shape model <read on virtual face model> that is glasses-less <read on before wearing glasses> from a facial tracking device, where the tinted glasses are applied at step S220 <read on after wearing glasses>, which changes the brightness <read on irradiated> of the eye-area region of the generated 3D face shape model; [0052]: teaches constructing a 3D face/head model of a user using camera 26 <read on virtual image acquisition device>), and
the modifying the brightness of the glasses light-transmitting area and the brightness of the non-glasses area based on a brightness attenuation parameter corresponding to the glasses, comprises:modifying the brightness of the glasses light-transmitting area and the brightness of the non-glasses area based on the first brightness attenuation coefficient (Ye, FIGS. 8a to 8c teaches applying an alpha percentage to the lenses of the virtual glasses <read on modifying brightness of glasses light-transmitting area>; [0050]: teaches applying a shadow effect to the face image of the user in real-time using the 3D face/head model; FIG. 11b teaches step S220, where the 3D face model is wearing the tinted virtual glasses based on the alpha percentage <read on first brightness attenuation coefficient>, where a shadow effect is applied to parts of the 3D face/head model <read on modifying brightness of non-glasses area>, which affects the brightness that area).
Regarding Claim 10, the combination of Ye and Kornilov discloses the image synthesis method of Claim 9. Additionally, Ye further discloses wherein, the brightness attenuation parameter further includes
a second brightness attenuation coefficient, which represents the change of the imaging brightness parameter of the virtual face model before and after the glasses are worn acquired by the virtual image acquisition device (Ye, [0047]: teaches changing the transparency <read on brightness attenuation parameter> of the lenses of a pair of 3D virtual eyeglasses based on the alpha channel value <read on second brightness attenuation coefficient>; Note: it is noted that the first and second brightness attenuation coefficients are interpreted to be identical to each other; FIG. 11b teaches a generated 3D face shape model <read on virtual face model> that is glasses-less <read on before wearing glasses> from a facial tracking device, where the tinted glasses are applied at step S220 <read on after wearing glasses>, which changes the brightness of the eye-area region of the generated 3D face shape model; [0052]: teaches constructing a 3D face/head model of a user using camera 26 <read on virtual image acquisition device>), and
the modifying the brightness of the glasses light-transmitting area and the brightness of the non-glasses area based on a brightness attenuation parameter corresponding to the glasses, comprises:modifying the brightness of the glasses light-transmitting area based on the second brightness attenuation coefficient (Ye, FIGS. 8a to 8c teaches applying an alpha percentage to the lenses of the virtual glasses <read on modifying brightness of glasses light-transmitting area>; [0048]: teaches applying an environmental mapping effect to the pair of lens, which shows the transient modified lens images as shown in FIG. 10a; FIGS. 10a to 10c teaches differing viewing angles of the 3D face/head model wearing dark tinted glasses, where the reflection of the environment changes <read on modified brightness of glasses light-emitting area> based on the angle direction, which shows more of the reflected light <read on second brightness attenuation coefficient>).
PNG
media_image5.png
479
327
media_image5.png
Greyscale
Regarding Claim 12, the combination of Ye and Kornilov discloses the image synthesis method of Claim 9. Additionally, Ye further discloses wherein, the glasses wearing parameter includes
glasses type and/or glasses wearing posture (Ye, [0043]: teaches a product catalog of groups of 2D images that include different types of eyeglasses; [0048]: teaches determining a set criteria based on the real-time posture <read on glasses wearing posture> of the face of the user).
Regarding Claims 19 and 22, the combination of Ye and Kornilov discloses the electronic device and the non-transitory computer-readable storage medium of Claims 15 and 16 respectively. Additionally, Ye further discloses wherein
[[the brightness parameter of the virtual face model includes a first imaging brightness parameter of the virtual light source acquired by a virtual image acquisition device located on the virtual face model;]]
the brightness attenuation parameter at least includes a first brightness attenuation coefficient, which is determined based on the first imaging brightness parameters before and after the glasses are worn (Ye, [0047]: teaches changing the transparency <read on brightness attenuation parameter> of the lenses of a pair of 3D virtual eyeglasses based on the alpha channel value <read on first brightness attenuation coefficient>; FIG. 11b teaches a generated 3D face shape model that is glasses-less <read on before wearing glasses> from a facial tracking device, where the tinted glasses are applied at step S220 <read on after wearing glasses>, which changes the brightness of the eye-area region of the generated 3D face shape model), and wherein,
[[the acquiring a brightness parameter of a virtual face model by using a virtual image acquisition device with the assistance of a virtual light source, comprises:binding the virtual image acquisition device at the position of the virtual face model, and]]
[[acquiring the first imaging brightness parameter based on the virtual image acquisition device when the virtual face model is irradiated by the virtual light source, and/or wherein,]]
[[the brightness parameter of the virtual face model includes a second imaging brightness parameter of the virtual light source located in the virtual face model at the virtual image acquisition device;]]
the brightness attenuation parameter further includes a second brightness attenuation coefficient, which is determined based on the second imaging brightness parameters before and after the glasses are worn (Ye, [0047]: teaches changing the transparency <read on brightness attenuation parameter> of the lenses of a pair of 3D virtual eyeglasses based on the alpha channel value <read on second brightness attenuation coefficient>; FIG. 11b teaches a generated 3D face shape model that is glasses-less <read on before wearing glasses> from a facial tracking device, where the tinted glasses are applied at step S220 <read on after wearing glasses>, which changes the brightness of the eye-area region of the generated 3D face shape model; Note: it is noted that the first and second brightness attenuation coefficients are being interpreted as the same coefficient), and wherein,
[[the acquiring a brightness parameter of a virtual face model by using a virtual image acquisition device with the assistance of a virtual light source, comprises:binding the virtual light source at the virtual face model, and]]
[[acquiring the first imaging brightness parameter of the virtual light source based on the virtual image acquisition device.]]
However, Ye does not expressly disclose
the brightness parameter of the virtual face model includes a first imaging brightness parameter of the virtual light source acquired by a virtual image acquisition device located on the virtual face model;
the acquiring a brightness parameter of a virtual face model by using a virtual image acquisition device with the assistance of a virtual light source, comprises:binding the virtual image acquisition device at the position of the virtual face model, and
acquiring the first imaging brightness parameter based on the virtual image acquisition device when the virtual face model is irradiated by the virtual light source, and/or wherein,
the brightness parameter of the virtual face model includes a second imaging brightness parameter of the virtual light source located in the virtual face model at the virtual image acquisition device;
the acquiring a brightness parameter of a virtual face model by using a virtual image acquisition device with the assistance of a virtual light source, comprises:binding the virtual light source at the virtual face model, and
acquiring the first imaging brightness parameter of the virtual light source based on the virtual image acquisition device.
Kornilov discloses
the brightness parameter of the virtual face model includes a first imaging brightness parameter of the virtual light source acquired by a virtual image acquisition device located on the virtual face model (Kornilov, [0032]: teaches a client device 104 including a camera device <read on virtual image acquisition device> to track and capture a set of images of the user's face/head; [0079]: teaches determining a set of lighting parameters <read on first imaging brightness parameter> that are associated with each predetermined set of segments of the identified area of the user's face <read on located on virtual face model>, where each segment includes "a direction of the light and a source of the light <read on virtual light source>"; Note: it is noted that a "set of lighting parameters" is interpreted to be more than one lighting parameter, where one of the lighting parameters is interpreted to be the first imaging brightness parameter);
the acquiring a brightness parameter of a virtual face model by using a virtual image acquisition device with the assistance of a virtual light source, comprises:binding the virtual image acquisition device at the position of the virtual face model (Kornilov, [0081]: teaches determining a lighting model based on the sets of lighting parameters determined <read on binding> from the plurality of images, which are images of the user's face/head <read on virtual face model> taken at differing positions via a camera <read on virtual image acquisition device>; Note: it is noted that "binding" is being interpreted broadly), and
acquiring the first imaging brightness parameter based on the virtual image acquisition device when the virtual face model is irradiated by the virtual light source (Kornilov, [0081]: teaches the average of the lighting parameters <read on first imaging brightness parameter>, such as the average source of the light <read on virtual light source> and the average intensity of the light <read on virtual face model being irradiated>, being used to determine the lighting model; Note: it is noted that "irradiated" is being interpreted as lighting up an area/region), and/or wherein,
the brightness parameter of the virtual face model includes a second imaging brightness parameter of the virtual light source located in the virtual face model at the virtual image acquisition device (Kornilov, [0032]: teaches a client device 104 including a camera device <read on virtual image acquisition device> to track and capture a set of images of the user's face/head; [0079]: teaches determining a set of lighting parameters <read on second imaging brightness parameter> that are associated with each predetermined set of segments of the identified area of the user's face <read on located on virtual face model>, where each segment includes "a direction of the light and a source of the light <read on virtual light source>"; Note: it is noted that a "set of lighting parameters" is interpreted to be more than one lighting parameter, where one of the lighting parameters is interpreted to be the second imaging brightness parameter);
the acquiring a brightness parameter of a virtual face model by using a virtual image acquisition device with the assistance of a virtual light source, comprises:binding the virtual light source at the virtual face model (Kornilov, [0081]: teaches determining a lighting model based on the sets of lighting parameters determined <read on binding> from the plurality of images, which are images of the user's face/head <read on virtual face model> taken at differing positions via a camera), and
acquiring the first imaging brightness parameter of the virtual light source based on the virtual image acquisition device (Kornilov, [0081]: teaches the average of the lighting parameters <read on first imaging brightness parameter>, such as the average source of the light <read on virtual light source> and the average intensity of the light, being used to determine the lighting model, where the lighting parameters are determined from differing positions of a camera <read on virtual image acquisition device>).
Kornilov is analogous art with respect to Ye because they are from the same field of endeavor, namely providing a virtual try-on glasses experience. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to have the system determine a set of lighting parameters for the user's face as taught by Kornilov into the teaching of Ye. The suggestion for doing so would allow for the generation of a lighting model, which would provide more accurate casted shadows from the virtual eyeglasses, further making the virtual glasses appear more realistic. Therefore, it would have been obvious to combine Kornilov with Ye.
Regarding Claims 21 and 24, the combination of Ye and Kornilov discloses the electronic device and the non-transitory computer-readable storage medium of Claims 15 and 16 respectively. Additionally, Ye further discloses wherein, the brightness attenuation parameter includes
a first brightness attenuation coefficient, which represents the change of the imaging brightness parameter of the virtual face model irradiated by the virtual light source before and after the glasses are worn, that is acquired by the virtual image acquisition device (Ye, [0047]: teaches changing the transparency <read on brightness attenuation parameter> of the lenses of a pair of 3D virtual eyeglasses based on the alpha channel value <read on first brightness attenuation coefficient>; FIG. 11b teaches a generated 3D face shape model <read on virtual face model> that is glasses-less <read on before wearing glasses> from a facial tracking device, where the tinted glasses are applied at step S220 <read on after wearing glasses>, which changes the brightness <read on irradiated> of the eye-area region of the generated 3D face shape model; [0052]: teaches constructing a 3D face/head model of a user using camera 26 <read on virtual image acquisition device>), and wherein
the modifying the brightness of the glasses light-transmitting area and the brightness of the non-glasses area based on a brightness attenuation parameter corresponding to the glasses, comprises:modifying the brightness of the glasses light-transmitting area and the brightness of the non-glasses area based on the first brightness attenuation coefficient (Ye, FIGS. 8a to 8c teaches applying an alpha percentage to the lenses of the virtual glasses <read on modifying brightness of glasses light-transmitting area>; [0050]: teaches applying a shadow effect to the face image of the user in real-time using the 3D face/head model; FIG. 11b teaches step S220, where the 3D face model is wearing the tinted virtual glasses based on the alpha percentage <read on first brightness attenuation coefficient>, where a shadow effect is applied to parts of the 3D face/head model <read on modifying brightness of non-glasses area>, which affects the brightness that area), and/or wherein,
the brightness attenuation parameter further includes a second brightness attenuation coefficient, which represents the change of the imaging brightness parameter of the virtual face model before and after the glasses are worn acquired by the virtual image acquisition device (Ye, [0047]: teaches changing the transparency <read on brightness attenuation parameter> of the lenses of a pair of 3D virtual eyeglasses based on the alpha channel value <read on second brightness attenuation coefficient>; Note: it is noted that the first and second brightness attenuation coefficients are interpreted to be identical to each other; FIG. 11b teaches a generated 3D face shape model <read on virtual face model> that is glasses-less <read on before wearing glasses> from a facial tracking device, where the tinted glasses are applied at step S220 <read on after wearing glasses>, which changes the brightness of the eye-area region of the generated 3D face shape model; [0052]: teaches constructing a 3D face/head model of a user using camera 26 <read on virtual image acquisition device>), and wherein
the modifying the brightness of the glasses light-transmitting area and the brightness of the non-glasses area based on a brightness attenuation parameter corresponding to the glasses, comprises:modifying the brightness of the glasses light-transmitting area based on the second brightness attenuation coefficient (Ye, FIGS. 8a to 8c teaches applying an alpha percentage <read on second brightness attenuation coefficient> to the lenses of the virtual glasses; [0048]: teaches applying an environmental mapping effect to the pair of lens, which shows the transient modified lens images as shown in FIG. 10a; FIGS. 10a to 10c teaches differing viewing angles of the 3D face/head model wearing dark tinted glasses, where the reflection of the environment changes <read on modified brightness of glasses light-emitting area> based on the angle direction, which shows more of the reflected light).
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Ye et al. (US 20160035133 A1), hereinafter referenced as Ye, in view of Kornilov et al. (US 20180374261 A1), hereinafter referenced as Kornilov as applied to Claim 8 above respectively, and further in view of Laffont Pierre-Yves et al. (WO 2017134275 A1), hereinafter referenced as Laffont.
Regarding Claim 11, the combination of Ye and Kornilov discloses the image synthesis method of Claim 8. Additionally, Ye further discloses wherein, the method further comprises:
adjusting a position of the glasses light-transmitting area based on an image distortion parameter, to obtain a light transmission modified area (Ye, [0048]: teaches applying an environmental mapping effect to the pair of lens images <read on glasses light-transmitting area> to show a transient modified lens image <read on light transmission modified area>, where a rotation angle <read on image distortion parameter> is determined, which affects the 2D lens image and shows the transient modified lens images), wherein
the image distortion parameter represents an image position change of the virtual face model before and after the glasses are worn, which is acquired by the virtual image acquisition device (Ye, FIGS. 10a to 10c teaches the rotation angle <read on image distortion parameter> differences <read on image position change> between each viewing angle),
[[if the size difference between the glass light-transmitting area and the light transmission modified area is equal to or larger than a preset difference, perform image inpainting on the light transmission modified area based on the light transmission modified area.]]
The combination of Ye and Kornilov does not expressly disclose
if the size difference between the glass light-transmitting area and the light transmission modified area is equal to or larger than a preset difference, perform image inpainting on the light transmission modified area based on the light transmission modified area.
Laffont discloses
if the size difference between the glass light-transmitting area and the light transmission modified area is equal to or larger than a preset difference, perform image inpainting on the light transmission modified area based on the light transmission modified area (Laffont, [00244]: teaches an image capture process where the user is recorded with and without eyeglasses, where the eyeglasses lens characteristics <read on light transmission modified area> and frames are estimated, and the texture and geometry of the regions potentially covered by the glasses frames are acquired via in-painting <read on image inpainting>; [00248]: teaches modifying the characteristics of the lens, where "the intersections between the refracted rays with the 3D face template are computed for each frame" such that a vector field is used to measure the amount of lens deviation <read on preset difference> at a given lens pixel; Note: it is noted that although the if conditional is not expressly stated, it would be obvious to one of ordinary skill in the art to try and implement a threshold value for changing the lens property to save on performance; additionally, it is unclear as to what "the size difference" is referring to).
Laffont is analogous art with respect to Ye, in view of Kornilov because they are from the same field of endeavor, namely providing a virtual try-on eyeglasses experience. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement an in-painting process to modify the reflective properties of the virtual lens in real-time based on viewpoint as taught by Laffont into the teaching of Ye, in view of Kornilov. The suggestion for doing so would allow for more realistic appearing virtual glasses on the user's face. Therefore, it would have been obvious to combine Laffont with Ye, in view of Kornilov.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Choukroun et al. (US 20160246078 A1) discloses generating a real-time realistic-looking image of a pair of virtual glasses positioned on a real user face image;
Coon et al. (US 20130321412 A1) discloses generating a virtual try-on glasses application;
Knorr et al. (US 20200013223 A1) discloses representing a virtual object in a view of a real environment;
Yang et al. (US 20160178936 A1) discloses virtually trying on eyeglasses that includes capturing a plurality of images of a user's face.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KARL TRUONG whose telephone number is (703)756-5915. The examiner can normally be reached 10:30 AM - 7:30 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571) 272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/K.D.T./Examiner, Art Unit 2614
/KENT W CHANG/Supervisory Patent Examiner, Art Unit 2614
