Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-10, 13, 15, 19 and 20 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Pandey et al. (Total Relighting: Learning to Relight Portraits for Background Replacement, 2021).
Regarding claim 1, Pandey et al. discloses a method comprising:
receiving, by one or more processors of a device, an image that includes data representing a depiction of an object (starting from just a single RGB image, page 2, para. 4);
generating, by the one or more processors, a segmentation of the data representing the depiction of the object (a learned matting module is used to extract a high-quality segmentation mask as well as the color of the foreground from the input portrait. In particular, we employ a deep convolutional network, sec. 3.1);
applying a machine learning model to a portion of the image to predict a surface normal tensor for the data representing the depiction of the object, the surface normal tensor representing surface normals of each pixel within the portion of the image (perform image-to-image translation of the input RGB foreground to an image of surface normals using a U-Net architecture (see figure 4, upper left), sec. 3.2.1); and
applying one or more augmented reality (AR) elements to the image based on the surface normal tensor (trained model infers per pixel surface normals, which are used as indices into the prefiltered lighting environments to form diffuse and specular light maps, sec. 2), page 2, para. 7).
Regarding claim 2, Pandey et al. discloses the method of claim 1, further comprising:
extracting a portion of the image corresponding to the segmentation of the data representing the object depicted in the image (The input to the relighting module is the predicted foreground 𝐹 generated by the matting network, sec. 3.2.1).
Regarding claim 3, Pandey et al. discloses the method of claim 1, wherein applying the one or more AR elements comprises:
determining that light is being focused on the data representing the depiction of the object from a first direction based on the surface normal tensor (relighting foreground and background replacement, which maintains high-frequency boundary details and accurately synthesizes the subject’s appearance as lit by novel illumination, page 1, para. 1); and
modifying pixel values of the portion of the image corresponding to the segmentation of the data representing the depiction of the object to re-focus the light on the depiction of the object from a second direction, wherein the pixel values are modified without modifying pixel values of portions of the image outside of the segmentation (composite the relit foreground into a new background extracted as an oriented view into the lighting panorama, using the matting, sec. 3.3).
Regarding claim 4, Pandey et al. discloses the method of claim 1, wherein applying the one or more AR elements comprises applying artificial light to the data representing the object depicted in the image based on the surface normal tensor (per-pixel lighting representation or light maps 𝐿, which encode or model the specular 𝑆 and diffuse 𝐷 components of surface reflection for a given omnidirectional target HDR lighting environment, sec. 3.2).
Regarding claim 5, Pandey et al. discloses the method of claim 1, further comprising: displaying the one or more AR elements on a first portion of the data representing the object depicted in a first frame of a video, wherein the object is positioned at a first location in the first frame (see figure 22);
determining that the object has moved from the first location to a second location in a second frame of the video and updating a display position of the one or more AR elements in the second frame to maintain the display of the one or more AR elements on the data representing the object depicted in the image based on the surface normal tensor (Although the full approach is designed to operate on still images, the technique also applies to videos, as demonstrated in our supplementary video using per-frame computation, sec. 7. AR elements are maintained on moving object(s) across frames, see figure 22).
Regarding claim 6, Pandey et al. discloses the method of claim 1, wherein the surface normal tensor is computed relative to a surface normal of a camera used to capture the image (output of the module is a surface normal image 𝑁 in camera space coordinates, sec. 3.2.1).
Regarding claim 7, Pandey et al. discloses the method of claim 1, wherein the one or more AR elements are applied to a real-time video feed comprising the image (applies to videos, as demonstrated in our supplementary video using per-frame computation, sec. 7.).
Regarding claim 8, Pandey et al. discloses the method of claim 1, wherein applying the one or more AR elements comprises replacing data representing a depiction of the object with one or more visual effects, further comprising:
determining light reflection directions on the object based on the surface normal tensor (per-pixel lighting representation or light maps 𝐿, which encode or model the specular 𝑆 and diffuse 𝐷 components of surface reflection for a given omnidirectional target HDR lighting environment, sec. 3.2); and
causing the one or more visual effects to reflect light along the light reflection directions using the surface normal tensor (accurately synthesizes the subject’s appearance as lit by novel illumination, page 1, para. 1).
Regarding claim 9, Pandey et al. discloses the method of claim 8, wherein applying the one or more AR elements comprises recoloring one or more portions of the object depicted in the image (producing realistic composite images for any desired scene. Examiner articulates that relighting a portrait or scene implies per-pixel color changes.).
Regarding claim 10, Pandey et al. discloses the method of claim 1, wherein applying the one or more AR elements comprises applying one or more animated fashion items to the object depicted in the image based on the surface normal tensor (wearing different clothing, section 5.1).
Regarding claim 13, Pandey et al. discloses the method of claim 1, wherein the machine learning model comprises a neural network, the neural network being trained to establish a relationship between image portions depicting different orientations of human bodies and surface normal directions of pixels of the human bodies (model is trained using relit portraits of subjects captured in a light stage computational illumination system, which records multiple lighting conditions, high quality geometry, and accurate alpha mattes; maintains high-frequency boundary details and accurately synthesizes the subject’s appearance as lit by novel illumination, page 1, para. 1).
Regarding claim 15, Pandey et al. discloses the method of claim 1, further comprising: detecting one or more wrinkles of clothing worn by the object depicted in the image based on the surface normal tensor (maintains high-frequency boundary details and accurately synthesizes the subject’s appearance as lit by novel illumination, thereby producing realistic composite images for any desired scene, page 1, para. 1).
Claim 19, a system claim, is rejected for the same reason as claim 1 (and see computing methodologies, page 1, para. 1).
Claim 20, a non-transitory computer-readable storage medium claim, is rejected for the same reason as claim 1 (and see computing methodologies, page 1, para. 1).
Allowable Subject Matter
Claims 11, 12, 14, and 16-18 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to THOMAS J LETT whose telephone number is (571)272-7464. The examiner can normally be reached Mon-Fri 9-6 ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tammy Goddard can be reached at (571) 272-7773. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/THOMAS J LETT/Primary Examiner, Art Unit 2611