DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Sollami et al., U.S. Patent Number 11,250,572 B2.
Regarding claim 1, Sollami discloses a method comprising: accessing, by a user system, a first image that depicts a real-world object (col. 3, lines 1-2, receive a first image that include at least a portion of a person); retrieving, by the user system, a second image that depicts an augmented reality (AR) fashion item (col. 3, lines 25-26, an image of a second fashion item (i.e., a second image that includes a second fashion item); providing as concurrent inputs to a machine learning model both the first image that depicts the real-world object and the second image that depicts the AR fashion item (col. 4, lines 36-40, the first image may be masked to occlude pixels of the first fashion item to be replaced; key points and masking may be convolved using the convolution encoder; col. 4, lines 42-45, portion of the GAN may be the second image that includes the second fashion item, which may be convolved by the convolution encoder; FIG. 3A): analyzing the first image and the second image by the machine learning model to estimate an occlusion map for overlaying the AR fashion item on the real-world object depicted in the first image (col. 4, lines 1-14, server may mask the first image to occlude pixels of the first fashion item that is to be replaced with the second fashion item; determine which portions of the body of the person are not to be covered with the second fashion item; masking of the image; occlusion of the portions of the person; figure 2A, col. 4, lines 7-9, masking of the image 200 so that the shirt 222a, pants 222b, and shoes 222c may be replace with the fashion item); and generating a modified image by overlaying the AR fashion item depicted in the second image on the real-world object depicted in the first image using the estimated occlusion map (col. 5, lines 6-10, using the masked first image, the second segmentation image, and the second image that includes the second fashion item, the server may generate a third image that includes the person with the second fashion item; generate the third image such that the second fashion item disposed on the person appears real; that is, the fashion item may have the desired positioning, texture, color, orientation, and the like on the person).
Regarding claim 2, Sollami discloses wherein the second image comprises a three-dimensional (3D) rendering of the AR fashion item overlaid on a portion of the real-world object depicted in the first image (col. 5, lines 18-20, generate a photo-realistic image of the person wearing the transferred fashion item based on the segmentation GAN).
Regarding claim 3, Sollami discloses wherein the AR fashion item comprises at least one of an AR shoe, AR glasses, an AR watch, an AR hat, or AR jewelry (col. 3, lines 2-4, fashion item (e.g., shirt, pants, dress, belt, shoes, jewelry, hat, scarf, garment, and/or fashion accessory)).
Regarding claim 4, Sollami discloses wherein the occlusion map indicates occlusions and visibility associated with the first image comprising indications of whether to replace each pixel in the first image with a pixel corresponding to the AR fashion item (col. 2, lines 21-25, the pixels of the occluded portions of the person may be located; the pixels of an identified hand may be retained in the input segmentation map (i.e., the hand is not masked); col. 4, lines 1-3, may mask the first item to occlude pixels of the first fashion item that is to be replaced with the second fashion item; col. 4, lines 16-19, image of the person may be occluded as shown in image 250, where portions of the person 252b and 252c are covered, portion 252d remains visible).
Regarding claim 5, Sollami discloses further comprising: selecting a region of the first image over which to overlay the AR fashion item (col. 54, lines 25-26, first image may be masked to occlude pixels of the first fashion item with the second fashion item); identifying a first pixel of the first image within the region of the first image (col. 29-30, predict pixel-wise semantic segmentation); and replacing, based on the occlusion map, the first pixel with a corresponding pixel of the AR fashion item to occlude the first pixel with the corresponding pixel of the AR fashion item (col. 4, lines 1-3, may mask the first time to occlude pixels of the first fashion item that is to be replaced with the second fashion item; col. 4, lines 16-19, image of the person may be occluded as shown in image 250, where portions of the person 252b and 252c are covered, portion 252d remains visible; col. 4, lines 36-38, first image may be masked to occlude pixels of the first fashion item with the second fashion item).
Regarding claim 6, Sollami discloses further comprising: identifying a second pixel of the first image within the region of the first image (col. 2, lines 23-24, pixels of an identified hand); and preventing replacing, based on the occlusion map, the second pixel with a corresponding pixel of the AR fashion item to prevent occluding the second pixel with the corresponding pixel of the AR fashion item (col. 2, lines 24-25, the hand may be retained; col. 6, lines 53-63, when the parts of the body of the person in the first image overlap or self-occlude, the pixels of the overlapped or self-occluded parts in the first semantic segmentation image may be retained; col. 6, lines 59-62, pixels associated with these portions of the body may be identified; the identified pixels may be retained in the input segmentation map).
Regarding claim 7, Sollami discloses wherein the AR fashion item is selected in response to input that identifies the AR fashion item from a list of AR fashion items (col. 2, lines 2-6, use two images: (1) a picture of the target fashion item, and (2) an image including the customer; generated output image includes the customer wearing the fashion item; col. 3, lines 2-4, person wearing a first fashion item (e.g., shirt, pants, dress, belt, shoes, jewelry, hat, scarf, garment, and/or fashion accessory; col.4 , lines 8-9, FIG. 2A, shows masking of the image so that the shirt 222a, pants 222b and shoes 222c may be replaced with a fashion item).
Regarding claim 8, Sollami discloses wherein the first image comprises a frame of a real-time video captured by a camera of the user system (col. 3, lines 5-8, first image may be received by the server from a camera, a computer and/or mobile device (e.g., computer 550 shown in FIG.4) that may capture; col. 3, lines 30-32, a plurality of images may be provided where the person is in a different pose, or the images are taken from different points of view).
Regarding claim 9, Sollami discloses further comprising: applying one or more machine learning models to the real-time video to generate tracking information of the real-world object depicted in the real-time video; continuously updating the real-time video; and modifying placement of the AR fashion item, adjusted based on the estimated occlusion map, on the depiction of the real-world object (col. 3, lines 7-8, may capture and/or store the first image, which Examiner interprets as real time; col. 3, lines 28-37, in some implementation, more than one image of person may be provided to the server; a plurality of images may be provided where the person is in a different pose, or the images are taken from different points of view; the semantic segmentation of the first image may retain the details of the one or more garments and/or fashion items in the image (e.g., color, cloth, texture and the like)
Regarding claim 10, Sollami discloses wherein the machine learning model comprises an image-to-image artificial neural network (col. 3, lines 23-26, GAN may use an image that includes a person (i.e., the first image including a person wearing a first fashion item) and an image of a fashion item (i.e., a second image that includes a second fashion item)).
Regarding claim 11, Sollami discloses further comprising training the machine learning model by performing training operations comprising: accessing training data comprising a first set of training images that depict one or more real-world objects, a second set of training images that depict training AR objects overlaid on the one or more real-world objects, and corresponding ground-truth occlusion maps (col. 4, line 61 – col. 5, line 5, form image 318 of the person wearing the transferred fashion item; m ay compare the image 318 with the first image and/or second image to determine whether the image is comparatively realistic in order to train the GAN); applying the machine learning model to a first training image of the first set of training images and a second training image of the second set of training images to estimate a training occlusion map (col. 18-30, segmentation GAN of FIG. 3A; occludes pixels of the first fashion item to be replaced with the second fashion item; key points and masking may be convolved by the convolution encoder so as to concatenate the key points and masking); computing a deviation between the training occlusion map and an individual ground-truth occlusion map of the ground-truth occlusion maps corresponding to the first and second training images (col. 7, lines 2-4, may determine a loss between the second semantic segmentation image and the first semantic segmentation image, and determine adversarial loss); and updating one or more parameters of the machine learning model based on the computed deviation (col. 7, lines 5-14, losses between the second semantic segmentation image and the first semantic segmentation image may be determined from the loss (i.e., the least absolute deviation); the determined loss may be used to train the server and/or one or more of the above-described GANs).
Regarding claim 12, Sollami discloses wherein the first and second training images are synthetically generated, further comprising: synthetically generating a depiction of the one or more real-world objects to provide the first training image (col. 3, lines 54-55, the first semantic segmentation which may form an image that appears to be non-realistic with the person and the second fashion item); and synthetically generating a depiction of the training AR objects of the one or more real- world objects to provide the second training image; col. 7, lines 27-37, the server may generate a fourth image of the person that includes the second fashion item by using the first semantic segmentation image, the second masked image, and the second image that includes the second fashion item; determine an error gradient by combining perceptual loss, feature matching loss, and adversarial loss; server may be trained by the determined error gradient).
Regarding claim 13, Sollami discloses further comprising: automatically generating the individual ground-truth occlusion map for the synthetically generated first and second training images (col. 7, lines 21-26, masking at the server, the first image to occlude pixels of the first fashion item to be replaced to form a second masked image; occluding may be performed by the server deleting minimal sub-images whose pixels are to be changed during the transference of the image of the second fashion onto the image of the first fashion item on the person).
Regarding claim 14, Sollami discloses further comprising: accessing the first training image depicting the one or more real-world objects; automatically overlaying the training AR objects on the one or more real-world objects depicted in the first training image to generate the second training image (col. 5, lines 2-5, the discriminator may compare the image with the first image and/or second image to determine whether the image is comparatively realistic in order to train the GAN); and receiving input that specifies portions of the first training image to occlude by a first set of portions of the training AR objects and portions of the first training image to prevent from being occluded by a second set of portions of the training AR objects (col. 5, lines 27-32, key points and masking may be convolved by using the convolution encoder so as to concatenate the key points and the masking; the output may be combined).
Regarding claim 15, Sollami discloses further comprising: generating the individual ground-truth occlusion map for the first and second training images based on the input that specifies the portions of the first training image to occlude by a first set of portions of the training AR objects and the portions of the first training image to prevent from being occluded by a second set of portions of the training AR objects (col. 2, lines 23-24, pixels of an identified hand; col. 2, lines 24-25, the hand may be retained; col. 6, lines 53-63, when the parts of the body of the person in the first image overlap or self-occlude, the pixels of the overlapped or self-occluded parts in the first semantic segmentation image may be retained; col. 6, lines 59-62, pixels associated with these portions of the body may be identified; the identified pixels may be retained in the input segmentation map; col. 5, lines 2-5, the discriminator may compare the image with the first image and/or second image to determine whether the image is comparatively realistic in order to train the GAN).
Regarding claims 16- 19, they are rejected based upon similar rational as above claims 1, 2, 11 and 4, respectively. Sollami further discloses a system comprising: at least one processor configured to perform operations (col 10, lines 61-67).
Regarding claim 20, it is rejected based upon similar rational as above claim 1. Sollami further discloses a non-transitory machine-readable storage medium that includes instructions that, when executed by one or more processors of a user system, cause the user system to perform operations (col. 10, lines 41-45).
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Motilewa Good-Johnson whose telephone number is (571)272-7658. The examiner can normally be reached Monday - Friday 6am-2:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jason Chan can be reached at 571-272-3022. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
MOTILEWA . GOOD JOHNSON
Primary Examiner
Art Unit 2616
/MOTILEWA GOOD-JOHNSON/Primary Examiner, Art Unit 2619
Prosecution Insights