DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-17 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-17 of U.S. Patent No. 12363249. Although the claims at issue are not identical, they are not patentably distinct from each other because application claims 1-17 are encompassed by patented claims 1-17, respectively. See also table below:
Application Claims
1.A method for generation of a plurality of portrait effects in an electronic device, the method comprising: feeding an image captured from the electronic device into an encoder pre-learned using a plurality of features corresponding to the plurality of portrait effects; extracting, using the encoder, at least one of one or more low level features and one or more high level features from the image; generating, for the image, one or more first portrait effects of the plurality of portrait effects based on the at least one of the one or more high level features and the one or more low level features by passing the image through one or more first decoders; and generating, for the image, one or more second portrait effects of the plurality of portrait effects based on the at least one of the one or more high level features and the one or more low level features by passing the image through one or more second decoders.
2. The method of The method of wherein the one or more first decoders comprises a Bokeh decoder, the one or more second decoders comprises a High Key decoder, and a Low Key decoder, and wherein the one or more first portrait effects is at least one of a Big circle effect, a studio effect or a Bokeh effect associated with the Bokeh decoder, the one or more second portrait effects is at least one of a High Key portrait effect, a Low Key portrait effect, a color backdrop effect, a color point effect, a spin effect, or a zoom effect.
3. The method of claim 1, wherein the one or more first portrait effects relates to depth-related camera features, and wherein the one or more second portrait effects relates to segmentation-related camera features.
4. The method of claim 1, wherein the encoder, the one or more first decoders, and the one or more second decoders are comprised within a single deep neural network (DNN) model.
5. The method of claim 4, further comprising: training the single DNN model, wherein the training of the single DNN model comprises: generating ground truth data for each of the one or more first portrait effects, and the one or more second portrait effects using a plurality of data modules, training the encoder in a plurality of stages using the generated ground truth to extract the at least one of the one or more low level features and the one or more high level features from the image, and training the encoder, the one or more first decoders, the one or more second decoders, and a defocus map decoder associated with the one or more first decoders in a plurality of stages using the generated ground truth data to generate the one or more first and the one or more second portrait effect.
6. The method of claim 5, wherein the generating of the ground truth data for the one or more first portrait effects comprises: generating the ground truth data using a shallow depth of field of an input color image captured using a first aperture of a lens of a camera and a wide depth of field of the input color image captured using a second aperture of the lens of the camera; and generating a ground truth defocus map by generating a depth map of the input color image and generating a defocus map based on the depth map to generate the ground truth data for the one or more first portrait effects.
7. The method of claim 5, wherein the generating of the ground truth data for the one or more second portrait effects comprises: generating a segmentation mask of an input color image; generating a high-resolution matting mask from the segmentation mask; refining the generated matting mask; and generating the one or more second portrait effects on the input color image by changing one or more color parameters of the input color image and replacing a background of the input color image with a predetermined background based on the one or more second portrait effects.
8. The method of claim 5, wherein the training of the encoder, the one or more first decoders, the one or more second decoders, and the defocus map decoder comprises: training the encoder, the one or more first decoders, and the one or more second decoders in a first stage of the plurality of stages to provide an input image as an output image; training the encoder, the one or more first decoders, the one or more second decoders, and the defocus map decoder in a series of second stages of the plurality of stages to generate the one or more first portrait effect, and the one or more second portrait effects using the ground truth data for each of the one or more first portrait effects, and the one or more second portrait effects and the input image; and training the encoder, the one or more first decoders and the defocus map decoder to generate the one or more first portrait effects using the ground truth data for the one or more first portrait effects and the input image.
9. A system for generation of a plurality of portrait effects in an electronic device, the system comprising: an encoder configured to: receive an image captured from the electronic device, wherein the encoder is pre-learned using a plurality of features corresponding to the plurality of portrait effects, and extract at least one of one or more low level features and one or more high level features from the image; one or more first decoders to generate, for the image, one or more first portrait effects of the plurality of portrait effects based on the at least one of the one or more high level features and the one or more low level features; and one or more second decoders to generate, for the image, one or more second portrait effects of the plurality of portrait effects based on the at least one of the one or more high level features and the one or more low level features.
10. The system of The system of wherein the one or more first decoders comprises a Bokeh decoder, and the one or more second decoders comprise a High Key decoder, and a Low Key decoder, and wherein the one or more first portrait effects is a Bokeh effect associated with the Bokeh decoder, the one or more second portrait effects is at least one of a High Key portrait effect, a Low Key portrait cffcct, a color backdrop cffcct, a color point effect, a spin effect, or a zoom effect.
11. The system of claim 9, wherein the encoder, the one or more first decoders, and the one or more second decoders are comprised within a single DNN model.
12. The system of The system of wherein the one or more first portrait effects relates to depth-related camera features, and wherein the one or more second portrait effects relates to segmentation-related camera features.
13. The system of claim 11, wherein the single DNN model is trained by: generating ground truth data for each of the one or more first portrait effects, and the one or more second portrait effects using a plurality of data modules; training the encoder in a plurality of stages using the generated ground truth to extract the at least one of the one or more low level features and the one or more high level features from the image; and training the encoder, the one or more first decoders, the one or more second decoders, and a defocus map decoder associated with the one or more first decoders in a plurality of stages using the generated ground truth data to generate the one or more first and the one or more second portrait effect.
14. The system of claim 13, wherein the ground truth data for the one or more first portrait effects is generated by; generating the ground truth data using a shallow depth of field of an input color image captured using a first aperture of a lens of a camera and a wide depth of field of the input color image captured using a second aperture of the lens of the camera; and generating a ground truth defocus map by generating a depth map of the input color image and generating a defocus map based on the depth map to generate the ground truth data for the one or more first portrait effects.
15. The system of claim 13, wherein the ground truth data for the one or more second portrait effects is generated by: generating a segmentation mask of an input color image; generating a high-resolution matting mask from the segmentation mask; refining the generated matting mask; and generating the one or more second portrait effects on the input color image by changing one or more color parameters of the input color image and replacing a background of the input color image with a predetermined background based on the one or more second portrait effects.
16. The system of claim 13, wherein the encoder, the one or more first decoders, the one or more second decoders, and the defocus map decoder are trained by: training the encoder, the one or more first decoders, and the one or more second decoders, in a first stage of the plurality of stages to provide an input image as an output image; training the encoder, the one or more first decoders, and the one or more second decoders, and the defocus map decoder in a series of second stages of the plurality of stages to generate the one or more first portrait effect, and the one or more second portrait effect, using the ground truth data for each of the one or more first portrait effects, and the one or more second portrait effects and the input image; and training the encoder, the one or more first decoders and the defocus map decoder to generate the one or more first portrait effects using the ground truth data for the one or more first portrait effects and the input image.
17. The system of claim 9, further comprising: a memory; a processor; a communicator; a display; one or more cameras; and an image processor, wherein the memory stores the plurality of portrait effects and information related to the generation of the plurality of portrait effects, and wherein the memory stores instructions to be executed by the processor for generating the plurality of portrait effects.
Patented Claims
1. A method for generation of a plurality of portrait effects in an electronic device, the method comprising: feeding an image captured from the electronic device into an encoder pre-learned using a plurality of features corresponding to the plurality of portrait effects; extracting, using the encoder, at least one of one or more low level features and one or more high level features from the image; generating, for the image, one or more first portrait effects of the plurality of portrait effects based on the at least one of the one or more high level features and the one or more low level features by passing the image through one or more first decoders; and generating, for the image, one or more second portrait effects of the plurality of portrait effects based on the at least one of the one or more high level features and the one or more low level features by passing the image through one or more second decoders,
wherein each of the one or more first portrait effect, and the one or more second portrait effects is generated in a single inference.
2. The method of claim 1, wherein the one or more first decoders comprises a Bokeh decoder, the one or more second decoders comprises a High Key decoder, and a Low Key decoder, and wherein the one or more first portrait effects is at least one of a Big circle effect, a studio effect or a Bokeh effect associated with the Bokeh decoder, the one or more second portrait effects is at least one of a High Key portrait effect, a Low Key portrait effect, a color backdrop effect, a color point effect, a spin effect, or a zoom effect.
3. The method of claim 1, wherein the one or more first portrait effects relates to depth-related camera features, and wherein the one or more second portrait effects relates to segmentation-related camera features.
4. The method of claim 1, wherein the encoder, the one or more first decoders, and the one or more second decoders are comprised within a single deep neural network (DNN) model.
5. The method of claim 4, further comprising: training the single DNN model, wherein training the single DNN model comprises: generating ground truth data for each of the one or more first portrait effects, and the one or more second portrait effects using a plurality of data modules, training the encoder in a plurality of stages using the generated ground truth to extract the at least one of the one or more low level features and the one or more high level features from the image, and training the encoder, the one or more first decoders, the one or more second decoders, and a defocus map decoder associated with the one or more first decoders in a plurality of stages using the generated ground truth data to generate the one or more first and the one or more second portrait effect.
6. The method of claim 5, wherein the generating of the ground truth data for the one or more first portrait effects comprises: generating the ground truth data using a shallow depth of field of an input color image captured using a first aperture of a lens of a camera and a wide depth of field of the input color image captured using a second aperture of the lens of the camera; and generating a ground truth defocus map by generating a depth map of the input color image and generating a defocus map based on the depth map to generate the ground truth data for the one or more first portrait effects.
7. The method of claim 5, wherein the generating of the ground truth data for the one or more second portrait effects comprises: generating a segmentation mask of an input color image; generating a high-resolution matting mask from the segmentation mask; refining the generated matting mask; and generating the one or more second portrait effects on the input color image by changing one or more color parameters of the input color image and replacing a background of the input color image with a predetermined background based on the one or more second portrait effects.
8. The method of claim 5, wherein the training of the encoder, the one or more first decoders, the one or more second decoders, and the defocus map decoder comprises: training the encoder, the one or more first decoders, and the one or more second decoders in a first stage of the plurality of stages to provide an input image as an output image; training the encoder, the one or more first decoders, the one or more second decoders, and the defocus map decoder in a series of second stages of the plurality of stages to generate the one or more first portrait effect, and the one or more second portrait effects using the ground truth data for each of the one or more first portrait effects, and the one or more second portrait effects and the input image; and training the encoder, the one or more first decoders and the defocus map decoder to generate the one or more first portrait effects using the ground truth data for the one or more first portrait effects and the input image.
9. A system for generation of a plurality of portrait effects in an electronic device, the system comprising: an encoder configured to: receive an image captured from the electronic device, wherein the encoder is pre-learned using a plurality of features corresponding to the plurality of portrait effects, and extract at least one of one or more low level features and one or more high level features from the image; one or more first decoders to generate, for the image, one or more first portrait effects of the plurality of portrait effects based on the at least one of the one or more high level features and the one or more low level features; and one or more second decoders to generate, for the image, one or more second portrait effects of the plurality of portrait effects based on the at least one of the one or more high level features and the one or more low level features,
wherein each of the one or more first, and the one or more second portrait effects is generated in a single inference.
10. The system of claim 9, wherein the one or more first decoders comprises a Bokeh decoder, and the one or more second decoders comprise a High Key decoder, and a Low Key decoder, and wherein the one or more first portrait effects is a Bokeh effect associated with the Bokeh decoder, the one or more second portrait effects is at least one of a High Key portrait effect, a Low Key portrait effect, a color backdrop effect, a color point effect, a spin effect, or a zoom effect.
11. The system of claim 9, wherein the encoder, the one or more first decoders, and the one or more second decoders are comprised within a single DNN model.
12. The system of claim 9, wherein the one or more first portrait effects relates to depth-related camera features, and wherein the one or more second portrait effects relates to segmentation-related camera features.
13. The system of claim 11, wherein the single DNN model is trained by: generating ground truth data for each of the one or more first portrait effects, and the one or more second portrait effects using a plurality of data modules; training the encoder in a plurality of stages using the generated ground truth to extract the at least one of the one or more low level features and the one or more high level features from the image; and training the encoder, the one or more first decoders, the one or more second decoders, and a defocus map decoder associated with the one or more first decoders in a plurality of stages using the generated ground truth data to generate the one or more first and the one or more second portrait effect.
14. The system of claim 13, wherein the ground truth data for the one or more first portrait effects is generated by: generating the ground truth data using a shallow depth of field of an input color image captured using a first aperture of a lens of a camera and a wide depth of field of the input color image captured using a second aperture of the lens of the camera; and generating a ground truth defocus map by generating a depth map of the input color image and generating a defocus map based on the depth map to generate the ground truth data for the one or more first portrait effects.
15. The system of claim 13, wherein the ground truth data for the one or more second portrait effects is generated by: generating a segmentation mask of an input color image; generating a high-resolution matting mask from the segmentation mask; refining the generated matting mask; and generating the one or more second portrait effects on the input color image by changing one or more color parameters of the input color image and replacing a background of the input color image with a predetermined background based on the one or more second portrait effects.
16. The system of claim 13, wherein the encoder, the one or more first decoders, the one or more second decoders, and the defocus map decoder are trained by: training the encoder, the one or more first decoders, and the one or more second decoders, in a first stage of the plurality of stages to provide an input image as an output image; training the encoder, the one or more first decoders, and the one or more second decoders, and the defocus map decoder in a series of second stages of the plurality of stages to generate the one or more first portrait effect, and the one or more second portrait effect, using the ground truth data for each of the one or more first portrait effects, and the one or more second portrait effects and the input image; and training the encoder, the one or more first decoders and the defocus map decoder to generate the one or more first portrait effects using the ground truth data for the one or more first portrait effects and the input image.
17. The system of claim 9, further comprising: a memory; a processor; a communicator; a display; one or more cameras; and an image processor, wherein the memory stores the plurality of portrait effects and information related to the generation of the plurality of portrait effects, and wherein the memory stores instructions to be executed by the processor for generating the plurality of portrait effects.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HUNG H LAM whose telephone number is (571)272-7367. The examiner can normally be reached 9AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, TWYLER HASKINS can be reached at (571) 272-7406. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HUNG H LAM/Primary Examiner, Art Unit 2639 09/03/26
Prosecution Insights