Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 8-10, 14, and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Bouguerra (U.S. Patent Application Publication No. 2012/0069028), referred herein as Bouguerra, in view of Mazurenko et al. (WO 2015/057262), referred herein as Mazurenko.
Regarding claim 1, Bouguerra teaches a method for streamlining the applying of an effect to an object appearing in a sequence of video frames (fig 4), the method comprising the steps of:
a. performing in parallel (paragraph 68, lines 5-9; paragraph 72, the last 10 lines; the effect rendering and tracking for landmark determination are performed in parallel, frame-by-frame, in real-time, during a live video chat)
i) effect rendering for the sequence of video frames (paragraph 69; numerous effects are rendered in real-time, augmenting the live video chat), and
ii) object landmark determining and occlusion detecting for the sequence of video frames, wherein the effect rendering applies an effect in association with landmarks determined for the object to define a sequence of output video frames with the effect applied (paragraph 65, lines 1-8 and the last 9 lines; paragraph 67, lines 1-8; paragraph 68, lines 1-9; real-time landmark tracking and occlusion detection are performed to determine facial features to which effects are applied); and
b. providing the sequence of output video frames for displaying (paragraph 49, lines 1-8 and the last 4 lines; paragraph 71, lines 4-5; the real-time effects-augmented video frames are displayed during the live video chat).
Bouguerra does not explicitly teach performing the different operations in first and second threads.
However, in a similar field of endeavor, Mazurenko teaches a method for streamlining image processing for an object appearing in a sequence of video frames, comprising performing multi-threaded parallel processing of object landmark determining and image processing (abstract; page 1, lines 27-33; page 2, line 28 through page 3, line 1), wherein the parallel processing comprises, i) in a first thread, effect rendering and image processing for the sequence of video frames, and ii) in a second thread, object landmark determining and detection for the sequence of video frames (page 3, lines 9-16; page 3, line 26 through page 4, line 2).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the multi-thread parallel processing of Mazurenko with the parallel processing of Bouguerra because this provides improved object landmark determining and image processing performance, even when processing resources may be limited (see, for example, Mazurenko, page 9, lines 16-24).
Regarding claim 2, Bouguerra in view of Mazurenko teaches the method of claim 1, wherein the sequence of frames comprises frame t-1, frame t and frame t+1 in sequence, and wherein step a. determines object landmarks for frame t in parallel with applying the effect to the object in association with object landmarks previously determined for frame t-1 (Bouguerra, paragraph 65, lines 1-8 and the last 9 lines; paragraph 68, lines 1-9; paragraph 72, the last 10 lines; continual video streaming comprises a stream of frame at times t, t-1, and t+1, where feature landmarks determined at a previous frame are used to apply effects to the current frame; this is performed continuously, in parallel, in real-time).
Regarding claim 3, Bouguerra in view of Mazurenko teaches the method of claim 2, wherein the effect rendering is guided by the occlusion as detected (Bouguerra, paragraph 67, lines 1-8).
Regarding claim 8, Bouguerra in view of Mazurenko teaches the method of claim 1, wherein for each of the video frames, the object landmark determining determines a bounding box within which the object is located, the bounding box comprising a subset of video frame pixels (Bouguerra, paragraph 66, lines 1-8; paragraph 74, lines 1-8).
Regarding claim 9, Bouguerra in view of Mazurenko teaches the method of claim 1, wherein steps a. and b. are performed by a first computing device and wherein step b. comprises communicating the sequence of output video frames via a communication network for displaying by at least one other computing device participating in a video chat, video conference or teleconsultation with the first computing device (Bouguerra, figs 1, 2; paragraph 49).
Regarding claim 10, Bouguerra in view of Mazurenko teaches the method of claim 1, wherein the method applies respective effects to a plurality of respective objects and step a. performs object landmark detection for each of the plurality of respective objects and effect rendering applies respective effects relative to at least some of the plurality of respective objects (Bouguerra, paragraph 51, the last 7 lines; paragraph 65, lines 1-19; paragraph 68, lines 1-9; paragraph 69).
Regarding claim 14, the limitations of this claim substantially correspond to the limitations of claim 1 (except for the device, storage, and processor, which are taught by Bouguerra, fig 2, device, 220, processor 222, and memory 230; paragraphs 54 and 55); thus they are rejected on similar grounds.
Regarding claim 15, the limitations of this claim substantially correspond to the limitations of claim 3; thus they are rejected on similar grounds.
Claims 4-6, 11-13, 16, 17, 19, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Bouguerra, in view of Mazurenko, and further in view of Troutman et al. (U.S. Patent Application Publication No. 2022/0202168), referred herein as Troutman.
Regarding claim 4, Bouguerra in view of Mazurenko teaches the method of claim 3, wherein step a. provides object information at a pixel level according to the occlusion as detected to guide the effects rendering (Bouguerra, paragraph 67, lines 1-8; paragraph 79).
Bouguerra in view of Mazurenko teaches image features being adversely affected by real backgrounds being removed and replaced with digital backgrounds, and inpainting areas that were adversely affected (Bouguerra, paragraph 79) – and both background replacement and inpainting can be considered types of masking, or as typically involving masks. Nevertheless, Bouguerra in view of Mazurenko does not explicitly teach mask information.
However, in a similar field of endeavor, Troutman teaches a method for applying effects to an object by determining object landmarks and rendering effects applied to the object (figs 3 and 8; paragraph 70, lines 1-9; paragraphs 71 and 77; paragraph 83, lines 1-8; paragraph 87), wherein object mask information at a pixel level is provided to guide the effects rendering (paragraph 78, lines 1-2; paragraph 79; paragraph 80, lines 4-14).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the masking of Troutman with the effect rendering of Bouguerra in view of Mazurenko because this can help increase the accuracy of determining how and where to apply the effects to the object, thereby improving the quality of the images that are output for display (see, for example, Troutman, paragraph 41, the last 8 lines; paragraph 73, lines 1-10; paragraph 83).
Regarding claim 5, Bouguerra in view of Mazurenko teaches the method of claim 1, wherein object landmark determining provides pixel locations for the object, the object landmark determining comprising detecting the pixel locations for at least some of the video frames using machine learning (Bouguerra, paragraph 66, the last 3 lines; paragraphs 69 and 79).
Bouguerra in view of Mazurenko does not explicitly teach using a deep neural network.
However, in a similar field of endeavor, Troutman teaches a method for applying effects to an object by determining object landmarks and rendering effects applied to the object (figs 3 and 8; paragraph 70, lines 1-9; paragraphs 71 and 77; paragraph 83, lines 1-8; paragraph 87), wherein the object landmark determination comprises using a deep neural network (fig 10; paragraph 78; paragraph 79, lines 1-5).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the deep neural network of Troutman with the machine learning of Bouguerra in view of Mazurenko because deep neural networks provide high accuracy landmark identification that help determine how and where to apply the effects to the object, thereby improving the quality of the images that are output for display (see, for example, Troutman, paragraph 41, the last 8 lines; paragraph 73, lines 1-10; paragraph 83).
Regarding claim 6, Bouguerra in view of Mazurenko, further in view of Troutman teaches the method of claim 5, wherein the sequence of frames comprises frame t-1, frame t and frame t+1 in sequence, and wherein step a. comprises stabilizing object landmarks for frame t in accordance with a prediction of the location of the object landmarks for frame t using an optical flow function (Bouguerra, paragraph 65, lines 1-19; paragraph 66, lines 1-8; paragraph 68, lines 1-9 and the last 2 lines; Troutman, paragraph 78, lines 1-4 and the last 3 lines; paragraph 80, the last 5 lines; the motivation to combine is similar to that discussed above in the rejection of claim 5).
Regarding claim 11, Bouguerra in view of Mazurenko teaches the method of claim 10, wherein the sequence of video frames includes a face, the plurality of objects comprises respective regions of the face and the respective effects comprise respective facial effects (Bouguerra, paragraph 60; paragraph 65, lines 1-19).
Bouguerra in view of Mazurenko does not explicitly teach that the facial effects are makeup effects.
However, in a similar field of endeavor, Troutman teaches a method for applying effects to an object by determining object landmarks and rendering effects applied to the object (figs 3 and 8; paragraph 70, lines 1-9; paragraphs 71 and 77; paragraph 83, lines 1-8; paragraph 87), wherein the effects comprise respective makeup effects for respective regions of the face (figs 6 and 7; paragraph 54; paragraphs 66 and 68).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the makeup facial effects with the facial effects of Bouguerra in view of Mazurenko because this provides the user with a more customizable experience in order to achieve particular desired looks or improvements, which can be especially helpful in video chat environments such as those described in Bouguerra (see, for example, Troutman, paragraphs 6 and 7).
Regarding claim 12, Bouguerra in view of Mazurenko, further in view of Troutman teaches the method of claim 11, wherein the regions comprise any one or more of: a left eye, a left brow, a right eye, a right brow, a nose, a mouth, a top lip or a bottom lip (Bouguerra, paragraph 65, lines 1-19; Troutman, paragraphs 81 and 83; the motivation to combine is similar to that discussed above in the rejection of claim 11).
Regarding claim 13, Bouguerra in view of Mazurenko teaches the method of claim 1, wherein the effect comprises a makeup effect, a hair effect or a nail effect and wherein the method comprises providing a user interface presenting a plurality of makeup, hair or nail effects for selection through user input (Bouguerra, figs 5-7; paragraph 60; paragraph 62, lines 1-11; paragraph 65, lines 1-19; paragraph 69).
Bouguerra in view of Mazurenko does not explicitly teach that the effects are associated with respective products, and wherein the user interface is configured to provide access to an e-commerce interface to conduct a product purchase transaction.
However, in a similar field of endeavor, Troutman teaches a method for applying effects to an object by determining object landmarks and rendering effects applied to the object (figs 3 and 8; paragraph 70, lines 1-9; paragraphs 71 and 77; paragraph 83, lines 1-8; paragraph 87), wherein the effects are associated with respective products, and wherein a user interface is configured to provide access to an e-commerce interface to conduct a product purchase transaction (figs 4, 6, and 11-13; paragraph 9; paragraph 54; paragraphs 66 and 68).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the e-commerce capability of Troutman with the user interface of Bouguerra in view of Mazurenko because this provides the user with a more customizable experience in order to achieve particular desired looks or improvements, which can be especially helpful in video chat environments such as those described in Bouguerra, and enables the capability to then follow that up with a corresponding real-life experience (see, for example, Troutman, paragraphs 6 and 7).
Regarding claims 16, 17, and 19, the limitations of these claims substantially correspond to the limitations of claims 5, 6, and 13, respectively; thus they are rejected on similar grounds as their corresponding claims.
Regarding claim 20, Bouguerra in view of Mazurenko teaches the computing device of claim 14, wherein: the applying applies respective effects to a plurality of respective objects and step a. performs object landmark detection for each of the plurality of respective objects and effect rendering applies respective effects relative to at least some of the plurality of respective objects (Bouguerra, paragraph 51, the last 7 lines; paragraph 65, lines 1-19; paragraph 68, lines 1-9; paragraph 69); the sequence of video frames includes a face, the plurality of objects comprises respective regions of the face, the respective effects comprise respective facial effects, and the regions comprise any one or more of: a left eye, a left brow, a right eye, a right brow, a nose, a mouth, a top lip or a bottom lip (Bouguerra, paragraph 60; paragraph 65, lines 1-19).
Bouguerra in view of Mazurenko does not explicitly teach that the facial effects are makeup effects.
However, in a similar field of endeavor, Troutman teaches a method for applying effects to an object by determining object landmarks and rendering effects applied to the object (figs 3 and 8; paragraph 70, lines 1-9; paragraphs 71 and 77; paragraph 83, lines 1-8; paragraph 87), wherein the effects comprise respective makeup effects for respective regions of the face (figs 6 and 7; paragraph 54; paragraphs 66 and 68).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the makeup facial effects with the facial effects of Bouguerra in view of Mazurenko because this provides the user with a more customizable experience in order to achieve particular desired looks or improvements, which can be especially helpful in video chat environments such as those described in Bouguerra (see, for example, Troutman, paragraphs 6 and 7).
Claims 7 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Bouguerra, in view of Mazurenko, further in view of Troutman, and further in view of Zhang et al. (U.S. Patent Application Publication No. 2006/0088191), referred herein as Zhang.
Regarding claim 7, Bouguerra in view of Mazurenko, further in view of Troutman teaches method of claim 5, wherein the sequence of frames comprises frame t-1, frame t and frame t+1 in sequence, and wherein object landmark determining comprises computing an optical flow function in relation to frame t for predicting locations within frame t responsive to locations in frame t-1, and using pixel locations responsive to the optical flow function (Bouguerra, paragraph 65, lines 1-19; paragraph 66, lines 1-8; paragraph 68, lines 1-9 and the last 2 lines; Troutman, paragraph 78, lines 1-4 and the last 3 lines; paragraph 80, the last 5 lines; the motivation to combine is similar to that discussed above in the rejection of claim 5).
Bouguerra in view of Mazurenko, further in view of Troutman does not explicitly teach determining an optical flow error for frame t, and skipping a detecting of the pixel locations for frame t responsive to the optical flow error.
However, in a similar field of endeavor, Zhang teaches a method for performing object landmark determination for a sequence of video frames (paragraph 26, lines 1-10; paragraph 28; paragraph 48, lines 1-6; paragraph 85), comprising determining an optical flow error for frame t, and skipping a detecting of the pixel locations for frame t responsive to the optical flow error (paragraph 8; paragraph 44; paragraphs 70 and 72).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the optical flow error processing of Zhang with the optical flow processing of Bouguerra in view of Mazurenko, further in view of Troutman because this provides a faster and more robust algorithm for object detection such that real-time video processing, such as that disclosed by Bouguerra, can be achieved and improved (see, for example, Zhang, paragraph 26, the last 9 lines; paragraph 27, the last 4 lines; paragraph 28, the last 4 lines).
Regarding claim 18, the limitations of this claim substantially correspond to the limitations of claim 7; thus they are rejected on similar grounds.
Response to Arguments
Applicant’s arguments with respect to the double patenting rejections have been fully considered and are persuasive. The approved terminal disclaimer obviates the double patenting rejections; thus they have been withdrawn.
Applicant’s arguments with respect to the 102 and 103 rejections have been fully considered, but they are not persuasive.
On page 8 of the Applicant’s Remarks, with respect to claim 1, the Applicant argues that 1) Bouguerra teaches performing operations in parallel, but is silent regarding any particular operations to be performed in parallel, and 2) Bouguerra does not teach the claimed multi-threading approach.
Regarding the first argument, cited paragraph 72 of Bouguerra points to the blocks of figure 4 as capable of being performed in parallel and concurrently. These referenced blocks include the object landmark determining with occlusion detection, and the effects rendering, inter alia. Additionally, as noted in the prior art rejections and disclosed by Bouguerra, the object landmark determining with occlusion detection, and the effects rendering, are performed in parallel, frame-by-frame, in real-time during the video stream – which requires parallel/concurrent processing of these components. Thus it is clear in Bouguerra that the disclosed cited functions are performed in parallel.
Regarding the second argument, it is respectfully submitted that this argument is moot in view of the new grounds of rejection presented above, as Mazurenko discloses such a multi-threaded approach and its advantages.
Conclusion
The following prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Dodge (U.S. Patent Application Publication No. 2005/0088420); Ordering of events between different input sources.
Treat (U.S. Patent Application Publication No. 2012/0174121); Processing user input events in a web browser.
Shotton (U.S. Patent Application Publication No. 2016/0086025); Pose tracker with multi-threaded architecture.
Chaland (U.S. Patent Application Publication No. 2015/0339006); Asynchronous preparation of displayable sections of a graphical user interface.
Li (U.S. Patent Application Publication No. 2020/0265231); Method, apparatus, and system for automatically annotating a target object in images.
Venkataraman (U.S. Patent Application Publication No. 2020/0073714); Serialization floors and deadline driven control for performance optimization of asymmetric multiprocessor systems.
Sideris (U.S. Patent Application Publication No. 2023/0062386); Graphics processing.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID T WELCH whose telephone number is (571)270-5364. The examiner can normally be reached Monday-Thursday, 8:30-5:30 EST, and alternate Fridays, 9:00-2:30 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached at 571-272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
DAVID T. WELCH
Primary Examiner
Art Unit 2613
/DAVID T WELCH/Primary Examiner, Art Unit 2613