Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5, 9-10, 13-16, and 19-22 are rejected under 35 U.S.C. 103 as being unpatentable over Sun (CN111640165A), Runxiang (CN110062176B), and Zhou (US20210217219A1).
Regarding claim 1, Sun teaches a video display method, comprising: displaying a first image (Sun; ABST, describes displaying AR character special effects of historical celebrities in a first display area of a display screen. This teaches displaying a first image.) and in response to an input for the first image (claims 1, 5, and S104, describe, while the AR character special effects are displayed in the first display area, detecting whether the user makes a preset group photo triggering action based on the video data and after detecting the triggering action, generating and displaying a second group photo image based on the AR character special effects displayed in the first area and the video frame images displayed in the second area. The input (triggering action) occurs while the first image is already being displayed and the input triggers generation of the second images. This reads on an input for the displayed first image, the user input (triggering action) occurs while the first image is displayed.)
displaying a video including a plurality of consecutive frames of second images, to present a target object image acquired in real time (S102, describes acquiring video data including the user (object) in real time and displaying the video data, and, ABST, wherein the video data includes multiple frames of video frame images. This teaches presenting a target object image acquired in real time across multiple frames).
However, Sun does not explicitly describe presenting a dynamic effect of a local feature in the first image; wherein each frame of the second images includes the target object image and a partial image of a third image, the third image being one frame of a plurality of consecutive frames of target images obtained after processing the local feature in the first image.
Runxiang teaches a plurality of consecutive frames (Runxiang; claims 1-2, and S101-104, describes acquiring a first image from a second video wherein the first image is the last image frame of the second video, determining a target object in the first image, acquiring a second image, and generating a first video wherein foreground images of a plurality of continuous image frames comprise the target object and background images of the plurality of continuous image frames comprise the second image, and merging the first video and the second video into one video. This teaches generating a video including a plurality of consecutive frames.), wherein each frame of the second images includes the target object image and a partial image of a third image (Runxiang; claims 1-2, S104, describes per-frame image composition and (Runxiang; claim 2) “for each of a plurality of successive image frames of the first video, rendering color parameters of pixels of the foreground image as: multiplying the color parameter of the pixel of the target object by X and multiplying the color parameter of the pixel in the image region corresponding to the foreground image in the second image by the sum of (1-X), wherein X <1 is greater than or equal to 0; wherein the value of X corresponding to a preceding image frame in the continuous multiple image frames of the first video is larger than the value of X corresponding to a following image frame”. This teaches that, in each frame, the target object is present and at least a portion of the background image (partial image) is present via blending, as (1-X) uses the background for each composed pixel. Runxiang’s “second image” corresponds to the third image and the “second image” may comprise images/frames used across the video frames). It would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Sun with Runxiang to generate and display a multi-frame video in response to Sun’s triggering action to improve the video result and yield a more clear multi-frame composite video.
However, Sun in view of Runxiang does not explicitly describe presenting a dynamic effect of a local feature in the first image and that the background/third image is one frame of a plurality of consecutive frames of target images obtained after processing the local feature in the first image.
Zhou teaches processing a local feature in the first image to generate a plurality of frames that present a dynamic effect of a local feature in the first image (Zhou; claims 1-2, describe generating a facial animation from a single image by processing facial features, including detecting facial feature points and performing deformation/optimization steps to generate facial animation frames (dynamic effect) from local facial features of the initial image.) and the third image being one frame of a plurality of consecutive frames of target images obtained after processing the local feature in the first image (Zhou; claims 1-2, describe generating facial animation frames from the initial single image by processing facial features through the multi-step process. Zhou’s animation frames are generated (obtained) after the facial feature processing steps are performed on the initial image and these generated animation frames read as the third image (background image).) It would have been obvious to one of ordinary skill in the art, before the effective filing date, to further modify the image sequence of Sun in view of Runxiang with the consecutive facial animation frames of Zhou so that the video presents a more dynamic, local-feature effect across multiple frames.
Claim 9 has similar limitations as of Claim 1, therefore it is rejected under the same rationale as Claim 1, except Claim 9 further recites “a processor, a memory and a computer program stored on the memory and executable on the processor that, when executed by the processor, implements the video display method according to any one of claims 1 to-7”. Sun states “In a third aspect, embodiments of the present disclosure also provide a computer device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processing The processor communicates with the memory through a bus, and the machine-readable instructions execute the above-mentioned first aspect or the steps in any possible implementation manner of the first aspect when executed by the processor.”
Claim 10 has similar limitations as of Claim(s) 1 except it is a CRM claim, therefore it is rejected under the same rationale as Claim 1. Sun; fourth aspect, “In a fourth aspect, the embodiments of the present disclosure also provide a computer-readable storage medium having a computer program stored on the computer-readable storage medium, and the computer program executes the above-mentioned first aspect or any of the first aspects when run by a processor. Steps in one possible implementation.”
Regarding claim 2, Sun in view of Runxiang and Zhou teaches the method according to claim 1, further comprising: acquiring the first image (Runxiang; ABST, describes acquiring a first image.), processing the local feature of the first image using a Generative Adversarial Networks algorithm, to obtain the plurality of consecutive frames of target images (Zhou; claim 1, steps 3 and 4, describe generating a facial animation from a single image by processing facial features through a multi-step process that includes optimizing texture of the facial area through a generative adversarial neural network and (Zhou; claim 2) feature points corresponding to a “current i-th frame” of a person. Successive i-th frames correspond to successive generated animation images. This teaches processing the local feature (facial features) of the first image using a GAN algorithm to obtain a plurality of consecutive frames of target images), acquiring the third image to be displayed from the plurality of consecutive frames of target images (Zhou; claims 1-2, as previously discussed in claim 1, Zhou teaches generating the plurality of facial animation frames (see claim 1). Sun describes (Sun; claim 1) determining at least one frame from multiple frames of video frame images based on the video data. This teaches selecting at least one frame from multiple frames, which would be applied to Zhou’s generated facial animation frames to acquire the third image to be displayed.) acquiring the target object image from an image acquired by a camera in real time (Sun; ABST and claim 1, describes acquiring video data including the user in real time, wherein the video data includes multiple frames of video frame images. The user in the video data corresponds to the target object. This teaches acquiring the target object image from an image acquired by a camera in real time), and fusing the third image with the target object image to obtain one frame of the second images (Runxiang; claim 2, describes per-frame composition where foreground images comprise the target object and background images comprise the second image and, for each frame, rendering color parameters by multiplying the color parameter of the pixel of the target object by X and multiplying the color parameter of the pixel in the image region corresponding to the foreground image in the second image by (1-X), wherein X <1 is greater than or equal to 0. This teaches fusing the third image (Runxiang “second image”/background) with the target object image (foreground) via blending to obtain one frame of the second images.) It would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify the combination of Sun in view of Zhou and Runxiang by using Zhou’s GAN processing to generate the animation frames, using Sun’s frame selection, and Runxiang’s blending with Sun’s real-time camera image in order to improve animation quality.
Claim 13, has similar limitations as of Claim 2, therefore it is rejected under the same rationale as Claim 2.
Claim 19, has similar limitations as of Claim 2, therefore it is rejected under the same rationale as Claim 2.
Regarding claim 3, Sun in view of Runxiang and Zhou teaches the method according to claim 2, wherein the processing the local feature of the first image using the Generative Adversarial Networks algorithm to obtain the plurality of consecutive frames of target images comprises: processing the local feature of the first image using the Generative Adversarial Networks algorithm, to obtain a plurality of consecutive frames of local feature images (Zhou; claim 1, steps 3 and 4, describe GAN processing of local facial features, including optimizing texture of the facial area, wherein the facial area does not comprise an oral cavity area, and synthesizing texture of the oral cavity area through a GAN network. These GAN operations produce processed facial area and oral cavity outputs for successive animation images which reads on a plurality of consecutive frames of local feature region images.) respectively fusing the first image with each of the plurality of consecutive frames of local feature images to obtain the plurality of consecutive frames of target images (Zhou; claim 5 step 4.2, describes combining the processed facial image with the non-facial area in the deformation image to obtain the final portrait image. This reads on fusing local feature region images with an image derived from the first image to obtain target images.) It would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify the combination in further view of Zhou’s two-stage GAN approach for optimization to generate higher quality animations.
Claim 14, has similar limitations as of Claim 3, therefore it is rejected under the same rationale as Claim 3.
Claim 20, has similar limitations as of Claim 3, therefore it is rejected under the same rationale as Claim 3.
Regarding claim 4, Sun in view of Runxiang and Zhou teaches the method according to claim 2, wherein prior to the fusing the third image with the target object image to obtain one frame of the second images, the method further comprises: in accordance with the image acquired by the camera in real time, determining a mask image in which a first mask region corresponding to the target object image is identified (Sun; ABST and S102, describes acquiring video data including the user in real time. Runxiang describes (Runxiang; claims 1-2 and s102) determining a target object by image segmentation, where the target object corresponds to a pixel region in the image. Applying segmentation in accordance with the real time camera image to identify the target object pixel region reads on determining a mask image with a first mask region for the target object) the acquiring the target object image from the image acquired by the camera in real time, comprising: in accordance with the mask image and the image acquired by the camera in real time, acquiring the target object image from the image acquired by the camera in real time (Sun, ABST and S102, describes acquiring video data including the user in real time. Runxiang describes (Runxiang; claims 1-2 and S102) that the target object corresponds to a pixel region determined by segmentation. This reads on using the pixel region (mask) to extract the target object from the real time camera image) the fusing the third image with the target object image to obtain one frame of the second images, comprising: in accordance with the mask image and the third image, acquiring, from the third image, a partial image of the third image that matches mask regions other than the first mask region (Runxiang describes (Runxiang; claims 1-2) per-frame foreground/background composition where the foreground comprises the target object and the background comprises the second image. In accordance with the mask image, the background/third image provides pixels for mask regions other than the first mask region (the non-target-object regions), where the partial image spatially matches those non-target-object mask regions.) and in accordance with the target object image and the partial image of the third image, forming the one frame of the second images by fusion (Runxiang; claims 2, describes forming each output frame by blending foreground and background pixels in accordance with both images, rendering color parameters by multiplying the target object pixel by X and the background pixel by (1-X), wherein X <1 is greater than or equal to 0. This teaches forming the output frame by fusion in accordance with the target object image and the partial image of the third image). It would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify the combination further with Runxiang’s image segmentation to Sun’s real time camera frames to identify which regions correspond to the target objects versus background for region-based composition to properly allocate pixels for improved accuracy.
Claim 15, has similar limitations as of Claim 4, therefore it is rejected under the same rationale as Claim 4.
Claim 21, has similar limitations as of Claim 4, therefore it is rejected under the same rationale as Claim 4.
Regarding claim 5, Sun in view of Runxiang and Zhou teaches the method according to claim 4, wherein in the mask image, the first mask region is identified by a target color channel (Runxiang describes (Runxiang; claims 1-2) determining a target object by image segmentation, where the target object corresponds to a pixel region in the image. Runxiang further describes (S103 pg. 6 ¶7) that pixels may be characterized as (x, y, r, g, b) where r, g, and b are RGB channel values. This teaches a maskable pixel region and channel-based pixel representation.) wherein the target color channel is any one of an R channel, a G channel and a B channel (Runxiang (S103 pg. 6 ¶7) describes R, G, and B channels as pixel color representation characterizing pixels as (x, y, r, g, b). This reads on using any one of the R, G, or B channels). It would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify the combination in further view of Runxiang such that the first mask region is identified by an RGB color channel for more efficient representation using standard color channels.
Claim 16, has similar limitations as of Claim 5, therefore it is rejected under the same rationale as Claim 5.
Claim 22, has similar limitations as of Claim 5, therefore it is rejected under the same rationale as Claim 5.
Claims 6, 17, and 23 are rejected under 35 U.S.C. 103 as being unpatentable over Sun (CN111640165A), Runxiang (CN110062176B), Zhou (US20210217219A1), Hao (CN109348276B), and Peng (CN113724140A).
Regarding claim 6, Sun in view of Runxiang and Zhou teaches the method according to claim 2, wherein the acquiring the third image to be displayed from the plurality of consecutive frames of target images comprises: acquiring one frame of the target images to be displayed from the plurality of consecutive frames of target images (Zhou describes (Zhou; claims 1-2) generating facial animation images from a single image by processing facial features to generate facial animation frames. Sun describes (Sun; claim 1) determining at least one frame from multiple frames of video frame images. Together, this teaches acquiring/selecting one frame from a plurality of frames for display.)
However, Sun in view of Runxiang and Zhou does not explicitly disclose in accordance with an aspect ratio of the one frame of target images and a screen aspect ratio, constructing a Model View Projection matrix and in accordance with the Model View Projection matrix, scaling the one frame of the target images to obtain the third image.
Hao describes, ABST, comparing a first aspect ratio of a picture/frame with a preset aspect ratio and obtaining a comparison result. This teaches aspect ratio comparison between a frame and a target/preset aspect ratio. It would have been obvious to modify the combination to use Hao’s aspect-ratio comparison and equal-ratio scaling to the selected frame, using the screen aspect ratio as the preset aspect ratio to better fit the frame to the display.
However, Hao fails to teach, but Peng teaches constructing a Model View Projection matrix (Peng; pg. 18 b), describes using a Model View Projection (MVP) matrix to adjust a virtual camera for image transformation/mapping. The aspect ratios (frame aspect ratio and screen aspect ratio from Hao’s comparison) serve as parameters for constructing Peng’s MVP matrix, as MVP matrices incorporate aspect ratio parameters to properly map images to display coordinates. The scaled output image produced by applying the MVP matrix is the third image used as the background in the composition. It would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Hao’s aspect-ratio-based scaling using Peng’s MVP matrix to better transform/scale the output image for display.
Claim 17, has similar limitations as of Claim 6, therefore it is rejected under the same rationale as Claim 6.
Claim 23, has similar limitations as of Claim 6, therefore it is rejected under the same rationale as Claim 6.
Claims 7 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Sun (CN111640165A), Runxiang (CN110062176B), Zhou (US20210217219A1), and Zou (CN112714336A).
Regarding claim 7, Sun in view of Runxiang and Zhou teaches the method according to claim 2, further comprising: acquiring a second image (Runxiang (Runxiang; claims 1) teaches “acquiring a second image”.) and processing the second image using the Generative Adversarial Networks algorithm, (Zhou describes (Zhou; claim 1 steps 3-4) generating facial animation images from a single image using GAN-based processing and, Figs. 1-5, showing the multiple images, produced by successive animation images/frames). However, Sun in view of Runxiang and Zhou does not explicitly describe to obtain and cache a plurality of consecutive frames of processed images and subsequent to the acquiring the first image, the method further comprising: in response to determining that the first image is different from the second image, clearing the cache.
Zou describes, pg. 6 ¶2-4, caching/buffering frames for reuse (¶2, “previously cached”) and clearing the cache when transitioning to a new sequence/image context (¶4 “immediately emptying the reference frame queue” and “the image after the IDR image will never use the data of the image before the IDR to decode”). This teaches caching processed frame data and clearing cached frame data when a new/different image/sequence is encountered. It would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify the combination of Sun in view of Runxiang and Zhou to cache the GAN processed frames and to clear the cache when the input image changes, as taught by Zou in order to increase efficiency and reduce cache misses/errors.
Claim 18, has similar limitations as of Claim 7, therefore it is rejected under the same rationale as Claim 7.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAN F KALHORI whose telephone number is (571)272-5475. The examiner can normally be reached Mon-Fri 8:30-5:30 ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona Faulk can be reached at (571) 272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DAN F KALHORI/Examiner, Art Unit 2618 /ZHENGXI LIU/Primary Examiner, Art Unit 2611