Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This communication is a non-Final office action in merits. Claims 1-20, after restriction requirement, are presently pending. Claims 1-14 have been elected and considered below.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 12/19/2024, 9/30/2025, 5/19/2026, and 7/16/2026 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. CN202210716223.2, filed on 06/23/2022.
Restriction Election
Applicant elects Group I (claims 1-14), without traverse, for further examination.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or
nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim 1 is rejected under 35 U.S.C. 103 as being unpatentable over US 2020/0134804 A1, Song et al. (hereinafter Song) in view of US 2020/0129784 A1, Beriault et al. (hereinafter Beriault).
As to claim 1, Song discloses a method comprising:
acquiring a reference sample frame and a plurality of consecutive to-be-encoded sample frames (Figs 3, 7; pars 0004-0005, 0016-0017, receiving a sequence of video frames including past frames as input of an encoder/transformer. Note any past video frame can be used as a reference frame for a training purpose);
deforming the reference sample frame by using a generator in an initial generative model to generate a reconstructed sample frame corresponding to each of the to-be-encoded sample frames (Figs 3, 7, input past frames to a generator with a convolutional transformer network model to generates a future frame; pars 0004-0005, 0040, 0042, 0086, past frames being used as initial input for generative model);
inputting each reconstructed sample frame and a corresponding to-be-encoded sample frame into a first discriminator in the initial generative model to obtain a first identification result (Figs 3-4; pars 0053-0055, 0057-0058);
according to a timestamp order, splicing the to-be-encoded sample frames to obtain a spliced to-be-encoded sample frame (Fig 3; pars 0040, 0048, the past frames and the generated frames being added and stacked and added together and the resulting images are input to a video discriminator, i.e. splicing reconstructed sample frames to obtain a spliced reconstructed sample frame);
inputting the spliced to-be-encoded sample frame and the spliced reconstructed sample frame into a second discriminator of the initial generative model to obtain a second identification result (Fig 3; pars 0040, 0048);
obtaining an adversarial loss value based on the first identification result and the second identification result (pars 0046, 0050, 0052, 0055, adversarial loss values being obtained for generated and reconstruction loss); and
training the initial generative model based on the adversarial loss value to obtain a trained generative model (Fig 7; pars 0004-0005, 0014, 0016-0017).
Song teaches past frames can serve as reference frame for initial generative modeling.
The reference frame can also be adopted separately rather than using a past frame of the video frames.
Nevertheless, Beriault, in the same or similar field of endeavor, further teaches acquiring a reference sample frame (pars 0024, 0070, 0077) and deforming the reference sample frame by using a generator in an initial generative model to generate a reconstructed sample frame corresponding to each of the to-be-encoded sample frames (pars 0022, 0024, 0071, 0077, 0092, 0097).
Therefore, consider Song and Beriault’s teachings as a whole, it would have been obvious to one of skill in the art before the filing date of invention to incorporate Beriault’s teachings in Song’s method to select a proper reference frame suited for a particular application.
Claims 2, 7-12 are rejected under 35 U.S.C. 103 as being unpatentable over Song in view of Beriault and further in view of US 2021/0125583 A1, Kaplanyan et al. (hereinafter Kaplanyan).
As to claim 2, Song as modified discloses the method according to claim 1, wherein the adversarial loss value comprises: a generative adversarial loss value (Song: pars 0046, 0050, a generative adversarial loss). Song as modified does not expressly teach a spatial or a temporal adversarial loss value.
Kaplanyan, in the same or similar field of endeavor further teaches various adversarial loss including a temporal loss value being used for generative and/or decimator (pars 0056, 0062). Therefore, consider Song as modified and Kaplanyan’s teachings as a whole, it would have been obvious to one of skill in the art before the filing date of invention to incorporate Kaplanyan’s teachings in Song as modified’s method to provide a variety of loss functions/values appropriate for corresponding modeling applications.
As to claim 7, Song as modified discloses the method according to claim 1, wherein prior to training the initial generative model based on the adversarial loss value to obtain the trained generative model, the method further comprises generating a perceptual loss value based on each reconstructed sample frame and each to-be-encoded sample frame (Kaplanyan: pars 0056, 0062, perceptual spatial loss being generated for each reconstructed frame).
As to claim 8, Song as modified discloses the method according to claim 7, wherein the training the initial generative model based on the adversarial loss value to obtain the trained generative model comprises training the initial generative model based on the adversarial loss value and the perceptual loss value to obtain the trained generative model (Song: pars 0046, 0050, 0052; Kaplanyan: pars 0056, 0062).
As to claim 9, Song as modified discloses the method according to claim 7, wherein the deforming the reference sample frame by using the generator in the initial generative model to generate reconstructed sample frames corresponding to each of the to-be-encoded sample frames comprises: performing, based on the reference sample frame, motion estimation on each to-be-encoded sample frame to obtain a motion estimation result for each to-be-encoded sample frame (Song: pars 0052, 0081-0082, obtaining motion estimate; Beriault: Figs 1, 7-8; pars 0036, 0041, 0043, 0045, patient state estimation using a preliminary motion model; Kaplanyan: Fig 9; pars 0048-0051, 0061, 0064); and inputting, for each to-be-encoded sample frame, the reference sample frame and motion estimation results of the to-be-encoded sample frames into the generator of the initial generative model, and deforming the reference sample frame through the generator to generate the reconstructed sample frames corresponding to the to-be-encoded sample frames (Song: pars 0052, 0081-0082; Beriault: pars 0068-0069, 0082; Kaplanyan: Fig 9; pars 0048-0051, 0061, 0064).
As to claim 10, Song as modified discloses the method according to claim 9, prior to training the initial generative model based on the adversarial loss value and the perceptual loss value to obtain the trained generative model, the method further comprises: inputting each to-be-encoded sample frame into a pre-trained motion prediction module to obtain actual motion results corresponding to each of the to-be-encoded sample frames (Beriault: Figs 1, 7-8, preliminary motion model; pars 0034, 0036, 0043, 0071-0072; Kaplanyan: Fig 9; pars 0048-0052, 0061, 0064); and generating an optical flow loss value based on a difference between the motion estimation results and the actual motion results of each of the to-be-encoded sample frames (Kaplanyan: pars 0016, 0027, 0056, 0064, optical flow loss being used to stimulate temporal consistency across frames).
As to claim 11, Song as modified discloses the method according to claim 10, wherein the training the initial generative model based on the adversarial loss value and the perceptual loss value to obtain the trained generative model comprises training the initial generative model based on the adversarial loss value, the perceptual loss value, and the optical flow loss value to obtain the trained generative model (Beriault: Figs 1, 7-8, preliminary motion model; Kaplanyan: pars 0044, 0051, 0055-0056, adversarial loss, perceptual spatial loss, and optical flow loss. Or weighted sum of several losses, also see rejection in claim 10).
As to claim 12, Song as modified discloses the method according to claim 10, wherein the generating the optical flow loss value based on the difference between the motion estimation results and actual motion results of each of the to-be-encoded sample frames comprises: calculating, for a respective to-be-encoded sample frame, the difference between the motion estimation result and the actual motion result thereof as a motion difference corresponding to the respective to-be-encoded sample frame; and summing each motion difference as the optical flow loss value (Kaplanyan: pars 0056, a weighted sum of several losses, also see rejection in claim 10).
Claims 3-6 are rejected under 35 U.S.C. 103 as being unpatentable over Song in view of Beriault and further in view of Kaplanyan and US 2023/0037782 A1, Ma et al. (hereinafter Ma).
As to claim 3, Song as modified discloses the method according to claim 2, wherein the obtaining the adversarial loss value based on the first identification result and the second identification result comprises: obtaining the generative adversarial loss value based on the first identification result of each reconstructed sample frame (Song: pars 0046, 0050, 0052, obtaining the generative adversarial loss value); obtaining the spatial adversarial loss value based on a difference between the first identification result of each reconstructed sample frame and the first identification result of the corresponding to-be-encoded sample frame (Kaplanyan: pars 0056, 0062-0063, obtaining the spatial adversarial loss value based on a difference between the result of reconstructed sample frame result of the corresponding to-be-encoded sample frame); and obtaining the temporal adversarial loss value based on a difference between the second identification result of the spliced to-be-encoded sample frame and the second identification result of the spliced reconstructed sample frame (Kaplanyan: pars 0062-0064, similarity or difference).
Song as modified does not expressly teach the generative adversarial loss and the temporal adversarial loss are associated with corresponding identification results. An ordinary skill in the art, however, would appreciate and understand object identification is one of a commonly encountered applications in neural network modeling (including GAN) applications.
Nevertheless, Ma, in the same or similar field of endeavor further teaches the loss calculations associated identification results (Figs 3-4; pars 0058-0059, 0061-0063, 0072).
Therefore, consider Song as modified and Ma’s teachings as a whole, it would have been obvious to one of skill in the art before the filing date of invention to incorporate Ma’s teachings in Song as modified’s method to utilize a GAN for object identification application.
As to claim 4, Song as modified discloses the method according to claim 3, wherein the obtaining the generative adversarial loss value based on the first identification result of each reconstructed sample frame comprises: acquiring a probability distribution of the first identification result of each reconstructed sample frame as a first reconstruction probability distribution for each reconstructed sample frame (Ma: pars 0057-0059, 0063, 0069-0071, loss function being calculated with respect to the probability distribution of reconstructed and/or ground truth data;); and obtaining the generative adversarial loss value based on an expected value of a first reconstruction probability distribution for each reconstructed sample frame (Ma: pars 0057-0059, 0063, 0069-0071).
As to claim 5, Song as modified discloses the method according to claim 3, wherein the obtaining the spatial adversarial loss value based on the difference between the first identification result of each reconstructed sample frame and the first identification result of the corresponding to-be-encoded sample frame comprises: acquiring a probability distribution of the first identification result of each to-be-encoded sample frame as a first to-be-encoded probability distribution for each to-be-encoded sample frame (Kaplanyan: pars 0056, 0062-0063; Ma: pars 0057-0059, 0063, 0069-0071); and obtaining the spatial adversarial loss value based on an expected difference between the expected value of a first reconstructed probability distribution for each reconstructed sample frame and an expected value of the first to-be-encoded probability distribution for each to-be-encoded sample frame (Kaplanyan: pars 0056, 0062-0063; Ma: pars 0057-0059, 0063, 0069-0071).
As to claim 6, Song as modified discloses the method according to claim 3, wherein the obtaining the temporal adversarial loss value based on the difference between the second identification result of the spliced to-be-encoded sample frame and the second identification result of the spliced reconstructed sample frame comprises: acquiring a probability distribution of the second identification result of the spliced reconstructed sample frame as a second reconstructed probability distribution (Kaplanyan: pars 0056, 0062-0063; Ma: pars 0057-0059, 0063, 0069-0071); acquiring a probability distribution of the second identification result of the spliced to-be-encoded sample frame as a second to-be-encoded probability distribution; and obtaining the temporal adversarial loss value based on an expected difference between an expected value of the second reconstructed probability distribution and an expected value of the second to-be-encoded probability distribution (Kaplanyan: pars 0056, 0062-0063; Ma: pars 0057-0059, 0063, 0069-0071).
15-20. (Withdrawn)
Allowable Subject Matter
Claims 13-14 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Reasons for Allowance
Prior art of record (Song, Beriault, Kaplanyan and Ma) neither discloses alone nor teaches in combination functions and features recited in claim 13. Claim 14 depends from claim 13.
Examiner’s Note
Examiner has cited particular column, line number, paragraphs and/or figure(s) in the reference(s) as applied to the claims for the convenience of the Applicant. Although the specified citations are representative of the teachings of the art and are applied to the specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested from the applicant in preparing responses, to fully consider the reference(s) in entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the Examiner.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to QUN SHEN whose telephone number is (571)270-7927. The examiner can normally be reached on Mon-Fri 8:30-5:50 PT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on 571-272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/QUN SHEN/
Primary Examiner, Art Unit 2662
