Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This communication is a non-Final office action in merits. Claims 1-20, as originally filed, are presently pending and have been elected and considered below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or
nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over US 2020/0084427 A1, Sun et al. (hereinafter Sun) in view of US 2023/0379469 A1, Besenbruch et al. (hereinafter Besenbruch).
As to claim 1, Sun discloses an apparatus to estimate an optical flow, the apparatus comprising: one or more memories configured to store a plurality of images; and one or more processors coupled to the one or more memories and configured to:
obtain the plurality of images including at least a first image and a second image (Figs 1B, 1D-1E, obtaining a first and a second images; pars 0006, 0028-0032);
process the first image and the second image using a first neural network to obtain a set of features representing the first image and the second image (Figs 1A, 1C, 2, image pairs being encoded to generate corresponding features; pars 0006, 0028-0033, 0036);
predict, based on the set of features representing the first image and the second image, a latent representation of a change in an optical flow between at least the first image and the second image (Figs 1A, 1C, 2; pars 0005, 0020, 0022-0023, predict the optical flow change (pixel motion representation) based on encoded features of the image (e.g. latent representation)); and
estimate the optical flow based on the predicted latent representation to generate an estimated optical flow, wherein the optical flow is associated with movement of pixels from at least the first image to the second image (Figs 1A-1B, 1D-1E, 2A; pars 0005, 0021-0025, 0031-0032, 0036, 0047-0052, optical flow estimate representing pixel movements).
Sen does not expressly disclose predicting laten representation of a change in an optical flow using a neural ordinary differential equation that uses a second neural network to generate a predicted latent representation.
Besenbruch, in the same or similar field of endeavor, further teaches process the first image and the second image using a first neural network to obtain a set of features representing the first image and the second image (Figs 1; pars 0008-0009, 0067-0068, 0036); predict, based on the set of features representing the first image and the second image, a latent representation between at least the first image and the second image (pars 0091-0092, 0103-0104, 0166-0167); and updating and minimizing a loss function for an neural network modeling for optical flow data using an ordinary differential equation (pars 0166-0171, 0415, 0431, 0445, 0461, 1111).
Therefore, consider Sun and Besenbruch’s teachings as a whole, it would have been obvious to one of skill in the art before the filing date of invention to incorporate Besenbruch’s teachings in Sun’s system to estimate or predict optical flow change using an ODE in neural network modeling.
As to claim 2, Sun as modified discloses the apparatus of claim 1, wherein the one or more processors are configured to: update parameters of the first neural network and the second neural network based on a loss function to generate updated parameters (Sun: Fig 2A; pars 0069-0070, parameters being updated to reduce a loss function; Besenbruch: pars 0009, 0014, 0061, 0067-0068, 0082-0083, 0086, 0105-0107, 0112, training a first and a second neural networks with respective loss function and weights updates), wherein the loss function is based on a difference between the estimated optical flow and a ground truth optical flow (Sun: Fig 2A; pars 0066-0067, 0069-0070);
obtain a third image (Sun: Fig 1B; par 0029, a third image or a sequence of mages beyond the first and second image); process the second image and the third image using the first neural network with the updated parameters to obtain a set of features representing the second image and the third image (Sun: Figs 1B; pars 0004-0006, 0022-0024, 0028, features representing a first, a second, and/or a third image (e.g. from frames of the video sequence) being input into a decoder to predict optical flow; Besenbruch: pars 0180, 0217, 0233, 0243, 0261, multiple neural networks including a first, a second, and a third neural networks being trained to produce respective latent representations); and predict, based on the set of features representing the second image and the third image, an updated latent representation of an optical flow between the second image and the third image using a neural ordinary different equation parameterized by the second neural network with the updated parameters (Sun: pars 0064, 0070, 0072, claim 21; Besenbruch: pars 0109, 0166-0172, 0272, 0415, 0431, 0445, 0461, 1111).
As to claim 3, Sun as modified discloses the apparatus of claim 2, wherein the loss function is based on a difference between the estimated optical flow and the ground truth optical flow (Sun: Fig 2A; pars 0066-0067, 0069-0070, a loss function representing differences between the ground-truth and optical flow, disparity, and occlusion estimate).
As to claim 4, Sun as modified discloses the apparatus of claim 1, wherein the first neural network comprises a feature encoder and a context encoder (Sun: Figs 1B-1D, context network/context process unit for context encoding; pars 0021, 0027, 0037, 0047, 0058).
As to claim 5, Sun as modified discloses the apparatus of claim 1, wherein the first neural network comprises a convolutional neural network (Sun: pars 0020, 0032, 0036, 0038, a convolutional neural network).
As to claim 6, Sun as modified discloses the apparatus of claim 1, wherein the set of features comprises a four-dimensional volume based on output from a feature encoder (Sun: pars 0005, 0024, 0082, 0088, features associated with movement of objects from frame-to-frame forms three-dimensional space/volume) and context encoder data output from a context encoder (Sun: Figs 1A-1D; pars 0021, 0027, 0037. 0047, the context encoder produces one-dimensional volume). Note combining 3D features volume and 1D context volume forming a 4D volume.
As to claim 7, Sun as modified discloses the apparatus of claim 1, wherein the second neural network comprises at least one of a multilayer perceptron, a transformer, or a convolutional neural network (Sun: pars 0020, 0032, 0034, 0047, a CNN with multiple layers).
As to claim 8, Sun as modified discloses the apparatus of claim 1, wherein, to estimate the optical flow based on the predicted latent representation, the one or more processors are configured to decode the predicted latent representation (Sun: Figs 1A-1B, 1D-1E, 2B; pars 0005, 0020-0022).
As to claim 9, Sun as modified discloses the apparatus of claim 1, wherein the optical flow is an estimate of per pixel movement from the first image to the second image (Sun: pars 0005, 0023-0024, estimating pixel movements between images).
As to claim 10, Sun as modified discloses the apparatus of claim 1, wherein the latent representation of the change in the optical flow is between multiple images and the second image (Sun: Fig 1A; pars 0022-0025, 0035-0036, optical flow being predicted from extracted features based on pixel motions in different images), wherein the multiple images comprise the first image and at least one other image (Sun: Figs 2; pars 0006, 0028-0029, first and second images and a sequence of image pairs).
As to claim 11, it is a method claim necessitated claim 1. Rejection of claim 1 is therefore incorporated herein.
As to claims 12-19, they are rejected with the same reason as set forth in claims 2-9, respectively.
As to claim 20, it recites a non-transitory CRM storing instructions executed to perform functions and features of claim 1. Rejection of claim 1 is therefore incorporated herein.
Examiner’s Note
Examiner has cited particular column, line number, paragraphs and/or figure(s) in the reference(s) as applied to the claims for the convenience of the Applicant. Although the specified citations are representative of the teachings of the art and are applied to the specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested from the applicant in preparing responses, to fully consider the reference(s) in entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the Examiner.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to QUN SHEN whose telephone number is (571)270-7927. The examiner can normally be reached on Mon-Fri 8:30-5:50 PT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on 571-272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/QUN SHEN/
Primary Examiner, Art Unit 2662
Prosecution Insights