DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 1 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 1 fails to identify what the recited “fourth size” represents or what object and/or parameter has that size. Claim 1 recites that “the machine learning model is trained using a third image set of a third size” and that “the second size is equal to or smaller than a fourth size”. The claim identifies the respective subject associated with the first second and third sizes (the image respected image sets) the claim does not identify what subject or parameter has the “fourth size” Because of this, it is unclear whether the “fourth size” refers to a size of an image or an image set, a maximum allowable input size of the machine learning model, a threshold size for the second image set, or some other size parameter determined by the third size. A person of ordinary skill in the art would not reasonably know the scope of the limitation requiring “the second size is equal to or smaller than a fourth size,” because the claim does not define what this fourth size actually represents. 2-11 are subsequently rejected through their dependency chain to claim 1.
Claim 7 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 7 recites “acquiring, as partial data, at least one of an image acquired based on the second image set and the motion vector, and the motion vector, and concatenating a plurality of partial data corresponding to a plurality of different partial areas.” The actual scope of these alternative branches is unclear because “the motion vector” appears both as part of the recitation which has to do with image and again as a separate item following “and,” to the point where it is unclear what respective alternatives encompass “at least one”. Furthermore “concatenating a plurality of partial data corresponding to a plurality of different partial areas.” Does not clearly identify which alternatively claimed data is the plurality being concatenated. The scope of the acquisition and concatenation of partial data cannot be determined with reasonable certainty.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 6, 7, 9, 10, 11, and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Lin et al (Lin hereinafter US 20220398747 A1) In view of Epstein et al (Epstein hereinafter US 20240197262 A1)
As per claim 1
Lin teaches first image set including a first image and a second image of a first size (Figure 2, Paragraph [0057] “ The optical flow estimation system 100 can process frames 103 to generate an optical flow map (e.g., an optical flow map 108) by performing optical flow estimation for pixels within a pair of frames of the frames 103. “) a second image set of a second size smaller than the first size (Figure 2, Paragraph [0047] “the optical flow estimation system can result in optical flow estimation being performed within flow search areas that include 20% of the pixels of the target frame, flow search areas that include 10% of the pixels of the target frame, flow search areas that include 5% of the pixels of the target frame, flow search areas that include 2.5% of the pixels of the target frame, or any other number based on the techniques described herein.”) Paragraph [0073] “The updated pixel P1 location 202B can include a (W, H) pixel location of (5, 2) relative to the top-left-most pixel location of (0, 0). An optical flow vector can be computed for the pixel P1, indicating the velocity, displacement, or optical flow of the pixel P1 from the first frame 201 to the second frame 203. In one illustrative example, the optical flow vector (illustrated by vector 205) for the pixel P1 between the frames 201 and 203 is (1, 1), indicating the pixel P1 has moved one pixel location to the right and one pixel location down. In some cases,”) which corresponds to partial areas of the first image set (Figure 2) and acquiring a motion vector by inputting the second image set into a machine learning model, ( Figure 3A Paragraph [0073] “The updated pixel P1 location 202B can include a (W, H) pixel location of (5, 2) relative to the top-left-most pixel location of (0, 0). An optical flow vector can be computed for the pixel P1, indicating the velocity, displacement, or optical flow of the pixel P1 from the first frame 201 to the second frame 203. In one illustrative example, the optical flow vector (illustrated by vector 205) for the pixel P1 between the frames 201 and 203 is (1, 1), indicating the pixel P1 has moved one pixel location to the right and one pixel location down. In some cases, the optical flow estimation system 100 can determine a total of W×H optical flow vectors, which can represent one optical flow vector for each pixel location in the source frame, including determining optical flow vectors for the remaining pixels P2-P7 (not shown for simplicity).” Paragraph [0074] “FIG. 3A illustrates two frames, a source frame 310 and a target frame 322. The source frame 310 and target frame 322 can have dimensions of a width=W pixels and a height=H pixels, for a total of W×H pixels in each frame” Paragraph [0075] “by having a flow search area for each pixel that is smaller than the full frame, the total size of the correlation volume can be smaller than a correlation volume that includes a full frame worth of pixels (e.g., W×H pixels corresponding to the target frame 322) for each pixel in the source frame 310” Paragraph [0127] “s described above, the optical flow estimation systems and techniques described herein can be implemented using a neural network-based machine learning system. Illustrative examples of neural networks that can be used include one or more convolutional neural networks (CNNs), autoencoders, deep belief nets (DBNs), Recurrent Neural Networks (RNNs), Generative Adversarial Networks (GANs), Gated Recurrent Unit (GRUs), any combination thereof, and/or any other suitable neural network” ) wherein the motion vector is a motion vector in the second image based on the first image (Figure2 ,Figure 3A, Paragraph [0073] “he updated pixel P1 location 202B can include a (W, H) pixel location of (5, 2) relative to the top-left-most pixel location of (0, 0). An optical flow vector can be computed for the pixel P1, indicating the velocity, displacement, or optical flow of the pixel P1 from the first frame 201 to the second frame 203…the optical flow vector (illustrated by vector 205) for the pixel P1 between the frames 201 and 203 is (1, 1), indicating the pixel P1 has moved one pixel location to the right and one pixel location down. In some cases, the optical flow estimation system 100 can determine a total of W×H optical flow vectors, which can represent one optical flow vector for each pixel location in the source frame, including determining optical flow vectors for the remaining pixels P2-P7 (not shown for simplicity)”) wherein the machine learning model is trained using a third image set of a third size (Paragraph [0133] “The process can be repeated for a certain number of iterations for each set of training images until the neural network 900 is trained well enough so that the weights of the layers are accurately tuned.” Each training set image has its own size)
Lin does not expressly teach wherein the second size is equal to or smaller than a fourth size, and wherein the fourth size is set based on the third size
Epstein teaches wherein the machine learning model is trained using a third image set of a third size (Paragraph [0070] “ The images were cropped to a fixed size: N.sub.x×N.sub.y (e.g., 48*48 pixels to include the full left ventricle region). Data augmentation was performed using a 90° rotation. Cine images (validation data set) were scaled to match the resolution range of DENSE training images and also cropped into 48*48 pixels…The model was trained with the Adam optimizer and a total of 300 epochs. The initial learning rate was set to 1e-4, with a halving schedule for each 100 epochs… The study trained the neural network using DENSE training datasets according to the table of FIG. 4A” This shows that Epstein trains the model using an image set which has a predetermined dimension that corresponds to the dimension of image data that will be processed by the trained model for inference. Epstein establishes the size of subsequently processed image data based upon the size of the training image data. “Paragraph [0048] “The acquired images are resized (204) as pre-processed images to match the size of the trained data set of the trained neural network (e.g., 102a, 102b) and cropped to a region of interest. The pre-processed images provided (206) as input to the trained neural network can then provide a displacement map.” Paragraph [0070] “Cine images (validation data set) were scaled to match the resolution range of DENSE training images and also cropped into 48*48 pixels. “Epstein shows using the size of the training image set as the basis for a size constraint applicable to image data processed by the trained neural network during inference)
In regard to “wherein the second size is equal to or smaller than a fourth size, and wherein the fourth size is set based on the third size” In a combined teaching Lin teaches application of neural network based optical flow (motion vector) estimation using a first image set which contain a first and second image frame and a second image set that are partial regions of the first image set which is used to output a motion vector. Epstein teaches the well known routine machine learning practice of configuring training image data according to the dimensional characteristic desired for the inference stage. That includes training a neural network using a third image set having a size and a fourth image set being based off the third and being correlated with the inference phase second image set. Lin’s first and second image sets would be reflected in selecting the dimensions of the third training image set. The third training image set size would be the base of the 4th size representing the absolute max dimension of the cropped/partial image data to be processed by the neural network. Lin’s second image set would have a second size equal to or smaller than the fourth size wherein the fourth size is set based on the third size of the training image set. The first image set corresponds to the third set (claim does not recite these are different sizes) and second set corresponds to at least the maximum size of the fourth image set. This is routine training/inference data relationship parallels.
Accordingly a person of ordinary skill in the art would have found it obvious, at the time this invention was effectively filed, to configure Lin’s neural network training data according to the size of the image data desired to be processed during the inference stage as taught by Epstein. A person of ordinary skill in the art understands this to be a routine machine learning design. A person of ordinary skill in the art knows that trained data is ordinarily selected and prepared to represent the characteristics of the data in which the trained model is intended to operate. Epstein shows evidence of this practice by teaching that acquired images are resized “to match the size of the trained data set of the trained neural network”. When running Lin’s optical flow estimator for a desired first image size and a smaller partial reason size, a person of ordinary skill in the art would have predictably trained the model with corresponding image sizes. Based on those trained sizes, a maximum size for the partial image regions would be established for the inference phase. Lin’s second image set (which is a portion of the first image set) would therefore be maintained at or below that training based size constraint brought by the 4th image set. This gives size consistency between the data used to train the model and partial image data subsequently processed for motion vector estimation.
As per claim 2
Lin and Epstein teach all claim limitations previously rejected in claim 1’s 103 rejection. See claim 1’s 103 rejection.
Epstein teaches wherein the fourth size is equal to or smaller than 1.5 times as large as the third size. (Paragraph [0048] “ The acquired images are resized (204) as pre-processed images to match the size of the trained data set of the trained neural network (e.g., 102a, 102b) and cropped to a region of interest. Paragraph [00710] “The images were cropped to a fixed size: N.sub.x×N.sub.y (e.g., 48*48 pixels to include the full left ventricle region)…Cine images (validation data set) were scaled to match the resolution range of DENSE training images and also cropped into 48*48 pixels. “ This shows within Epstein they are equal in size.)
In a combined teaching, Lin teaches processing smaller partial regions for the second image set based off the first image set for motion vector estimation. Epstein teaches sizing inference data based on the training image size. Consistent with claim 1’s 103 rejection, applying Epstein to Lin would establish the fourth size based on the third size (The training set sizes) with Lin’s smaller partial regions (second image set) maintained in that training constraint. Since Epstein teaches matching these sizes, the fourth size is 1 times the third size and therefore no greater than the claimed 1.5 times the third size.
As per claim 3
Lin and Epstein teach all claim limitations previously rejected in claim 1’s 103 rejection. See claim 1’s 103 rejection.
Lin teaches wherein the first size is the number of pixels on one side of each of the first image and the second image. (Paragraph [0074] “ The source frame 310 and target frame 322 can have dimensions of a width=W pixels and a height=H pixels, for a total of W×H pixels in each frame.” Lin defines the size of both the source frame and target frame according to the number of pixels along each image dimension. One side of each image has W pixels and the other has H pixels. Either pixel dimension constitutes the claimed “first size” representing the number of pixels on one side of the first and second image in the set. )
As per claim 4
Lin and Epstein teach all claim limitations previously rejected in claim 1’s 103 rejection. See claim 1’s 103 rejection.
Lin in view of Epstein teach wherein the first size is larger than the fourth size. (Lin: Paragraph [0046] “For example, the optical flow estimation system can result in optical flow estimation being performed within flow search areas that include 20% of the pixels of the target frame, flow search areas that include 10% of the pixels of the target frame, flow search areas that include 5% of the pixels of the target frame, flow search areas that include 2.5% of the pixels of the target frame, or any other number based on the techniques described herein.” Paragraph [0074] “ The source frame 310 and target frame 322 can have dimensions of a width=W pixels and a height=H pixels, for a total of W×H pixels in each frame.” Paragraph [0075] “by having a flow search area for each pixel that is smaller than the full frame, the total size of the correlation volume can be smaller than a correlation volume that includes a full frame worth of pixels (e.g., W×H pixels corresponding to the target frame 322) for each pixel in the source frame 310. “ ) As established in claim 1, Epstein teaches creating the models input size limit based upon the size of the training data. So, when Epstein’s training consistent sizing is applied to Lin’s smaller partial region processing, the fourth size corresponds to the training based size constraint for those smaller partial regions rather than the dimensions full original frame. This follows the idea that in the combined Lin/Epstein pipeline, Lin’s second image set is the partial image based off the first image set. The model’s training is based off the desired inference, therefore just as the second image set is smaller than the first, the second image set’s counterpart that was used to train the network, the fourth image set, is smaller than the first image set. The first image set is to the third image set as the second image set is to the fourth image set.
As per claim 6
Lin and Epstein teach all claim limitations previously rejected in claim 1’s 103 rejection. See claim 1’s 103 rejection.
Lin teaches wherein the second image set is acquired by reducing the partial areas of the first image set. (Figure 2, Figure 3BParagraph [0046] “For example, the optical flow estimation system can result in optical flow estimation being performed within flow search areas that include 20% of the pixels of the target frame, flow search areas that include 10% of the pixels of the target frame, flow search areas that include 5% of the pixels of the target frame, flow search areas that include 2.5% of the pixels of the target frame, or any other number based on the techniques described herein.” Paragraph [0075] “by having a flow search area for each pixel that is smaller than the full frame, the total size of the correlation volume can be smaller than a correlation volume that includes a full frame worth of pixels (e.g., W×H pixels corresponding to the target frame 322) for each pixel in the source frame 310. This reduction of the correlation volume can reduce the required size of TCM as well as bring the total computational effort necessary to perform optical flow estimation between the source frame 310 and the target frame 322” )
As per claim 7
Lin and Epstein teach all claim limitations previously rejected in claim 1’s 103 rejection. See claim 1’s 103 rejection.
Lin teaches acquiring, as partial data… the motion vector (Paragraph [0073] “ An optical flow vector can be computed for the pixel P1, indicating the velocity, displacement, or optical flow of the pixel P1 from the first frame 201 to the second frame 203...optical flow estimation system 100 can determine a total of W×H optical flow vectors, which can represent one optical flow vector for each pixel location in the source frame” Lin gets an optical flow vector for respective pixel locations. The vectors are determined using corresponding partial flow search areas. Those vectors correspond to “claimed partial data corresponding to the respective partial areas”) and concatenating a plurality of partial data corresponding to a plurality of different partial areas. (Paragraph [0075] “the search areas in the target frame 322 corresponding to each pixel of the source frame 310 can be concatenated to form a correlation volume 332.”
As per claim 9
Lin and Epstein teach all claim limitations previously rejected in claim 1’s 103 rejection. See claim 1’s 103 rejection.
Lin in view of Epstein teach determining the second size based on at least one of the first size, the fourth size, and the machine learning model.
Lin teaches determining the second size based on at least one of the first size (Figure 2, Figure 3A, Figure 3B Paragraph [0111] “ the width and height of a square flow search area 722 can be varied depending on the characteristics of the video data being analyzed…an optical flow estimation system can use a 32×32-pixel square flow search area 722 when no optical flow history is available and a 16×16-pixel square flow search area when one or more frames of optical flow history is available.” Lin also contrasts partial areas with a search area encompassing the entire WxH of target. Therefore second size is based on the first size.
Epstein teaches determining the second size based on the fourth size and machine learning model. (Paragraph [0070] “ The images were cropped to a fixed size: N.sub.x×N.sub.y (e.g., 48*48 pixels to include the full left ventricle region). Data augmentation was performed using a 90° rotation. Cine images (validation data set) were scaled to match the resolution range of DENSE training images and also cropped into 48*48 pixels” Epstein correlates those crop dimensions with the ML model for example paragraph [0070] “ The neural network is configured with an input size of N.sub.x×N.sub.y×N.sub.t (e.g., 48*48*N.sub.t). This shows Epstein selecting size of the cropped input data according to both the size from training data and the input size of the trained neural network.
In a combined teaching, Lin teaches determining the dimensions of partial image regions used for optical flow estimation relative to the size of the original image data. Epstein teaches cropping subsequent processed image data according to the dimension of the training data and corresponding input size configuration/limitations of the network. This includes cropping images to 48x48 pixels where the neural network is configured for a corresponding 48x48 size input. Within the Lin/Epstein pipeline a person of ordinary skill in the art would find it obvious to base the second size on the first size of the original images (where the second size is cropped from) , the fourth size (which parallels the second size as inference inputs may parallel training inputs) that came from training and the input size characteristic/constraint the machine learning model is running on based on training.
As per claim 10
Lin and Epstein teach all claim limitations previously rejected in claim 1’s 103 rejection. See claim 1’s 103 rejection.
Lin teaches wherein the first image and the second image correspond to a plurality of frames at different times in moving image data. (Figure 1, Figure 2, Paragraph [0054] “In some examples, the optical flow estimation system 100 can capture the frames 103. In some examples, the optical flow estimation system 100 can obtain the frames 103 from a frame source. In some cases, the frame source can include one or more image capture devices and/or one or more video capture devices” Paragraph [0071] “FIG. 2 is a diagram illustrating an example of the optical flow estimation process that may be performed by the optical flow estimation system 100 of FIG. 1. FIG. 2 illustrates an example of a first frame 201 (also referred to as the source frame) and a second frame 203 (also referred to as the target frame) of a sequence of frames. The frames 201 and 203 can correspond to one of frames 103 of FIG. 1.” Paragraph [0080] “In one example, a sequence of frames in a video may contain a ball moving from point A to point B. The ball may for example, be slowing in velocity or lowering in height due to gravity. The optical flow estimation system (e.g., using flow target engine 102) can produce an estimated trajectory of the ball and estimate the likely position of the ball in the target frame” )
As per claim 11
Lin and Epstein teach all claim limitations previously rejected in claim 1’s 103 rejection. See claim 1’s 103 rejection.
Lin in view of Epstein teach processing cropped partial image regions using a CNN. Lin shows carrying and reducing the dimensions of partial regions (Paragraph [0046] “ For example, the optical flow estimation system can result in optical flow estimation being performed within flow search areas that include 20% of the pixels of the target frame, flow search areas that include 10% of the pixels of the target frame, flow search areas that include 5% of the pixels of the target frame, flow search areas that include 2.5% of the pixels of the target frame, or any other number based on the techniques described herein” Epstein shows a multilayer CNN having an effective receptive field extending across image region (Figure 4A Figure 4B Figure 9 Paragraph [109] “As shown in FIG. 9, a three-dimensional (3D) (2D+time) U-Net (26) was trained to predict intramyocardial displacement, a continuous variable, from contour data. The network depth was three, and the numbers of filters were 64, 128, 256, and 512, from the top to bottom levels, respectively. The kernel sizes were 3×3×3 for the convolutional layers and 4×4×2 for the pooling layers.”). It would have been obvious to select Lin’s reduced partial region size to fall within Epstein’s receptive field i.e. be smaller than the receptive field. A person of ordinary skill knows this lets the model use its available context field while reducing the amount of data used for interference. Lin is using a maximum of 32 pixels in either dimension for the partial area. Epstein’s disclosure of convolution and pooling “The network depth was three, and the numbers of filters were 64, 128, 256, and 512, from the top to bottom levels, respectively. The kernel sizes were 3×3×3 for the convolutional layers and 4×4×2 for the pooling layers” puts the receptive field well over Lin’s reduced spatial amount
As per claim 21
Lin and Epstein teach all claim limitations previously rejected in claim 1’s 103 rejection. See claim 1’s 103 rejection.
Lin in view of Epstein teach A non-transitory computer-readable storage medium storing a program that causes a computer to execute the image processing method according to claim 1. (Lin: Paragraph [0126] “Additionally, the processes described herein (including process 800 and/or other processes described herein) may be performed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) …the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.)
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Lin et al (Lin hereinafter US 20220398747 A1) In view of Epstein et al (Epstein hereinafter US 20240197262 A1) in further view of Xiao et al (Xiao hereinafter US 11367165 B2)
As per claim 8
Lin and Epstein teach all claim limitations previously rejected in claim 1’s 103 rejection. See claim 1’s 103 rejection.
Lin nor Epstein teach estimating a resolution-improved image corresponding to the second image set by inputting the second image set and the motion vector into the machine Learning model.
Lin in view of Epstein in further view of Xiao teach estimating a resolution-improved image corresponding to the second image set by inputting the second image set and the motion vector into the machine Learning model. (Xiao: Paragraph (21) “a computing system may take common inputs from modern game engines, i.e., color, depth and motion vectors at a lower resolution, and significantly up-sample the input imagery to the target high resolution using a temporal convolutional neural network) by inputting the second image set and the motion vector into the machine Learning model. (Paragraph (18) “Our super-sampling method may take the color, depth, and motion vectors of multiple low-resolution frames, and produce high-fidelity reconstructions by reducing aliasing and recovering scene details.” Paragraph (21) “Paragraph (21) “a computing system may take common inputs from modern game engines, i.e., color, depth and motion vectors at a lower resolution, and significantly up-sample the input imagery to the target high resolution using a temporal convolutional neural network”)
In a combined teaching Lin teaches acquiring a cropped second image set corresponding to partial areas of an original fist image set and corresponding optical flow motion vectors. Xiao teaches providing lower resolution image data together with corresponding motion vectors to a CNN to upsample the image and get a high resolution output. Lin/Epstein modified by Xiao would input Lin’s cropped second image set and its optical flow motion vectors into the learned super resolution model to estimate a resolution improved image arising to the second image set.
Accordingly a person of ordinary skill in the art, at the time this invention was effectively filed would have found it obvious to modify the Lin/Epstein pipeline to include Xiao’s super resolution acquisition method. A person of ordinary skill in the art would take Lin’s cropped partial image data and optical flow vectors as input to a learned super resolution model. Xiao teaches exploitation of lower resolution image information together with motion vectors in a CNN network to upsample the input imagery while producing a higher fidelity result. A person of ordinary skill in the art would have been motivated to use Lin’s readily available partial images and motion information as inputs to Xiao’s CNN to improve the spatial resolution of the partial images This modification uses both image content and motion information to make high resolution partial images with improved spatial and time consistency. This uses motion information to account for displacement of image content between frames and allows for accurate reconstruction of pixel detail in resolution improved images.
Claims 12, 13, 17, 18, 19, 20, and 22 are rejected under 35 U.S.C. 103 as being unpatentable over Liu et al (Liu hereinafter US 20230138053 A1) In view of Niklaus et al (Niklaus WO 2019168765 A1 “CONTEXT-AWARE SYNTHESIS FOR VIDEO FRAME INTERPOLATION“ )
As per claim 12
Liu teaches reducing a first image and a second image that include at least a portion of a same object at different positions (Paragraph [0027] “ The optical flow module 200 first estimates optical flow using downsampled input images and then adaptively upsamples the low-resolution flow back to the original high-resolution…input images 202 comprising at least two sequential image frames at a first resolution are input to a downscaling module 206….The downscaling module 206 takes the input images 202, say a pair of images {I.sub.1, I.sub.2}, at their original resolution, and downscales the input images 202 to generate low-resolution input images” and generating a third image corresponding to the first image and a fourth image corresponding to the second image (Paragraph [0027] “The downscaling module 206 takes the input images 202, say a pair of images {I.sub.1, I.sub.2}, at their original resolution, and downscales the input images 202 to generate low-resolution input images”) generating a first motion vector based on the third image and the fourth image using a first machine learning model; (Figure 5, Paragraph [0047] “At 515, method 500 generates a low-resolution optical flow estimation at a third resolution for the two downsampled input images with a first neural network…The first neural network may comprise the LFE module 210, for example, and so may comprise a convolutional neural network configured to estimate the optical flow between two image frames.”) generating a second motion vector by enlarging the first motion vector; (Figure 2 Figure 5, Paragraph [0034] “The UFE module 230 takes the low-resolution optical flow 218 as input and transforms the low-resolution optical flow 218 into a high-resolution optical flow 240…where the resolution of the high-resolution optical flow 240 matches the resolution of the original input images 202. ” Paragraph [0049] “ At 525, method 500 generates a flow estimation at the first resolution with a second neural network that iteratively up samples the low-resolution flow estimation guided by the downshuffled input images.” The first motion vector is from the low resolution optical flow. The upsampled high resolution optical flow estimation is the second motion vector made by upsampling the first low resolution flow estimation motion vector.
Liu does not teach generating a fifth image based on the first image, the second image, and the second motion vector using a second machine Learning model.
Liu in view of Niklaus teach generating a fifth image based on the first image, the second image, and the second motion vector using a second machine Learning model. (Niklaus: Figure 2, EXAMPLE EMBODIMENTS VIDEO FRAME INTERPOLATION: “the OFEs 210 first estimate bidirectional flows between the two consecutive input frames h and h…the SWEs 215 then pre warp the input frames h and h and their corresponding context maps The warped frames and context maps are then be fed into the FSN 300 to generate the interpolate result R at a desirable temporal position t e [0, 1]…the CAS may feed the warped input frames and corresponding context maps into a deep FSN 300 to generate the interpolation result R… a synthesis neural network that takes the two pre-warped images as input and directly generates the final output,” Niklaus’s result (Interpolation resulting in the R of figure 2) is the fifth image in this combined teaching. The fifth image is generated based on the first and second input images and the corresponding optical flow information. The optical flow pushes the pre warping of those images using the FSN model.
In a combined teaching Liu shows reducing first and second sequential images to create third and fourth reduced images. That generates third and fourth reduced images. A first low resolution optical flow motion vector comes from third and fourth images using the first network. Liu then “enlarges” the first motion vector to make a second high resolution motion vector corresponding to the resolution of the original images. Niklaus further teaches using optical flow information coming from the two input images to pre warp images and providing motion compensated image information to a neural network. That network synthesizes an interpolated frame. In the modified system Liu’s second motion vector would be used to motion compensate for the original first and second images as Niklaus teaches. The result would be the processed frame from Niklaus’s synthesis network. This is the fifth image.
Accordingly a person of ordinary skill in the art at the time this invention was effectively filed would have found it obvious to modify Liu’s pipeline by using Liu’s optical flow motion vector in the synthesis network processing taught by Niklaus. Liu shows that video interpolation is the next step in the full resolution optical flow estimation (Paragraph [0044] “For example, method 400 may output the optical flow estimation to one or more of memory 124, the display 180, or another algorithm for additional processing (e.g., video interpolation, video compression, and so on). Method 400 then returns.” Paragraph [0040] “For example, method 500 may output the optical flow estimation to one or more of memory 124, the display 180, or another algorithm for additional processing (e.g., video interpolation, video compression, and so on)”). Niklaus teaches using optical flow information to guide synthesis of an interpolated image from two input frames using a frame synthesis network. A skilled artisan would employ Niklaus’s known framework as Liu’s expressly contemplates video interpolation as the next step. This uses Liu’s full resolution motion information to guide generation of the output image. This combination is consistent with the closely related development of the two techniques. Feng Liu is a named inventor in both Liu and Niklaus with Niklaus likewise directed to neural network processing of optical flow for video frame generation.
As per claim 13
Liu and Niklaus teach all claim limitations previously rejected in claim 12’s 103 rejection. See claim 12’s 103 rejection.
Liu teaches wherein the first image and the second image are images extracted from the same moving image. (Paragraph [0046] “Method 500 begins at 505. At 505, method 500 receives two input images at a first resolution. For example, method 500 may receive two input images 202 comprising sequential image frames of a video acquired via the video source 110 at a high resolution.”)
As per claim 17
Liu and Niklaus teach all claim limitations previously rejected in claim 12’s 103 rejection. See claim 12’s 103 rejection.
Liu in view of Niklaus teaches wherein the fifth image is an image that constitutes a moving image acquired by increasing a frame rate of a moving image including the first image and the second image. (Niklaus states in section 3 “EXAMPLE EMBODIMENTS OF VIDEO FRAME INTERPOLATION”: “Given two consecutive video frames h and h. a goal may to generate an intermediate frame …at the temporal location t in between the two input frame…The warped frames and context maps are then be fed into the FSN 300 to generate the interpolate result R at a desirable temporal position t e [0, 1]” Niklaus also identifies the two images as consecutive image frames “Given two consecutive input frames h and h. the OFEs 210 first estimate bidirectional flows between the two consecutive input frames h and h” This shows Niklaus inserts an additional synthesized frame temporarily between the first and second consecutive frames. That increases the frame rate over the same time interval and increases the frame rate of the moving image. Generating one or more intermediate frames in-between the original frames gives the practical effect of increased frame rate.
As per claim 18
Liu and Niklaus teach all claim limitations previously rejected in claim 12’s 103 rejection. See claim 12’s 103 rejection.
Liu teaches wherein enlarging the first motion vector is performed using interpolation processing (Paragraph [0031] “Both PWC-Net and IRR-PWC apply an m-level processing architecture to predict the flow up to a level n (wherein n<m) and use a simply upsampling method to upscale the optical flow to the input resolution” This is the interpolation route. Then Liu states “ both networks first produce optical flow at 0.25 times smaller resolution before upsampling the flow back to input resolution using a simple method such as bilinear upsampling.” Liu’s true implementation can be seen as “the optical flow estimate 732 for the first input image 702 is obtained with PWC-Net configured to generate, from half-resolution input images, an estimate at half resolution which is then bilinearly upsampled to the original resolution (PWC-Net 0.5x+Bilinear 2x). “ In paragraph [0052]. The low resolution optical flow corresponds to the claimed first motion vector. Bilinear upsampling of that low resolution optical flow to the original image resolution is the claimed “enlarging the first motion vector using interpolation processing” )
As per claim 19
Liu and Niklaus teach all claim limitations previously rejected in claim 12’s 103 rejection. See claim 12’s 103 rejection.
Liu teaches wherein both the third image and the fourth image are images of a first size, (Paragraph [0027] The downscaling module 206 takes the input images 202, say a pair of images {I.sub.1, I.sub.2}, at their original resolution, and downscales the input images 202 to generate low-resolution input images {I.sub.1.sup.d, I.sub.2.sup.d}, wherein d is a downsampling factor applied to each side of each image... wherein d is a downsampling factor applied to each side of each image. Therefore, if the original resolution of the input images 202 is given by the width w times the height h of the images 202, such that the resolution is w×h, then the resolution of the downscaled or downsampled images generated by the downscaling module 206 is dw×dh. For example, if the downsampling factor d is 0.5 and the original resolution of the input images 202 is 1920×1080, then the resolution of the downsampled or downscaled images is 960×540.” Both third and fourth images are downsampled at the same time and are at the same size ) wherein a first training image set that is used to train the first machine learning model consists of images of the first size or larger.(Paragraph [0047] “ At 515, method 500 generates a low-resolution optical flow estimation at a third resolution for the two downsampled input images with a first neural network…The first neural network may comprise the LFE module 210, for example, and so may comprise a convolutional neural network configured to estimate the optical flow between two image frames. ” This is a first neural network which is a low resolution flow estimator. Paragraph [0055] “the LFE module 210 and the UFE module 230 may be trained in a multi-stage training procedure using the end-point-error (EPE) as the training loss For the first stage, the optical flow module 200 may be trained to produce high-resolution optical flows as similar to the results of running the original LFE module 210 on high-resolution input images (I.sub.1, I.sub.2). Specifically, a pre-trained model of LFE 210 is used to produce a high-resolution reference flow:” Paragraph [0057] “optical flow estimation optionally including one or more of the first through fourth examples, the method further comprises pre-training the first neural network with input images at the first resolution.” Yet Liu’s inference images supplied to the first network are downsampled below that first resolution (Paragraph [0042]). Liu also states in Paragraph [0047] “By initially downsampling the original input images to the second resolution, the speed performance of the first neural network is increased relative to the performance of the first neural network estimating optical flow directly from the input images.” Liu’s “first size” is the downsampled images. This means Liu’s training images at the first resolution are necessarily of the same general size or truly larger than the first size)
As per claim 20
Liu and Niklaus teach all claim limitations previously rejected in claim 12’s 103 rejection. See claim 12’s 103 rejection.
Liu teaches wherein both the first image and the second image are images of a third size, (Paragraph [0027] “The downscaling module 206 takes the input images 202, say a pair of images {I.sub.1, I.sub.2}, at their original resolution” Paragraph [0046] “Method 500 begins at 505. At 505, method 500 receives two input images at a first resolution…method 500 may receive two input images 202 comprising sequential image frames of a video acquired via the video source 110 at a high resolution…method 500 may receive two input images 202 comprising sequential image frames of a video acquired via the video source 110 at a high resolution.” The image pair already mapped in claim 12 are the first and second image. Both are provided at the same resolution. That common resolution constitutes the claimed third size. )
Niklaus teaches wherein a second training image set that is used to train the second machine learning model. (Section 3.2 Training “the frame synthesis neural network may be trained on examples of size 256 .sup.c 256 …50,000 training examples may be collected and the neural network may be trained on these for 50 epochs…the training samples may be collected from videos by splitting each video into sets of three frames, such that the center frame in each triplet serves as the ground truth.” Niklaus teaches a training image set used to trin the second machine learning model that was incorporated into Liu for claim 12’s 103 rejection) consists of images of a fourth size or smaller (“ Patches with a size of 300x300 may be extracted from these frame triplets…While the raw samples in the training dataset have a resolution of 300 .sup.c 300 pixels, in some implementations, only patches with a size of 256x256 may be used during training his allows for augmentation of the training data on the fly by choosing a different randomly cropped patch from a training sample each time it is being used.” Niklaus teaches that the second ML model is trained using training images having a predetermined size including images obtained by cropping larger training samples.)
In a combined teaching Liu gives the first and second images having the common resolution corresponding to the claimed third size. Niklaus then teaches training the second frame machine learning model using second training image set having predetermined dimension such as patches of 256x256 cropped from 300x300 samples In the modified system the first and second images have the common same size as taught by Liu while the second machine learning model is trained using the fourth size or smaller training image data as taught my Niklaus.
As per claim 22
Liu and Niklaus teach all claim limitations previously rejected in claim 12’s 103 rejection. See claim 12’s 103 rejection.
Liu in view of Niklaus teach A non-transitory computer-readable storage medium storing a program that causes a computer to execute the image processing method according to claim 1. (Liu: Paragraph [0021] “The memory 124 of the computing device 120 may comprise one or more physical, non-transitory devices configured to hold data and/or instructions executable by the processor 122 to implement the methods and processes described herein. When such methods and processes are implemented, the state of the memory 124 may be transformed (for example, to hold different data).”)
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Liu et al (Liu hereinafter US 20230138053 A1) In view of Niklaus et al (Niklaus WO 2019168765 A1 “CONTEXT-AWARE SYNTHESIS FOR VIDEO FRAME INTERPOLATION“ ) in further view of Niklaus et al (Niklaus’830 hereinafter US 20220321830 A1)
As per claim 14
Liu and Niklaus teach all claim limitations previously rejected in claim 12’s 103 rejection. See claim 12’s 103 rejection.
Liu and Niklaus do not clearly teach wherein the first image and the second image are images acquired by dividing a first original image and a second original image, respectively.
Niklaus’830 teaches wherein the first image and the second image are images acquired by dividing a first original image and a second original image, respectively. (Paragraph [0033] “, a singular prediction can be enhanced by combining the predictions of multiple transformed versions of the same input. Such transforms can include rotations, mirroring, or cropping. Any reversible image transforms can be used. Such self-ensembling can be used to enhance the frame interpolation quality. For example, the pair of frame inputs can be transformed (e.g., rotated, mirrored, cropped, etc.) and then processed again by the neural network to predict kernel coefficients. A new output image is then generated using theses kernel coefficients. This may be performed multiple times and the outputs combined (e.g., by taking the average, median, or otherwise merge the output images).”)
In a combined teaching Liu teaches the processing of the first and second images by reducing the images and then estimates a first motion vector from the reduced images using a machine learning model and enlarging the first motion vector to give a higher resolution information. Niklaus uses the motion information associated with the first and second input images in a frame synthesis machine learning process to generate an interpolated output image. Niklaus’830 teaches that the input images provided for frame interpolation processing may be obtained from transformed version of original input image pair. These original images include cropped images. They are then processed again by the network. In the modified system, Liu’s first and second image would comprise cropped/divided portions of each respected original image with the resulting image pair processed according to Liu and Niklaus’s motion estimation and image generation pipeline.
Accordingly, a person of ordinary skill in the art, at the time this invention was effectively filed would have found it obvious to further modify the Liu/Niklaus pipeline with Niklaus’830’s methodology of acquiring first and second images by transforming/cropping the image pairs. Niklaus’830 expresses applying reversible transformations, including cropping, to the pair of frame inputs and processing the transformed pair through a neural network to improve interpolation of frames and their quality. A person of ordinary skill in the art recognizes that there is reason to apply Niklaus’830’s cropping technique to the sequential image par of the previously modified Liu/Niklaus system as a known preprocessing technique for providing transformed portions of the original images to the interpolation pipeline with the goal of enhancing the interpolation quality. This enables the modified system to take cropped portions of the original frame pair to be processed through the learned motion vector and interpolation system. This allows predictions from cropped in inputs to be used instead of whole images which subsequently lowers processing load.
Claim 15 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Liu et al (Liu hereinafter US 20230138053 A1) In view of Niklaus et al (Niklaus WO 2019168765 A1 “CONTEXT-AWARE SYNTHESIS FOR VIDEO FRAME INTERPOLATION“ ) in further view of Qin et al (Qin hereinafter US 20210266496 A1)
As per claim 15
Liu and Niklaus teach all claim limitations previously rejected in claim 12’s 103 rejection. See claim 12’s 103 rejection.
Liu nor Niklaus teach the fifth image is an image corresponding to the first image and having a resolution higher than that of the first image.
Liu in view of Niklaus in further view of Qin teach the fifth image is an image corresponding to the first image and having a resolution higher than that of the first image. (Qin: Paragraph [0010] “ . A video super resolution deep neural network processor module is used for processing the uncompressed stream in conjunction with the motion vector information to produce a video super resolution stream “ Paragraph [0029] “The VSR DNN module 306 will process the incoming sequence of consecutive LR frames to generate a single HR frame, focusing on obtaining high-quality reconstruction results for each single frame. The VSR DNN module 306 will advantageously utilize the motion vector information obtained from the motion vector-based reconstruction module 304. The motion vector information greatly assists in the performance of higher accuracy VSR. The VSR DNN module 306 implements a DNN-based video super resolution engine that incorporates the motion vector information to generate super resolution images with high accuracy.” Qin teaches generating a high resolution output image corresponding to low resolution inputs and using motion vector information within the neural super resolution network. )
Liu already identifies optical flow as useful for downstream tasks stating in paragraph [00036] that ‘Optical flow estimation is one of the basic problems of computer vision, and serves as an important step for many computer vision tasks such as video super resolution,” Furthermore the existing modification of Liu and Niklaus already generates an image from temporally related frames using motion information. Qin teaches that the same type of time based frame data and motion information can be processed by a DNN to generate higher resolution images. This is a predictable modification of the output of an already existent motion guided learned processing pipeline. A person of ordinary skill in the art can see that instead of merely outputting the fifth image as an interpolated image, the image generation stage can be configured for super resolution using time related image information and the already available motion information to reconstruct high resolution output that corresponds to the first image.
Accordingly a person of ordinary skill, at the time this invention was effectively filed would have found it obvious to configure the Liu/Niklaus motion guided neural image generation system to include Qin’s methodology in order to generate the fifth image at a resolution higher than the corresponding first image. Niklaus already allows for generating an output image from time related input images using optical flow to guide a networks processing. Qin teaches that consecutive low resolution frames and their motion vector information can be processed by a video super resolution DNN to generate a high resolution frame. A person of ordinary skill in the art would have had reason to employ Qin’s known motion assisted super resolution pipeline in the Liu/Niklaus system to increase the spatial resolution of the generated image whilst using the motion information already produced and used by the combined system. All three references are already exploiting motion correspondence between video frames to improve image generation. Qin simply shows using that information for super resolution rather than only temporal interpolation.
As per claim 16
Liu and Niklaus teach all claim limitations previously rejected in claim 12’s 103 rejection. See claim 12’s 103 rejection.
Liu in view of Niklaus in further view of Qin teach wherein the fifth image is an image acquired by upscaling the first image. (Qin: Paragraph [0029] “The VSR DNN module 306 will process the incoming sequence of consecutive LR frames to generate a single HR frame, focusing on obtaining high-quality reconstruction results for each single frame. “ Qin states that “Video super-resolution (VSR) is the task of upscaling a video from a low-resolution to a high-resolution.” In paragraph [0002].)
In a combined teaching Liu in view of Niklaus teaches generating a fifth image from the first and second images using motion information in image generation processing. Qin further teaches configuring motion assisted learned video processing to upscale low resolution input frame images into a higher resolution output frame. The modified system of Liu, Niklaus and Qin would generate the fifth image by upscaling the first image using the motion information already available in the Liu/Niklaus system. Upscaling a low resolution image into a high resolution image is the task Qin is already set out to do.
Allowable Subject Matter
Claim 5 is objected to as being dependent upon a rejected base claim but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHANE WRENSFORD CODRINGTON whose telephone number is (571)272-8130. The examiner can normally be reached 8:00am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at (571) 272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHANE WRENSFORD CODRINGTON/Examiner, Art Unit 2667 /MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667
Prosecution Insights