Detailed Action
1. Claims1-16, 18-19 and 22-23 are pending in this Application.
Notice of Pre-AIA or AIA Status
2. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103, which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
3. Claims 1 and 18-19 are rejected under 35 U.S.C. 103(a) as being unpatentable over ZHENG He, (hereafter ZHENG ), CN 113365110 A, pub., 09/07/2021 ,in view of WANG et al., (hereafter WANG ), WO 2021163928 A1, pub. on 08/26/2012.
As to claim 1, ZHENG teaches A video processing method (Abstract, Video Frame Inserting Method, Model Training Method And Corresponding Device ), comprising:
determining a first optical flow of adjacent video frames (Abstract, claims1,8 and 16, The video acquisition unit is configured to acquire the target video; The second optical flow calculation unit is configured to determine the first optical flow and the second optical flow between two adjacent video frames based on two consecutive video frames in the target video and an optical flow estimation network trained by the method described in any one of claims 1-7); and
synthesizing an intermediate video frame according to the first video frame, the second video frame, the first optical flow and the second optical flow(Abstract, claims1,8 and 16, The video interpolation unit is configured to synthesize an intermediate video frame from two adjacent video frames based on the first optical flow and the second optical flow)wherein the intermediate video frame is an estimated video frame to be inserted between the first video frame and the second video frame (inherent In video interpolation, an intermediate frame is indeed generated and inserted between two consecutive frames);
however, it is noted that ZHENG does not specifical teach “a first image block in a first and a second image block in a second video, each of the first image block and the second image block is an image area comprising a plurality of pixel points”:
On the other hand in the same field of endeavor video image processing, and in particular to an optical flow acquisition method of WANG, teaches a first image block in a first and a second image block in a second video, each of the first image block and the second image block is an image area comprising a plurality of pixel points (claim 1, page 2 4th par., A method for obtaining optical flow, which is characterized in that it comprises: Determine the first similarity between the first pixel block in the i-1 frame image and the second pixel block in the i frame image; wherein the coordinates of the second pixel block are the same as the coordinates of the first pixel block , I is a positive integer)
It would have been obvious to a person of ordinary skill in the art before the effective
filing date of the claimed invention to replace a method of determining the optical flow based on the entire video frame (often called dense optical flow) taught by ZHENG with a well-known method of determining an optical flow based on image block (often called block-based or sparse optical flow block matching) of the video frame taught by WANG
The motivation for doing so is to provide ZHENG users with massive computational speed and improved noise robustness.
As to claim18, ZHENG An electronic device, comprising: a processor; and a memory configured to store executable instructions of the processor; wherein the processor is configured to read the executable instructions from the memory, and execute the executable instructions (claim 17, An electronic device, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, the memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7 or the method of claim 8);
regarding the remaining limitation of claim 18, all the remaining limitations are rejected the same as claim 1 except claim 18 is directed to a device claim. Thus, argument analogous to that presented above for claim 1 is applicable to claim 18.
As to claim 19, ZHENG teaches A non-transitory computer-readable storage medium, having stored therein instructions that, when run on a terminal device (Claim 18, A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-7 or the method of claim 8);
regarding the remaining limitation of claim 19, all the remaining limitations are rejected the same as claim 1 except claim 1 is directed to a computer program claim. Thus, argument analogous to that presented above for claim 1 is applicable to claim 19.
4. Claims 2-3 and 22-23 are rejected under 35 U.S.C. 103(a) as being unpatentable over ZHENG, CN 113365110 A ,in view of WANG, WO 2021163928 A1, further in view of HENG et al., (hereafter HENG), CN112104830, pub. 12/18/2020.
As claim 2 the combination ZHENG and WANG teaches “wherein the determining of the first optical flow of the first image block in the first video frame moving to the second video frame, and the second optical flow of the second image block in the second video frame moving to the first video (ZHENG: Abstract, claims1, 8 and 16, WANG: claim 1, page 2 4th par., this limitation is discussed in claim 1 above);
however, it is noted that modified ZHENG does not specifical teach “scaling the first video frame to obtain a first image set corresponding to the first video frame, and scaling the second video frame to obtain a second image set corresponding to the second video frame, wherein the first image set and the second image set each comprise a plurality of image layers with different resolutions; starting from a lowest resolution image layer in the first image set, calculating an initial optical flow of an image block pre-divided in a current layer image in the first image set, calculating an initial optical flow of an image block pre-divided in a next layer resolution image”
On the other hand HENG teaches scaling the first video frame to obtain a first image set corresponding to the first video frame, and scaling the second video frame to obtain a second image set corresponding to the second video frame, wherein the first image set and the second image set each comprise a plurality of image layers with different resolutions ( [0081]-[0089], Figure 4 illustrates a method for constructing a first image and a second image using a feature pyramid; After constructing the two feature pyramids, the system traverses each pyramid layer by layer from the top layer down, and the two feature maps located in the same layer are respectively designated as J1 and J2 of the optical flow calculation module corresponding to that layer. Since the feature map size in the feature pyramid gradually increases from the top to the bottom, the top layer corresponds to a smaller and less accurate feature map, while the bottom layer corresponds to a larger and more accurate feature map. Thus, starting from the top layer of the feature pyramid, the feature maps are input to the corresponding optical flow calculation modules layer by layer, which is conducive to gradually refining the optical flow calculation)
Starting from a lowest resolution image layer in the first image set, calculating an initial optical flow of an image block pre-divided in a current layer image in the first image set, calculating an initial optical flow of an image block pre-divided in a next layer resolution image; starting from a lowest resolution image layer in the second image set, calculating an initial optical flow of an image block pre-divided in a current layer image in the second image set, calculating an initial optical flow of an image block pre-divided in a next layer resolution image in the second image set according to the initial optical flow of the image block in the current layer image in the second image set, until an initial optical flow of an image block pre-divided in a highest resolution image layer in the second image set is calculated, and determining the initial optical flow of the image block pre-divided in the highest resolution image layer as the second optical flow of the second image block moving to the first video frame. ([0081]- [0089], [0202], video frame interpolation device 300, the first optical flow calculation unit 320 uses the image obtained after down sampling the first video frame as the first image input to each optical flow calculation module, and uses the image obtained after down sampling the second video frame as the second image input to each optical flow calculation module. This includes: down sampling the first video frame and the second video frame respectively to form an image pyramid of the first video frame and an image pyramid of the second video frame. Each layer of the image pyramid, starting from the top, corresponds to an optical flow calculation module of the first neural network, starting from the first optical flow calculation module. Starting from the top of the two image pyramids, the system traverses layer by layer downwards, using two down sampled images located at the same layer as the first image and the second image input to the optical flow calculation module corresponding to that layer, respectively. )
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify the method of ZHENG by incorporating the feature pyramid construction method taught by HENG to gradually refine optical flow.
The suggestion and motivation to do so stem from the known benefit of estimating rough, large-scale motions at lower resolutions and refining them at higher resolutions, a strategy that ensures highly accurate, artifact-free vector fields ,
As claim 3 the combination ZHENG and WANG teaches “wherein the calculating of the initial optical flow of the image block pre-divided in the current layer image in the first image set or the calculating of the initial optical flow of the image block pre-divided in the current layer image in the second image set first video (ZHENG: Abstract, claims1, 8 and 16, WANG: claim 1, page 2 4th par., this limitation is discussed in claim 1 above);
however, it is noted that modified ZHENG does not specifical teach “obtaining a first direction gradient value and a second direction gradient value of each pixel of the image block in the current layer image; determining a first pixel matrix, a second pixel matrix and a third pixel matrix corresponding to the image block in the current layer image according to the first direction gradient value and the second direction gradient value of the each pixel; and processing the first pixel matrix, the second pixel matrix and the third pixel matrix according
to a preset algorithm to obtain the initial optical flow corresponding to the image block in the
current layer image”
On the other hand HENG teaches obtaining a first direction gradient value and a second direction gradient value of each pixel of the image block in the current layer image; determining a first pixel matrix, a second pixel matrix and a third pixel matrix corresponding to the image block in the current layer image according to the first direction gradient value and the second direction gradient value of the each pixel; and processing the first pixel matrix, the second pixel matrix and the third pixel matrix according to a preset algorithm to obtain the initial optical flow corresponding to the image block in the current layer image ( [0013] In a possible implementation, the gradient information includes a sum of X-direction gradient values, an X-direction gradient value sum of squares, a sum of Y-direction gradient values, a sum of a sum of Y-direction gradient values, and an X-direction Y-direction gradient value product, and the method further includes: calculating a convolution of each pixel point of the first pixel block and the X-direction Sobel operator to obtain an X-direction gradient matrix, and calculating a convolution of each pixel point of the first pixel block and the Y-direction Sobel operator to obtain a Y-direction gradient matrix; accumulating and summing all gradient values of the X-direction gradient matrix to obtain a sum of the X-direction gradient values, and accumulating and summing each gradient value of the X-direction gradient matrix to obtain a sum of squares of the X-direction gradient values; Cumulative summation is performed on all gradient values of the Y-direction gradient matrix to obtain a sum of the Y-direction gradient values, and each gradient value of the Y-direction gradient matrix is squared and summed to obtain a sum of squares of the Y-direction gradient values; and the sum of the gradient values of the same position of the X-direction gradient matrix and the Y-direction gradient matrix is multiplied and accumulated to obtain a product of the X-direction Y-direction gradient values and the sum of the gradient values of the Y-direction gradient values.)
Claim 22 is rejected the same as claim 2 except claim 22 is directed to a device claim. Thus, argument analogous to that presented above for claim 2 is applicable to claim 22.
Claim 23 is rejected the same as claim 2 except claim 23 is directed to a computer program claim. Thus, argument analogous to that presented above for claim 2 is applicable to claim 23.
Allowable Subject Matter
5. Claims 4-16 is objected to as being dependent upon a rejected base claims, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
6. Regarding dependent claim 4 no prior art is found to anticipate or render the flowing limitation obvious;
“calculating a first offset vector between the first optical flow of the first image block currently to be detected and the second optical flow of the second image block in the second video frame and corresponding to the first optical flow, and comparing the first offset vector with a first threshold preset; in response to the first offset vector being greater than the first threshold, comparing a length of a vector of the first optical flow of the first image block currently to be detected with a length of an inverse vector of the second optical flow of the second image block in the second video frame and corresponding to the first optical flow; and in response to the length of the inverse vector of the second optical flow being less than the length of the vector of the first optical flow, adjusting the first optical flow of the first image block currently to be detected to the inverse vector of the second optical flow of the second image block in the second video frame and corresponding to the first optical flow.”
7. Regarding dependent claim 5 no prior art is found to anticipate or render the flowing limitation obvious:
“calculating a second offset vector between the second optical flow of the second image block currently to be detected and the first optical flow of the first image block in the first video frame and corresponding to the second optical flow, and comparing the second offset vector with a second threshold preset; in response to the second offset vector being greater than the second threshold, comparing a length of a vector of the second optical flow of the second image block currently to be detected with a length of an inverse vector of the first optical flow of the first image block in the first video frame and corresponding to the second optical flow; and in response to the length of the inverse vector of the first optical flow being less than the length of the vector of the second optical flow, adjusting the second optical flow of the second image block currently to be detected to the inverse vector of the first optical flow of the first image block in the first video frame and corresponding to the second optical flow.”
8. Regarding dependent claim 6 no prior art is found to anticipate or render the flowing limitation obvious:
“comparing the length of the vector corresponding to the first optical flow of the first image block of the row boundary or the column boundary currently to be detected with a preset threshold value; and in response to a number of the length of the vector less than the preset threshold value being greater than a third threshold preset, adjusting the first optical flow of the first image block of the row boundary or the column boundary currently to be detected to a first optical flow of a first image block of a row or column adjacent to the row boundary or the column boundary currently to be detected; and/or, performing anomaly detection on a second image block corresponding to a row boundary or a column boundary in the second video frame to obtain a length of a vector corresponding to the second optical flow of the second image block of the row boundary or the column boundary currently to be detected; comparing the length of the vector corresponding to the second optical flow of the second image block of the row boundary or the column boundary currently to be detected with a preset threshold value; and in response to a number of the length of the vector less than the preset threshold value being greater than a third threshold preset, adjusting the second optical flow of the second image block of the row boundary or the column boundary currently to be detected to a second optical flow of a second image block of a row or column adjacent to the row boundary or the column boundary currently to be detected.”
9. Regarding dependent claim 7 no prior art is found to anticipate or render the flowing limitation obvious:
“performing motion search adjustment on the first optical flow of the first image block moving to the second video frame to obtain a third optical flow of the first image block moving to the second video frame, and performing motion search adjustment on the second optical flow of the second image block moving to the first video frame to obtain a fourth optical flow of the second image block moving to the first video frame; and synthesizing the intermediate video frame according to the first video frame, the second video frame, the third optical flow of the first image block moving to the second video frame and the fourth optical flow of the second image block moving to the first video frame.”.
10. Regarding dependent claim 12 no prior art is found to anticipate or render the flowing limitation obvious.
“accumulating a square of the first direction gradient value of each pixel of each image block in the current layer image to a sum to obtain an element value of the each image block in the first pixel matrix and corresponding to the each image block, and filling the first pixel matrix according to a positional relationship between image blocks to obtain the first pixel matrix; accumulating a square of the second direction gradient value of each pixel in each image block in the current layer image to a sum to obtain an element value of the each image block in the second pixel matrix and corresponding to the each image block, and filling the second pixel matrix according to the positional relationship between the image blocks to obtain the second pixel matrix; and accumulating a product of the first direction gradient value and the second direction gradient value of each pixel in each image block in the current layer image to a sum to obtain an element value of the each image block in the third pixel matrix and corresponding to the each image block, and filling the third pixel matrix according to the positional relationship between the image blocks to obtain the third pixel matrix.”
11. Claims 8-11 and 13-16 are objected because they are dependent of the objected claim dependent claims 4-7.
Prior art not used in rejections but pertinent to the claims or disclosure
“METHOD FOR VIDEO FRAME INTERPOLATION ROBUST TO EXCEPTIONAL MOTION AND THE APPARATUS THEREOF”, KR 102244187 B1, pub. 04/26/2021, to RO YONGMAN disclosed:
A first prediction frame corresponding to the first frame and a second prediction frame corresponding to the second frame are obtained using a space-time automatic encoder, and the first prediction frame is obtained based on a difference between the first frame and the first prediction frame. Estimating the exceptional motion information for the region of the exceptional motion pattern in one frame, and calculating the exceptional motion information for the region of the exceptional motion pattern in the first frame based on the difference between the second frame and the second prediction frame. Video frame interpolation device, characterized in that to estimate.
Generation of estimating an optical flow between consecutive first and second frames of a video sequence and generating an intermediate frame between the first frame and the second frame through interpolation using the estimated optical flow part;
A detector for detecting exceptional motion information in the intermediate frame; and
An acquisition unit for obtaining an interpolated frame by correcting the intermediate frame based on the detected exceptional motion information,
A residual between the intermediate frame and the actual frame corresponding to the intermediate frame is predicted by using a pre-learned second neural network using the detected exceptional motion information and the intermediate frame as inputs, and the intermediate frame and the intermediate frame A video frame interpolation apparatus, comprising acquiring the interpolated frame by combining residuals (see claim 14,)
Using a pre-learned third neural network to estimate the optical flow in both directions between the first frame and the second frame, and using the optical flow in both directions between the first frame and the second frame, the first The optical flow between the frame and the intermediate frame and the optical flow between the second frame and the intermediate frame are calculated, and spatial warping using the calculated optical flow and the first frame, and the calculated optical flow and the second frame The video frame interpolation apparatus, characterized in that generating the intermediate frame based on the spatial warping used(see claim 16).
Contact Information
Any inquiry concerning this communication or earlier communication from the examiner should be directed to Mekonen Bekele whose telephone number is (469) 295-9077.The examiner can normally be reached on Monday-Friday from 9:00AM to 6:50 PM Eastern Time.
If attempt to reach the examiner by telephone are unsuccessful, the examiner’s supervisor Eng, George can be reached on (571) 272-7495.The fax phone number for the organization where the application or proceeding is assigned is 571-237-8300. Information regarding the status of an application may be obtained from the patent Application Information Retrieval (PAIR) system. Status information for published application may be obtained from either Private PAIR or Public PAIR. Status information for unpublished application is available through Privet PAIR only.
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have question on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866.
/MEKONEN T BEKELE/Primary Examiner, Art Unit 2699
Prosecution Insights