Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on June 03, 2026 has been entered.
Response to Arguments
Applicant’s arguments with respect to claims 1, 3-14, and 16-31 have been considered but are moot because the new ground of rejection.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 3-14, and 16-31 are rejected under 35 U.S.C. 103 as being unpatentable over JIANG et al. (US 2022/0217371 A1) in view of LU (CN 101064849 A).
Regarding claim 1, JIANG et al. discloses a device (Figs. 12 and 13) comprising: a memory configured to store image data (see Fig. 3, buffer memory 303, page 2, paragraph #0037, “A receiver 302 may receive one or more codec video sequences to be decoded by the decoder 300; … . To combat network jitter, a buffer memory 303 may be coupled in between receiver 302 and entropy decoder/parser 304 (“parser” henceforth). When receiver 302 is receiving data from a store/forward device of sufficient bandwidth and controllability, or from an isosychronous network, the buffer 303 may not be needed, or can be small. For use on best effort packet networks such as the Internet, the buffer 303 may be required, can be comparatively large and can advantageously of adaptive size”); and one or more processors coupled to the memory, the one or more processors (see, page 1, paragraph #0006, "According to exemplary embodiments, there is included a method and apparatus comprising memory configured to store computer program code and a processor or processors configured to access ") configured to: obtain a bitstream corresponding to an encoded version of an image frame (see page 8, paragraph #0091, "On the decoder side, such as described for example with respect to the flowchart 1100 of FIG. 11 and various modules of FIG. 12, received encoded bitstreams, at S111, "); based on determining that the bitstream includes a virtual reference frame usage indicator, generate a virtual reference frame based on synthesis support data included in the bitstream, wherein the synthesis support data includes motion-base data (see page 5, paragraphs #0065-#0066, "A Predictive picture (P picture) may be one that may be coded and decoded using intra prediction or inter prediction using at most one motion vector and reference index to predict the sample values of each block" and "A Bi-directionally Predictive Picture (B Picture) may be one that may be coded and decoded using intra prediction or inter prediction using at most two motion vectors and reference indices to predict the sample values of each block. Similarly, multiple-predictive pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block" and page 3, paragraph #0038, “The video decoder 300 may include a parser 304 to reconstruct symbols 313 from the entropy coded video sequence. … . The entropy decoder/parser may also extract from the coded video sequence information such as transform coefficients, quantizer parameter values, motion vectors, and so forth”); and generate a decoded version of the image frame based on the virtual reference frame (see page 8, paragraph #0092, "At S114, the decoded EFA features F.sub.b,1, F.sub.b,2, and the up-sampled sequence X=x.sub.1, x.sub.2, are aggregated together by a Fusion module 139 to generate the final reconstructed video sequence X={circumflex over (x)}.sub.1, {circumflex over (x)}.sub.2, …, data 140. The Fusion module can be a small DNN, where for generating {circumflex over (x)}.sub.i at time stamp i, the Fusion module can use only L.sub.i, B.sub.i, andx.sub.ifrom the same time stamp, or use L.sub.i-n,, L.sub.i+m, B.sub.i-n, B.sub.i+m, and use x.sub.i-n, x.sub.i+1 from a few neighbouring time stamps. " and page 8, paragraph #0094, "Also, according to exemplary embodiments, a fusion module, a compute adversarial loss module 241, a compute reconstruction loss module 242, a compute perceptual loss module 243, and the workflow 1300 also includes various data 221, 224, 225, 229, 228, 232, 233, 236, 238, and 240").
However, JIANAG et al. does not specifically discloses that the motion-based data indicates a global motion of the image frame, a change in position of an image capture device that captured the image frame, or both.
LU teaches in the same field of coding/decoding video signal using global motion vector of the image (see pages 3-4 of the translation of CN 101064849 A, “if there is other frame in one of pixel blocks and the current pixel block is the same or similar, then it is called the best matching block of the current pixel block. movement of the image is small, the same pixel block position in the preceding frame as the best matching block. movement of the image, the position or P of motion vector for motion estimation in frame and B frame to find the best matching block, such as using the global motion vector of the image. when one moving object, motion vector recording the displacement of it, when the next frame coding, it can find the optimal matching block using the motion vector. I frame has no motion vector, displacement with respect to the previous frame can be composed of P-frame or B-frame motion vectors. such as I1 and I5 in FIG. 6, is an intra-prediction encoded image, P2, P3 and P4 is a forward prediction encoded image, the pixel block of I5 602 an optimal matching block in I1 can use the preceding P frame motion vector of the corresponding pixel block and 603 + 604 + 605 + 606, wherein I5 relative displacement vector of the P4 603 is equal to the motion vector 604. FIG. 7 is another example of I1 and I4 is an intra-prediction encoded image, P2 and P3 are forward prediction encoded image, 703 is I4 relative to the displacement vector of the P3, making 703 equal to P3 relative to the global motion vector of the P2 pixel block of the I4 702 using 703 for searching for an optimal matching block, points 701, 701 702 as the best matching block. In a word, the one pixel block and the relative displacement of the optimal matching block can be the frame according to their display order (display order), using a motion vector or a motion vector extension (extrapolate)”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the encoder/decoder as taught by LU into JIANG et al.’s system in order to effectively reduce the amount of calculation, save resource and meet the requirement of real-time encoding.
Regarding claim 3, JIANG et al. also discloses wherein the bitstream indicates a first set of reference candidates that includes the virtual reference frame (see page 5, paragraphs #0065-#0066, "A Predictive picture (P picture) may be one that may be coded and decoded using intra prediction or inter prediction using at most one motion vector and reference index to predict the sample values of each block" and "A Bi-directionally Predictive Picture (B Picture) may be one that may be coded and decoded using intra prediction or inter prediction using at most two motion vectors and reference indices to predict the sample values of each block. Similarly, multiple-predictive pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block", page 3, paragraph #0038, “The video decoder 300 may include a parser 304 to reconstruct symbols 313 from the entropy coded video sequence. … . The entropy decoder/parser may also extract from the coded video sequence information such as transform coefficients, quantizer parameter values, motion vectors, and so forth” and page 8, paragraph #0092, "At S114, the decoded EFA features F.sub.b,1, F.sub.b,2, and the up-sampled sequence X=x.sub.1, x.sub.2, are aggregated together by a Fusion module 139 to generate the final reconstructed video sequence X={circumflex over (x)}.sub.1, {circumflex over (x)}.sub.2, …, data 140. The Fusion module can be a small DNN, where for generating {circumflex over (x)}.sub.i at time stamp i, the Fusion module can use only L.sub.i, B.sub.i, andx.sub.ifrom the same time stamp, or use L.sub.i-n,, L.sub.i+m, B.sub.i-n, B.sub.i+m, and use x.sub.i-n, x.sub.i+1 from a few neighbouring time stamps. ", and page 8, paragraph #0094, "Also, according to exemplary embodiments, a fusion module, a compute adversarial loss module 241, a compute reconstruction loss module 242, a compute perceptual loss module 243, and the workflow 1300 also includes various data 221, 224, 225, 229, 228, 232, 233, 236, 238, and 240").
Regarding claim 4, JIANG et al. further discloses wherein the bitstream indicates one or more additional first sets of reference candidates that include one or more additional virtual reference frames associated with one or more additional image frames of a sequence of image frames (see page 5, paragraphs #0065-#0066, "A Predictive picture (P picture) may be one that may be coded and decoded using intra prediction or inter prediction using at most one motion vector and reference index to predict the sample values of each block" and "A Bi-directionally Predictive Picture (B Picture) may be one that may be coded and decoded using intra prediction or inter prediction using at most two motion vectors and reference indices to predict the sample values of each block. Similarly, multiple-predictive pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block", page 3, paragraph #0038, “The video decoder 300 may include a parser 304 to reconstruct symbols 313 from the entropy coded video sequence. … . The entropy decoder/parser may also extract from the coded video sequence information such as transform coefficients, quantizer parameter values, motion vectors, and so forth”, and page 8, paragraph #0092, "At S114, the decoded EFA features F.sub.b,1, F.sub.b,2, and the up-sampled sequence X=x.sub.1, x.sub.2, are aggregated together by a Fusion module 139 to generate the final reconstructed video sequence X={circumflex over (x)}.sub.1, {circumflex over (x)}.sub.2, …, data 140. The Fusion module can be a small DNN, where for generating {circumflex over (x)}.sub.i at time stamp i, the Fusion module can use only L.sub.i, B.sub.i, andx.sub.ifrom the same time stamp, or use L.sub.i-n,, L.sub.i+m, B.sub.i-n, B.sub.i+m, and use x.sub.i-n, x.sub.i+1 from a few neighbouring time stamps. ").
Regarding claim 5, JIANG et al. discloses wherein the bitstream further indicates a second set of reference candidates including one or more previously decoded image frames (see page 5, paragraphs #0065-#0066, "A Predictive picture (P picture) may be one that may be coded and decoded using intra prediction or inter prediction using at most one motion vector and reference index to predict the sample values of each block" and "A Bi-directionally Predictive Picture (B Picture) may be one that may be coded and decoded using intra prediction or inter prediction using at most two motion vectors and reference indices to predict the sample values of each block. Similarly, multiple-predictive pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block").
Regarding claim 6, JIANG et al. discloses wherein the bitstream includes a supplemental enhancement information (SEI) message indicating the synthesis support data (see page 3, paragraph #0038, "The video decoder 300 … The control information for the rendering device(s) may be in the form of Supplementary Enhancement Information (SEI messages) or Video Usability Information parameter set fragments (not depicted). ..").
Regarding claim 7, JIANG et al. discloses wherein the bitstream includes an encoded version of a previous image frame that is prior to the image frame in a sequence of image frames (see page 5, paragraphs #0065-#0066, "A Predictive picture (P picture) may be one that may be coded and decoded using intra prediction or inter prediction using at most one motion vector and reference index to predict the sample values of each block" and "A Bi-directionally Predictive Picture (B Picture) may be one that may be coded and decoded using intra prediction or inter prediction using at most two motion vectors and reference indices to predict the sample values of each block. Similarly, multiple-predictive pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block"); the synthesis support data includes facial landmark data indicating locations of facial features in the image frame (see page 5, paragraph #0066, "A Bi-directionally Predictive Picture (B Picture) may be one that may be coded and decoded using intra prediction or inter prediction using at most two motion vectors and reference indices to predict the sample values of each block. Similarly, multiple-predictive pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block", page 7, paragraph #0088, " According to exemplary embodiments, to locate a pre-determined set of facial landmarks for each detected face (e.g., landmarks around left/right eyes, nose, mouse, etc.). ...", and page 8, paragraph #0091, "On the decoder side, and the decoded facial landmark features F.sub.l,1, F.sub.1,2, data 128. …"), and wherein the one or more processors are configured to: generate a decoded version of the previous image frame, wherein the facial features have previous positions in the decoded version of the previous image frame (see page 5, paragraph #0066, "A Bi-directionally Predictive Picture (B Picture) may be one that may be coded and decoded using intra prediction or inter prediction using at most two motion vectors and reference indices to predict the sample values of each block. Similarly, multiple-predictive pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block", page 7, paragraph #0088, " According to exemplary embodiments, to locate a pre-determined set of facial landmarks for each detected face (e.g., landmarks around left/right eyes, nose, mouse, etc.). ...", and page 8, paragraph #0091, "On the decoder side, and the decoded facial landmark features F.sub.l,1, F.sub.1,2, data 128. …"); and adjust the previous positions of the facial features based on the locations of the facial features to generate the virtual reference frame having adjusted positions of the facial features (see page 5, paragraph #0066, "A Bi-directionally Predictive Picture (B Picture) may be one that may be coded and decoded using intra prediction or inter prediction using at most two motion vectors and reference indices to predict the sample values of each block. Similarly, multiple-predictive pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block", page 7, paragraph #0088, " According to exemplary embodiments, to locate a pre-determined set of facial landmarks for each detected face (e.g., landmarks around left/right eyes, nose, mouse, etc.). ...", and page 8, paragraph #0091, "On the decoder side, and the decoded facial landmark features F.sub.l,1, F.sub.1,2, data 128. …").
Regarding claim 8, LU teaches wherein the synthesis support data includes the motion-based data indicating the global motion of the image frame, and wherein the one or more processors are configured to generate the virtual reference frame based at least in part on a previously decoded image frame and the global motion of the image (see pages 3-4 of the translation of CN 101064849 A, “if there is other frame in one of pixel blocks and the current pixel block is the same or similar, then it is called the best matching block of the current pixel block. movement of the image is small, the same pixel block position in the preceding frame as the best matching block. movement of the image, the position or P of motion vector for motion estimation in frame and B frame to find the best matching block, such as using the global motion vector of the image. when one moving object, motion vector recording the displacement of it, when the next frame coding, it can find the optimal matching block using the motion vector. I frame has no motion vector, displacement with respect to the previous frame can be composed of P-frame or B-frame motion vectors. such as I1 and I5 in FIG. 6, is an intra-prediction encoded image, P2, P3 and P4 is a forward prediction encoded image, the pixel block of I5 602 an optimal matching block in I1 can use the preceding P frame motion vector of the corresponding pixel block and 603 + 604 + 605 + 606, wherein I5 relative displacement vector of the P4 603 is equal to the motion vector 604. FIG. 7 is another example of I1 and I4 is an intra-prediction encoded image, P2 and P3 are forward prediction encoded image, 703 is I4 relative to the displacement vector of the P3, making 703 equal to P3 relative to the global motion vector of the P2 pixel block of the I4 702 using 703 for searching for an optimal matching block, points 701, 701 702 as the best matching block. In a word, the one pixel block and the relative displacement of the optimal matching block can be the frame according to their display order (display order), using a motion vector or a motion vector extension (extrapolate)”).
The motivation to combine the references has been discussed in claim 1 above.
Regarding claim 9, JIANG et al. discloses wherein the one or more processors are configured to use the motion-based data to warp a previously decoded image frame to generate the virtual reference frame, and wherein the synthesis support data includes the motion-based data (page 5, paragraphs #0065-#0066, "A Predictive picture (P picture) may be one that may be coded and decoded using intra prediction or inter prediction using at most one motion vector and reference index to predict the sample values of each block" and "A Bi-directionally Predictive Picture (B Picture) may be one that may be coded and decoded using intra prediction or inter prediction using at most two motion vectors and reference indices to predict the sample values of each block. Similarly, multiple-predictive pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block").
Regarding claim 10, JIANG et al. discloses wherein the one or more processors are configured to use a model to generate the virtual reference frame, and wherein an input to the model includes the synthesis support data and at least one previously decoded image frame (see page 5, paragraphs #0065-#0066, "A Predictive picture (P picture) may be one that may be coded and decoded using intra prediction or inter prediction using at most one motion vector and reference index to predict the sample values of each block" and "A Bi-directionally Predictive Picture (B Picture) may be one that may be coded and decoded using intra prediction or inter prediction using at most two motion vectors and reference indices to predict the sample values of each block. Similarly, multiple-predictive pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block" and page 8, paragraph #0094, "Also, according to exemplary embodiments, there are several components in the proposed framework that needs to be trained, and such training will be described with respect to FIG. 13 which illustrates a workflow 1300 of an exemplary training process according to exemplary embodiments. …").
Regarding claim 11, JIANG et al. discloses all the claimed limitations as discussed in claim 1 above including that the terminals 101, 102, 103, and 104 are connected to each other using wireline and/or wireless communication networks (see page 2, paragraph #0033. "In FIG. 1, the terminals 101, 102, 103 and 104 may be illustrated as servers, personal computers and smart phones but the principles of the present disclosure are not so limited. The network 105 represents any number of networks that convey coded video data among the terminals 101, 102, 103 and 104, including for example wireline and/or wireless communication networks." ...) except for providing a modem configured to transmit the bitstream to a second device.
The use of modem to transmit data between terminals is old and well-known in the art and Official Notice is taken.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the well-known modem to transmit data between terminals 101, 102, 103, and 104 of JIANG et al. since it merely amounts to selecting equivalent available transmitters between terminals.
Regarding claim 12, JIANG et al. discloses further comprising a display device configured to display the decoded version of the image frame (see page 2, paragraph #0032, "FIG. 1 illustrates a second pair of terminals 101 and 104 provided to support bidirectional transmission of coded video that may occur, for example, during videoconferencing. may decode the coded data and may display the recovered video data at a local display device.").
The method claim 13 is rejected for the same reason as discussed in the corresponding apparatus claim 1 above.
Regarding claim 14, JIANG et al. discloses a device (Figs. 12-13) comprising: a memory configured to store image data (see Fig. 3, buffer memory 303, page 2, paragraph #0037, “A receiver 302 may receive one or more codec video sequences to be decoded by the decoder 300; … . To combat network jitter, a buffer memory 303 may be coupled in between receiver 302 and entropy decoder/parser 304 (“parser” henceforth). When receiver 302 is receiving data from a store/forward device of sufficient bandwidth and controllability, or from an isosychronous network, the buffer 303 may not be needed, or can be small. For use on best effort packet networks such as the Internet, the buffer 303 may be required, can be comparatively large and can advantageously of adaptive size”); and one or more processors coupled to the memory (see, page 1, paragraph #0006, "According to exemplary embodiments, there is included a method and apparatus comprising memory configured to store computer program code and a processor or processors configured to access ") configured to: obtain synthesis support data associated with an image frame of a sequence of image frames, wherein the synthesis support data includes motion-base data (see page 8, paragraph #0091, "On the decoder side, such as described for example with respect to the flowchart 1100 of FIG. 11 and various modules of FIG. 12, received encoded bitstreams, at S111, "); based on determining that the bitstream includes a virtual reference frame usage indicator, generate a virtual reference frame based on synthesis support data included in the bitstream, wherein the synthesis support data includes motion-base data (see page 5, paragraphs #0065-#0066, "A Predictive picture (P picture) may be one that may be coded and decoded using intra prediction or inter prediction using at most one motion vector and reference index to predict the sample values of each block" and "A Bi-directionally Predictive Picture (B Picture) may be one that may be coded and decoded using intra prediction or inter prediction using at most two motion vectors and reference indices to predict the sample values of each block. Similarly, multiple-predictive pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block" and page 3, paragraph #0038, “The video decoder 300 may include a parser 304 to reconstruct symbols 313 from the entropy coded video sequence. … . The entropy decoder/parser may also extract from the coded video sequence information such as transform coefficients, quantizer parameter values, motion vectors, and so forth”); selectively generate a virtual reference frame based on the synthesis support data (see page 5, paragraph #0066, "A Bi-directionally Predictive Picture (B Picture) may be one that may be coded and decoded using intra prediction or inter prediction using at most two motion vectors and reference indices to predict the sample values of each block. Similarly, multiple-predictive pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block" and page 8, paragraph #0092, "At S114, the decoded EFA features F.sub.b,1, F.sub.b,2, and the up-sampled sequence X=x.sub.1, x.sub.2, are aggregated together by a Fusion module 139 to generate the final reconstructed video sequence X={circumflex over (x)}.sub.1, {circumflex over (x)}.sub.2, data 140. … "); and generate a bitstream corresponding to an encoded version of the image frame that is at least partially based on the virtual reference frame (see page 8, paragraph #0092, "At S114, the decoded EFA features F.sub.b,1, F.sub.b,2, and the up-sampled sequence X=x.sub.1, x.sub.2, are aggregated together by a Fusion module 139 to generate the final reconstructed video sequence X={circumflex over (x)}.sub.1, {circumflex over (x)}.sub.2, data 140. …").
However, JIANAG et al. does not specifically discloses that the motion-based data indicates a global motion of the image frame, a change in position of an image capture device that captured the image frame, or both.
LU teaches in the same field of coding/decoding video signal using global motion vector of the image (see pages 3-4 of the translation of CN 101064849 A, “if there is other frame in one of pixel blocks and the current pixel block is the same or similar, then it is called the best matching block of the current pixel block. movement of the image is small, the same pixel block position in the preceding frame as the best matching block. movement of the image, the position or P of motion vector for motion estimation in frame and B frame to find the best matching block, such as using the global motion vector of the image. when one moving object, motion vector recording the displacement of it, when the next frame coding, it can find the optimal matching block using the motion vector. I frame has no motion vector, displacement with respect to the previous frame can be composed of P-frame or B-frame motion vectors. such as I1 and I5 in FIG. 6, is an intra-prediction encoded image, P2, P3 and P4 is a forward prediction encoded image, the pixel block of I5 602 an optimal matching block in I1 can use the preceding P frame motion vector of the corresponding pixel block and 603 + 604 + 605 + 606, wherein I5 relative displacement vector of the P4 603 is equal to the motion vector 604. FIG. 7 is another example of I1 and I4 is an intra-prediction encoded image, P2 and P3 are forward prediction encoded image, 703 is I4 relative to the displacement vector of the P3, making 703 equal to P3 relative to the global motion vector of the P2 pixel block of the I4 702 using 703 for searching for an optimal matching block, points 701, 701 702 as the best matching block. In a word, the one pixel block and the relative displacement of the optimal matching block can be the frame according to their display order (display order), using a motion vector or a motion vector extension (extrapolate)”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the encoder/decoder as taught by LU into JIANG et al.’s system in order to effectively reduce the amount of calculation, save resource and meet the requirement of real-time encoding.
Regarding claim 16, JIANG et al. discloses wherein the bitstream includes the synthesis support data (see page 5, paragraph #0066, "A Bi-directionally Predictive Picture (B Picture) may be one that may be coded and decoded using intra prediction or inter prediction using at most two motion vectors and reference indices to predict the sample values of each block. Similarly, multiple-predictive pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block" and page 8, paragraph #0092, "At S114, the decoded EFA features F.sub.b,1, F.sub.b,2, and the up-sampled sequence X=x.sub.1, x.sub.2, are aggregated together by a Fusion module 139 to generate the final reconstructed video sequence X={circumflex over (x)}.sub.1, {circumflex over (x)}.sub.2, data 140. … ").
Regarding claim 17, JIANG et al. discloses wherein the one or more processors are configured to generate a first set of reference candidates that includes the virtual reference frame (see page 8, paragraph #0092, "At S114, the decoded EFA features F.sub.b,1, F.sub.b,2, and the up-sampled sequence X=x.sub.1, x.sub.2, are aggregated together by a Fusion module 139 to generate the final reconstructed video sequence X={circumflex over (x)}.sub.1, {circumflex over (x)}.sub.2, "") data 140. The Fusion module can be a small DNN, where for generating {circumflex over (x)}.sub.i at time stamp i, the Fusion module can use only L.sub.i, B.sub.i, and x.sub.i from the same time stamp, or use L.sub.i-n, L.sub.i+m, B.sub.i-n, ,B.sub.i+m, and use x.sub.i-n, x.sub.i+1 from a few neighbouring time stamps. …").
Regarding claim 18, JIANG et al. discloses wherein the bitstream indicates the first set of reference candidates (see page 8, paragraph #0092, "At S114, the decoded EFA features F.sub.b,1, F.sub.b,2, and the up-sampled sequence X=x.sub.1, x.sub.2, are aggregated together by a Fusion module 139 to generate the final reconstructed video sequence X={circumflex over (x)}.sub.1, {circumflex over (x)}.sub.2, data 140. The Fusion module can be a small DNN, where for generating {circumflex over (x)}.sub.i at time stamp i, the Fusion module can use only L.sub.i, B.sub.i, and x.sub.i from the same time stamp, or use L.sub.i-n, L.sub.i+m, B.sub.i-n, B.sub.i+m, and use x.sub.i-n,, x.sub.i+1 from a few neighbouring time stamps …").
Regarding claim 19, JIANG et al. discloses wherein the one or more processors are configured to generate one or more additional first sets of reference candidates that include one or more additional virtual reference frames associated with one or more additional image frames of the sequence of image frames (see page 8, paragraph #0092, "At S114, the decoded EFA features F.sub.b,1, F.sub.b,2, and the up-sampled sequence X=x.sub.1, x.sub.2, are aggregated together by a Fusion module 139 to generate the final reconstructed video sequence X={circumflex over (x)}.sub.1, {circumflex over (x)}.sub.2, data 140. The Fusion module can be a small DNN, where for generating {circumflex over (x)}.sub.i at time stamp i, the Fusion module can use only L.sub.i, B.sub.i, and x.sub.i from the same time stamp, or use L.sub.i-n, L.sub.i+m, B.sub.i-n, B.sub.i+m, and use x.sub.i-n, x.sub.i+1 from a few neighbouring time stamps …").
Regarding claim 20, JIANG et al. discloses wherein the bitstream further indicates a second set of reference candidates including one or more previously decoded image frames, and wherein the one or more processors are configured to generate the virtual reference frame based at least in part on determining that a count of reference frames in the second set of reference candidates is less than a threshold reference count of a coding configuration (see page 5, paragraph #0066, "A Bi-directionally Predictive Picture (B Picture) may be one that may be coded and decoded using intra prediction or inter prediction using at most two motion vectors and reference indices to predict the sample values of each block. Similarly, multiple-predictive pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block").
Regarding claim 21, JIANG et al. discloses wherein the one or more processors are configured to, based at least in part on detecting a face in the image frame, generate the virtual reference frame (see page 7, paragraph #0088, " According to exemplary embodiments, The Face Detection & Facial Landmark Extraction module 122 can use any face detector to locate face areas in each video frame xi, such as to locate a pre-determined set of facial landmarks for each detected face (e.g., landmarks around left/right eyes, nose, mouse, etc.). …").
Regarding claim 22, JIANG et al. discloses wherein the one or more processors are configured to: obtain the motion-based data associated with the image frame (see page 5, paragraphs #0065-#0066, "A Predictive picture (P picture) may be one that may be coded and decoded using intra prediction or inter prediction using at most one motion vector and reference index to predict the sample values of each block" and "A Bi-directionally Predictive Picture (B Picture) may be one that may be coded and decoded using intra prediction or inter prediction using at most two motion vectors and reference indices to predict the sample values of each block. Similarly, multiple-predictive pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block"); and based at least in part on determining that the motion-based data indicates global motion that is greater than a global motion threshold (see page 5, paragraphs #0065-#0066 of JIANG, "A Predictive picture (P picture) may be one that may be coded and decoded using intra prediction or inter prediction using at most one motion vector and reference index to predict the sample values of each block" and "A Bi-directionally Predictive Picture (B Picture) may be one that may be coded and decoded using intra prediction or inter prediction using at most two motion vectors and reference indices to predict the sample values of each block. Similarly, multiple-predictive pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block" and see pages 3-4 of the translation of CN 101064849 A of LU, “if there is other frame in one of pixel blocks and the current pixel block is the same or similar, then it is called the best matching block of the current pixel block. movement of the image is small, the same pixel block position in the preceding frame as the best matching block. movement of the image, the position or P of motion vector for motion estimation in frame and B frame to find the best matching block, such as using the global motion vector of the image. when one moving object, motion vector recording the displacement of it, when the next frame coding, it can find the optimal matching block using the motion vector. I frame has no motion vector, displacement with respect to the previous frame can be composed of P-frame or B-frame motion vectors. such as I1 and I5 in FIG. 6, is an intra-prediction encoded image, P2, P3 and P4 is a forward prediction encoded image, the pixel block of I5 602 an optimal matching block in I1 can use the preceding P frame motion vector of the corresponding pixel block and 603 + 604 + 605 + 606, wherein I5 relative displacement vector of the P4 603 is equal to the motion vector 604. FIG. 7 is another example of I1 and I4 is an intra-prediction encoded image, P2 and P3 are forward prediction encoded image, 703 is I4 relative to the displacement vector of the P3, making 703 equal to P3 relative to the global motion vector of the P2 pixel block of the I4 702 using 703 for searching for an optimal matching block, points 701, 701 702 as the best matching block. In a word, the one pixel block and the relative displacement of the optimal matching block can be the frame according to their display order (display order), using a motion vector or a motion vector extension (extrapolate)”), generate the virtual reference frame (see page 8, paragraph #0092, "At S114, the decoded EFA features F.sub.b,1, F.sub.b,2, , and the up-sampled sequence X=x.sub.1, x.sub.2, are aggregated together by a Fusion module 139 to generate the final reconstructed video sequence X={circumflex over (x)}.sub.1, {circumflex over (x)}.sub.2, data 140. The Fusion module can be a small DNN, where for generating {circumflex over (x)}.sub.i at time stamp i, the Fusion module can use only L.sub.i, B.sub.i, and x.sub.i from the same time stamp, or use L.sub.i-n, L.sub.i+m, B.sub.i-n, B.sub.i+m, and use x.sub.i-n, x.sub.i+1 from a few neighbouring time stamps. …").
The motivation to combine the references has been discussed in claim 1 above.
Regarding claim 23, JIANG et al. discloses wherein the synthesis support data includes facial landmark data that indicates locations of facial features in the image frame (see page 7, paragraph #0088, " According to exemplary embodiments, The Face Detection & Facial Landmark Extraction module 122 can use any face detector to locate face areas in each video frame xi, such as to locate a pre-determined set of facial landmarks for each detected face (e.g., landmarks around left/right eyes, nose, mouse, etc.). …").
Regarding claim 24, JIANG et al. discloses wherein the motion-based data includes motion sensor data indicating motion of the image capture device (see page 5, paragraphs #0065-#0066, "A Predictive picture (P picture) may be one that may be coded and decoded using intra prediction or inter prediction using at most one motion vector and reference index to predict the sample values of each block" and "A Bi-directionally Predictive Picture (B Picture) may be one that may be coded and decoded using intra prediction or inter prediction using at most two motion vectors and reference indices to predict the sample values of each block. Similarly, multiple-predictive pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block").
Regarding claim 25, JIANG et al. discloses wherein the image capture device includes at least one of an extended reality (XR) device, a vehicle, or a camera (see page 4, paragraph #0052, "The video source 401 may provide the source the video source 401 may be a camera that captures local image information as a video sequence. …").
Regarding claim 26, JIANG et al. discloses wherein the one or more processors are configured to use the motion-based data to warp a previously decoded image frame to generate the virtual reference frame (see page 5, paragraphs #0065-#0066, "A Predictive picture (P picture) may be one that may be coded and decoded using intra prediction or inter prediction using at most one motion vector and reference index to predict the sample values of each block" and "A Bi-directionally Predictive Picture (B Picture) may be one that may be coded and decoded using intra prediction or inter prediction using at most two motion vectors and reference indices to predict the sample values of each block. Similarly, multiple-predictive pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block").
Regarding claim 27, JIANG et al. discloses wherein the bitstream includes a supplemental enhancement information (SEI) message indicating virtual reference frame usage to generate a decoded version of the image frame (see page 3, paragraph #0038, "The video decoder 300 The control information for the rendering device(s) may be in the form of Supplementary Enhancement Information (SEI messages) or Video Usability Information parameter set fragments (not depicted). ").
Regarding claim 28, JIANG et al. discloses wherein the one or more processors are configured to use a model to generate the virtual reference frame, and wherein input to the model includes the synthesis support data and at least one previously decoded image frame (see page 5, paragraph #0066, "A Bi-directionally Predictive Picture (B Picture) may be one that may be coded and decoded using intra prediction or inter prediction using at most two motion vectors and reference indices to predict the sample values of each block. Similarly, multiple-predictive pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block" and page 8, paragraph #0094, "Also, according to exemplary embodiments, there are several components in the proposed framework that needs to be trained, and such training will be described with respect to FIG. 13 which illustrates a workflow 1300 of an exemplary training process according to exemplary embodiments. …").
Claim 29 is rejected for the same reasons as discussed in claim 11 above.
The method claim 30 is rejected for the same reason as discussed in the corresponding apparatus claim 14 above.
Regarding claim 31, JIANG et al. discloses that the image capture device configured to capture the image frame (see page 4, paragraph #0052, "The video source 401 may provide the source the video source 401 may be a camera that captures local image information as a video sequence. …").
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
The cited reference related to encoding/decoding video signal using global motion of image frame.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to THAI Q TRAN whose telephone number is (571)272-7382. The examiner can normally be reached Monday to Friday from 10:00am to 6:30pm..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Colleen Fauz can be reached at (571)272-1667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/THAI Q TRAN/ Supervisory Patent Examiner, Art Unit 2484
Prosecution Insights