DETAILED ACTION
This Office action is in response to the Application filed on September 4, 2024, which is a continuation application of International Application No. PCT/JP2022/014239, filed on March 25, 2022. An action on the merits follows. Claims 1-20 are pending on the application.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 5-6 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 5 recites the limitation “an area determined to be a non-effective area… reconstructed image data in which the area is set as an invalid image” in lines 2-4 of the claim. However, it is not clear if the claimed “an area” recited in line 2 of claim 5 encompass embodiments corresponding to any of the claimed “target area” and “non-target area” previously recited in lines 3-4 of claim 1, or if the claimed “an area” recited in line 2 of claim 5 encompass embodiments corresponding another “area” different from any of the claimed “target area” and “non-target area” previously recited in lines 3-4 of claim 1, for example. Additionally, it is not clear if the claimed “the area” recited in line 4 of claim 5 encompass embodiments corresponding to the claimed “an area” recited in line 2 of claim 5, or if the claimed “the area” recited in line 4 of claim 5 encompass embodiments corresponding to any of the claimed “target area” and “non-target area” previously recited in lines 3-4 of claim 1, for example. Therefore, the metes and bounds of the claim are not clearly set forth and the examiner cannot clearly determine which elements are encompassed by the claim language, which renders the claim indefinite.
Claim 6 recites the limitation “an area determined to be a non-effective area… the quantization value of the area becomes maximum” in lines 2-5 of the claim. However, it is not clear if the claimed “an area” recited in line 2 of claim 6 encompass embodiments corresponding to any of the claimed “target area” and “non-target area” previously recited in lines 3-4 of claim 1, or if the claimed “an area” recited in line 2 of claim 6 encompass embodiments corresponding another “area” different from any of the claimed “target area” and “non-target area” previously recited in lines 3-4 of claim 1, for example. Additionally, it is not clear if the claimed “the area” recited in line 4 of claim 6 encompass embodiments corresponding to the claimed “an area” recited in line 2 of claim 6, or if the claimed “the area” recited in line 4 of claim 6 encompass embodiments corresponding to any of the claimed “target area” and “non-target area” previously recited in lines 3-4 of claim 1, for example. Therefore, the metes and bounds of the claim are not clearly set forth and the examiner cannot clearly determine which elements are encompassed by the claim language, which renders the claim indefinite.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 9-12, 15, and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Malakhov et al. (US PG Publication No. 2021/0168408 A1), hereafter referred to as Malakhov, in view of HATTORI et al. (US PG Publication No. 2016/0234523 A1), hereafter referred to as HATTORI.
Regarding claim 1, Malakhov discloses an image processing system (Par. [0007]: an apparatus is provided for encoding samples of a video image comprising) comprising:
a hierarchical encoder (Par. [0007-10]: an apparatus is provided for encoding samples of a video image comprising a processing circuitry configured to… divide the video image into equally sized coding tree units (CTUs), hierarchically split one of the coding tree units into a plurality of coding units; Par. [0029-30]: a method is provided for encoding samples of a video image comprising… dividing the video image into equally sized coding tree units, hierarchically splitting one of the coding tree units into a plurality of coding units) that determines, based on a result of recognition processing, a target area needed to recognize a recognition target and a non-target area other than the target area in image data (Par. [0005-18]: detect whether interesting objects… even exist in the video image apart from the image background… processing circuitry of the apparatus is further configured to divide the video image into equally sized coding tree units (CTUs), hierarchically split one of the coding tree units into a plurality of coding units, determine motion vector for each of the plurality of the coding units, input to the machine-learning-based model the coding tree unit and the respective motion vectors determined for the plurality of the coding units of the coding tree unit… the machine-learning-based model is configured to classify the input sample as belonging to a region of interest (ROI) or to a region of non-interest (RONI)… obtain as the output of the machine-learning-based model a map specifying the ROI and RONI of the video image … the machine-learning-based model is configured to detect, using the machine-learning-based model, an object within the video image; Par. [0029-35]: method is provided for encoding samples of a video image comprising… dividing the video image into equally sized coding tree units, hierarchically splitting one of the coding tree units into a plurality of coding units… classifying by the machine-learning-based model the input sample as belonging to a ROI or to a RONI… obtaining as the output of the machine-learning-based model a map specifying the ROI and RONI of the video image… detecting, using the machine-learning-based model, an object within the video image; Par. [0155-161]: detection and tracking (and often also identification) of objects... in a video image, respectively, in its frames apart from the usually unimportant image background. In general, before a more detailed object identification is performed, the video, respectively, its frames are segmented first into a foreground image and background image, wherein the foreground image contains in most cases image details to be processed further… a region in the video image may be identified as relevant, corresponding to an interesting region. Those regions may be also referred to as ROI… In turn, regions in the video image, which are not of interest or of less interest, may be referred to as RONI… perform foreground-background segmentation of the video image. The segmentation result of the machine-learning model 820 is output and provided as input to the module 830 to perform the coding parameter adaptation… the image segmentation by machine-learning model is based on classification of the sample or the group of samples belonging to a ROI or a RONI; Par. [0177-200]: using a video encoder (e.g. H.264/AVC or HEVC (H.265)… divide the video image into equally-sized coding-tree units CTUs 1010. The CTU is further hierarchically split into a plurality of coding units CUs. As mentioned, in H.264/AVC or HEVC (H.265), these CTUs/CUs are the basic block units, which are referred to as blocks… the machine-learning-based model 820 or 910 uses this data as input and performs classification of the input sample as to belonging to a region of interest ROI or a region of non-interest RONI. This ROI-RONI classification of a sample of a block results in an image segmentation of the video image, respectively, of its frames… obtain as the output of the machine-learning-based model a map, which specifies the ROI and RONI of the video image. The map includes for each sample or block of samples an indication whether the said sample or block of samples belongs to the ROI or RONI… the machine-learning-based model is configured to detect an object within the video image… The output of the machine-learning model includes the detection result whether or not an object or multiple objects is/are in the video image. This output is also used by the coding parameter adaptation modules to adapt a coding parameter in accordance with the presence of an object; Par. [0236-237]: map is provided as output by the machine-learning model 1310… As described before, the ROI-RONI map may be a bitmap, wherein it is indicated bit-wise whether a sample or a block of samples belongs to ROI or RONI… In case of standard HEVC codecs, this indication is done on a CTU/CU basis for compatibility reasons. When a bitmap is used, its entries contain for each CTU of a video frame either a value of “1” or “0” if it is an ROI or RONI; that determines, based on a result of recognition processing, a target area needed to recognize a recognition target and a non-target area other than the target area in image data (e.g. apparatus for encoding samples of a video image (i.e. an image processing system) includes processing circuitry configured to divide (i.e. split, partition, segment, separate, etc.) the video image into coding tree units (CTUs) and hierarchically split the coding tree units into a plurality of coding units (i.e. comprising a hierarchical encoder), for example, including machine-learning-based models configured to detect (i.e. recognize, identify, distinguish, etc.) an interesting object (i.e. a recognition target) within the video image, for example, including a machine-learning-based model that is configured to classify input encoding samples of the video image as belonging to a region (i.e. area, section, portion, etc.) of interest (i.e. a hierarchical encoder that determines, based on a result of recognition processing, a target area needed to recognize a recognition target including a region of interest (ROI), such as foreground or object regions, for example), or belonging to a region of non-interest (i.e. and a non-target area other than the target area in image data including a region of non-interest (RONI), such as a less relevant background region, for example), as indicated above), for example),
a quantization value of the target area needed to recognize the recognition target, and a quantization value of the non-target area (Par. [0005]: coding parameters, for example… quantization parameter (QP), used for video image encoding are often preset and signaled to the encoder… a key aspect is to detect whether interesting objects… even exist in the video image apart from the image background, whether these objects move and how fast. Dependent on these factors, the coding parameters may be adapted; Par. [0023-27]: determine the coding parameter by adapting spatial resolution based on whether or not an object was detected in the video image… the coding parameter related to the spatial resolution is QP… the processing circuitry is further configured to set the quantization parameter for a sample lower in a case when the sample belongs to ROI than in a case when the sample belongs to RONI; Par. [0039-41]: method includes the step of determining the coding parameter by adapting spatial resolution based on whether or not an object was detected in the video image… method steps include setting the quantization parameter for a sample lower in a case when the sample belongs to ROI than in a case when the sample belongs to RONI; Par. [0101]: quantize the transformed coefficients 107 to obtain quantized coefficients 109, e.g. by applying scalar quantization or vector quantization. The quantized coefficients 109 may also be referred to as quantized residual coefficients 109. For example, for scalar quantization, different scaling may be applied to achieve finer or coarser quantization. Smaller quantization step sizes correspond to finer quantization, whereas larger quantization step sizes correspond to coarser quantization. The applicable quantization step size may be indicated by a QP; Par. [0161]: segmentation result of the machine-learning model 820 is output and provided as input to the module 830 to perform the coding parameter adaptation combined with the motion information… the image segmentation by machine-learning model is based on classification of the sample or the group of samples belonging to a ROI or a RONI; Par. [0235-236]: the quantization parameter for a sample is set lower when the sample belongs to the ROI than in case the sample belongs to RONI… the adaption (i.e. the setting) of the quantization parameter is performed by the QP map generator 1360, which uses the importance map (i.e. the ROI-RONI map) as input. This map is provided as output by the machine-learning model 1310. The QP map generator uses this importance map to generate a corresponding map of the quantization parameters; a quantization value of the target area needed to recognize the recognition target, and a quantization value of the non-target area (e.g. apparatus for encoding samples of a video image includes processing circuitry configured to adapt coding parameters, including a quantization parameter (i.e. a quantization value) QP, for example, based on a result of recognition processing that classifies input encoding samples of the video image as belonging to a region (i.e. area, section, portion, etc.) of interest including a region of interest ROI (i.e. the target area needed to recognize the recognition target), such as foreground or object regions, for example, or belonging to a region of non-interest including a region of non-interest RONI (i.e. the non-target area), such as a less relevant background region, as indicated above, for example, by setting the quantization parameter for a sample lower in a case when the sample belongs to a ROI (i.e. a quantization value of the target area needed to recognize the recognition target), than in a case when the sample belongs to a RONI (i.e. and a quantization value of the non-target area), as indicated above), for example),
encodes an entire area of the image data with the quantization value of the target area to generate first encoded data, and encodes the entire area of the image data with the quantization value of the non-target area to generate second encoded data (Par. [0023-29]: determine the coding parameter by adapting spatial resolution based on whether or not an object was detected in the video image… the coding parameter related to the spatial resolution is QP… the processing circuitry is further configured to set the quantization parameter for a sample lower in a case when the sample belongs to ROI than in a case when the sample belongs to RONI… method is provided for encoding samples of a video image comprising the steps of determining motion information for samples of a block in the video image, determining a coding parameter based on an output of a machine-learning-based model, to which the motion information and the samples of the block are input, and encoding the video frame by applying the coding parameter; Par. [0039-41]: method includes the step of determining the coding parameter by adapting spatial resolution based on whether or not an object was detected in the video image… method steps include setting the quantization parameter for a sample lower in a case when the sample belongs to ROI than in a case when the sample belongs to RONI; Par. [0159-167]: a machine-learning-based model, which is input with motion information and frames of a video image, for adapting one or multiple coding parameters used for encoding the video image… an apparatus is provided wherein a video image, including a plurality of video frames, is encoded using coding parameters in order to provide a video bitstream as the output… segmentation result of the machine-learning model 820 is output and provided as input to the module 830 to perform the coding parameter adaptation combined with the motion information… the image segmentation by machine-learning model is based on classification of the sample or the group of samples belonging to a ROI or a RONI… the machine-learning model may be trained to directly deliver the coding parameters, which are then input to the encoder 830. The coding parameters may be determined for one sample or a group of samples of the video image. The coding parameters of different single samples or different groups of samples may thus be different… The sample is encoded by applying the determined coding parameters. The encoding of a sample or the samples is performed by an encoder 430 and provides the encoded video image as a bitstream video; Par. [0231-242]: adapted coding parameter picture size is signaled by the module 1350 to the video encoder 1320… the coding parameter related to spatial resolution is a QP… The quantization parameter refers to the inverse number of bits used for encoding. Thus, a small QP value corresponds to a large number of bits, respectively, a large number of smaller increments. As a result, finer spatial variations within the video image may be encoded, improving image quality… the quantization parameter for a sample is set lower when the sample belongs to the ROI than in case the sample belongs to RONI… the adaption (i.e. the setting) of the quantization parameter is performed by the QP map generator 1360, which uses the importance map (i.e. the ROI-RONI map) as input. This map is provided as output by the machine-learning model 1310. The QP map generator uses this importance map to generate a corresponding map of the quantization parameters… method is provided for encoding samples of a video image comprising the steps of determining motion information for samples of a block in the video image, determining a coding parameter based on an output of a machine-learning-based model, to which the motion information and the samples of the block are input, and encoding the video frame by applying the coding parameter; encodes an entire area of the image data with the quantization value of the target area to generate first encoded data, and encodes the entire area of the image data with the quantization value of the non-target area to generate second encoded data (e.g. apparatus for encoding samples of a video image includes processing circuitry configured to adapt coding parameters, including a quantization parameter (i.e. the quantization value) QP, for example, based on a result of recognition processing that classifies input encoding samples of the video image as belonging to a region (i.e. area, section, portion, etc.) of interest including a region of interest ROI (i.e. the target area), for example, or belonging to a region of non-interest including a region of non-interest RONI (i.e. the non-target area), for example, by setting the quantization parameter for a sample lower in a case when the sample belongs to a ROI (i.e. the quantization value of the target area) than in a case when the sample belongs to a RONI (i.e. the quantization value of the non-target area), as indicated above, for example, and encodes each video frame of the video image (i.e. an entire area of the image data) by applying corresponding coding parameters to the ROI region to generate first encoded data and to the RONI region to generate second encoded data, respectively (i.e. encodes an entire area of the image data with the quantization value of the target area to generate first encoded data, and encodes the entire area of the image data with the quantization value of the non-target area to generate second encoded data), as indicated above), for example); and
a transcoder that generates reconstructed image data by using the target area in first decoded data obtained by decoding the first encoded data and the non-target area in second decoded data obtained by decoding the second encoded data, and re-encodes the reconstructed image data to generate re-encoded data (Par. [0064-66]: Video coding typically refers to the processing of a sequence of pictures, which form the video or video sequence… Video coding comprises two parts, video encoding and video decoding. Video encoding is performed at the source side… Video decoding is performed at the destination side and typically comprises the inverse processing compared to the encoder to reconstruct the video pictures. Embodiments referring to “coding” of video pictures (or pictures in general, as will be explained later) shall be understood to relate to both, “encoding” and “decoding” of video pictures. The combination of the encoding part and the decoding part is also referred to as CODEC (COding and DECoding)… Each picture of a video sequence is typically partitioned into a set of non-overlapping blocks and the coding is typically performed on a block level. That is, at the encoder the video is typically processed, i.e. encoded, on a block (video block) level, e.g. using spatial (intra picture) prediction and temporal (inter picture) prediction to generate a prediction block, subtracting the prediction block from the current block (block currently processed/to be processed) to obtain a residual block, transforming the residual block and quantizing the residual block in the transform domain to reduce the amount of data to be transmitted (compression), whereas at the decoder the inverse processing compared to the encoder is applied to the encoded or compressed block to reconstruct the current block for representation… the encoder duplicates the decoder processing loop such that both will generate identical predictions (e.g. intra- and inter predictions) and/or re-constructions for processing, i.e. coding, the subsequent blocks; Par. [0077-84]: receive the encoded picture data 171 and to directly transmit it to another device, e.g. the destination device 320 or any other device, for storage or direct reconstruction, or to process the encoded picture data 171 for respectively before storing the encoded data 330 and/or transmitting the encoded data 330 to another device, e.g. the destination device 320 or any other device for decoding or storing… receive the encoded picture data 171 and provide decoded picture data 231 or a decoded picture 231; Par. [0091-93]: FIG. 1 shows a schematic/conceptual block diagram of an embodiment of an encoder 100, e.g. a picture encoder 100, which comprises an input 102, a residual calculation unit 104, a transformation unit 106, a quantization unit 108, an inverse quantization unit 110, and inverse transformation unit 112, a reconstruction unit 114, a buffer 118, a loop filter 120, a decoded picture buffer (DPB) 130… the backward signal path of the encoder corresponds to the signal path of the decoder (see decoder 200 in FIG. 2)… encoder is configured to receive, e.g. by input 102, a picture 101 or a picture block 103 of the picture 101, e.g. picture of a sequence of pictures forming a video or video sequence. The picture block 103 may also be referred to as current picture block or picture block to be coded, and the picture 101 as current picture or picture to be coded (in particular in video coding to distinguish the current picture from other pictures, e.g. previously encoded and/or decoded pictures of the same video sequence, i.e. the video sequence which also comprises the current picture); Par. [0105-114]: combine the inverse transformed block 113 and the prediction block 165 to obtain a reconstructed block 115 in the sample domain… The DPB 130 is configured to receive and store the filtered block 121. The decoded picture buffer 130 may be further configured to store other previously filtered blocks, e.g. previously reconstructed and filtered blocks 121, of the same current picture or of different pictures, e.g. previously reconstructed pictures, and may provide complete previously reconstructed, i.e. decoded, pictures (and corresponding reference blocks and samples) and/or a partially reconstructed current picture (and corresponding reference blocks and samples), for example for inter estimation and/or inter prediction… receive or obtain the picture block 103 (current picture block 103 of the current picture 101) and decoded or at least reconstructed picture data, e.g. reference samples of the same (current) picture from buffer 116 and/or decoded picture data 231 from one or a plurality of previously decoded pictures from decoded picture buffer 130; a transcoder that generates reconstructed image data by using the target area in first decoded data obtained by decoding the first encoded data and the non-target area in second decoded data obtained by decoding the second encoded data, and re-encodes the reconstructed image data to generate re-encoded data (e.g. apparatus for encoding samples of a video image includes processing circuitry configured to adapt coding parameters, including a quantization parameter QP, for example, based on a result of recognition processing that classifies input encoding samples of the video image as belonging to a region (i.e. area, section, portion, etc.) of interest including a region of interest ROI (i.e. the target area), for example, or belonging to a region of non-interest including a region of non-interest RONI (i.e. the non-target area), for example, and encodes each video frame of the video image by applying corresponding coding parameters to the ROI region (i.e. the target area used to generate first encoded data) and to the RONI region, to generate second encoded data (i.e. the non-target area used to generate second encoded data), respectively, as indicated above, for example, including video coding that comprises two parts, video encoding and video decoding (i.e. a transcoder, CODEC, etc.), in which video encoding is performed at a source side and video decoding is performed at a destination side and typically comprises the inverse processing compared to the encoder to reconstruct the video pictures (i.e. a transcoder that generates reconstructed image), for example, including decoded pictures and corresponding reference blocks and samples and/or a partially reconstructed current picture and corresponding reference blocks and samples (i.e. a transcoder that generates reconstructed image data by using the target area in first decoded data obtained by decoding the first encoded data and the non-target area in second decoded data obtained by decoding the second encoded data) used for inter estimation and/or inter prediction, for example, and the encoder duplicates decoder processing loop such that both will generate identical predictions (i.e. intra- and inter predictions) and/or re-constructions for processing (i.e. coding, re-encode, etc.) subsequent blocks (i.e. and re-encodes the reconstructed image data to generate re-encoded data), as indicated above), for example).
Malakhov teachings above disclose the apparatus for encoding samples of a video image, detecting an interesting object within the video image, in which “an interesting object” corresponds to the claimed “a recognition target”, by classifying input encoding samples of the video image as belonging to a region of interest ROI, in which a ROI corresponds to the claimed “target area”, such as foreground or object regions, and a region of non-interest RONI, in which a RONI corresponds to the claimed “non-target area”, such as a less relevant background region, for example, but do not expressly disclose “target”. Additionally, Malakhov teachings above disclose video coding that comprises two parts, video encoding and video decoding, in which video encoding and video decoding corresponds to the claimed “transcoder” process, including a CODEC, for example, but do not expressly disclose “a transcoder”.
However, HATTORI teaches “target” and “transcoder” (Par. [0045-60]: Referring to FIG. 1, the video encoding device 1 generates an entire region bitstream by performing an encoding process, in a form that enables the video transcoding device 2 in the next stage to process the entire region bitstream, on the picture of an entire region (a region including an entire frame) of an inputted video, and multiplexes hint information used for transcoding (the details of the hint information will be described below) into the entire region bitstream and outputs the entire region bitstream after multiplexing of the hint information to the video transcoding device 2… video encoding device 1 performs a process of determining coding parameters for a coding target block in a picture belonging to a GOP (Group of pictures) and generating a prediction image by using the coding parameter, and also compression-encoding a difference image between the coding target block and the prediction image and multiplexing encoded data which is the result of the encoding and the coding parameters to generate an entire region bitstream… video transcoding device 2 is configured with an entire region stream decoder 3 and a partial region transcoder 4, and performs a process of decoding the image of the entire region from the entire region bitstream generated by the video encoding device 1, and outputting the image of the entire region (referred to as the “entire region decoded image” from here on) to an entire region display device 5… The entire region stream decoder 3 performs a process of extracting the encoded data and the coding parameters of the entire region, and the hint information which are included in the entire region bitstream generated by the video encoding device 1, and decoding the encoded data and the coding parameters of the entire region into an entire region decoded image and outputting the entire region decoded image to the partial region transcoder 4 and the entire region display device 5, and also outputting the encoded data and the coding parameters of the entire region, and the hint information to the partial region transcoder 4… The partial region transcoder 4 performs a process of referring to the motion vector limitation information, the GOP size limitation information and the reference configuration specification information which are included in the hint information outputted from the entire region stream decoder 3, to specify an indispensable encoded region which is a region required at the time of decoding the display area of a picture… An indispensable encoded region determinator is configured with the entire region stream decoder 3 and the partial region transcoder 4… The partial region transcoder 4 also performs a process of extracting the encoded data and the coding parameters of a coding target block which is included in the above-mentioned indispensable encoded region from among the encoded data and the coding parameters of the entire region which are outputted from the entire region stream decoder 3, and generating a partial region bitstream in conformity with the encoding codec set in advance from the encoded data and the coding parameters of the coding target block…The partial region transcoder 4 constructs a parameter extractor and a partial region stream generator… The entire region display device 5 is display equipment to display the entire region decoded image outputted from the entire region stream decoder 3…The video decoding device 6 decodes an image of a partial region from the partial region bitstream outputted from the partial region transcoder 4, and outputs the image of the partial region (referred to as the “partial region decoded image” from here on) to a partial region display device 7; Par. [0107]: transcode controller 41 performs a process of referring to the motion vector limitation information, the GOP size limitation information and the reference configuration specification information which are included in the hint information outputted from the entire region stream decoder 3, to specify a region which is a target for transcoding (a target region to be transcoded) from the display area of a picture, the display area being indicated by the display area information provided therefor from the outside thereof, and also specify an indispensable encoded region which is a region required at the time of decoding the target region to be transcoded (a region to which the coding parameters need to be applied at the time of transcoding), and outputting target region to be transcoded information indicating the target region to be transcoded and indispensable encoded region information indicating the indispensable encoded region; “target” and “transcoder” (e.g. video encoding device performs a process of determining coding parameters for a coding target (“target”) block in a picture belonging to a GOP (Group of pictures) and generating a prediction image by using the coding parameter, and also compression-encoding a difference image between the coding target block and the prediction image and multiplexing encoded data which is the result of the encoding and the coding parameters to generate an entire region bitstream, for example, including video transcoding device, or transcoder (“transcoder”), for example, which performs a process of extracting encoded data and coding parameters of a coding target block which is included in an indispensable encoded region from among encoded data and the coding parameters of an entire region which are outputted from the entire region stream decoder, and generating a partial region bitstream in conformity with the encoding codec set in advance from the encoded data and the coding parameters of the coding target block, for example, by specify a region which is a target (“target”) for transcoding (a target region to be transcoded) from the display area of a picture, and also by specifying the indispensable encoded region which is a region required at the time of decoding the target region to be transcoded (a region to which the coding parameters need to be applied at the time of transcoding), and outputting target region to be transcoded information indicating the target region to be transcoded and indispensable encoded region information indicating the indispensable encoded region, as indicated above), for example).
Malakhov and HATTORI are considered to be analogous art because they pertain to image processing techniques related to video coding applications. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify the apparatus for encoding samples of a video image including machine-learning-based models configured to detect an interesting object within the video image and classify input encoding samples of the video image as belonging to a region of interest, or belonging to a region of non-interest, encode each video frame of the video image by applying corresponding coding parameters, including video coding that comprises two parts, video encoding and video decoding, in which video encoding is performed at a source side and video decoding is performed at a destination side and typically comprises the inverse processing compared to the encoder to reconstruct the video pictures (as disclosed by Malakhov) with “target” and “transcoder” (as taught by HATTORI, Abstract, Par. [0045-60, 107]) to generate an arbitrary partial region bitstream from an entire region bitstream, to generate a partial region bitstream including encoded data and coding parameters of the partial region, and to improve the efficiency of generation of partial region bitstream (HATTORI, Abstract, Par. [005-10, 13-18, 20, 45-60, 107]).
Regarding claim 2, claim 1 is incorporated and the combination of Malakhov and HATTORI, as a whole, teaches the system (Malakhov, Par. [0007]), wherein
the hierarchical encoder separates the image data into first image data in which the non-target area is set as an invalid image and second image data in which the target area is set as an invalid image, encodes an entire area of the first image data with the quantization value of the target area, and encodes an entire area of the second image data with the quantization value of the non-target area (Malakhov, Par. [0007-10]: an apparatus is provided for encoding samples of a video image comprising a processing circuitry configured to… divide the video image into equally sized coding tree units (CTUs), hierarchically split one of the coding tree units into a plurality of coding units; Par. [0023-35]: determine the coding parameter by adapting spatial resolution based on whether or not an object was detected in the video image… the coding parameter related to the spatial resolution is QP… the processing circuitry is further configured to set the quantization parameter for a sample lower in a case when the sample belongs to ROI than in a case when the sample belongs to RONI… method is provided for encoding samples of a video image comprising… dividing the video image into equally sized coding tree units, hierarchically splitting one of the coding tree units into a plurality of coding units… classifying by the machine-learning-based model the input sample as belonging to a ROI or to a RONI… obtaining as the output of the machine-learning-based model a map specifying the ROI and RONI of the video image… detecting, using the machine-learning-based model, an object within the video image; Par. [0039-41]: method includes the step of determining the coding parameter by adapting spatial resolution based on whether or not an object was detected in the video image… method steps include setting the quantization parameter for a sample lower in a case when the sample belongs to ROI than in a case when the sample belongs to RONI; Par. [0155-167]: detection and tracking (and often also identification) of objects, including persons, motor vehicle or non-motor vehicles or the like, in a video image, respectively, in its frames apart from the usually unimportant image background. In general, before a more detailed object identification is performed, the video, respectively, its frames are segmented first into a foreground image and background image, wherein the foreground image contains in most cases image details to be processed further. Commonly, the segmentation is performed through classification of a pixel or a group of pixels of the video image, respectively, of a frame whether the pixel or the pixel group belongs to the foreground or background based on certain classification criteria. These criteria are specified beforehand… a region in the video image may be identified as relevant, corresponding to an interesting region. Those regions may be also referred to as ROI… In turn, regions in the video image, which are not of interest or of less interest, may be referred to as RONI… perform foreground-background segmentation of the video image. The segmentation result of the machine-learning model 820 is output and provided as input to the module 830 to perform the coding parameter adaptation… the image segmentation by machine-learning model is based on classification of the sample or the group of samples belonging to a ROI or a RONI… the machine-learning model may be trained to directly deliver the coding parameters, which are then input to the encoder 830. The coding parameters may be determined for one sample or a group of samples of the video image. The coding parameters of different single samples or different groups of samples may thus be different… The sample is encoded by applying the determined coding parameters. The encoding of a sample or the samples is performed by an encoder 430 and provides the encoded video image as a bitstream video; Par. [0198-199]: machine-learning model may classify the entire CTUs or CUs as a ROI or RONI rather than each sample separately. That is, the map may include an indication not for each sample of a video frame, but rather for each block (CU) of the video frame. The indication may be one or more bits. A one bit indication may take values “1” or “0”, respectively for RONI or ROI… the simultaneous use of texture and motion information improves the ROI-RONI classification of the image samples (pixels). As a result, the foreground-background image segmentation becomes more accurate and reliable; Par. [0231-242]: adapted coding parameter picture size is signaled by the module 1350 to the video encoder 1320… the coding parameter related to spatial resolution is a QP… The quantization parameter refers to the inverse number of bits used for encoding. Thus, a small QP value corresponds to a large number of bits, respectively, a large number of smaller increments. As a result, finer spatial variations within the video image may be encoded, improving image quality… the quantization parameter for a sample is set lower when the sample belongs to the ROI than in case the sample belongs to RONI… the adaption (i.e. the setting) of the quantization parameter is performed by the QP map generator 1360, which uses the importance map (i.e. the ROI-RONI map) as input. This map is provided as output by the machine-learning model 1310. The QP map generator uses this importance map to generate a corresponding map of the quantization parameters… method is provided for encoding samples of a video image comprising the steps of determining motion information for samples of a block in the video image, determining a coding parameter based on an output of a machine-learning-based model, to which the motion information and the samples of the block are input, and encoding the video frame by applying the coding parameter; wherein the hierarchical encoder separates the image data into first image data in which the non-target area is set as an invalid image and second image data in which the target area is set as an invalid image, encodes an entire area of the first image data with the quantization value of the target area, and encodes an entire area of the second image data with the quantization value of the non-target area (e.g. apparatus for encoding samples of a video image includes processing circuitry configured to divide (i.e. split, partition, segment, separate, etc.) the video image into coding tree units (CTUs) and hierarchically split the coding tree units into a plurality of coding units CU (i.e. wherein the hierarchical encoder separates the image data), for example, including machine-learning-based models configured to detect (i.e. recognize, identify, distinguish, etc.) an interesting object (i.e. a recognition target) within the video image, for example, including a machine-learning-based model that is configured to classify input encoding samples of the video image as belonging to a region (i.e. area, section, portion, etc.) of interest (i.e. hierarchical encoder separates the image data into second image data in which a target area is set including a region of interest ROI), such as foreground or object regions, for example, or belonging to a region of non-interest (i.e. hierarchical encoder separates the image data into first image data in which a non-target area is set including a region of non-interest RONI), such as foreground or object regions, for example) including a region of non-interest RONI), such as a less relevant background region, for example, in which segmentation is performed through classification of a pixel or a group of pixels of the video image, respectively, of a frame whether the pixel or the pixel group belongs to the foreground or background based on certain classification criteria (i.e. a predetermined pixel value condition), for example, including a map that includes an indication for each block CU of the video frame that includes a one bit indication with values “1” or “0”, respectively, for RONI or ROI regions (i.e. wherein the hierarchical encoder separates the image data into first image data in which the non-target area is set as an invalid image and second image data in which the target area is set as an invalid image), for example, and encodes each video frame of the video image (i.e. an entire area of the image data) by applying corresponding coding parameters to the ROI region to generate first encoded data and to the RONI region to generate second encoded data (i.e. encodes an entire area of the first image data with the quantization value of the target area, and encodes an entire area of the second image data with the quantization value of the non-target area), respectively, as indicated above), for example).
Regarding claim 3, claim 1 is incorporated and the combination of Malakhov and HATTORI, as a whole, teaches the system (Malakhov, Par. [0007]), wherein
the transcoder (HATTORI, Par. [0107]: transcode controller 41 performs a process) generates a quantization value map based on the quantization value of the target area and the quantization value of the non-target area, and re-encodes the reconstructed image data by using the generated quantization value map to generate the re-encoded data (Malakhov, Par. [0172]: processing circuitry of the apparatus 900 (marked by the dashed box) is configured to perform the tasks of the machine-learning based model 910, the analysis of motion by a motion analyzer 930 based on the motion information, and the adaption of the coding parameters including frame rate adaptation 940, picture size adaptation 950, and generation of a QP map by a QP map generator 960 for QP adaptation; Par. [0231-242]: adapted coding parameter picture size is signaled by the module 1350 to the video encoder 1320… the coding parameter related to spatial resolution is a QP… The quantization parameter refers to the inverse number of bits used for encoding. Thus, a small QP value corresponds to a large number of bits, respectively, a large number of smaller increments. As a result, finer spatial variations within the video image may be encoded, improving image quality… the quantization parameter for a sample is set lower when the sample belongs to the ROI than in case the sample belongs to RONI… the adaption (i.e. the setting) of the quantization parameter is performed by the QP map generator 1360, which uses the importance map (i.e. the ROI-RONI map) as input. This map is provided as output by the machine-learning model 1310. The QP map generator uses this importance map to generate a corresponding map of the quantization parameters… method is provided for encoding samples of a video image comprising the steps of determining motion information for samples of a block in the video image, determining a coding parameter based on an output of a machine-learning-based model, to which the motion information and the samples of the block are input, and encoding the video frame by applying the coding parameter; generates a quantization value map based on the quantization value of the target area and the quantization value of the non-target area, and re-encodes the reconstructed image data by using the generated quantization value map to generate the re-encoded data (e.g. apparatus for encoding samples of a video image includes processing circuitry configured to adapt coding parameters, including a quantization parameter QP (i.e. a quantization value), for example, based on a result of recognition processing that classifies input encoding samples of the video image as belonging to a region (i.e. area, section, portion, etc.) of interest including a region of interest ROI (i.e. the target area), for example, or belonging to a region of non-interest including a region of non-interest RONI (i.e. the non-target area), to generate an importance map (i.e. a ROI-RONI map), for example, and the adaption (i.e. adjustment) of the quantization parameter is performed by a QP map (i.e. a quantization value map) generator, which uses the importance map (i.e. the ROI-RONI map) as input (i.e. generates a quantization value map based on the quantization value of the target area and the quantization value of the non-target area), for example, and encodes each video frame of the video image by applying corresponding coding parameters to the ROI region and to the RONI region, respectively (i.e. and re-encodes the reconstructed image data by using the generated quantization value map to generate the re-encoded data), as indicated above, for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 4, claim 3 is incorporated and the combination of Malakhov and HATTORI, as a whole, teaches the system (Malakhov, Par. [0007]), wherein
the transcoder (HATTORI, Par. [0107]: transcode controller 41 performs a process) generates the quantization value map based on:
information that indicates the target area and information that indicates the quantization value of the target area included in the first encoded data or transmitted in association with the first encoded data; and
information that indicates the non-target area and information that indicates the quantization value of the non-target area included in the second encoded data or transmitted in association with the second encoded data (Malakhov, Par. [0172]: processing circuitry of the apparatus 900 (marked by the dashed box) is configured to perform the tasks of the machine-learning based model 910, the analysis of motion by a motion analyzer 930 based on the motion information, and the adaption of the coding parameters including frame rate adaptation 940, picture size adaptation 950, and generation of a QP map by a QP map generator 960 for QP adaptation; Par. [0198-199]: machine-learning model may classify the entire CTUs or CUs as a ROI or RONI rather than each sample separately. That is, the map may include an indication not for each sample of a video frame, but rather for each block (CU) of the video frame. The indication may be one or more bits. A one bit indication may take values “1” or “0”, respectively for RONI or ROI… the simultaneous use of texture and motion information improves the ROI-RONI classification of the image samples (pixels). As a result, the foreground-background image segmentation becomes more accurate and reliable; Par. [0231-242]: adapted coding parameter picture size is signaled by the module 1350 to the video encoder 1320… the coding parameter related to spatial resolution is a QP… The quantization parameter refers to the inverse number of bits used for encoding. Thus, a small QP value corresponds to a large number of bits, respectively, a large number of smaller increments. As a result, finer spatial variations within the video image may be encoded, improving image quality… the quantization parameter for a sample is set lower when the sample belongs to the ROI than in case the sample belongs to RONI… the adaption (i.e. the setting) of the quantization parameter is performed by the QP map generator 1360, which uses the importance map (i.e. the ROI-RONI map) as input. This map is provided as output by the machine-learning model 1310. The QP map generator uses this importance map to generate a corresponding map of the quantization parameters… method is provided for encoding samples of a video image comprising the steps of determining motion information for samples of a block in the video image, determining a coding parameter based on an output of a machine-learning-based model, to which the motion information and the samples of the block are input, and encoding the video frame by applying the coding parameter; generates the quantization value map based on:
information that indicates the target area and information that indicates the quantization value of the target area included in the first encoded data or transmitted in association with the first encoded data; and
information that indicates the non-target area and information that indicates the quantization value of the non-target area included in the second encoded data or transmitted in association with the second encoded data (e.g. apparatus for encoding samples of a video image includes processing circuitry configured to adapt coding parameters, including a quantization parameter QP (i.e. a quantization value), for example, based on a result of recognition processing that classifies input encoding samples of the video image as belonging to a region (i.e. area, section, portion, etc.) of interest including a region of interest ROI (i.e. the target area), for example, or belonging to a region of non-interest including a region of non-interest RONI (i.e. the non-target area), to generate an importance map (i.e. a ROI-RONI map), for example, and the adaption (i.e. adjustment) of the quantization parameter is performed by a QP map (i.e. a quantization value map) generator, which uses the importance map (i.e. the ROI-RONI map) as input (i.e. generates a quantization value map based on the quantization value of the target area and the quantization value of the non-target area), for example, in which segmentation is performed through classification of a pixel or a group of pixels of the video image, respectively, of a frame whether the pixel or the pixel group belongs to the foreground or background based on certain classification criteria (i.e. a predetermined pixel value condition), for example, including a map that includes an indication for each block CU of the video frame that includes a one bit indication with values “1” or “0” (i.e. information that indicates the quantization value of the target area and non-target area), respectively, for RONI or ROI regions (i.e. generates the quantization value map based on: information that indicates the target area and information that indicates the quantization value of the target area included in the first encoded data or transmitted in association with the first encoded data and information that indicates the non-target area and information that indicates the quantization value of the non-target area included in the second encoded data or transmitted in association with the second encoded data), as indicated above), for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 9, claim 1 is incorporated and the combination of Malakhov and HATTORI, as a whole, teaches the system (Malakhov, Par. [0007]), wherein
the transcoder (HATTORI, Par. [0107]: transcode controller 41 performs a process) generates a quantization value map (Malakhov, Par. [0172]: processing circuitry of the apparatus 900 (marked by the dashed box) is configured to perform the tasks of the machine-learning based model 910, the analysis of motion by a motion analyzer 930 based on the motion information, and the adaption of the coding parameters including frame rate adaptation 940, picture size adaptation 950, and generation of a QP map by a QP map generator 960 for QP adaptation; Par. [0231-242]: adapted coding parameter picture size is signaled by the module 1350 to the video encoder 1320… the coding parameter related to spatial resolution is a QP… The quantization parameter refers to the inverse number of bits used for encoding. Thus, a small QP value corresponds to a large number of bits, respectively, a large number of smaller increments. As a result, finer spatial variations within the video image may be encoded, improving image quality… the quantization parameter for a sample is set lower when the sample belongs to the ROI than in case the sample belongs to RONI… the adaption (i.e. the setting) of the quantization parameter is performed by the QP map generator 1360, which uses the importance map (i.e. the ROI-RONI map) as input. This map is provided as output by the machine-learning model 1310. The QP map generator uses this importance map to generate a corresponding map of the quantization parameters… method is provided for encoding samples of a video image comprising the steps of determining motion information for samples of a block in the video image, determining a coding parameter based on an output of a machine-learning-based model, to which the motion information and the samples of the block are input, and encoding the video frame by applying the coding parameter; generates the quantization value map (e.g. apparatus for encoding samples of a video image includes processing circuitry configured to adapt coding parameters, including a quantization parameter QP (i.e. a quantization value), for example, based on a result of recognition processing that classifies input encoding samples of the video image as belonging to a region (i.e. area, section, portion, etc.) of interest including a region of interest ROI (i.e. the target area), for example, or belonging to a region of non-interest including a region of non-interest RONI (i.e. the non-target area), to generate an importance map (i.e. a ROI-RONI map), for example, and the adaption (i.e. adjustment) of the quantization parameter is performed by a QP map (i.e. generates the quantization value map) generator, as indicated above, for example) based on a value that represents an attribute of the image data and a value that represents an attribute of the reconstructed image data (Malakhov, Par. [0065]: the original video pictures can be reconstructed, i.e. the reconstructed video pictures have the same quality as the original video pictures (assuming no transmission loss or other data loss during storage or transmission). In case of lossy video coding, further compression, e.g. by quantization, is performed, to reduce the amount of data representing the video pictures, which cannot be completely reconstructed at the decoder, i.e. the quality of the reconstructed video pictures is lower or worse compared to the quality of the original video pictures; based on a value that represents an attribute of the image data and a value that represents an attribute of the reconstructed image data (e.g. apparatus for encoding samples of a video image includes processing circuitry configured to adapt coding parameters, including a quantization parameter QP (i.e. a quantization value), for example, including original video pictures reconstructed (i.e. reconstructed image data), which have the same quality (i.e. attribute) as the original video pictures (i.e. a value that represents an attribute of the image data), assuming no transmission loss or other data loss during storage or transmission, for example, and in a case of lossy video coding, further compression by quantization is performed to reduce the amount of data representing the video pictures, which cannot be completely reconstructed at the decoder, and the quality of the reconstructed video pictures is lower or worse compared to the quality of the original video pictures (i.e. based on a value that represents an attribute of the image data and a value that represents an attribute of the reconstructed image data), as indicated above), for example), and re-encodes the reconstructed image data by using the generated quantization value map to generate the re-encoded data (Malakhov, Par. [0172]: processing circuitry of the apparatus 900 (marked by the dashed box) is configured to perform the tasks of the machine-learning based model 910, the analysis of motion by a motion analyzer 930 based on the motion information, and the adaption of the coding parameters including frame rate adaptation 940, picture size adaptation 950, and generation of a QP map by a QP map generator 960 for QP adaptation; Par. [0231-242]: adapted coding parameter picture size is signaled by the module 1350 to the video encoder 1320… the coding parameter related to spatial resolution is a QP… The quantization parameter refers to the inverse number of bits used for encoding. Thus, a small QP value corresponds to a large number of bits, respectively, a large number of smaller increments. As a result, finer spatial variations within the video image may be encoded, improving image quality… the quantization parameter for a sample is set lower when the sample belongs to the ROI than in case the sample belongs to RONI… the adaption (i.e. the setting) of the quantization parameter is performed by the QP map generator 1360, which uses the importance map (i.e. the ROI-RONI map) as input. This map is provided as output by the machine-learning model 1310. The QP map generator uses this importance map to generate a corresponding map of the quantization parameters… method is provided for encoding samples of a video image comprising the steps of determining motion information for samples of a block in the video image, determining a coding parameter based on an output of a machine-learning-based model, to which the motion information and the samples of the block are input, and encoding the video frame by applying the coding parameter; generates a quantization value map based on the quantization value of the target area and the quantization value of the non-target area, and re-encodes the reconstructed image data by using the generated quantization value map to generate the re-encoded data (e.g. apparatus for encoding samples of a video image includes processing circuitry configured to adapt coding parameters, including a quantization parameter QP (i.e. a quantization value), for example, based on a result of recognition processing that classifies input encoding samples of the video image as belonging to a region (i.e. area, section, portion, etc.) of interest including a region of interest ROI (i.e. the target area), for example, or belonging to a region of non-interest including a region of non-interest RONI (i.e. the non-target area), to generate an importance map (i.e. a ROI-RONI map), for example, and the adaption (i.e. adjustment) of the quantization parameter is performed by a QP map (i.e. a quantization value map) generator, which uses the importance map (i.e. the ROI-RONI map) as input (i.e. generates a quantization value map based on the quantization value of the target area and the quantization value of the non-target area), for example, and encodes each video frame of the video image by applying corresponding coding parameters to the ROI region and to the RONI region, respectively (i.e. and re-encodes the reconstructed image data by using the generated quantization value map to generate the re-encoded data), as indicated above, for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 10, claim 9 is incorporated and the combination of Malakhov and HATTORI, as a whole, teaches the system (Malakhov, Par. [0007]), wherein
the transcoder (HATTORI, Par. [0107]: transcode controller 41 performs a process) uses the quantization value calculated based on a difference between the value that represents the attribute of the image data and the value that represents the attribute of the reconstructed image data to generate the quantization value map (Malakhov, Par. [0124]: encoder 100… configured to select a reference block from a plurality of reference blocks of the same or different pictures of the plurality of other pictures and provide a reference picture (or reference picture index) and/or an offset (spatial offset) between the position (x, y coordinates) of the reference block and the position of the current block as inter estimation parameters 143 to the inter prediction unit 144. This offset is also called MV. The inter estimation is also referred to as motion estimation (ME) and the inter prediction also motion prediction (MP); Par. [0172]: processing circuitry of the apparatus 900 (marked by the dashed box) is configured to perform the tasks of the machine-learning based model 910, the analysis of motion by a motion analyzer 930 based on the motion information, and the adaption of the coding parameters including frame rate adaptation 940, picture size adaptation 950, and generation of a QP map by a QP map generator 960 for QP adaptation; Par. [0205-207]: calculate motion for a sample based on a metric of the motion vector of a block in which the sample is located… The term motion refers to a temporal change of the sample or a group of samples of a block, including changes of luma and/or color intensity within a video image and/or between different (e.g. adjacent) frames of a video; Par. [0231-242]: adapted coding parameter picture size is signaled by the module 1350 to the video encoder 1320… the coding parameter related to spatial resolution is a QP… The quantization parameter refers to the inverse number of bits used for encoding. Thus, a small QP value corresponds to a large number of bits, respectively, a large number of smaller increments. As a result, finer spatial variations within the video image may be encoded, improving image quality… the quantization parameter for a sample is set lower when the sample belongs to the ROI than in case the sample belongs to RONI… the adaption (i.e. the setting) of the quantization parameter is performed by the QP map generator 1360, which uses the importance map (i.e. the ROI-RONI map) as input. This map is provided as output by the machine-learning model 1310. The QP map generator uses this importance map to generate a corresponding map of the quantization parameters… method is provided for encoding samples of a video image comprising the steps of determining motion information for samples of a block in the video image, determining a coding parameter based on an output of a machine-learning-based model, to which the motion information and the samples of the block are input, and encoding the video frame by applying the coding parameter; uses the quantization value calculated based on a difference between the value that represents the attribute of the image data and the value that represents the attribute of the reconstructed image data to generate the quantization value map (e.g. apparatus for encoding samples of a video image includes processing circuitry configured to adapt coding parameters, including a quantization parameter QP (i.e. a quantization value), for example, including adaption of the coding parameter based on motion information, including aspatial offset between x, y coordinates of a reference block and a position of the current block (i.e. based on a difference between the value that represents the attribute of the image data and the value that represents the attribute of the reconstructed image data), for example, including original video pictures reconstructed (i.e. reconstructed image data), which have the same quality (i.e. attribute) as the original video pictures (i.e. a value that represents an attribute of the image data), assuming no transmission loss or other data loss during storage or transmission, for example, and in a case of lossy video coding, further compression by quantization is performed to reduce the amount of data representing the video pictures, which cannot be completely reconstructed at the decoder, and the quality of the reconstructed video pictures is lower or worse compared to the quality of the original video pictures, for example, in which the adaption (i.e. adjustment) of the quantization parameter is performed by a QP map (i.e. a quantization value map) generator, which uses the importance map (i.e. the ROI-RONI map) as input (i.e. generates a quantization value map based on the quantization value of the target area and the quantization value of the non-target area), for example, and encodes each video frame of the video image by applying corresponding coding parameters to the ROI region and to the RONI region, respectively, (i.e. uses the quantization value calculated based on a difference between the value that represents the attribute of the image data and the value that represents the attribute of the reconstructed image data to generate the quantization value map), as indicated above), for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 11, claim 10 is incorporated and the combination of Malakhov and HATTORI, as a whole, teaches the system (Malakhov, Par. [0007]), wherein
the transcoder (HATTORI, Par. [0107]: transcode controller 41 performs a process), when the image data is a P-picture, generates the quantization value map after adjusting the quantization value according to a number of encoding blocks to which an intra prediction mode is applied by a first encodement unit (Par. [0066]: video coding standards since H.261… combine spatial and temporal prediction in the sample domain and two-dimensional (2D) transform coding for applying quantization in the transform domain… Each picture of a video sequence is typically partitioned into a set of non-overlapping blocks and the coding is typically performed on a block level. That is, at the encoder the video is typically processed, i.e. encoded, on a block (video block) level, e.g. using spatial (intra picture) prediction and temporal (inter picture) prediction to generate a prediction block, subtracting the prediction block from the current block (block currently processed/to be processed) to obtain a residual block, transforming the residual block and quantizing the residual block in the transform domain to reduce the amount of data to be transmitted (compression), whereas at the decoder the inverse processing compared to the encoder is applied to the encoded or compressed block to reconstruct the current block for representation. Furthermore, the encoder duplicates the decoder processing loop such that both will generate identical predictions (e.g. intra- and inter predictions) and/or re-constructions for processing, i.e. coding, the subsequent blocks; Par. [0172]: processing circuitry of the apparatus 900 (marked by the dashed box) is configured to perform the tasks of the machine-learning based model 910, the analysis of motion by a motion analyzer 930 based on the motion information, and the adaption of the coding parameters including frame rate adaptation 940, picture size adaptation 950, and generation of a QP map by a QP map generator 960 for QP adaptation; Par. [0231-242]: adapted coding parameter picture size is signaled by the module 1350 to the video encoder 1320… the coding parameter related to spatial resolution is a QP… The quantization parameter refers to the inverse number of bits used for encoding. Thus, a small QP value corresponds to a large number of bits, respectively, a large number of smaller increments. As a result, finer spatial variations within the video image may be encoded, improving image quality… the quantization parameter for a sample is set lower when the sample belongs to the ROI than in case the sample belongs to RONI… the adaption (i.e. the setting) of the quantization parameter is performed by the QP map generator 1360, which uses the importance map (i.e. the ROI-RONI map) as input. This map is provided as output by the machine-learning model 1310. The QP map generator uses this importance map to generate a corresponding map of the quantization parameters… method is provided for encoding samples of a video image comprising the steps of determining motion information for samples of a block in the video image, determining a coding parameter based on an output of a machine-learning-based model, to which the motion information and the samples of the block are input, and encoding the video frame by applying the coding parameter; when the image data is a P-picture, generates the quantization value map after adjusting the quantization value according to a number of encoding blocks to which an intra prediction mode is applied by a first encodement unit (e.g. apparatus for encoding samples of a video image includes processing circuitry configured to adapt coding parameters, including a quantization parameter QP (i.e. a quantization value), for example, based on a result of recognition processing that classifies input encoding samples of the video image as belonging to a region (i.e. area, section, portion, etc.) of interest including a region of interest ROI (i.e. the target area), for example, or belonging to a region of non-interest including a region of non-interest RONI (i.e. the non-target area), to generate an importance map (i.e. a ROI-RONI map), for example, and the adaption (i.e. adjusting) of the quantization parameter is performed by a QP map (i.e. a quantization value map) generator (i.e. generates the quantization value map after adjusting the quantization value), which uses the importance map (i.e. the ROI-RONI map) as input (i.e. generates a quantization value map based on the quantization value of the target area and the quantization value of the non-target area), for example, and at the encoder the video is encoded on a video block level (i.e. a number of encoding blocks) using spatial intra picture prediction (i.e. when the image data is a P-picture, generates the quantization value map after adjusting the quantization value according to a number of encoding blocks to which an intra prediction mode is applied by a first encodement unit), as indicated above), for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 12, claim 1 is incorporated and the combination of Malakhov and HATTORI, as a whole, teaches the system (Malakhov, Par. [0007]), wherein
the transcoder (HATTORI, Par. [0107]: transcode controller 41 performs a process) generates a quantization value map (Malakhov, Par. [0172]: processing circuitry of the apparatus 900 (marked by the dashed box) is configured to perform the tasks of the machine-learning based model 910, the analysis of motion by a motion analyzer 930 based on the motion information, and the adaption of the coding parameters including frame rate adaptation 940, picture size adaptation 950, and generation of a QP map by a QP map generator 960 for QP adaptation; Par. [0231-242]: adapted coding parameter picture size is signaled by the module 1350 to the video encoder 1320… the coding parameter related to spatial resolution is a QP… The quantization parameter refers to the inverse number of bits used for encoding. Thus, a small QP value corresponds to a large number of bits, respectively, a large number of smaller increments. As a result, finer spatial variations within the video image may be encoded, improving image quality… the quantization parameter for a sample is set lower when the sample belongs to the ROI than in case the sample belongs to RONI… the adaption (i.e. the setting) of the quantization parameter is performed by the QP map generator 1360, which uses the importance map (i.e. the ROI-RONI map) as input. This map is provided as output by the machine-learning model 1310. The QP map generator uses this importance map to generate a corresponding map of the quantization parameters… method is provided for encoding samples of a video image comprising the steps of determining motion information for samples of a block in the video image, determining a coding parameter based on an output of a machine-learning-based model, to which the motion information and the samples of the block are input, and encoding the video frame by applying the coding parameter; generates the quantization value map (e.g. apparatus for encoding samples of a video image includes processing circuitry configured to adapt coding parameters, including a quantization parameter QP (i.e. a quantization value), for example, based on a result of recognition processing that classifies input encoding samples of the video image as belonging to a region (i.e. area, section, portion, etc.) of interest including a region of interest ROI (i.e. the target area), for example, or belonging to a region of non-interest including a region of non-interest RONI (i.e. the non-target area), to generate an importance map (i.e. a ROI-RONI map), for example, and the adaption (i.e. adjustment) of the quantization parameter is performed by a QP map (i.e. generates the quantization value map) generator, as indicated above, for example) based on a value that represents an attribute of the reconstructed image data (Malakhov, Par. [0065]: the original video pictures can be reconstructed, i.e. the reconstructed video pictures have the same quality as the original video pictures (assuming no transmission loss or other data loss during storage or transmission). In case of lossy video coding, further compression, e.g. by quantization, is performed, to reduce the amount of data representing the video pictures, which cannot be completely reconstructed at the decoder, i.e. the quality of the reconstructed video pictures is lower or worse compared to the quality of the original video pictures; based on a value that represents an attribute of the image data and a value that represents an attribute of the reconstructed image data (e.g. apparatus for encoding samples of a video image includes processing circuitry configured to adapt coding parameters, including a quantization parameter QP (i.e. a quantization value), for example, including original video pictures reconstructed (i.e. reconstructed image data), which have the same quality (i.e. attribute) as the original video pictures (i.e. a value that represents an attribute of the image data), assuming no transmission loss or other data loss during storage or transmission, for example, and in a case of lossy video coding, further compression by quantization is performed to reduce the amount of data representing the video pictures, which cannot be completely reconstructed at the decoder, and the quality of the reconstructed video pictures is lower or worse compared to the quality of the original video pictures (i.e. based on a value that represents an attribute of the image data and a value that represents an attribute of the reconstructed image data), as indicated above), for example), and re-encodes the reconstructed image data by using the generated quantization value map to generate the re-encoded data(Malakhov, Par. [0172]: processing circuitry of the apparatus 900 (marked by the dashed box) is configured to perform the tasks of the machine-learning based model 910, the analysis of motion by a motion analyzer 930 based on the motion information, and the adaption of the coding parameters including frame rate adaptation 940, picture size adaptation 950, and generation of a QP map by a QP map generator 960 for QP adaptation; Par. [0231-242]: adapted coding parameter picture size is signaled by the module 1350 to the video encoder 1320… the coding parameter related to spatial resolution is a QP… The quantization parameter refers to the inverse number of bits used for encoding. Thus, a small QP value corresponds to a large number of bits, respectively, a large number of smaller increments. As a result, finer spatial variations within the video image may be encoded, improving image quality… the quantization parameter for a sample is set lower when the sample belongs to the ROI than in case the sample belongs to RONI… the adaption (i.e. the setting) of the quantization parameter is performed by the QP map generator 1360, which uses the importance map (i.e. the ROI-RONI map) as input. This map is provided as output by the machine-learning model 1310. The QP map generator uses this importance map to generate a corresponding map of the quantization parameters… method is provided for encoding samples of a video image comprising the steps of determining motion information for samples of a block in the video image, determining a coding parameter based on an output of a machine-learning-based model, to which the motion information and the samples of the block are input, and encoding the video frame by applying the coding parameter; generates a quantization value map based on the quantization value of the target area and the quantization value of the non-target area, and re-encodes the reconstructed image data by using the generated quantization value map to generate the re-encoded data (e.g. apparatus for encoding samples of a video image includes processing circuitry configured to adapt coding parameters, including a quantization parameter QP (i.e. a quantization value), for example, based on a result of recognition processing that classifies input encoding samples of the video image as belonging to a region (i.e. area, section, portion, etc.) of interest including a region of interest ROI (i.e. the target area), for example, or belonging to a region of non-interest including a region of non-interest RONI (i.e. the non-target area), to generate an importance map (i.e. a ROI-RONI map), for example, and the adaption (i.e. adjustment) of the quantization parameter is performed by a QP map (i.e. a quantization value map) generator, which uses the importance map (i.e. the ROI-RONI map) as input (i.e. generates a quantization value map based on the quantization value of the target area and the quantization value of the non-target area), for example, and encodes each video frame of the video image by applying corresponding coding parameters to the ROI region and to the RONI region, respectively (i.e. and re-encodes the reconstructed image data by using the generated quantization value map to generate the re-encoded data), as indicated above, for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 15, claim 3 is incorporated and the combination of Malakhov and HATTORI, as a whole, teaches the system (Malakhov, Par. [0007]), wherein
the transcoder (HATTORI, Par. [0107]: transcode controller 41 performs a process) calculates a correction coefficient that corrects the quantization value map based on a change in a recognition rate (Malakhov, Par. [0098-101]: calculate a residual block 105 based on the picture block 103 and a prediction block 165… by subtracting sample values of the prediction block 165 from sample values of the picture block 103, sample by sample (pixel by pixel) to obtain the residual block 105 in the sample domain. Residual values (forming residual signal) correspond to prediction error (prediction error signal)… apply a transformation… on the sample values of the residual block 105 to obtain transformed coefficients 107 in a transform domain. The transformed coefficients 107 may also be referred to as transformed residual coefficients and represent the residual block 105 in the transform domain… quantize the transformed coefficients 107 to obtain quantized coefficients… by applying scalar quantization or vector quantization. The quantized coefficients 109 may also be referred to as quantized residual coefficients 109. For example, for scalar quantization, different scaling may be applied to achieve finer or coarser quantization. Smaller quantization step sizes correspond to finer quantization, whereas larger quantization step sizes correspond to coarser quantization. The applicable quantization step size may be indicated by a QP; Par. [0172-173]: processing circuitry of the apparatus 900 (marked by the dashed box) is configured to perform the tasks of the machine-learning based model 910, the analysis of motion by a motion analyzer 930 based on the motion information, and the adaption of the coding parameters including frame rate adaptation 940, picture size adaptation 950, and generation of a QP map by a QP map generator 960 for QP adaptation… The coding parameters picture size and quantization parameter QP refer to parameters of the spatial resolution; Par. [0231-239]: adapted coding parameter picture size is signaled by the module 1350 to the video encoder 1320… the coding parameter related to spatial resolution is a QP… The quantization parameter refers to the inverse number of bits used for encoding. Thus, a small QP value corresponds to a large number of bits, respectively, a large number of smaller increments. As a result, finer spatial variations within the video image may be encoded, improving image quality… the quantization parameter for a sample is set lower when the sample belongs to the ROI than in case the sample belongs to RONI… the adaption (i.e. the setting) of the quantization parameter is performed by the QP map generator 1360, which uses the importance map (i.e. the ROI-RONI map) as input. This map is provided as output by the machine-learning model 1310. The QP map generator uses this importance map to generate a corresponding map of the quantization parameters… Based on the importance (bit)map, the QP map generator 1360 generates a corresponding QP map, which contains the adapted QP value for a CTU/CU of the frame. This QP map is signaled by the module 1360 to the video encoder 1320; calculates a correction coefficient that corrects the quantization value map based on a change in a recognition rate (e.g. apparatus for encoding samples of a video image includes machine-learning-based models configured to detect (i.e. recognize, identify, distinguish, etc.) an interesting object (i.e. a recognition target) within the video image, processing circuitry configured to adapt coding parameters, including a quantization parameter QP (i.e. a quantization value), for example, based on a result of recognition processing (i.e. based on a change in a recognition rate) that classifies input encoding samples of the video image as belonging to a region (i.e. area, section, portion, etc.) of interest including a region of interest ROI (i.e. the target area), for example, or belonging to a region of non-interest including a region of non-interest RONI (i.e. the non-target area), to generate an importance map (i.e. a ROI-RONI map), for example, and the adaption (i.e. adjustment, correction, etc.) of the quantization parameter is performed by a QP map generator (i.e. corrects the quantization value map), which uses the importance map (i.e. the ROI-RONI map) as input, for example, and applying a transformation on the sample values of a residual block to obtain transformed coefficients, also referred to as transformed residual coefficients (i.e. calculates a correction coefficient), and represent the residual block, for example, by quantizing the transformed coefficients to obtain quantized coefficients (i.e. calculates a correction coefficient that corrects the quantization value map based on a change in a recognition rate), and multiplies the generated quantization value map by the correction coefficient to correct the quantization value map (Malakhov, Par. [0101]: quantization unit 108 is configured to quantize the transformed coefficients 107 to obtain quantized coefficients 109, e.g. by applying scalar quantization or vector quantization. The quantized coefficients 109 may also be referred to as quantized residual coefficients 109. For example, for scalar quantization, different scaling may be applied to achieve finer or coarser quantization… applicable quantization step size may be indicated by a QP… The quantization may include division by a quantization step size and corresponding or inverse dequantization, e.g. by inverse quantization 110, may include multiplication by the quantization step size; Par. [0172-173]: processing circuitry of the apparatus 900 (marked by the dashed box) is configured to perform the tasks of the machine-learning based model 910, the analysis of motion by a motion analyzer 930 based on the motion information, and the adaption of the coding parameters including frame rate adaptation 940, picture size adaptation 950, and generation of a QP map by a QP map generator 960 for QP adaptation… The coding parameters picture size and quantization parameter QP refer to parameters of the spatial resolution; multiplies the generated quantization value map by the correction coefficient to correct the quantization value map (e.g. apparatus for encoding samples of a video image includes processing circuitry configured to adapt coding parameters, including a quantization parameter QP (i.e. a quantization value), for example, including quantizing transformed coefficients, also referred to as quantized residual coefficients, that are obtained by applying scalar quantization or vector quantization to achieve finer or coarser quantization, for example, in which an applicable quantization step is indicated by a QP and the quantization include division by a quantization step size and corresponding or inverse dequantization, including multiplication by the quantization step size, for example, and generates a QP map for the QP adaptation (i.e. multiplies the generated quantization value map by the correction coefficient to correct the quantization value map), as indicated above), for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 18, Malakhov discloses an image processing device (Par. [0007]: an apparatus is provided for encoding samples of a video image comprising) that acquires a first encoded data in which an entire area of image data is encoded with a quantization value of a target area needed to recognize a recognition target in the image data and a second encoded data in which an entire area of the image data is encoded with a quantization value of a non-target area other than the target in the image data, the target area, the non-target area, the quantization value of the target area, and the quantization value of non-target area are decided based on a result of recognition processing (Par. [0005-18]: coding parameters, for example… quantization parameter (QP), used for video image encoding are often preset and signaled to the encoder… a key aspect is to detect whether interesting objects… even exist in the video image apart from the image background, whether these objects move and how fast. Dependent on these factors, the coding parameters may be adapted… processing circuitry of the apparatus is further configured to divide the video image into equally sized coding tree units (CTUs), hierarchically split one of the coding tree units into a plurality of coding units, determine motion vector for each of the plurality of the coding units, input to the machine-learning-based model the coding tree unit and the respective motion vectors determined for the plurality of the coding units of the coding tree unit… the machine-learning-based model is configured to classify the input sample as belonging to a region of interest (ROI) or to a region of non-interest (RONI)… obtain as the output of the machine-learning-based model a map specifying the ROI and RONI of the video image … the machine-learning-based model is configured to detect, using the machine-learning-based model, an object within the video image; Par. [0023-29]: determine the coding parameter by adapting spatial resolution based on whether or not an object was detected in the video image… the coding parameter related to the spatial resolution is QP… the processing circuitry is further configured to set the quantization parameter for a sample lower in a case when the sample belongs to ROI than in a case when the sample belongs to RONI… method is provided for encoding samples of a video image comprising the steps of determining motion information for samples of a block in the video image, determining a coding parameter based on an output of a machine-learning-based model, to which the motion information and the samples of the block are input, and encoding the video frame by applying the coding parameter; Par. [0039-41]: method includes the step of determining the coding parameter by adapting spatial resolution based on whether or not an object was detected in the video image… method steps include setting the quantization parameter for a sample lower in a case when the sample belongs to ROI than in a case when the sample belongs to RONI; Par. [0159-167]: a machine-learning-based model, which is input with motion information and frames of a video image, for adapting one or multiple coding parameters used for encoding the video image… an apparatus is provided wherein a video image, including a plurality of video frames, is encoded using coding parameters in order to provide a video bitstream as the output… segmentation result of the machine-learning model 820 is output and provided as input to the module 830 to perform the coding parameter adaptation combined with the motion information… the image segmentation by machine-learning model is based on classification of the sample or the group of samples belonging to a ROI or a RONI… the machine-learning model may be trained to directly deliver the coding parameters, which are then input to the encoder 830. The coding parameters may be determined for one sample or a group of samples of the video image. The coding parameters of different single samples or different groups of samples may thus be different… The sample is encoded by applying the determined coding parameters. The encoding of a sample or the samples is performed by an encoder 430 and provides the encoded video image as a bitstream video; Par. [0231-242]: adapted coding parameter picture size is signaled by the module 1350 to the video encoder 1320… the coding parameter related to spatial resolution is a QP… The quantization parameter refers to the inverse number of bits used for encoding. Thus, a small QP value corresponds to a large number of bits, respectively, a large number of smaller increments. As a result, finer spatial variations within the video image may be encoded, improving image quality… the quantization parameter for a sample is set lower when the sample belongs to the ROI than in case the sample belongs to RONI… the adaption (i.e. the setting) of the quantization parameter is performed by the QP map generator 1360, which uses the importance map (i.e. the ROI-RONI map) as input. This map is provided as output by the machine-learning model 1310. The QP map generator uses this importance map to generate a corresponding map of the quantization parameters… method is provided for encoding samples of a video image comprising the steps of determining motion information for samples of a block in the video image, determining a coding parameter based on an output of a machine-learning-based model, to which the motion information and the samples of the block are input, and encoding the video frame by applying the coding parameter; that acquires a first encoded data in which an entire area of image data is encoded with a quantization value of a target area needed to recognize a recognition target in the image data and a second encoded data in which an entire area of the image data is encoded with a quantization value of a non-target area other than the target in the image data the target area, the non-target area, the quantization value of the target area, and the quantization value of non-target area are decided based on a result of recognition processing (e.g. apparatus for encoding samples of a video image (i.e. an image processing device) includes machine-learning-based models configured to detect (i.e. recognize, identify, distinguish, etc.) an interesting object within the video image (i.e. a recognition target in an entire area of image data), for example, including a machine-learning-based model that is configured to classify input encoding samples of the video image as belonging to a region (i.e. area, section, portion, etc.) of interest (i.e. a target area needed to recognize a recognition target including a region of interest (ROI), such as foreground or object regions, for example), or belonging to a region of non-interest RONI (i.e. a non-target area other than the target in the image data), such as a less relevant background region, for example, including processing circuitry configured to adapt coding parameters, including a quantization parameter (i.e. a quantization value) QP, for example, based on a result of recognition processing that classifies input encoding samples of the video image as belonging to a region of interest ROI (i.e. a quantization value of a target area needed to recognize a recognition target in the image data), or belonging to a region of non-interest including a region of non-interest RONI (i.e. a quantization value of a non-target area other than the target in the image data), for example, and encodes each video frame of the video image (i.e. an entire area of the image data) by applying corresponding coding parameters to the ROI region to generate first encoded data and to the RONI region to generate second encoded data, respectively (i.e. acquires a first encoded data in which an entire area of image data is encoded with a quantization value of a target area needed to recognize a recognition target in the image data and a second encoded data in which an entire area of the image data is encoded with a quantization value of a non-target area other than the target in the image data), for example, by using machine-learning-based models configured to detect an interesting object within the video image, classify input encoding samples of the video image as belonging to a region of interest ROI, or belonging to a region of non-interest RONI, adapt coding parameters, including a quantization parameter QP, for example, based on a result of recognition processing that classifies input encoding samples of the video image as belonging to a region of interest ROI (i.e. the target area, the non-target area, the quantization value of the target area, and the quantization value of non-target area are decided based on a result of recognition processing), as indicated above), for example), the image processing device comprising:
a transcoder that generates reconstructed image data by using the target area in first decoded data obtained by decoding the first encoded data and the non-target area in second decoded data obtained by decoding the second encoded data; and re-encodes the reconstructed image data to generate re-encoded data (e.g. apparatus for encoding samples of a video image includes processing circuitry configured to adapt coding parameters, including a quantization parameter QP, for example, based on a result of recognition processing that classifies input encoding samples of the video image as belonging to a region (i.e. area, section, portion, etc.) of interest including a region of interest ROI (i.e. the target area), for example, or belonging to a region of non-interest including a region of non-interest RONI (i.e. the non-target area), for example, and encodes each video frame of the video image by applying corresponding coding parameters to the ROI region (i.e. the target area used to generate first encoded data) and to the RONI region, to generate second encoded data (i.e. the non-target area used to generate second encoded data), respectively, as indicated above, for example, including video coding that comprises two parts, video encoding and video decoding (i.e. a transcoder, CODEC, etc.), in which video encoding is performed at a source side and video decoding is performed at a destination side and typically comprises the inverse processing compared to the encoder to reconstruct the video pictures (i.e. a transcoder that generates reconstructed image), for example, including decoded pictures and corresponding reference blocks and samples and/or a partially reconstructed current picture and corresponding reference blocks and samples (i.e. a transcoder that generates reconstructed image data by using the target area in first decoded data obtained by decoding the first encoded data and the non-target area in second decoded data obtained by decoding the second encoded data) used for inter estimation and/or inter prediction, for example, and the encoder duplicates decoder processing loop such that both will generate identical predictions (i.e. intra- and inter predictions) and/or re-constructions for processing (i.e. coding, re-encode, etc.) subsequent blocks (i.e. and re-encodes the reconstructed image data to generate re-encoded data), as indicated above), for example).
Malakhov teachings above disclose the apparatus for encoding samples of a video image, detecting an interesting object within the video image, in which “an interesting object” corresponds to the claimed “a recognition target”, by classifying input encoding samples of the video image as belonging to a region of interest ROI, in which a ROI corresponds to the claimed “target area”, such as foreground or object regions, and a region of non-interest RONI, in which a RONI corresponds to the claimed “non-target area”, such as a less relevant background region, for example, but do not expressly disclose “target”. Additionally, Malakhov teachings above disclose video coding that comprises two parts, video encoding and video decoding, in which video encoding and video decoding corresponds to the claimed “transcoder” process, including a CODEC, for example, but do not expressly disclose “a transcoder”.
However, HATTORI teaches “target” and “transcoder” (Par. [0045-60]: Referring to FIG. 1, the video encoding device 1 generates an entire region bitstream by performing an encoding process, in a form that enables the video transcoding device 2 in the next stage to process the entire region bitstream, on the picture of an entire region (a region including an entire frame) of an inputted video, and multiplexes hint information used for transcoding (the details of the hint information will be described below) into the entire region bitstream and outputs the entire region bitstream after multiplexing of the hint information to the video transcoding device 2… video encoding device 1 performs a process of determining coding parameters for a coding target block in a picture belonging to a GOP (Group of pictures) and generating a prediction image by using the coding parameter, and also compression-encoding a difference image between the coding target block and the prediction image and multiplexing encoded data which is the result of the encoding and the coding parameters to generate an entire region bitstream… video transcoding device 2 is configured with an entire region stream decoder 3 and a partial region transcoder 4, and performs a process of decoding the image of the entire region from the entire region bitstream generated by the video encoding device 1, and outputting the image of the entire region (referred to as the “entire region decoded image” from here on) to an entire region display device 5… The entire region stream decoder 3 performs a process of extracting the encoded data and the coding parameters of the entire region, and the hint information which are included in the entire region bitstream generated by the video encoding device 1, and decoding the encoded data and the coding parameters of the entire region into an entire region decoded image and outputting the entire region decoded image to the partial region transcoder 4 and the entire region display device 5, and also outputting the encoded data and the coding parameters of the entire region, and the hint information to the partial region transcoder 4… The partial region transcoder 4 performs a process of referring to the motion vector limitation information, the GOP size limitation information and the reference configuration specification information which are included in the hint information outputted from the entire region stream decoder 3, to specify an indispensable encoded region which is a region required at the time of decoding the display area of a picture… An indispensable encoded region determinator is configured with the entire region stream decoder 3 and the partial region transcoder 4… The partial region transcoder 4 also performs a process of extracting the encoded data and the coding parameters of a coding target block which is included in the above-mentioned indispensable encoded region from among the encoded data and the coding parameters of the entire region which are outputted from the entire region stream decoder 3, and generating a partial region bitstream in conformity with the encoding codec set in advance from the encoded data and the coding parameters of the coding target block…The partial region transcoder 4 constructs a parameter extractor and a partial region stream generator… The entire region display device 5 is display equipment to display the entire region decoded image outputted from the entire region stream decoder 3…The video decoding device 6 decodes an image of a partial region from the partial region bitstream outputted from the partial region transcoder 4, and outputs the image of the partial region (referred to as the “partial region decoded image” from here on) to a partial region display device 7; Par. [0107]: transcode controller 41 performs a process of referring to the motion vector limitation information, the GOP size limitation information and the reference configuration specification information which are included in the hint information outputted from the entire region stream decoder 3, to specify a region which is a target for transcoding (a target region to be transcoded) from the display area of a picture, the display area being indicated by the display area information provided therefor from the outside thereof, and also specify an indispensable encoded region which is a region required at the time of decoding the target region to be transcoded (a region to which the coding parameters need to be applied at the time of transcoding), and outputting target region to be transcoded information indicating the target region to be transcoded and indispensable encoded region information indicating the indispensable encoded region; “target” and “transcoder” (e.g. video encoding device performs a process of determining coding parameters for a coding target (“target”) block in a picture belonging to a GOP (Group of pictures) and generating a prediction image by using the coding parameter, and also compression-encoding a difference image between the coding target block and the prediction image and multiplexing encoded data which is the result of the encoding and the coding parameters to generate an entire region bitstream, for example, including video transcoding device, or transcoder (“transcoder”), for example, which performs a process of extracting encoded data and coding parameters of a coding target block which is included in an indispensable encoded region from among encoded data and the coding parameters of an entire region which are outputted from the entire region stream decoder, and generating a partial region bitstream in conformity with the encoding codec set in advance from the encoded data and the coding parameters of the coding target block, for example, by specify a region which is a target (“target”) for transcoding (a target region to be transcoded) from the display area of a picture, and also by specifying the indispensable encoded region which is a region required at the time of decoding the target region to be transcoded (a region to which the coding parameters need to be applied at the time of transcoding), and outputting target region to be transcoded information indicating the target region to be transcoded and indispensable encoded region information indicating the indispensable encoded region, as indicated above), for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 19, Malakhov discloses an image processing method (Par. [0029-30]: a method is provided for encoding samples of a video image comprising… dividing the video image into equally sized coding tree units, hierarchically splitting one of the coding tree units into a plurality of coding units) implemented by a computer of an image processing device (Par. [0007]: an apparatus is provided for encoding samples of a video image comprising; Par. [0042]: a computer-readable non-transitory medium is provided for storing a program, including instructions which when executed on a processor cause the processor to perform the method steps; Par. [0153] a non-transitory computer readable medium comprising a program code that, when executed by a processor, causes a computer system to perform any of the methods described herein; Par. [0243] a computer-readable non-transitory medium stores a program, including instructions which when executed on a processor cause the processor to perform the steps of the method) that acquires a first encoded data in which an entire area of image data is encoded with a quantization value of a target area needed to recognize a recognition target in the image data and a second encoded data in which an entire area of the image data is encoded with a quantization value of a non-target area other than the target in the image data, the target area, the non-target area, the quantization value of the target area, and the quantization value of non-target area are decided based on a result of recognition processing (Par. [0005-18]: coding parameters, for example… quantization parameter (QP), used for video image encoding are often preset and signaled to the encoder… a key aspect is to detect whether interesting objects… even exist in the video image apart from the image background, whether these objects move and how fast. Dependent on these factors, the coding parameters may be adapted… processing circuitry of the apparatus is further configured to divide the video image into equally sized coding tree units (CTUs), hierarchically split one of the coding tree units into a plurality of coding units, determine motion vector for each of the plurality of the coding units, input to the machine-learning-based model the coding tree unit and the respective motion vectors determined for the plurality of the coding units of the coding tree unit… the machine-learning-based model is configured to classify the input sample as belonging to a region of interest (ROI) or to a region of non-interest (RONI)… obtain as the output of the machine-learning-based model a map specifying the ROI and RONI of the video image … the machine-learning-based model is configured to detect, using the machine-learning-based model, an object within the video image; Par. [0023-29]: determine the coding parameter by adapting spatial resolution based on whether or not an object was detected in the video image… the coding parameter related to the spatial resolution is QP… the processing circuitry is further configured to set the quantization parameter for a sample lower in a case when the sample belongs to ROI than in a case when the sample belongs to RONI… method is provided for encoding samples of a video image comprising the steps of determining motion information for samples of a block in the video image, determining a coding parameter based on an output of a machine-learning-based model, to which the motion information and the samples of the block are input, and encoding the video frame by applying the coding parameter; Par. [0039-41]: method includes the step of determining the coding parameter by adapting spatial resolution based on whether or not an object was detected in the video image… method steps include setting the quantization parameter for a sample lower in a case when the sample belongs to ROI than in a case when the sample belongs to RONI; Par. [0159-167]: a machine-learning-based model, which is input with motion information and frames of a video image, for adapting one or multiple coding parameters used for encoding the video image… an apparatus is provided wherein a video image, including a plurality of video frames, is encoded using coding parameters in order to provide a video bitstream as the output… segmentation result of the machine-learning model 820 is output and provided as input to the module 830 to perform the coding parameter adaptation combined with the motion information… the image segmentation by machine-learning model is based on classification of the sample or the group of samples belonging to a ROI or a RONI… the machine-learning model may be trained to directly deliver the coding parameters, which are then input to the encoder 830. The coding parameters may be determined for one sample or a group of samples of the video image. The coding parameters of different single samples or different groups of samples may thus be different… The sample is encoded by applying the determined coding parameters. The encoding of a sample or the samples is performed by an encoder 430 and provides the encoded video image as a bitstream video; Par. [0231-242]: adapted coding parameter picture size is signaled by the module 1350 to the video encoder 1320… the coding parameter related to spatial resolution is a QP… The quantization parameter refers to the inverse number of bits used for encoding. Thus, a small QP value corresponds to a large number of bits, respectively, a large number of smaller increments. As a result, finer spatial variations within the video image may be encoded, improving image quality… the quantization parameter for a sample is set lower when the sample belongs to the ROI than in case the sample belongs to RONI… the adaption (i.e. the setting) of the quantization parameter is performed by the QP map generator 1360, which uses the importance map (i.e. the ROI-RONI map) as input. This map is provided as output by the machine-learning model 1310. The QP map generator uses this importance map to generate a corresponding map of the quantization parameters… method is provided for encoding samples of a video image comprising the steps of determining motion information for samples of a block in the video image, determining a coding parameter based on an output of a machine-learning-based model, to which the motion information and the samples of the block are input, and encoding the video frame by applying the coding parameter; that acquires a first encoded data in which an entire area of image data is encoded with a quantization value of a target area needed to recognize a recognition target in the image data and a second encoded data in which an entire area of the image data is encoded with a quantization value of a non-target area other than the target in the image data the target area, the non-target area, the quantization value of the target area, and the quantization value of non-target area are decided based on a result of recognition processing (e.g. apparatus for encoding samples of a video image (i.e. an image processing device) includes machine-learning-based models configured to detect (i.e. recognize, identify, distinguish, etc.) an interesting object within the video image (i.e. a recognition target in an entire area of image data), for example, including a machine-learning-based model that is configured to classify input encoding samples of the video image as belonging to a region (i.e. area, section, portion, etc.) of interest (i.e. a target area needed to recognize a recognition target including a region of interest (ROI), such as foreground or object regions, for example), or belonging to a region of non-interest RONI (i.e. a non-target area other than the target in the image data), such as a less relevant background region, for example, including processing circuitry configured to adapt coding parameters, including a quantization parameter (i.e. a quantization value) QP, for example, based on a result of recognition processing that classifies input encoding samples of the video image as belonging to a region of interest ROI (i.e. a quantization value of a target area needed to recognize a recognition target in the image data), or belonging to a region of non-interest including a region of non-interest RONI (i.e. a quantization value of a non-target area other than the target in the image data), for example, and encodes each video frame of the video image (i.e. an entire area of the image data) by applying corresponding coding parameters to the ROI region to generate first encoded data and to the RONI region to generate second encoded data, respectively (i.e. acquires a first encoded data in which an entire area of image data is encoded with a quantization value of a target area needed to recognize a recognition target in the image data and a second encoded data in which an entire area of the image data is encoded with a quantization value of a non-target area other than the target in the image data), for example, by using machine-learning-based models configured to detect an interesting object within the video image, classify input encoding samples of the video image as belonging to a region of interest ROI, or belonging to a region of non-interest RONI, adapt coding parameters, including a quantization parameter QP, for example, based on a result of recognition processing that classifies input encoding samples of the video image as belonging to a region of interest ROI (i.e. the target area, the non-target area, the quantization value of the target area, and the quantization value of non-target area are decided based on a result of recognition processing), as indicated above), for example), the image processing method comprising:
generating reconstructed image data by using the target area in first decoded data obtained by decoding the first encoded data and the non-target area in second decoded data obtained by decoding the second encoded data; and
re-encoding the reconstructed image data to generate re-encoded data (e.g. apparatus for encoding samples of a video image includes processing circuitry configured to adapt coding parameters, including a quantization parameter QP, for example, based on a result of recognition processing that classifies input encoding samples of the video image as belonging to a region (i.e. area, section, portion, etc.) of interest including a region of interest ROI (i.e. the target area), for example, or belonging to a region of non-interest including a region of non-interest RONI (i.e. the non-target area), for example, and encodes each video frame of the video image by applying corresponding coding parameters to the ROI region (i.e. the target area used to generate first encoded data) and to the RONI region, to generate second encoded data (i.e. the non-target area used to generate second encoded data), respectively, as indicated above, for example, including video coding that comprises two parts, video encoding and video decoding (i.e. a transcoder, CODEC, etc.), in which video encoding is performed at a source side and video decoding is performed at a destination side and typically comprises the inverse processing compared to the encoder to reconstruct the video pictures (i.e. a transcoder that generates reconstructed image), for example, including decoded pictures and corresponding reference blocks and samples and/or a partially reconstructed current picture and corresponding reference blocks and samples (i.e. generating reconstructed image data by using the target area in first decoded data obtained by decoding the first encoded data and the non-target area in second decoded data obtained by decoding the second encoded data) used for inter estimation and/or inter prediction, for example, and the encoder duplicates decoder processing loop such that both will generate identical predictions (i.e. intra- and inter predictions) and/or re-constructions for processing (i.e. coding, re-encode, etc.) subsequent blocks (i.e. and re-encoding the reconstructed image data to generate re-encoded data), as indicated above), for example).
Malakhov teachings above disclose the apparatus for encoding samples of a video image, detecting an interesting object within the video image, in which “an interesting object” corresponds to the claimed “a recognition target”, by classifying input encoding samples of the video image as belonging to a region of interest ROI, in which a ROI corresponds to the claimed “target area”, such as foreground or object regions, and a region of non-interest RONI, in which a RONI corresponds to the claimed “non-target area”, such as a less relevant background region, for example, but do not expressly disclose “target”. Additionally, Malakhov teachings above disclose video coding that comprises two parts, video encoding and video decoding, in which video encoding and video decoding corresponds to the claimed “transcoder” process, including a CODEC, for example, but do not expressly disclose “a transcoder”.
However, HATTORI teaches “target” and “transcoder” (Par. [0045-60]: Referring to FIG. 1, the video encoding device 1 generates an entire region bitstream by performing an encoding process, in a form that enables the video transcoding device 2 in the next stage to process the entire region bitstream, on the picture of an entire region (a region including an entire frame) of an inputted video, and multiplexes hint information used for transcoding (the details of the hint information will be described below) into the entire region bitstream and outputs the entire region bitstream after multiplexing of the hint information to the video transcoding device 2… video encoding device 1 performs a process of determining coding parameters for a coding target block in a picture belonging to a GOP (Group of pictures) and generating a prediction image by using the coding parameter, and also compression-encoding a difference image between the coding target block and the prediction image and multiplexing encoded data which is the result of the encoding and the coding parameters to generate an entire region bitstream… video transcoding device 2 is configured with an entire region stream decoder 3 and a partial region transcoder 4, and performs a process of decoding the image of the entire region from the entire region bitstream generated by the video encoding device 1, and outputting the image of the entire region (referred to as the “entire region decoded image” from here on) to an entire region display device 5… The entire region stream decoder 3 performs a process of extracting the encoded data and the coding parameters of the entire region, and the hint information which are included in the entire region bitstream generated by the video encoding device 1, and decoding the encoded data and the coding parameters of the entire region into an entire region decoded image and outputting the entire region decoded image to the partial region transcoder 4 and the entire region display device 5, and also outputting the encoded data and the coding parameters of the entire region, and the hint information to the partial region transcoder 4… The partial region transcoder 4 performs a process of referring to the motion vector limitation information, the GOP size limitation information and the reference configuration specification information which are included in the hint information outputted from the entire region stream decoder 3, to specify an indispensable encoded region which is a region required at the time of decoding the display area of a picture… An indispensable encoded region determinator is configured with the entire region stream decoder 3 and the partial region transcoder 4… The partial region transcoder 4 also performs a process of extracting the encoded data and the coding parameters of a coding target block which is included in the above-mentioned indispensable encoded region from among the encoded data and the coding parameters of the entire region which are outputted from the entire region stream decoder 3, and generating a partial region bitstream in conformity with the encoding codec set in advance from the encoded data and the coding parameters of the coding target block…The partial region transcoder 4 constructs a parameter extractor and a partial region stream generator… The entire region display device 5 is display equipment to display the entire region decoded image outputted from the entire region stream decoder 3…The video decoding device 6 decodes an image of a partial region from the partial region bitstream outputted from the partial region transcoder 4, and outputs the image of the partial region (referred to as the “partial region decoded image” from here on) to a partial region display device 7; Par. [0107]: transcode controller 41 performs a process of referring to the motion vector limitation information, the GOP size limitation information and the reference configuration specification information which are included in the hint information outputted from the entire region stream decoder 3, to specify a region which is a target for transcoding (a target region to be transcoded) from the display area of a picture, the display area being indicated by the display area information provided therefor from the outside thereof, and also specify an indispensable encoded region which is a region required at the time of decoding the target region to be transcoded (a region to which the coding parameters need to be applied at the time of transcoding), and outputting target region to be transcoded information indicating the target region to be transcoded and indispensable encoded region information indicating the indispensable encoded region; “target” and “transcoder” (e.g. video encoding device performs a process of determining coding parameters for a coding target (“target”) block in a picture belonging to a GOP (Group of pictures) and generating a prediction image by using the coding parameter, and also compression-encoding a difference image between the coding target block and the prediction image and multiplexing encoded data which is the result of the encoding and the coding parameters to generate an entire region bitstream, for example, including video transcoding device, or transcoder (“transcoder”), for example, which performs a process of extracting encoded data and coding parameters of a coding target block which is included in an indispensable encoded region from among encoded data and the coding parameters of an entire region which are outputted from the entire region stream decoder, and generating a partial region bitstream in conformity with the encoding codec set in advance from the encoded data and the coding parameters of the coding target block, for example, by specify a region which is a target (“target”) for transcoding (a target region to be transcoded) from the display area of a picture, and also by specifying the indispensable encoded region which is a region required at the time of decoding the target region to be transcoded (a region to which the coding parameters need to be applied at the time of transcoding), and outputting target region to be transcoded information indicating the target region to be transcoded and indispensable encoded region information indicating the indispensable encoded region, as indicated above), for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 20, Malakhov discloses a non-transitory computer readable recording medium storing an image processing program for causing a computer of an image processing device (Par. [0042] a computer-readable non-transitory medium is provided for storing a program, including instructions which when executed on a processor cause the processor to perform the method steps; Par. [0153] a non-transitory computer readable medium comprising a program code that, when executed by a processor, causes a computer system to perform any of the methods described herein; Par. [0243] a computer-readable non-transitory medium stores a program, including instructions which when executed on a processor cause the processor to perform the steps of the method). The steps of the program further recited in claim 20 correspond to claim 19 when executed and are rejected as applied to method claim 19 above.
Claims 7-8, 13, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Malakhov, in view of HATTORI, as applied to claim 1 above, in further view of Jang et al. (US PG Publication No. 2013/0322524 A1), hereafter referred to as Jang.
Regarding claim 7, claim 3 is incorporated and the combination of Malakhov and HATTORI, as a whole, teaches the system (Malakhov, Par. [0007]), wherein
the transcoder (HATTORI, Par. [0107]: transcode controller 41 performs a process) generates the quantization value map and re-encodes the reconstructed image data by using the generated quantization value map to generate the re-encoded data (Malakhov, Par. [0172]: processing circuitry of the apparatus 900 (marked by the dashed box) is configured to perform the tasks of the machine-learning based model 910, the analysis of motion by a motion analyzer 930 based on the motion information, and the adaption of the coding parameters including frame rate adaptation 940, picture size adaptation 950, and generation of a QP map by a QP map generator 960 for QP adaptation; Par. [0231-242]: adapted coding parameter picture size is signaled by the module 1350 to the video encoder 1320… the coding parameter related to spatial resolution is a QP… The quantization parameter refers to the inverse number of bits used for encoding. Thus, a small QP value corresponds to a large number of bits, respectively, a large number of smaller increments. As a result, finer spatial variations within the video image may be encoded, improving image quality… the quantization parameter for a sample is set lower when the sample belongs to the ROI than in case the sample belongs to RONI… the adaption (i.e. the setting) of the quantization parameter is performed by the QP map generator 1360, which uses the importance map (i.e. the ROI-RONI map) as input. This map is provided as output by the machine-learning model 1310. The QP map generator uses this importance map to generate a corresponding map of the quantization parameters… method is provided for encoding samples of a video image comprising the steps of determining motion information for samples of a block in the video image, determining a coding parameter based on an output of a machine-learning-based model, to which the motion information and the samples of the block are input, and encoding the video frame by applying the coding parameter; generates the quantization value map and re-encodes the reconstructed image data by using the generated quantization value map to generate the re-encoded data (e.g. apparatus for encoding samples of a video image includes processing circuitry configured to adapt coding parameters, including a quantization parameter QP (i.e. a quantization value), for example, based on a result of recognition processing that classifies input encoding samples of the video image as belonging to a region (i.e. area, section, portion, etc.) of interest including a region of interest ROI (i.e. the target area), for example, or belonging to a region of non-interest including a region of non-interest RONI (i.e. the non-target area), to generate an importance map (i.e. a ROI-RONI map), for example, and the adaption (i.e. adjustment) of the quantization parameter is performed by a QP map (i.e. a quantization value map) generator, which uses the importance map (i.e. the ROI-RONI map) as input, for example, and encodes each video frame of the video image by applying corresponding coding parameters to the ROI region and to the RONI region, respectively (i.e. and re-encodes the reconstructed image data by using the generated quantization value map to generate the re-encoded data), as indicated above, for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Malakhov and HATTORI, as a whole, teach the apparatus, as indicated above, but fail to teach the following as further recited in claim 7.
However, Jang teaches quantization value to achieve a re-bit rate at a time of transmitting the re-encoded data, the re-bit rate being determined based on a bit rate at a time of transmitting the first encoded data and the second encoded data (Par. [0005-6]: inventive concept provides a technology of controlling bit rates that prevents degradation in image quality during multi-layered video encoding… provides a rate control method for multi-layered video coding, performed to control bit rates at an upper layer by using encoding statistical information and/or region-of-interest (ROI) information generated based on the result of performing encoding on a lower layer. A second rate controller generates a plurality of second quantization parameters based on the encoding statistical information, to be used when encoding is performed on a plurality of regions at the second (upper) layer… a video encoder for controlling bit rates at an upper layer by using encoding statistical information and/or ROI information generated based on the result of performing encoding on a lower layer; Par. [0128-135]: base layer encoder 121 generates a base layer bitstream of video having a first quality level by encoding the video data at a first bit rate… first rate controller 20A may determine target numbers of bits and quantization parameters assigned to regions of a first layer according to a first target bit rate… The first rate controller 20A determines a first quantization parameter QP1 to be assigned to the first encoding processor 10A by adjusting quantization parameters assigned to the regions of the first layer according to the differences between target numbers of bits assigned to macroblocks according to the first target bit rate… second encoding processor 30A of the enhancement layer encoder 122A generates an enhancement layer bitstream having a second resolution by encoding the up-sampled frame data at a bit rate determined based on a quantization parameter QP2 received from the second rate controller 40A… The second rate controller 40A determines quantization parameters to be assigned to regions of a second layer. For example, the quantization parameters that are to be assigned to the regions of the second layer may be determined according to a second target bit rate; Par. [0153-162]: enhancement layer bitstream having a second resolution is generated by encoding frame data input to the enhancement layer encoder 122B at a bit rate determined based on a second quantization parameter QP2 supplied by the second rate controller 40B… The second rate controller 40B determines quantization parameters assigned to regions of the second layer. For example, the quantization parameters assigned to the regions of the second layer may be determined according to a second target bit rate… first encoding processor 10C of the base layer encoder 121C generates a base layer bitstream having a first quality level by encoding original video data at a bit rate determined based on a first quantization parameter QP1 received from the first rate controller 20C… The first rate controller 20C determines the first quantization parameter QP1 to be assigned to the first encoding processor 10C by adjusting quantization parameters assigned to regions of the base layer according to the difference between a target number of bits assigned to a macroblock according to a first target bit rate; quantization value to achieve a re-bit rate at a time of transmitting the re-encoded data, the re-bit rate being determined based on a bit rate at a time of transmitting the first encoded data and the second encoded data (e.g. technology of controlling bit rates that prevents degradation in image quality during multi-layered video encoding includes an encoder that generates a bitstream of video having a first quality level by encoding original video data at a bit rate determined (i.e. a bit rate at a time of transmitting the first encoded data and the second encoded data) based on a first quantization parameter (i.e. quantization value to achieve a re-bit rate at a time of transmitting the re-encoded data), for example, including a rate controller that determines a quantization parameter to be assigned to a first encoding processor by adjusting quantization parameters assigned to regions according to the difference between a target number of bits assigned to a macroblock according to a first target bit rate (i.e. quantization value to achieve a re-bit rate at a time of transmitting the re-encoded data, the re-bit rate being determined based on a bit rate at a time of transmitting the first encoded data and the second encoded data), as indicated above), for example).
Malakhov, HATTORI, and Jang are considered to be analogous art because they pertain to image processing techniques related to video coding applications Therefore, the combined teachings of Malakhov, HATTORI, and Jang, as a whole, would have rendered obvious the invention recited in claim 7 with a reasonable expectation of success in order to modify the apparatus for encoding samples of a video image including machine-learning-based models configured to detect an interesting object within the video image and classify input encoding samples of the video image as belonging to a region of interest, or belonging to a region of non-interest, encode each video frame of the video image by applying corresponding coding parameters, including video coding that comprises two parts, video encoding and video decoding, in which video encoding is performed at a source side and video decoding is performed at a destination side and typically comprises the inverse processing compared to the encoder to reconstruct the video pictures (as disclosed by Malakhov) with quantization value to achieve a re-bit rate at a time of transmitting the re-encoded data, the re-bit rate being determined based on a bit rate at a time of transmitting the first encoded data and the second encoded data (as taught by Jang, Abstract, Par. [0005-6, 128-135, 153-162]) to meet the need to compress video data for a variable bandwidth network, to prevent degradation in image quality during multi-layered video encoding, to provide a rate control method for multi-layered video coding, performed to control bit rates at an upper layer by using encoding statistical information and/or region-of-interest (ROI) information generated based on the result of performing encoding on a lower layer (Jang, Abstract, Par. [0003-5]).
Regarding claim 8, claim 3 is incorporated and the combination of Malakhov and HATTORI, as a whole, teaches the system (Malakhov, Par. [0007]), wherein
the transcoder (HATTORI, Par. [0107]: transcode controller 41 performs a process) calculates a correction coefficient that corrects the quantization value map (Malakhov, Par. [0098-101]: calculate a residual block 105 based on the picture block 103 and a prediction block 165… by subtracting sample values of the prediction block 165 from sample values of the picture block 103, sample by sample (pixel by pixel) to obtain the residual block 105 in the sample domain. Residual values (forming residual signal) correspond to prediction error (prediction error signal)… apply a transformation… on the sample values of the residual block 105 to obtain transformed coefficients 107 in a transform domain. The transformed coefficients 107 may also be referred to as transformed residual coefficients and represent the residual block 105 in the transform domain… quantize the transformed coefficients 107 to obtain quantized coefficients… by applying scalar quantization or vector quantization. The quantized coefficients 109 may also be referred to as quantized residual coefficients 109. For example, for scalar quantization, different scaling may be applied to achieve finer or coarser quantization. Smaller quantization step sizes correspond to finer quantization, whereas larger quantization step sizes correspond to coarser quantization. The applicable quantization step size may be indicated by a QP; Par. [0172-173]: processing circuitry of the apparatus 900 (marked by the dashed box) is configured to perform the tasks of the machine-learning based model 910, the analysis of motion by a motion analyzer 930 based on the motion information, and the adaption of the coding parameters including frame rate adaptation 940, picture size adaptation 950, and generation of a QP map by a QP map generator 960 for QP adaptation… The coding parameters picture size and quantization parameter QP refer to parameters of the spatial resolution; Par. [0231-239]: adapted coding parameter picture size is signaled by the module 1350 to the video encoder 1320… the coding parameter related to spatial resolution is a QP… The quantization parameter refers to the inverse number of bits used for encoding. Thus, a small QP value corresponds to a large number of bits, respectively, a large number of smaller increments. As a result, finer spatial variations within the video image may be encoded, improving image quality… the quantization parameter for a sample is set lower when the sample belongs to the ROI than in case the sample belongs to RONI… the adaption (i.e. the setting) of the quantization parameter is performed by the QP map generator 1360, which uses the importance map (i.e. the ROI-RONI map) as input. This map is provided as output by the machine-learning model 1310. The QP map generator uses this importance map to generate a corresponding map of the quantization parameters… Based on the importance (bit)map, the QP map generator 1360 generates a corresponding QP map, which contains the adapted QP value for a CTU/CU of the frame. This QP map is signaled by the module 1360 to the video encoder 1320; calculates a correction coefficient that corrects the quantization value map (e.g. apparatus for encoding samples of a video image includes processing circuitry configured to adapt coding parameters, including a quantization parameter QP (i.e. a quantization value), for example, based on a result of recognition processing that classifies input encoding samples of the video image as belonging to a region (i.e. area, section, portion, etc.) of interest including a region of interest ROI (i.e. the target area), for example, or belonging to a region of non-interest including a region of non-interest RONI (i.e. the non-target area), to generate an importance map (i.e. a ROI-RONI map), for example, and the adaption (i.e. adjustment, correction, etc.) of the quantization parameter is performed by a QP map generator (i.e. corrects the quantization value map), which uses the importance map (i.e. the ROI-RONI map) as input, for example, and applying a transformation on the sample values of a residual block to obtain transformed coefficients, also referred to as transformed residual coefficients (i.e. calculates a correction coefficient), and represent the residual block, for example, by quantizing the transformed coefficients to obtain quantized coefficients (i.e. calculates a correction coefficient that corrects the quantization value map), as indicated above), for example), multiplies the generated quantization value map by the correction coefficient to correct the quantization value map (Malakhov, Par. [0101]: quantization unit 108 is configured to quantize the transformed coefficients 107 to obtain quantized coefficients 109, e.g. by applying scalar quantization or vector quantization. The quantized coefficients 109 may also be referred to as quantized residual coefficients 109. For example, for scalar quantization, different scaling may be applied to achieve finer or coarser quantization… applicable quantization step size may be indicated by a QP… The quantization may include division by a quantization step size and corresponding or inverse dequantization, e.g. by inverse quantization 110, may include multiplication by the quantization step size; Par. [0172-173]: processing circuitry of the apparatus 900 (marked by the dashed box) is configured to perform the tasks of the machine-learning based model 910, the analysis of motion by a motion analyzer 930 based on the motion information, and the adaption of the coding parameters including frame rate adaptation 940, picture size adaptation 950, and generation of a QP map by a QP map generator 960 for QP adaptation… The coding parameters picture size and quantization parameter QP refer to parameters of the spatial resolution; multiplies the generated quantization value map by the correction coefficient to correct the quantization value map (e.g. apparatus for encoding samples of a video image includes processing circuitry configured to adapt coding parameters, including a quantization parameter QP (i.e. a quantization value), for example, including quantizing transformed coefficients, also referred to as quantized residual coefficients, that are obtained by applying scalar quantization or vector quantization to achieve finer or coarser quantization, for example, in which an applicable quantization step is indicated by a QP and the quantization include division by a quantization step size and corresponding or inverse dequantization, including multiplication by the quantization step size, for example, and generates a QP map for the QP adaptation (i.e. multiplies the generated quantization value map by the correction coefficient to correct the quantization value map), as indicated above), for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Malakhov and HATTORI, as a whole, teach the apparatus, as indicated above, but fail to teach the following as further recited in claim 8.
However, Jang teaches correction coefficient corrects based on a bit rate at a time of transmitting the first encoded data and the second encoded data and a re-bit rate at a time of transmitting the re-encoded data (Par. [0005-6]: inventive concept provides a technology of controlling bit rates that prevents degradation in image quality during multi-layered video encoding… provides a rate control method for multi-layered video coding, performed to control bit rates at an upper layer by using encoding statistical information and/or region-of-interest (ROI) information generated based on the result of performing encoding on a lower layer. A second rate controller generates a plurality of second quantization parameters based on the encoding statistical information, to be used when encoding is performed on a plurality of regions at the second (upper) layer… a video encoder for controlling bit rates at an upper layer by using encoding statistical information and/or ROI information generated based on the result of performing encoding on a lower layer; Par. [0099-103]: Transformation is performed to transform a set of pixel difference values into residual transformation coefficients representing the energies of the pixel difference values in a frequency domain…Each of the base layer encoder 121 and the enhancement layer encoder 122 quantize residual transformation coefficients by using a quantization parameter QP… Quality scalability may be realized through residual quantization. For example, the base layer encoder 121 that encodes video data to have a lowest quality level quantizes coefficients; Par. [0128-135]: base layer encoder 121 generates a base layer bitstream of video having a first quality level by encoding the video data at a first bit rate… first rate controller 20A may determine target numbers of bits and quantization parameters assigned to regions of a first layer according to a first target bit rate… The first rate controller 20A determines a first quantization parameter QP1 to be assigned to the first encoding processor 10A by adjusting quantization parameters assigned to the regions of the first layer according to the differences between target numbers of bits assigned to macroblocks according to the first target bit rate… second encoding processor 30A of the enhancement layer encoder 122A generates an enhancement layer bitstream having a second resolution by encoding the up-sampled frame data at a bit rate determined based on a quantization parameter QP2 received from the second rate controller 40A… The second rate controller 40A determines quantization parameters to be assigned to regions of a second layer. For example, the quantization parameters that are to be assigned to the regions of the second layer may be determined according to a second target bit rate; Par. [0153-162]: enhancement layer bitstream having a second resolution is generated by encoding frame data input to the enhancement layer encoder 122B at a bit rate determined based on a second quantization parameter QP2 supplied by the second rate controller 40B… The second rate controller 40B determines quantization parameters assigned to regions of the second layer. For example, the quantization parameters assigned to the regions of the second layer may be determined according to a second target bit rate… first encoding processor 10C of the base layer encoder 121C generates a base layer bitstream having a first quality level by encoding original video data at a bit rate determined based on a first quantization parameter QP1 received from the first rate controller 20C… The first rate controller 20C determines the first quantization parameter QP1 to be assigned to the first encoding processor 10C by adjusting quantization parameters assigned to regions of the base layer according to the difference between a target number of bits assigned to a macroblock according to a first target bit rate; correction coefficient corrects based on a bit rate at a time of transmitting the first encoded data and the second encoded data and a re-bit rate at a time of transmitting the re-encoded data (e.g. technology of controlling bit rates that prevents degradation in image quality during multi-layered video encoding includes an encoder that generates a bitstream of video having a first quality level by encoding original video data at a bit rate determined (i.e. a bit rate at a time of transmitting the first encoded data and the second encoded data) based on a first quantization parameter (i.e. quantization value to achieve a re-bit rate at a time of transmitting the re-encoded data), including quantized residual transformation coefficients (i.e. correction coefficients) by using a quantization parameter QP, for example, including a rate controller that determines a quantization parameter to be assigned to a first encoding processor by adjusting quantization parameters assigned to regions according to the difference between a target number of bits assigned to a macroblock according to a first target bit rate (i.e. correction coefficient corrects based on a bit rate at a time of transmitting the first encoded data and the second encoded data and a re-bit rate at a time of transmitting the re-encoded data), as indicated above), for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 7.
Regarding claim 13, claim 12 is incorporated and the combination of Malakhov and HATTORI, as a whole, teaches the system (Malakhov, Par. [0007]), wherein
the transcoder (HATTORI, Par. [0107]: transcode controller 41 performs a process) generates the quantization value map (Malakhov, Par. [0172]: processing circuitry of the apparatus 900 (marked by the dashed box) is configured to perform the tasks of the machine-learning based model 910, the analysis of motion by a motion analyzer 930 based on the motion information, and the adaption of the coding parameters including frame rate adaptation 940, picture size adaptation 950, and generation of a QP map by a QP map generator 960 for QP adaptation; Par. [0231-242]: adapted coding parameter picture size is signaled by the module 1350 to the video encoder 1320… the coding parameter related to spatial resolution is a QP… The quantization parameter refers to the inverse number of bits used for encoding. Thus, a small QP value corresponds to a large number of bits, respectively, a large number of smaller increments. As a result, finer spatial variations within the video image may be encoded, improving image quality… the quantization parameter for a sample is set lower when the sample belongs to the ROI than in case the sample belongs to RONI… the adaption (i.e. the setting) of the quantization parameter is performed by the QP map generator 1360, which uses the importance map (i.e. the ROI-RONI map) as input. This map is provided as output by the machine-learning model 1310. The QP map generator uses this importance map to generate a corresponding map of the quantization parameters… method is provided for encoding samples of a video image comprising the steps of determining motion information for samples of a block in the video image, determining a coding parameter based on an output of a machine-learning-based model, to which the motion information and the samples of the block are input, and encoding the video frame by applying the coding parameter; generates the quantization value map (e.g. apparatus for encoding samples of a video image includes processing circuitry configured to adapt coding parameters, including a quantization parameter QP (i.e. a quantization value), for example, based on a result of recognition processing that classifies input encoding samples of the video image as belonging to a region (i.e. area, section, portion, etc.) of interest including a region of interest ROI (i.e. the target area), for example, or belonging to a region of non-interest including a region of non-interest RONI (i.e. the non-target area), to generate an importance map (i.e. a ROI-RONI map), for example, and the adaption (i.e. adjustment) of the quantization parameter is performed by a QP map (i.e. generates the quantization value map) generator, as indicated above, for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Malakhov and HATTORI, as a whole, teach the apparatus, as indicated above, but fail to teach the following as further recited in claim 13.
However, Jang teaches a relationship between the value that represents the attribute of the reconstructed image data and a re-bit rate at a time of transmitting the re-encoded data is determined in advance for each different quantization value, and by referring to the relationship and deriving the quantization value at which the re-bit rate that corresponds to the value that represents the attribute of the reconstructed image data becomes a target bit rate (Par. [0005-6]: inventive concept provides a technology of controlling bit rates that prevents degradation in image quality during multi-layered video encoding… provides a rate control method for multi-layered video coding, performed to control bit rates at an upper layer by using encoding statistical information and/or region-of-interest (ROI) information generated based on the result of performing encoding on a lower layer. A second rate controller generates a plurality of second quantization parameters based on the encoding statistical information, to be used when encoding is performed on a plurality of regions at the second (upper) layer… a video encoder for controlling bit rates at an upper layer by using encoding statistical information and/or ROI information generated based on the result of performing encoding on a lower layer; Par. [0128-135]: base layer encoder 121 generates a base layer bitstream of video having a first quality level by encoding the video data at a first bit rate… first rate controller 20A may determine target numbers of bits and quantization parameters assigned to regions of a first layer according to a first target bit rate… The first rate controller 20A determines a first quantization parameter QP1 to be assigned to the first encoding processor 10A by adjusting quantization parameters assigned to the regions of the first layer according to the differences between target numbers of bits assigned to macroblocks according to the first target bit rate… second encoding processor 30A of the enhancement layer encoder 122A generates an enhancement layer bitstream having a second resolution by encoding the up-sampled frame data at a bit rate determined based on a quantization parameter QP2 received from the second rate controller 40A… The second rate controller 40A determines quantization parameters to be assigned to regions of a second layer. For example, the quantization parameters that are to be assigned to the regions of the second layer may be determined according to a second target bit rate; Par. [0153-162]: enhancement layer bitstream having a second resolution is generated by encoding frame data input to the enhancement layer encoder 122B at a bit rate determined based on a second quantization parameter QP2 supplied by the second rate controller 40B… The second rate controller 40B determines quantization parameters assigned to regions of the second layer. For example, the quantization parameters assigned to the regions of the second layer may be determined according to a second target bit rate… first encoding processor 10C of the base layer encoder 121C generates a base layer bitstream having a first quality level by encoding original video data at a bit rate determined based on a first quantization parameter QP1 received from the first rate controller 20C… The first rate controller 20C determines the first quantization parameter QP1 to be assigned to the first encoding processor 10C by adjusting quantization parameters assigned to regions of the base layer according to the difference between a target number of bits assigned to a macroblock according to a first target bit rate; a relationship between the value that represents the attribute of the reconstructed image data and a re-bit rate at a time of transmitting the re-encoded data is determined in advance for each different quantization value, and by referring to the relationship and deriving the quantization value at which the re-bit rate that corresponds to the value that represents the attribute of the reconstructed image data becomes a target bit rate (e.g. technology of controlling bit rates that prevents degradation in image quality during multi-layered video encoding includes an encoder that generates a bitstream of video having a first quality level (i.e. attribute of the reconstructed image data) by encoding original video data at a bit rate determined based on a first quantization parameter (i.e. a relationship between the value that represents the attribute of the reconstructed image data and a re-bit rate at a time of transmitting the re-encoded data is determined in advance for each different quantization value, and by referring to the relationship), for example, including a rate controller that determines a quantization parameter to be assigned to a first encoding processor by adjusting quantization parameters assigned to regions according to the difference between a target number of bits assigned to a macroblock according to a first target bit rate (i.e. and deriving the quantization value at which the re-bit rate that corresponds to the value that represents the attribute of the reconstructed image data becomes a target bit rate), as indicated above), for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 7.
Regarding claim 17, claim 3 is incorporated and the combination of Malakhov and HATTORI, as a whole, teaches the system (Malakhov, Par. [0007]), wherein
the transcoder (HATTORI, Par. [0107]: transcode controller 41 performs a process) calculates a correction coefficient that corrects the quantization value map (Malakhov, Par. [0098-101]: calculate a residual block 105 based on the picture block 103 and a prediction block 165… by subtracting sample values of the prediction block 165 from sample values of the picture block 103, sample by sample (pixel by pixel) to obtain the residual block 105 in the sample domain. Residual values (forming residual signal) correspond to prediction error (prediction error signal)… apply a transformation… on the sample values of the residual block 105 to obtain transformed coefficients 107 in a transform domain. The transformed coefficients 107 may also be referred to as transformed residual coefficients and represent the residual block 105 in the transform domain… quantize the transformed coefficients 107 to obtain quantized coefficients… by applying scalar quantization or vector quantization. The quantized coefficients 109 may also be referred to as quantized residual coefficients 109. For example, for scalar quantization, different scaling may be applied to achieve finer or coarser quantization. Smaller quantization step sizes correspond to finer quantization, whereas larger quantization step sizes correspond to coarser quantization. The applicable quantization step size may be indicated by a QP; Par. [0172-173]: processing circuitry of the apparatus 900 (marked by the dashed box) is configured to perform the tasks of the machine-learning based model 910, the analysis of motion by a motion analyzer 930 based on the motion information, and the adaption of the coding parameters including frame rate adaptation 940, picture size adaptation 950, and generation of a QP map by a QP map generator 960 for QP adaptation… The coding parameters picture size and quantization parameter QP refer to parameters of the spatial resolution; Par. [0231-239]: adapted coding parameter picture size is signaled by the module 1350 to the video encoder 1320… the coding parameter related to spatial resolution is a QP… The quantization parameter refers to the inverse number of bits used for encoding. Thus, a small QP value corresponds to a large number of bits, respectively, a large number of smaller increments. As a result, finer spatial variations within the video image may be encoded, improving image quality… the quantization parameter for a sample is set lower when the sample belongs to the ROI than in case the sample belongs to RONI… the adaption (i.e. the setting) of the quantization parameter is performed by the QP map generator 1360, which uses the importance map (i.e. the ROI-RONI map) as input. This map is provided as output by the machine-learning model 1310. The QP map generator uses this importance map to generate a corresponding map of the quantization parameters… Based on the importance (bit)map, the QP map generator 1360 generates a corresponding QP map, which contains the adapted QP value for a CTU/CU of the frame. This QP map is signaled by the module 1360 to the video encoder 1320; calculates a correction coefficient that corrects the quantization value map (e.g. apparatus for encoding samples of a video image includes processing circuitry configured to adapt coding parameters, including a quantization parameter QP (i.e. a quantization value), for example, based on a result of recognition processing that classifies input encoding samples of the video image as belonging to a region (i.e. area, section, portion, etc.) of interest including a region of interest ROI (i.e. the target area), for example, or belonging to a region of non-interest including a region of non-interest RONI (i.e. the non-target area), to generate an importance map (i.e. a ROI-RONI map), for example, and the adaption (i.e. adjustment, correction, etc.) of the quantization parameter is performed by a QP map generator (i.e. corrects the quantization value map), which uses the importance map (i.e. the ROI-RONI map) as input, for example, and applying a transformation on the sample values of a residual block to obtain transformed coefficients, also referred to as transformed residual coefficients (i.e. calculates a correction coefficient that corrects the quantization value map), as indicated above), for example), and multiplies the generated quantization value map by the correction coefficient to correct the quantization value map (Malakhov, Par. [0101]: quantization unit 108 is configured to quantize the transformed coefficients 107 to obtain quantized coefficients 109, e.g. by applying scalar quantization or vector quantization. The quantized coefficients 109 may also be referred to as quantized residual coefficients 109. For example, for scalar quantization, different scaling may be applied to achieve finer or coarser quantization… applicable quantization step size may be indicated by a QP… The quantization may include division by a quantization step size and corresponding or inverse dequantization, e.g. by inverse quantization 110, may include multiplication by the quantization step size; Par. [0172-173]: processing circuitry of the apparatus 900 (marked by the dashed box) is configured to perform the tasks of the machine-learning based model 910, the analysis of motion by a motion analyzer 930 based on the motion information, and the adaption of the coding parameters including frame rate adaptation 940, picture size adaptation 950, and generation of a QP map by a QP map generator 960 for QP adaptation… The coding parameters picture size and quantization parameter QP refer to parameters of the spatial resolution; multiplies the generated quantization value map by the correction coefficient to correct the quantization value map (e.g. apparatus for encoding samples of a video image includes processing circuitry configured to adapt coding parameters, including a quantization parameter QP (i.e. a quantization value), for example, including quantizing transformed coefficients, also referred to as quantized residual coefficients, that are obtained by applying scalar quantization or vector quantization to achieve finer or coarser quantization, for example, in which an applicable quantization step is indicated by a QP and the quantization include division by a quantization step size and corresponding or inverse dequantization, including multiplication by the quantization step size, for example, and generates a QP map for the QP adaptation (i.e. multiplies the generated quantization value map by the correction coefficient to correct the quantization value map), as indicated above), for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Malakhov and HATTORI, as a whole, teach the apparatus, as indicated above, but fail to teach the following as further recited in claim 17.
However, Jang teaches based on ratio between a re-bit rate at a time of transmitting the re-encoded data and a specified bit rate (Par. [0005-6]: inventive concept provides a technology of controlling bit rates that prevents degradation in image quality during multi-layered video encoding… provides a rate control method for multi-layered video coding, performed to control bit rates at an upper layer by using encoding statistical information and/or region-of-interest (ROI) information generated based on the result of performing encoding on a lower layer. A second rate controller generates a plurality of second quantization parameters based on the encoding statistical information, to be used when encoding is performed on a plurality of regions at the second (upper) layer… a video encoder for controlling bit rates at an upper layer by using encoding statistical information and/or ROI information generated based on the result of performing encoding on a lower layer; Par. [0099-103]: Transformation is performed to transform a set of pixel difference values into residual transformation coefficients representing the energies of the pixel difference values in a frequency domain…Each of the base layer encoder 121 and the enhancement layer encoder 122 quantize residual transformation coefficients by using a quantization parameter QP… Quality scalability may be realized through residual quantization. For example, the base layer encoder 121 that encodes video data to have a lowest quality level quantizes coefficients; Par. [0128-135]: base layer encoder 121 generates a base layer bitstream of video having a first quality level by encoding the video data at a first bit rate… first rate controller 20A may determine target numbers of bits and quantization parameters assigned to regions of a first layer according to a first target bit rate… The first rate controller 20A determines a first quantization parameter QP1 to be assigned to the first encoding processor 10A by adjusting quantization parameters assigned to the regions of the first layer according to the differences between target numbers of bits assigned to macroblocks according to the first target bit rate… second encoding processor 30A of the enhancement layer encoder 122A generates an enhancement layer bitstream having a second resolution by encoding the up-sampled frame data at a bit rate determined based on a quantization parameter QP2 received from the second rate controller 40A… The second rate controller 40A determines quantization parameters to be assigned to regions of a second layer. For example, the quantization parameters that are to be assigned to the regions of the second layer may be determined according to a second target bit rate; Par. [0153-162]: enhancement layer bitstream having a second resolution is generated by encoding frame data input to the enhancement layer encoder 122B at a bit rate determined based on a second quantization parameter QP2 supplied by the second rate controller 40B… The second rate controller 40B determines quantization parameters assigned to regions of the second layer. For example, the quantization parameters assigned to the regions of the second layer may be determined according to a second target bit rate… first encoding processor 10C of the base layer encoder 121C generates a base layer bitstream having a first quality level by encoding original video data at a bit rate determined based on a first quantization parameter QP1 received from the first rate controller 20C… The first rate controller 20C determines the first quantization parameter QP1 to be assigned to the first encoding processor 10C by adjusting quantization parameters assigned to regions of the base layer according to the difference between a target number of bits assigned to a macroblock according to a first target bit rate; based on ratio between a re-bit rate at a time of transmitting the re-encoded data and a specified bit rate (e.g. technology of controlling bit rates that prevents degradation in image quality during multi-layered video encoding includes an encoder that generates a bitstream of video having a first quality level by encoding original video data at a bit rate determined based on a first quantization parameter, including quantized residual transformation coefficients (i.e. correction coefficients) by using a quantization parameter QP, for example, including a rate controller that determines a quantization parameter to be assigned to a first encoding processor by adjusting quantization parameters assigned to regions according to (i.e. based on) the difference between a target number of bits assigned to a macroblock according to a first target (i.e. specific) bit rate (i.e. based on ratio between a re-bit rate at a time of transmitting the re-encoded data and a specified bit rate), as indicated above), for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 7.
Claims 14 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Malakhov, in view of HATTORI, as applied to claim 1 above, in further view of KAWAI et al. (US PG Publication No. 2020/0021832 A1), hereafter referred to as KAWAI.
Regarding claim 14, claim 3 is incorporated and the combination of Malakhov and HATTORI, as a whole, teaches the system (Malakhov, Par. [0007]), wherein
the transcoder (HATTORI, Par. [0107]: transcode controller 41 performs a process) calculates a correction coefficient that corrects the quantization value map (Malakhov, Par. [0098-101]: calculate a residual block 105 based on the picture block 103 and a prediction block 165… by subtracting sample values of the prediction block 165 from sample values of the picture block 103, sample by sample (pixel by pixel) to obtain the residual block 105 in the sample domain. Residual values (forming residual signal) correspond to prediction error (prediction error signal)… apply a transformation… on the sample values of the residual block 105 to obtain transformed coefficients 107 in a transform domain. The transformed coefficients 107 may also be referred to as transformed residual coefficients and represent the residual block 105 in the transform domain… quantize the transformed coefficients 107 to obtain quantized coefficients… by applying scalar quantization or vector quantization. The quantized coefficients 109 may also be referred to as quantized residual coefficients 109. For example, for scalar quantization, different scaling may be applied to achieve finer or coarser quantization. Smaller quantization step sizes correspond to finer quantization, whereas larger quantization step sizes correspond to coarser quantization. The applicable quantization step size may be indicated by a QP; Par. [0172-173]: processing circuitry of the apparatus 900 (marked by the dashed box) is configured to perform the tasks of the machine-learning based model 910, the analysis of motion by a motion analyzer 930 based on the motion information, and the adaption of the coding parameters including frame rate adaptation 940, picture size adaptation 950, and generation of a QP map by a QP map generator 960 for QP adaptation… The coding parameters picture size and quantization parameter QP refer to parameters of the spatial resolution; Par. [0231-239]: adapted coding parameter picture size is signaled by the module 1350 to the video encoder 1320… the coding parameter related to spatial resolution is a QP… The quantization parameter refers to the inverse number of bits used for encoding. Thus, a small QP value corresponds to a large number of bits, respectively, a large number of smaller increments. As a result, finer spatial variations within the video image may be encoded, improving image quality… the quantization parameter for a sample is set lower when the sample belongs to the ROI than in case the sample belongs to RONI… the adaption (i.e. the setting) of the quantization parameter is performed by the QP map generator 1360, which uses the importance map (i.e. the ROI-RONI map) as input. This map is provided as output by the machine-learning model 1310. The QP map generator uses this importance map to generate a corresponding map of the quantization parameters… Based on the importance (bit)map, the QP map generator 1360 generates a corresponding QP map, which contains the adapted QP value for a CTU/CU of the frame. This QP map is signaled by the module 1360 to the video encoder 1320; calculates a correction coefficient that corrects the quantization value map (e.g. apparatus for encoding samples of a video image includes processing circuitry configured to adapt coding parameters, including a quantization parameter QP (i.e. a quantization value), for example, based on a result of recognition processing that classifies input encoding samples of the video image as belonging to a region (i.e. area, section, portion, etc.) of interest including a region of interest ROI (i.e. the target area), for example, or belonging to a region of non-interest including a region of non-interest RONI (i.e. the non-target area), to generate an importance map (i.e. a ROI-RONI map), for example, and the adaption (i.e. adjustment, correction, etc.) of the quantization parameter is performed by a QP map generator (i.e. corrects the quantization value map), which uses the importance map (i.e. the ROI-RONI map) as input, for example, and applying a transformation on the sample values of a residual block to obtain transformed coefficients, also referred to as transformed residual coefficients (i.e. calculates a correction coefficient that corrects the quantization value map), as indicated above), for example), and multiplies the generated quantization value map by the correction coefficient to correct the quantization value map (Malakhov, Par. [0101]: quantization unit 108 is configured to quantize the transformed coefficients 107 to obtain quantized coefficients 109, e.g. by applying scalar quantization or vector quantization. The quantized coefficients 109 may also be referred to as quantized residual coefficients 109. For example, for scalar quantization, different scaling may be applied to achieve finer or coarser quantization… applicable quantization step size may be indicated by a QP… The quantization may include division by a quantization step size and corresponding or inverse dequantization, e.g. by inverse quantization 110, may include multiplication by the quantization step size; Par. [0172-173]: processing circuitry of the apparatus 900 (marked by the dashed box) is configured to perform the tasks of the machine-learning based model 910, the analysis of motion by a motion analyzer 930 based on the motion information, and the adaption of the coding parameters including frame rate adaptation 940, picture size adaptation 950, and generation of a QP map by a QP map generator 960 for QP adaptation… The coding parameters picture size and quantization parameter QP refer to parameters of the spatial resolution; and multiplies the generated quantization value map by the correction coefficient to correct the quantization value map (e.g. apparatus for encoding samples of a video image includes processing circuitry configured to adapt coding parameters, including a quantization parameter QP (i.e. a quantization value), for example, including quantizing transformed coefficients, also referred to as quantized residual coefficients, that are obtained by applying scalar quantization or vector quantization to achieve finer or coarser quantization, for example, in which an applicable quantization step is indicated by a QP and the quantization include division by a quantization step size and corresponding or inverse dequantization, including multiplication by the quantization step size, for example, and generates a QP map for the QP adaptation (i.e. multiplies the generated quantization value map by the correction coefficient to correct the quantization value map), as indicated above), for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Malakhov and HATTORI, as a whole, teach the apparatus, as indicated above, but fail to teach the following as further recited in claim 14.
However, KAWAI teaches based on a change in a peak signal-to-noise ratio (PSNR) calculated for re-decoded data obtained by re-decoding the re-encoded data (Par. [0412-431]: tap coefficient learning is performed using an image during decoding as student data and using an original image corresponding to the image during decoding as teacher data, and tap coefficients obtained by the tap coefficient learning are supplied as initial coefficients form the coefficient calculation section 154 to the reduction apparatus 132 (FIG. 10)… The reduction number determination section 161 determines a plurality of candidates for a reduction number P, for example, in response to the quantization parameter QP, a bit rate of encoded data, an image feature amount of an image during decoding, an image feature amount of an original image and so forth, and supplies the candidates to the coefficient reduction section 162 … coefficient reduction section 162 performs, determining each class except nonconforming classes from among all classes of initial coefficients as a target class that is a target of main component analysis, main component analysis of tap coefficients of the target glass to determine, for each of the plurality of candidates for a reduction number, a reduction coefficient W′(c) for each target class and a transform coefficient A′.sup.−1 common to all target classes… The nonconforming class detection section 164 detects a class in which the PSNR (Peak signal-to-noise ratio) of a post-filter image as a second image obtained by the classification adaptive process as a filter process that uses the reconstruction tap coefficients w′(c,n) for each target class is reduced significantly (class whose PSNR is equal to or lower than a threshold value) as a nonconforming class. Then, the nonconforming class detection section 164 deletes the reduction coefficient of the nonconforming class from the adopted reduction coefficients W′(c); based on a change in a peak signal-to-noise ratio (PSNR) calculated for re-decoded data obtained by re-decoding the re-encoded data (e.g. adaptive process as a filter process uses reconstruction tap coefficients for each target class by obtaining a nonconforming class that detects a class in which a PSNR (Peak signal-to-noise ratio) of a post-filter image as a second image obtained by the classification adaptive process as a filter process that uses the reconstruction tap coefficients (i.e. based on a change in a peak signal-to-noise ratio (PSNR) calculated), for example, in which tap coefficient learning is performed using an image during decoding as student data and using an original image corresponding to the image during decoding as teacher data, and in response to the quantization parameter QP, a bit rate of encoded data, an image feature amount of an image during decoding (i.e. calculated for re-decoded data obtained by re-decoding the re-encoded data), as indicated above), for example).
Malakhov, HATTORI, and KAWAI are considered to be analogous art because they pertain to image processing techniques related to video coding applications Therefore, the combined teachings of Malakhov, HATTORI, and KAWAI, as a whole, would have rendered obvious the invention recited in claim 14 with a reasonable expectation of success in order to modify the apparatus for encoding samples of a video image including machine-learning-based models configured to detect an interesting object within the video image and classify input encoding samples of the video image as belonging to a region of interest, or belonging to a region of non-interest, encode each video frame of the video image by applying corresponding coding parameters, including video coding that comprises two parts, video encoding and video decoding, in which video encoding is performed at a source side and video decoding is performed at a destination side and typically comprises the inverse processing compared to the encoder to reconstruct the video pictures (as disclosed by Malakhov) with based on a change in a peak signal-to-noise ratio (PSNR) calculated for re-decoded data obtained by re-decoding the re-encoded data (as taught by KAWAI, Abstract, Par. [0412-431]) to appropriately improve the compression efficiency of an image (KAWAI, Abstract, Par. [0001-3, 6-8]).
Regarding claim 16, claim 3 is incorporated and the combination of Malakhov and HATTORI, as a whole, teaches the system (Malakhov, Par. [0007]), wherein
the transcoder (HATTORI, Par. [0107]: transcode controller 41 performs a process) calculates a correction coefficient that corrects the quantization value map (Malakhov, Par. [0098-101]: calculate a residual block 105 based on the picture block 103 and a prediction block 165… by subtracting sample values of the prediction block 165 from sample values of the picture block 103, sample by sample (pixel by pixel) to obtain the residual block 105 in the sample domain. Residual values (forming residual signal) correspond to prediction error (prediction error signal)… apply a transformation… on the sample values of the residual block 105 to obtain transformed coefficients 107 in a transform domain. The transformed coefficients 107 may also be referred to as transformed residual coefficients and represent the residual block 105 in the transform domain… quantize the transformed coefficients 107 to obtain quantized coefficients… by applying scalar quantization or vector quantization. The quantized coefficients 109 may also be referred to as quantized residual coefficients 109. For example, for scalar quantization, different scaling may be applied to achieve finer or coarser quantization. Smaller quantization step sizes correspond to finer quantization, whereas larger quantization step sizes correspond to coarser quantization. The applicable quantization step size may be indicated by a QP; Par. [0172-173]: processing circuitry of the apparatus 900 (marked by the dashed box) is configured to perform the tasks of the machine-learning based model 910, the analysis of motion by a motion analyzer 930 based on the motion information, and the adaption of the coding parameters including frame rate adaptation 940, picture size adaptation 950, and generation of a QP map by a QP map generator 960 for QP adaptation… The coding parameters picture size and quantization parameter QP refer to parameters of the spatial resolution; Par. [0231-239]: adapted coding parameter picture size is signaled by the module 1350 to the video encoder 1320… the coding parameter related to spatial resolution is a QP… The quantization parameter refers to the inverse number of bits used for encoding. Thus, a small QP value corresponds to a large number of bits, respectively, a large number of smaller increments. As a result, finer spatial variations within the video image may be encoded, improving image quality… the quantization parameter for a sample is set lower when the sample belongs to the ROI than in case the sample belongs to RONI… the adaption (i.e. the setting) of the quantization parameter is performed by the QP map generator 1360, which uses the importance map (i.e. the ROI-RONI map) as input. This map is provided as output by the machine-learning model 1310. The QP map generator uses this importance map to generate a corresponding map of the quantization parameters… Based on the importance (bit)map, the QP map generator 1360 generates a corresponding QP map, which contains the adapted QP value for a CTU/CU of the frame. This QP map is signaled by the module 1360 to the video encoder 1320; calculates a correction coefficient that corrects the quantization value map (e.g. apparatus for encoding samples of a video image includes processing circuitry configured to adapt coding parameters, including a quantization parameter QP (i.e. a quantization value), for example, based on a result of recognition processing that classifies input encoding samples of the video image as belonging to a region (i.e. area, section, portion, etc.) of interest including a region of interest ROI (i.e. the target area), for example, or belonging to a region of non-interest including a region of non-interest RONI (i.e. the non-target area), to generate an importance map (i.e. a ROI-RONI map), for example, and the adaption (i.e. adjustment, correction, etc.) of the quantization parameter is performed by a QP map generator (i.e. corrects the quantization value map), which uses the importance map (i.e. the ROI-RONI map) as input, for example, and applying a transformation on the sample values of a residual block to obtain transformed coefficients, also referred to as transformed residual coefficients (i.e. calculates a correction coefficient that corrects the quantization value map), as indicated above), for example), and multiplies the generated quantization value map by the correction coefficient to correct the quantization value map (Malakhov, Par. [0101]: quantization unit 108 is configured to quantize the transformed coefficients 107 to obtain quantized coefficients 109, e.g. by applying scalar quantization or vector quantization. The quantized coefficients 109 may also be referred to as quantized residual coefficients 109. For example, for scalar quantization, different scaling may be applied to achieve finer or coarser quantization… applicable quantization step size may be indicated by a QP… The quantization may include division by a quantization step size and corresponding or inverse dequantization, e.g. by inverse quantization 110, may include multiplication by the quantization step size; Par. [0172-173]: processing circuitry of the apparatus 900 (marked by the dashed box) is configured to perform the tasks of the machine-learning based model 910, the analysis of motion by a motion analyzer 930 based on the motion information, and the adaption of the coding parameters including frame rate adaptation 940, picture size adaptation 950, and generation of a QP map by a QP map generator 960 for QP adaptation… The coding parameters picture size and quantization parameter QP refer to parameters of the spatial resolution; and multiplies the generated quantization value map by the correction coefficient to correct the quantization value map (e.g. apparatus for encoding samples of a video image includes processing circuitry configured to adapt coding parameters, including a quantization parameter QP (i.e. a quantization value), for example, including quantizing transformed coefficients, also referred to as quantized residual coefficients, that are obtained by applying scalar quantization or vector quantization to achieve finer or coarser quantization, for example, in which an applicable quantization step is indicated by a QP and the quantization include division by a quantization step size and corresponding or inverse dequantization, including multiplication by the quantization step size, for example, and generates a QP map for the QP adaptation (i.e. multiplies the generated quantization value map by the correction coefficient to correct the quantization value map), as indicated above), for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Malakhov and HATTORI, as a whole, teach the apparatus, as indicated above, but fail to teach the following as further recited in claim 16.
However, KAWAI teaches based on a ratio between a PSNR calculated for re-decoded data obtained by re-decoding the re-encoded data and a specified PSNR (Par. [0412-431]: tap coefficient learning is performed using an image during decoding as student data and using an original image corresponding to the image during decoding as teacher data, and tap coefficients obtained by the tap coefficient learning are supplied as initial coefficients form the coefficient calculation section 154 to the reduction apparatus 132 (FIG. 10)… The reduction number determination section 161 determines a plurality of candidates for a reduction number P, for example, in response to the quantization parameter QP, a bit rate of encoded data, an image feature amount of an image during decoding, an image feature amount of an original image and so forth, and supplies the candidates to the coefficient reduction section 162 … coefficient reduction section 162 performs, determining each class except nonconforming classes from among all classes of initial coefficients as a target class that is a target of main component analysis, main component analysis of tap coefficients of the target glass to determine, for each of the plurality of candidates for a reduction number, a reduction coefficient W′(c) for each target class and a transform coefficient A′.sup.−1 common to all target classes… The nonconforming class detection section 164 detects a class in which the PSNR (Peak signal-to-noise ratio) of a post-filter image as a second image obtained by the classification adaptive process as a filter process that uses the reconstruction tap coefficients w′(c,n) for each target class is reduced significantly (class whose PSNR is equal to or lower than a threshold value) as a nonconforming class. Then, the nonconforming class detection section 164 deletes the reduction coefficient of the nonconforming class from the adopted reduction coefficients W′(c); based on a ratio between a PSNR calculated for re-decoded data obtained by re-decoding the re-encoded data and a specified PSNR (e.g. adaptive process as a filter process uses reconstruction tap coefficients for each target class by obtaining a nonconforming class that detects a class in which a PSNR (Peak signal-to-noise ratio) of a post-filter image as a second image obtained by the classification adaptive process as a filter process that uses the reconstruction tap coefficients (i.e. based on a change in a peak signal-to-noise ratio (PSNR) calculated and a specified PSNR), for example, in which tap coefficient learning is performed using an image during decoding as student data and using an original image corresponding to the image during decoding as teacher data, and in response to the quantization parameter QP, a bit rate of encoded data, an image feature amount of an image during decoding (i.e. c calculated for re-decoded data obtained by re-decoding the re-encoded data), as indicated above), for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 14.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GUILLERMO M RIVERA-MARTINEZ whose telephone number is (571) 272-4979. The examiner can normally be reached on 9 am to 5 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached on 571-270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GUILLERMO M RIVERA-MARTINEZ/ Primary Examiner, Art Unit 2677
Prosecution Insights