DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Notice to Applicants
2. This communication is in response to the application filled on 10/22/2024.
3. Claims 1-20 are pending.
4. Limitations appearing inside {} are intended to indicate the limitations not taught by said prior art(s)/combinations.
Information Disclosure Statement
5. The information disclosure statements (IDS) submitted on 10/22/2024, 04/09/2025, 01/27/2026 have been considered by the examiner.
Claim Objections
6. Claim 2 and 9 are objected to because of the following informalities:
In ln. 6, the Claim 2 recites “…etermined number…”, consider correcting to “…predetermined number…”. Appropriate correction is required.
Regarding Claims 9, a series of singular dependent claims is permissible in which a dependent claim refers to a preceding claim which, in turn, refers to another preceding claim.
A claim which depends from a dependent claim should not be separated by any claim which does not also depend from said dependent claim. It should be kept in mind that a dependent claim may refer to any preceding independent claim. In general, applicant's sequence will not be changed. See MPEP § 608.01(n). Claims 6-8 are not dependent on claim 3 and separate claim 9 form claim 3.
Claim Rejections - 35 USC § 112
7. The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
8. Claim 19 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
9. Claim 19 recites the limitation "…the pre-stored first reference model” and “…the second reference model trained based on the first model…" in ln. 4-5. There is insufficient antecedent basis for this limitation in the claim. Specifically, there is not a reference to a first reference model or a trained second reference model in claim 12. The examiner recommends changing the dependency of claim 19 to be dependent on claim 14, which would obviate the antecedent issues.
Claim Rejections - 35 USC § 103
10. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
11. Claims 1-2, 6-7, 12-13, and 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over “Higher Quality Live Streaming under Lower Uplink Bandwidth: An approach of Super-Resolution Based Video Coding” to Chen et al. (hereinafter Chen), and further in view of “Delay-Sensitive and Power-Efficient Quality Control of Dynamic Video Streaming using Adaptive Super-Resolution” to Choi et al. (hereinafter Choi).
12. Regarding Claim 1, Choi discloses an image processing device comprising:
memory storing one or more instructions; and
one or more processors including processing circuitry, operatively coupled to the memory,
wherein the one or more instructions, when executed by the one or more processors individually or collectively, cause the image processing device to ([pg. 77, col. 1, par. 4, ln. 12 to col. 2, par. 1, ln. 2] “…On the broadcaster side, we use an ordinary PC with one CPU (Intel Core i7) for key frame compression, and on the cloud server, we use a 2080Ti GPU for super-resolution...”, [pg. 79, col. 1, 4.1 Methodology, par. 1, ln. 6-11] “…We implement the Smart Cloud in the Pytorch framework and use two GeForce RTX 2080Ti GPUs on the smart cloud, one for inference and the other for the training. We use thebenchmarkDIV2Kdataset [2] for training offline pre-trained model. We initialize our specific model for training with weights from the pertained model.”):
store {a cumulative quality of} content comprising a plurality of input images based on a viewing frequency of the content ([pg. 77, col. 1, par. 2, ln. 1-13] “On the broadcaster side, a sequence of raw frames is encoded to a desired high-resolution determined by our live bitrate adaptation algorithm based on the current network condition and the status of our cloud computing resources. Downsampling of
K
F
is the critical step of the proposed coding module that improves the compression capability of an encoded video stream. First, the high resolution
K
F
H
is extracted from a GOP of an encoded video stream. Second, the
K
F
H
is down-sampled to lower resolution, yielding
K
F
L
, and the downsample scale is decided by the live bitrate adaptation algorithm. We do not downsample non-key frames to preserve its spatial and temporal characteristics of the high-resolution GOP. Finally, we use encoded
K
F
L
to replace
K
F
H
in the encoded stream.”, [pg. 77, col. 1, par. 3, ln. 1-10] “On the server side, we receive the video sequence with
K
F
L
and
N
K
F
H
. First, we extract and decode the key frame and do not deal with non-key frames since they are already at the desired resolution. Then, we upsample
K
F
L
to their original resolution, and the missing details of the
K
F
L
caused by down-sampling are recovered by SR. Finally, we replace
K
F
L
in the live stream with
K
F
H
in high resolution. At this point, we have both key and non-key frames at the desired high resolution. The whole process is transparent to viewers, and the player can decode and obtain high resolution videos without any modification.”, [pg. 78, col. 1, par. 2, ln. 1-17] “Online incremental training provides better video enhancement effects than using offline pre-trained models. However, online training and inference for each channel are computing-resource-hungry. Due to limited computing resources, we classify live channels into three different categories according to their popularity (i.e., the number of viewers): the hot channel, the normal channel, the cold channel. For the hot channel, the cloud server trains the specific SR model in real-time. For the normal channel, two SR models can be used: a specific model (SP-similar) trained by other hot channels with similar content and the offline pre-trained model. For the cold channel, no computing power support is provided. Fig. 5 shows snapshots of video quality enhancement results of a 360P video by the upsample scale x3, and the bottom right corner of each picture is the VMAF score. With the pre-trained model, the video quality has been significantly improved compared to bicubic. The specific model achieves the highest video quality. The video enhancement quality of the SP-similar model outperforms the pre-trained model.”),
determine a model storing condition based on the viewing frequency and the {cumulative quality} ([pg. 78, col. 1, par. 2, ln. 1-17] see specifically hot channel vs. normal channel vs. cold channel),
obtain a reference model corresponding to the model storing condition ([pg. 78, col. 1, par. 2, ln. 1-17] see specifically hot channel vs. normal channel vs. cold channel, specifically real-time SR for hot channel, vs. SP-similar and offline pre-trained model for normal channel, and no SR for cold channel),
store the reference model in the memory ([pg. 77, col. 1, par. 4, ln. 12 to col. 2, par. 1, ln. 2], [pg. 79, col. 1, 4.1 Methodology, par. 1, ln. 6-11]), and
generate a target model corresponding to a first image by training the reference model stored in the memory by using training data corresponding to a quality of the first image ([pg. 77, col. 2, 3.3. Online super-resolution model training, par. 1, ln. 1 to pg. 78, col. 1, par. 1, ln. 12] “One single model for super-resolution has a finite capability, which results in the great variance of the super-resolution outputs for different types of videos. To further enhance super-resolution performance, we can train the corresponding specific SR model for a live channel through real-time training. Due to the characteristic of live broadcast, we cannot obtain all content in advance, and scene switching in the live stream may occur at any time. We first evaluate the effects of using different training sets on the model performance: 𝐴 represents high-definition video chunks at the beginning of a live video stream, and 𝐵 represents the chunk obtained after scene switch. We compare three training ways: A, we only use set 𝐴 for training; A+B, we first use set𝐴, and after the detection of a scene change, we add the set 𝐵; A+aB, similar to A+B, but set B has a larger weight 𝑎 to be sampled for training [14]. We set𝑎 to 0.7. Figure 4 illustrates the online learning performance on online gaming. As shown in Figure 4, with the increase of training epoch, the effect of video enhancement increases, and training gain tends to be saturated in the late stage. Comparing the three curves, we see that A+aB can obtain the highest VMAF. Therefore, we adopt A+aB to ensure the enhancement effect of the super-resolution model. The cloud and the broadcaster jointly assist online model training: the cloud server regularly checks the video quality enhanced by the super-resolution model; the broadcaster detects scene change during encoding. When the training gain tends to be saturated, we stop online training and restart online training after scene switching”, [pg. 78, col. 1, par. 2, ln. 1-17], [pg. 78, Figure 3-5] see online training and image super-resolution from Specific, SP-similar and pre-trained (a), (b), and (c)).
One of ordinary skill in the art, before the effective filling date of the claimed invention, would specifically recognize GPUs contain both a processor and a memory for storing one or more instructions to be performed by the processor, and that the reference model of Chen, given that it is trained on the GPU, is thus stored in the memory. Chen does not specifically disclose storing a cumulative quality.
However, Choi specifically teaches to store a cumulative quality of content and to modify conditions of a super-resolution model based on the cumulative quality of content ([pg. 6, col. 2, 4.1 Problem Formulation, par. 1, ln. 1 to pg. 7, col. 1, par. 1, ln. 14] “As explained in Section 3, the proposed video delivery scheme jointly makes decisions on the number of transmitting chunks, the transcoding rate and the transmit power at the transmitter side, the number of depths of the ASR-GAN, and the number of CPU cores at the receiver side in every time slot. We suppose that the perfect channel state information (CSI) is known at the central controller or the transmitter. After they observe their own queue and buffer states respectively, the decisions are made for pursuing the average image quality. The joint optimization problem is described as follows: … see Equations (15-22)… where
P
-
is the maximum quality measure,
η
is the threshold for the average transmit power,
ξ
is the threshold for the average GPU usage, and
P
0
is the power budget. Also, 𝒟 and 𝒰 are the sets of available depths of the ASRGAN and the available CPU cores, respectively. Since we adaptively choose the number of transmitting and receiving chunks, the objective function in (15) is the long-term time-average quality degradation of the received chunks. Also,
N
=
[
N
0
,
N
1
,
…
,
N
(
T
)
]
, and r, P, d, and u are defined in a similar manner. Specifically, the expectation of (15)-(19) is with respect to random channel realizations. The constraint of (16) and (17) are for limiting the queuing delay and the chunk processing delay. The transmitting power consumption and usage of CPU cores are limited by the constraints of (18) and (19), respectively, and the constraint (20) comes from (1), which demonstrates that decision on N(t) and r(t) depend on the channel capacity.”). Specifically, one of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize Chen and Choi as within the same field of image super-resolution processing based on user and network characteristics, and as analogous to the claimed invention. Specifically, the motivation to combine is disclosed in Choi, wherein incorporating cumulative quality metrics allows for more joint optimization of video quality, processing requirement, and latency ([pg. 6, col. 2, 4 Joint Optimization of Dynamic Video Delivery and Quality Enhancement, par. 1, ln. 1-6] “This section introduces the joint optimization problem of dynamic image delivery and quality enhancement that pursues the high-quality video, the limited latency, and the efficient uses of transmit power and receiver CPU. Also, the Lyapunov-based decision method for solving the problem is presented.”, [pg. 6, col. 2, 4.1 Problem Formulation, par. 1, ln. 1 to pg. 7, col. 1, par. 1, ln. 14]). One of ordinary skill in the art, before the effective filling date of the claimed invention, would have combined the system of Chen with the cumulative quality metric of Choi through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results. Specifically, one of ordinary skill in the art would have combined cumulative quality metric of Choi with the system of Chen such that the model of Chen is further stored based on the cumulative quality metric, analogous to the training of the model based on latency optimization as already disclosed in Chen ([Chen, pg. 78, col. 1, 3.4, Live bitrate adaptation algorithm, par. 1, ln. 1-13] “Problem and goal. Super-resolution based key frame compression cannot be easily adopted in live streaming since we need to determine the desired resolution for live video and a suitable compression ratio to yield high quality and low latency. For example, with limited bandwidth, consistently uploading high-bitrate video chunks may induce high delay. Conversely, if you continue to up load low-bitrate videos, although the transmission time is short, the video quality will decrease. To balance multiple transmission quality factors, we propose a live bitrate adaptation algorithm to determine the encoding bitrate and the optimal downs-sampling scale of a live video based on the uplink bandwidth condition and the status of cloud computing power. Considering perceptual video quality features, we refer to the QoE model defined in [11]: see Equation (1)…”).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the device of Chen with the cumulative quality metrics of Choi to obtain the invention as specified in claim 1.
13. Regarding Claim 2, a combination of Chen and Choi teaches the device of claim 1. Chen further discloses wherein the model storing condition comprises at least one of information about the content with a high viewing frequency, resolution information of the content, a high frequency point of the cumulative quality, or a number of models to be stored ([pg. 77, col. 1, par. 2, ln. 1-13], [pg. 77, col. 1, par. 3, ln. 1-10], [pg. 78, col. 1, par. 2, ln. 1-17], [pg. 77, col. 2, 3.3. Online super-resolution model training, par. 1, ln. 1 to pg. 78, col. 1, par. 1, ln. 12], [pg. 79, Table 2], [Chen, pg. 78, col. 1, 3.4, Live bitrate adaptation algorithm, par. 1, ln. 1-13] see Equation (1)), wherein the content with the high viewing frequency represents content that is viewed at least a determined number of times within a determined interval, and wherein the high frequency point of the cumulative quality represents a largest quality value included in the cumulative quality. Specifically, one of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize the model storing condition includes a number of models to be stored (e.g., one for hot channel, two for normal channel, none for cold channel in [pg. 77, col. 2, 3.3. Online super-resolution model training, par. 1, ln. 1 to pg. 78, col. 1, par. 1, ln. 12]), information about the content with a high viewing frequency (e.g., see three training ways described in [pg. 77, col. 2, 3.3. Online super-resolution model training, par. 1, ln. 1 to pg. 78, col. 1, par. 1, ln. 12], specifically scene switch), and resolution information of the content (see [Chen, pg. 78, col. 1, 3.4, Live bitrate adaptation algorithm, par. 1, ln. 1-13] see Equation (1), and [pg. 79, Table 2]). Chen does not specifically disclose a high frequency point of the cumulative quality, or wherein the content with the high viewing frequency represents content that is viewed at least a determined number of times within a determined interval, and wherein the high frequency point of the cumulative quality represents a largest quality value included in the cumulative quality. However, the examiner notes that only one of the listed limitations is required given the broadest reasonable interpretation (BRI) of “…at least one of…”, and thus given that Chen discloses the model storing condition comprises a number of models to be stored and resolution information of the content, Chen would disclose the BRI of claim 2. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the device of Chen with the cumulative quality metrics of Choi to obtain the invention as specified in claim 2.
14. Regarding Claim 6, a combination of Chen and Choi teaches the device of claim 1. Chen further discloses storing the reference model for each type of classification information ([pg. 78, Figure 3-5] see Live-news, Live chat, and Online gaming categories, see also [pg. 77, col. 2, 3.3. Online super-resolution model training, par. 1, ln. 1 to pg. 78, col. 1, par. 1, ln. 12] specifically training sets A and B), and the classification information comprises at least one of a type of content, a type of over-the-top (OTT) content, a type of broadcast channel, a type of game content, a resolution of the content, or a combination of the type of content and the resolution of the content ([pg. 78, Figure 3-5] see Live-news, Live chat, and Online gaming categories, see also [pg. 77, col. 2, 3.3. Online super-resolution model training, par. 1, ln. 1 to pg. 78, col. 1, par. 1, ln. 12] specifically training sets A and B). Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the method of Chen with the cumulative quality metrics of Choi to obtain the invention as specified in claim 6.
15. Regarding Claim 7, a combination of Chen and Choi teaches the device of claim 1. The claim language is analogous to claim 6 with the exception of “…store the cumulative quality of the content for each type of classification information…”. Chen does not specifically disclose cumulative quality.
However, Choi discloses cumulative quality metrics ([pg. 6, col. 2, 4.1 Problem Formulation, par. 1, ln. 1 to pg. 7, col. 1, par. 1, ln. 14]). Specifically, the motivation to combine remains analogous to claim 1 ([pg. 6, col. 2, 4 Joint Optimization of Dynamic Video Delivery and Quality Enhancement, par. 1, ln. 1-6]). One of ordinary skill in the art, before the effective filling date of the claimed invention, would have combined the system of Chen with the cumulative quality metric of Choi through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results. Specifically, one of ordinary skill in the art would have combined cumulative quality metric of Choi with the system of Chen such that the cumulative quality of the content for each type of classification information is also stored, such that the reference model for each type of classification information can be optimized as taught in Choi ([pg. 6, col. 2, 4.1 Problem Formulation, par. 1, ln. 1 to pg. 7, col. 1, par. 1, ln. 14], [pg. 6, col. 2, 4 Joint Optimization of Dynamic Video Delivery and Quality Enhancement, par. 1, ln. 1-6]).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the method of Chen with the cumulative quality metrics of Choi to obtain the invention as specified in claim 7.
16. Regarding Claim 12, the claim language is analogous to claim 1, with the exception of “An operating method of an image processing device, the operating method comprising…”. Chen specifically discloses an operating method ([pg. 76, col. 1, par. 2, ln. 1-9] “We conducted extensive trace-driven simulations to evaluate LiveSRVC. The results show that LiveSRVC can save up to 50% of the required bandwidth if the live streaming method uploads the same quality as LiveSRVC. We evaluate our proposed key frame coding module compared to other state-of-the-art super-resolution based coding methods. The result shows that the compression rate of our coding method is on average 10% higher. Compared to the method of reconstructing all frames with super-resolution, LiveS RVC saves more than 10 × GPU occupation time.”). Rejections analogous to claim 1 are further applicable to the remainder of claim 12 in view of the analogous claim language. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the method of Chen with the cumulative quality metrics of Choi to obtain the invention as specified in claim 12.
17. Regarding Claims 13, 17, and 18 a combination of Chen and Choi teaches the method of claim 12. The claim language of claims 13, 17, and 18 is analogous to claims 2, 6, and 7 respectively, and rejections analogous to claims 2, 6, and 7 are further applicable to claims 13, 17, and 18 in view of the method of Chen. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the method of Chen with the cumulative quality metrics of Choi to obtain the invention as specified in claims 13, 17, and 18.
18. Claims 3-5, 8, and 14-16 are rejected under 35 U.S.C. 103 as being unpatentable over “Higher Quality Live Streaming under Lower Uplink Bandwidth: An approach of Super-Resolution Based Video Coding” to Chen, and further in view of “Delay-Sensitive and Power-Efficient Quality Control of Dynamic Video Streaming using Adaptive Super-Resolution” to Choi, and further in view of CN-113409192-A to Zheng et al. (hereinafter Zheng).
19. Regarding Claim 3, a combination of Chen and Choi teaches the device of claim 2. Chen further discloses {wherein the memory is a non-volatile memory, and} wherein the one or more instructions, when executed by the one or more processors individually or collectively, further cause the image processing device to: generate a second reference model trained from a pre-stored first reference model in response to a quality of the plurality of input images, and store, in the memory, the second reference model, based on the second reference model corresponding to the model storage condition ([pg. 77, col. 1, par. 2, ln. 1-13], [pg. 77, col. 1, par. 3, ln. 1-10], [pg. 78, col. 1, par. 2, ln. 1-17], [pg. 77, col. 2, 3.3. Online super-resolution model training, par. 1, ln. 1 to pg. 78, col. 1, par. 1, ln. 12], [pg. 79, col. 1, 4.1 Methodology, par. 1, ln. 6-11]). Specifically, one of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize that the second reference model (i.e., the specific model for hot channels, see [pg. 78, col. 1, par. 2, ln. 1-17]) is trained from a pre-stored first reference model (i.e., pre-trained model for normal channels via initializing weights, see [pg. 78, col. 1, par. 2, ln. 1-17] and [pg. 79, col. 1, 4.1 Methodology, par. 1, ln. 6-11]) in response to a quality of the plurality of input images ([pg. 77, col. 1, par. 2, ln. 1-13], [pg. 77, col. 1, par. 3, ln. 1-10], [pg. 78, col. 1, par. 2, ln. 1-17]). Chen does not specifically disclose that the memory is non-volatile memory. Likewise, Choi does not specifically teach that the memory is non-volatile memory.
However, Zheng teaches wherein the memory is a non-volatile memory ([pg. 11, par. 3, ln. 1-4] “Any reference to memory, storage, database, or other media used in this application may include non-volatile and/or volatile memory. Non-volatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory.…”). Specifically, one of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize Chen, Choi, and Zheng as within the same field of image super-resolution processing based on user characteristics, and as analogous to the claimed invention. Specifically, one of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize that you could substitute volatile memory of the combination of Chen and Choi with a non-volatile memory as taught in Zheng. The motivation to do this would have been obvious to one of ordinary skill in the art, in that non-volatile memory is generally larger in terms of storage capacity and thus reduces can be used to reduce memory usage, as well as retaining the stored information after power off. One of ordinary skill in the art, before the effective filling date of the claimed invention, would have combined the device of the combination of Chen and Choi with the non-volatile memory of Zheng through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results. Specifically, one of ordinary skill in the art would have combined the non-volatile memory of Zheng to store the models of the device of the combination of Chen and Choi, and thus freed up the volatile memory (e.g., of the GPU) for other processing.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the device of Chen with the cumulative quality metrics of Choi and the non-volatile memory of Zheng to obtain the invention as specified in claim 3.
20. Regarding Claim 4, a combination of Chen, Choi, and Zheng teaches the method of claim 3. Chen does not specifically disclose to store the second reference model in the memory based on the high frequency point of the cumulative quality.
However, Choi teaches to store the second reference model in the memory based on the high frequency point of the cumulative quality ([pg. 6, col. 2, 4.1 Problem Formulation, par. 1, ln. 1 to pg. 7, col. 1, par. 1, ln. 14] see Equation (15), [pg. 10, col. 1, 6 Performance Evaluation, par. 1, ln. 1 to col. 2, 6.1 Adaptive Super-Resolution Network, par. 3, ln. 11] “This section verifies the advantages of the proposed delay sensitive and power-efficient quality control of dynamic video streaming compared to comparison techniques introduced in Section 5.3. We first show the reliability of the adaptive SR whose performance is evaluated by observing PSNRs of output images… Our proposed ASRGAN is trained with 4,000 iterations and 32 batches. The generator G is the ASRGAN and the VGG19 is adopted as discriminator D. The DIV2K high resolution dataset is used to train and test the ASRGAN [54]. High-resolution images are preprocessed as explained in Sec. 3.2.1. The preprocessed dataset with various resolutions (e.g. r = 2, r = 3, r = 4, and r = 5) is randomly cropped with the size of 120 × 120 × 3… As a training result, Fig. 2 shows the average PSNR and average SSIM in the training phase. The average PSNR and average SSIM increase during training phase, and approach 27.97 dB and 0.8051 separately at the end of the training phase (4,000 iterations). Fig. 3 shows output images of the ASRGAN depending on the depth of the ASRGAN and the compression rate of input images. The test images shown in Fig. 3 are from DIV2K test dataset. We can see that when the feature is extracted from the deeper depth of the ASRGAN, output images have a better resolution… The information of the trained ASRGAN is described in Table 2. The quality measures (i.e., PSNR and SSIM) of output images of the ASRGAN depending on the depth and the compression rate, and the required weights (i.e., δkτ) and CPU clocks for operating the ASRGAN with different depths are given. We can see that as the ASRGAN extracts the feature from the deeper depth, the quality measures as well as the required CPU clocks increase regardless of the compression rate r. In conclusion, we can confirm that there is a tradeoff between the image quality and the computational task, and it can be controlled by adjusting the depth of the ASRGAN.”, [pg. 10, Fig 2], [pg. 12, Fig. 4 (c)]). Specifically, one of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize that the model is specifically stored in memory based on the high frequency point of the cumulative quality since the training phase completion is determined by the average PSNR and SSSIM. The motivation to combine would have been obvious to one of ordinary skill in the art, in that storing the model based on the high frequency point of the cumulative quality (e.g., PSNR) allows for accurate determination as to the quality of the outputs of the model for super-resolution. One of ordinary skill in the art, before the effective filling date of the claimed invention, would have combined the device of the combination of Chen with the cumulative quality metrics and high frequency point of the cumulative quality Choi, and further combined the device of the combination of Chen and Choi with the non-volatile memory of Zheng through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results. Specifically, one of ordinary skill in the art would have further combined the device of the combination of Chen, Choi, and Zheng such that the models are stored based on high frequency point of the cumulative quality as taught in Choi.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the device of Chen with the cumulative quality metrics and high frequency point of the cumulative quality of Choi and the non-volatile memory of Zheng to obtain the invention as specified in claim 4.
21. Regarding Claim 5, a combination of Chen, Choi, and Zheng teaches the device of claim 3. Chen further discloses to store the second reference model in the memory for the content having the high viewing frequency ([pg. 77, col. 1, par. 2, ln. 1-13], [pg. 77, col. 1, par. 3, ln. 1-10], [pg. 78, col. 1, par. 2, ln. 1-17], [pg. 77, col. 2, 3.3. Online super-resolution model training, par. 1, ln. 1 to pg. 78, col. 1, par. 1, ln. 12], [pg. 79, col. 1, 4.1 Methodology, par. 1, ln. 6-11]). Specifically, one of ordinary skill in the art would recognize the specific model (i.e., the second reference model) trained in Chen is stored in the memory for the content of the “hot” channel, which is the channel that has a high viewing frequency. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the device of Chen with the cumulative quality metrics and high frequency point of the cumulative quality of Choi and the non-volatile memory of Zheng to obtain the invention as specified in claim 5.
22. Regarding Claim 8, a combination of Chen and Choi teaches the method of claim 1. Chen discloses to store the {cumulative quality of} the plurality of input images corresponding to content with a high viewing frequency, and wherein the content with the high viewing frequency represents content that is viewed ([pg. 77, col. 1, par. 2, ln. 1-13], [pg. 77, col. 1, par. 3, ln. 1-10], [pg. 78, col. 1, par. 2, ln. 1-17], [pg. 77, col. 2, 3.3. Online super-resolution model training, par. 1, ln. 1 to pg. 78, col. 1, par. 1, ln. 12], [pg. 79, col. 1, 4.1 Methodology, par. 1, ln. 6-11]) {at least a determined number of times within a determined interval}. Chen does not specifically disclose a cumulative quality, or wherein high viewing frequency is content that is viewed at least a determined number of times within a determined interval.
However, Choi teaches to store a cumulative quality ([pg. 6, col. 2, 4.1 Problem Formulation, par. 1, ln. 1 to pg. 7, col. 1, par. 1, ln. 14]). Specifically, arguments analogous to claim 1 are further applicable to claim 8. The motivation to combine remains analogous to claim 1 ([pg. 6, col. 2, 4 Joint Optimization of Dynamic Video Delivery and Quality Enhancement, par. 1, ln. 1-6]). One of ordinary skill in the art, before the effective filling date of the claimed invention, would have combined the device of the combination of Chen with the cumulative quality metrics of Choi through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results.
Choi does not specifically teach wherein high viewing frequency content is content that has been viewed at least a determined number of times within a determined interval. Therefore, the device of the combination of Chen and Choi does not specifically teach wherein high viewing frequency is content that is viewed at least a determined number of times within a determined interval.
However, Zheng specifically teaches wherein high frequency content is content viewed at least a determined number of times within a determined interval ([pg. 6, par. 4, ln. 1-11] “It is understandable that when the user uses the electronic device, the electronic device can record the user's use of each candidate application, including the use frequency and use time of each candidate application. The electronic device determines the target application from each candidate application based on the usage frequency of each candidate application. In one embodiment, the electronic device counts the usage frequency of each candidate application, and determines the candidate application with the highest usage frequency as the target application. In another implementation manner, the electronic device counts the usage frequency of each candidate application, and determines the candidate application with the second highest usage frequency as the target application. In another implementation manner, the electronic device counts the usage frequency of each candidate application within a preset time period, and determines the candidate application with the highest usage frequency within the preset time period as the target application. The way the electronic device determines the target application is not limited and is not limited here.”). The motivation to combine would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, in that using a viewing frequency threshold or maximum selection as taught in Zheng is an easily implementable and customizable means by which to determine “high” frequency viewing content. One of ordinary skill in the art, before the effective filling date of the claimed invention, would have combined the device of the combination of Chen with the cumulative quality metrics and high frequency point of the cumulative quality Choi, and further combined the device of the combination of Chen and Choi with the non-volatile memory and determined number of views within an interval of Zheng through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results. Specifically, one of ordinary skill in the art would have combined the determined number of views across an interval of Zheng with the device of the combination of Chen and Choi such that the hot, normal, and cold channels are determined by a determined number of views within a determined interval (e.g., hot channel >=100 views in 10 minutes, normal channel >=40 views in 10 minutes, cold <=10 views in 10 minutes).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the device of Chen with the cumulative quality metrics and high frequency point of the cumulative quality of Choi and the non-volatile memory and determined number of views within a determined interval of Zheng to obtain the invention as specified in claim 8.
23. Regarding Claims 14-16, a combination of Chen and Choi teaches the method of claim 13. The claim language of claim 14-16 is analogous to claim 3-5, and rejections analogous to claims 3-5 are further applicable to claims 14-16 in view of the method of the combination of Chen and Choi. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the method of Chen with the cumulative quality metrics and high frequency point of the cumulative quality of Choi and the non-volatile memory of Zheng to obtain the invention as specified in claims 14-16.
24. Claim 9 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over “Higher Quality Live Streaming under Lower Uplink Bandwidth: An approach of Super-Resolution Based Video Coding” to Chen, and further in view of “Delay-Sensitive and Power-Efficient Quality Control of Dynamic Video Streaming using Adaptive Super-Resolution” to Choi, in view of CN-113409192-A to Zheng, and further in view of U.S. Publication No. 2023/0048386 to Wang et al. (hereinafter Wang).
25, Regarding Claim 9, a combination of Chen, Choi, and Zhang teaches the device of claim 3. Chen discloses {identifying} a model that outputs an image having a quality {that is closest to the quality of the first image among} the pre-stored first reference model and the second reference model trained based on the first model ([pg. 77, col. 1, par. 2, ln. 1-13], [pg. 77, col. 1, par. 3, ln. 1-10], [pg. 78, col. 1, par. 2, ln. 1-17], [pg. 77, col. 2, 3.3. Online super-resolution model training, par. 1, ln. 1 to pg. 78, col. 1, par. 1, ln. 12], [pg. 79, col. 1, 4.1 Methodology, par. 1, ln. 6-11]); and generating the target model by training the {identified} model based on the quality of the first image ([pg. 77, col. 1, par. 2, ln. 1-13], [pg. 77, col. 1, par. 3, ln. 1-10], [pg. 78, col. 1, par. 2, ln. 1-17], [pg. 77, col. 2, 3.3. Online super-resolution model training, par. 1, ln. 1 to pg. 78, col. 1, par. 1, ln. 12], [pg. 79, col. 1, 4.1 Methodology, par. 1, ln. 6-11]). Specifically, one of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize that Chen discloses a pre-stored first reference model (i.e. pre-trained mode see [Fig. 5 (c)]), and a second reference model trained based on the first model (i.e., specific model see [Fig. 5 (a)] and [pg. 79, col. 1, 4.1 Methodology, par. 1, ln. 6-11]). Chen does not specifically disclose identifying a model that outputs an image have a quality that is closest to the quality of the first image among the first and second reference models, or wherein the identified model (i.e., the first or second reference model that is closest) is trained based on the quality of the first image. Likewise, Choi and Zheng does not specifically teach identifying a model that outputs an image have a quality that is closest to the quality of the first image among the first and second reference models.
However, Wang specifically teaches identifying a model that outputs an image having a quality that is closest to the quality of the first image among a first and second reference model ([par. 0292, ln. 1-41] “…the target model may be determined from the at least two trained models preferentially based on the F1 score. As mentioned above, the F1 score is an index in consideration of both accuracy rate and recall rate, and in general, the closer the F1 score is to 1, the better the model is trained, and conversely, the closer the F1 score is to 0, the worse the model is trained. Furthermore, for example, it may further determine whether the target model determined satisfies a predetermined requirement according to the confusion matrix; and may update the target model by retraining or adjusting the confidence threshold in response to the target model determined not satisfying the predetermined requirement. The test result for each category may be understood in more detail based on the confusion matrix, so that a more comprehensive and detailed understanding of the training of the model for different categories may be obtained. For example, a prediction result distribution, a prediction accuracy rate, the recall rate and the F1 score of the model for some or all categories may be verified based on the confusion matrix, and whether it satisfies the predetermined requirement may be determined. For example, for the screen defect detection application scenario described above, it may be determined whether the prediction accuracy rate, the recall rate and/or the F1 score of the model for some defect categories with higher priorities (or all defect categories) are greater than a predetermined threshold. If the prediction of the target model for one or several categories does not satisfy the predetermined requirement, the target model may be updated by adjusting the confidence threshold for the corresponding category, or the target model may be trained again by supplementing the sample data for the corresponding category to update the target model. Or, if the prediction of the target model for more categories does not satisfy the predetermined requirement, or the prediction of the target model cannot be adjusted to satisfy the predetermined requirement by adjusting the confidence threshold, supplementing the sample data and the like, the target model may be retrained by supplementing the training data set, optimizing the model parameter or the like, or the initial deep learning model may be retrained and the target model may be re-selected.”). Specifically, one of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize the device of the combination of Chen, Choi, and Zhang and Wang as within the same field of multi-model machine learning image processing, and as analogous to the claimed invention. The motivation to combine is disclosed in Wang, wherein comparing two reference model capabilities allows for retraining and updating of the less accurate model ([par. 0292, ln. 1-41]). One of ordinary skill in the art, before the effective filling date of the claimed invention, would have combined the device of the combination of Chen, Choi, and Zheng with the closeness identification and training of the target model as taught in Wang, through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results. Specifically, in combining the closeness identification and target model training of Wang with the device of the combination of Chen, Choi, and Zhang, one of ordinary skill in the art would have used to quality of the first image as the metric to decide between the two reference models of the combination of Chen, Choi, and Zhang. Specifically, for the case of the normal channel of the combination of Chen, Choi, and Zhang, one of ordinary skill in the art would recognize you could perform an analogous re-training of the identified model as the target model, as taught in Wang, using the PSNR of the super-resolved results of the pre-trained model and the SP-similar model.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the device of Chen with the cumulative quality metrics of Choi, the non-volatile memory of Zheng, and the closeness identification and training of the target model as taught in Wang to obtain the invention as specified in claims 19.
26. Regarding Claim 10, a combination of Chen, Choi, Zheng, and Wang teaches the device of claim 9. Chen further discloses to obtain, based on the target model, a second image that is quality-processed from the first image ([pg. 77, col. 1, par. 2, ln. 1-13], [pg. 77, col. 1, par. 3, ln. 1-10], [pg. 78, col. 1, par. 2, ln. 1-17], [pg. 77, col. 2, 3.3. Online super-resolution model training, par. 1, ln. 1 to pg. 78, col. 1, par. 1, ln. 12], [pg. 79, col. 1, 4.1 Methodology, par. 1, ln. 6-11]). Specifically, one of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize Chen discloses that the models are trained corresponding to the first image to generate a second image that is quality-processed (i.e., upscaled in the case of Chen) from the first image and analogous training data. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the device of Chen with the cumulative quality metrics of Choi, the non-volatile memory of Zheng, and the closeness identification and training of the target model as taught in Wang to obtain the invention as specified in claims 20.
27. Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over “Higher Quality Live Streaming under Lower Uplink Bandwidth: An approach of Super-Resolution Based Video Coding” to Chen, and further in view of “Delay-Sensitive and Power-Efficient Quality Control of Dynamic Video Streaming using Adaptive Super-Resolution” to Choi, in view of CN-113409192-A to Zheng, and further in view of U.S. Publication No. 2023/0048386 to Wang, and further in view of U.S. Publication No. 2021/0390152 to Jang et al. (hereinafter Jang).
28. Regarding Claim 11, a combination of Chen, Choi, Zheng, and Wang teaches the method of claim 10. Chen discloses the device further comprises {a communication interface}, and controlling {the communication interface} to transmit the model storing condition to a server; and based on receiving a reference model corresponding to the model storing condition from the server, storing the reference model in the memory ([pg. 77, col. 1, par. 2, ln. 1-13], [pg. 77, col. 1, par. 3, ln. 1-10], [pg. 78, col. 1, par. 2, ln. 1-17], [pg. 77, col. 2, 3.3. Online super-resolution model training, par. 1, ln. 1 to pg. 78, col. 1, par. 1, ln. 12], [pg. 79, col. 1, 4.1 Methodology, par. 1, ln. 6-11]). Specifically, Chen does not disclose the device comprises a communication interface, because Chen instead simulates transmitting the model storing condition to a server; and based on receiving a reference model corresponding to the model storing condition from the server, storing the reference model in the memory ([pg. 76, Fig. 1], [pg. 77, Fig. 2], [pg. 79, col. 1, 4.1 Methodology, par. 2, ln. 1 to col. 2, par. 2, ln. 13] “Evaluation Setup. We select three different stream categories (live news, chatting, and games) from YouTube. To simulate the dynamic changes between broadcasters and the smart cloud in a realistic network, we use the Linux Traffic Control tool [12] to emu late 100 4G networks uplink traces [22]. More specifically, we select network traces whose average uplink bandwidth below 2.5Mbps to model a bandwidth-constrained environment. We evaluate the performance of the proposed LiveSRVC in various aspects by com paring it with two baseline methods: The original method: broadcasters upload video encoded by H264 codec according to the uplink bandwidth, and the cloud server does not perform video enhancement processes. • LiveNAS: broadcasters send low-resolution video streams, and the cloud server super-resolves all frames of the low resolution streams used in [14].”). Likewise, while Choi teaches an analogous transmitting/receiving to a server ([pg. 4, Fig. 1]), Choi does not disclose a communication interface. Zheng and Wang likewise do not disclose a communication interface.
However, Jang specifically teaches a communication interface ([par. 0046, ln. 1-25] “The communication module 213, 223 may provide a function for communication between the electronic device 110 and the server 150 over the network 170 and may provide a function for communication between the electronic device 110 and/or the server 150 and another electronic device, for example, the electronic device 120 or another server (for example the server 160). For example, the processor 212 of the electronic device 110 may transfer a request created based on a program code stored in the storage device such as the memory 211, to the server 150 over the network 170 under control of the communication module 213. Inversely, a control signal, an instruction, content, a file, etc., provided under control of the processor 222 of the server 150 may be received at the electronic device 110 through the communication module 213 of the electronic device 110 by going through the communication module 223 and the network 170. For example, a control signal, an instruction, content, a file, etc., of the server 150 received through the communication module 213 may be transferred to the processor 212 or the memory 211, and content, a file, etc., may be stored in a storage medium, for example, the permanent storage device, further includable in the electronic device 110. The communication module 213, 223 may, for example, include processors, a bus, an antenna, a connection port, and/or the like.”). Specifically, one of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize the device of the combination of Chen, Choi, Zhang, and Wang and Jang as within the same field of machine learning based on network characteristics, and as analogous to the claimed invention. The motivation to combine would have been obvious to one of ordinary skill in the art, in that communications interface would be required for both the server and client side to perform communications as taught in Chen and Choi, and thus the communication interface taught in Jang offers a real-world application to the device of the combination of Chen Choi, Zhang, and Wang. One of ordinary skill in the art, before the effective filling date of the claimed invention, would have combined the device of the combination of Chen Choi, Zhang, and Wang with the communication interface of Jang through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results. Specifically, one of ordinary skill in the art would have combined the communication interface of Jang such that the device of the combination of Chen Choi, Zhang, and Wang controls the communication interface to transmit and receive from the server as taught in Chen ([pg. 76, Fig. 1], [pg. 77, Fig. 2], [pg. 79, col. 1, 4.1 Methodology, par. 2, ln. 1 to col. 2, par. 2, ln. 13]).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the device of the combination of Chen Choi, Zhang, and Wang and the communication interface of Jang to obtain the invention as specified in claim 11.
29. Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over “Higher Quality Live Streaming under Lower Uplink Bandwidth: An approach of Super-Resolution Based Video Coding” to Chen, and further in view of “Delay-Sensitive and Power-Efficient Quality Control of Dynamic Video Streaming using Adaptive Super-Resolution” to Choi, and further in view of U.S. Publication No. 2023/0048386 to Wang et al. (hereinafter Wang).
30. Regarding Claim 19, a combination of Chen and Choi teaches the method of claim 12. Rejections analogous to claim 9 are further applicable to claim 19 in view of the analogous claim langue. Specifically, the motivation remains analogous to claim 9. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the method of Chen with the cumulative quality metrics of Choi and the closeness identification and training of the target model as taught in Wang to obtain the invention as specified in claims 19.
31. Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over “Higher Quality Live Streaming under Lower Uplink Bandwidth: An approach of Super-Resolution Based Video Coding” to Chen, and further in view of “Delay-Sensitive and Power-Efficient Quality Control of Dynamic Video Streaming using Adaptive Super-Resolution” to Choi, and further in view of U.S. Publication No. 2021/0390152 to Jang et al. (hereinafter Jang).
32. Regarding Claim 20, a combination of Chen and Choi teaches the method of claim 12. The claim language of claim 20 is analogous to claim 11. Rejections analogous to claim 11 are further applicable to claim 20 in view of the analogous claim language and the method of the combination of Chen and Choi. Specifically, the motivation remains analogous to claim 11. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the method of Chen with the cumulative quality metrics of Choi and the communication interface of Jang to obtain the invention as specified in claims 20.
Conclusion
33. The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure. See PTO-892.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PAULO ANDRES GARCIA whose telephone number is (703)756-5493. The examiner can normally be reached Mon-Fri, 8-4:30PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chan Park can be reached on (571)272-7409. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PAULO ANDRES GARCIA/Examiner, Art Unit 2669 /CHAN S PARK/Supervisory Patent Examiner, Art Unit 2669
Prosecution Insights