DETAILED ACTION
Response to Amendment
This Action is responsive to Applicant’s response filed on 12/31/2025. All claims are still pending
in the present application. This Action is made FINAL.
Response to Arguments
In regards to Argument(s), Applicant(s) state(s) that, Kim et al (US 2020/0389658 A1) in view of Zhang et al (US 10,652,565 B1) does not disclose/teach/suggest on the amended claim(s) “reconstructing, as reconstructed image data, the first image data based at least in part on the fourth image data and a feature vector, the feature vector carrying style information that includes at least color and texture information”, therefore, the rejection of 35 U.S.C. 103 should be removed, (Emphasis added, Remarks, page 9-10).
Applicant’s arguments have been considered but are moot in view of the new ground(s) of rejection in view of Kim et al (US 2020/0389658 A1) and Tomar et al (US 2021/0382936 A1).
Office Action Summary
Claim(s) 1-2, 5-6, 8-10, 13-14, and 16-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kim et al (US 2020/0389658 A1) in view of Tomar et al (US 2021/0382936 A1).
Claim(s) 3, 11, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kim et al (US 2020/0389658 A1) in view of Tomar et al (US 2021/0382936 A1), further in view of Lu et al (US 2022/0237740 A1).
Claim(s) 4, 12, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kim et al (US 2020/0389658 A1) in view of Tomar et al (US 2021/0382936 A1) and Lu et al (US 2022/0237740 A1), further in view of Park et al (Fast Adaptation to Super-Resolution Networks via Meta-Learning).
Claim(s) 7 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kim et al (US 2020/0389658 A1) in view of Tomar et al (US 2021/0382936 A1), further in view of Park et al (Fast Adaptation to Super-Resolution Networks via Meta-Learning).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-2, 5-6, 8-10, 13-14, and 16-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kim et al (US 2020/0389658 A1) in view of Tomar et al (US 2021/0382936 A1).
Regarding claim(s) 1, 9, and 17, Kim teaches one or more non-transitory computer readable media storing executable instructions that, when executed by one or more processors (Paragraph [0061]), cause the one or more processors to perform acts comprising:
receiving first image data (Figure 2B; Figure 3: Original Image 300; Paragraph [0006]: “determining a compressed image by performing downsampling using a deep neural network (DNN) on an image”; and Paragraph [0065]: “an original image 300”);
downsampling the first image data to second image data (Figure 2B: S210; Figure 3: Downsampling 302; Paragraph [0006]: “determining a compressed image by performing downsampling using a deep neural network (DNN) on an image”; and Paragraph [0066]: “a compressed image 303 obtained through downsampling 302 with respect to the original signal 300 may be generated”);
encoding the second image data to third image data, the third image data being a bitstream (Figure 2B: S216; Figure 3: Encoding 304 and Bitstream; Paragraph [0123]: “In operation S216, the bitstream generator 170 of the image compressing device 150 may generate a bitstream including information related to the encoded residual signal”; and Paragraph [0066]: “[…] the encoding process 304 may be performed on the compressed image 303”);
decoding the third image data to fourth image data (Figure 2A: S202; Paragraph [0037]: “decoding the compressed image by using the residual signal and a prediction signal obtained by performing prediction”; and Paragraph [0066]: “As a result of the decoding process 306 with respect to a bitstream including a result of the encoding process 304, a decoded compressed image 307 may be determined, upsampling 308 may be performed on the decoded compressed image 307”); and
reconstructing, as reconstructed image data, the first image data based at least in part on the fourth image data(Figure 2A: S204; Paragraph [0037]: “reconstructing the image by performing upsampling using a deep neural network (DNN) on the decoded compressed image”; and Paragraph [0066]: “[…] upsampling 308 may be performed on the decoded compressed image 307, and thus a reconstructed image 309 may be determined”).
Kim fails to teach a
However, Tomar teaches a feature vector (Paragraph [0022]: “In an illustrative example of the generating a similarity score, some embodiments convert (e.g., via a deep learning machine learning model) the one or more features (e.g., the line texture and shading patterns) into a feature vector (e.g., a vector of numbers that represent the one or more features) that is embedded in feature space”), the feature vector carrying style information that includes at least color and texture information (Paragraph [0021]: “These one or more features may correspond to an image style of the target image, as opposed to the content of the target image. For example, some embodiments extract the line texture or shading patterns from the target image that make up the content of the target image”; and Paragraph [0001]: “image style can refer to the color, lighting, shading, texture, line patterns, fading or other image effects of an object representing the content”).
Therefore, it would have been obvious to one of ordinary skill in the art to combine Kim and Tomar before the effective filing date of the claimed invention. The motivation for this combination of references would have been to represent image style characteristics such as color and texture using a feature vector representation in order to enable automated analysis and comparison of image styles, as taught by Tomar in Paragraph [0021] – [0022]. This motivation for the combination of Kim and Tomar is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Regarding claim(s) 2, 10, and 18, Kim as modified by Tomar teaches the method of claim 1, where Kim teaches the method further comprising:
generating a stack of kernels based at least in part on a weight vector (Figure 4A, 4D, and 4E; and Paragraph [0076]: “The deep convolutional neural network may determine weights of nodes included in each of the convolution layers. The nodes included in each of the convolution layers may generate feature maps by using different filter kernels. The deep convolutional neural network may adjust the weights of the nodes and thus may adjust weights of the filter kernels that generate the feature maps”); and
where Tomar teaches generating the feature vector based at least in part on the stack of kernels (Paragraph [0022]: “In an illustrative example of the generating a similarity score, some embodiments convert (e.g., via a deep learning machine learning model) the one or more features (e.g., the line texture and shading patterns) into a feature vector (e.g., a vector of numbers that represent the one or more features) that is embedded in feature space”).
Regarding claim(s) 5 and 13 Kim as modified by Tomar teaches the method of claim 1, where Kim teaches wherein encoding the second image data to the third image data or decoding the third image data to the fourth image data comprises using one or more compression methods, the one or more compression methods comprising one or more of:
JPEG, JPEG 2000, H.264/MPEG4, H.265/HEVC, VCC, a DNN-based learned image compression method, or a DNN-based learned video compression method (Paragraph [0002]: “Image data is encoded according to a predetermined data compression standard, for example, a codec according to the Moving Picture Expert Group (MPEG) standard”; and Paragraph [0006]: “an image compressing method may include determining a compressed image by performing downsampling using a deep neural network (DNN) on an image”).
Regarding claim(s) 6 and 14, Kim as modified by Tomar teaches the method of claim 1, where Kim teaches wherein the first image data comprises at least one of an image, a video frame, or a sequence of video frames (Paragraph [0111]: “According to an embodiment, the reconstructor 120 may perform a process of decoding, upsampling, and downsampling an image by using various data units including a video, a sequence, a frame, a slice, a slice segment, a largest coding unit, a coding unit, a prediction unit, a transform unit, a processing unit, or the like”).
Regarding claim(s) 8 and 16, Kim as modified by Tomar teaches the method of claim 1, where Kim teaches wherein the third image data and the feature vector are sent from an encoder (Figure 2B: S216; Figure 3: Encoding 304 and Bitstream; Paragraph [0123]: “In operation S216, the bitstream generator 170 of the image compressing device 150 may generate a bitstream including information related to the encoded residual signal”; and Paragraph [0066]: “[…] the encoding process 304 may be performed on the compressed image 303”) to a decoder (Figure 2A: S202; Paragraph [0037]: “decoding the compressed image by using the residual signal and a prediction signal obtained by performing prediction”; and Paragraph [0066]: “As a result of the decoding process 306 with respect to a bitstream including a result of the encoding process 304, a decoded compressed image 307 may be determined, upsampling 308 may be performed on the decoded compressed image 307”); and
where Tomar teaches feature vector (Paragraph [0022]: “In an illustrative example of the generating a similarity score, some embodiments convert (e.g., via a deep learning machine learning model) the one or more features (e.g., the line texture and shading patterns) into a feature vector (e.g., a vector of numbers that represent the one or more features) that is embedded in feature space”).
Claim(s) 3, 11, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kim et al (US 2020/0389658 A1) in view of Tomar et al (US 2021/0382936 A1), further in view of Lu et al (US 2022/0237740 A1).
Regarding claim(s) 3, 11, and 19, Kim as modified by Tomar teaches the method of claim 2, where Kim teaches the method further comprising:
(Paragraph [0046]: “the compressed image may include generating the compressed image by performing filtering in each of the plurality of hidden layers by using at least one of a plurality of filter kernels”; Paragraph [0039]: “the performing of the upsampling by using the deep convolutional neural network may include performing the upsampling by performing filtering in each of the plurality of hidden layers by using at least one of a plurality of filter kernels”; and Paragraph [0043]: “the DNN used by the image reconstructing method may trained to allow a sum of at least one lossy information to be decreased”).
Kim and Tomar fails to teach to obtaining a set of parameters indicating a compression quality of the fourth image dataLu teaches to obtaining a set of parameters indicating a compression quality of the fourth image data (Figure 2 and Figure 14 => Quantization bin size and Scaling -> Latent -> decoder; and Paragraph [0031]: “[…] In other machine-learning-based compression schemes, an encoder and a decoder network may be dependent on the β parameter so that a single model can adapt to different rate-distortion tradeoffs. Other machine-learning-based compression schemes may learn to adjust quantization step sizes of generated latents”).
Therefore, it would have been obvious to one of ordinary skill in the art to combine Kim, Tomar and Lu before the effective filing date of the claimed invention. The motivation for this combination of references would have been to enable adaptive control of the weight/feature vector based on compression quality in order to improve rate–distortion performance and reconstruction efficiency, which is a well-known and predictable design goal in neural-network-based image compression systems. This motivation for the combination of Kim, Tomar and Lu is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Claim(s) 4, 12, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kim et al (US 2020/0389658 A1) in view of Tomar et al (US 2021/0382936 A1) and Lu et al (US 2022/0237740 A1), further in view of Park et al (Fast Adaptation to Super-Resolution Networks via Meta-Learning).
Regarding claim(s) 4, 12, and 20, Kim as modified by Tomar and Lu teaches the method of claim 3, but do not specifically teach the method further comprising: computing a distortion loss value based at least in part on the first image data and the reconstructed image data; determining a step size based at least in part on the distortion loss value; and updating the set of parameters based at least in part on the distortion loss value and the step size.
However, Park teaches the method further comprising:
computing a distortion loss value based at least in part on the first image data and the reconstructed image data (Page 6, Section 3.3 Proposed Method and Equation 1 teaches computing a distortion loss value based on original and reconstructed image data, as shown by the loss function (See Equation 1), which directly represents a distortion between a reconstructed image and a corresponding reference image);
determining a step size based at least in part on the distortion loss value (Page 7, Algorithm 1 and Equation 2 teaches determining a step size for updating network parameters based on the computed distortion loss, as shown by the update rule θi ← θ−α∇θL, where α is a step size (learning rate) that scales the gradient of the distortion loss during parameter adaptation); and
updating the set of parameters based at least in part on the distortion loss value and the step size (Page 7, Algorithm 1 and Page 8, Algorithm 2 teaches updating a set of network parameters based on the distortion loss value and the step size, as shown by the inner-loop update θi ← θ−α∇θLi and the outer-loop update θ ← θ−β∇θ∑Li, in which the parameters θ are updated using both the distortion loss and the step size).
Therefore, it would have been obvious to one of ordinary skill in the art to combine Kim, Tomar, Lu and Park before the effective filing date of the claimed invention. The motivation for this combination of references would have been to enable adaptive refinement of compression and reconstruction parameters based on real-time reconstruction error in order to improve reconstruction accuracy and robustness to input variations, which is a well-known and predictable design objective in neural-network-based image compression and super-resolution systems. This motivation for the combination of Kim, Tomar, Lu and Park is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Claim(s) 7 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kim et al (US 2020/0389658 A1) in view of Tomar et al (US 2021/0382936 A1), further in view of Park et al (Fast Adaptation to Super-Resolution Networks via Meta-Learning).
Regarding claim(s) 7 and 15, Kim as modified by Tomar teaches the method of claim 1, where Kim teaches wherein reconstructing, as the reconstructed image data, the first image data based at least in part on the fourth image data (Figure 2A: S204; Paragraph [0037]: “reconstructing the image by performing upsampling using a deep neural network (DNN) on the decoded compressed image”; and Paragraph [0066]: “[…] upsampling 308 may be performed on the decoded compressed image 307, and thus a reconstructed image 309 may be determined”),
where Tomar teaches the feature vector (Paragraph [0022]: “In an illustrative example of the generating a similarity score, some embodiments convert (e.g., via a deep learning machine learning model) the one or more features (e.g., the line texture and shading patterns) into a feature vector (e.g., a vector of numbers that represent the one or more features) that is embedded in feature space”).
Kim and Tomar fails to teaches a meta-controlled super-resolution method. However, Park teaches a meta-controlled super-resolution method (Figure 2, Abstract, Section 3.3, and Algorithms 1 and 2 teaches a meta-learning-based super-resolution framework in which a super-resolution network is trained through meta-learning and is adapted at test time using a small number of gradient-update steps based on a reconstruction loss, thereby controlling the super-resolution process using meta-learned parameters).
Therefore, it would have been obvious to one of ordinary skill in the art to combine Kim, Tomar and Park before the effective filing date of the claimed invention. The motivation for this combination of references would have been to enable adaptive, image-specific control of a super-resolution network in order to improve reconstruction performance across varying input degradations, which represents a predictable and well-understood improvement in the field of deep-learning-based super-resolution. This motivation for the combination of Kim, Zhang and Park is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Relevant Prior Art Directed to State of Art
El-Khamy et al (US 2018/0293707 A1) are relevant prior art not applied in the rejection(s) above. El-Khamy discloses a method for super resolution imaging, the method comprising: receiving, by a processor, a low resolution image; generating, by the processor, an intermediate high resolution image having an improved resolution compared to the low resolution image; generating, by the processor, a final high resolution image based on the intermediate high resolution image and the low resolution image; and transmitting, by the processor, the final high resolution image to a display device for display thereby.
Kim et al (US 2020/0162751 A1) are relevant prior art not applied in the rejection(s) above. Kim discloses an image reconstructing method comprising: obtaining a bitstream generated by encoding a first image; reconstructing a second image by decoding the bitstream, the second image corresponding to the first image; and obtaining a third image upsampled from the second image by using a deep neural network (DNN) for upsampling, based on upsampling target information, wherein the first image is generated by downsampling an original image by using a DNN for downsampling.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JONGBONG NAH whose telephone number is (571)272-1361. The examiner can normally be reached M - F: 7:30am - 4:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Edward Urban can be reached on 571-272-7899. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JONGBONG NAH/Examiner, Art Unit 2674
/ONEAL R MISTRY/Supervisory Patent Examiner, Art Unit 2674