DETAILED ACTION
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on March 12, 2026, has been entered.
Response to Amendment
Claims 1-2, 5, 7-10, and 12-20 were previously pending. Applicant’s amendment filed March 12, 2026, has been entered in full. Claims 1, 5, 7-10, 14 and 17 are amended. Claims 13 and 15 are cancelled. No new claims are added. Accordingly, claims 1-2, 5, 7-10, 12, 14, and 16-20 are now pending.
Response to Arguments
Applicant argues that the previous objection to claim 7 has been overcome by amendment (Remarks filed March 12, 2026, hereinafter Remarks: Page 7). Examiner agrees. The previous objection is withdrawn.
Applicant traverses the previous rejections of claims 9, 10 and 13 under 35 U.S.C. 112 (Remarks: Pages 7-8).
Applicant has cancelled claim 13 (Remarks: Page 7), so the previous rejections of that claim are withdrawn as moot.
Applicant argues that claims 9 and 10 are definite (Remarks: Pages 7-8). Examiner agrees. The previous rejections of these claims under 35 U.S.C. 112(b) are withdrawn.
Applicant argues that claims 9 and 10 comply with the written description requirement under 35 U.S.C. 112(a) (Remarks: Pages 7-8). Examiner agrees with respect to claim 9 and the previous ‘112(a) rejection of this claim is withdrawn. However, Examiner respectfully disagrees regarding claim 10. Neither pars. [0050] and [0060] cited in the Remarks, nor any other portion of the specification describes receiving metadata indicative of a luminance value of the plurality of image portions as required by amended claim 10. See the ‘112(a) rejection below for further explanation.
Applicant argues that the previous rejection of claim 17 under 35 U.S.C. 112(a) has been overcome by amendment (Remarks: Page 9). Examiner agrees. The previous rejection is withdrawn.
Applicant argues that the previous rejection of claims 1-2, 5, 7-10 and 12-20 under 35 U.S.C. 112(b) has been overcome by amendment (Remarks: Page 9). Examiner agrees. The previous rejection is withdrawn.
Applicant traverses the previous rejections under 35 U.S.C. 102 and 103 (Remarks: Pages 9-11). In particular, Applicant argues that the previously-cited ‘Yang’ reference does not teach all elements of the amended claims (Remarks: Pages 9-10). Examiner does not necessarily agree. Nevertheless, the previous rejections over Yang have been withdrawn and replaced by new rejections over the previously-cited ‘Zhang’ reference, which appears to be closer to Applicant’s inventive concept as expressed in the newly-amended claims.
Claim Interpretation
Claims are given their broadest reasonable interpretation (BRI) during examination. MPEP 2111. Under BRI, the words of a claim are given their plain meaning, unless such meaning is inconsistent with the specification. MPEP 2111.01, Subsection I. The plain meaning of a term is the ordinary and customary meaning given to the term by those of ordinary skill in the art at the relevant time. Id.
Claim 1 recites “a first image upscaling process” and “a second image upscaling process”. The plain meaning of these phrases is that the “first” and “second” image upscaling processes are different. However, such a plain meaning is inconsistent with the specification. The specification describes that the purpose of storing image portions in the cache is to re-use already-upscaled images (e.g., [0064] as-published), and such re-use does not require that the image upscaling processes are different. Furthermore, the specification does not describe any example where the image upscaling process applied to cached portions from a previous frame is different from the image upscaling process applied to other portions in a current frame. Therefore, the BRI of claim 1 includes embodiments where the “first” and “second” image upscaling processes may be the same image upscaling process.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claim(s) 10 is/are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claim 10 depends from claim 1. Claim 1 recites “receiving, from the server device and by the client device, metadata” at the fifth line. Claim 10 has been amended to further recite “the metadata further comprising metadata indicative of a luminance value of the plurality of image portions” at lines 3-6.
Examiner has reviewed the specification, but finds no disclosure of metadata indicative of a luminance value.
In one example, pars. [0050] and [0060] of the as-published specification generally describe a server providing information to a client device as metadata, but do not specifically mention any luminance values being transmitted as metadata.
In another example, pars. [0071]-[0077] state that “information sent to the client can include: … Edge detection models, as described in the ’Tile selection algorithm based on edge detection’ section below,” but a luminance value is not an edge detection model.
In another example, Fig. 6 is described at par. [0120] et seq. Par. [0120] states that “This method may be used by the client if no tile map is provided by the server.” The method described in Fig. 6 does involve determining luminance values and thresholding them, but there is no disclosure of transmitting metadata indicating the luminance values as recited in amended claim 10.
Claim 10 does not comply with the written description requirement of 35 U.S.C. 112(a) because it requires receiving metadata indicative of a luminance value of the plurality of image portions, but the specification does not describe transmitting or receiving such metadata.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim(s) 16 is/are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 16 recites the limitation "the image portions" in lines 1-2. There is insufficient antecedent basis for this limitation in the claim.
Claim 16 depends from claim 1, which introduces “a plurality of image portions” at line 4, “a first group of one or more image portions” at line 10, and “a second group of one or more image portions” at line 17. It is unclear which of these previously-recited image portions are being referred to as “the image portions” in claim 16.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-2, 5, 7, 14, and 16-20 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by ‘Zhang’ (US 2017/0339431 A1; previously cited).
Regarding claim 1, Zhang discloses a computer-implemented (e.g., Figure 5) method for image upscaling at a client device (e.g., Figs. 1-2), the method comprising:
receiving, from a server device and by the client device (e.g., Fig. 1, compressed bit stream 110 is received; The device from which it is received is a server device and the device that receives it is the client device), an image which is one of a plurality of images forming an image stream (e.g., Fig. 1, bitstream 110 encodes a plurality of frames 130, including a first frame 131 and a second frame 132), wherein the image comprises a plurality of image portions (e.g., [0025], [0056], Fig. 1, each frame image is divided into plural blocks, each block being an image portion);
receiving, from the server device and by the client device, metadata relating to processing of the image (e.g., [0024]-[0025], Fig. 1, syntax elements 135), wherein the metadata indicates, for each of the plurality of image portions, that upscaling is foregone for the image portion or one of a plurality of available image upscaling processes to apply to the image portion (e.g., [0055] including Table 1, the syntax elements for each block indicate what type of block it is, which in turn indicates whether upscaling is foregone [for Inter zero residual blocks] or which of a plurality of available upscaling processes to apply to the block [SR algorithm is applied to Anchored-type blocks, while bicubic upsampling is applied to Intra-type blocks, for example]);
determining a first group of one or more image portions from the plurality of image portions to apply a first image upscaling process to from the plurality of available image upscaling processes based on the metadata for the first group of one or more image portions (e.g., [0055] including Table 1, determination based on metadata/syntax elements of which blocks are Anchored-type blocks, all of which will have an SR algorithm applied as a first image upscaling process);
selecting the first group of one or more image portions based on the determination (e.g., [0055] including Table 1, Anchored blocks are selected for SR algorithm);
applying the first image upscaling process to the first group of one or more image portions (e.g., [0055] including Table 1, SR algorithm is applied to the Anchored blocks);
retrieving, from a cache, a second group of one or more image portions to which a second image upscaling process of the plurality of available image upscaling processes was previously applied (e.g., [0025]-[0027], Fig. 1, second blocks that were previously upscaled for a prior frame are retrieved for insertion into a current frame; The plain meaning of a cache is a place where data is stored for later use and this plain meaning is not inconsistent with the specification; Wherever Zhang stores upscaled blocks from a prior frame so that they can be retrieved and inserted into a later frame falls within the BRI of a cache; See, e.g., [0062], Fig. 5, memory 506; Also note the interpretation of “second” – see Claim Interpretation above); and
generating an upscaled version of the image based at least in part on combining the first group of one or more image portions to which the first image upscaling process was applied and the second group of one or more image portions (e.g., [0027], Fig. 1, SR upscaling is applied to a portion of frame 170, while other portions are transferred from a previous frame – i.e., the first and second groups are combined to generate an upscaled version of the image).
Regarding claim 2, Zhang discloses the computer-implemented method of claim 1, wherein the first image upscaling process comprises a neural network based super resolution model (e.g., [0026], [0034], [0004], computationally-intensive super-resolution (SR) algorithm may be neural network based).
Regarding claim 5, Zhang discloses the computer-implemented method of claim 1, wherein the metadata comprises a tile map comprising the plurality of image portions (e.g., [0024]-[0025], Fig. 1, syntax element metadata maps locations of the block image portions), the tile map indicating which image portions to be selected (e.g., [0055] including table 1, syntax elements for each block indicate what type of block it is, which indicates which blocks are to be selected for SR algorithm upscaling).
Regarding claim 7, Zhang discloses the computer-implemented method of claim 1, wherein additional metadata is sent from the server device for each image of the plurality of images which form the image stream (e.g., [0024]-[0025], Fig. 1, syntax element metadata is received for each of plural images 130).
Regarding claim 14, Zhang discloses the computer-implemented method of claim 1, further comprising
selecting a third group of one or more image portions (e.g., [0055] including Table 1, Intra and/or Inter-predicted blocks with zero motion vector are selected as a third group of blocks); and
applying a third image upscaling process to the third group of one or more image portions (e.g., [0055] including Table 1, bicubic upsampling is applied to the “third” blocks), wherein the third image upscaling process is less computationally demanding than the first image upscaling process (e.g., [0034]).
Regarding claim 16, Zhang discloses the computer-implemented method of claim 1, wherein one or more of the image portions are received from the server device (Note the ‘112(b) rejection; See, e.g., mapping in claim 1 and Fig. 1, encoded bitstream 110 that includes the image portions is received from a server).
Regarding claim 17, Zhang discloses the method of claim 1, wherein a resolution of the image subsequent to the applying of the first image upscaling process to the first group of one or more image portions is 3840 x 2160 pixels or above (e.g., [0030], “3840x2160 (4K)”).
Regarding claim 18, Zhang discloses the computer-implemented method of claim 1, further comprising:
determining a fourth group of one or more image portions of the image to be used in a subsequent image in the image stream comprising at least the image and the subsequent image (e.g., [0028], further transfer processing into third frame, etc.), the method comprising:
calculating, for each image portion of the plurality of image portions, an average pixel intensity difference between the image and the subsequent image (e.g., [0040], “mean absolute magnitude of the residual block”; Note that the residual block indicates intensity difference between subsequent image block/portion and prediction from current image block/portion);
if the average pixel intensity difference of one or more of the plurality of image portions is below a predetermined threshold (e.g., [0040], less than
η
), adding the one or more image portions to the fourth group of one or more image portions and storing a location of the fourth group of one or more image portions (e.g., [0040], “If the magnitude of the residual is below the residual thresholdr [sic]
η
, a transfer is performed”; Note from [0019] that transferring a block means replicating/copying it from a first frame to a second frame); and
using the fourth group of one or more image portions in the subsequent image (e.g., [0027], Fig. 1).
Regarding claim 19, Zhang discloses a client computing device (e.g., Fig. 5) comprising one or more processors (Fig. 5, processor 502) that are associated with a memory (Fig. 5, memory 506), the one or more processors configured with executable instructions which, when executed, cause the client computing device to carry out the computer-implemented method of claim 1 (e.g., [0059]-[0060], [0064]-[0065]).
Regarding claim 20, Zhang discloses a system (e.g., Fig. 5) comprising,
a memory (Fig. 5, memory 506); and
one or more processors configured to perform the method of claim 1 (e.g., [0058]-[0060], [0064]-[0065]).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang and further in view of ‘Ozer’ (“Beginners’ Guide to Adaptive Bitrate Streaming,” 10 May 2021; previously cited).
Regarding claim 8, Zhang teaches the computer-implemented method of claim 5.
Zhang teaches (e.g., [0024], Fig. 1, bitstream 110 of image data and syntax elements) video streaming, but does not explicitly teach receiving a manifest file from the server device.
However, Ozer does teach techniques for video streaming that include receiving, from a server device, a manifest file (e.g., page 5, master playlist) comprising information relating to the interpretation of metadata at the client device (e.g., page 5, includes information relating to interpretation of specific media metadata at the client device, such as metadata pertaining to specific rungs on the encoding ladder), wherein the manifest file comprises one or more of:
i) a number of columns in a tile grid;
ii) a number of rows in a tile grid;
iii) a library each index of the plurality of indices and their related image upscaling process of the plurality of available image upscaling processes;
iv) an indication whether the server device supports embedding the tile map for each image in the plurality of images forming the image stream;
v) an indication of availability of an additional stream of data comprising full resolution data (e.g., Page 5, Fig. 4, top rung of encoding ladder is an indication of availability of an additional stream of data comprising full (i.e., highest) resolution data); and/or
vi) instructions for compositing the tiles in the tile grid to generate the image.
Ozer teaches that manifest files allow clients to access adaptive bitrate (ABR) streaming (e.g., page 4), which advantageously delivers optimum viewing experience for a range of client devices and connection speeds (e.g., page 2, 1st paragraph).
Ozer also demonstrates that manifest files can be used to communicate information about video coding from a server to a client (e.g., page 6). Yang requires sharing such information between client and server, such as a library matching each of different indices to a corresponding image upscaling process (e.g., [0353]-[0354]; [0363], correspondence between identifier and matching upscaling DNN settings needs to be provided to the client so that it can select appropriate upscaling/second DNN settings).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify the method of Zhang as applied above with the manifest file of Ozer in order to improve the method with the reasonable expectation that this would result in a method that allowed clients to access ABR streaming, thereby advantageously allowing delivery of optimum viewing experiences for a range of client devices and connection speeds, and/or allowed clients to receive necessary information about video coding. This technique for improving the method of Zhang was within the ordinary ability of one of ordinary skill in the art based on the teachings of Ozer.
Therefore, it would have been obvious to one of ordinary skill in the art to combine the teachings of Zhang and Ozer to obtain the invention as specified in claim 8.
Claim(s) 9, 10 and 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of ‘Wang’ (US 2024/0153033 A1; previously cited).
Regarding claim 9, Zhang teaches the method of claim 1.
Zhang selects groups of image portions to subject to computationally-expensive super resolution or computationally-inexpensive upsampling depending on various metadata (e.g., [0055]). Nevertheless, Zhang does not explicitly teach determining a third group of portions based on the metadata further comprising metadata indicative of image portions containing one or more salient regions of the image, wherein a salient region of the image comprises one or more image portions which have a saliency value above a predetermined threshold.
However, Wang does teach techniques for selecting groups of image portions to subject to computationally-expensive super resolution or computationally-inexpensive upsampling depending on various metadata (e.g., Fig. 1), including determining a third group of portions (i.e., salient image blocks that warrant using a neural-network-based super resolution algorithm – [0031]) based on the metadata further comprising metadata indicative of image portions containing one or more salient regions of the image (e.g., [0031], Fig. 1, object detection results metadata are communicated), wherein a salient region of the image comprises one or more image portions which have a saliency value above a predetermined threshold (e.g., [0031], object recognition is applied “to find objects in the image that are sufficiently important to warrant using a NN algorithm on a section with an important object”).
The purpose of Zhang’s FAST technique is to accelerate video upscaling by applying super resolution (SR) to only a subset of video data (e.g., [0019]). Wang’s teachings indicate that video SR can be further accelerated by applying computation-intensive, neural-network-based SR only to important, salient regions (e.g., [0039], [0026], [0029]).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify the method of Zhang with the saliency-based portion selection of Wang in order to improve the method with the reasonable expectation that this would result in a method that could further accelerate video upscaling. This technique for improving the method of Zhang was within the ordinary ability of one of ordinary skill in the art based on the teachings of Wang.
Therefore, it would have been obvious to one of ordinary skill in the art to combine the teachings of Zhang and Wang to obtain the invention as specified in claim 9.
Regarding claim 10, Zhang teaches the method of claim 1.
Zhang selects groups of image portions to subject to computationally-expensive super resolution or computationally-inexpensive upsampling depending on various metadata (e.g., [0055]). Nevertheless, Zhang does not explicitly teach determining a third group of image portions based on metadata indicative of a luminance value of the plurality of image portions, and selecting one or more image portions having a luminance value above a predetermined threshold.
However, Wang does teach techniques for selecting groups of image portions to subject to computationally-expensive super resolution or computationally-inexpensive upsampling depending on various metadata (e.g., Fig. 1), including determining a third group of image portions based on metadata indicative of a luminance value of the plurality of image portions (e.g., [0055], [0053], groups of image portions/sections with same/similar luminance standard deviation values are identified), and selecting one or more image portions having a luminance value above a predetermined threshold (e.g., [0054], image sections/portions with luminance standard deviation values over threshold are selected for NN-based super resolution processing).
The purpose of Zhang’s FAST technique is to accelerate video upscaling by applying super resolution (SR) to only a subset of video data (e.g., [0019]). Wang’s teachings indicate that video SR can be further accelerated by applying computation-intensive, neural-network-based SR only to highly-textured image portions where noticeable quality improvement may be achieved (e.g., [0026]-[0027]). These textured (i.e., more complex, less smooth) regions are identified by Wang’s luminance value threshold (e.g., [0053]-[0055]).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify the method of Zhang with the saliency-based portion selection of Wang in order to improve the method with the reasonable expectation that this would result in a method that could further accelerate video upscaling. This technique for improving the method of Zhang was within the ordinary ability of one of ordinary skill in the art based on the teachings of Wang.
Therefore, it would have been obvious to one of ordinary skill in the art to combine the teachings of Zhang and Wang to obtain the invention as specified in claim 10.
Regarding claim 12, Zhang teaches the method of claim 1.
Zhang selects groups of image portions to subject to computationally-expensive super resolution or computationally-inexpensive upsampling depending on various metadata (e.g., [0055]). While Zhang recognizes that high-performance upscaling may be too computationally-expensive for some client devices (e.g., [0004]), Zhang generally tunes its thresholds to maximize image quality (e.g., [0041], maximizing PSNR).
Zhang does not explicitly teach applying a local calibration test on the client device to determine a calibration score to determine an upper limit on a number of image portions to which the first image upscaling process can be applied.
However, Wang does teach applying a local calibration test on the client device to determine a calibration score to determine an upper limit on a number of image portions to which the first image upscaling process can be applied (e.g., [0038], application developer, device manufacturer, etc., can adjust threshold to make an appropriate trade-off between performance gain and computation cost; e.g., [0037], the threshold controls and determines an upper limit on the number of image portions to which the first/NN image upscaling process can be applied – i.e., the lower the threshold, the fewer the portions to which first/NN upscaling will be applied, and vice versa; For at least this reason, the threshold can be seen as a calibration score and the process of setting it can be seen as a calibration test).
Wang’s techniques advantageously consider the tradeoff between image quality and computational expense and allow an adjustment of the number of regions of interest in accordance with a given client device’s capability. Considering this tradeoff is advantageous because user experience is affected by more than just image quality. For example, even if given very high-quality video, a user’s experience may be relatively poorer if that video freezes and buffers due to a client device’s computational capability being exceeded.
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify the method of Zhang with the calibration test of Wang in order to improve the method with the reasonable expectation that this would result in a method that advantageously considered the tradeoff between image quality and computational expense. This technique for improving the method of Zhang was within the ordinary ability of one of ordinary skill in the art based on the teachings of Wang.
Therefore, it would have been obvious to one of ordinary skill in the art to combine the teachings of Zhang and Wang to obtain the invention as specified in claim 12.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GEOFFREY E SUMMERS whose telephone number is (571)272-9915. The examiner can normally be reached Monday-Friday, 7:00 AM to 3:30 PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chan Park can be reached at (571) 272-7409. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GEOFFREY E SUMMERS/Examiner, Art Unit 2669
Prosecution Insights