DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
1. This action is in response to the amendment filed on 12/4/2025. Claims 1-3, 5, 13, and 17-18 have been amended. Claims 1-20 remain rejected in the application. Applicant’s amendments to the specification have overcome each and every objection previously set forth in the Non-Final Office Action mailed 9/4/2025.
Response to Arguments
2. Applicant’s arguments with respect to claim 1, and similarly claims 17 and 18, with respect to the rejection under 35 U.S.C. 102 regarding that the prior art does not teach the limitation(s): “based on an interpolation feature describing complexity of image content of a first pixel block in the first image, selecting between a first interpolation and a second interpolation for upscaling the first pixel block” have been considered but are moot because of the new ground of rejection. The claims are now disclosed by Lin and Matsuoka.
3. Regarding arguments to claims 2-16 and 19-20, they are dependent on independent claims 1 and 18 respectively. Applicant does not argue anything other than independent claim 1, and similarly claims 17 and 18. The limitations in those claims, in conjunction with their combination, has previously been established and explained.
Claim Rejections - 35 USC § 103
4. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
5. Claims 1-2, 7-13, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Lin et al. (US-9123138-B2, hereinafter "Lin") in view of Matsuoka (US-6263120-B1).
6. As per claim 1, Lin discloses: An image processing method executed in a computer device, the method comprising:
acquiring a first image having a first resolution; (Lin, col. 4, lines 64-67, “An input image is upscaled by a scale factor to generate an upscaled input image (block 402). For example, input image “It” may be processed by an upscale module 308 to increase a resolution of the input image ...” and col. 3, lines 33-34, “The computing device 102 is illustrated as including an image processing module 104.”)
based on an interpolation feature describing complexity of image content of a first pixel block in the first image, [[selecting between]] a first interpolation and a second interpolation for upscaling the first pixel block; (Lin, col. 9, lines 8-13, “Additionally, variations to the techniques may be made to address different color spaces and color channels. For example, the upscale techniques employed by the image upscale system 110 may be applied to a luminance channel “Y,” and the result may be combined with different algorithms applied to the “Cb” and “Cr” color channels.” and col. 6, lines 35-38, “As before, the factor upscale module 210 may be used to identify which of a plurality of patches in a neighborhood of a location in an image is most similar to a patch in an upscaled input image (block 602).” and col. 5, lines 1-3, “A variety of different techniques may be employed to perform this upscaling, an example of which is Bicubic interpolation.” and col. 8, lines 23-28, ”Accordingly, the image upscale system 110 may be configured to adaptively limit a number of iterations performed by the factor upscale module 210, and then use an adjustment upscale module 216 (e.g., analytical interpolation) to resize the output ...”)
obtaining an interpolated pixel block by:
performing the first interpolation on the first pixel block if the interpolation feature of the first pixel block does not satisfy a feature determination condition regarding complexity of the image content of the first pixel block; or
performing the second interpolation on the first pixel block if the interpolation feature satisfies the feature determination condition; and (Lin, col. 5, lines 1-3, “A variety of different techniques may be employed to perform this upscaling, an example of which is Bicubic interpolation.” and col. 8, lines 23-28, ”Accordingly, the image upscale system 110 may be configured to adaptively limit a number of iterations performed by the factor upscale module 210, and then use an adjustment upscale module 216 (e.g., analytical interpolation) to resize the output ...” and col. 2, line 59-col. 3, line 3, “In a further example, algorithm parameters may be adapted with respect to algorithm iterations, which may be performed to increase efficiency of computing device resource utilization and speed of performance. For instance, algorithm parameters may be adapted to enforce a minimum and/or maximum number to iterations, cease iterations for image sizes over a threshold amount, set sampling step sizes for patches, employ techniques based on color channels (which may include independence and joint processing techniques), and so on as further described in the Iteration-Adaptive Parameter Update Section.” and col. 9, lines 8-13, “Additionally, variations to the techniques may be made to address different color spaces and color channels. For example, the upscale techniques employed by the image upscale system 110 may be applied to a luminance channel “Y,” and the result may be combined with different algorithms applied to the “Cb” and “Cr” color channels.” and col. 9, lines 30-38, “Yet further, at each iteration performed by the factor upscale module 210, a detail-preserving upscaling algorithm may be employed to upscale its input by a factor of “s.” If the original image is noisy, for instance, an image denoising algorithm may be applied before the upscaling iterations to suppress noise boosting. Additionally, in the initial denoising step, the search region may be adapted by setting its radius proportional to the input denoising parameter to speed-up the process for images with relatively small amounts of noise.”)
outputting a second image with a second resolution based on the interpolated pixel block, the second resolution being greater than the first resolution, wherein the first interpolation and the second interpolation comprise up-sampling of the first pixel block, and computational resource consumption of the second interpolation is greater than computational resource consumption of the first interpolation. (Lin col. 8, lines 48-53, “Further, the image upscale system 110 may be configured to cease performance of the iterations if the image size would exceed a threshold. For example, an expression “w×h>M” may be employed where “w” and “h” are the width and height of the input image for current iteration and “M” is the threshold.” and col. 8, lines 23-33, “Accordingly, the image upscale system 110 may be configured to adaptively limit a number of iterations performed by the factor upscale module 210, and then use an adjustment upscale module 216 (e.g., analytical interpolation) to resize the output of the factor upscale module 210 to the target image size of the upscaled image data 112. In such a way, efficiency of the image upscale system 110 may be increased (e.g., in terms of computational resources and time spent performing the processing) with little compromise on the final image quality. This may be performed in a variety of ways.” and col. 9, lines 39-42, “In a further example, variations may be made for setting algorithm parameters. For example, a scale factor “s” may be set differently over iterations, e.g. increasing “s” over iterations in the range of 1:1 to 1:5.”)
7. Lin doesn't explicitly disclose but Matsuoka discloses: [[based on an interpolation feature describing complexity of image content of a first pixel block in the first image,]] selecting between [[a first interpolation and a second interpolation for upscaling the first pixel block;]] (Matsuoka, Fig. 1; Abstract, “An original image containing different kinds of components-areas is adaptively processed by applying interpolating methods suitably selected for respective areas. The processing method comprises the steps of extracting data of a partial image from data of a multi-gradation original image (STEP 1), ... selecting a filter for surface interpolation or linear interpolation on the basis of the discrimination result (STEP 5) and interpolating the data of the partial image (STEP 6).” and col. 4, lines 46-52, “The filter selecting step (STEP 5) selects either a bilinear interpolation filter or a cubic-convolution interpolation filter depending on whether the partial image contains or does not contains an edge portion (i.e., according to the discriminating result). The image-data interpolating step (STEP 6) executes interpolation of the partial image data by using the selected filter.” and col. 2, lines 16-23, “An object of the present invention is to provide an image-data interpolation processing method that can easily process an image including different kinds of image areas (e.g., a character image area and a photographic-image area) through performing adaptive interpolation to create the high-resolution character image-area and the smooth gradational photographic image area with no need of separation of areas before interpolation, ...”)
8. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of Lin to include selecting between a first interpolation and a second interpolation for upscaling a first pixel block based on an interpolation feature of Matsuoka. The motivation for this modification could have been to adaptively apply an interpolation method to best suit the pixel block region. For instance, one interpolation method might be better for handling text instead of photographic regions. In addition, an interpolation method might also be chosen based on computational resources. For instance, an interpolation method might be chosen for producing high-quality results (utilizing heavy computation resources) whereas another method might be chosen to generate fast results for a real-time application by sacrificing image quality.
9. As per claim 2, Lin in view of Matsuoka discloses: The method according to claim 1, further comprising determining the interpolation feature by:
calculating the interpolation feature according to a plurality of second pixel blocks, the plurality of second pixel blocks comprising adjacent pixel blocks located around the first pixel block. (Lin col. 6, lines 35-38, “As before, the factor upscale module 210 may be used to identify which of a plurality of patches in a neighborhood of a location in an image is most similar to a patch in an upscaled input image (block 602).”)
10. As per claim 7, Lin in view of Matsuoka discloses: The method according to claim 1, further comprising:
determining the feature determination condition according to the first image. (Lin, col. 9, lines 8-13, “Additionally, variations to the techniques may be made to address different color spaces and color channels. For example, the upscale techniques employed by the image upscale system 110 may be applied to a luminance channel “Y,” and the result may be combined with different algorithms applied to the “Cb” and “Cr” color channels.” and col. 6, lines 35-38, “As before, the factor upscale module 210 may be used to identify which of a plurality of patches in a neighborhood of a location in an image is most similar to a patch in an upscaled input image (block 602).” and col. 9, lines 30-38, “Yet further, at each iteration performed by the factor upscale module 210, a detail-preserving upscaling algorithm may be employed to upscale its input by a factor of “s.” If the original image is noisy, for instance, an image denoising algorithm may be applied before the upscaling iterations to suppress noise boosting. Additionally, in the initial denoising step, the search region may be adapted by setting its radius proportional to the input denoising parameter to speed-up the process for images with relatively small amounts of noise.”)
11. As per claim 8, Lin in view of Matsuoka discloses: The method according to claim 7, wherein the determining the feature determination condition comprises:
determining the feature determination condition according to position information about the first pixel block in the first image. (Lin, col. 6, lines 35-42, “As before, the factor upscale module 210 may be used to identify which of a plurality of patches in a neighborhood of a location in an image is most similar to a patch in an upscaled input image (block 602). For example, the content-adaptive patch finding module 502 of the factor upscale module 210 may be employed as part of a search for a most-similar patch in the local neighborhood 314 in smoothed image “B” 306 to patch from the upscaled image “U” 310.”)
12. As per claim 9, Lin in view of Matsuoka discloses: The method according to claim 8, wherein the determining the feature determination condition further comprises:
determining that the feature determination condition comprises complexity of image content of the first pixel block exceeding a first target threshold if a position of the first pixel block is within a target region that is a partial region of the first image; or (Lin, col. 6, line 62-65, “A threshold is then applied using the computed image distances to determine whether to use the identified patch or the patch at the location in the image to predict a patch for use in generating the upscaled image (block 606).” and col. 6, lines 35-45, “As before, the factor upscale module 210 may be used to identify which of a plurality of patches in a neighborhood of a location in an image is most similar to a patch in an upscaled input image (block 602). ... This similarity may be expressed as relative “image distances” between the patches and used as part of a determination as to which of the patches are to be used as part of the upscale operation.” and col. 6, lines 49-53, “The content-adaptive patch finding module 402, for instance, may estimate “p2” as the position “p” in the local neighborhood of “p1” (which is the center location), such that “∥U(q1)−B(p)∥” is minimized. An image distance may then be computed between the patch in the upscaled input image and the identified patch (block 604).”)
determining that the feature determination condition comprises the complexity of the image content of the first pixel block exceeding a second target threshold if the position of the first pixel block is outside the target region, wherein the first target threshold is less than the second target threshold. (Lin, col. 6, line 62-65, “A threshold is then applied using the computed image distances to determine whether to use the identified patch or the patch at the location in the image to predict a patch for use in generating the upscaled image (block 606).” and col. 7, lines 14-17, “In this way, preference may be given to the “in place” patch. For instance, the threshold may be used such that an “in place” patch is used unless a patch is found such that a ratio of the image distances is closer than the threshold.” and col. 6, lines 49-54, “The content-adaptive patch finding module 402, for instance, may estimate “p2” as the position “p” in the local neighborhood of “p1” (which is the center location), such that “∥U(q1)−B(p)∥” is minimized. An image distance may then be computed between the patch in the upscaled input image and the identified patch (block 604).”)
13. As per claim 10, Lin in view of Matsuoka discloses: The method according to claim 8, wherein the determining the feature determination condition further comprises:
determining the feature determination condition according to image content of the first image and the position information about the first pixel block in the first image. (Lin, col. 9, lines 8-13, “Additionally, variations to the techniques may be made to address different color spaces and color channels. For example, the upscale techniques employed by the image upscale system 110 may be applied to a luminance channel “Y,” and the result may be combined with different algorithms applied to the “Cb” and “Cr” color channels.” and col. 6, lines 35-38, “As before, the factor upscale module 210 may be used to identify which of a plurality of patches in a neighborhood of a location in an image is most similar to a patch in an upscaled input image (block 602).” and col. 9, lines 30-38, “Yet further, at each iteration performed by the factor upscale module 210, a detail-preserving upscaling algorithm may be employed to upscale its input by a factor of “s.” If the original image is noisy, for instance, an image denoising algorithm may be applied before the upscaling iterations to suppress noise boosting. Additionally, in the initial denoising step, the search region may be adapted by setting its radius proportional to the input denoising parameter to speed-up the process for images with relatively small amounts of noise.”)
14. As per claim 11, Lin in view of Matsuoka discloses: The method according to claim 10, wherein the determining the feature determination condition further comprises:
determining an image body region in the first image according to the image content of the first image; and (Lin, col. 6, lines 35-45, “As before, the factor upscale module 210 may be used to identify which of a plurality of patches in a neighborhood of a location in an image is most similar to a patch in an upscaled input image (block 602). ... This similarity may be expressed as relative “image distances” between the patches and used as part of a determination as to which of the patches are to be used as part of the upscale operation.”)
determining that the feature determination condition comprises either:
complexity of image content of the first pixel block exceeding a third target threshold if a position of the first pixel block is within the image body region; or (Lin, col. 6, lines 35-45, “As before, the factor upscale module 210 may be used to identify which of a plurality of patches in a neighborhood of a location in an image is most similar to a patch in an upscaled input image (block 602). ... This similarity may be expressed as relative “image distances” between the patches and used as part of a determination as to which of the patches are to be used as part of the upscale operation.” and col. 6, line 62-65, “A threshold is then applied using the computed image distances to determine whether to use the identified patch or the patch at the location in the image to predict a patch for use in generating the upscaled image (block 606).” and col. 6, lines 35-45, “As before, the factor upscale module 210 may be used to identify which of a plurality of patches in a neighborhood of a location in an image is most similar to a patch in an upscaled input image (block 602). ... This similarity may be expressed as relative “image distances” between the patches and used as part of a determination as to which of the patches are to be used as part of the upscale operation.” and col. 6, lines 49-53, “The content-adaptive patch finding module 402, for instance, may estimate “p2” as the position “p” in the local neighborhood of “p1” (which is the center location), such that “∥U(q1)−B(p)∥” is minimized. An image distance may then be computed between the patch in the upscaled input image and the identified patch (block 604).”)
complexity of the image content of the first pixel block exceeding a fourth target threshold if the position of the first pixel block is outside the image body region, wherein the third target threshold is less than the fourth target threshold. (Lin, col. 6, lines 35-45, “As before, the factor upscale module 210 may be used to identify which of a plurality of patches in a neighborhood of a location in an image is most similar to a patch in an upscaled input image (block 602). ... This similarity may be expressed as relative “image distances” between the patches and used as part of a determination as to which of the patches are to be used as part of the upscale operation.” and col. 6, line 62-65, “A threshold is then applied using the computed image distances to determine whether to use the identified patch or the patch at the location in the image to predict a patch for use in generating the upscaled image (block 606).” and col. 6, lines 35-45, “As before, the factor upscale module 210 may be used to identify which of a plurality of patches in a neighborhood of a location in an image is most similar to a patch in an upscaled input image (block 602). ... This similarity may be expressed as relative “image distances” between the patches and used as part of a determination as to which of the patches are to be used as part of the upscale operation.” and col. 6, lines 49-53, “The content-adaptive patch finding module 402, for instance, may estimate “p2” as the position “p” in the local neighborhood of “p1” (which is the center location), such that “∥U(q1)−B(p)∥” is minimized. An image distance may then be computed between the patch in the upscaled input image and the identified patch (block 604).” and col. 7, lines 14-17, “In this way, preference may be given to the “in place” patch. For instance, the threshold may be used such that an “in place” patch is used unless a patch is found such that a ratio of the image distances is closer than the threshold.”)
15. As per claim 12, Lin in view of Matsuoka discloses: The method according to claim 11, wherein the determining an image body region in the first image according to the image content of the first image comprises:
invoking a first image recognition model to identify a target object in the first image, and determining a display region of the target object as the image body region in the first image; or (Lin, col. 6, lines 35-48, “As before, the factor upscale module 210 may be used to identify which of a plurality of patches in a neighborhood of a location in an image is most similar to a patch in an upscaled input image (block 602). For example, the content-adaptive patch finding module 502 of the factor upscale module 210 may be employed as part of a search for a most-similar patch in the local neighborhood 314 in smoothed image “B” 306 to patch from the upscaled image “U” 310. This similarity may be expressed as relative “image distances” between the patches and used as part of a determination as to which of the patches are to be used as part of the upscale operation. The search may be performed to give preference to a patch at an “in place” location as opposed to patches at other locations to reduce likelihood of blurred edges and texture.” and col. 7, lines 53-56, “Weights are then assigned to the identified patch (i.e., the most similar patch) and the patch at the location in the image (i.e., the in-place patch) based on respective content metrics (block 808).”)
invoking a second image recognition model to determine an image type of the first image in the first image, and determining a corresponding image body region according to the image type. (Lin, col. 7, lines 40-43, “The content metric may describe consistency of gradient orientations to distinguish informative local patches (such as edges and corners) versus uniform or noisy patches.” and col. 9, lines 32-38, “If the original image is noisy, for instance, an image denoising algorithm may be applied before the upscaling iterations to suppress noise boosting. Additionally, in the initial denoising step, the search region may be adapted by setting its radius proportional to the input denoising parameter to speed-up the process for images with relatively small amounts of noise.”)
16. As per claim 13, Lin in view of Matsuoka discloses: The method according to claim 1, wherein the performing the first interpolation comprises:
performing the first interpolation on the first pixel block according to third pixel blocks to obtain the interpolated pixel block, wherein the third pixel blocks comprise adjacent pixel blocks located around the first pixel block; or (Lin, col. 8, lines 23-28, “Accordingly, the image upscale system 110 may be configured to adaptively limit a number of iterations performed by the factor upscale module 210, and then use an adjustment upscale module 216 (e.g., analytical interpolation) to resize the output ...” and, col. 6, lines 35-38, “As before, the factor upscale module 210 may be used to identify which of a plurality of patches in a neighborhood of a location in an image is most similar to a patch in an upscaled input image (block 602).” and col. 5, line 62-col. 6, line 2, “Portions of patches “It+1(q1)” that are overlapping may then be averaged to arrive at a result for the image upscale iteration “It+1” 316. This process may then be repeated to predict patches for each location of the upscaled image data. Further, as previously described in relation to FIG. 2, this technique may be repeated for a plurality of different iterations such that the image upscale iteration “It+1” is used as the input image “It” 302 for the next iteration.”)
the performing the second interpolation comprises:
performing the second interpolation on the first pixel block according to fourth pixel blocks to obtain the interpolated pixel block, wherein the fourth pixel blocks comprise adjacent pixel blocks located around the first pixel block, and wherein a number of the fourth pixel blocks is greater than a number of the third pixel blocks. (Lin, col. 8, lines 23-28, “Accordingly, the image upscale system 110 may be configured to adaptively limit a number of iterations performed by the factor upscale module 210, and then use an adjustment upscale module 216 (e.g., analytical interpolation) to resize the output ...” and, col. 6, lines 35-38, “As before, the factor upscale module 210 may be used to identify which of a plurality of patches in a neighborhood of a location in an image is most similar to a patch in an upscaled input image (block 602).” and col. 5, lines 25-30, “For instance, the blurred and smoothed images may be processed by a prediction module 312 to make a prediction to upscale the input image “It” 302. The underlying approach may be based on patches, which may include predefined areas of an image, e.g., five by five pixels although other examples are also contemplated.” and col. 9, lines 39-42, “In a further example, variations may be made for setting algorithm parameters. For example, a scale factor “s” may be set differently over iterations, e.g. increasing “s” over iterations in the range of 1:1 to 1:5.”)
17. Claim 17, which is similar in scope to claim 1, is thus rejected under the same rationale as described above. In addition, the rational for combining Matsuoka with Lin is the same as claim 1 above.
18. Claims 3-5 are rejected under 35 U.S.C. 103 as being unpatentable over Lin et al. (US-9123138-B2, hereinafter "Lin") in view of Matsuoka (US-6263120-B1), and further in view of Hoshii et al. (US-2002/0015162-A1, hereinafter "Hoshii").
19. As per claim 3, Lin in view of Matsuoka discloses: The method according to claim 2, wherein color information about the first image comprises a luminance factor; and
the method comprises determining the interpolation feature by: (Lin, col. 9, lines 8-13, “Additionally, variations to the techniques may be made to address different color spaces and color channels. For example, the upscale techniques employed by the image upscale system 110 may be applied to a luminance channel “Y,” and the result may be combined with different algorithms applied to the “Cb” and “Cr” color channels.” and col. 5, lines 1-3, “A variety of different techniques may be employed to perform this upscaling, an example of which is Bicubic interpolation.”)
20. Lin in view of Matsuoka doesn't explicitly disclose but Hoshii discloses: calculating a direction feature of the first pixel block according to luminance factors of the plurality of second pixel blocks, wherein the direction feature describes a luminance difference between the first pixel block and the plurality of second pixel blocks; and
determining the direction feature as the interpolation feature. (Hoshii, page 13, ¶ [0142], “At the same time, the edge angle and line thickness of the source pixels are decided, based on the luminance values of given pixels that fall within the zone comprising 5×5 pixels.” and page 11, ¶ [0142], “It is assumed that the pattern of the pixels shown in FIG. 11B matches one of the prepared edge patterns. Because of matching with an edge pattern, the pattern shown in FIG. 11B shows a characteristic that the difference between the luminance values of the pixels positioned on the upper side in the j direction of the figure and those of the pixels positioned on the lower side in the j direction of the figure tends to be great, but the luminance difference in the horizontal direction on paper is little. Thus, this pattern is regarded as a horizontally parallel edge. To this edge pattern, pixel interpolation processing is then executed, according to a predetermined rule.”)
21. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of claim 2 of Lin in view of Matsuoka to include calculating a direction feature of the first pixel block according to luminance factors of the plurality of second pixel blocks, wherein the direction feature describes a luminance difference between the first pixel block and the plurality of second pixel blocks and determine the direction feature as the interpolation feature of Hoshii. The motivation for this modification could have been to improve an interpolation technique where accounting for the direction of the luminance difference between two pixel blocks, the interpolation can account for image features, such as edges and produce higher quality results.
22. As per claim 4, Lin in view of Matsuoka, and further in view of Hoshii discloses: The method according to claim 3, wherein the calculating a direction feature of the first pixel block according to luminance factors of the plurality of second pixel blocks comprises:
determining luminance differences between the first pixel block and the plurality of second pixel blocks in a first direction and a second direction; and (Hoshii, page 13, ¶ [0142], “At the same time, the edge angle and line thickness of the source pixels are decided, based on the luminance values of given pixels that fall within the zone comprising 5×5 pixels.” and page 15, ¶ [0200], “When the 16 grid points surrounding the point Puv to be interpolated are given their tone values, the luminance of the point Puv is determined, subject to the influence of these tone values on the point Puv.”)
encapsulating the luminance differences into two-dimensional floating-point data to determine a luminance feature of the first pixel block; and (Hoshii, page 15, ¶ [0201], “The 16 grid points act on the point Puv to be interpolated with the degree of their influence depending on their distance from the point Puv, which is expressed as described above. The degree of influence of the tone data of all grid points on the point Puv in the X and Y directions in the aggregate can be expressed in a general equation as below ...”)
determining a sum of a first direction component and a second direction component of the luminance feature as the direction feature of the first pixel block, wherein the first direction and the second direction are perpendicular to each other in the first image. (Hoshii, page 12, ¶ [0159], “For the calculation to obtain the tone values of pixels b, c, and f of interpolations, source pixels to be used differ, according to the luminance values of given pixels that fall within the zone comprising 5×5 pixels. If the pattern is a right-angled edge, source L, M, R pixels are used to generate. If the pattern is not a right-angled edge, source G, H, N, and S pixels are used to generate pixels b, c, and f of interpolations.” and page 12, ¶ [0158], “In the present embodiment of the invention, interpolation processing is executed for a horizontal edge, according to the above rules, with luminance change in the direction perpendicular to the edge being taken into consideration.” and page 12, ¶ [0150],”In the present embodiment of the invention, interpolation is executed such that change of luminance in the direction perpendicular to the above horizontally lengthening edge is also reflected in the interpolation. If luminance changes in the direction perpendicular to the horizontally lengthening edge, the tone values of the source Q, R, and S pixels are assigned to Pn so that the appearance of edge change will be reflected in the tone values of the g, h, and i pixels of interpolations.”)
23. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of claim 3 of Lin in view of Matsuoka to include determining the luminance differences between the first pixel block and the plurality of second pixel blocks in a first direction and a second direction, encapsulating the luminance differences into two-dimensional floating-point data, and determining a sum of a first direction component and a second direction component of the luminance feature wherein the first direction and the second direction are perpendicular to each other of Hoshii. The motivation for this modification could have been to improve an interpolation technique where accounting for the direction of the luminance difference between two pixel blocks, the interpolation can account for image features, such as edges and produce higher quality results. Encapsulating the luminance difference also provides a distribution of values for the interpolation method to consider. By also accounting for luminance directions that are perpendicular to each other, this will help account for right angle gradients or structure in an image during interpolation.
24. As per claim 5, Lin in view of Matsuoka, and further in view of Hoshii discloses: The method according to claim 4, wherein the determining luminance differences comprises:
determining a first luminance difference of the first pixel block in the first direction according to a difference in luminance factor between: (Hoshii, page 13, ¶ [0142], “At the same time, the edge angle and line thickness of the source pixels are decided, based on the luminance values of given pixels that fall within the zone comprising 5×5 pixels.” and page 15, ¶ [0200], “When the 16 grid points surrounding the point Puv to be interpolated are given their tone values, the luminance of the point Puv is determined, subject to the influence of these tone values on the point Puv.”)
a second pixel block at a front side of the first pixel block; and
a second pixel block at a rear side of the first pixel block in the first direction in the plurality of second pixel blocks; and (Hoshii, page 15, ¶ [0200], “When the 16 grid points surrounding the point Puv to be interpolated are given their tone values, the luminance of the point Puv is determined, subject to the influence of these tone values on the point Puv. ... When taking notice of the X axis direction, we express the distance of the above 16 grid points from the point Puv as follows: the distance to the outer left grid points is x1; the distance to the inner left grid points is x2; the distance to the inner right grid points is x3; and the distance to the outer right grid points is x4. We use function f (x) to express the degree of influence of the tone of the gird points on the luminance of the point Puv, according to the above distance.”)
determining a second luminance difference of the first pixel block in the second direction according to a difference in luminance factor between: (Hoshii, page 13, ¶ [0142], “At the same time, the edge angle and line thickness of the source pixels are decided, based on the luminance values of given pixels that fall within the zone comprising 5×5 pixels.” and page 15, ¶ [0200], “When the 16 grid points surrounding the point Puv to be interpolated are given their tone values, the luminance of the point Puv is determined, subject to the influence of these tone values on the point Puv.”)
a second pixel block at a front side of the first pixel block; and
a second pixel block at a rear side of the first pixel block in the second direction in the plurality of second pixel blocks; and (Hoshii, page 15, ¶ [0200], “When the 16 grid points surrounding the point Puv to be interpolated are given their tone values, the luminance of the point Puv is determined, subject to the influence of these tone values on the point Puv. ... When taking notice of the Y axis direction, we express the distance of the above 16 grid points from the point Puv as follows: the distance to the top grid points is y1, the distance to the inner grid points above the point Puv is y2, the distance to the inner grid points below the point Puv is y3, and the distance to the bottom grid points is x4. Similarly, we use function f (y) to express the degree of influence of the tone of the gird points on the luminance of the point Puv, according to the above distance.”)
the encapsulating the luminance differences comprises encapsulating the first luminance difference and the second luminance difference into the two-dimensional floating-point data to determine the luminance feature of the first pixel block. (Hoshii, page 15, ¶ [0200], “We use function f (x) to express the degree of influence of the tone of the gird points on the luminance of the point Puv, according to the above distance. ... Similarly, we use function f (y) to express the degree of influence of the tone of the gird points on the luminance of the point Puv, according to the above distance.”)
25. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of claim 4 of Lin in view of Matsuoka to include determining a first luminance difference of the first pixel block in the first direction according to a difference in luminance factor between a second pixel block at a front side and rear side of the first pixel block, determining a second luminance difference of the first pixel block in the second direction according to a difference in luminance factor between a second pixel block at a front side and rear side of the first pixel block, and encapsulate the luminance differences into the two-dimensional floating-point data value of Hoshii. The motivation for this modification could have been to improve an interpolation technique where accounting for the direction of the luminance difference between two pixel blocks, the interpolation can account for image features, such as edges and produce higher quality results. Encapsulating the luminance difference also provides a distribution of values for the interpolation method to consider.
26. Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Lin et al. (US-9123138-B2, hereinafter "Lin") in view of Matsuoka (US-6263120-B1), and further in view of Tang (US-10339632-B2).
27. As per claim 6, Lin in view of Matsuoka discloses: The method according to claim 1, further comprising:
dividing the first image into at least two pixel blocks according to a division rule, the first pixel block being any pixel block of the at least two pixel blocks; and
wherein the outputting further comprises: (Lin, col. 6, lines 35-38, “As before, the factor upscale module 210 may be used to identify which of a plurality of patches in a neighborhood of a location in an image is most similar to a patch in an upscaled input image (block 602).” and col. 5, lines 25-30, “For instance, the blurred and smoothed images may be processed by a prediction module 312 to make a prediction to upscale the input image “It” 302. The underlying approach may be based on patches, which may include predefined areas of an image, e.g., five by five pixels although other examples are also contemplated.”)
28. Lin in view of Matsuoka doesn't explicitly disclose but Tang discloses: concatenating the interpolated pixel block and the second image according to a combination rule that comprises an inverse ordering rule to the division rule. (Tang, col. 14, lines 36-53, “The determining module 410 is configured to determine a high-frequency region of the color-block image. The first converting module 420 is configured to convert a part of the color-block image within the high-frequency region into a first image using a first interpolation algorithm. The first image includes first simulation pixels arranged in an array, and each photosensitive pixel 212 corresponds to one first simulation pixel. The second converting module 430 is configured to convert a part of the color-block image beyond the high-frequency region into a second image using a second interpolation algorithm. The second image includes second simulation pixels arranged in an array, and each photosensitive pixel 212 corresponds to one second simulation pixel. A complexity of the second interpolation algorithm is less than that of the first interpolation algorithm. The merging module 440 is configured to merge the first image and the second image into a simulation image corresponding to the color-block image.”)
29. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of claim 1 of Lin in view of Matsuoka to include concatenating the interpolated pixel block and the second image according to a combination rule that comprises an inverse ordering rule to the division rule of Tang. The motivation for this modification could have been used to assist the interpolation process so that the process by more efficiently recombine and up-scale an image based on the pixel block’s complexity.
30. Claims 14-16 and 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Lin et al. (US-9123138-B2, hereinafter "Lin") in view of Matsuoka (US-6263120-B1), and further in view of Colenbrander (US-11731043-B2).
31. As per claim 14, Lin in view of Matsuoka discloses: The method of claim 1, wherein the computer device comprises … (see rejection of claim 1 above).
32. Lin in view of Matsuoka doesn't explicitly disclose but Colenbrander discloses: a game device, wherein the first resolution comprises an output resolution of a game engine, and the second resolution comprises a display resolution of the game device, and wherein acquiring the first image comprises acquiring the first image from the game engine. (Colenbrander, col. 5, lines 13-18, “By way of example without limitation, the client device 122 can be a game console ... capable of streaming gameplay of a video game from a cloud gaming provider as in the present disclosure.” and col. 1, line 65-col. 2, line 1, “In some implementations, a method is provided, including the following operations: executing a video game by a cloud game machine, the execution of the video game includes rendering gameplay video ...” and col. 4, lines 19-30, “There are a few settings which can be adjusted such as the bitrate of the audio or video encoder, frame rate and video resolution. However, these knobs are suboptimal because the streaming software receives completed audio or video frames to encode. While the server can adjust resolution, it does so by scaling the already rendered image ... These adjustments ... process video and audio that has already been rendered.” and col. 9, lines 25-29, “In current cloud gaming systems, the streaming logic/server receives the completed video frames from the video game, and can only scale the video frames in dimensions and in color depth after the fact.”)
33. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of claim 1 of Lin in view of Matsuoka to include a game device, wherein the first resolution comprises an output resolution of a game engine, and the second resolution comprises a display resolution of the game device, and wherein acquiring the first image comprises acquiring the first image from the game engine of Colenbrander. The motivation for this modification could have been utilize interpolation techniques with a game engine and game console device in order to produce higher quality output and resolutions, especially in relation to the resources available to render and display the video game.
34. As per claim 15, Lin in view of Matsuoka, and further in view of Colenbrander discloses: The method according to claim 14, further comprising determining the first resolution based on attribute information about the game device, wherein the attribute information about the game device comprises at least one of the following:
a computing power of the game device, a load condition of the game device, a temperature of the game device, or a model feature of the game device. (Colenbrander, col. 7, lines 19-33, “By way of example without limitation, the game stream quality falling below a given threshold can be defined by various measures, such as packet loss exceeding a predefined threshold, bandwidth falling below a predefined threshold, latency exceeding a predefined threshold, etc. The resulting reduction in game quality setting can include various measures described in further detail herein, such as reducing resolution or framerate, reducing texture quality, reducing detail settings, reducing the scope of the virtual environment that is rendered, etc. Similar to the above, if the game stream quality rises above a given threshold, then no action may be taken in some implementations, whereas in other implementations, then at method operation 210, the video game is optionally instructed to increase a game quality setting.” and col. 17, lines 59-65, “An advantage of using a distributed game engine is that it is possible to take advantage of elastic computing, wherein computing resources can be scaled up or down depending upon needs. For example, in a large multiplayer game executed traditionally on a single hardware server, after for example about 100 players, hardware resources become limited, so that more players cannot be added.” and col. 18, lines 5-10, “Thus, a cloud game engine can have functionality distributed to different processing entities. It will be appreciated that different functions can be executed in different frameworks. For example, some functions (e.g. social) might be easier to run in a container, whereas graphics might be better run using a VM connected to a GPU.” and col. 10, lines 7-10, “In some implementations, detection of switching from one device to another can trigger adjustments performed by the video game, such as the above-mentioned changes in video resolution, frame rate, or aspect ratio.”)
35. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of claim 14 of Lin in view of Matsuoka to include the that the attribute information about the game device comprises at least one of the following: a computing power of the game device, a load condition of the game device, a temperature of the game device, or a model feature of the game device of Colenbrander. The motivation for this modification could have been provide information about how a video game is running so that the video game engine or game device can dynamically adjust to produce the highest quality game experience while still within available computing resources.
36. As per claim 16, Lin in view of Matsuoka, and further in view of Colenbrander discloses: The method according to claim 15, wherein the determining the first resolution comprises:
determining the first resolution as A1 by B1 if the attribute information about the game device satisfies a target condition; or
determining the first resolution as A2 by B2 if the attribute information about the game device does not satisfy the target condition, wherein A1 is greater than A2 and/or B1 is greater than B2, and the target condition comprises at least one of the following: (Colenbrander, col. 2, lines 43-60, “In some implementations, a method is provided, including the following method operations: executing a video game by a cloud game machine, the execution of the video game includes rendering gameplay video ... responsive to detecting a change in the connection quality between the streaming server and the client device, then adjusting the rendering of the gameplay video by the cloud game machine, wherein adjusting the rendering of the gameplay video by the cloud game machine includes adjusting a frame rate or a resolution for the gameplay video.” and col. 9, lines 36-48, “However, rather than scaling such a video frame after it has already been generated, the video game itself can be instructed to (or caused to) render the video frame at a different resolution, which enables the video game to optimize the rendering for the chosen resolution, e.g. so that text quality is optimized. In this way, the resolution can be dynamically changed to any arbitrary resolution without penalty from scaling after the fact. For example, when stream quality or network conditions worsen, then the video game 104 can be caused to decrease the video resolution; and when stream quality or network conditions improve, then the video game 104 can be caused to increase the video resolution.”)
the computing power of the game device being greater than a target power threshold, the load condition of the game device being less than a target load threshold, the temperature of the game device being less than a target temperature threshold, or the model feature of the game device exceeding a target model feature. (Colenbrander, col. 7, lines 19-33, “By way of example without limitation, the game stream quality falling below a given threshold can be defined by various measures, such as packet loss exceeding a predefined threshold, bandwidth falling below a predefined threshold, latency exceeding a predefined threshold, etc. The resulting reduction in game quality setting can include various measures described in further detail herein, such as reducing resolution or framerate, reducing texture quality, reducing detail settings, reducing the scope of the virtual environment that is rendered, etc. Similar to the above, if the game stream quality rises above a given threshold, then no action may be taken in some implementations, whereas in other implementations, then at method operation 210, the video game is optionally instructed to increase a game quality setting.” and col. 17, lines 59-65, “An advantage of using a distributed game engine is that it is possible to take advantage of elastic computing, wherein computing resources can be scaled up or down depending upon needs. For example, in a large multiplayer game executed traditionally on a single hardware server, after for example about 100 players, hardware resources become limited, so that more players cannot be added.” and col. 18, lines 5-10, “Thus, a cloud game engine can have functionality distributed to different processing entities. It will be appreciated that different functions can be executed in different frameworks. For example, some functions (e.g. social) might be easier to run in a container, whereas graphics might be better run using a VM connected to a GPU.” and col. 10, lines 7-10, “In some implementations, detection of switching from one device to another can trigger adjustments performed by the video game, such as the above-mentioned changes in video resolution, frame rate, or aspect ratio.”)
37. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of claim 15 of Lin in view of Matsuoka to include determining the first resolution as A1 by B1 if the attribute information about the game device satisfies a target condition or determining the first resolution as A2 by B2 if the attribute information about the game device does not satisfy the target condition, wherein the computing power of the game device being greater than a target power threshold, the load condition of the game device being less than a target load threshold, the temperature of the game device being less than a target temperature threshold, or the model feature of the game device exceeding a target model feature of Colenbrander. The motivation for this modification could have been increase the quality of the video game output by determining which screen resolution is best suited for gameplay and also depending on the resources or conditions with the game device.
38. Claim 18 is similar in scope to claim 1 except for additional limitations that Lin in view of Matsuoka, and further in view of Colenbrander discloses: One or more non-transitory computer-readable media storing instructions that, when executed, cause: (Lin, col. 11, lines 8-17, ““Computer-readable storage media” may refer to media and/or devices that enable persistent and/or non-transitory storage of information ... The computer-readable storage media includes ... a method or technology suitable for storage of information such as computer readable instructions …”)
determining a first resolution and a second resolution, the first resolution being an output resolution of a game engine, and the second resolution being a display resolution of a game device; (Colenbrander, col. 5, lines 13-18, “By way of example without limitation, the client device 122 can be a game console ... capable of streaming gameplay of a video game from a cloud gaming provider as in the present disclosure.” and col. 1, line 65-col. 2, line 1, “In some implementations, a method is provided, including the following operations: executing a video game by a cloud game machine, the execution of the video game includes rendering gameplay video ...” and col. 4, lines 19-30, “There are a few settings which can be adjusted such as the bitrate of the audio or video encoder, frame rate and video resolution. However, these knobs are suboptimal because the streaming software receives completed audio or video frames to encode. While the server can adjust resolution, it does so by scaling the already rendered image ... These adjustments ... process video and audio that has already been rendered.” and col. 9, lines 25-29, “In current cloud gaming systems, the streaming logic/server receives the completed video frames from the video game, and can only scale the video frames in dimensions and in color depth after the fact.”)
39. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the non-transitory computer-readable media of Lin in view of Matsuoka to include determining a first resolution and a second resolution, the first resolution being an output resolution of a game engine, and the second resolution being a display resolution of a game device of Colenbrander. The motivation for this modification could have been utilize interpolation techniques with a game engine and game console device in order to produce higher quality output and resolutions, especially in relation to the resources available to render and display the video game.
40. Claim 19, which is similar in scope to dependent claim 2 and independent claim 18, is thus rejected under the same rationale as described above. In addition, the rational for combining Colenbrander with Lin in view of Matsuoka is the same as claim 18 above.
41. Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Lin et al. (US-9123138-B2, hereinafter "Lin") in view of Matsuoka (US-6263120-B1), further in view of Colenbrander (US-11731043-B2), and further in view of Hoshii et al. (US-2002/0015162-A1, hereinafter "Hoshii").
42. Claim 20, which is similar in scope to dependent claims 3, 19 and independent claim 18 is thus rejected under the same rationale as described above. In addition, the rational for combining Hoshii with Lin in view of Matsuoka, and further in view of Colenbrander is the same as claim 3 above.
Conclusion
43. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. These are as follows: Hasegawa (US-2009/0136153-A1), Huang (US-2013/0084014-A1), Yang (US-2017/0070747-A1), and Taoka (US-2019/0205019-A1). Each prior art discloses selecting between interpolation methods based on various image criteria.
44. Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
45. Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW CLOTHIER whose telephone number is (571)272-4667. The examiner can normally be reached Mon-Fri 8:00am-4:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571)272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW CLOTHIER/Examiner, Art Unit 2614
/KENT W CHANG/Supervisory Patent Examiner, Art Unit 2614
Prosecution Insights