DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 11/20/2024, 02/12/2025, 08/25/2026 was filed. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 13-14 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 13 recites the limitation " and responsive to the portion of the original image not including… the text". There is insufficient antecedent basis for this limitation in the claim.
Claim 14 recites the limitation “determining whether the portion of the original image includes text; responsive to the portion of the original image including the face, outputting a face super resolution layer.” There is insufficient antecedent basis for “the face” because neither claim 10 nor claim 14 previously introduces a face.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1, 4-7, 9-10, 13-16, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Kim (US 20210073945 A1) in view of Zhang (US 20190378242 A1).
Regarding independent claim 1, Kim teaches: A computer-implemented method comprising:
providing a user interface to a user that includes an original image and an option to generate a high-resolution portion of the original image, (Kim – Fig. 7, [0075] [0075] In addition, the user terminal 100 may include the user interface 140 to receive instructions from a user and transmit output information to the user. [0076] The user may select an area of an image to be processed in the user terminal 100 through the user interface 140. Fig. 7 [0215] When the super resolution neural network to be used is determined, the processor may apply the determined super resolution neural network to the image of the area selected to be displayed (S260). [0216] The super resolution neural network may process the received image and output a final high-resolution image (S270).)
wherein the high-resolution portion of the original image is associated with a higher resolution than the original image; (Kim – Fig. 7 [0216] The super resolution neural network may process the received image and output a final high-resolution image (S270). [0020] A method for enhancing image resolution according to an embodiment of the present disclosure may include receiving a low resolution image, selecting an image processing area for the low resolution image, selecting a neural network for image processing according to an attribute of the selected area among neural network groups for image processing, and generating a high resolution image for the area by processing the selected image processing area according to the selected neural network for image processing.)
and dimensions of a portion of the original image; (Kim – Fig. 7 [0205-0210] [0205] For example, when a user's finger touches an image display to perform pinch-in/out instruction, selects the target area to be displayed, clips that selected area, and determines the area according to zoom and the number of pixels or the resolution of the area that is eventually displayed according to zoom in/out.)
providing the portion of the original image as input to a machine-learning model; (Kim – Fig. 7 [0215] When the super resolution neural network to be used is determined, the processor may apply the determined super resolution neural network to the image of the area selected to be displayed (S260). In terms of the processing process inside the device, it can be understood as a process in which an area clipped to be displayed is input to a super resolution neural network determined to match the magnification.)
generating, with the machine-learning model, the high-resolution portion of the original image; (Kim – Fig. 7 [0216] The super resolution neural network may process the received image and output a final high-resolution image (S270). The output may be through a display of an apparatus for enhancing image resolution or through transmission to another device having a display, so the enlarged high-resolution image may be displayed on the display of the user terminal.)
and updating the user interface to include the high-resolution portion of the original image. (Kim – [0075] In addition, the user terminal 100 may include the user interface 140 to receive instructions from a user and transmit output information to the user. [0215] The output may be through a display of an apparatus for enhancing image resolution or through transmission to another device having a display, so the enlarged high-resolution image may be displayed on the display of the user terminal.)
Kim does not explicitly teach: receiving a selection of the option to generate the high-resolution portion of the original image
However, Zhang teaches: receiving a selection of the option to generate the high-resolution portion of the original image (Zhang − [0096] LR image module 148 obtains a LR image to be super-resolved and generates a scaled image. In one example, a user loads a LR image into system 200 and selects an option in a user interface exposed by UI module 152 to super-resolve the LR image, such as a “super-resolve now” button.)
and updating the user interface to include the high-resolution portion of the original image. (Zhang − [0147] In one example, super-resolution image 514 is exposed in user interface 500 responsive to a user selection to enable super-resolution, such as a “super-resolve now” button in toolbar 512.)
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Kim and Zhang, as each invention teaches enhancing a low resolution image. Adding the teaching of Zhang provides a selectable user-interface option to initiate a super-resolution processing. One of ordinary skill in the art would have been motivated to make such a modification to allow the user to control when super-resolution processing is initiated.
Regarding dependent claim 4, depends on claim 1, Kim teaches: wherein the machine-learning model generates the high-resolution portion of the original image by: (Kim – Fig. 7 [0216] The super resolution neural network may process the received image and output a final high-resolution image (S270). [0020] A method for enhancing image resolution according to an embodiment of the present disclosure may include receiving a low resolution image, selecting an image processing area for the low resolution image, selecting a neural network for image processing according to an attribute of the selected area among neural network groups for image processing, and generating a high resolution image for the area by processing the selected image processing area according to the selected neural network for image processing.)
determining whether the portion of the original image meets a threshold resolution value; responsive to the portion of the original image meeting the threshold resolution value, (Kim – Fig. 7 [0207-0212] determining the zoom level and/or the number of pixels or resolution (threshold) and determining the super-resolution neural network according to the determined value)
determining whether the portion of the original image includes a face or text; (Kim − [0160] Furthermore, the training computing system 300 may generate a neural network for image processing that can perform the same type of training on various types of objects such as a human face image, an animal image, and a car image, and may optimally improve images of the type of objects. [0180] In the same way, by training the neural network with the training data including images of a specific object such as a human face, text, or an animal can obtain the neural network for image processing optimized for enhancing the resolution of the image of the object. [0194] Here, various types of objects that can be identified through the neural network for object recognition may include a person, a logo, text, an animal, a human face and the like, and the attribute of the image area may be determined according to the type of the identified objects.)
and responsive to the portion of the original image not including the face or the text, outputting the high-resolution portion of the original image. (Kim − [0192-0202] Fig. 5 objects other than face or text, such as a logo or animal, may be identified and outputs the final resolution image at S160.)
Regarding dependent claim 5, depends on claim 1, Kim teaches: wherein the machine-learning model generates the high-resolution portion of the original image by: (Kim – Fig. 7 [0216] The super resolution neural network may process the received image and output a final high-resolution image (S270). [0020] A method for enhancing image resolution according to an embodiment of the present disclosure may include receiving a low resolution image, selecting an image processing area for the low resolution image, selecting a neural network for image processing according to an attribute of the selected area among neural network groups for image processing, and generating a high resolution image for the area by processing the selected image processing area according to the selected neural network for image processing.)
generating a base super resolution layer; (Kim – Fig. 5 [0189] applying a primary super resolution model to the entire image to generate an image having enhance resolution)
determining whether the portion of the original image includes a face; (Kim − [0192-0194] [0192] The processor of the apparatus for enhancing image resolution may recognize the object in the image by applying the neural network for object recognition to the low-resolution image. [0194] Here, various types of objects that can be identified through the neural network for object recognition may include a human face;)
responsive to the portion of the original image including the face, outputting a face super resolution layer; (Kim – [0180] training the neural network with the training data including images of a specific object such as a human face, text, or an animal can obtain the neural network for image processing optimized for enhancing the resolution of the image of the object. [0194-0201] selects the object-specific SR model, applies it to the recognized object, and obtains the corresponding high resolution object image.)
and blending the base super resolution layer and the face super resolution layer to form the high-resolution portion of the original image. (Kim – [0201-0202] [0202] The processor may output the final image by combining the high-resolution images for each object and the entire image having the enhanced resolution (S160). The output may be through a display of the apparatus for enhancing image resolution or through transmission to another device having a display.)
Regarding dependent claim 6, depends on claim 1, Kim teaches: wherein the machine-learning model generates the high-resolution portion of the original image by: (Kim – Fig. 7 [0216] The super resolution neural network may process the received image and output a final high-resolution image (S270). [0020] A method for enhancing image resolution according to an embodiment of the present disclosure may include receiving a low resolution image, selecting an image processing area for the low resolution image, selecting a neural network for image processing according to an attribute of the selected area among neural network groups for image processing, and generating a high resolution image for the area by processing the selected image processing area according to the selected neural network for image processing.)
generating a base super resolution layer; determining whether the portion of the original image includes text; (Kim – [0189] Fig. 5 applying a primary super resolution model to the entire image to generate an image having enhanced resolution, corresponding to the claimed base super resolution layer; [0192-0194] identifying objects in the image using an object-recognition neural network, and identified objects include text.)
responsive to the portion of the original image including the text, outputting a text super resolution layer of the original image; (Kim – [0172] a super resolution model trained for a text image, [0196] [0199-0201] selecting and applying an object-specific super resolution model to the recognized object, and obtaining a high-resolution image for the object, corresponding to the claimed text super resolution layer.)
and blending the base super resolution layer and the text super resolution layer to form the high-resolution portion of the original image. (Kim – [0201-0202] [0202] The processor may output the final image by combining the high-resolution images for each object and the entire image having the enhanced resolution (S160). The output may be through a display of the apparatus for enhancing image resolution or through transmission to another device having a display.)
Regarding dependent claim 7, depends on claim 1, Kim teaches: further comprising receiving an indication of a corresponding level of magnification for the portion of the original image, (Kim – Fig. 6, 7 [0205] For example, when a user's finger touches an image display to perform pinch-in/out, an enlargement or reduction instruction may be transmitted, so an image area to be displayed may be selected (S210). [0207] The processor may calculate the zoom-in/zoom-out level by a moving distance of a pinch depending on whether an enlargement instruction is received (that is, whether zoom-in needs to be performed) or whether a reduction instruction is received (that is, whether zoom-out needs to be performed) (S230 and S240). [0208] Here, as the expression representing the zoom-in/zoom-out level, the enlargement or reduction magnification may be used. When the image is enlarged two times, the enlargement or reduction magnification is expressed as two times, and when the image is reduced to ½, the enlargement or reduction magnification is expressed as ½ times.)
wherein the high-resolution portion of the original image is based on the corresponding level of magnification. (Kim – [0210-0212] [0215] When the super resolution neural network to be used is determined, the processor may apply the determined super resolution neural network to the image of the area selected to be displayed (S260). In terms of the processing process inside the device, it can be understood as a process in which an area clipped to be displayed is input to a super resolution neural network determined to match the magnification. [0216] The super resolution neural network may process the received image and output a final high-resolution image (S270). The output may be through a display of an apparatus for enhancing image resolution or through transmission to another device having a display, so the enlarged high-resolution image may be displayed on the display of the user terminal.)
Regarding dependent claim 9, depends on claim 1, Kim does not explicitly teach: extracting a random crop of an input image
However, Zhang teaches: wherein the machine-learning model is trained using training data that includes a lower-resolution image generated from a higher-resolution image by performing one or more operations selected from a group of extracting a random crop of an input image, applying an inverse gamma correction to the input image based on a random gamma correction value, augmenting the input image by randomly shifting pixel values by a constant factor, blurring the input image by adding noise to the input image, applying gamma correction to the input image, and combinations thereof. (Zhang – [0138] Neural network 408 can be trained with LR images obtained by downsampling randomly-cropped patches of a HR image set as input images. Reference images for training can be obtained from other patches of the HR image set than the randomly-cropped patches. [0139] The systems described herein constitute an improvement over systems that constrain reference images to have similar or same content as a LR image being super-resolved and that directly transfer pixel content of a reference image to a super-resolution image. )
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Kim and Zhang, as each invention teaches enhancing a low resolution image. Adding the teaching of Zhang provides a selectable user-interface option to initiate a super-resolution processing. One of ordinary skill in the art would have been motivated to make such a modification to allow the user to control when super-resolution processing is initiated.
Regarding independent claim 10, is directed to a non-transitory computer-readable medium and recites limitations similar to those of claim 1. Claim 10 is rejected under the same rationale as above with respect to claim 1.
Regarding dependent claim 13, depends on claim 10, Kim teaches: wherein the machine-learning model generates the high-resolution portion of the original image by: (Kim – Fig. 7 [0216] The super resolution neural network may process the received image and output a final high-resolution image (S270). [0020] A method for enhancing image resolution according to an embodiment of the present disclosure may include receiving a low resolution image, selecting an image processing area for the low resolution image, selecting a neural network for image processing according to an attribute of the selected area among neural network groups for image processing, and generating a high resolution image for the area by processing the selected image processing area according to the selected neural network for image processing.)
determining whether the portion of the original image meets a threshold resolution value; responsive to the portion of the original image meeting the threshold resolution value, (Kim – Fig. 7 [0207-0212] determining the zoom level and/or the number of pixels or resolution (threshold) and determining the super-resolution neural network according to the determined value)
determining whether the portion of the original image includes a face or text; (Kim − [0160] Furthermore, the training computing system 300 may generate a neural network for image processing that can perform the same type of training on various types of objects such as a human face image, an animal image, and a car image, and may optimally improve images of the type of objects. [0180] In the same way, by training the neural network with the training data including images of a specific object such as a human face, text, or an animal can obtain the neural network for image processing optimized for enhancing the resolution of the image of the object. [0194] Here, various types of objects that can be identified through the neural network for object recognition may include a person, a logo, text, an animal, a human face and the like, and the attribute of the image area may be determined according to the type of the identified objects.)
and responsive to the portion of the original image not including the face or the text, outputting the high-resolution portion of the original image. (Kim − [0192-0202] Fig. 5 objects other than face or text, such as a logo or animal, may be identified and outputs the final resolution image at S160.)
Regarding dependent claim 14, depends on claim 10, Kim teaches: wherein the machine-learning model generates the high-resolution portion of the original image by: (Kim – Fig. 7 [0216] The super resolution neural network may process the received image and output a final high-resolution image (S270). [0020] A method for enhancing image resolution according to an embodiment of the present disclosure may include receiving a low resolution image, selecting an image processing area for the low resolution image, selecting a neural network for image processing according to an attribute of the selected area among neural network groups for image processing, and generating a high resolution image for the area by processing the selected image processing area according to the selected neural network for image processing.)
generating a base super resolution layer; (Kim – Fig. 5 [0189] applying a primary super resolution model to the entire image to generate an image having enhance resolution)
determining whether the portion of the original image includes a face; (Kim − [0192-0194] [0192] The processor of the apparatus for enhancing image resolution may recognize the object in the image by applying the neural network for object recognition to the low-resolution image. [0194] Here, various types of objects that can be identified through the neural network for object recognition may include a human face;)
responsive to the portion of the original image including the face, outputting a face super resolution layer; (Kim – [0180] training the neural network with the training data including images of a specific object such as a human face, text, or an animal can obtain the neural network for image processing optimized for enhancing the resolution of the image of the object. [0194-0201] selects the object-specific SR model, applies it to the recognized object, and obtains the corresponding high resolution object image.)
and blending the base super resolution layer and the face super resolution layer to form the high-resolution portion of the original image. (Kim – [0201-0202] [0202] The processor may output the final image by combining the high-resolution images for each object and the entire image having the enhanced resolution (S160). The output may be through a display of the apparatus for enhancing image resolution or through transmission to another device having a display.)
Regarding dependent claim 15, depends on claim 10, Kim teaches: wherein the machine-learning model generates the high-resolution portion of the original image by: (Kim – Fig. 7 [0216] The super resolution neural network may process the received image and output a final high-resolution image (S270). [0020] A method for enhancing image resolution according to an embodiment of the present disclosure may include receiving a low resolution image, selecting an image processing area for the low resolution image, selecting a neural network for image processing according to an attribute of the selected area among neural network groups for image processing, and generating a high resolution image for the area by processing the selected image processing area according to the selected neural network for image processing.)
generating a base super resolution layer; determining whether the portion of the original image includes text; (Kim – [0189] Fig. 5 applying a primary super resolution model to the entire image to generate an image having enhanced resolution, corresponding to the claimed base super resolution layer; [0192-0194] identifying objects in the image using an object-recognition neural network, and identified objects include text.)
responsive to the portion of the original image including the text, outputting a text super resolution layer of the original image; (Kim – [0172] a super resolution model trained for a text image, [0196] [0199-0201] selecting and applying an object-specific super resolution model to the recognized object, and obtaining a high-resolution image for the object, corresponding to the claimed text super resolution layer.)
and blending the base super resolution layer and the text super resolution layer to form the high-resolution portion of the original image. (Kim – [0201-0202] [0202] The processor may output the final image by combining the high-resolution images for each object and the entire image having the enhanced resolution (S160). The output may be through a display of the apparatus for enhancing image resolution or through transmission to another device having a display.)
Regarding independent claim 16, is directed to a system and recites limitations similar to those of claim 1. Claim 16 is rejected under the same rationale as above with respect to claim 1.
Regarding dependent claim 19, depends on claim 16, Kim teaches: wherein the machine-learning model generates the high-resolution portion of the original image by: (Kim – Fig. 7 [0216] The super resolution neural network may process the received image and output a final high-resolution image (S270). [0020] A method for enhancing image resolution according to an embodiment of the present disclosure may include receiving a low resolution image, selecting an image processing area for the low resolution image, selecting a neural network for image processing according to an attribute of the selected area among neural network groups for image processing, and generating a high resolution image for the area by processing the selected image processing area according to the selected neural network for image processing.)
determining whether the portion of the original image meets a threshold resolution value; responsive to the portion of the original image meeting the threshold resolution value, (Kim – Fig. 7 [0207-0212] determining the zoom level and/or the number of pixels or resolution (threshold) and determining the super-resolution neural network according to the determined value)
determining whether the portion of the original image includes a face or text; (Kim − [0160] Furthermore, the training computing system 300 may generate a neural network for image processing that can perform the same type of training on various types of objects such as a human face image, an animal image, and a car image, and may optimally improve images of the type of objects. [0180] In the same way, by training the neural network with the training data including images of a specific object such as a human face, text, or an animal can obtain the neural network for image processing optimized for enhancing the resolution of the image of the object. [0194] Here, various types of objects that can be identified through the neural network for object recognition may include a person, a logo, text, an animal, a human face and the like, and the attribute of the image area may be determined according to the type of the identified objects.)
and responsive to the portion of the original image not including the face or the text, outputting the high-resolution portion of the original image. (Kim − [0192-0202] Fig. 5 objects other than face or text, such as a logo or animal, may be identified and outputs the final resolution image at S160.)
Regarding dependent claim 20, depends on claim 16, Kim teaches: wherein the machine-learning model generates the high-resolution portion of the original image by: (Kim – Fig. 7 [0216] The super resolution neural network may process the received image and output a final high-resolution image (S270). [0020] A method for enhancing image resolution according to an embodiment of the present disclosure may include receiving a low resolution image, selecting an image processing area for the low resolution image, selecting a neural network for image processing according to an attribute of the selected area among neural network groups for image processing, and generating a high resolution image for the area by processing the selected image processing area according to the selected neural network for image processing.)
generating a base super resolution layer; (Kim – Fig. 5 [0189] applying a primary super resolution model to the entire image to generate an image having enhance resolution)
determining whether the portion of the original image includes a face; (Kim − [0192-0194] [0192] The processor of the apparatus for enhancing image resolution may recognize the object in the image by applying the neural network for object recognition to the low-resolution image. [0194] Here, various types of objects that can be identified through the neural network for object recognition may include a human face;)
responsive to the portion of the original image including the face, outputting a face super resolution layer; (Kim – [0180] training the neural network with the training data including images of a specific object such as a human face, text, or an animal can obtain the neural network for image processing optimized for enhancing the resolution of the image of the object. [0194-0201] selects the object-specific SR model, applies it to the recognized object, and obtains the corresponding high resolution object image.)
and blending the base super resolution layer and the face super resolution layer to form the high-resolution portion of the original image. (Kim – [0201-0202] [0202] The processor may output the final image by combining the high-resolution images for each object and the entire image having the enhanced resolution (S160). The output may be through a display of the apparatus for enhancing image resolution or through transmission to another device having a display.)
Claim(s) 2, 11 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Kim and Zhang as applied to claims 1, 10 and 16 above, and further in view of Voroshilov (US 20190272622 A1).
Regarding dependent claim 2, depends on claim 1, Kim teaches: wherein the machine-learning model generates the high-resolution portion of the original image by: (Kim – Fig. 7 [0216] The super resolution neural network may process the received image and output a final high-resolution image (S270). [0020] A method for enhancing image resolution according to an embodiment of the present disclosure may include receiving a low resolution image, selecting an image processing area for the low resolution image, selecting a neural network for image processing according to an attribute of the selected area among neural network groups for image processing, and generating a high resolution image for the area by processing the selected image processing area according to the selected neural network for image processing.)
Kim does not explicitly teach: dividing the portion of the original image into a plurality of tiles;
However, Voroshilov teaches: dividing the portion of the original image into a plurality of tiles; (Voroshilov − [0019] One technique for creating such high- or super-resolution pictures is to divide the work that needs to be done into parts, sometimes called “tiles”. Using tiles lets the user's GPU spread the work out across more steps, breaking it into smaller parts that can be handled individually. [0020] The low-resolution image of the scene is then divided into tiles just like described above with the super-resolution image of a scene.)
for each tile of the plurality of tiles generating a super resolution tile that includes one or more of a base super resolution layer, a face super resolution layer, a text super resolution layer, and combinations thereof; (Voroshilov – [0059-[0060] intermediate tile 322 is separately processed to produce a final high-resolution tile 332 containing the high-resolution image data, corresponding to the claimed based super resolution layer.)
and aggregating the super resolution tiles to form the high-resolution portion of the original image.(Voroshilov – [0021] [0060] The Final high-resolution tile 332 is combined with other generated final high-resolution tiles to generate the final high-resolution image.)
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further combine the teaching of Kim, Zhang, and Voroshilov, as each invention teaches generating a high-resolution image. Adding the teaching of Voroshilov provides tiled processing of the image. One of ordinary skill in the art would have been motivated to make such a modification to allow the image to be processed in smaller portions that can be handled individually, thereby allowing generation of larger super-resolution images.
Regarding dependent claim 11, depends on claim 10, Kim teaches: wherein the machine-learning model generates the high-resolution portion of the original image by: (Kim – Fig. 7 [0216] The super resolution neural network may process the received image and output a final high-resolution image (S270). [0020] A method for enhancing image resolution according to an embodiment of the present disclosure may include receiving a low resolution image, selecting an image processing area for the low resolution image, selecting a neural network for image processing according to an attribute of the selected area among neural network groups for image processing, and generating a high resolution image for the area by processing the selected image processing area according to the selected neural network for image processing.)
Kim does not explicitly teach: dividing the portion of the original image into a plurality of tiles;
However, Voroshilov teaches: dividing the portion of the original image into a plurality of tiles; (Voroshilov − [0019] One technique for creating such high- or super-resolution pictures is to divide the work that needs to be done into parts, sometimes called “tiles”. Using tiles lets the user's GPU spread the work out across more steps, breaking it into smaller parts that can be handled individually. [0020] The low-resolution image of the scene is then divided into tiles just like described above with the super-resolution image of a scene.)
for each tile of the plurality of tiles generating a super resolution tile that includes one or more of a base super resolution layer, a face super resolution layer, a text super resolution layer, and combinations thereof; (Voroshilov – [0059-[0060] intermediate tile 322 is separately processed to produce a final high-resolution tile 332 containing the high-resolution image data, corresponding to the claimed based super resolution layer.)
and aggregating the super resolution tiles to form the high-resolution portion of the original image.(Voroshilov – [0021] [0060] The Final high-resolution tile 332 is combined with other generated final high-resolution tiles to generate the final high-resolution image.)
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further combine the teaching of Kim, Zhang, and Voroshilov, as each invention teaches generating a high-resolution image. Adding the teaching of Voroshilov provides tiled processing of the image. One of ordinary skill in the art would have been motivated to make such a modification to allow the image to be processed in smaller portions that can be handled individually, thereby allowing generation of larger super-resolution images.
Regarding dependent claim 17, depends on claim 16, Kim teaches: wherein the machine-learning model generates the high-resolution portion of the original image by: (Kim – Fig. 7 [0216] The super resolution neural network may process the received image and output a final high-resolution image (S270). [0020] A method for enhancing image resolution according to an embodiment of the present disclosure may include receiving a low resolution image, selecting an image processing area for the low resolution image, selecting a neural network for image processing according to an attribute of the selected area among neural network groups for image processing, and generating a high resolution image for the area by processing the selected image processing area according to the selected neural network for image processing.)
Kim does not explicitly teach: dividing the portion of the original image into a plurality of tiles;
However, Voroshilov teaches: dividing the portion of the original image into a plurality of tiles; (Voroshilov − [0019] One technique for creating such high- or super-resolution pictures is to divide the work that needs to be done into parts, sometimes called “tiles”. Using tiles lets the user's GPU spread the work out across more steps, breaking it into smaller parts that can be handled individually. [0020] The low-resolution image of the scene is then divided into tiles just like described above with the super-resolution image of a scene.)
for each tile of the plurality of tiles generating a super resolution tile that includes one or more of a base super resolution layer, a face super resolution layer, a text super resolution layer, and combinations thereof; (Voroshilov – [0059-[0060] intermediate tile 322 is separately processed to produce a final high-resolution tile 332 containing the high-resolution image data, corresponding to the claimed based super resolution layer.)
and aggregating the super resolution tiles to form the high-resolution portion of the original image.(Voroshilov – [0021] [0060] The Final high-resolution tile 332 is combined with other generated final high-resolution tiles to generate the final high-resolution image.)
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further combine the teaching of Kim, Zhang, and Voroshilov, as each invention teaches generating a high-resolution image. Adding the teaching of Voroshilov provides tiled processing of the image. One of ordinary skill in the art would have been motivated to make such a modification to allow the image to be processed in smaller portions that can be handled individually, thereby allowing generation of larger super-resolution images.
Claim(s) 3, 12 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Kim and Zhang as applied to claims 1, 10 and 16 above, and further in view of Wang (US 20130058588 A1).
Regarding dependent claim 3, depends on claim 1, Kim teaches: wherein the machine-learning model generates the high-resolution portion of the original image by: (Kim – Fig. 7 [0216] The super resolution neural network may process the received image and output a final high-resolution image (S270). [0020] A method for enhancing image resolution according to an embodiment of the present disclosure may include receiving a low resolution image, selecting an image processing area for the low resolution image, selecting a neural network for image processing according to an attribute of the selected area among neural network groups for image processing, and generating a high resolution image for the area by processing the selected image processing area according to the selected neural network for image processing.)
determining whether the portion of the original image meets a threshold resolution value; and responsive to the portion of the original image failing to meet the threshold resolution value, (Kim – Fig. 7 [0207-0212] determining the zoom level and/or the number of pixels or resolution (threshold) and determining the super-resolution neural network according to the determined value)
Kim does not explicitly teach: generating an unblurred portion of the original image and upscaling the unblurred portion of the original image to a target resolution.
However, Wang teaches: generating an unblurred portion of the original image and upscaling the unblurred portion of the original image to a target resolution. (Wang − [0044] Some embodiments include a means for performing deblurring through iterative upscaling of an image. For example, in some embodiments a deblurring module may receive input identifying an image for which deblurring is, desired and may repeat from a course to fine scale generating an estimate of a latent image from a blurred image using an upsampling super-resolution function, and estimating a blur kernel based on the estimate of the latent image and the blurred image. [0049] estimate a sharp latent image at the finer scale using super-resolutions)
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further combine the teaching of Kim, Zhang, and Wang, as each invention teaches generating an enhanced-resolution image. Adding the teachings of Wang provides deblurring through image upscaling. One of ordinary skill in the art would have been motivated to make such a modification to reduce blurring while increasing the resolution of the image, thereby generating a clearer high-resolution image.
Regarding dependent claim 12, depends on claim 10, Kim teaches: wherein the machine-learning model generates the high-resolution portion of the original image by: (Kim – Fig. 7 [0216] The super resolution neural network may process the received image and output a final high-resolution image (S270). [0020] A method for enhancing image resolution according to an embodiment of the present disclosure may include receiving a low resolution image, selecting an image processing area for the low resolution image, selecting a neural network for image processing according to an attribute of the selected area among neural network groups for image processing, and generating a high resolution image for the area by processing the selected image processing area according to the selected neural network for image processing.)
determining whether the portion of the original image meets a threshold resolution value; and responsive to the portion of the original image failing to meet the threshold resolution value, (Kim – Fig. 7 [0207-0212] determining the zoom level and/or the number of pixels or resolution (threshold) and determining the super-resolution neural network according to the determined value)
Kim does not explicitly teach: generating an unblurred portion of the original image and upscaling the unblurred portion of the original image to a target resolution.
However, Wang teaches: generating an unblurred portion of the original image and upscaling the unblurred portion of the original image to a target resolution. (Wang − [0044] Some embodiments include a means for performing deblurring through iterative upscaling of an image. For example, in some embodiments a deblurring module may receive input identifying an image for which deblurring is, desired and may repeat from a course to fine scale generating an estimate of a latent image from a blurred image using an upsampling super-resolution function, and estimating a blur kernel based on the estimate of the latent image and the blurred image. [0049] estimate a sharp latent image at the finer scale using super-resolutions)
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further combine the teaching of Kim, Zhang, and Wang, as each invention teaches generating an enhanced-resolution image. Adding the teachings of Wang provides deblurring through image upscaling. One of ordinary skill in the art would have been motivated to make such a modification to reduce blurring while increasing the resolution of the image, thereby generating a clearer high-resolution image.
Regarding dependent claim 18, depends on claim 16, Kim teaches: wherein the machine-learning model generates the high-resolution portion of the original image by: (Kim – Fig. 7 [0216] The super resolution neural network may process the received image and output a final high-resolution image (S270). [0020] A method for enhancing image resolution according to an embodiment of the present disclosure may include receiving a low resolution image, selecting an image processing area for the low resolution image, selecting a neural network for image processing according to an attribute of the selected area among neural network groups for image processing, and generating a high resolution image for the area by processing the selected image processing area according to the selected neural network for image processing.)
determining whether the portion of the original image meets a threshold resolution value; and responsive to the portion of the original image failing to meet the threshold resolution value, (Kim – Fig. 7 [0207-0212] determining the zoom level and/or the number of pixels or resolution (threshold) and determining the super-resolution neural network according to the determined value)
Kim does not explicitly teach: generating an unblurred portion of the original image and upscaling the unblurred portion of the original image to a target resolution.
However, Wang teaches: generating an unblurred portion of the original image and upscaling the unblurred portion of the original image to a target resolution. (Wang − [0044] Some embodiments include a means for performing deblurring through iterative upscaling of an image. For example, in some embodiments a deblurring module may receive input identifying an image for which deblurring is, desired and may repeat from a course to fine scale generating an estimate of a latent image from a blurred image using an upsampling super-resolution function, and estimating a blur kernel based on the estimate of the latent image and the blurred image. [0049] estimate a sharp latent image at the finer scale using super-resolutions)
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further combine the teaching of Kim, Zhang, and Wang, as each invention teaches generating an enhanced-resolution image. Adding the teachings of Wang provides deblurring through image upscaling. One of ordinary skill in the art would have been motivated to make such a modification to reduce blurring while increasing the resolution of the image, thereby generating a clearer high-resolution image.
Claim(s) 8 is rejected under 35 U.S.C. 103 as being unpatentable over Kim and Zhang as applied to claim 1 above, and further in view of Voroshilov (US 11948274 B1).
Regarding dependent claim 8, depends on claim 1, Kim does not explicitly teach: herein the machine-learning model is trained using a combination of multiple losses, and a sharpened perceptual feature loss.
However Zhang teaches: teaches: wherein the machine-learning model is trained using a combination of multiple losses, and a sharpened perceptual feature loss. (Zhang – [0133-0135] the loss includes content loss and texture loss, the content loss includes reconstruction loss, perceptual loss, and adversarial loss, and the perceptual loss is based on feature maps extracted from a VGG model.)
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further combine the teaching of Kim, Zhang, and Voroshilov, as each invention teaches generating a high-resolution image. Adding the teaching of Voroshilov provides tiled processing of the image. One of ordinary skill in the art would have been motivated to make such a modification to allow the image to be processed in smaller portions that can be handled individually, thereby allowing generation of larger super-resolution images.
Kim does not explicitly teach: a color mismatch loss,
However, Vavilala teaches: a color mismatch loss, (Vavilala − [Col. 9 ll. 5-30] The loss function may include one or more of a discriminator loss, an L1 or L2 loss, a color shift loss, or a feature loss. The discriminator loss may be used to maximize the probability of assigning the correct type to scaled or rendered images. The L1 loss may penalize the difference between the rendered input image and the scaled image. The color shift loss may penalize the L1 loss between an input low-resolution rendered image, and a downscaled low-resolution image. The downscaled low-resolution image can be generated by upscaling the low-resolution rendered image to create an upscaled high-resolution image, and then downscaling the upscaled high-resolution image. The feature loss may be similar to a visual geometry group (VGG) loss, which penalizes differences between intermediate feature maps of the network between the scaled image and the ground truth (reference) image. The loss function may be used as a quality metric by the discriminator. [Col. 10, ll. 16-22] The generator 420 and the discriminator 430 may be trained in an alternating fashion. Various loss functions may be combined with relative weights.)
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Kim, Zhang and Vavilala, as each teach training a neural networking for image resolution enhancement using loss function. Adding the teaching of Vavilala provides a color-based loss for comparing image data. One of ordinary skill in the art would have been motivated to make such a modification to account for color differences when training the image enhancement model and thereby improve the quality of the generated image.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Bai US 20210065337 A1 teaches segmenting an image into patches, processing the patches by using training CNN models, and generating an enhance image based on the processed patches.
Kim US 20210118095 A1 teaches neural networking based super-resolution processing of image regions, including a super resolution processing layer and interpolation processing for upscaling an image.
El-Khamy US 20180293707 A1 teaches deep learning image super-resolution, including user-requested image processing and combining outputs to generate a high-resolution image.
Chen US 20200342572 A1 teaches generating training images for image enhancement using image degradation and augmentation operations.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CARL E BARNES JR whose telephone number is (571)270-3395. The examiner can normally be reached Monday-Friday 9am-6pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Stephen Hong can be reached at (571) 272-4124. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CARL E BARNES JR/Examiner, Art Unit 2178
/STEPHEN S HONG/Supervisory Patent Examiner, Art Unit 2178
Prosecution Insights