DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Claim Notes
Claims 4 and 14 recite limitations of the form “at one or more of A, B, and C”. Claims 5 and 15 recite a limitation of “one or more…from among A, B, C and D”. In accordance with the U.S. Court of Appeals for the Federal Circuit in SuperGuide Corp v. DirecTV Enterprises, Inc., these limitations are conjunctive in nature and to be construed as “at least one of A, at least one of B, at least one of C (and at least one of D)”. Therefore, these claims are addressed herein as requiring each of these steps rather than the alternative of A or B or C (or D).
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-3, 6-9, 11-13 and 16-19 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Han et al. (KR 10-2072106, utilizing a machine translation).
Regarding claim 1, Han et al. discloses an image processing device comprising:
at least one processor including processing circuitry (processor of computer described in conjunction with Figure 14); and
memory storing one or more instructions that, when executed by the at least one processor individually or collectively (“Meanwhile, the processes illustrated in FIG. 14 can be implemented as computer-readable code on a computer-readable recording medium” at paragraph 0052, line 1), cause the image processing device to:
obtain a neural network model corresponding to a quality of a first image and viewing information related to the first image (“At this time, the coefficients of the CNN-based filter are pre-set by being trained for each profile using ground truth images in which the image quality has been corrected for each of the multiple pre-set profiles by combining the viewing environment for the original image, as described above” at paragraph 0048, paragraph 3, line 1; the profile determines the preferred quality and environment as these elements are used to initialize the neural network);
generate training data, based on the quality of the first image and the viewing information (“During the training process, the video to be improved in quality, that is, the image, is input as data into the input layer. The convolution kernel coefficients of each convolution layer are initialized before training, and are trained by the Error Backpropagation algorithm to minimize the error between the output layer data (images) and the output layer labels, i.e., the images of the original quality” at paragraph 0009, line 2; “The correct answer video setting unit (310) sets the correct answer video for each of these seven profiles” at paragraph 0027, line 1);
obtain a trained neural network model by training the neural network model based on the training data (“The learning unit (320) of the present embodiment learns and obtains the coefficients of a CNN-based filter for each preset profile as described above, using the correct answer image generated by the correct answer image setting unit (310)” at paragraph 0030); and
obtain a second image based on the first image by performing a first image quality processing operation on the first image based on the trained neural network model (“In the inference unit (340) according to the present embodiment, the coefficients of a CNNbased filter assigned to the profile determined by the profile determination unit (330) are applied to a target image requested by a user terminal to generate a preferred image” at paragraph 0033).
Regarding claim 11, Han et al. discloses an operating method of an image processing device, the operating method comprising:
obtaining a neural network model corresponding to a quality of a first image and viewing information related to the first image (“At this time, the coefficients of the CNN-based filter are pre-set by being trained for each profile using ground truth images in which the image quality has been corrected for each of the multiple pre-set profiles by combining the viewing environment for the original image, as described above” at paragraph 0048, paragraph 3, line 1; the profile determines the preferred quality and environment as these elements are used to initialize the neural network);
generating training data, based on the quality of the first image and the viewing information (“During the training process, the video to be improved in quality, that is, the image, is input as data into the input layer. The convolution kernel coefficients of each convolution layer are initialized before training, and are trained by the Error Backpropagation algorithm to minimize the error between the output layer data (images) and the output layer labels, i.e., the images of the original quality” at paragraph 0009, line 2; “The correct answer video setting unit (310) sets the correct answer video for each of these seven profiles” at paragraph 0027, line 1);
obtain a trained neural network model by training the neural network model based on the training data (“The learning unit (320) of the present embodiment learns and obtains the coefficients of a CNN-based filter for each preset profile as described above, using the correct answer image generated by the correct answer image setting unit (310)” at paragraph 0030); and
obtaining a second image based on the first image by performing a first image quality processing operation on the first image based on the trained neural network model (“In the inference unit (340) according to the present embodiment, the coefficients of a CNNbased filter assigned to the profile determined by the profile determination unit (330) are applied to a target image requested by a user terminal to generate a preferred image” at paragraph 0033).
Regarding claims 2 and 12, Han et al. discloses a device and method wherein the viewing information comprises at least one of resolution information, bitrate information, encoding information, a content type, a content genre, an ambient environment, a viewing distance, or user information of the first image (“The viewing environment of the present embodiment may include, but is not limited to, weather, time, place, age, gender, type of video content, and encoding quality of video content” at paragraph 0025, line 1).
Regarding claims 3 and 13, Hang et al. discloses a device and method wherein the one or more instructions, when executed by the at least one processor individually or collectively, further cause the image processing device to:
obtain, based on the viewing information comprising at least one of the content type or the content genre of the first image (“The viewing environment of the present embodiment may include, but is not limited to, weather, time, place, age, gender, type of video content, and encoding quality of video content” at paragraph 0025, line 1), a training image corresponding to the content type or the content genre as first data (“The answer video setting unit (310) according to the present embodiment sets the answer video, in which the quality of the original video has been corrected, according to a preset profile by combining one or more viewing environments” at paragraph 0021, line 1); and
obtain an image quality degraded image as second data, the image quality degraded image being obtained by performing image quality degradation on the training image in the first data to have an image quality corresponding to a quality value of the first image (“That is, as illustrated in FIG. 4, the correct answer image setting unit (310) of the present embodiment generates various combinations of candidate images (410, 420, and 430) by increasing or decreasing brightness, contrast, sharpness, and saturation, etc., in the original image (400) provided by the content provider” at paragraph 0022, line 3; the decrease of sharpness, contrast or brightness constitute a degradation), and
wherein the training data comprises the first data and the second data (both the original data and the corrected data are used in the training process, thereby constituting the overall training data).
Regarding claims 6 and 16, Han et al. discloses a device and method wherein the one or more instructions, when executed by the at least one processor individually or collectively, further cause the image processing device to train the neural network model to allow a difference between an image output by inputting the image quality degraded image in the second data to the neural network model and the training image comprised in the first data to be a minimum (“The convolution kernel coefficients of each convolution layer are initialized before training, and are trained by the Error Backpropagation algorithm to minimize the error between the output layer data (images) and the output layer labels, i.e., the images of the original quality.” At paragraph 0009, line 3).
Regarding claims 7 and 17, Han et al. discloses a device and method
wherein the one or more instructions, when executed by the at least one processor individually or collectively, further cause the image processing device to obtain, via a plurality of reference models (“The viewing environment of the present embodiment may include, but is not limited to, weather, time, place, age, gender, type of video content, and encoding quality of video content. In this embodiment, a profile for each viewing environment for a CNN is generated by combining one or more of these, taking into account situations among these viewing environments where the same image quality processing is required.” at paragraph 0025), a neural network model corresponding to the image quality of the first image and the viewing information with respect to the first image (“At this time, the coefficients of the CNN-based filter are pre-set by being trained for each profile using ground truth images in which the image quality has been corrected for each of the multiple pre-set profiles by combining the viewing environment for the original image, as described above.” at paragraph 0048, line 8), and
at least one of the plurality of reference models comprises at least one of:
a first image quality processing model trained based on a plurality of training images having different quality values (“That is, as illustrated in FIG. 4, the correct answer image setting unit (310) of the present embodiment generates various combinations of candidate images (410, 420, and 430) by increasing or decreasing brightness, contrast, sharpness, and saturation, etc., in the original image (400) provided by the content provider” at paragraph 0022, line 3),
a second image quality processing model trained based on a plurality of training images corresponding to different types of content (“The viewing environment of the present embodiment may include, but is not limited to, weather, time, place, age, gender, type of video content, and encoding quality of video content. In this embodiment, a profile for each viewing environment for a CNN is generated by combining one or more of these, taking into account situations among these viewing environments where the same image quality processing is required.” at paragraph 0025), or
a third image quality processing model trained based on a plurality of training images corresponding to different genres of content.
Regarding claims 8 and 18, Han et al. discloses a device and method wherein the one or more instructions, when executed by the at least one processor individually or collectively, further cause the image processing device to:
identify one or more reference models from among the plurality of reference models by comparing a content type or a content genre corresponding to the plurality of reference models with the content type or the content genre of the first image (“The profile determination unit (330) that receives such viewing environment information analyzes the information about the acquired viewing environment and determines the profile that the user terminal corresponds to among a plurality of preset profiles by combining one or more viewing environments (S1420)” at paragraph 0048, line 1); and
obtain the neural network model based on the one or more reference models (“The profile determination unit (330) transmits the determined profile to the inference unit (340), and the inference unit (340) applies the coefficients of the CNN-based filter assigned to the determined profile to the target image to generate a preferred image (S1430).” at paragraph 0048, line 5).
Regarding claims 9 and 19, Han et al. discloses a device and method wherein the one or more instructions, when executed by the at least one processor individually or collectively, further cause the image processing device to:
identify one or more reference models from among the plurality of reference models by comparing a quality value corresponding to the plurality of reference models with the quality value of the first image (“The profile determination unit (330) that receives such viewing environment information analyzes the information about the acquired viewing environment and determines the profile that the user terminal corresponds to among a plurality of preset profiles by combining one or more viewing environments (S1420)” at paragraph 0048, line 1; “The viewing environment of the present embodiment may include, but is not limited to, weather, time, place, age, gender, type of video content, and encoding quality of video content. In this embodiment, a profile for each viewing environment for a CNN is generated by combining one or more of these, taking into account situations among these viewing environments where the same image quality processing is required.” at paragraph 0025); and
obtain the neural network model based on the one or more reference models (“The profile determination unit (330) transmits the determined profile to the inference unit (340), and the inference unit (340) applies the coefficients of the CNN-based filter assigned to the determined profile to the target image to generate a preferred image (S1430).” at paragraph 0048, line 5).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 4 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Han et al. and Yang et al. (US 10,817,990).
Han et al. discloses a device and method wherein the one or more instructions, when executed by the at least one processor individually or collectively, further cause the image processing device to:
obtain one or more pieces of compression information based on the encoding information (“The viewing environment of the present embodiment may include, but is not limited to, weather, time, place, age, gender, type of video content, and encoding quality of video content.” at paragraph 0025, line 1);
obtain one or more quality values related to the first image (“That is, as illustrated in FIG. 4, the correct answer image setting unit (310) of the present embodiment generates various combinations of candidate images (410, 420, and 430) by increasing or decreasing brightness, contrast, sharpness, and saturation, etc., in the original image (400) provided by the content provider” at paragraph 0022, line 3); and
obtain the image quality degraded image by performing a second image quality processing operation on the first image, based on the one or more quality values and the one or more pieces of compression information (the decrease of sharpness, contrast or brightness constitute a degradation; the corrected image is done according to the retrieved profile parameters).
Han et al. does not explicitly disclose that the compression information is based on the bitrate information and the resolution information and the quality values are based on one or more of a compression image quality, a blur image quality, and noise related to the first image.
Yang et al. teaches a device and method in the same field of endeavor of video encoding and decoding, wherein the one or more instructions, when executed by the at least one processor individually or collectively, further cause the image processing device to:
obtain one or more pieces of compression information based on one or more of the bitrate information, the resolution information, and the encoding information (“Through an embodiment according to FIG. 5, it will be determined that AI encoding and AI decoding processes according to an embodiment of the disclosure do not only consider a change of resolution. As shown in FIG. 5, DNN setting information may be selected considering resolution, such as standard definition (SD), high definition (HD), or full HD, a bitrate, such as 10 Mbps, 15 Mbps, or 20 Mbps, and codec information, such as AV1, H.264, or HEVC, individually or collectively. For such consideration of the resolution, the bitrate and the codec information, training in consideration of each element should be jointly performed with encoding and decoding processes during an AI training process” at col. 17, line 4);
obtain one or more quality values based on one or more of a compression image quality, a blur image quality, and noise related to the first image (“Referring to FIG. 24B, the low-pass filter may be a filter formed in a shape of a 3×3 mask 2450. The AI decoding apparatus 2300 may adjust strength of the low-pass filter by a method of adjusting values of coefficients in the 3×3 mask 2450. For example, the AI decoding apparatus 2300 may change a coefficient located at a center of the 3×3 mask 2450 among the coefficients in the 3×3 mask 2450 from 2 to 5, based on an artifact map. In this case, a degree of blurring of an image may be weakened through filtering. Alternatively, the AI decoding apparatus 2300 may change the remaining coefficients except for the coefficient located at the center among the coefficients in the 3×3 mask 2450 from 1 to 2, based on the artifact map” at col. 48, line 50; “Herein, an artifact map may be a map representing an artifact region including an aliasing artifact in unit of a block. The aliasing artifact is noise that is generated when a sampling frequency of an image is not sufficiently greater than twice a maximum frequency of signals in the image or when neighboring signal spectrums overlap each other due to improper filtering of an image” at col. 33, line 5; “According to an embodiment, the AI down-scaler 612 may determine the down-scaling target based on at least one of a compression ratio (for example, a resolution difference between the original image 105 and the first image 115, target bitrate, or the like), compression quality (for example, type of bitrate), compression history information, or a type of the original image 105” at col. 22, line 9).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize the additional quality values and information as taught by Yang et al. for the profiles of Han et al. to be able to further control the quality of the generated images.
Claim(s) 10 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Han et al. and Kim et al. (KR20230120325, utilizing US 20240404004 as the translation).
Han et al. discloses a device and method as described in claims 7 and 17 above.
Han et al. does not explicitly disclose applying a weight to each of the plurality of reference models; and obtaining the neural network model by performing weighted sum on each of the plurality of reference models to which the weight is applied, and the weight applied to each of the plurality of reference models is determined based on a difference between a quality value corresponding to a respective reference model and the quality value of the first image.
Kim et al. teaches a device and method in the same field of endeavor of display processing, wherein the one or more instructions, when executed by the at least one processor individually or collectively, further cause the image processing device to:
apply a weight to each of the plurality of reference models (“The processor 120 according to one or more embodiments may obtain an adaptive neural network model 1 by applying different weight values to each of a plurality of neural network models based on the probability value for each of the plurality of clusters included in the weight value information” at paragraph 0067); and
obtain the neural network model by performing weighted sum on each of the plurality of reference models to which the weight is applied (as demonstrated in Figure 2, the total weight is the sum of the individually weighted clusters), and
the weight applied to each of the plurality of reference models is determined based on a difference between a quality value corresponding to a respective reference model and the quality value of the first image (“In an example, if a probability of the input image being included in the first cluster is 0.5, a probability of being included in the second cluster is 0.25, and a probability of being included in the third cluster is 0.25 according to the weight value information, the processor 120 may obtain the adaptive neural network model 1 by applying a relatively high weight value to the neural network model corresponding to the first cluster, and applying a relatively low weight value to the neural network model corresponding to the second cluster and the neural network model corresponding to the third cluster. For example, the adaptive neural network model 1 obtained by the processor 120 using the weight value information may be a model for the input image (or, corresponding to the input image)” at paragraph 0068).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize a weighted modeling as taught by Kim et al. in the system of Han et al. to be able to adapt the neural network to generate the expected quality match.
Claim(s) 5 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Han et al. and Yang et al. as applied to claims 4 and 14 above, and further in view of Kang et al. (US 2022/0292635).
The Han et al. and Yang et al. combination discloses a device and method further comprising:
one or more sensors (“illuminance sensor and an RGB sensor” Han et al. at paragraph 0003, line 3),
wherein the one or more instructions, when executed by the at least one processor individually or collectively, further cause the image processing device to:
obtain at least one of the ambient environment or the viewing distance corresponding to the viewing information via the one or more sensors (“The viewing environment of the present embodiment may include, but is not limited to, weather, time, place, age, gender, type of video content, and encoding quality of video content” Han et al. at paragraph 0025, line 1; the above sensors are able to determine the environment, which would be applicable in the case of weather);
determine a target image quality of an image, based on the at least one of ambient environment or the viewing distance (“As shown in Fig. 5, the location viewing environment is divided into two areas, outdoor and indoor, and in the case of the outdoor environment, the time viewing environment is divided into daytime and nighttime, and in the case of the daytime environment, the weather viewing environment is divided into clear day and cloudy day, so that three profiles can be generated as the outdoor environment.” Han et al. at paragraph 0026, line 3; see also paragraph 0027 for specific target parameters depending on detected environment); and
generate the training data by adjusting an image quality of the first image, based on the target image quality (“The correct answer video setting unit (310) sets the most suitable video for each viewing environment among the candidate videos generated in the above manner as the correct answer video.” Han et al. At paragraph 0024), and
wherein the target image quality of the image comprises one or more image qualities from among sharpness, brightness, a contrast (“That is, as illustrated in FIG. 4, the correct answer image setting unit (310) of the present embodiment generates various combinations of candidate images (410, 420, and 430) by increasing or decreasing brightness, contrast, sharpness, and saturation, etc., in the original image (400) provided by the content provider.” Han et al. at paragraph 0022, line 3).
The Han et al. and Yang et al. combination does not explicitly disclose that the target image quality of the image comprises a chroma.
Kang et al. teaches a device and method in the same field of endeavor of image quality adjustment, wherein the target image quality of the image comprises a chroma (“In an example, the image correction apparatus may change, to a target chromaticity vector, chromaticity information corresponding to a target illuminant among a plurality of illuminants. For example, a user may select the target illuminant. Thus, the target chromaticity vector may be a vector indicating how a user selects to change a chromaticity. The image correction apparatus may calculate a partial illumination map having a target chromaticity of the target illuminant by multiplying the determined target chromaticity vector and mixture coefficient maps. The image correction apparatus may generate a relighting map by adding together the partial illumination map of the target illuminant and a remaining illumination map of a remaining illuminant” at paragraph 0088, line 1).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize the chromaticity information as taught by Kang et al. for the target quality of the Han et al. and Yang et al. combination as a way to adjust the white balance of the image to be more perceptible to the user.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KATRINA R FUJITA whose telephone number is (571)270-1574. The examiner can normally be reached Monday - Friday 9:30-5:30 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at 5712723638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KATRINA R FUJITA/Primary Examiner, Art Unit 2672
Prosecution Insights