DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 11, and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Tai et al. (KR 20210062477 A) (Hereinafter referred to as Tai) in view of Kim et al. (US 20160100147 A1) (Hereinafter referred to as Kim) and in further view of Yoo et al. (KR 20150103602 A) (Hereinafter referred to as Yoo).
Regarding Claim 1, Tai discloses An electronic device, comprising: a display; memory storing at least one instruction; and one or more processors connected to the display and the memory to control the electronic device, wherein the at least one instruction, when executed by the one or more processors, causes the electronic device to: (See Page 3 Paragraphs 10 - 11, “The electronic device 100 is a device that processes an input image, and has a display . . . Referring to FIG. 2, the electronic device 100 includes a memory 110 and a processor 120.”)
identify sensitivity information of an image according to each of at least one type of image quality processing; (See Abstract, “first artificial intelligence models trained to perform different image processing, and information on a second artificial intelligence model trained to identify a type of image by predicting the processing result of the image by each of the plurality of first artificial intelligence models;”
Also See Page 5 Paragraph 7, “Meanwhile, the processor 120 may input the input image into the second artificial intelligence model to obtain weights for a plurality of types related to the input image, and identify the type having the largest weight among the plurality of types as the type of the input image.”
In summary, Tai teaches a first AI model used to perform image processing (at least one type of image quality processing), a second AI model that obtains weights (sensitivity information) which identifies the type of the image, and the second AI model identifies the type of image by predicting the processing results of the image according to the first AI model (identify sensitivity information of an image according to each of at least one type of image quality processing).
Note that sensitivity information is being interpreted as simply being the weights associated with the type of image for the context of image quality processing. The basis for this interpretation is derived from the instant application’s Fig. 9 showing that based on whether the type of image quality is a first type or a second type, the sensitivity identified for the focus map is either relatively low or high, and thus sensitivity information is interpreted as being information associated with the type of image quality.)
image-quality process the input image according to the at least one type of image quality processing based on the image and the sensitivity information; and (See Abstract, “first artificial intelligence models trained to perform different image processing, and information on a second artificial intelligence model trained to identify a type of image by predicting the processing result of the image by each of the plurality of first artificial intelligence models;”
See Page 7 Paragraph - Page 8 Paragraph 1, “As described above, the electronic device can improve the processing performance of the input image by identifying the type of the input image using the artificial intelligence model and adaptively processing the input image using another artificial intelligence model corresponding to the identified type.”
See Page 5 Paragraph 5, “Meanwhile, the image processing of the same type may include at least one of up-scaling processing, noise removal processing, and detail enhancement processing.”)
control the display to display the image-quality processed input image. (See Page 3 Paragraph 10, “Alternatively, the electronic device 100 may be a device that does not have a display such as a set-top box (STB), a speaker, or a computer body, and may be a device that provides a processed image to a display device.”)
However, Tai fails to explicitly disclose obtain a focus map based on importance information per region included in an input image;
obtain reliability information per region of the focus map based on at least one of brightness information or contrast information of the input image and information included in the focus map;
identify sensitivity information of the focus map according to each of at least one type of image quality processing;
image-quality process the input image according to the at least one type of image quality processing based on the focus map, the reliability information per region of the focus map, and the sensitivity information; and
Kim teaches obtain a focus map; (See [0052], “The input unit 110 receives the metadata including the image content and information on the image content.” Further see [0019], “The metadata may include a focus map for enhancing a brightness of a specific area of the image content . . .”)
identify sensitivity information of the focus map according to each of at least one type of image quality processing; (See Kim [0019] teaching to obtain a focus map for enhancing image content (one type of image quality processing).
In combination with Tai Abstract and Page 5 Paragraph 7 already teaching to obtain weights for the type of image (sensitivity information) which is according to the image quality processing performed for said image type (according to each of at least one type of image quality processing), since the sensitivity information taught by Tai and the focus map taught by Kim are both derived from an input image/image content, then sensitivity information which identifies the type of image can be also be considered the sensitivity information of the focus map as they originate from the same source.)
image-quality process the input image according to the at least one type of image quality processing based on the focus map and the sensitivity information; and (See [0085], “Further, as illustrated in FIG. 6B, the image processor 120 may acquire a focus map for enhancing the brightness of a specific area of the image content from the metadata, extract the specific area by using the acquired focus map, and control a gain value of contrast of the specific area to perform the image processing on the image content. In detail, to enhance the contrast for the specific area (for example, characters, and the like) within the image content, the image processor 120 may extract the specific area and perform the image processing on the image content by improving the contrast for the extracted area.”
Here, Kim teaches to use the focus map for enhancing the brightness or contrast of a specific area of the image (image-quality process the input image according to the at least one type of image quality processing based on the focus map). In combination with Tai Abstract, Page 7 Paragraph - Page 8 Paragraph 1, and Page 5 Paragraph 5 teaching to image-quality process the input image according to the type of image which is obtained based on weights (the sensitivity information), the above limitations are taught.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Tai with Kim to include obtaining a focus map and image-quality process an image based on the focus map.
The motivation to combine Tai with Kim would have been obvious as both are within the same field of image processing and enhancing the contents of images (See Kim Abstract and [0085]). The benefit of including a focus map is that it allows for enhancing more salient or important areas (See Kim [0085), which would help guide algorithms to allocate processing power to the most relevant regions.
However, Tai in view of Kim still fails to explicitly disclose obtain a focus map based on importance information per region included in an input image;
obtain reliability information per region of the focus map based on at least one of brightness information or contrast information of the input image and information included in the focus map;
image-quality process the input image according to the at least one type of image quality processing based on the focus map, the reliability information per region of the focus map, and the sensitivity information; and
Yoo teaches obtain a focus map based on importance information per region included in an input image; (See Page 3 Paragraphs 4-5, “The FDM generation unit 130 generates a focus depth map from the image. The focus depth map is a depth map of the degree of blur in the image. The focus depth map is based on the assumption that the object to be photographed increases in focus blur as it is moved away from the device from which the image was taken. Therefore, the focus depth map is determined by the amount of focus blur.”
In this case, a focus depth map corresponds to a “focus map” and it is based on the amount of focus blur (importance information). Note it would be implied that the amount of focus blur is region based (per region) as focus blur would change if a region is a foreground region versus a background region.)
obtain reliability information per region of the focus map based on at least one of brightness information or contrast information of the input image and information included in the focus map; (See Page 4 Paragraph 2, “When the image has a ground, the FDM reliability calculation unit 120 calculates the reliability of the focus depth map based on the number of focus pixels located in a lower region of the image. A focus pixel means a pixel having a laplacian focus value of the edge pixels of 10% or more in the image. The Laplacian focus value is the value that the edge pixels return when applying the Laplacian function to the pixels of the image. The larger the Laplacian focus value, the more pixels can be understood to be focused.”
Also see Page 4 Paragraph 3, “Referring to FIG. 3 (a), the FDM reliability calculation unit 120 detects an edge from the image 310 using a Laplacian filter. The detected edge pixels are displayed in the edge pixel image 320.”
In this case, calculating reliability based on focus pixels can be considered as obtaining reliability information based on “contrast information” as the calculation for focus pixels can be considered as a measure of edge contrast. Here, a Laplacian filter is used to detect edges in an image and thus a higher Laplacian focus value represents stronger edge contrast and greater sharpness, and thus the pixel being considered as a focus pixel.)
image-quality process the input image according to the at least one type of image quality processing based on the focus map, the reliability information per region of the focus map, and the sensitivity information; and (See Yoo Page 3 Paragraphs 4-5 teaching a focus depth map And Yoo Page 4 Paragraph 2 teaching reliability information per region of the focus map.
See Page 3 Paragraph 7, “The depth map generator 140 may generate a final depth map using various depth maps as well as a focus depth map. For example, the depth map generator 140 may generate a depth map including a pseudo depth map, a context depth map, an object depth map based on saliency, The alpha depth map, vertical / edge depth map, etc. can be combined to create a final depth map. Also see Page 3 Paragraph 10, “In Equation (1 ), each of the pseudo depth map and the focus depth map is adjusted by the reliability of the focus depth map. That is, the independent reliability of the focus depth map affects the final depth map generation.”
In summary, Yoo teaches to combine multiple depth maps (include a focus depth map) and weighting them based on the calculated reliability. The idea being that a higher reliability focus depth map should be given more weight. In combination with Tai Abstract, Page 7 Paragraph - Page 8 Paragraph 1, and Page 5 Paragraph 5 already teaching to image-quality process the input image according to the type of image which is obtained based on weights (the sensitivity information) and Kim [0085] already teaching to use the focus map for image-quality processing, the above limitations are taught. In this case, using focus maps with higher reliability information would result in better image-quality processing.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Tai in view of Kim with Yoo to include obtaining a focus map based on importance information and calculating reliability information.
The motivation to combine Tai in view of Kim with Yoo would have been obvious as both Kim and Yoo are both within the same field of focus maps (See Yoo Page 3 Paragraphs 4-5). The benefit of generating a focus map with reliability information is that reliability information can be useful when determining the usefulness of a generated focus map. In the case where there is low reliability information, those focus maps would be less accurate and using those maps in processes like image enchantments can lead to noise and unwanted effect. Thus, calculating reliability information alongside a focus map would be obvious to include.
Regarding Claim 11, Tai in view of Kim and Yoo discloses A method of processing an image of an electronic device, comprising: (See Tai Page 4 Paragraph 1, “Also, the image processing method of the of the plurality of first artificial intelligence models . . . ”)
obtaining a focus map based on importance information per region included in an input image; obtaining reliability information per region of the focus map based on at least one of brightness information or contrast information of the input image and information included in the focus map; identifying sensitivity information of the focus map according to each of at least one type of image quality processing; image-quality processing the input image according to the at least one type of image quality processing based on the focus map, the reliability information per region of the focus map, and the sensitivity information; and displaying the image-quality processed input image. (The above limitations are similar to those of Claim 1 and are therefore rejected under a similar rationale as that of Claim 1.)
Regarding Claim 15, Tai in view of Kim and Yoo discloses A non-transitory computer readable medium storing computer instructions that when executed by a processor of an electronic device, cause the electronic device to: (See Tai Page 12 Paragraph 3, “A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium.” Also see Tai Page 3 Paragraphs 10 - 11, “The electronic device 100 is a device that processes an input image, and has a display . . . Referring to FIG. 2, the electronic device 100 includes a memory 110 and a processor 120.”)
obtain a focus map based on importance information per region included in an input image; obtain reliability information per region of the focus map based on at least one of brightness information or contrast information of the input image and information included in the focus map; identify sensitivity information of the focus map according to each of at least one type of image quality processing; image-quality process the input image according to the at least one type of image quality processing based on the focus map, the reliability information per region of the focus map, and the sensitivity information; and display the image-quality processed input image. (The above limitations are similar to those of Claim 1 and are therefore rejected under a similar rationale as that of Claim 1.)
Claims 3, 13, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Tai in view of Kim and Yoo and in further view of Scalise et al. (US 20100215259 A1) (Hereinafter referred to as Scalise).
Regarding Claim 3, Tai in view of Kim and Yoo discloses The electronic device of claim 1, wherein the at least one instruction, when executed by the one or more processors, causes the electronic device to: downscale the input image; (See Tai Page 8 Paragraph 5, “In this case, the prediction module 320 may downscale the input image and input the downscaled image to the second artificial intelligence model to identify the type.”)
obtain the focus map based on importance information, (See Yoo Page 3 Paragraphs 4-5, “The FDM generation unit 130 generates a focus depth map from the image. The focus depth map is a depth map of the degree of blur in the image. The focus depth map is based on the assumption that the object to be photographed increases in focus blur as it is moved away from the device from which the image was taken. Therefore, the focus depth map is determined by the amount of focus blur.” Once again, the amount of focus blur corresponds to “importance information”.)
However, Tai in view of Kim and Yoo fails to explicitly disclose identify the downscaled image as a plurality of regions and obtain a region map;
obtain a plurality of importance values according to a plurality of different characteristics with respect to each of the plurality of regions; and
obtain the focus map based on the plurality of importance values, wherein the plurality of different characteristics include at least one of color difference information, skin color information, face probability information, or high frequency information.
Scalise teaches identify the downscaled image as a plurality of regions and obtain a region map; (See Fig. 1 element 10, “Receive Digital Image” and then element 28, “Segment Digital Image Into Regions”. In this case, the segmented image can be considered as a “region map”. In combination with Tai Page 8 Paragraph 5 already teaching to downscale the image, the above limitations is taught.)
obtain a plurality of importance values according to a plurality of different characteristics with respect to each of the plurality of regions; and (See Fig. 1 showing in element 22, that after the image is segmented into regions, a main subject importance map is generated.
Also see [0050], “In the embodiment of FIGS. 1-2, the main subject map is the above-described main subject importance map and the subject values are importance values.”
Lastly, see [0051], “Referring to FIG. 1, the region map and blended map are both input to a main subject importance detector (MSI) that uses those maps to generate a main subject importance map. The MSI relates the skin color values to respective segments and considers additional information in generating importance values for each of the regions of the main subject importance map.” Here, skin color values corresponds to a “plurality of different characteristics”.)
obtain the focus map based on the plurality of importance values, wherein the plurality of different characteristics include at least one of color difference information, skin color information, face probability information, or high frequency information. (See [0070], “The importance values can be associated with the respective segments to provide a main subject importance map.”
Also see [0074], “Referring now to FIG. 2, in the cropping step (24), the main subject map is thresholded (30) to define a main subject and background.”
Lastly, see [0051], “Referring to FIG. 1, the region map and blended map are both input to a main subject importance detector (MSI) that uses those maps to generate a main subject importance map. The MSI relates the skin color values to respective segments and considers additional information in generating importance values for each of the regions of the main subject importance map.”
In this case, a main subject map corresponds to a focus map as a main subject map, similar to a focus map, is used to define a main subject and background. In particular, Scalise teaches importance values associated with said focus map. Scalise also teaches skin color values (skin color information) for the main subject map.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Tai in view of Kim and Yoo with Scalise to include obtaining a region map, importance values based on different characteristics, and a focus map based on the importance values.
The motivation to combine Tai in view of Kim and Yoo with Scalise would have been obvious as Kim, Yoo, and Scalise are all within the same field of focus maps (See Scalise [0070] and [0074]). The benefits of obtaining a region map is that it can be used to identify more objects or sections than just the foreground versus background. Importance values based on different characteristics can be beneficial when deciding what determines that an area is salient, and in the case for Scalise, skin color would be a metric that draws salience to an area.
Regarding Claim 13, Claim 13 contains similar limitations as to Claim 3 and is therefore rejected under a similar rationale as that of Claim 3.
Regarding Claim 17, Claim 17 recites similar limitations as to Claim 3 and is therefore rejected under a similar rationale as that of Claim 3.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Tai in view of Kim and Yoo and in further view of Cao et al. (CN 107665347 A) (Hereinafter referred to as Cao)
Regarding Claim 10, Tai in view of Kim and Yoo discloses The electronic device of claim 1,wherein the at least one instruction, when executed by the one or more processors, causes the electronic device to: obtain a filtered focus map by applying at least one of temporal filtering or spatial filtering to the input image; and (See Yoo Page 4 Paragraph 3, “Referring to FIG. 3 (a), the FDM reliability calculation unit 120 detects an edge from the image 310 using a Laplacian filter. The detected edge pixels are displayed in the edge pixel image 320.” In this case, a Laplacian filter is a “spatial filter”, which is used to detect edge pixels.)
obtain the reliability information per region of the focus map based on at least one of the brightness information or the contrast information of the input image and information included in the filtered input image. (See Yoo Page 4 Paragraph 2, “When the image has a ground, the FDM reliability calculation unit 120 calculates the reliability of the focus depth map based on the number of focus pixels located in a lower region of the image. A focus pixel means a pixel having a laplacian focus value of the edge pixels of 10% or more in the image. The Laplacian focus value is the value that the edge pixels return when applying the Laplacian function to the pixels of the image. The larger the Laplacian focus value, the more pixels can be understood to be focused.” In this case, calculating reliability based on focus pixels can be considered as obtaining reliability information based on “contrast information” as the calculation for focus pixels can be considered as a measure of edge contrast.)
However, Tai in view of Kim and Yoo fails to explicitly disclose obtain a filtered focus map by applying at least one of temporal filtering or spatial filtering to the focus map;
obtain the reliability information per region of the focus map based on at least one of the brightness information or the contrast information of the input image and information included in the filtered focus map.
Cao teaches obtain a filtered focus map by applying at least one of temporal filtering or spatial filtering to the focus map; (See Abstract, “The invention relates to a filter-based optimization of visual significance target detection method, the method is specifically as follows: 1) using image division method divides the original image into multiple areas . . . step 3) calculating the saliency map value of each area is formed; 5) applying quick guiding and filtering method to optimize the saliency map.” In this case, Cao teaches applying a filter to a saliency map (focus map). In combination with Yoo Page 4 Paragraph 3 teaching a Laplacian filter (spatial filter), the above limitations are taught.)
obtain the reliability information per region of the focus map based on at least one of the brightness information or the contrast information of the input image and information included in the filtered focus map. (See Cao Abstract teaching a filtered the saliency map (filtered focus map). In combination with Yoo Page 4 Paragraph 2 teaching to calculate reliability of the focus depth map, instead of a normal focus map, the reliability would be calculated for the filtered focus map.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Tai in view of Kim and Yoo with Cao to include applying a filtering method to a focus map.
The motivation to combine Tai in view of Kim and Yoo with Cao would have been obvious as Kim, Yoo, and Cao are all within the same field pertaining to focus maps (See Cao Abstract). The benefit of a filtering a focus map would be that it optimizes the map while maintaining the subjective visual edge or texture information of significant targets (See Cao Page 4 Paragraph 4).
Allowable Subject Matter
Claims 2, 4-9, 12, 14, 16, and 18-20 objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding Claim 2, the cited prior art does not disclose or render obvious the combination of elements cited in the claims as a whole. Specifically, the cited prior art fails to disclose or render obvious the limitations: identify the sensitivity information of the focus map with respect to a first type of image quality processing as relatively low and scale a value included in the focus map to become large; identify the sensitivity information of the focus map with respect to a second type of image quality processing as relatively high and scale the value included in the focus map to become small; and image-quality process the input image according to the at least one type of image quality processing based on the scaled focus map, the reliability information per region of the focus map, and the sensitivity information, wherein the first type of image quality processing includes at least one of noise reduction processing or detail enhancement processing, and wherein the second type of image quality processing includes at least one of contrast ratio enhancement processing, color enhancement processing, or brightness processing. Thus, Claim 2 contains allowable subject matter.
Regarding Claim 4, the cited prior art does not disclose or render obvious the combination of elements cited in the claims as a whole. Specifically, the cited prior art fails to disclose or render obvious the limitations: downscale the brightness information of the input image to a size of the focus map; identify a background brightness value of the input image based on an inverse value of the value included in the focus map and the downscaled brightness information; and obtain a first reliability gain value based on the background brightness value of the input image. Thus, Claim 4 contains allowable subject matter.
Regarding Claims 5-9, Claims 5-9 are dependent upon the base Claim of 4 and thus therefore also contains allowable subject matter.
Regarding Claim 12, Claim 12 recites similar limitations as to Claim 2 and therefore also contains allowable subject matter.
Regarding Claim 14, Claim 14 contains similar limitations as to Claim 4 and therefore also contains allowable subject matter.
Regarding Claim 16, Claim 16 recites similar limitations as to Claim 2 and therefore also contains allowable subject matter.
Regarding Claim 18, Claim 18 contains similar limitations as to Claim 4 and therefore also contains allowable subject matter.
Regarding Claims 19-20, Claims 19-20 are dependent upon the base Claim of 4 and thus therefore also contains allowable subject matter.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to THANG G HUYNH whose telephone number is (571)272-5432. The examiner can normally be reached Mon-Thu 7:30am-4:30pm EST | Fri 7:30am-11:30am EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KEE M TUNG/Supervisory Patent Examiner, Art Unit 2611
/T.G.H./Examiner, Art Unit 2611
Prosecution Insights