DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Election/Restrictions
Restriction to one of the following inventions is required under 35 U.S.C. 121:
I. Claims 1-10, drawn to a method of generating estimated images by combining images from different models, classified in G06T5/50: for using two or more images, e.g. averaging or subtraction
II. Claims 11-24, drawn to a method for generating images and imparted components using a first and second machine learning model, classified in G06N3/045: for combination of networks.
Inventions I and II are related as subcombinations disclosed as usable together in a single combination. The subcombinations are distinct if they do not overlap in scope and are not obvious variants, and if it is shown that at least one subcombination is separately usable.
In the instant case, subcombination I has separate utility such as determining different weights for different areas of a combined image.
Further, subcombination II has separate utility such as generating images, imparted components, and adjustment parameters by utilizing a first and second machine learning model.
See MPEP § 806.05(d).
The examiner has required restriction between combination and subcombination inventions. Where applicant elects a subcombination, and claims thereto are subsequently found allowable, any claim(s) depending from or otherwise requiring all the limitations of the allowable subcombination will be examined for patentability in accordance with 37 CFR 1.104. See MPEP § 821.04(a). Applicant is advised that if any claim presented in a continuation or divisional application is anticipated by, or includes all the limitations of, a claim that is allowable in the present application, such claim may be subject to provisional statutory and/or nonstatutory double patenting rejections over the claims of the instant application.
In a voicemail left by Richard Schachner (48,342) on September 3rd, 2026 a provisional election was made without traverse to prosecute the invention of group I, claims 1-10. Affirmation of this election must be made by applicant in replying to this Office action. Claims 11-24 are withdrawn from further consideration by the examiner, 37 CFR 1.142(b), as being drawn to a non-elected invention.
Claims 1-10 are pending regarding this application.
Priority
The present application claims foreign priority benefits from JP2024-010400 filed on
01/26/2024 and JP2024-050589 filed on 03/26/2024. The certified copies of the priority documents were electronically retrieved on 03/05/2025.
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Acknowledgment is made of applicant's claim for foreign priority based on an application filed in Japan (JP2025-007096) on 01/17/2025. It is noted, however, that applicant has not filed a certified copy of the JP2025-007096 application as required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 01/23/2025 is considered and attached.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Claim Objections
Claim 9 is objected to because of the following informalities:
Please amend “wherein the estimated image is an image having a defocus blur with a shape different from a shape of a defocus blur of the input image” in claim 9 to recite “wherein the estimated image is an image having a first defocus blur with a first shape different from a second shape of a second defocus blur of the input image” in order to promote clarity within the distinction between the shape of the defocus blur of the estimated image and the shape of the defocus blur of the input image.
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 10 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because signals per se do not fall into one of the four statutory categories.
Claim 10 recites, inter alia, “A computer-readable storage medium storing a computer program…” After close inspection, the Examiner respectfully notes that the disclosure, as a whole, does not definitively describe what can and cannot be considered the “computer-readable storage medium”. Applicant’s specification discusses the “computer-readable storage medium” in paragraph 0186. Applicant describes the “computer-readable storage medium” as possibly being a non-transitory storage medium. However, applicant uses the word “may” to describe the fact that it might be non-transitory. Thus, the “computer-readable storage medium” could be other forms – such as a signal. See also that “The storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read-only memory (ROM), a storage of distributed computing systems, an optical disc (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)TM), a flash memory device, a memory card, and the like.” as recited in para. [0186] (emphasis added).
An Examiner is obliged to give claims their broadest reasonable interpretation consistent with the specification during examination. The broadest reasonable interpretation of a claim drawn to a computer program product (also called a computer readable medium, machine readable medium and other such variations) typically covers forms of non-transitory tangible media and transitory propagating signals per se in view of the ordinary and customary meaning of computer readable media, particularly when the specification is silent. See MPEP 2111.01. When the broadest reasonable interpretation of a claim covers a signal, per se, the claim must be rejected under 35 U.S.C. § 101 as covering non-statutory subject matter.
Therefore, given the non-definitive disclosure and the broadest reasonable interpretation, the machine-readable storage medium of the claim may include transitory propagating signals. As a result, the claim pertains to non-statutory subject matter.
However, the Examiner respectfully submits a claim drawn to such a computer program product or computer readable storage medium that covers both transitory and non-transitory embodiments may be amended to narrow the claim to cover only statutory embodiments to avoid a rejection under 35 U.S.C. § 101 by adding the limitation “non-transitory” to the claim. Such an amendment would typically not raise the issue of new matter, even when the specification is silent because the broadest reasonable interpretation relies on the ordinary and customary meaning that includes signals per se. For additional information, please see the Patents’ Official Gazette notice published February 23, 2010 (1351 OG 212).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 5-7, and 10 are rejected under 35 U.S.C. 103 as being unpatentable over He (CN 113781309 A, see attached English translation for citations) in view of Lee et al. (U.S. Publication No. 2024/0412325 A1), hereinafter Lee.
Regarding claim 1, He teaches an image processing method comprising:
a step of generating an estimated image from an input image by using a plurality of machine learning models including a generative model and a non-generative model (He teaches “the image processing module is used for respectively inputting the image to be processed into the CNN network model and the GAN network model for image processing to obtain a first image output by the CNN network model and a second image output by the GAN network model” in para. [0040]. Here, the CNN is the non-generative model and the GAN is the generative model),
wherein, in the step, the estimated image is generated by assigning different weights to output of the generative model and output of the non-generative model (He teaches that, “by setting the weight, the image fusion is carried out, and different fusion weights can be set according to actual conditions so as to obtain better image effect” as shown in para. [0066]. Here, since the fusion weight may be determined based on the image conditions, it is clear that He teaches assigning different weights based on information regarding the input image).
He fails to teach assigning different weights to output of the generative model and output of the non-generative model for each of a plurality of areas of the input image (emphasis added).
However, Lee similarly teaches a method of generating an estimated image from an input image by using a plurality of machine learning models (Lee teaches that “the plurality of neural network models may include a first neural network model and a second neural network model, and the processor 120 may acquire a first image by inputting the input image to the first neural network model, acquire a second image by inputting the input image to the second neural network mode, and acquire an output image by weighting the first image and the second image based on the weight map” as shown in para. [0072]), and additionally teaches wherein the estimated image is generated by assigning different weights to output of the [first] model and output of the [second] model for each of a plurality of areas of the input image based on information regarding the input image (Lee teaches that “the output image may be an image in which the input image is upscaled, a weight of the first image processed by the first neural network model may be highly reflected in the object area, and a weight of the second image processed by the second neural network model may be highly reflected in the background area” as shown in para. [0073]. See also para. [0074]).
He and Lee are both considered to be analogous to the claimed invention because they are in the same field of using multiple models to generate images with improved image quality. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of He to incorporate the teachings of Lee and include “assigning different weights to output of the generative model and output of the non-generative model for each of a plurality of areas of the input image” (emphasis added). The motivation for doing so would have been that “as the weights are reflected differently for each area, upscaling processing in which area characteristics are more reflected for each area may be possible” and additionally, “as the electronic apparatus 100 uses a plurality of neural network models specialized for each object type, image processing performance may be improved”, as suggested by Lee in para. [0074] and [0104], respectively. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine He with Lee to obtain the invention specified in claim 1.
Regarding claim 2, He and Lee teach the image processing method according to claim 1, wherein the information regarding the input image includes at least one of
distance information regarding the input image,
a segmentation map (Lee teaches “the processor 120 may acquire a weight map based on an object area included in an input image” as shown in para. [0067]. This weight map is interpreted as equivalent to the claimed segmentation map),
information regarding a saturated area, and
an optical performance map.
**Please note: only one of the above limitations need be found in the prior art due to the “at least one of” language.
Similar motivations as applied to claim 1 can be applied here to claim 2.
Regarding claim 5, He and Lee teach the image processing method according to claim 1
wherein the information regarding the input image includes a segmentation map (Lee teaches “the processor 120 may acquire a weight map based on an object area included in an input image” as shown in para. [0067]. This weight map is interpreted as equivalent to the claimed segmentation map), and
wherein the estimated image is generated with the weight of output of the non-generative model being larger than the weight of output of the generative model for an area determined to be an area including a person among the plurality of areas based on the segmentation map (Lee teaches “the processor 120 may give the object area included in the first image a higher weight than the object area included in the second image” as shown in para. [0074]. Here, the first image is generated by the first neural network model, which may be non-generative as shown in para. [0103], and the second image is the output of the second neural network model, which may be a generative model as shown in para. [0103]. Furthermore, the object area as taught by Lee may be a face area/person as shown in para. [0067]. See also that He teaches the generative and non-generative models specifically as shown in claim 1), and
generated with the weight of output of the generative model being larger than the weight of output of the non-generative model for an area determined to be an area including no person among the plurality of areas based on the segmentation map (Lee teaches that “the background area included in the first image a lower weight than the background area included in the second image to weight the first image and the second image” as shown in para. [0074]. Here, the first image is generated by the first neural network model, which may be non-generative as shown in para. [0103], and the second image is the output of the second neural network model, which may be a generative model as shown in para. [0103]. Furthermore, the object area as taught by Lee may be a face area/person as shown in para. [0067]. Here, background area is interpreted as equivalent to the claimed area including no person among the plurality of areas. See also that He teaches the generative and non-generative models specifically as shown in claim 1).
Similar motivations as applied to claim 1 can be applied here to claim 5.
Regarding claim 6, He and Lee teach the image processing method according to claim 1,
wherein the estimated image is generated by using output of the non-generative model for an area determined to be a first area among the plurality of areas based on information regarding the input image, and generated by using an output of the generative model for an area determined to be a second area among the plurality of areas based on information regarding the input image (Lee teaches generating an output image wherein the object area reflects the output of a first neural network and the background area reflects the output of a second neural network, wherein the first neural network may be a non-generative model, and the second neural network may be a generative model as shown in para. [0103]. See also that He teaches the non-generative and generative model specifically, as shown in claim 1).
Similar motivations as applied to claim 1 can be applied here to claim 6.
Regarding claim 7, He and Lee teach the image processing method according to claim 1, wherein, in the step,
the estimated image is generated by weighted-averaging a first image and a second image based on information regarding the input image, the first image being generated by inputting the input image to the generative model, the second image being generated by inputting the input image to the non-generative model (He teaches generating a target image by performing a weighted fusion of a first and second image as shown in para. [0061]-[0066], wherein the weight can be set based on the conditions of the input image, and the first image is output from a (non-generative) CNN (see para. [0054]) and the second image is output from a (generative) GAN (see para. [0040]). The weighted fusion as taught by He is interpreted as equivalent to the claimed weighted averaging process).
Regarding claim 10, He and Lee teach a computer-readable storage medium storing a computer program that causes a computer to execute the image processing method according to claim 1 (He teaches “an embodiment of the present application further provides an electronic device, which includes a processor and a memory, where the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the method in the first aspect” in para. [0020]).
Claims 3 and 4 are rejected under 35 U.S.C. 103 as being unpatentable over He (CN 113781309 A, see attached English translation for citations) in view of Lee et al. (U.S. Publication No. 2024/0412325 A1), hereinafter Lee and Nakada (U.S. Publication No. 2012/0301044 A1).
Regarding claim 3, He and Lee teach the image processing method according to claim 2.
He and Lee fail to teach wherein the distance information regarding the input image is a defocus map or a depth map
However, Nakada teaches wherein the distance information regarding the input image is a defocus map or a depth map (Nakada teaches “the region division processing unit 305 divides a scene between a region to be put into focus and another region, and generates region-divided image data 315 based on the virtual focus parameter 310 acquired in step S402 and the depth image data 314 generated in step S404” in para. [0064], wherein “the distance estimation processing unit 304 serving as a functional unit of the CPU 101 performs stereo matching to estimate a depth value of a captured scene region by region and generates depth image data 314 based on the captured image data 308… [and] The depth image data 314 is image data that includes depth information with respect to an object in each region (or at each pixel)” as shown in para. [0038]. Here, the captured image data 308 is interpreted as equivalent to the claimed input image and the depth image information is interpreted as equivalent to the claimed depth map).
He, Lee, and Nakada are all considered to be analogous to the claimed invention because they are in the same field of generating images with improved image quality. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of He (as modified by Lee) to incorporate the teachings of Nakada and include “wherein the distance information regarding the input image is a defocus map or a depth map”. The motivation for doing so would have been that “it is possible to suppress a blur in an in-focus region (region within the depth of field) and improve the image quality of a blurred region outside the depth of field when generating a synthesized image based on image data 308 ”, as suggested by Nakada in para. [0083]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine He and Lee with Nakada to obtain the invention specified in claim 3.
Regarding claim 4, He and Lee teach the image processing method according to claim 1
wherein the estimated image is generated with the weight of output of the non-generative model being greater than the weight of output of the generative model for an area determined to be an (Lee teaches “the processor 120 may give the object area included in the first image a higher weight than the object area included in the second image” as shown in para. [0074]. Here, the first image is generated by the first neural network model, which may be non-generative as shown in para. [0103], and the second image is the output of the second neural network model, which may be a generative model as shown in para. [0103]. Furthermore, the object area as taught by Lee may be a face area/person as shown in para. [0067]. See also that He teaches the generative and non-generative models specifically as shown in claim 1), and generated with the weight of output of the generative model being larger than the weight of output of the non-generative model for an area determined to be an (Lee teaches that “the background area included in the first image a lower weight than the background area included in the second image to weight the first image and the second image” as shown in para. [0074]. Here, the first image is generated by the first neural network model, which may be non-generative as shown in para. [0103], and the second image is the output of the second neural network model, which may be a generative model as shown in para. [0103]. Furthermore, the object area as taught by Lee may be a face area/person as shown in para. [0067]. Here, background area is interpreted as equivalent to the claimed area including no person among the plurality of areas. See also that He teaches the generative and non-generative models specifically as shown in claim 1).
While Lee and He teach wherein the estimated image is generated with the weight of output of the non-generative model being greater than the weight of output of the generative model for an area determined to be an in-focus area among the plurality of areas and generated with the weight of output of the generative model being larger than the weight of output of the non-generative model for an area determined to be an out-of-focus area among the plurality of areas as shown above, He and Lee fail to teach in-focus and out-of-focus areas wherein the information regarding the input image includes distance information regarding the input image and weighting the in-focus and out-of-focus areas based on the distance information regarding the input image. However, Nakada teaches wherein the information regarding the input image includes distance information regarding the input image and weighting the in-focus and out-of-focus areas based on the distance information regarding the input image (Nakada teaches “weighting factors Win for interpolation image data 312 in an in-focus region only need to be smaller than weighting factors Wout in at least a part of an out-of-focus region. This can suppress a blur in the in-focus region caused by synthesis, and produce a blur of high image quality in at least a part of the out-of-focus region” in para. [0084], wherein the in-focus and out-of-focus areas are determined based on the depth image data 314 as shown in para. [0064]-[0066]).
He, Lee, and Nakada are all considered to be analogous to the claimed invention because they are in the same field of generating images with improved image quality. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of He (as modified by Lee) to incorporate the teachings of Nakada and include “wherein the information regarding the input image includes distance information regarding the input image and weighting the in-focus and out-of-focus areas based on the distance information regarding the input image”. The motivation for doing so would have been that “it is possible to suppress a blur in an in-focus region (region within the depth of field) and improve the image quality of a blurred region outside the depth of field when generating a synthesized image based on image data 308”, as suggested by Nakada in para. [0083]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine He and Lee with Nakada to obtain the invention specified in claim 4.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over He (CN 113781309 A, see attached English translation for citations) in view of Lee et al. (U.S. Publication No. 2024/0412325 A1), hereinafter Lee and Koumura et al. (U.S. Publication No. 2026/0030482 A1), hereinafter Koumura.
Regarding claim 8, He and Lee teach the image processing method according to claim 1.
While He and Lee teach the generative and non-generative model generating a first image and the estimated image (see claim 1), He and Lee fail to teach wherein, in the step, a first image is generated by inputting the input image and the information regarding the input image to one of the generative model and the non-generative model, and the estimated image is generated by inputting the first image and the information regarding the input image to the other of the generative model and the non-generative model.
However, Koumura teaches wherein, in the step, a first image is generated by inputting the input image and the information regarding the input image to one of the (Koumura teaches generating a feature map (first image) by inputting input image data into a global coarse-scale network (first model), and generating a depth map (estimated image) by inputting the input image data and the feature map (first image) into a local fine-scale network (second image) as shown in para. [0128] and FIG. 3. See also para. [0152], wherein the depth map is output as image data). This teaching of Koumura can be combined with He and Lee’s teaching of specifically generating the first image and the estimated image using a non-generative and generative model, respectively, to teach the entirety of the limitation.
He, Lee, and Koumura are all considered to be analogous to the claimed invention because they are in the same field of generating images with improved image quality. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of He (as modified by Lee) to incorporate the teachings of Koumura and include “wherein, in the step, a first image is generated by inputting the input image and the information regarding the input image to one of the 200 or the like” and such that “ arithmetic operation can be performed using the feature map [first image] in the network RNT [second model] described below”, as suggested by Koumura in para. [0156] and [0141], respectively. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine He and Lee with Koumura to obtain the invention specified in claim 8.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over He (CN 113781309 A, see attached English translation for citations) in view of Lee et al. (U.S. Publication No. 2024/0412325 A1), hereinafter Lee and Kobayashi (U.S. Publication No. 2021/0097650 A1).
Regarding claim 9, He and Lee teach the image processing method according to claim 1.
He and Lee fail to teach wherein the estimated image is an image having a defocus blur with a shape different from a shape of a defocus blur of the input image.
However, Kobayashi teaches wherein the estimated image is an image having a defocus blur with a shape different from a shape of a defocus blur of the input image (Kobayashi teaches a method of correcting a defocus blur shape in an input image and generating an estimated image with the corrected defocus blur shape as shown in para. [0098]-[0100] and [0108]-[0109]. Here, since the defocus blur shape is corrected in the estimated image, it is inherent that the shape of the defocus blur shape is changed from the input image to the estimated image).
He, Lee, and Kobayashi are all considered to be analogous to the claimed invention because they are in the same field of generating images with improved image quality. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of He (as modified by Lee) to incorporate the teachings of Kobayashi and include “wherein the estimated image is an image having a defocus blur with a shape different from a shape of a defocus blur of the input image”. The motivation for doing so would have been to “correct a defocus blur shape in a captured image into various shapes while suppressing increases of a learning load and a stored data amount for a machine learning model”, as suggested by Kobayashi in para. [0005]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine He and Lee with Kobayashi to obtain the invention specified in claim 9.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner
should be directed to KYLA G ALLEN whose telephone number is (703)756-5315. The examiner can
normally be reached M-F 7:30am - 4:30pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a
USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use
the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor,
John Villecco can be reached on (571) 272-7319. The fax phone number for the organization where this
application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from
Patent Center. Unpublished application information in Patent Center is available to registered users. To
file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit
https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and
https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional
questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like
assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or
571-272-1000.
/Kyla Guan-Ping Tiao Allen/
Examiner, Art Unit 2661
/AARON W CARTER/Primary Examiner, Art Unit 2661
Prosecution Insights