Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed December 16, 2025 have been fully considered but they are not persuasive.
On page 8 of Remarks, Applicant stated on last paragraph, “The Examiner appears to have failed to fully appreciate the application of different image processing before and after demosaicing”. However Baqai reference teaches “the application of different image processing before and after demosaicing” (column 10, line 44-45; column 11, lines 14-19; noise reduction (second processing unit) can be performed on fused image that is demosaiced; column 19, lines 43-55; noise reduction 176 (second processing unit) executes noise reduction processing (that is different than first processing unit that performs synthesis)). As stated in column 11, lines 14-19 of Baqai, noise reduction processing (second processing) is performed after demosaicing of the image performed in the first processing including synthesizing of image (first processing). The first processing of synthesizing is different from the second processing of noise reduction following the demosaicing process. However in Baqai it does not clearly teach when the demosaicing process is performed before the noise reduction processing. The system of Yano was relied on to teach demosaicing after the synthesis process as claimed in claim 1. Regarding argument on page 9 of Remarks which states “However, the applicant respectfully submits that the Examiner fails to explain why it would have been obvious to a Person Having Ordinary Skill in the Art to modify Baquai's system based on Yano's disclosure to arrive at the feature as recited in amended claim 1 of the present application: "applying different image processing before and after demosaicing" because Yano merely discloses raw image and demosaicing”, Examiner disagrees because Yano was only relied on to teach demosaicing process after the synthesis process. Baqai teaches different processing before and after demosaicing.
The motivation to modify the Baqai such that the demosaicing is performed on image data after synthesis process is to simplify the image processing performed in the first processing by performing the demosaicing after synthesis. It would be more complicated processing if the demosaicing is performed before the synthesis requiring synthesis operation on demosaiced image data which requires more computation since multiple demosaiced R, G, B image plane data that have to be synthesized.
Claim Objections
Claim 1 is objected to because of the following informalities: Claim 1 discloses “a storage medium stores a program for executing a first processing unit, a map generation unit, and a second processing unit”. However the program cannot execute by itself the processing units. Examiner suggest amending “a storage medium that stores a program that is executed by a processor for implementing a first processing unit, a map generation unit, and a second processing unit”. Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 2, 6, 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 11189017 to Baqai in view of US 20140270518 to Yano.
Regarding claim 1, Baqai discloses an image processing device comprising (column 26, lines 56-62; image processor):
a storage medium stores a program for executing a first processing unit, a map
generation unit, and a second processing unit (column 4, lines 22-55; program stored in memory that executes first processing of determining corresponding pixels, fusion, map generation, and second processing of noise reduction processing),
the first processing unit, detecting corresponding pixels between reference image data selected from among a plurality of pieces of image data and each of a plurality of pieces of comparison image data comprised in the plurality of pieces of image data (column 9, lines 40-55; step 106 selects reference image from plurality of image; column 11, lines 22-30, 40-60; detecting whether pixel in reference image is similar to pixel in each of candidate image (comparison image) from plurality of images; column 9, lines 57-59; processor; column 11, lines 21; processor performing “adaptively fusing images 110” is first processing unit), synthesizing the reference image data and the plurality of pieces of comparison image data on the basis of the corresponding pixels, generating synthesized image data (column 17, lines 14-21; column 18, lines 42-60; column 20, lines 15-18; the reference image is fused with each of the candidate images (comparison image data) on basis of the weight map which is indication of similarity (correspondence) of the pixels and generating fused image), and executing image processing on the reference image data and the plurality of pieces of comparison image data (column 9, lines 50-58; column 18, lines 8-15; first processing further includes application of image registration processing (executing image processing) to candidate images and reference image in step 106 before the fusion process in step 110);
the map generation unit (column 19, lines 65-67; fusion-adaptive noise reduction processing unit is used to generate map), generating map information in which a pixel position of the synthesized image data is associated with information derived from at least one piece of image data of the plurality of pieces of image data (column 18, lines 65-67; column 19, lines 1-49; column 19, lines 32-41, 42-55; column 21, lines 16-22; noise map generated in 190; pixel position in fused image is associated with entry pixel in the fusion adaptive noise map which is associated with weight maps derived from the images since fusion adaptive noise map is combination of weight map and noise maps); and
the second processing unit, executing image processing different from that of the first processing unit on the synthesized image data that is demosaiced, on the basis of the map information generated by the map generation unit (column 10, line 44-45; column 11, lines 14-19; noise reduction (second processing unit) can be performed on fused image that is demosaiced; column 19, lines 43-55; noise reduction 176 (second processing unit) executes noise reduction processing (that is different than first processing unit that performs synthesis) by performing noise reduction pixel-by-pixel on the fused image data using the noise map generated).
However Baqai does not disclose the plurality of pieces of raw image data and
the synthesized raw image data that has been demosaiced.
Yano discloses the plurality of pieces of raw image data and the synthesized raw image data that has been demosaiced (paragraph 3, 127, 136 144; the synthesized image is RAW image and plurality of RAW image; demosaicing is performed on synthesized image).
It would have been obvious to one of ordinary skill in the art at the time of the invention was made to modify the system of Baqai as taught by Yano to perform demosaicing process on synthesized RAW image data.
The motivation to combine the references is to simplify the image processing performed in the first processing by performing the demosaicing after synthesis process in contrast to performing the demosaicing before the synthesis requiring synthesis operation on demosaiced image data which requires more computation since there are multiple R, G, B image plane data that have to be synthesized if it is demosaiced and further motivation is to provide full color image data that can be processed in subsequent processing for display (paragraph 144).
Regarding claim 2, Baqai discloses the image processing device according to claim 1, wherein the information derived from the at least one piece of image data is information regarding presence or absence of the corresponding pixels (column 18, lines 65-67; column 19, lines 1-49; column 19, lines 32-41, 42-55; column 21, lines 16-22; noise map generated in 190; pixel position in fused image is associated with entry pixel in the fusion adaptive noise map which is associated with weight maps derived from the images; column 11, lines 46-60; candidate pixel being similar to reference image pixel is associated with presence and candidate pixel being NOT similar to reference image pixel is associated with absence in weight map). Further Yano discloses raw image data (paragraph 3, 127, 136 144; the synthesized image is RAW image and plurality of RAW image).
Regarding claim 6, Baqai discloses a non-transitory computer-readable medium storing an image processing program causing a computer to function as (column 26, lines 56-62; image processor; column 26, lines 61-67; column 27, lines 1-17; memory 460 storing programs for image processing to be executed by processor 405):
a first processing unit, detecting corresponding pixels between reference image data selected from among a plurality of pieces of image data and each of a plurality of pieces of comparison image data comprised in the plurality of pieces of image data (column 9, lines 40-55; step 106 selects reference image from plurality of image; column 11, lines 22-30, 40-60; detecting whether pixel in reference image is similar to pixel in each of candidate image (comparison image) from plurality of images; column 9, lines 57-59; processor; column 11, lines 21; processor performing “adaptively fusing images 110” is first processing unit), synthesizing the reference image data and the plurality of pieces of comparison image data on the basis of the corresponding pixels, generating synthesized image data (column 17, lines 14-21; column 18, lines 42-60; column 20, lines 15-18; the reference image is fused with each of the candidate images on basis of the weight map which is indication of similarity (correspondence) of the pixels and generating fused image), and
executing image processing on the reference image data and the plurality of pieces of comparison image data (column 9, lines 50-58; column 18, lines 8-15; first processing further includes application of image registration processing (executing image processing) to candidate images and reference image in step 106 before the fusion process in step 110);
a map generation unit (column 19, lines 65-67; fusion-adaptive noise reduction processing unit is used to generate map), generating map information in which a pixel position of the synthesized image data is associated with information derived from at least one piece of image data of the plurality of pieces of image data (column 18, lines 65-67; column 19, lines 1-49; column 19, lines 32-41, 42-55; column 21, lines 16-22; noise map generated in 190; pixel position in fused image is associated with entry pixel in the fusion adaptive noise map which is associated with weight maps derived from the images since fusion adaptive noise map is combination of weight map and noise maps); and
a second processing unit, executing image processing different from that of the first processing unit on the synthesized image data that is demosaiced, on the basis of the map information generated by the map generation unit (column 10, line 44-45; column 11, lines 14-19; noise reduction (second processing unit) can be performed on fused image that is demosaiced; column 19, lines 43-55; noise reduction 176 (second processing unit) executes noise reduction processing (that is different than first processing unit that performs synthesis) by performing noise reduction pixel-by-pixel on the fused image data using the noise map generated).
However Baqai does not disclose the plurality of pieces of raw image data and
the synthesized raw image data that has been demosaiced.
Yano discloses the plurality of pieces of raw image data and the synthesized raw image data that has been demosaiced (paragraph 3, 127, 136 144; the synthesized image is RAW image and plurality of RAW image; demosaicing is performed on synthesized image).
It would have been obvious to one of ordinary skill in the art at the time of the invention was made to modify the system of Baqai as taught by Yano to perform demosaicing process on synthesized RAW image data.
The motivation to combine the references is to simplify the image processing performed in the first processing by performing the demosaicing after synthesis process in contrast to performing the demosaicing before the synthesis requiring synthesis operation on demosaiced image data which requires more computation since there are multiple R, G, B image plane data that have to be synthesized if it is demosaiced and further motivation is to provide full color image data that can be processed in subsequent processing for display (paragraph 144).
Regarding claim 7, see rejection of claim 1.
Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 11189017 to Baqai in view of US 20140270518 to Yano further in view of CN 105976308 to Lai.
Regarding claim 3, Baqai discloses the image processing device according to claim 2, wherein
the first processing unit executes image processing that averages the corresponding pixels of the reference image data and the corresponding pixels of the plurality of pieces of comparison image data (column 13, lines 50-67; fused pixel values can be determined by averaging the x pixel values from each of reference image and comparison images at each position via Eqn. 5); and
the second processing unit executes image processing that applies a filter to a pixel value of the pixel position of the synthesized image data associated with information indicating the absence of the corresponding pixels (column 19, lines 42-55; second processing unit involves the noise reduction 176 that applies noise reduction by looking up entry in the fusion noise map for specific pixel position in the fused image and applying this entry to reduce noise of this pixel thereby performing filtering using the noise map; column 13, lines 14-30; noise reduction is performed at all pixel positions including positions wherein weight value is 0 for each of comparison images (weight value=0 is information indicating absence of the corresponding pixels)). Further Yano discloses the synthesized raw image data (paragraph 3, 127, 136 144; the synthesized image is RAW image and plurality of RAW image; demosaicing is performed on synthesized image).
However Baqai in view of Yano does not disclose processing unit that executes image processing that applies a smoothing filter to a pixel value of the pixel position.
Lai discloses processing unit that executes image processing that applies a smoothing filter to a pixel value of the pixel position (paragraph 14; GPU hardware (second processing unit) performs image processing; paragraph 22, 33-36; image denoising in s113 uses smoothing filtering that involves formula having a K value representing strength of filtering which is associated with amount of noise to be removed for pixel at position (i, j)).
It would have been obvious to one of ordinary skill in the art at the time of the invention was made to modify the system of Baqai in view of Yano as taught by Lai to provide smoothing filter image processing to pixels.
The motivation to combine the references is to apply smoothing filter to synthesized image data to denoise the image data by controlling the filter parameters such that the smoothing is maximized resulting in smoothed pixel value close to the average value (paragraph 32-36).
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 11189017 to Baqai in view of US 20140270518 to Yano further in view of US 20080152256 to Ou.
Regarding claim 4, Yano discloses the image processing device according to claim 1, wherein the information derived from the at least one piece of raw image data (paragraph 79, 127; pixel value derived from RAW image).
However Yano does not disclose wherein the information derived from the at least one piece of image data is a noise amount estimated on the basis of a statistic indicating a variation in pixel value between a pixel of interest and a peripheral pixel located around the pixel of interest.
Ou discloses wherein the information derived from the at least one piece of image data is a noise amount estimated on the basis of a statistic indicating a variation in pixel value between a pixel of interest and a peripheral pixel located around the pixel of interest (paragraph 32-35; noise estimation value N (noise amount) derived from images based on variance of pixels of interest and surround 8 pixels).
It would have been obvious to one of ordinary skill in the art at the time of the invention was made to modify the system of Baqai as taught by Ou to provide estimation of noise amount using statistical data of pixels.
The motivation to combine the references is to provide accurate image processing by estimating noise amount using both temporal and spatial information and adjusting the noise estimate accordingly (paragraph 8-11, 56).
4. Claim(s) 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 11189017 to Baqai in view of US 20140270518 to Yano further in view of US 20080152256 to Ou further in view of CN 105976308 to Lai.
Regarding claim 5, Baqai in view of Yano disclose the image processing device according to claim 4, wherein the second processing unit executes image processing that applies a filter to a pixel value of the pixel position of the synthesized raw image data (see Baqai; column 19, lines 42-55; second processing unit involves the noise reduction 176 that applies noise reduction by looking up entry in the fusion noise map for specific pixel position in the fused image and applying this entry to reduce noise of this pixel thereby performing filtering using the noise map).
However Baqai does not disclose executing image processing that applies a smoothing filter with a smoothing strength corresponding to the noise amount to a pixel value of the pixel position.
Lai discloses executing image processing (paragraph 14; GPU hardware (second processing unit) performs image processing) that applies a smoothing filter with a smoothing strength corresponding to the noise amount to a pixel value of the pixel position (paragraph 22, 33-36; image denoising in s113 uses smoothing filtering that involves formula having a K value representing strength of filtering which is associated with amount of noise to be removed for pixel at position (i, j)).
It would have been obvious to one of ordinary skill in the art at the time of the invention was made to modify the system of Baqai in view of Yano further in view Ou as taught by Lai to provide smoothing filter image processing to pixels.
The motivation to combine the references is to apply smoothing filter to synthesized image data to denoise the image data by controlling the filter parameters such that the smoothing is maximized resulting in smoothed pixel value close to the average value (paragraph 32-36).
Other Prior Art Cited
14. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
JP 2012119852 to Imaide.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BENIYAM MENBERU whose telephone number is (571) 272-7465. The examiner can normally be reached on Monday-Friday, 10:00am-6:30pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Akwasi Sarpong can be reached on (571) 270-3438. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Any inquiry of a general nature or relating to the status of this application or proceeding should be directed to the customer service office whose telephone number is (571) 272-2600. The group receptionist number for TC 2600 is (571) 272-2600.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only.
For more information about the PAIR system, see <http://pair-direct.uspto.gov/>. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Patent Examiner
Beniyam Menberu
/BENIYAM MENBERU/Primary Examiner, Art Unit 2681
03/12/2026