DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 5, 9-11, 13, and 17-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tico et al. (US 2019/0370948) in view of Shoa Hassani Lashdan et al. (US 2019/0141287).
Regarding claim 1, Tico teaches/suggests: A method, comprising:
receiving, by a processor (Tico Fig. 4: processor 405), a first image frame depicting a scene at a first resolution (Tico [0021] “the process 100 may begin by obtaining an incoming image stream”);
determining, by the processor, a fourth image frame by fusing a second image frame depicting the scene with a third image frame retrieved from a memory (Tico [0032] “the process may select two or more images from the incoming image stream for fusion (Step 106), including which image should serve as the reference image for the fusion operation” [The reference image meets the third image frame.] [0060] “display 410 may display a video stream as it is captured while processor 405 and/or graphics hardware 420 and/or image capture circuitry contemporaneously generate and store the video stream in memory 460 and/or storage 465”);
Tico does not teach/suggest:
wherein each of the second image frame, the third image frame and the fourth image frame is at a second resolution less than the first resolution;
upscaling, by the processor, the fourth image frame from the second resolution to the first resolution; and
determining, by the processor, an output image frame by fusing the first image frame and the fourth image frame after upscaling the fourth image frame.
Shoa Hassani Lashdan, however, teaches/suggests a second resolution less than the first resolution (Shoa Hassani Lashdan [0037] “video source 102 generates a first series of frames 200 that includes high-resolution frames and low-resolution frames”). Shoa Hassani Lashdan further discloses in [0003]: “High frame rate in video is desirable in various applications … However, high frame rate bitstreams of encoded video data may be very large, especially if the frames are high resolution.” Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to modify the image stream of Tico to be that of Shoa Hassani Lashdan for a high frame rate that’s memory efficient. As such, Tico as modified by Shoa Hassani Lashdan teaches/suggests:
wherein each of the second image frame, the third image frame and the fourth image frame is at a second resolution less than the first resolution (Tico [0032] “the process may select two or more images from the incoming image stream for fusion (Step 106), including which image should serve as the reference image for the fusion operation” Shoa Hassani Lashdan [0037] “video source 102 generates a first series of frames 200 that includes high-resolution frames and low-resolution frames”);
upscaling, by the processor, the fourth image frame from the second resolution to the first resolution (Shoa Hassani Lashdan [0038] “A conversion process produces a second series of frames 202 ... each frame in the second series of frames 202 is a high-resolution frame”); and
determining, by the processor, an output image frame by fusing the first image frame and the fourth image frame after upscaling the fourth image frame (Tico [0032] “the process may select two or more images from the incoming image stream for fusion (Step 106), including which image should serve as the reference image for the fusion operation” Shoa Hassani Lashdan [0038] “A conversion process produces a second series of frames 202 ... each frame in the second series of frames 202 is a high-resolution frame” [In view of Tico and Shoa Hassani Lashdan, the fusion of the high-resolution frame and the fused low-resolution frames meets the determining.]).
Regarding claim 2, Tico as modified by Shoa Hassani Lashdan teaches/suggests: The method of claim 1, further comprising determining the second image frame based on the first image frame (Shoa Hassani Lashdan [0041] “rate conversion unit 106 generates a down-sampled frame by down-sampling the preliminary high-resolution frame”). The same rationale to combine as set forth in the rejection of claim 1 above is incorporated herein.
Regarding claim 3, Tico as modified by Shoa Hassani Lashdan teaches/suggests: The method of claim 1, wherein the second image frame is determined by downscaling the first image frame from the first resolution to the second resolution (Shoa Hassani Lashdan [0041] “rate conversion unit 106 generates a down-sampled frame by down-sampling the preliminary high-resolution frame”). The same rationale to combine as set forth in the rejection of claim 1 above is incorporated herein.
Regarding claim 5, Tico as modified by Shoa Hassani Lashdan teaches/suggests: The method of claim 1, wherein:
the processor is an image signal processor (Tico [0023] “the encoding operation may be performed directly by an Image Signal Processor (ISP)”),
the memory is an external memory, and the second and third image frames are fused based on data retrieved from the external memory (Tico [0032] “the process may select two or more images from the incoming image stream for fusion (Step 106), including which image should serve as the reference image for the fusion operation” [0060] “display 410 may display a video stream as it is captured while processor 405 and/or graphics hardware 420 and/or image capture circuitry contemporaneously generate and store the video stream in memory 460 and/or storage 465”).
Claims 9-11 and 13 recite limitation(s) similar in scope to those of claims 1-3 and 5, respectively, and are rejected for the same reason(s). Tico as modified by Shoa Hassani Lashdan further teaches/suggests a memory storing processor-readable code; and at least one processor coupled to the memory (Tico Fig. 4: processor 405 and memory 460).
Claims 17 and 18 recite limitation(s) similar in scope to those of claims 1 and 3, respectively, and are rejected for the same reason(s). Tico as modified by Shoa Hassani Lashdan further teaches/suggests an image sensor; a memory storing processor-readable code; and at least one processor coupled to the memory and to the image sensor (Tico Fig. 4: processor 405, memory 460, and image capturing device 450).
Claim(s) 6-8, 14-16, and 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tico et al. (US 2019/0370948) in view of Shoa Hassani Lashdan et al. (US 2019/0141287) as applied to claims 1, 9, and 17 above, and further in view of Numata et al. (US 2012/0169894).
Regarding claim 6, Tico as modified by Shoa Hassani Lashdan teaches/suggests: The method of claim 1, wherein the plurality of image frames includes the first image frame and the second image frame (Shoa Hassani Lashdan [0037] “video source 102 generates a first series of frames 200 that includes high-resolution frames and low-resolution frames” [0032] “Storage media 104 may store the frames generated by video source 102”). The same rationale to combine as set forth in the rejection of claim 1 above is incorporated herein.
Tico and Shoa Hassani Lashdan are silent regarding receiving a plurality of image frames from a buffer. Numata, however, teaches/suggests receiving a plurality of image frames from a buffer (Numata Fig. 5: base frame buffer section 1611). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to modify the image stream of Tico as modified by Shoa Hassani Lashdan to be stored in the buffer of Numata for processing.
Regarding claim 7, Tico as modified by Shoa Hassani Lashdan and Numata teaches/suggests: The method of claim 6, wherein:
the buffer includes a first sequence of image frames that alternates between a second sequence of image frames at the second resolution and an image frame at the first resolution, the plurality of image frames includes a single image frame at the first resolution in the first sequence and a plurality of image frames at the second resolution in the first sequence, the single image frame at the first resolution is the first image frame, and the plurality of image frames at the second resolution includes the second image frame (Shoa Hassani Lashdan Fig. 2: high-resolution frame 204 and low-resolution frames 208; Numata Fig. 5: base frame buffer section 1611).
The same rationales to combine as set forth in the rejection of claims 1 and 6 above are incorporated herein.
Regarding claim 8, Tico as modified by Shoa Hassani Lashdan and Numata teaches/suggests: The method of claim 6, wherein the plurality of image frames includes a plurality of second image frames (Shoa Hassani Lashdan [0037] “video source 102 generates a first series of frames 200 that includes high-resolution frames and low-resolution frames”). The low-resolution frames meet the plurality of second image frames. The same rationale to combine as set forth in the rejection of claim 1 above is incorporated herein.
Tico as modified by Shoa Hassani Lashdan and Numata does not teach/suggest:
determining a fourth image frame for each second image frame of the plurality of second image frames;
upscaling each fourth image frame from the second resolution to the first resolution; and
determining the output image frame by fusing the first image frame with all of the fourth image frames after upscaling each of the fourth image frames.
Numata, in view of Tico, teaches/suggests:
determining a fourth image frame for each second image frame of the plurality of second image frames (Tico [0032] “the process may select two or more images from the incoming image stream for fusion (Step 106), including which image should serve as the reference image for the fusion operation” Numata [0098] “the reduced frame NR image stored in the memory 103 may be used as a reduced frame reference image” [In other words, the reference image of Numata is different from the incoming image stream of Tico.]);
Before the effective filing date of the claimed invention, the substitution of one known element (the reference image of Numata) for another (the reference image of Tico) would have been obvious to one of ordinary skill in the art because such substitutions would have yielded predictable results, namely for the fusion. As such, Tico as modified by Shoa Hassani Lashdan and Numata teaches/suggests:
upscaling each fourth image frame from the second resolution to the first resolution (Shoa Hassani Lashdan [0038] “A conversion process produces a second series of frames 202 ... each frame in the second series of frames 202 is a high-resolution frame”); and
determining the output image frame by fusing the first image frame with all of the fourth image frames after upscaling each of the fourth image frames (Tico [0032] “the process may select two or more images from the incoming image stream for fusion (Step 106), including which image should serve as the reference image for the fusion operation” Shoa Hassani Lashdan [0038] “A conversion process produces a second series of frames 202 ... each frame in the second series of frames 202 is a high-resolution frame” Numata [0098] “the reduced frame NR image stored in the memory 103 may be used as a reduced frame reference image”).
Claims 14-16 recite limitation(s) similar in scope to those of claims 6-8, respectively, and are rejected for the same reason(s).
Claims 19 and 20 recite limitation(s) similar in scope to those of claims 6 and 7, respectively, and are rejected for the same reason(s).
Allowable Subject Matter
Claims 4 and 12 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: The limitation “wherein the fourth image frame is used as the third image frame in the determination of the subsequent output image frame,” taken as a whole, renders the claims patentably distinct over the prior art.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
US 2003/0090571 – intersperse high and low resolution images
US 2011/0019082 – temporal noise reduction
US 2022/0138964 – zero shutter lag buffer
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANH-TUAN V NGUYEN whose telephone number is 571-270-7513. The examiner can normally be reached on M-F 9AM-5PM ET. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, JASON CHAN can be reached on 571-272-3022. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANH-TUAN V NGUYEN/
Primary Examiner, Art Unit 2619