DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The Office Action is in response to an AMENDMENT entered 6/23/2026.
Status of Claims
Claims 1-18 are pending.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 4, 5, 7, 10, 11, 13, 14 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over US Pub. No. 20240022827 A1 to Srivastava in view of US Patent. No. 10,200,599 B1 to Baldwin.
As to claims 1 and 10, Srivastava discloses an image fusion method, comprising:
receiving a first image stream with a first frame rate (Srivastava Fig. 1-6, ¶0063, 0064, 0065, 0067, first image data is received from the first camera captured at the first frame rate, the first frame rate may be 30 FPS and second image data is received from the first camera captured at the second frame rate, the second frame rate may be 60 FPS or 90 FPS);
receiving a second image stream with a second frame rate (Srivastava Fig. 1-6, ¶0063, 0064, 0065, 0067, first image data is received from the first camera captured at the first frame rate, the first frame rate may be 30 FPS and second image data is received from the first camera captured at the second frame rate, the second frame rate may be 60 FPS or 90 FPS); and
fusing a first reused image frame of the first image stream with a set of second image frames of the second image stream respectively for outputting a set of fused image frames (Srivastava Fig. 1-6, ¶0060, 0068, output image frame 232 (F1) may be determined as F1=Fuse(S12,L1) with S12=Fuse(S1,S2) and additional output image frames may be generated by reusing intermediate image frame 222 (S12) for fusion with both adjacent long image frames, image frame S1 meets the first reused image frame and S2 meets the another first reused image frame).
Srivastava does not expressly disclose wherein at least one image frame of the first image stream has a timing offset from at least one corresponding image frame of the second image stream.
Baldwin discloses wherein at least one image frame of the first image stream has a timing offset from at least one corresponding image frame of the second image stream (Baldwin col. 10, lines, 1-44, col. 13, lines 9-19, 24-63, first image captured by the first camera has an offset in time with the second image frame captured by the second camera).
It would have been obvious to a person of ordinary skilled in the art before the effective filing date of the claimed invention to modify Srivastava by wherein at least one image frame of the first image stream has a timing offset from at least one corresponding image frame of the second image stream as disclosed by Baldwin. The suggestion/motivation would have been in order to capture a plurality of images using a plurality of cameras at different times using different capture settings thereby enhancing the user experience and/or image capture quality.
As to claims 2 and 11, Srivastava discloses wherein the first frame rate is different from the second frame rate (Srivastava Fig. 1-6, ¶0063, 0064, 0065, 0067, first image data is received from the first camera captured at the first frame rate, the first frame rate may be 30 FPS and second image data is received from the first camera captured at the second frame rate, the second frame rate may be 60 FPS or 90 FPS).
As to claims 4 and 13, Srivastava discloses wherein at least one image frame of the first image stream comprises data from a plurality of exposure settings (Srivastava ¶0032, the different exposure durations may be obtained by controlling a frame rate of the camera to switch between a first frame rate for capturing one or more long image frames and a second frame rate for capturing two or more short image frames).
As to claims 5 and 14, Srivastava discloses wherein the plurality of exposure settings comprises a first exposure setting and a second exposure setting, and wherein the second image stream comprises data from the second exposure setting (Srivastava ¶0032, capturing long exposure frames and fusing one or more long exposure frames with multiple short exposure frames captured at different exposure durations and the different exposure durations may be obtained by controlling a frame rate of the camera to switch between a first frame rate for capturing one or more long image frames and a second frame rate for capturing two or more short image frames).
As to claims 7 and 16, Srivastava discloses wherein fusing the first reused image frame comprises: outputting the first reused image frame when the first reused image frame comprises merged exposure data (Srivastava Fig. 1-6, ¶0033, 0077, 0077, a first short exposure frame 204 (S1) is fused with a second short exposure frame (S2) to determine an intermediate image frame 222 (S12), Long exposure frame 202 (L1) is then fused with S12 to determine a first output frame 232 (F1). When determining output image frame 232, the intermediate image frame S12 is used as the anchor frame. Long exposure frame 208 (L2) is fused with intermediate image frame 222 (S12) to determine a second output image frame (F2). When determining output image frame 234, the long exposure frame 208 (L2) is used as the anchor frame).
Claims 3 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over US Pub. No. 20240022827 A1 to Srivastava in view of US Patent. No. 10,200,599 B1 to Baldwin and in further view of US Pub. No. 20180308523 A1 to Silvestri.
As to claims 3 and 12, Baldwin discloses wherein: the first image stream has the timing offset from the second image stream (Baldwin col. 13, lines 24-63, first image captured by the first camera has an offset in time with the second image frame captured by the second camera); and
Srivastava and Baldwin do not expressly disclose the timing offset is one frame timecode.
Silvestri discloses the timing offset is one frame timecode (Silvestri ¶0062, a timing reference identified in the timecode packet of the frame and a frame identifier for retrieving the frame from storage server (e.g., byte offset, file offset, timecode offset, frame number, etc.)
It would have been obvious to a person of ordinary skilled in the art before the effective filing date of the claimed invention to modify Srivastava and Baldwin by the timing offset is one frame timecode as disclosed by Silvestri. The suggestion/motivation would have been in order to provide a timecode packet of the frame and a frame identifier having the timecode offset that allows the system to properly identify the timing offset.
Claims 6 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over US Pub. No. 20240022827 A1 to Srivastava in view of US Patent. No. 10,200,599 B1 to Baldwin and in further view of US Pub. No. 20230143443 A1 to Cotoros.
As to claims 6 and 15, Srivastava and Baldwin do not expressly disclose wherein the first frame rate and the second frame rate are each lower than an output frame rate of the set of fused image frames.
Cotoros discloses wherein the first frame rate and the second frame rate are each lower than an output frame rate of the set of fused image frames (Cotoros ¶0054, 0057, 0059-0060, the output fused image frames having a higher/greater frame rate than the input images with lower frame rates).
It would have been obvious to a person of ordinary skilled in the art before the effective filing date of the claimed invention to modify Srivastava and Baldwin by wherein the first frame rate and the second frame rate are each lower than an output frame rate of the set of fused image frames as disclosed by Cotoros. The suggestion/motivation would have been in order to provide a higher frame rate output increasing the quality of the video thereby enhancing the user’s experience.
Claims 8 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over US Pub. No. 20240022827 A1 to Srivastava in view of US Patent. No. 10,200,599 B1 to Baldwin and in further view of US Pub. No. 20210266445 A1 to Miyake.
As to claims 8 and 17, Srivastava and Baldwin do not expressly disclose performing exposure fusion between the first reused image frame and an image frame in the set of second image frames when the first reused image frame does not comprise merged exposure data.
Miyake discloses performing exposure fusion between the first reused image frame and an image frame in the set of second image frames when the first reused image frame does not comprise merged exposure data (Miyake ¶0077, non-combining exposure to set the main exposure image as an exposure image (the third exposure image) without combining the pre-exposure image and the main exposure image).
It would have been obvious to a person of ordinary skilled in the art before the effective filing date of the claimed invention to modify Srivastava and Baldwin by performing exposure fusion between the first reused image frame and an image frame in the set of second image frames when the first reused image frame does not comprise merged exposure data as disclosed by Miyake. The suggestion/motivation would have been in order to provide fusion of frames without having exposure to provide the non-combining exposure fusion providing various detail enhancement to the image.
Claims 9 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over US Pub. No. 20240022827 A1 to Srivastava in view of US Patent. No. 10,200,599 B1 to Baldwin and in further view of US Pub. No. 20170208231 A1 to Lewkow.
As to claims 9 and 18, Srivastava and Baldwin do not expressly disclose wherein at least one image frame of the first image stream comprises a different image size than at least one image frame of the second image stream.
Lewkow discloses wherein at least one image frame of the first image stream comprises a different image size than at least one image frame of the second image stream (Lewkow ¶0020, 0029, different image/frame sizes).
It would have been obvious to a person of ordinary skilled in the art before the effective filing date of the claimed invention to modify Srivastava and Baldwin by wherein at least one image frame of the first image stream comprises a different image size than at least one image frame of the second image stream as disclosed by Lewkow. The suggestion/motivation would have been in order to provide images/frames in different sizes that improve the image/frame quality with wider range of details.
Response to Arguments
Applicant's arguments filed 6/23/2026 related to claims 1-18 have been fully considered but they are not persuasive.
In reference to Applicant's arguments:
Applicant respectfully submits that the rejection of independent claim 1 is improper.
A. Srivastava fails to disclose receiving a first image stream and a second image stream each having a respective frame rate
Independent claim 1 recites, in relevant part, "receiving a first image stream with a first frame rate" and "receiving a second image stream with a second frame rate." Properly construed, these limitations require two image streams, each ongoing at its respective frame rate, such that frames of the first stream and frames of the second stream may be paired and fused as further recited in the claim.
Srivastava does not disclose this configuration. Srivastava is directed to a single camera whose image sensor is reconfigured back and forth between two frame-rate modes. Srivastava expressly teaches that "the different exposure durations may be obtained by controlling a frame rate of the camera to switch between a first frame rate for capturing one or more long image frames and a second frame rate for capturing two or more short image frames" (Srivastava, 1 0032). The method of Srivastava correspondingly configures the camera for the first frame rate, receives first image data, and then reconfigures the same camera for the second frame rate before receiving second image data (Srivastava, TT 0063-0067; Fig. 3). Because the single sensor is set to only one frame rate at any given time, Srivastava captures interleaved batches of data from one sensor-not two image streams, each ongoing at its own frame rate, as recited in claim 1.
The Office action maps Srivastava's long-exposure data to the claimed first image stream and Srivastava's short-exposure data to the claimed second image stream. However, that mapping disregards that the long-exposure and short-exposure data in Srivastava originate from the same sensor operating at different times under successive configurations, rather than from two streams concurrently received at respective frame rates. Srivastava therefore does not teach "receiving a first image stream with a first frame rate" and "receiving a second image stream with a second frame rate" as those limitations are recited in claim 1.
B. Baldwin does not teach the claimed "timing offset" in the context of image fusion
Claim 1 further recites "wherein at least one image frame of the first image stream has a timing offset from at least one corresponding image frame of the second image stream." The Office action relies on Baldwin to supply this "timing offset" limitation. Baldwin, however, is directed to an entirely different problem and teaches a fundamentally different use of a temporal offset.
Baldwin concerns "image capture setting determination in devices having access to multiple cameras." In Baldwin, a second camera captures a second image frame after a first camera captures a first image frame, and the second image frame is used to determine an image capture setting, such as an automatic exposure control, automatic white balance, or automatic focus setting, for the first camera (Baldwin, col. 10:1-44; col. 13:24-63. The temporal offset in Baldwin exists so that the setting estimate is closer in time to the upcoming capture, thereby improving the responsiveness of the setting determination. Critically, Baldwin does not fuse image data from the two cameras. The second camera's frame in Baldwin is used only to compute capture settings for the first camera, not to be combined with the first camera's frame to produce a fused output image frame.
By contrast, the claimed "timing offset" is a characteristic of the relationship between two image streams that are fused together: at least one frame of the first image stream is offset in time from a corresponding frame of the second image stream, and the reused frame of the first image stream is fused with a set of frames of the second image stream to output fused image frames. Baldwin's offset, directed to camera-setting estimation rather than image fusion, does not teach this limitation.
C. Combination does not teach fusing a reused frame of the first image stream with a set of frames of the second image stream.
Claim 1 also recites "fusing a first reused image frame of the first image stream with a set of second image frames of the second image stream respectively for outputting a set of fused image frames." This limitation requires that a single frame of the first image stream be reused that is, fused respectively with each of multiple frames of the second image stream to output a corresponding set of fused image frames.
The Office action maps Srivastava's intermediate frame S12 to the claimed "first reused age frame of the first image stream." That mapping is internally inconsistent. As Srivastava describes, S12 is not an as-received frame of any image stream; rather, it is a derived intermediate produced by fusing short-exposure frames S1 and S2, i.e., S12 = Fuse(S1, S2) (Srivastava, 0058). The short-exposure frames S1 and S2, however, correspond to the data that the Office action elsewhere maps to the claimed second image stream. The same data cannot simultaneously serve as a derived product of the second image stream and as a "reused image frame of the first image stream."
Furthermore, the reuse pattern in Srivastava proceeds in the opposite direction from that recited in claim 1. In Srivastava, the short-frame intermediate 512 is fused with each of the adjacent long-exposure frames L1 and L2, e.g., F1 = Fuse(S 12, L1) and F2 = Fuse(L2, S12) (Srivastava, 0058, 0060). Accordingly, a short-frame composite is reused across multiple long frames. Claim 1 recites the reverse: a frame of the first image stream is reused and fused respectively with a set of frames of the second image stream.
The cited combination thus does not teach this limitation as claimed.
Examiners Response:
The examiner respectfully disagrees. Srivastava does disclose receiving a first image stream with a first frame rate and receiving a second image stream with a second frame rate.
In particular, Srivastava discloses first image data is received from the first camera captured at the first frame rate, the first frame rate may be 30 FPS and second image data is received from the first camera captured at the second frame rate, the second frame rate may be 60 FPS or 90 FPS (Srivastava Fig. 1-6, ¶0063, 0064, 0065, 0067).
In response to applicant's argument that the references fail to show certain features of the invention, it is noted that the features upon which applicant relies (i.e., However, that mapping disregards that the long-exposure and short-exposure data in Srivastava originate from the same sensor operating at different times under successive configurations, rather than from two streams concurrently received at respective frame rates) are not recited in the rejected claim(s). Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993).
Baldwin does disclose wherein at least one image frame of the first image stream has a timing offset from at least one corresponding image frame of the second image stream. In particular, Baldwin discloses the first image captured by the first camera has an offset in time with the second image frame captured by the second camera (Baldwin col. 10, lines, 1-44, col. 13, lines 9-19, 24-63). Furthermore, it was not Baldwin to disclose the fusion of the image data. Baldwin was cited to disclose the timing offset.
In response to applicant's argument that the references fail to show certain features of the invention, it is noted that the features upon which applicant relies (i.e., “timing offset” is a characteristic of the relationship between two image streams that are fused together) are not recited in the rejected claim(s). Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993).
Srivastava does disclose fusing a first reused image frame of the first image stream with a set of second image frames of the second image stream respectively for outputting a set of fused image frames. In particular Srivastava discloses output image frame 232 (F1) may be determined as F1=Fuse(S12,L1) with S12=Fuse(S1,S2) and additional output image frames may be generated by reusing intermediate image frame 222 (S12) for fusion with both adjacent long image frames (Srivastava Fig. 1-6, ¶0060, 0068). Reusing intermediate image frame 222 (S12) is derived by S1 and S2. F1 is determined by fusing S12 and L1. F2 is derived by fusing L2 and S12. In other words, reusing intermediate image frame 222 (S12) which is derived by S1 and S2 is fused with L1 and L2 for outputting the fused image frames of F1 and F2. Therefore, applicant’s arguments are not persuasive and the examiner respectfully disagrees.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Claims 1-18 have been rejected.
Correspondence Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KYU CHAE whose telephone number is (571)270-5696. The examiner can normally be reached on 8:00am -4:30pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, NASSER MOAZZAMI can be reached on 571-272-4195. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KYU CHAE/
Primary Examiner, Art Unit 2426
Prosecution Insights