DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed 20 July 2026 have been fully considered but they are not persuasive. Applicant amends the claims to recite “a temporal association” rather than a temporal alignment and argues that the claimed “temporal association” and “synchronize the plurality of events with the image frame, according to the temporal association” is supported by paragraphs 0029, 0049, 0068 and 0074 of the specification. The examiner disagrees.
The paragraphs are reproduced below:
[0029] Separately, as indicated in step 202b, events are also accumulated during a longer window, wdw, including the short-exposure time (SET) of the image frame in an event frame ef3. The length of the window, wdw, is proportional and may, although not necessarily, be the same as a desired long-exposure time (LET) which the method is to simulate e.g., half the frame interval. The window is typically no longer than the frame interval - as this would mean that certain events could affect the blurring of more than one frame. The window is typically but not necessarily centered around the SET of the image frame, as this means that events occurring before and after the SET can be given equal weight. In any case, knowledge of the desired exposure time employed by a cinematographic camera whose output it is desired to simulate is useful in determining the width of the window, wdw.
[0049] In any case, it should be noted that where the relative timing of event frames ef1 ... ef3 and their respective images frames 12a ... 12c is known, information from the event frames can be suitably weighted as required when determining the blur kernel for the pixels of each image frame to be blurred.
[0068] Referring now to Figure 6, in order to ensure that each of the cameras 100, 102 and a cinematographic camera 68 share a common field of view, a 3-way beam splitter 62, similar to those used 3CCD cameras, can be employed. In a preferred embodiment, the incident light is split into three paths, one leading to the event camera 102, whereas the second frame camera 100 records image frames with short exposure times and the third cinematographic camera 68 records image frames with a long exposure time. The camera 68 may have a neutral density filter at the front of its sensor to allow for longer exposure times while keeping the gain and other parameters the same as for the frame camera 100. The trichroic prism is designed to work with a single lens, so eliminating parallax errors and matching the optical axes and geometry of the cameras in the system. The cameras are synchronised in the time domain with timestamps and exposure times recorded. Ideally, short exposure frames captured by the camera 100 are exposed in the middle of the long exposure frame window to minimise the distance of the target blur.
[0071] Nonetheless, in some embodiments, the acquisition of the video frames from cameras 100 and 68 and the events from the camera 102 may be realised using side-by-side cameras. Such an approach would require additional processing to perform geometrical registration between image frames and event frames, in order to compensate for alignment and parallax errors.
The claims as written define:
a processor configured to communicate with the frame-based camera and the event-based camera, and further configured to:
perform a temporal association based on:
the timestamp for each event of the plurality of events; and
at least one time signal from the frame-based camera, wherein the at least one time signal corresponds to the image frame; and
synchronize the plurality of events with the image frame, according to the temporal association to generate a synchronized frame.
As claimed, the processor is configured to perform at least two functions, a temporal association function and a synchronize function.
The temporal association is defined as being based on timestamps for each of the plurality of events and at least one time signal from the frame based camera, wherein the at least one time signal corresponds to the image frame.
Applicant argues that the temporal alignment/association is described in reference to the fact that “the cameras are synchronised in the time domain with timestamps [i.e., “timestamp for each event”] and exposure times [e.g., “the at least one time signal …[that] corresponds to the image frame”] recorded.” (Specification, para. [0049]) (see p. 2 of the response).
Paragraph 0049 states weighting event frames when determining a blur kernel for the pixels when the relative timing of event frames ef1 … ef3 and their respective image frames 12a … 12c is known.
This paragraph merely states that the timing between frames may be known, and when the relative timing is known the event frames can be suitable weighted. However, the paragraph does not state that a temporal association is performed by the processor based on timestamps of the event frames and a time signal from the frame-based camera which corresponds to the image frame.
As claimed, the temporal association is an active step which the processor must be configured to perform. Merely knowing a relative timing between event and image frames does not properly disclose the configuration for performing a temporal association, much less a temporal association based on timestamps of event frames and at least one time signal received from the frame camera as required by the claim.
Applicant further references paragraph 0029:
Applicant respectfully submits that temporal alignment is disclosed through the process of accumulating events within a window centered around the frame camera's exposure time, while the synchronizing is referenced as “events [being] also accumulated during a longer window, wdw, including the short-exposure time (SET) of the image frame in an event frame ef3,” where the former “is typically but not necessarily centered around the SET of the image frame, as this means that events occurring before and after the SET can be given equal weight. (Specification, para. [0029]). This is further supported by the first quote (para. [0049]) of the Office action, which describes that “relative timing of event frames ef1... ef3 and their respective images.” Applicant submits that specific associations between event frames and image frames are an example of an alignment, while the above also establishes the temporal basis.
While paragraph 0029 and 0049 as reference by applicant may show that a temporal association between the frames exists, the paragraphs do not show that the processor itself is configured to perform a temporal association based on timestamps and at least one time signal as required by the claim. The fact that the frames were captured within a time window does not necessarily imply that the processor performs a temporal association (see e.g. Figure 1 of the specification and note no involvement by the processor). In fact, it would more likely be the case that the frame-based camera and the event-based camera cooperate to generate frames which are temporarily associated by virtue of their common capture times which would negate the need for any active temporal association to be performed by the processor.
Applicant further argues “regardless of whether the exact phrasing is present, the concept is described in reference to the frame camera and array camera output”. While the examiner agrees that no in haec verba requirement exists and that a temporal association is described in reference to the frame camera and array camera output, this nonetheless does properly support a processor which is configured to perform a temporal association based on timestamps and at least one time signal as required by the claims.
On page 3, applicant argues:
Second, Applicant respectfully submits that the step of “synchronizing the plurality of events with the image frame, according to the temporal alignment, to generate a synchronized frame” is described in reference to the fact that “[e]mbodiments use an image frame acquired with a short exposure time in combination with events from an event camera, in order to simulate an image frame captured with a longer exposure time.” (Specification, Summary). This “combination” of event data with the image frame constitutes synchronization according to the temporal alignment discussed above. The output of this synchronization is described as follows: “[i]n step 218, the blur kernel is then applied to the image frame pixels of the mask to produce a blurred image.” (Specification, para. [0044]). Applicant submits that this “blurred image” is the synchronized output generated by combining temporally-aligned event information with the image frame. This interpretation is further supported by the disclosure that “where the relative timing of event frames ef1 ... ef3 and their respective images frame[ ] 12a ... 12c is known, information from the event frames can be suitably weighted as required when determining the blur kernel for the pixels of each image frame to be blurred.” (Specification, para. [0048]). Applicant submits that the association between event frames and their “respective” image frames, combined with the weighting based on “relative timing,” constitutes synchronization according to temporal association. As such, Applicant respectfully submits that, regardless of whether the exact phrasing “synchronized frame” is present, the concept of synchronizing events with an image frame to generate an output frame is described in the specification.
Applicant does not describe how this process of processing “the image frame acquired with a short exposure time in combination with events from an event camera, in order to simulate an image frame captured with a longer exposure time” is equivalent to the “synchronize the plurality of events with the image frame, according to the temporal association to generate a synchronized frame” function of the processor.
As typically defined, synchronize is defined as “to happen at the same time” or “to make synchronous in operation” (Merriam-Webster definition of “synchronize”) where synchronous is defined as “happening, existing, or arising at precise the same time” (Merriam-Webster definition of “synchronous”).
The forming of a simulated long exposure image does not change the timing of the plurality of event frames and the image frame, according to the temporal association in a way which would “synchronize” these frames as required by the claims. Additionally, weighting of the event frames does not alter their timing in a way which would perform a synchronization. The weighting would merely change the contribution of each frame to the final combined frame. Furthermore, the simulated long exposure image is not disclosed as being a “synchronized frame” in the specification.
While applicant is entitled to be their own lexicographer, “[t]o act as their own lexicographer, the applicant must clearly set forth a special definition of a claim term in the specification that differs from the plain and ordinary meaning it would otherwise possess. CCS Fitness, Inc. v. Brunswick Corp., 288 F.3d 1359, 1366, 62 USPQ2d 1658, 1662 (Fed. Cir. 2002).” (MPEP 2111.01(IV)).
Additionally, “it is important to note that any special meaning assigned to a term “must be sufficiently clear in the specification that any departure from common usage would be so understood by a person of experience in the field of the invention.” Multiform Desiccants Inc. v. Medzam Ltd., 133 F.3d 1473, 1477, 45 USPQ2d 1429, 1432 (Fed. Cir. 1998)” and “[i]n some cases, the meaning of a particular claim term may be defined by implication, that is, according to the usage of the term in the context in the specification. See Phillips v. AWH Corp., 415 F.3d 1303, 1320-21, 75 USPQ2d 1321, 1332 (Fed. Cir. 2005) (en banc); Vitronics Corp. v. Conceptronic Inc., 90 F.3d 1576, 1583, 39 USPQ2d 1573, 1577 (Fed. Cir. 1996). But where the specification is ambiguous as to whether the inventor used claim terms inconsistent with their ordinary meaning, the ordinary meaning will apply. Merck & Co. v. Teva Pharms. USA, Inc., 395 F.3d 1364, 1370, 73 USPQ2d 1641, 1646 (Fed. Cir. 2005)” (MPEP 2111.01(IV)(A)).
Applicant has clearly avoided the use of “synchronize” or “synchronization” in drafting the specification and thus does not clearly redefine these terms to have a meaning consistent with generating a simulated long exposure image the ordinary meanings will apply.
For these reasons, applicant’s arguments are not considered persuasive and the 35 USC 112(a) rejections are maintained below.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 13-32 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.[claims 13-32]
Claims 13-24 as amended on 31 March 2026 recite:
A system for performing image capture, the system comprising:
…
and a processor … configured to:
perform a temporal association based on: the timestamp for each event of the plurality of events; and at least one time signal from the frame-based camera, wherein the at least one time signal corresponds to the image frame; and
synchronize the plurality of events with the image frame, according to the temporal association, to generate a synchronized frame.
Claims 25-32 recite similar limitations in a CRM form.
As claimed, claims 13-32 recite performing a temporal association and a synchronization, however the specification does not support such a combination of features.
At best, the specification describes:
[0049]In any case, it should be noted that where the relative timing of event frames ef1 ... ef3 and their respective images frames 12a ... 12c is known, information from the event frames can be suitably weighted as required when determining the blur kernel for the pixels of each image frame to be blurred.
[0068]Referring now to Figure 6, in order to ensure that each of the cameras 100, 102 and a cinematographic camera 68 share a common field of view, a 3-way beam splitter 62, similar to those used 3CCD cameras, can be employed. In a preferred embodiment, the incident light is split into three paths, one leading to the event camera 102, whereas the second frame camera 100 records image frames with short exposure times and the third cinematographic camera 68 records image frames with a long exposure time. The camera 68 may have a neutral density filter at the front of its sensor to allow for longer exposure times while keeping the gain and other parameters the same as for the frame camera 100. The trichroic prism is designed to work with a single lens, so eliminating parallax errors and matching the optical axes and geometry of the cameras in the system. The cameras are synchronised in the time domain with timestamps and exposure times recorded. Ideally, short exposure frames captured by the camera 100 are exposed in the middle of the long exposure frame window to minimise the distance of the target blur.
[0071]Nonetheless, in some embodiments, the acquisition of the video frames from cameras 100 and 68 and the events from the camera 102 may be realised using side-by-side cameras. Such an approach would require additional processing to perform geometrical registration between image frames and event frames, in order to compensate for alignment and parallax errors.
Paragraph 0049 describes weighting the event frames when determining the blur kernel when relative timings of event frames and respective image frames are known.
Paragraph 0068 describes synchronizing the event camera and the frame camera, but does not describe the processor synchronizing the plurality of events with the image frame based on a temporal association to generate a synchronized frame as recited in the claims.
Paragraph 0071 describes performing geometrical registration to compensate for alignment and parallax errors, but does not disclose a temporal alignment based on timestamps and a time signal.
However, none of these paragraphs disclose temporal association and synchronization of events with an image frame according to the temporal association to generate a synchronized frame as amended. Additionally, nowhere else in the specification is any sort of alignment or synchronization disclosed.
The 31 March 2026 amendment states “Support for the amendments can be found through the specification” without citing any specific figures or paragraphs. However, the closest paragraphs as discussed above do not appear to fully support the amendments.
Therefore, the claims contain subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Subject Matter Not Taught by the Prior Art
Claims 13-32 contain subject matter not taught by the prior art, but cannot be considered allowable due the 35 USC 112(a) new matter rejections above.[claims 13-32]
Regarding claims 13-24, the prior art does not teach or reasonably suggest a system for performing image capture, the system comprising: a frame-based camera, wherein the frame-based camera is configured to generate an image frame depicting a first field of view at an exposure time; an event-based camera, wherein the event-based camera is configured to generate event information for a plurality of events at the exposure time; wherein the event information is generated in a second field of view; wherein the second field of view overlaps, at least in part, with the first field of view; and wherein the event information for each event of the plurality of events comprises: a coordinate; a polarity for a change of detected light intensity incident at the coordinate; and a timestamp for the event; a beam-splitter configured to split incident light to the frame-based camera and the event-based camera; and a processor configured to communicate with the frame-based camera and the event-based camera, and further configured to: perform a temporal association based on: the timestamp for each event of the plurality of events; and at least one time signal from the frame-based camera, wherein the at least one time signal corresponds to the image frame; and synchronize the plurality of events with the image frame, according to the temporal association, to generate a synchronized frame; and a display configured to display the synchronized frame.
Regarding claims 25-32 the prior art does not teach or reasonably suggest a non-transitory computer-readable medium, comprising instructions that, when executed on at least one processor, are configured to perform a process for: generating an image frame depicting a first field of view with a frame-based camera; generating, using an event-based camera, event information for a plurality of events, in a second field of view that overlaps, at least in part, with the first field of view, wherein the image frame and the event information for the plurality of events are generated at a shared exposure time by using a beam-splitter to split incident light to the frame-based camera and the event-based camera; and wherein the event information for each event of the plurality of events comprises: a coordinate; a polarity for a change of detected light intensity incident at the coordinate; and a timestamp for the event; performing a temporal association based on: the timestamp for each event of the plurality of events; and at least one time signal from the frame-based camera, wherein the at least one time signal corresponds to the image frame; synchronizing the plurality of events with the image frame, according to the temporal association, to generate a synchronized frame; and displaying the synchronized frame on a display.
Krishnaswamy discloses a system for generating motion blur (Figure 1; c. 4, ll. 7-24; c. 6, ll. 4-16), the system comprising: a frame camera configured to provide an image frame at an exposure time covering a field of view of the frame camera (c. 4, ll. 7-11; camera capturing a series of images); and a processor (c. 6, ll. 17-33) configured to: determine one or more areas of movement within the field of view (Figure 2, Step 204; c. 7, ll. 15-39); determine a direction of movement for the one or more areas of movement (c. 7, ll. 15-54; determining magnitude and direction based on optical flow) and apply blur in one or more areas of the image frame corresponding to the one or more determined areas of movement in accordance with at least the direction of movement for each of the one or more areas of movement to produce a blurred image (Figure 2, Step 206; c. 7, ll. 40-54; blurring image according to optical flow).
Krishnaswamy does not disclose an event camera having a field of view overlapping the field of view of the frame camera and configured to provide event information for a plurality of events, each event indicating an x,y location within said field of view, a polarity for a change of detected light intensity incident at said x,y location and an event cycle at which said event occurred; an accumulator for accumulating event information from a plurality of events occurring within a window around the exposure time in a plurality of event frames, the event frames having a temporal relationship with one another; wherein the movement and direction are determined from the events in at least a first of said plurality of event frames of the event camera. However, it is noted that Krishnaswamy discloses that various techniques for determining optical flow may be utilized (e.g. c. 7, ll. 22-28).
Daniilidis discloses a system for estimating optical flow including an event camera having a field of view and configured to provide event information for a plurality of, each event indicating an x,y location within said field of view, a polarity for a change of detected light intensity incident at said x,y location and an event cycle at which said event occurred events (Figure 12, Item 300; Figure 15, Step 600; Paragraphs 0162); an accumulator for accumulating event information from a plurality of events occurring within a window around the exposure time in a plurality of event frames, the event frames having a temporal relationship with one another (Figure 12, Item 302; Paragraphs 0162); wherein the movement and direction are determined from the events in at least a first of said plurality of event frames of the event camera (Figure 15, Steps 604-606; Paragraphs 0164-0165). Alternatively see Figure 8 of Daniilidis. The system of Daniilidis provides accurate prediction of optical flow (Paragraph 0161). Therefore, it would have been obvious to utilize an optical flow prediction method as taught by Daniilidis in the system of Krishnaswamy so that the optical flow may be accurately predicted.
Furthermore, it would have been obvious to align the event camera of Daniilidis and the frame camera of Krishnaswamy so that the fields of view overlap so that the optical flow of the scene being captured by the frame camera could be accurately predicted.
Additionally, Yamaguchi teaches a similar imaging system comprising a frame camera and an event camera (e.g. Figures 1-5) and further teaches synchronizing times in timestamp information included in an image signal and event data by using a clock mechanism for understanding the exact time and may correct a time gap between the RGB sensor 100 and the EVS 200 (Paragraph 0084, 0173) but does not teach a separate temporal association and synchronization using the temporal association as required in the claims.
Garcia Capel teaches a similar temporal association operation (Paragraphs 0038-0043) but also does not teach a separate temporal association and synchronization using the temporal association as required in the claims. Lee et al. (US 2016/0140731 A1 – Figures 3-6) teaches a similar system but similarly does not teach the specific requirements of the amended claims.
Therefore, while the prior art teaches similar systems, the prior art does not teach or reasonably suggest a processor or instructions configured to: perform a temporal association based on: the timestamp for each event of the plurality of events; and at least one time signal from the frame-based camera, wherein the at least one time signal corresponds to the image frame; and synchronize the plurality of events with the image frame, according to the temporal association, to generate a synchronized frame; and a display configured to display the synchronized frame as recited in the amended claims.
However, the claims cannot be considered allowable due to the 35 USC 112(a) new matter rejections above.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TIMOTHY J HENN whose telephone number is (571)272-7310. The examiner can normally be reached Monday-Friday ~10-6.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Twyler Haskins can be reached at (571) 272-7406. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Timothy J Henn/ Primary Examiner, Art Unit 2639
Prosecution Insights