DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. JP2023-060319, filed on 04/03/2023.
Status of Claims
Currently pending Claim(s):
Amended claim(s):
New Claim(s):
1-22
1, 3, 9-10, 12, and 15
17-22
Response to Arguments
This office action is responsive to Applicant’s Arguments/Remarks made in an Amendment received on 05/11/2026.
In view of amendments filed on 05/11/2026 to the title, the objection to the specification is withdrawn.
In view of amendments filed on 05/11/2026 to claim 15, the objections to claims 15 and 16 is withdrawn. However, in view of applicant’s argument, Remarks filed on 05/11/2026, with respect to claims 6 and 7, the claim objections have been carefully considered and the arguments are found to be not persuasive. The Applicant has not provided any rational that the claims are different. The phrase “corresponding to at least part of a region”, as recited in claim 4 line 6, may correspond to the region and to something else. Further, claim 6 line 6 the phrase “corresponding to the part of the region” does not preclude the second data from corresponding to something else. Accordingly, no meaningful distinction between the limitations has been established; therefore, the claim objections to claims 6 and 7 are maintained.
In view of Applicant’s lack of written response with respect to 35 U.S.C. 112(f) claim interpretation, the interpretation made to claims 8 and 9 is maintained. Applicant has not clarified why the claimed limitations do not invoke 112(f). According to MPEP 2181, 35 USC 112(f) is applicable to claim limitation if it meets the 3-prong analysis set forth in the previous Office Action. Applicant did not specifically point out why any of these prongs have not been met, and as such, the claims continue to be treated under 112(f). Applicant is welcome to amend the claim so that the limitations no longer invoke 112(f) by, e.g., modifying the “means” or generic placeholder with specific structure, with careful consideration that no new matter is introduced.
In view of new claim amendments and applicant arguments, Remarks filed on 05/11/2026, with respect to U.S.C. 112(b), claim rejection have been carefully considered and the claim rejection to claims 1-16 under 35 U.S.C. 112(b) is withdrawn.
PNG
media_image1.png
200
678
media_image1.png
Greyscale
PNG
media_image2.png
155
688
media_image2.png
Greyscale
In view of applicant’s argument, Remarks filed on 05/11/2026, with respect to independent claims 1, 9, and 10 under 35 U.S.C. 103, claim rejections have been fully considered but they are not persuasive. The Applicant argues on pages 10-11:
The Examiner respectfully disagrees. Claim 1 as amended, now recites “second frame data from the first frame data and the motion of the object, the second frame data corresponding to a time point later than a time point to which the first frame data corresponds”. Page 7 of the Office Action dated February 10, 2026, the Examiner was explicit in their interpretation of “time later than a time” as a interval relative to an earlier time. However, in view of the amended limitation specifying “time later than a time” to be “time point later than a time point” the Applicant’s argument to Honda failing to teach this limitation is moot, as the rejection is modified in view of the amended claim language. Burns discloses on Paragraph [0028] "insert a new interpolated frame between two existing frames (e.g., at a 2x up-convert rate), the pixel events may be integrated over half of the frame capture period to generate motion vectors used to predict the new frame at the halfway point.". The generated interpolated frame is inserted between two existing frames and; therefore, the new interpolated frame (i.e. the second frame data) corresponds to a time point later than a first frame.
Accordingly, the rejection has been updated in view of the amended claim language as explained in the body of the rejection below.
Claim Objections
Claim 6 is objected to under 37 CFR 1.75 as being a substantial duplicate of claim 4 (despite slight difference in wording).
Claim 7 is objected to under 37 CFR 1.75 as being a substantial duplicate of claim 5 (despite slight difference in wording).
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
“optical device” in claim 8
“a control device” in claim 8
“a signal processing device” in claim 8
“a display device” in claim 8
“a storage device” in claim 8
“a mechanical device” in claim 8
“a capturing device” in claim 9
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
Claim 8: “optical device” corresponds to figure 9, element 940 “Examples of the optical device 940 include a lens, a shutter, and a mirror.” (Application Pub., Paragraph [0063]).
Claim 8: “a control device” corresponds to figure 9, element 950 “The control device 950 is a semiconductor device such as an ASIC.” (Application Pub., Paragraph [0063]).
Claim 8: “a signal processing device” corresponds to figure 9, element 960 “The processing device 960 processes signals output from the semiconductor device 930. The processing device 960 is a semiconductor device such as a CPU or an ASIC for constituting an AFE (Analog Front End) or a DFE (Digital Front End).” (Application Pub., Paragraph [0064]).
Claim 8: “a display device” corresponds to figure 9, element 970 “The display device 970 is an EL display device or a liquid crystal display device that displays information (image) obtained by the semiconductor device 930.” ).” (Application Pub., Paragraph [0064]).
Claim 8: “a storage device” corresponds to figure 9, element 980 “The storage device 980 is a magnetic device or a semiconductor device for storing information (image) obtained by the semiconductor device 930. The storage device 980 is a volatile memory such as SRAM or DRAM, or a non-volatile memory such as a flash memory or a hard disk drive.” (Application Pub., Paragraph [0064]).
Claim 8: “a mechanical device” corresponds to figure 9, element 990 “The mechanical device 990 includes a movable unit or a driving unit such as a motor and an engine.” (Application Pub., Paragraph [0064]).
Claim 9: “a capturing device” corresponds to figure 8, element 31 “The image capturing device 31 includes an event sensor unit 311, a frame sensor unit 312, and a transmission unit 313. The event sensor unit 311 has the functions of the event data obtaining unit 11 and the motion detection unit 131” (Application Pub, Paragraph [0058].
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-3, 8, 10-12, 17, 19, and 21 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Burns et al. (US 2018/0098082 A1) (hereinafter, “Burns”).
Regarding claim 1, Burns discloses processing device comprising (Paragraph [0013] “the disclosed techniques can be implemented, for example, in a computing system or a graphics processing system”):
a first obtaining circuit (Figure 3 element 304, video signal processor, equates to first obtaining unit) configured to obtain first frame data (image frames in paragraph [0026] equates to first frame data) that is frame data of an image of an object at a predetermined interval (Paragraph [0026] “The video signal processor 304 is configured to receive a sequence of image frames generated by a frame-based video camera 104 at a frame sampling rate (or period).”);
a second obtaining circuit (Figure 3 element 302, event signal processor, equates to second obtaining unit) configured to obtain event data that is obtainable at an interval shorter than the predetermined interval (high temporal resolution in paragraph [0025] equates to an interval shorter than…) and is a detection result of a change in a pixel value of the object (Paragraph [0025] “The event signal processor 302 is configured to receive a sequence of pixel events, generated asynchronously by an event-based video camera 102. The events, which represent illumination change in a pixel, are captured asynchronously with relatively high temporal resolution.”); and
a processor configured to detect motion of the object (image dynamics in paragraph [0017] equates to motion of the object) at an interval shorter than the predetermined interval by using the event data (Paragraph [0017] “the event-based camera may employ a dynamic (or differential) vision sensor (DVS) to record the image dynamics. As such, events associated with changes in any pixel can be captured with a much greater temporal resolution than is possible with a frame-based video camera.”; Paragraph [0028] “event integration circuit 306 is configured to integrate a subset (or all) of the sequence of pixel events, occurring within the frame sampling period between pairs of captured image frames. The integration is employed to generate a pixel motion vector representing motion of the pixel between those frames”.),
and generate second frame data (the generated interpolated video frames in paragraph [0028] equates to second video frame) from the first frame data and the motion of the object, the second frame data corresponding to a time point later than a time point to which the first frame data corresponds (Paragraph [0028] “insert a new interpolated frame between two existing frames (e.g., at a 2× up-convert rate), the pixel events may be integrated over half of the frame capture period to generate motion vectors used to predict the new frame at the halfway point.”; Paragraph [0034] “hybrid frame rate up-converter circuit 112 is configured to perform frame rate up-conversion on the sequence of image frames using motion compensated interpolation based, at least in part, on the estimated tile motion vectors. The estimated motion vectors, which are generated from the pixel motion vectors based on the pixel events… motion compensated interpolation circuit 502 is configured to generate interpolated video frames corresponding to time periods between the captured video frames by applying the tile motion vectors to tiles of a captured video frame to predict a new video frame at the next up-converted time period.”; Examiner interprets the generated interpolated frame is inserted between two existing frames and; therefore, the new interpolated frame (i.e. the second frame data) corresponds to a time point later than a first frame).
Regarding claim 2, which claim 1 is incorporated, Burns discloses wherein the first obtaining circuit captures the object at the predetermined interval (Paragraph [0026] “The video signal processor 304 is configured to receive a sequence of image frames generated by a frame-based video camera 104 at a frame sampling rate (or period).”), and
the second obtaining circuit detects the change in the pixel value (Paragraph [0025] “The event signal processor 302 is configured to receive a sequence of pixel events, generated asynchronously by an event-based video camera 102. The events, which represent illumination change in a pixel, are captured asynchronously with relatively high temporal resolution.”; Paragraph [0040] “Events are associated with a change in pixel illumination, either an increase or a decrease, which exceeds a threshold value.”).
Regarding claim 3, which claim 1 is incorporated, Burns discloses wherein the processor generates the second frame data based on the first frame data and the motion of the object detected by using the event data corresponding to the time point later than the time point to which the first frame data corresponds (Paragraph [0028] “insert a new interpolated frame between two existing frames (e.g., at a 2× up-convert rate), the pixel events may be integrated over half of the frame capture period to generate motion vectors used to predict the new frame at the halfway point.”; Paragraph [0034] “hybrid frame rate up-converter circuit 112 is configured to perform frame rate up-conversion on the sequence of image frames using motion compensated interpolation based, at least in part, on the estimated tile motion vectors. The estimated motion vectors, which are generated from the pixel motion vectors based on the pixel events… motion compensated interpolation circuit 502 is configured to generate interpolated video frames corresponding to time periods between the captured video frames by applying the tile motion vectors to tiles of a captured video frame to predict a new video frame at the next up-converted time period.”; Examiner interprets the generated interpolated frame is inserted between two existing frames and; therefore, the new interpolated frame (i.e. the second frame data) corresponds to a time point later than a first frame).
Regarding claim 8, which claim 1 is incorporated, Burns discloses an apparatus comprising the processing device (Paragraph [0013] “the disclosed techniques can be implemented, for example, in a computing system or a graphics processing system”)…and further comprising at least one of:
an optical device corresponding to the processing device (Paragraph [0035] “The images captured by the frame-based camera 104 may not align precisely with the view from the dynamic vision sensor of the event-based camera 102 due to differences in viewing angle or perspective, or differences in characteristics of the lenses or other features of the devices.”);
a control device configured to control the processing device (Paragraph [0055] “hardware elements may include processors… ASICs, programmable logic devices, digital signal processors…semiconductor devices,”);
a signal processing device configured to process a signal output from the processing device (Paragraph [0024] “hybrid motion estimation circuit 106 is shown to include an event signal processor 302, a video signal processor 304”; Paragraph [0046] “processor (or processor cores) may be any type of processor… for example, a micro-processor… a digital signal processor (DSP)”);
a display device configured to display information obtained by the processing device (Note that the claim requires only one of an optical device corresponding to the processing device; a control device configured to control the processing device; a signal processing device configured to process a signal output from the processing device; a display device configured to display information obtained by the processing device; a storage device configured to store information obtained by the processing device; and a mechanical device configured to operate based on information obtained by the processing device.);
a storage device configured to store information obtained by the processing device (Paragraph [0045] “platform 810 may comprise any combination of a processor 820, a memory 830…a user interface 860, a display element 890, and a storage system 870.”); and
a mechanical device configured to operate based on information obtained by the processing device (Paragraph [0055] “embodiments may be implemented using hardware elements, software elements, or a combination of both. Examples of hardware elements may include processors, microprocessors, circuits, circuit elements…integrated circuits, ASICs, programmable logic devices, digital signal processors, FPGAs, logic gates, registers, semiconductor devices…”).
Figure 8
PNG
media_image3.png
766
546
media_image3.png
Greyscale
Regarding claim 10, Burns discloses a processing method comprising (Paragraph [0013] “the disclosed techniques can be implemented, for example, in a computing system or a graphics processing system”):
obtaining first frame data (image frames in paragraph [0026] equate to first frame data) that is frame data of an image of an object at a predetermined interval (Paragraph [0026] “The video signal processor 304 is configured to receive a sequence of image frames generated by a frame-based video camera 104 at a frame sampling rate (or period).”);
obtaining event data that is obtainable at an interval shorter than the predetermined interval and is a detection result of a change in a pixel value of the object (Paragraph [0025] “The event signal processor 302 is configured to receive a sequence of pixel events, generated asynchronously by an event-based video camera 102. The events, which represent illumination change in a pixel, are captured asynchronously with relatively high temporal resolution.”);
detecting motion of the object (image dynamics in paragraph [0017] equates to motion of the object) at an interval shorter than the predetermined interval by using the event data (Paragraph [0017] “the event-based camera may employ a dynamic (or differential) vision sensor (DVS) to record the image dynamics. As such, events associated with changes in any pixel can be captured with a much greater temporal resolution than is possible with a frame-based video camera.”; Paragraph [0028] “event integration circuit 306 is configured to integrate a subset (or all) of the sequence of pixel events, occurring within the frame sampling period between pairs of captured image frames. The integration is employed to generate a pixel motion vector representing motion of the pixel between those frames”); and
generating second frame data (generated interpolated video frames in paragraphs [0028] and [0034] equate to second frame data) from the first frame data and the motion of the object, the second frame data corresponding to a time point later than a time point to which the first frame data corresponds (Paragraph [0028] “insert a new interpolated frame between two existing frames (e.g., at a 2× up-convert rate), the pixel events may be integrated over half of the frame capture period to generate motion vectors used to predict the new frame at the halfway point.”; Paragraph [0034] “hybrid frame rate up-converter circuit 112 is configured to perform frame rate up-conversion on the sequence of image frames using motion compensated interpolation based, at least in part, on the estimated tile motion vectors. The estimated motion vectors, which are generated from the pixel motion vectors based on the pixel events… motion compensated interpolation circuit 502 is configured to generate interpolated video frames corresponding to time periods between the captured video frames by applying the tile motion vectors to tiles of a captured video frame to predict a new video frame at the next up-converted time period.”; Examiner interprets the generated interpolated frame is inserted between two existing frames and; therefore, the new interpolated frame (i.e. the second frame data) corresponds to a time point later than a first frame).
Regarding claim 11 (drawn to a method), claim 11 is rejected the same as claim 2 and the arguments similar to that presented above for claim 2 are equally applicable to the claim 11, and all the other limitations similar to claim 2 are not repeated herein, but incorporated by reference.
Regarding claim 12 (drawn to a method), claim 12 is rejected the same as claim 3 and the arguments similar to that presented above for claim 3 are equally applicable to the claim 12, and all the other limitations similar to claim 3 are not repeated herein, but incorporated by reference.
Regarding claim 17, which claim 1 is incorporated, Burns discloses wherein the event data includes a polarity of an event (Paragraph [0017] “Events may include…a polarity to indicate whether the pixel increased or decreased in illumination”).
Regarding claim 19, which claim 17 is incorporated, Burns discloses wherein the event data includes coordinate values corresponding to an event (Paragraph [0017 “Events may include an address of the location of the pixel that generated the event”; Paragraph [0040 “the generated events may include an address of the pixel associated with the event and a timestamp. The address may be a location relative to a 2-dimensional pixel array of the event-based video camera.”).
Regarding claim 21, which claim 19 is incorporated, Burns discloses wherein the event data includes time stamp of a time point at which an event occurred (Paragraph [0040 “the generated events may include an address of the pixel associated with the event and a timestamp.”).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Burns et al. (US 2018/0098082 A1) (hereinafter, “Burns”) in view of Honda et al. (US 2022/0030185 A1) (hereinafter, “Honda”).
Regarding claim 9, Burns discloses a system comprising: a capturing device (Paragraph [0044] “system 800 to perform motion estimation using hybrid video imaging, configured in accordance with certain embodiments of the present disclosure. In some embodiments, system 800 comprises a platform 810 which may host, or otherwise be incorporated into a personal computer, workstation, laptop computer, ultra-laptop computer, tablet, touchpad, portable computer, handheld computer, palmtop computer… and so forth”); and
a display device (Paragraph [0045] “platform 810 may comprise any combination of a processor 820…an input/output (I/O) system 850, an event-based video camera 102… a display element 890”),
wherein the capturing device includes: a first obtaining circuit (Figure 3 element 304, video signal processor, equates to first obtaining unit) configured to obtain first frame data (image frames in paragraph [0026] equate to first frame data) that is frame data of an image of an object by capturing the object at a predetermined interval (Paragraph [0026] “The video signal processor 304 is configured to receive a sequence of image frames generated by a frame-based video camera 104 at a frame sampling rate (or period).”);
a second obtaining circuit (Figure 3 element 302, event signal processor, equates to second obtaining unit) configured to obtain event data that is obtainable at an interval shorter than the predetermined interval and is a detection result of a change in a pixel value of the object by detecting the change in the pixel value of the object (Paragraph [0025] “The event signal processor 302 is configured to receive a sequence of pixel events, generated asynchronously by an event-based video camera 102. The events, which represent illumination change in a pixel, are captured asynchronously with relatively high temporal resolution.”); and
a first processor configured to detect motion of the object (image dynamics in paragraph [0017] equates to motion of the object) at an interval shorter than the predetermined interval by using the event data (Paragraph [0028] “event integration circuit 306 is configured to integrate a subset (or all) of the sequence of pixel events, occurring within the frame sampling period between pairs of captured image frames. The integration is employed to generate a pixel motion vector representing motion of the pixel between those frames”); and
[a transmission interface configured to transmit] the first frame data and motion information regarding the motion of the object (Paragraph [0026] “The video signal processor 304 is configured to receive a sequence of image frames generated by a frame-based video camera 104 at a frame sampling rate (or period).”; Paragraph [0028] “event integration circuit 306 is configured to integrate a subset (or all) of the sequence of pixel events, occurring within the frame sampling period between pairs of captured image frames. The integration is employed to generate a pixel motion vector representing motion of the pixel between those frames”) [to outside of the capturing device, and the display device includes:
a receiving interface configured to receive] the first frame data and the motion information (Paragraph [0026] “The video signal processor 304 is configured to receive a sequence of image frames generated by a frame-based video camera 104 at a frame sampling rate (or period).”; Paragraph [0028] “event integration circuit 306 is configured to integrate a subset (or all) of the sequence of pixel events, occurring within the frame sampling period between pairs of captured image frames. The integration is employed to generate a pixel motion vector representing motion of the pixel between those frames”) [from outside of the display device];
a second processor configured to generate second frame data (generated interpolated video frames in paragraphs [0028] and [0034] equate to second frame data) from the first frame data and the motion information, the second frame data corresponding to a time point later than a time point to which the first frame data corresponds (Paragraph [0028] “insert a new interpolated frame between two existing frames (e.g., at a 2× up-convert rate), the pixel events may be integrated over half of the frame capture period to generate motion vectors used to predict the new frame at the halfway point.”; Paragraph [0034] “hybrid frame rate up-converter circuit 112 is configured to perform frame rate up-conversion on the sequence of image frames using motion compensated interpolation based, at least in part, on the estimated tile motion vectors. The estimated motion vectors, which are generated from the pixel motion vectors based on the pixel events… motion compensated interpolation circuit 502 is configured to generate interpolated video frames corresponding to time periods between the captured video frames by applying the tile motion vectors to tiles of a captured video frame to predict a new video frame at the next up-converted time period.”; Examiner interprets the generated interpolated frame is inserted between two existing frames and; therefore, the new interpolated frame (i.e. the second frame data) corresponds to a time point later than a first frame); and
[a display configured to display an image] based on the second frame data (Paragraph [0034] “The frame insertion circuit 504 is configured to insert the predicted frames into the video frame stream to generate the up-converted frame stream.”; Paragraph [0044] “system 800 to perform motion estimation using hybrid video imaging…system 800 comprises a platform 810 which may host, or otherwise be incorporated into a personal computer, workstation, laptop computer, ultra-laptop computer…”).
However, Burns fails to teach a transmission interface configured to transmit [the first frame data and motion information regarding the motion of the object] to outside of the capturing device, and the display device includes: a receiving interface configured to receive [the first frame data and the motion information] from outside of the display device; and a display configured to display an image based on the second frame data.
Honda teaches a transmission interface configured to transmit [the first frame data and motion information regarding the motion of the object] to outside of the capturing device (Paragraph [0290] “The sound/image output section 12052 transmits an output signal of at least one of a sound or an image to an output device capable of visually or auditorily notifying an occupant of the vehicle or the outside of the vehicle of information.”), and
the display device includes: a receiving interface configured to receive [the first frame data and the motion information] from outside of the display device (Paragraph [0290] “transmits an output signal of at least one of a sound or an image to an output device capable of visually or auditorily notifying an occupant of the vehicle or the outside of the vehicle of information. In the example of FIG. 23, an audio speaker 12061, a display section 12062… are exemplified as the output device.”);
a display configured to display an image (Paragraph [0290] “transmits an output signal of at least one of a sound or an image to an output device capable of visually or auditorily notifying an occupant of the vehicle or the outside of the vehicle of information. In the example of FIG. 23, an audio speaker 12061, a display section 12062… are exemplified as the output device.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective
filing date to modify Burns’ reference to include a transmission interface configured to transmit [the first frame data and motion information regarding the motion of the object] to outside of the capturing device, and the display device includes: a receiving interface configured to receive [the first frame data and the motion information] from outside of the display device; and a display configured to display an image taught by Honda’s reference. The motivation for doing so would have been to notify and display the information to the user, as well as to improve the result’s reliability to obtain frame data that produces an image with high visibility and smooth movement as suggested by Honda (see Honda, Paragraph [0090], Paragraph [0138], and Paragraph [190]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Honda with Burns to obtain the invention specified in claim 9.
Claims 4-7 and 13-14 are rejected under 35 U.S.C. 103 as being unpatentable over Burns et al. (US 2018/0098082 A1) (hereinafter, “Burns”) in view of He et al. ("Timereplayer: Unlocking the potential of event cameras for video interpolation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.) (hereinafter, “He”).
Regarding claim 4, which claim 1 is incorporated, Burns discloses [wherein the processor further generates,] in a case where a difference between the first frame data and the second frame data is larger than a threshold (Paragraph [0025] “The event signal processor 302 is configured to receive a sequence of pixel events, generated asynchronously by an event-based video camera 102. The events, which represent illumination change in a pixel, are captured asynchronously with relatively high temporal resolution.”; Paragraph [0040] “Events are associated with a change in pixel illumination, either an increase or a decrease, which exceeds a threshold value.”).
However, Burns fails to teach wherein the processor further generates…third frame data by using third data that has a value between first data and second data, the first data corresponding to at least part of a region of the object in the first frame data, and the second data corresponding to at least part of a region of the object in the second frame data.
He teaches wherein the processor further generates…third frame data (target frame on Page 17807 right column last paragraph equates to third frame data) by using third data (average of two warped frames on Page 17807 right column last paragraph equates to third date) that has a value between first data and second data (Page 17807 right column last paragraph continued on to Page 17808 left column first paragraph “The target frame Ît can be synthesized by blending the warped input frames using refined optical flows. The blending process is taken as a weighted 17807 average of two warped frames with the product of time interval and visibility map as weights… as long as two input frames at two time stamps t0 and t1, and the event streams between these two time stamps and the targeted one t are given, we could synthesize the desired frame at that time stamp”), the first data corresponding to at least part of a region of the object in the first frame data (input frame at t0 on Page 17808 left column first paragraph equates to first frame) (Page 17807 right column last paragraph continued on to Page 17808 left column first paragraph “The target frame Ît can be synthesized by blending the warped input frames using refined optical flows. The blending process is taken as a weighted 17807 average of two warped frames with the product of time interval and visibility map as weights… as long as two input frames at two time stamps t0 and t1, and the event streams between these two time stamps and the targeted one t are given, we could synthesize the desired frame at that time stamp”), and the second data corresponding to at least part of a region of the object in the second frame data (input frame at t1 on Page 17808 left column first paragraph equates to second frame) (Page 17807 right column last paragraph continued on to Page 17808 left column first paragraph “The target frame Ît can be synthesized by blending the warped input frames using refined optical flows. The blending process is taken as a weighted 17807 average of two warped frames with the product of time interval and visibility map as weights… as long as two input frames at two time stamps t0 and t1, and the event streams between these two time stamps and the targeted one t are given, we could synthesize the desired frame at that time stamp”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective
filing date to modify Burns’ reference to include wherein the processor further generates…third frame data by using third data that has a value between first data and second data, the first data corresponding to at least part of a region of the object in the first frame data, and the second data corresponding to at least part of a region of the object in the second frame data taught by He’s reference. The motivation for doing so would have been to address complex motion and reconstruct high-quality intermediate frames as suggested by He (see He, Page 17811, Section 5 Conclusion right column).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine He with Burns to obtain the invention specified in claim 4.
Regarding claim 5, which claim 4 is incorporated, Burns fails to teach wherein the value of the third data is an average value between a value of the first data and a value of the second data.
He teaches wherein the value of the third data (average of two warped frames on Page 17807 right column last paragraph equates to third date) is an average value between a value of the first data and a value of the second data (Page 17807 right column last paragraph continued on to Page 17808 left column first paragraph “The target frame Ît can be synthesized by blending the warped input frames using refined optical flows. The blending process is taken as a weighted 17807 average of two warped frames with the product of time interval and visibility map as weights… as long as two input frames at two time stamps t0 and t1, and the event streams between these two time stamps and the targeted one t are given, we could synthesize the desired frame at that time stamp”)
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Burns’ reference to include wherein the value of the third data is an average value between a value of the first data and a value of the second data taught by He’s reference. The motivation for doing so would have been to address complex motion and reconstruct high-quality intermediate frames as suggested by He (see He, Page 17811, Section 5 Conclusion right column).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine He with Burns to obtain the invention specified in claim 5.
Regarding claim 6, which claim 1 is incorporated, Burns teaches [wherein the processor further generates], in a case where a difference between the first frame data and the second frame data is larger than a threshold (Paragraph [0025] “The event signal processor 302 is configured to receive a sequence of pixel events, generated asynchronously by an event-based video camera 102. The events, which represent illumination change in a pixel, are captured asynchronously with relatively high temporal resolution.”; Paragraph [0040] “Events are associated with a change in pixel illumination, either an increase or a decrease, which exceeds a threshold value.”).
However, Burns fails to teach wherein the processor further generates…third frame data by using third data that has a value between first data and second data, the first data corresponding to a part of a region of the object in the first frame data, and the second data corresponding to the part of a region of the object in the second frame data.
He teaches wherein the processor further generates…third frame data by using third data that has a value between first data and second data (Page 17807 right column last paragraph continued on to Page 17808 left column first paragraph “The target frame Ît can be synthesized by blending the warped input frames using refined optical flows. The blending process is taken as a weighted 17807 average of two warped frames with the product of time interval and visibility map as weights… as long as two input frames at two time stamps t0 and t1, and the event streams between these two time stamps and the targeted one t are given, we could synthesize the desired frame at that time stamp”), the first data corresponding to a part of a region of the object in the first frame data (Page 17807 right column last paragraph continued on to Page 17808 left column first paragraph “The target frame Ît can be synthesized by blending the warped input frames using refined optical flows. The blending process is taken as a weighted 17807 average of two warped frames with the product of time interval and visibility map as weights… as long as two input frames at two time stamps t0 and t1, and the event streams between these two time stamps and the targeted one t are given, we could synthesize the desired frame at that time stamp”), and the second data corresponding to the part of a region of the object in the second frame data (Page 17807 right column last paragraph continued on to Page 17808 left column first paragraph “The target frame Ît can be synthesized by blending the warped input frames using refined optical flows. The blending process is taken as a weighted 17807 average of two warped frames with the product of time interval and visibility map as weights… as long as two input frames at two time stamps t0 and t1, and the event streams between these two time stamps and the targeted one t are given, we could synthesize the desired frame at that time stamp”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective
filing date to modify Burns’ reference to include wherein the processor further generates…third frame data by using third data that has a value between first data and second data, the first data corresponding to a part of a region of the object in the first frame data, and the second data corresponding to the part of a region of the object in the second frame data taught by He’s reference. The motivation for doing so would have been to address complex motion and reconstruct high-quality intermediate frames as suggested by He (see He, Page 17811, Section 5 Conclusion right column).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine He with Burns to obtain the invention specified in claim 6.
Regarding claim 7, which claim 6 is incorporated, Burns fails to teach wherein the value of the third data is an average value between a value of the first data and a value of the second data.
He teaches wherein the value of the third data is an average value between a value of the first data and a value of the second data (Page 17807 right column last paragraph continued on to Page 17808 left column first paragraph “The target frame Ît can be synthesized by blending the warped input frames using refined optical flows. The blending process is taken as a weighted 17807 average of two warped frames with the product of time interval and visibility map as weights… as long as two input frames at two time stamps t0 and t1, and the event streams between these two time stamps and the targeted one t are given, we could synthesize the desired frame at that time stamp”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective
filing date to modify Burns’ reference to include wherein the value of the third data is an average value between a value of the first data and a value of the second data taught by He’s reference. The motivation for doing so would have been to address complex motion and reconstruct high-quality intermediate frames as suggested by He (see He, Page 17811, Section 5 Conclusion right column).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine He with Burns to obtain the invention specified in claim 7.
Regarding claim 13 (drawn to a method), claim 13 is rejected the same as claim 4 and the arguments similar to that presented above for claim 4 are equally applicable to the claim 13, and all the other limitations similar to claim 4 are not repeated herein, but incorporated by reference.
Regarding claim 14 (drawn to a method), claim 14 is rejected the same as claim 5 and the arguments similar to that presented above for claim 5 are equally applicable to the claim 14, and all the other limitations similar to claim 5 are not repeated herein, but incorporated by reference.
Claims 15-16 are rejected under 35 U.S.C. 103 as being unpatentable over Burns et al. (US 2018/0098082 A1) (hereinafter, “Burns”) in view of He et al. ("Timereplayer: Unlocking the potential of event cameras for video interpolation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.) (hereinafter, “He”); and further in view of Miyabe et al. (US12,340,545 B2) (hereinafter, “Miyabe”).
Regarding claim 15, which claim 10 is incorporated, Burns discloses [further comprising: generating,] in a case where a difference between the first frame data and the second frame data is larger than a threshold (Paragraph [0025] “The event signal processor 302 is configured to receive a sequence of pixel events, generated asynchronously by an event-based video camera 102. The events, which represent illumination change in a pixel, are captured asynchronously with relatively high temporal resolution.”; Paragraph [0040] “Events are associated with a change in pixel illumination, either an increase or a decrease, which exceeds a threshold value.”), [third frame data by using third data that has a value between first data and second data, the first data corresponding to a part of a region of the object in the first frame data, and the second data corresponding to the part of a region of the object in the second frame data,
wherein the third data is generated based on the difference between the first data and the second data].
However, Burn’s fails to teach further comprising: generating…third frame data by using third data that has a value between first data and second data, the first data corresponding to a part of a region of the object in the first frame data, and the second data corresponding to the part of a region of the object in the second frame data, wherein the third data is generated based on the difference between the first data and the second data.
He teaches further comprising: generating…third frame data by using third data that has a value between first data and second data (Page 17807 right column last paragraph continued on to Page 17808 left column first paragraph “The target frame Ît can be synthesized by blending the warped input frames using refined optical flows. The blending process is taken as a weighted 17807 average of two warped frames with the product of time interval and visibility map as weights… as long as two input frames at two time stamps t0 and t1, and the event streams between these two time stamps and the targeted one t are given, we could synthesize the desired frame at that time stamp”), the first data corresponding to a part of a region of the object in the first frame data (Page 17807 right column last paragraph continued on to Page 17808 left column first paragraph “The target frame Ît can be synthesized by blending the warped input frames using refined optical flows. The blending process is taken as a weighted 17807 average of two warped frames with the product of time interval and visibility map as weights… as long as two input frames at two time stamps t0 and t1, and the event streams between these two time stamps and the targeted one t are given, we could synthesize the desired frame at that time stamp”), and the second data corresponding to the part of a region of the object in the second frame data, [wherein the third data is generated based on the difference] between the first data and the second data (Page 17807 right column last paragraph continued on to Page 17808 left column first paragraph “The target frame Ît can be synthesized by blending the warped input frames using refined optical flows. The blending process is taken as a weighted 17807 average of two warped frames with the product of time interval and visibility map as weights… as long as two input frames at two time stamps t0 and t1, and the event streams between these two time stamps and the targeted one t are given, we could synthesize the desired frame at that time stamp”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Burns’ reference to include further comprising: generating…third frame data by using third data that has a value between first data and second data, the first data corresponding to a part of a region of the object in the first frame data, and the second data corresponding to the part of a region of the object in the second frame data, [wherein the third data is generated based on the difference] between the first data and the second data taught by He’s reference. The motivation for doing so would have been to address complex motion and reconstruct high-quality intermediate frames as suggested by He (see He, Page 17811, Section 5 Conclusion right column).
However, Burns and He both fail to teach wherein the third data is generated based on the difference [between the first data and the second data].
Miyabe teaches wherein the third data is generated based on the difference between the first data and the second data (Column 10 [lines 40-42) “FIG. 6 is a diagram illustrating an operation example of generating frame data (also referred to as difference data)”; Column 12 [lines 45-53] “example of generating frame data according to a first example of this embodiment…a part of the integration period in each frame period overlaps with the integration period in the previous frame period. In this way, by providing an overlap period in which the integration period overlaps between the preceding and subsequent frame periods, it is possible to spread the image of the object in the reconstructed frame data along the movement trajectory”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective
filing date to modify Burns in view of He to include wherein the third data is generated based on the difference between the first data and the second data taught by Miyabe’s reference. The motivation for doing so would have been to suppress a decrease in processing accuracy due to loss of the object in object tracking as suggested by Miyabe (see Miyabe, Column 12 [lines 54-56]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Miyabe with Burns and He to obtain the invention specified in claim 15.
Regarding claim 16, which claim 15 is incorporated, Burns and Miyabe both fail to teach wherein the value of the third data is an average value between a value of the first data and a value of the second data.
He teaches wherein the value of the third data is an average value between a value of the first data and a value of the second data (Page 17807 right column last paragraph continued on to Page 17808 left column first paragraph “The target frame Ît can be synthesized by blending the warped input frames using refined optical flows. The blending process is taken as a weighted 17807 average of two warped frames with the product of time interval and visibility map as weights… as long as two input frames at two time stamps t0 and t1, and the event streams between these two time stamps and the targeted one t are given, we could synthesize the desired frame at that time stamp”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Burns’ reference to include wherein the value of the third data is an average value between a value of the first data and a value of the second data taught by He’s reference. The motivation for doing so would have been to address complex motion and reconstruct high-quality intermediate frames as suggested by He (see He, Page 17811, Section 5 Conclusion right column).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine He with Burns and Miyabe to obtain the invention specified in claim 16.
Claims 18, 20, and 22 are rejected under 35 U.S.C. 103 as being unpatentable over Burns et al. (US 2018/0098082 A1) (hereinafter, “Burns”) in view of Jiang et al. (US 10,776,688 B2) (hereinafter, “Jiang”).
Regarding claim 18, which claim 17 is incorporated, Burns discloses wherein the processor detects, as the motion of the object, [a type and] amount of the motion of the object (Paragraph [0032] “The estimated motion vectors, which are generated from the pixel motion vectors based on the pixel events, represent the total motion of tiles between two image frames.”).
However, Burns fails to teach a type [of motion].
Jiang teaches a type [of motion] (Column 5 [lines 62-65 “Optical flow data comprises a flow field indicating a direction and magnitude of image data movement from the first frame to the second frame.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective
filing date to modify Burns’ reference to include a type [of motion] taught by Jiang’s reference. The motivation for doing so would have been to reduce artifacts in the interpolated frames as suggested by Jiang (see Jiang, Column 3 [lines 24-26]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Jiang with Burns to obtain the invention specified in claim 18.
Regarding claim 20, which claim 19 is incorporated, Burns discloses wherein the processor detects, as the motion of the object, [a type and] amount of the motion of the object (Paragraph [0032] “The estimated motion vectors, which are generated from the pixel motion vectors based on the pixel events, represent the total motion of tiles between two image frames.”).
However, Burns fails to teach a type [of motion].
Jiang teaches a type [of motion] (Column 5 [lines 62-65 “Optical flow data comprises a flow field indicating a direction and magnitude of image data movement from the first frame to the second frame.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective
filing date to modify Burns’ reference to include a type [of motion] taught by Jiang’s reference. The motivation for doing so would have been to reduce artifacts in the interpolated frames as suggested by Jiang (see Jiang, Column 3 [lines 24-26]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Jiang with Burns to obtain the invention specified in claim 20.
Regarding claim 22, which claim 21 is incorporated, Burns discloses wherein the processor detects, as the motion of the object, [a type and] amount of the motion of the object (Paragraph [0032] “The estimated motion vectors, which are generated from the pixel motion vectors based on the pixel events, represent the total motion of tiles between two image frames.”).
However, Burns fails to teach a type [of motion].
Jiang teaches a type [of motion] (Column 5 [lines 62-65 “Optical flow data comprises a flow field indicating a direction and magnitude of image data movement from the first frame to the second frame.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective
filing date to modify Burns’ reference to include a type [of motion] taught by Jiang’s reference. The motivation for doing so would have been to reduce artifacts in the interpolated frames as suggested by Jiang (see Jiang, Column 3 [lines 24-26]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Jiang with Burns to obtain the invention specified in claim 22.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Uekusa et al. (US 10,880,457 B2) discloses an image processing apparatus which tracks points across images to detects when a moving object enters a region.
Madsen (US 2019/0188861 A1) discloses a video processing method that detects motion deviation by generating change grids, calculating movement vectors and comparing them with reference vectors.
Cho et al. (US 11,508,075 B2) discloses a method and apparatus for target tracking on an input image obtained at one time period and a second time period.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to UROOJ FATIMA whose telephone number is (571)272-2096. The examiner can normally be reached M-F 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Henok Shiferaw can be reached at (571) 272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/UROOJ FATIMA/Examiner, Art Unit 2676
/Henok Shiferaw/Supervisory Patent Examiner, Art Unit 2676
Prosecution Insights