DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Claims 1 and 4-6 have been amended.
No claims have been cancelled.
No new claims have been introduced.
Claims 1-6 are currently pending.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-2 and 4-5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kohara (US 20190246068 A1) in view of Dabral (US 11528453 B2).
REGARDING CLAIM 1, Kohara discloses, an image storage unit (Kohara: [0023] The display control device 9 is an electronic control device including an input signal processing unit 11, an image processing unit 13, a storage unit 15, and an output signal processing unit 17. The input signal processing unit 11 is an interface for outputting captured images that are inputted from each of the cameras 2 to 5 to the image processing unit 13) configured to cumulatively store captured images (Kohara: [0023] The display control device 9 is an electronic control device including an input signal processing unit 11, an image processing unit 13, a storage unit 15, and an output signal processing unit 17. The input signal processing unit 11 is an interface for outputting captured images that are inputted from each of the cameras 2 to 5 to the image processing unit 14; [0044] history images are acquired at every standard cycle that is the same as the frame rate of the captured images, and the history area 43 is sequentially updated), each obtained by capturing an image of a surrounding environment (Kohara: [0016] FIG. 3 is an explanatory diagram illustrating a method of synthesizing a display bird's-eye image in a normal state) by an imaging device provided in a vehicle (Kohara: [0043-0045] in the example in FIG. 3, a historical image group C of time series corresponding to consecutive frames is accumulated in the history area 43. The display image creating unit 29 creates a supplementary image D corresponding to the non-imaging area B by sequentially joining partial areas corresponding to the non-imaging area B from the new history image of the accumulated history image group C), in association with vehicle information when capturing the respective captured images (Kohara: [0030] when the imaging region of the history image reaches a predetermined position, a new history image is acquired (examiner: implies embedded location information associated with an image as well as a time-stamp)); a processor configured to synthesize the cumulatively stored captured images (Kohara: [0023] The display control device 9 is an electronic control device including an input signal processing unit 11, an image processing unit 13, a storage unit 15, and an output signal processing unit 17. The input signal processing unit 11 is an interface for outputting captured images that are inputted from each of the cameras 2 to 5 to the image processing unit 13) based on the vehicle information (Kohara: [0023] The display control device 9 is an electronic control device including an input signal processing unit 11, an image processing unit 13, a storage unit 15, and an output signal processing unit 17. The input signal processing unit 11 is an interface for outputting captured images that are inputted from each of the cameras 2 to 5 to the image processing unit 13; [0030] when the imaging region of the history image reaches a predetermined position, a new history image is acquired (examiner: implies embedded location information associated with an image as well as a time-stamp)) to generate a route surrounding image indicating a surrounding environment along a route on which the vehicle travels (Kohara: [0019] The display system 1 is an in-vehicle device that displays a captured image of the surroundings of the vehicle … [0020] Each of the cameras 2 to 5 is arranged so as to be able to capture the entire surroundings of the host vehicle, with the front, rear, left side, and right side of the vehicle as the respective imaging regions ... [0022] A bird's-eye image representing a region including the surroundings of the host vehicle inputted from the display control device 9 is displayed on the display device 7); and an image display unit configured to display the route surrounding image along a route on which the vehicle is currently traveling (Kohara: [0046] the display control process at the current frame timing and resumes the process from S100 at the next frame timing; [0049-0050]), wherein the imaging device is provided to capture an image of a region including a side region located laterally with respect to the vehicle (Kohara: [0019] The display system 1 is an in-vehicle device that displays a captured image of the surroundings of the vehicle … [0020] Each of the cameras 2 to 5 is arranged so as to be able to capture the entire surroundings of the host vehicle, with the front, rear, left side, and right side of the vehicle as the respective imaging regions ... [0022] A bird's-eye image representing a region including the surroundings of the host vehicle inputted from the display control device 9 is displayed on the display device 7), and the image storage unit extracts and stores only a part, which includes the side region, in the captured image captured by the imaging device (Kohara: [0027] The history image acquiring unit 27 extracts a partial area ahead in the traveling direction of the host vehicle as a history image from the captured bird's-eye image created by the bird's-eye view converting unit 23 at a predetermined timing according to the traveling state of the vehicle. The history image acquiring unit 27 stores the extracted history images in a time series in the history area 43 of the storage unit; [0041] In S114, the image processing unit 13 extracts a portion corresponding to an area to be saved as a history image from the latest captured bird's-eye (examiner: "bird's-eye" suggests or implies the complete perimeter of the vehicle, including sides) image stored in the real area 41, and saves the extracted history image in the history area 43; see [0052-0055]), wherein the side region is a region located on a lateral side of the vehicle perpendicular to a traveling direction of the vehicle (Kohara: [0031] Note that the captured bird's-eye image represents the latest situation of a predetermined region in the front, rear, left and right directions of the host vehicle of the surroundings of the host vehicle. On the other hand, although the history image is not the latest situation, it is may represent a situation in a region not covered by the captured bird's-eye image such as a place underneath the host vehicle due to the movement of the host vehicle; [FIG. 3(43)]).
Kohara does not explicitly recite the terminology “store captured images, each obtained by capturing an image of a surrounding environment by an imaging device provided in a vehicle, in association with vehicle information when capturing the respective captured images”. However, in considering the disclosure of a reference, it is proper to take into account not only specific teachings of the reference but also the inferences which one skilled in the art would reasonably be expected to draw therefrom [see MPEP §2144.01]. In this case, Kohara discloses “[0030] when the imaging region of the history image reaches a predetermined position, a new history image is acquired”. Which, suggests or implies, for one of ordinary skill, embedded location information of the vehicle and a time-stamp associated with an image. Which, discloses “information when capturing the respective captured images”.
The examiner respectfully submits, Kohara discloses storing side images ().
However, in the alternative, and in the same field of endeavor, Dabral discloses, the image storage unit extracts and stores only a part, which includes the side region, in the captured image captured by the imaging device (Dabral: (Col. 6, Ln. 4-24) one or more history buffers may be provided to store images captured by one or more cameras positioned about a vehicle ... a separate history buffer may be provided for each camera, or a central history buffer may be provided for some or all of the cameras ... the history buffer may be sufficiently large to buffer images for a set timeframe and/or distance for the one or more cameras supported by the history buffer. This history buffer may be used to provide images for a temporal camera in a way similar to live camera images for a virtual camera. FIG. 5 is a flow chart 500 illustrating a technique for enhanced surround view, ... the vehicle having two or more cameras disposed about the vehicle ... a vehicle includes one or more camera, such as a camera in the front, right side, left side, and back of the vehicle, configured to capture images of areas around the vehicle; [FIG. 1A]), wherein the side region is a region located on a lateral side of the vehicle perpendicular to a traveling direction of the vehicle (Dabral: (Col. 6, Ln. 4-24) one or more history buffers may be provided to store images captured by one or more cameras positioned about a vehicle ... a separate history buffer may be provided for each camera, or a central history buffer may be provided for some or all of the cameras ... the history buffer may be sufficiently large to buffer images for a set timeframe and/or distance for the one or more cameras supported by the history buffer. This history buffer may be used to provide images for a temporal camera in a way similar to live camera images for a virtual camera. FIG. 5 is a flow chart 500 illustrating a technique for enhanced surround view, ... the vehicle having two or more cameras disposed about the vehicle ... a vehicle includes one or more camera, such as a camera in the front, right side, left side, and back of the vehicle, configured to capture images of areas around the vehicle; [FIG. 1A]), for the benefit of improved sensor fusion for enhanced perceptual surround view for determining poses and nearby objects.
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to modify the process disclosed by Kohara to include saving side images taught by Dabral. One of ordinary skill in the art would have been motivated to make this modification, with a reasonable expectation of success, in order to improved sensor fusion for enhanced perceptual surround view for determining poses and nearby objects.
REGARDING CLAIM 2, Kohara, as modified, remains as applied above to claim 1. Further, Kohara also discloses, the image storage unit extracts and stores an image (Kohara: [0027] image acquiring unit 27 stores the extracted history images in a time series in the history area 43 of the storage unit 15) in an elongated rectangular region perpendicular to an axle extending in a front-rear direction of the vehicle in the captured image captured by the imaging device (Kohara: [0006] the imaging region of the camera covers a wide region; [0057] the region synthesized from history images captured at the same time for the non-imaging area becomes wider than at a normal time, and the number of times the history images captured at different times are joined is reduced. Therefore, it is possible to suppress discontinuity of images of a road surface rendered in a non-imaging area in a synthesized display bird's-eye image).
REGARDING CLAIM 4, Kohara, as modified, remains as applied above to claim 1. Further, Kohara also discloses, the image synthesizing unit (Kohara: [0023] The display control device 9 is an electronic control device including an input signal processing unit 11, an image processing unit 13, a storage unit 15, and an output signal processing unit 17. The input signal processing unit 11 is an interface for outputting captured images that are inputted from each of the cameras 2 to 5 to the image processing unit 13) generates the route surrounding image along the route on which the vehicle is currently traveling (Kohara: [0019] The display system 1 is an in-vehicle device that displays a captured image of the surroundings of the vehicle … [0020] Each of the cameras 2 to 5 is arranged so as to be able to capture the entire surroundings of the host vehicle, with the front, rear, left side, and right side of the vehicle as the respective imaging regions ... [0022] A bird's-eye image representing a region including the surroundings of the host vehicle inputted from the display control device 9 is displayed on the display device 7) in a direction opposite to a traveling direction with respect to a current position of the vehicle (Kohara: [0019] The display system 1 is an in-vehicle device that displays a captured image of the surroundings of the vehicle … [0020] Each of the cameras 2 to 5 is arranged so as to be able to capture the entire surroundings of the host vehicle, with the front, rear, left side, and right side of the vehicle as the respective imaging regions ... [0022] A bird's-eye image representing a region including the surroundings of the host vehicle inputted from the display control device 9 is displayed on the display device 7), and the image display unit displays the route surrounding image along the route on which the vehicle is currently traveling in the direction opposite to the traveling direction with respect to the current position of the vehicle (Kohara: [0019] The display system 1 is an in-vehicle device that displays a captured image of the surroundings of the vehicle … [0020] Each of the cameras 2 to 5 is arranged so as to be able to capture the entire surroundings of the host vehicle, with the front, rear, left side, and right side of the vehicle as the respective imaging regions ... [0022] A bird's-eye image representing a region including the surroundings of the host vehicle inputted from the display control device 9 is displayed on the display device 7).
REGARDING CLAIM 5, Kohara, as modified, remains as applied above to claim 2. Further, Kohara also discloses, the image synthesizing unit (Kohara: [0023] The display control device 9 is an electronic control device including an input signal processing unit 11, an image processing unit 13, a storage unit 15, and an output signal processing unit 17. The input signal processing unit 11 is an interface for outputting captured images that are inputted from each of the cameras 2 to 5 to the image processing unit 13) generates the route surrounding image along the route on which the vehicle is currently traveling (Kohara: [0019] The display system 1 is an in-vehicle device that displays a captured image of the surroundings of the vehicle … [0020] Each of the cameras 2 to 5 is arranged so as to be able to capture the entire surroundings of the host vehicle, with the front, rear, left side, and right side of the vehicle as the respective imaging regions ... [0022] A bird's-eye image representing a region including the surroundings of the host vehicle inputted from the display control device 9 is displayed on the display device 7) in a direction opposite to a traveling direction with respect to a current position of the vehicle (Kohara: [0019] The display system 1 is an in-vehicle device that displays a captured image of the surroundings of the vehicle … [0020] Each of the cameras 2 to 5 is arranged so as to be able to capture the entire surroundings of the host vehicle, with the front, rear, left side, and right side of the vehicle as the respective imaging regions ... [0022] A bird's-eye image representing a region including the surroundings of the host vehicle inputted from the display control device 9 is displayed on the display device 7), and the image display unit displays the route surrounding image along the route on which the vehicle is currently traveling in the direction opposite to the traveling direction with respect to the current position of the vehicle (Kohara: [0019] The display system 1 is an in-vehicle device that displays a captured image of the surroundings of the vehicle … [0020] Each of the cameras 2 to 5 is arranged so as to be able to capture the entire surroundings of the host vehicle, with the front, rear, left side, and right side of the vehicle as the respective imaging regions ... [0022] A bird's-eye image representing a region including the surroundings of the host vehicle inputted from the display control device 9 is displayed on the display device 7).
Claim(s) 3 and 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kohara (US 20190246068 A1) in view of Dabral (US 11528453 B2) as applied above to claim 1, and in further view of Watanabe (US 11475676 B2).
REGARDING CLAIM 3, Kohara, as modified, remains as applied above to claim 1. Further, Dabral also discloses, the image storage unit stores the captured image having (Dabral: (Col. 6, Ln. 4-24) one or more history buffers may be provided to store images captured by one or more cameras positioned about a vehicle ... a separate history buffer may be provided for each camera, or a central history buffer may be provided for some or all of the cameras ... the history buffer may be sufficiently large to buffer images for a set timeframe and/or distance for the one or more cameras supported by the history buffer. This history buffer may be used to provide images for a temporal camera in a way similar to live camera images for a virtual camera. FIG. 5 is a flow chart 500 illustrating a technique for enhanced surround view, ... the vehicle having two or more cameras disposed about the vehicle ... a vehicle includes one or more camera, such as a camera in the front, right side, left side, and back of the vehicle, configured to capture images of areas around the vehicle; [FIG. 1A]), as a projection surface (Dabral: (Col. 4, Ln. 60-64) as projected onto the bowl mesh 302 to pixels of the camera 308 with a fish-eye lens 310. This mapping may be prepared, for example, during a calibration phase, and stored, for example, in a look-up table), a curved surface that becomes further away from a ground surface as a distance from the vehicle increases (Dabral: [FIG. 1A]; see (Col. 4, Ln. 28 – Col. 5, Ln. 12) for creating bowl mesh).
However, in the alternative, and in the same field of endeavor, Watanabe discloses, having, as a projection surface, a curved surface that becomes further away from a ground surface as a distance from the vehicle increases (Watanabe: (Col. 2, Ln. 15-27) the acquisition unit on a second virtual projection surface that is set to rise from an end part of the first virtual projection surface or from a vicinity of the end part in the three-dimensional virtual space, and can display a boundary between the first virtual projection surface and the second virtual projection surface to be distinguishable. With this configuration, for example, by displaying the boundary between the first virtual projection surface that is displayed using the past image and the second virtual projection surface that is displayed using the current image so that the boundary can be discriminated, the driver can clearly discriminate between a portion of the past image and a portion of the current image in display), for the benefit of discriminating a boundary.
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to modify the imaging disclosed by modified Kohara to include raised edges taught by Watanabe. One of ordinary skill in the art would have been motivated to make this modification, with a reasonable expectation of success, in order to discriminating a boundary.
REGARDING CLAIM 6, Kohara, as modified, remains as applied above to claim 6. Further, Kohara also discloses, the image synthesizing unit (Kohara: [0023] The display control device 9 is an electronic control device including an input signal processing unit 11, an image processing unit 13, a storage unit 15, and an output signal processing unit 17. The input signal processing unit 11 is an interface for outputting captured images that are inputted from each of the cameras 2 to 5 to the image processing unit 13) generates the route surrounding image along the route on which the vehicle is currently traveling (Kohara: [0019] The display system 1 is an in-vehicle device that displays a captured image of the surroundings of the vehicle … [0020] Each of the cameras 2 to 5 is arranged so as to be able to capture the entire surroundings of the host vehicle, with the front, rear, left side, and right side of the vehicle as the respective imaging regions ... [0022] A bird's-eye image representing a region including the surroundings of the host vehicle inputted from the display control device 9 is displayed on the display device 7) in a direction opposite to a traveling direction with respect to a current position of the vehicle (Kohara: [0019] The display system 1 is an in-vehicle device that displays a captured image of the surroundings of the vehicle … [0020] Each of the cameras 2 to 5 is arranged so as to be able to capture the entire surroundings of the host vehicle, with the front, rear, left side, and right side of the vehicle as the respective imaging regions ... [0022] A bird's-eye image representing a region including the surroundings of the host vehicle inputted from the display control device 9 is displayed on the display device 7), and the image display unit displays the route surrounding image along the route on which the vehicle is currently traveling in the direction opposite to the traveling direction with respect to the current position of the vehicle (Kohara: [0019] The display system 1 is an in-vehicle device that displays a captured image of the surroundings of the vehicle … [0020] Each of the cameras 2 to 5 is arranged so as to be able to capture the entire surroundings of the host vehicle, with the front, rear, left side, and right side of the vehicle as the respective imaging regions ... [0022] A bird's-eye image representing a region including the surroundings of the host vehicle inputted from the display control device 9 is displayed on the display device 7).
Response to Arguments
Applicant’s arguments, beginning on 6, received 08-05-2026, with respect to the drawing objection, minor informalities, §112(f) interpretations and accompanying §112(a)(b) rejections of record have been fully considered and are persuasive. The drawing objection, minor informalities, §112(f) interpretations and accompanying §112(a)(b) rejections of record have been withdrawn.
While the examiner does not agree with applicant’s remarks regarding the claim 1 obviousness rejection, applicant’s arguments with respect to the claim(s) 1 rejection under 35 USC §103, obviousness, have been considered but are moot because the new ground of rejection does not rely on the reference combination applied in the prior rejection of record for matter specifically challenged in the argument.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Dabral (US 11528453 B2)
Watanabe (US 11383642 B2)
Yousefian (US 20190031101 A1)
Abe (US 20080143835 A1)
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AARRON SANTOS whose telephone number is (571)272-5288. The examiner can normally be reached Monday - Friday: 8:00am - 4:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ANGELA ORTIZ can be reached at (571) 272-1206. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/A.S./Examiner, Art Unit 3663
/ANGELA Y ORTIZ/Supervisory Patent Examiner, Art Unit 3663
