Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
The United States Patent & Trademark Office appreciates the application that is submitted by the inventor/assignee. The United States Patent & Trademark Office reviewed the following application and has made the following comments below.
Priority
This application claims benefit of foreign priority under 35 U.S.C. 119(a)-(d) of DE102022125210.8, filed in Germany on 09/29/2022, and PCT/EP2023/067344, filed in Europe on 06/26/2023.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 01/14/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Preliminary Amendment
Applicant submitted a preliminary amendment on 01/14/2025. The Examiner acknowledges the amendment and has reviewed the claims accordingly.
Status of Claims
Claims 16-31 are pending.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 16-17, 21-22, 28, and 30 are rejected under 35 U.S.C. 102(a)(1)/(a)(2) as being anticipated by Asai (U.S. Patent Pub. No. US 20190075250 A1, hereafter referred to as Asai).
Regarding Claim 16, Asai teaches a method for merging partial images of surroundings of a vehicle to form an overall image (Paragraphs [0009], [0022], Asai teaches an image synthesizing process for synthesizing the rear camera image, the rear left camera image, and the rear right camera image.), wherein the method comprises: acquiring a first partial image in a first area of the surroundings (Fig. 3(a), Paragraphs [0020-21], [0024], Asai teaches a rear left camera (11BL) configured to image a rear left area located on the rear of the vehicle (1) and on the left of the vehicle (1), namely, located at the backward side and left [AltContent: arrow]from the vehicle. See Fig. 3(a), the rear left camera image (111BL).),
PNG
media_image1.png
320
499
media_image1.png
Greyscale
wherein the first area is located laterally behind the vehicle (Paragraphs [0020-21], Fig. 2, Asai teaches the rear left area is located on the rear of the vehicle and on the left of the vehicle. In addition, the right rear camera images a rear right area located on the rear of the vehicle and on the right of the vehicle.);
[AltContent: arrow]
PNG
media_image2.png
668
631
media_image2.png
Greyscale
acquiring a second partial image in a second area of the surroundings (Paragraph [0019], Fig. 3(a), Asai teaches the rear camera (11B) captures an image of a rear area located on the rear of the vehicle.), wherein the second area is located behind the vehicle (Paragraph [0019], Fig. 2, Asai teaches the rear area corresponds to an area including at least a portion of an area that is reflected in a back mirror placed in a vehicle interior of a vehicle.); acquiring an object (Paragraph [0027], Fig 5(b), Fig 5(c), Asai teaches a rear vehicle (alternatively, any object other than the vehicle) is in the image corresponding to the rear camera image (111B).),
[AltContent: arrow]
PNG
media_image3.png
392
445
media_image3.png
Greyscale
at least one section of which is located in one of the areas (Paragraph [0027], Fig. 5(b), Fig. 5(c), Asai teaches a rear vehicle is in a portion of the synthesized image corresponding to the rear camera image (111B).); determining a distance of the object from the vehicle (Paragraph [0044], Asai teaches a detection apparatus configured to detect the distance between the rear vehicle and the vehicle, such as a radar or a lidar. Alternatively, the image synthesizing unit may calculate the distance between the rear vehicle and the vehicle by performing an image process.); adapting the second partial image depending on the distance (Paragraphs [0074], [0077], Asai teaches changing a proportion of an area in which the first image is displayed in the synthetized image and a proportion of an area in which the second image is displayed in the synthesized image on the basis of a distance between an object in the synthesized image and the vehicle in generating the synthetized image.); and merging the partial images to form the overall image (Paragraph [0022], Asai teaches generating a synthesized image by performing an image synthesizing process for synthesizing the rear camera image (111B), the rear left camera image (111BL) and the rear right camera image (111BR). Fig. 3(a) illustrates a synthesized image.).
In regards to Claim 17, Asai teaches the method according to claim 16, wherein the overall image comprises a first section (Paragraph [0024], Fig. 3(a), Asai teaches generating a synthesized image in which the rear left area is on the left of the rear area and the rear right area is on the right of the rear right area. The synthesized image has the rear left camera image (111BL) on the left of the rear camera image (111B) and the rear right camera image (111BR) is on the right of the rear camera image (111B). See Fig. 3(a), the Examiner interprets the rear left image (111BL) to be “a first section.”), which is taken from the first partial image (Paragraph [0022], Asai teaches the rear left camera image (111BL) is an image that is captured by the rear left camera (11BL).) and represents an area of the surroundings outside the vehicle (Paragraph [0020], Fig. 3(a), Asai teaches the rear left camera images an area in a rear left area located on the rear of the vehicle and on the left of the vehicle.);
PNG
media_image4.png
405
793
media_image4.png
Greyscale
and a second section (Paragraph [0022], Fig. 3(a), Asai teaches the synthesized image includes the rear camera image (111B), the rear left camera image (111BL), and the rear right camera image (111BR). See Fig. 3(a), the Examiner interprets the rear camera image (111B) to be “a second section.”), which is taken from the second partial image (Paragraph [0019], Asai teaches the rear camera (11B) captures an image of a rear area located on the rear of the vehicle. See rear camera image (111B) in Fig. 3(a).) and represents an area of the surroundings located behind the vehicle (Paragraph [0020], Fig. 3(a), Asai teaches the rear left camera images an area in a rear left area located on the rear of the vehicle and on the left of the vehicle. See Fig. 3(a), rear camera image (111B) represents an area of the surroundings located behind the vehicle (i.e., rear vehicle, trees, roadway, guard rail, etc.).).
PNG
media_image5.png
360
568
media_image5.png
Greyscale
In regards to Claim 21, Asai discloses the method according to claim 16, wherein adaptations of the second partial image are predetermined for multiple predetermined distances between the vehicle and the object (Paragraphs [0047-49], [0041-43], Asai teaches changing a proportion (a ratio or percentage) of an area in which at least one of the rear camera image (111B), the rear left camera image (111BL), and the rear right camera (111BR) is displayed in the synthesized image based on the distance (threshold value) between the rear vehicle in the synthesized image and the vehicle. First and second threshold values are disclosed. The second threshold value may be different from the first threshold value. The Examiner interprets first and second “threshold values” to be “multiple” predetermined distances.).
In regards to Claim 22, Asai discloses the method according to claim 21, wherein an adaptation is interpolated with respect to a first predetermined adaptation (Paragraph [0041], Asai teaches generating the synthesized image that includes at least the border image (112L), if the distance between the rear vehicle located in the rear left area and the vehicle is smaller than a first threshold value. The Examiner interprets the border image (112L) is “interpolated” since the border is inserted/added between the partial images. The Examiner interprets “interpolated” to mean inserting something into something else. Therefore, inserting the border image into the synthesized image is interpreted as interpolating.), an assigned distance of which is less than one of the predetermined distances (Paragraphs [0041], [0044], Asai teaches a distance condition relating to the distance between the rear vehicle in the synthesized image and the vehicle and may be used as one example of the predetermined condition. A first example of a distance condition that the distance between the rear vehicle in the synthesized image and the vehicle is smaller than a first threshold value.), and a second predetermined adaptation, the assigned distance of which is greater than the determined distance and is applied to the second partial image (Paragraphs [0042], [004], Asai teaches generating the synthesized image that does not include the border images if the distance between the rear vehicle and the vehicle is not smaller (i.e., larger) than a first threshold value.).
[AltContent: arrow]In regards to Claim 28, Asai teaches a device for merging partial images of surroundings of a vehicle to form an overall image (Paragraph [0006], Fig. 1, Asai teaches a synthesizing device that is configured to generate a synthesized image. An ECU (Electronic Control Unit) is one example of a “synthesizing device.”),
PNG
media_image6.png
487
385
media_image6.png
Greyscale
wherein the device comprises: a first sensor for acquiring a first partial image in a first area of the surroundings, wherein the first area is located laterally behind the vehicle (Paragraphs [0017-18], [0020-21], Fig. 2, Asai teaches a rear left camera (11BL) is an imaging device that is configured to image a rear left area located on the rear of the vehicle on the left of the vehicle.); a second sensor for acquiring a second partial image in a second area of the surroundings, wherein the second area is located behind the vehicle (Paragraphs [0017-19], Fig. 2, Asai teaches the rear camera (11B) is an imaging device that is configured to capture an image of a rear area located on the rear of the vehicle as illustrated in Fig. 2.); a third sensor for determining a distance of an object (Paragraph [0044], Asai teaches the vehicle has a detection apparatus that is configured to detect the distance between the rear vehicle and the vehicle. A radar, a lidar, and the like are examples of the detection apparatus.), at least one section of which is located in one of the areas (Paragraph [0027], Fig. 5(b), Fig. 5(c), Asai teaches a rear vehicle is in a portion of the synthesized image corresponding to the rear camera image (111B).), from the vehicle (Paragraph [0044], Asai teaches the vehicle detection apparatus detects the distance between the rear vehicle and the vehicle.); and a processing device (Paragraph [0022], Asai teaches an electronic control unit which includes processing blocks or processing circuits physically realized in the ECU.) for adapting the second partial image depending on the distance (Paragraphs [0074], [0077], Asai teaches changing a proportion of an area in which the first image is displayed in the synthetized image and a proportion of an area in which the second image is displayed in the synthesized image on the basis of a distance between an object in the synthesized image and the vehicle in generating the synthetized image.); and for merging the partial images to form the overall image (Paragraph [0022], Asai teaches generating a synthesized image by performing an image synthesizing process for synthesizing the rear camera image (111B), the rear left camera image (111BL) and the rear right camera image (111BR). Fig. 3(a) illustrates a synthesized image.).
In regards to Claim 30, Asai teaches a vehicle (Paragraph [0007], Figs. 1 and 2, Asai teaches a vehicle.) comprising: a device for merging partial images of surroundings of the vehicle to form an overall image (Paragraph [0006], Fig. 1, Asai teaches a synthesizing device that is configured to generate a synthesized image.), wherein the device comprises: a first sensor for acquiring a first partial image in a first area of the surroundings, wherein the first area is located laterally behind the vehicle (Paragraphs [0017-18], [0020-21], Fig. 2, Asai teaches a rear left camera (11BL) is an imaging device that is configured to image a rear left area located on the rear of the vehicle on the left of the vehicle.); a second sensor for acquiring a second partial image in a second area of the surroundings, wherein the second area is located behind the vehicle (Paragraphs [0017-19], Fig. 2, Asai teaches the rear camera (11B) is an imaging device that is configured to capture an image of a rear area located on the rear of the vehicle as illustrated in Fig. 2.); a third sensor for determining a distance of an object (Paragraph [0044], Asai teaches the vehicle has a detection apparatus that is configured to detect the distance between the rear vehicle and the vehicle. A radar, a lidar, and the like are examples of the detection apparatus.), at least one section of which is located in one of the areas (Paragraph [0027], Fig. 5(b), Fig. 5(c), Asai teaches a rear vehicle is in a portion of the synthesized image corresponding to the rear camera image (111B).), from the vehicle (Paragraph [0044], Asai teaches the vehicle detection apparatus detects the distance between the rear vehicle and the vehicle.); and a processing device (Paragraph [0022], Asai teaches an electronic control unit which includes processing blocks or processing circuits physically realized in the ECU.) for adapting the second partial image depending on the distance (Paragraphs [0074], [0077], Asai teaches changing a proportion of an area in which the first image is displayed in the synthetized image and a proportion of an area in which the second image is displayed in the synthesized image on the basis of a distance between an object in the synthesized image and the vehicle in generating the synthetized image.); and for merging the partial images to form the overall image (Paragraph [0022], Asai teaches generating a synthesized image by performing an image synthesizing process for synthesizing the rear camera image (111B), the rear left camera image (111BL) and the rear right camera image (111BR). Fig. 3(a) illustrates a synthesized image.).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103(a) are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 18-20, 23, 25-27, 29, and 31 are rejected under 35 U.S.C. 103(a) as being unpatentable over Asai (U.S. Patent Pub. No. 20190075250, hereafter referred to as Asai) in view of Ren et al. (U.S. Patent Pub No. 20230319218, hereafter referred to as Ren).
Regarding Claim 18, Ren teaches the method according to claim 17, wherein a joining line (Paragraph [0037], Asai teaches a border image (112L). The border image (112L) indicates (represents) the border between the rear camera image (111B) and the rear left camera image (111BL) in the synthetized image. In Figs. 7(a) and 7(b), the border image (112L) is a liner-shaped (bar-shaped) image that extends along the border between the rear camera image (111B) and the rear left camera image (111BL).),
PNG
media_image7.png
572
567
media_image7.png
Greyscale
which delimits the first section on the overall image (Paragraph [0037], Asai teaches the border image (112L) indicates (represents) the border between the rear camera image (111B) and the rear left camera image (111BL) in the synthesized image.),
Asai does not explicitly disclose (a joining line) is permanently predetermined based on perspectives and imaging parameters of the partial images.
Ren is in the same field of art of stitching together multiple images to form a composite view of a vehicle’s surroundings. Further, Ren teaches (a joining line) is permanently predetermined (Paragraphs [0071], [0142], [0156], Ren teaches a default seam, such as a horizontal seam (or vertical, diagonal, etc.). For example, when the distance to the closest object is more than a threshold proximity, the default seam placement is used.) based on perspectives and imaging parameters of the partial images (Paragraphs [0174-175], Ren teaches assessing frames of image data (e.g., images) representing overlapping views of an environment, and using corresponding camera parameters to align the frames and create an aligned composite image with regions of overlapping image data. It is determined whether to use default placement for a seam or a dynamic placement based on several factors.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Asai by using a predetermined “default seam” such as a horizontal, vertical, or diagonal seam to stitch the images in cases where the closest object is beyond a threshold distance, that is taught by Ren, to make the invention that avoids placing the seam (“joining line”) through/on top of salient (important) objects in the vehicle’s surroundings; thus, one of ordinary skilled in the art would be motivated to combine the references since the seam (i.e., vertical seam) should be placed to avoid stitching in portions in the images that include or intersect the salient object (vehicle, etc.), thereby reducing distortion of the vehicles in a display or visualization (Ren, Paragraph [0130]).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 19, Asai teaches the method according to claim 16, wherein the overall image comprises a third section (Fig. 3(a), Asai teaches a synthesized image with three sections.),
PNG
media_image8.png
227
443
media_image8.png
Greyscale
Asai does not explicitly disclose which conceals an area located between the vehicle and the object, if a further object is located in this area.
Ren is in the same field of art of stitching together multiple images to form a composite view of a vehicle’s surroundings. Further, Ren teaches which conceals an area located between the vehicle and the object (Paragraph [0156], Ren teaches there may be no candidate seam that avoids objects or salient pixels i.e., the seam intersects or covers objects or salient pixels. See Fig. 4A below, the seam passes through/covers trees, etc. in the combined image.), if a further object is located in this area (Paragraphs [0130], [0004], Fig. 4A, Ren teaches a position of one or more of the seams is adjusted to avoid regions assigned higher weights in a saliency mask. For example, two vehicles on either side of the ego-vehicle may be assigned higher weights in a saliency mask, and a determination is made to use a vertical seam (404) to avoid stitching portions of the underlying images that include or intersect the neighboring vehicles.).
PNG
media_image9.png
598
654
media_image9.png
Greyscale
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Asai by choosing to cover/obstruct the less critical regions in the stitched images that is taught by Ren, to make the invention does not cover the salient/important objects in the combined images; thus, one of ordinary skilled in the art would be motivated to combine the references since placing the seams over less important regions of the combined image and not potentially important objects (vehicles, pedestrians, etc.) enables the driver to safely maneuver the vehicle (Ren, Paragraph [0004]) since stitching images can introduce a variety of artifacts such as misalignments, blurring, object distortions, object disappearance, etc. which could obscure useful visual information and be distracting to the driver/interfere with the safe operation of the vehicle (Ren, Paragraph [0005]).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
[AltContent: arrow][AltContent: arrow]In regards to Claim 20, Asai in view of Ren teaches the method according to claim 19, wherein the third section is taken from the first partial image and shows the vehicle (Paragraph [0030], Figs. 6(b) and 6(c), Asai teaches a situation where the rear vehicle is relatively close to the vehicle. Fig. 6(b) and Fig. 6(c) show the rear vehicle in the “third section.” The Examiner interprets (111BR) to be “the first partial image”. Further, the “third” section (111BR) shows the rear vehicle in the section.).
PNG
media_image10.png
367
416
media_image10.png
Greyscale
In regards to Claim 23, Asai teaches the method according to claim 16.
Asai does not explicitly disclose wherein, of multiple objects in the surroundings of the vehicle, one is selected, the distance of which to the vehicle is least.
Ren is in the same field of art of stitching together multiple images to form a composite view of a vehicle’s surroundings. Further, Ren teaches wherein, of multiple objects in the surroundings of the vehicle (Paragraph [0189], Ren teaches detected objects beyond some threshold distance to the ego-object (e.g., 3m, 7m, 15 m, etc.) may be filtered out.), one is selected, the distance of which to the vehicle is least (Paragraphs [0189], [0066], Ren teaches identifying one or more of the closest detected objects within a variable range threshold. Detected objects outside the applicable range may be filtered out or otherwise ignored from consideration.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Asai by identifying the closest object to the vehicle that is taught by Ren, to make the invention that assigns higher priority to closer and/or salient objects and ignores objects deemed non-salient and/or outside a threshold distance to the ego-vehicle; thus, one of ordinary skilled in the art would be motivated to combine the references since filtering out/ignoring detected objects beyond a threshold distance to the ego-vehicle (e.g., 3 m, 7 m, 15 m, etc.) may free up computational resources (Ren, Paragraph [0148]).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 25, Asai teaches the method according to claim 16.
Asai does not explicitly disclose wherein, of multiple objects in the surroundings of the vehicle, one is selected, which is most sensitive with respect to a collision with the vehicle.
Ren is in the same field of art of stitching together multiple images to form a composite view of a vehicle’s surroundings. Further, Ren teaches wherein, of multiple objects in the surroundings of the vehicle (Paragraphs [0130], [0133], Fig. 4A, Ren teaches parking between two cars. Another example is when driving at low speeds and/or whenever an object (e.g., another vehicle) passes by closely (e.g., within a threshold distance), the driver may need to pay closer attention to the object passing by.), one is selected, which is most sensitive with respect to a collision with the vehicle (Paragraph [0133], Ren teaches a saliency mask may allocate higher weights to a region or pixels corresponding to the passing object (instead of the other vehicle), and a seam may be placed or relocated to avoid cutting through the passing object.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Asai by identifying and assigning a higher importance level to an object deemed more critical (such as a passing object such as a pedestrian) compared to another vehicle that is taught by Ren, to make the invention that ensures the higher weighted object is not “cut through” with the stitching seam; thus, one of ordinary skilled in the art would be motivated to combine the references since objects such as two vehicles on each lateral side of the center ego-vehicle may be assigned higher weights in a saliency mask to ensure the seam is placed to avoid stitching portions of the images that include or intersect the neighboring vehicles (most likely objects for collision), thereby reducing distortion of the vehicles in a display or visualization (Ren, Paragraph [0030]) .
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 26, Asai teaches the method according to claim 16, (Paragraph [0020], Asai teaches the rear left camera (11BL) is an imaging device that is configured to image a rear left area located on the rear of the vehicle and on the left of the vehicle. The rear left area corresponds to an area including at least one portion of an area that is reflected in a door mirror placed at a left door when the left door is in the closed state (in other words, the left door is closed), under the assumption that the door mirror is placed at the left door of the vehicle 1 that is placed at the left side of the vehicle 1. The lens of the rear left camera (11BL) is referred to as an “optical system.”).
Asai does not explicitly disclose wherein a first sensor for providing the first partial image is calibrated based on a feature of the vehicle.
Ren is in the same field of art of stitching together multiple images to form a composite view of a vehicle’s surroundings. Further, Ren teaches wherein a first sensor for providing the first partial image is calibrated based on a feature of the vehicle (Paragraph [0236], Ren teaches the cameras are calibrated against the vehicle rig coordinate system, and relative ego-motion provides a relative vehicle rig transformation matrix (e.g., representing rotation and translation matrices).).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Asai by calibrating the cameras against the vehicle rig coordinate system that is taught by Ren, to make the invention that uses the camera calibration parameters to map each respective camera view into a stitched space to identify any overlapping regions between adjacent cameras; thus, one of ordinary skilled in the art would be motivated to combine the references to align the camera’s coordinate system in the world space and may enable realistic image stitching (Ren, Paragraph [0221]).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 27, Asai teaches the method according to claim 16.
Asai does not explicitly disclose wherein a second sensor for providing the second partial image is calibrated by an orientation sensor.
Ren is in the same field of art of stitching together multiple images to form a composite view of a vehicle’s surroundings. Further, Ren wherein a second sensor for providing the second partial image is calibrated (Paragraphs [0121-122], Ren teaches utilizing various intrinsic and extrinsic parameters of the cameras which may include information such as the relative poses or orientation of these different cameras, such that at least an initial stitching position and orientation may be determined for each image. The camera calibration parameters may be used to map the camera views into a stitched space) by an orientation sensor (Paragraph [0109], Ren teaches determining absolute or relative ego-motion data (e.g., orientation) may be determined using a vehicle speed sensor, gyroscope, accelerator, IMU, and/or others.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Asai by using an orientation sensor such as a gyroscope to calibrate the imaging sensor/camera that is taught by Ren, to make the invention that captures multiple images of an object/environment from different orientations and poses and stitches them together to form a composite image; thus, one of ordinary skilled in the art would be motivated to combine the references since the process of stitching can introduce a variety of artifacts, such as geometric distortions, texture distortions, and color distortions. When images are captured at different poses/orientations, they may not perfectly align and the artifacts can obscure useful visual information and can be distracting to the driver (Ren, Paragraph [0005]). Use of orientation sensors such as gyroscopes enables the image data to be aligned in the global coordinate system such that more accurate stitching may be performed (Ren, Paragraphs [0109], [0121]).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 29, Asai teaches the device according to claim 28.
Asai does not explicitly disclose wherein an orientation sensor is provided for determining an orientation of the second sensor in relation to the surroundings.
Ren is in the same field of art of stitching together multiple images to form a composite view of a vehicle’s surroundings. Further, Ren teaches wherein an orientation sensor is provided (Paragraph [0109], Ren teaches determining absolute or relative ego-motion data (e.g., orientation) may be determined using a vehicle speed sensor, gyroscope, accelerator, IMU, and/or others.) for determining an orientation of the second sensor in relation to the surroundings (Paragraph [0121], Ren teaches utilizing various intrinsic and extrinsic parameters of the cameras which may include information such as the relative poses or orientation of these different cameras, such that at least an initial stitching position and orientation may be determined for each image.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Asai by using an orientation sensor such as a gyroscope to calibrate the imaging sensor/camera that is taught by Ren, to make the invention that captures multiple images of an object/environment from different orientations and poses and stitches them together to form a composite image; thus, one of ordinary skilled in the art would be motivated to combine the references since the process of stitching can introduce a variety of artifacts, such as geometric distortions, texture distortions, and color distortions. When images are captured at different poses/orientations, they may not perfectly align and the artifacts can obscure useful visual information and can be distracting to the driver (Ren, Paragraph [0005]). Use of orientation sensors such as gyroscopes enables the image data to be aligned in the global coordinate system such that more accurate stitching may be performed (Ren, Paragraphs [0109], [0121]).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 31, Asai teaches the vehicle according to claim 30.
Asai does not explicitly disclose wherein the device further comprises an orientation sensor configured to determine an orientation of the second sensor in relation to the surroundings.
Ren is in the same field of art of stitching together multiple images to form a composite view of a vehicle’s surroundings. Further, Ren teaches wherein the device further comprises an orientation sensor (Paragraph [0109], Ren teaches determining absolute or relative ego-motion data (e.g., orientation) may be determined using a vehicle speed sensor, gyroscope, accelerator, IMU, and/or others.) configured to determine an orientation of the second sensor in relation to the surroundings (Paragraph [0121], Ren teaches utilizing various intrinsic and extrinsic parameters of the cameras which may include information such as the relative poses or orientation of these different cameras, such that at least an initial stitching position and orientation may be determined for each image.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Asai by using an orientation sensor such as a gyroscope to calibrate the imaging sensor/camera that is taught by Ren, to make the invention that captures multiple images of an object/environment from different orientations and poses and stitches them together to form a composite image; thus, one of ordinary skilled in the art would be motivated to combine the references since the process of stitching can introduce a variety of artifacts, such as geometric distortions, texture distortions, and color distortions. When images are captured at different poses/orientations, they may not perfectly align and the artifacts can obscure useful visual information and can be distracting to the driver (Ren, Paragraph [0005]). Use of orientation sensors such as gyroscopes enables the image data to be aligned in the global coordinate system such that more accurate stitching may be performed (Ren, Paragraphs [0109], [0121]).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim 24 is rejected under 35 U.S.C. 103(a) as being unpatentable over Asai (U.S. Patent Pub. No. 20190075250, hereafter referred to as Asai) in view of Stein et al. (U.S. Patent Pub. No. 20150210312, hereafter referred to as Stein).
Regarding Claim 24, Asai discloses the method according to claim, wherein, of multiple objects in the surroundings of the vehicle (Fig. 3(a), Asai teaches a synthesized image. The synthetized image shows multiple “objects” in the surroundings of the vehicle. For example, in Fig. 3(a), the image shows a “rear vehicle” as well as other objects such as trees, a guard rail, the roadway, etc.) one is selected (Paragraph [0041], Asai teaches a distance condition between the rear vehicle in the synthesized image and the vehicle as an example of a predetermined condition.),
Asai does not explicitly disclose an expected time of which until reaching the vehicle is least.
Stein is in the same field of art of detecting objects and determining time of collision with a vehicle. Further, Stein teaches an expected time of which until reaching the vehicle is least (Paragraph [0149], Stein teaches identifying the amount of time to collision (e.g., the amount of time for vehicle to reach the object).).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Asai by identifying the object in the vehicle’s surroundings with the shortest expected time of collision that is taught by Stein, to make the invention that recognizes objects in a vehicle’s environment and takes appropriate and timely action to avoid collisions; thus, one of ordinary skilled in the art would be motivated to combine the references since identifying critical and/or potential objects that could cause collisions is crucial for control of autonomous driving systems (i.e., change lanes, turn at intersections, change roadways) in order to navigate safely and accurately (Stein, Paragraph [0005]).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Pertinent Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Pan et al. (U.S. Patent Pub. No. 20190253625 A1) teaches a rear-stitched panorama system and method for computing an optimal left seam and an optimal right seam based on a transformed disparity map and stitching the transformed images together based on respective optimal seams to generate a panorama of the rear view of the vehicle.
Schmidt et al. (U.S. Patent Pub. No. 20170116710 A1) teaches a method for merging partial images to form an image of a contiguous surrounding region of a mode of transport (vehicle). The first and second partial images of the surrounding region are joined by a first joining line.
Okamoto et al. (U.S. Patent Pub. No. 20030085999 A1) teaches a vehicle surroundings monitoring system for synthesizing and displaying images captured by a plurality of cameras to provide images that, near the border between partial images, are more accurate and readily understood by the user.
Kosugi et al. (U.S. Patent Pub. No. 20180086271 A1) teaches capturing, outside a vehicle, plural exterior capture images in which portions of the images overlap with each other. The plural vehicle exterior images are combined and displayed. See Fig. 6 below, example of a combined image.
PNG
media_image11.png
254
487
media_image11.png
Greyscale
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SYDNEY L BLACKSTEN whose telephone number is (571)272-7120. The examiner can normally be reached 8:30am-4:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Oneal Mistry can be reached at 313-446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SYDNEY L BLACKSTEN/Examiner, Art Unit 2674
/ONEAL R MISTRY/Supervisory Patent Examiner, Art Unit 2674
Prosecution Insights