DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 1, 8 and 15, and therefore all claims depending therefrom, are objected to because of the following informalities: claims 1, 8 and 15 all recite “obtain[ing] a first conversion image” but later refer back to this limitation as “the first converted image,” when instead it should recite “the first conversion image.” Appropriate correction is required.
Claims 4, 11 and 18, and therefore claims 5, 12–13 and 19 which respectively depend therefrom, are objected to because of the following informalities: claims 4, 11 and 18 recite in part “a preset parameter matric” where “matric” should instead be “matrix.” Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1–3, 8–10, and 15–17 are rejected under 35 U.S.C. 103 as being unpatentable over Kim et al., KR 20190075034 (with reference to the provided machine English language translation, herein “Kim”).
Regarding claims 1, 8 and 15, where deficiencies of Kim are noted in square brackets [], and substantive differences between the claims are noted in curly brackets {}, Kim teaches {an image processing method using a vehicle-mounted device, the method comprising: - claim 1 / a vehicle-mounted device comprising: a storage device; at least one processor; and the storage device storing one or more programs, which when executed by the at least one processor, cause the at least one processor to: - claim 8 / a non-transitory storage medium having instructions stored thereon, when the instructions are executed by a processor of a vehicle-mounted device, the processor is caused to perform an image processing method, wherein the method comprises: - claim 15} (Kim Abstract, a device and method for generating a vehicle image, including image generation cores performing image processing, and where the device includes a plurality of wide-angle cameras attached to a vehicle, and page 30, claim 19, teaching a computer program stored on a computer readable medium for executing a method for generating a vehicle image when executed on a computer or a processor)
acquiring a plurality of images of an external vehicle out of a current vehicle, the plurality of images comprising a first image of a front part of the external vehicle acquired by using a first camera device and a second image of a rear part of the external vehicle acquired by using a second camera device, a first view angle of the first camera device being different from a second view angle of the second camera device (Kim fig. 2, reproduced below for convenience, and fig. 1, page 4, multiple camera unit including for example 5 wide-angle cameras, with at least four attached to the vehicle for outside images, where C1 (first camera device) has a field of view including a front part, C4 (second camera device) has a field of view of the rear part, also one omni camera provided on the top roof of the vehicle which would also capture a front and back part of the vehicle, and where page 5, first and third paragraph teach that an image from each of the wide-angle cameras is received by the vehicle image generating apparatus);
PNG
media_image1.png
514
416
media_image1.png
Greyscale
determining a target visual plane based on a first shooting field of view of the first camera device and a second shooting field of view of the second camera device (Kim page 7, images generated respective to each wide-angle camera are synthesized into an output image in accordance with a configuration, thus the field of view represented by the output image realizing a configured target visual plane, for example, the images respective to individual wide-angle cameras corresponding to image cores 701, 702 and 703, are synthesized into an output image) and obtaining a plurality of spatial coordinates of a plurality of vertices of the target visual plane (Kim page 8, first paragraph, a camera coordinate system represents the coordinates in each image taken by one of the wide-angle cameras, and a world coordinate system discussed further on page 9 also called a scene coordinate system (target visual plane) to which origin points of a virtual camera coordinate system {n} (the n corresponding to a particular one of n wide-view cameras) for projection onto the scene/world coordinate system);
projecting each of the plurality of the spatial coordinates into a first coordinate system of the first image and obtaining a first projection coordinate corresponding to each of the plurality of the spatial coordinates, and projecting each of the plurality of the spatial coordinates into a second coordinate system of the second image and obtaining a second projection coordinate corresponding to each of the plurality of the spatial coordinates (Kim page 9, starting with the paragraph beginning with the text “FIG. 6 shows..”, teaches transforming (projecting) each unit vector {1,0,0}, {0,1,0}, and {0,0,1} (three are listed) on the virtual camera coordinate system {n}into coordinates on the scene coordinate system);
generating a first transformation matrix corresponding to the first image according to the first projection coordinate corresponding to each of the plurality of the spatial coordinates and the second projection coordinate corresponding to each of the plurality of the spatial coordinates and/or generating a second transformation matrix corresponding to the second image according to the first projection coordinate corresponding to each of the plurality of the spatial coordinates and the second projection coordinate corresponding to each of the plurality of the spatial coordinates (Kim page 9 of machine English translation, corresponding to the formulas shown in ¶¶91–95 of the original document, a transformation matrix
T
n
0
which is defined by a 3-D translational matrix and rotation matrices in the equation in ¶91 of the original document:
PNG
media_image2.png
48
545
media_image2.png
Greyscale
, is transformed by the transform module using a point J of the image shown at the viewpoint of the virtual camera using the coordinates for that virtual camera, where each virtual camera has its own virtual camera coordinate system {n} (according to the first/second projection coordinate));
obtaining a first conversion image corresponding to the first image by converting the first image according to the first transformation matrix, and/or obtaining a second conversion image corresponding to the second image by converting the second image according to a second conversion matrix (given that the claim recites and/or thus having a broadest reasonable interpretation requiring one or the other of the listed limitations, Kim page 13, 2nd paragraph, the pixel value in the input image I (first image) is converted to a point J from the viewpoint of the virtual camera using the transformation matrix, and then coordinate calculation module calculates a point on the scene coordinate system to generate an image viewed at one or more arbitrary viewpoints (first conversion image)); and
generating an image of a whole body of the external vehicle based on the first converted image and the second image and/or generating the image of the whole body of the external vehicle based on the second converted image and the first image (Kim bottom of page 15–16, fig. 18, reproduced for convenience below, the output image displays an image that is simultaneously viewed from multiple viewpoints, such as 18a showing the synthesis of a rear wide angle camera C4 and a top view image, and where the image includes the whole body of the external vehicle
PNG
media_image3.png
841
681
media_image3.png
Greyscale
).
While Kim teaches determining an image of a whole body of an external vehicle using images obtained from various vehicle mounted cameras, and applying projections and transformation matrices, Kim does not teach applying the transformation matrix in the same embodiment as fig. 18, which is given as an example of one combination of input images, and the resulting output image. Nonetheless, it would have been obvious to a person having ordinary skill in the art (herein “PHOSITA”) to have modified the specific images projected, transformed and synthesized as disclosed in the embodiment on pages 4 and 9–13 with the specific camera synthesis for an output image as taught on page 16, fig. 18, at least because Kim teaches a finite set of cameras (C1–C5) and the specific transformations/projections applied to each camera input image, and therefore, various combinations of camera images to be combined together is a finite/closed set, accordingly, such a combination would have been obvious to try with predictable solutions, and with a reasonable expectation of success. See MPEP §2143(I)(E).
Regarding claims 2, 9 and 16, Kim teaches wherein the target visual plane is in an overlapping space area between the first shooting field of view and the second shooting field of view (Kim figs. 2 and 18, shown above in the rationale for claim 1, illustrating that the field of view for rear camera C4 overlaps with other camera fields of view, and in fig. 18 that this overlap is reflected in the output image (target visual plane)).
Regarding claims 3, 10 and 17, Kim teaches wherein projecting each of the plurality of the spatial coordinates into the first coordinate system of the first image and obtaining the first projection coordinate corresponding to each of the plurality of the spatial coordinates comprises: projecting each of the plurality of the spatial coordinates into the first coordinate system of the first image by using an orthogonal projection method or a perspective projection method (Kim page 7, image generating cores respective to each wide angle cameras and generating an image from an input image from the wide angle cameras, each image generating core including a virtual camera modeling unit which processes images according to a perspective projection), and obtaining the first projection coordinate corresponding to each of the plurality of the spatial coordinates (Kim bottom of page 7 - page 8, virtual cameras are modeled to output a plurality of images for one input image such that various projections such as a perspective or orthogonal projection are output, thus the input image being entirely projected (each pixel coordinate) and thus corresponding to each of the plurality of spatial coordinates).
Claims 6 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Kim, further in view of Sun et al., "3D Estimation of Single Image based on Homography Transformation," 2022 IEEE 6th Information Technology and Mechatronics Engineering Conference (ITOEC), Chongqing, China, 2022, pp. 727-732, doi: 10.1109/ITOEC53115.2022.9734605 (herein “Sun”).
Regarding claims 6 and 20, Kim teaches wherein obtaining a first conversion image corresponding to the first image by converting the first image according to the first transformation matrix, as set forth above in the independent claims. Kim does not explicitly teach where Sun teaches determining a first target coordinate corresponding to each first initial coordinate by multiplying a homogeneous coordinate vector corresponding to a first initial value of each pixel of the first image by the first transformation matrix (Sun page 728, equation (1), u0 and v0 are central coordinates of the imaging plane (a first initial value of each pixel of the first image) which are part of a column vector including a 1 (homogenous coordinate vector) and where the column vector is multiplied by a transformation matrix
R
T
0
1
where R and T are translation vectors (transformation matrix)); and
replacing a pixel value of a pixel of each first target coordinate of the first image with a pixel value of a pixel of the first initial coordinate and obtaining the first converted image (Sun page 729, fig. 3, a 2D plane image is converted to a two-dimensional perspective image (each first target coordinate of the first image), then using a homography matrix, is transformed (replacing pixel values) to corresponding spatial coordinates (first converted image)).
Therefore, taking the teachings of Kim and Sun together as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified the method and apparatus for generating a vehicle image disclosed by Kim to include the homography transformation disclosed specifically above by Sun at least because doing so would provide a way to obtain camera parameters with lower computational complexity and accuracy, thus leading to generating transformed images more efficiently. See Sun page 727, bottom right column.
Claims 7 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Kim, further in view of Wu et al., US Patent No. 10,609,862 B2 (herein “Wu”).
Regarding claims 7 and 14, while Kim teaches the claimed generating an image of a whole body of the external vehicle based on the first converted image and the second image, and determining overlapping areas in the images which is used for the obtaining the image of the whole body of the vehicle image, as discussed above in the rejection rationale for the independent claims, including the fig. 18 from Kim illustrating the overlap between the front and rear images. However, Kim does not explicitly teach where Wu teaches cropping a first overlapping area between the first converted image and the second image from the first converted image and obtaining a first cropped image; and splicing the first cropped image and the second image (Wu col. 38, ll. 29–53, image splicing unit 400 including converting module 410 and splicing module 420, where the converting module converts image information collected by two cameras into first image data and second image data (first converted image and the second image) and overlaps the first image data and the second image data through logic operation and clips (crops) the overlapped section, then produces a spliced image which is a seamless overall image spliced by the two cameras).
Therefore, taking the teachings of Kim and Wu together as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified the method and apparatus for generating a vehicle image disclosed by Kim to include the cropping and splicing disclosed specifically above by Wu at least because doing so would prevent image distortion in an environment-aware vehicle which uses images to determine environment interaction. See Wu col. 6, ll. 56–67.
Allowable Subject Matter
Claims 4, 11 and 18, and therefore claims 5, 12–13 and 19 which respectively depend therefrom are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Specifically, the closest cited art of record includes Kim as applied above to the independent claims and further, Sun et al., "3D Estimation of Single Image based on Homography Transformation," 2022 IEEE 6th Information Technology and Mechatronics Engineering Conference (ITOEC), Chongqing, China, 2022, pp. 727-732, doi: 10.1109/ITOEC53115.2022.9734605 (herein “Sun”).
Kim, as set forth in the rejection for the independent claims, teaches the claimed wherein generating the first transformation matrix corresponding to the first image according to the first projection coordinate corresponding to each of the plurality of the spatial coordinates and the second projection coordinate corresponding to each of the plurality of the spatial coordinates, Kim does not specifically detail the generating step as including obtaining a multiplication vector as further limited in claims 4, 11 and 18.
Sun teaches obtaining a multiplication vector of each first projection coordinate by multiplying a first homogeneous coordinate vector of each first projection coordinate with a preset parameter matric - on page 728, section II, as shown in equation (1), the first matrix on the right of the equation is an internal parameter matrix of the camera (preset parameter matrix) which is shown to be multiplied by a world coordinate vector corresponding to homogenous coordinates via a homography matrix H. While Sun also teaches generating the first transformation matrix based on the multiplication vector, as set forth in Sun in equation (2) on page 728, Sun does not teach that the transformation matrix is based on “a second homogenous coordinate vector of the second projection coordinate. Sun also teaches some of the limitations of claim 5 regarding the “establishing an equal relationship … solving the plurality of parameter equations” interpreted to mean that a system of equations is determined and solved to find the parameters in the preset parameter matrix. Here, Sun teaches on the bottom of page 728–page 729 that a system of 8 linear equations is established to obtain the homography matrix H, which is then used to solve for the camera parameters in the preset parameter matrix. However, Sun does not teach using multiple input images from which first and second projection coordinates are respectively obtained, and thus does not teach any of the specific relationships of the homography matrix or parameter matrix in association with a second input image, much less that the first image is of a front part of the external vehicle, and a second image is of a rear part of the external vehicle, as recited in claim 1.
Accordingly, in view of the above, none of the cited art, whether considered alone or in a combination obvious to a PHOSITA teaches or suggests the limitations of claims 4, 11 and 18, and therefore claims 5, 12–13 and 19 which respectively depend therefrom.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Kido et al., US Patent Application Publication No. US 20260196055 A1, directed towards an external world recognition device including image processing upon different camera images acquired from a vehicle camera.
Lee et al., US Patent Application Publication No. US 20240394914 A1, directed towards a vehicle driving assistance system that processes images from vehicle cameras.
Benhimane et al., US Patent Application Publication No. US 20160321811 A1, directed towards a self moving robot that performs image processing on an image taken by a camera on the robot.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHELLE M KOETH whose telephone number is (571)272-5908. The examiner can normally be reached Monday-Thursday, 09:00-17:00, Friday 09:00-13:00, EDT/EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vincent Rudolph can be reached at 571-272-8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
MICHELLE M. KOETH
Primary Examiner
Art Unit 2671
/MICHELLE M KOETH/Primary Examiner, Art Unit 2671
