Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This office action is in response to application filed 02/19/2025 in which claims 23-44 are pending.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 02/19/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 23-44 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-22 of U.S. Patent No. US 12261991 B2. Although the claims at issue are not identical, they are not patentably distinct from each other because instant claim 23 is anticipated by the conflicting patented claim 1 as shown in the table below. The difference between the instant examined claim and the conflicting patented claim is that the conflicting patented claim is narrower in scope and falls within the scope of the examined claim.
Patent No.: US 12261991 B2
Instant application: 19/057332
1. A system for generating three-dimensional information of a real-life scene comprising:
a camera array made up of a plurality of cameras, the camera array configured to be positioned to view the real-life scene; the plurality of cameras comprising: at least a first camera that is configured to collect spectral data from a first field of view of the real-life scene; at least a second camera that is configured to collect spectral data from a second field of view of the real-life scene;
at least a third camera that is configured to collect spectral data from a third field of view of the real-life scene; and at least a fourth camera that is configured to collect spectral data from a fourth field of view of the real-life scene; the first camera configured to generate at least a first image from the first field of view; the second camera configured to generate at least a second image from the second field of view; the third camera configured to generate at least a third image from the third field of view; and the fourth camera configured to generate at least a fourth image from the fourth field of view; the camera array being configured to transmit the first image, the second image, the third image, and the fourth image to a processing engine; and the processing engine is configured to: obtain image data from the plurality of cameras including the first camera, the second camera, the third camera, and the fourth camera;
generate a representation of a 3D volume representative at least in part of a portion of the real-life scene by extracting associated pixel data that is selected from at least a substantial portion of the pixel data based on the projection of the 3D volume in at least one of the camera images; and using the associated pixel data to determine the likelihood that the 3D volume contains a physical surface.
23. A method for generating three-dimensional information of a real-life scene, comprising: (a) positioning a camera array comprising a plurality of optical cameras that are configured, at least in part, to view a real-life scene; (b) capturing, via a
first optical camera, spectral data from a first field of view to produce a first image;
(c) capturing, via a second optical camera, spectral data from a second field of view to produce a second image; (d) capturing, via a third optical camera, spectral data from a third field of view to produce a third image; (e) capturing, via a fourth optical camera, spectral data from a fourth field of view to produce a fourth image;
(f) transmitting the first, second, third, and fourth images to a processing engine;
(g) obtaining, by the processing engine, two-dimensional (2D) image data from the plurality of optical cameras; (h) extracting pixel data associated with the 2D image data, the pixel data being selected from at least a substantial portion of the pixel data based on a projection of a three-dimensional (3D) volume in at least one of the camera images; and (i) using the extracted pixel data to determine a likelihood that the 3D volume contains a physical surface within the real-life scene.
2. The system of example 1,
wherein one or more of the 3D volumes are one or more 3D neighborhoods.
24. The method of claim 23, further comprising generating a plurality of 3D volumes representative of portions of the real-life scene, wherein one or more of the 3D volumes define respective 3D neighborhoods.
3. The system of examples 1 or 2, wherein the 3D volume is a 3D neighborhood.
25. The method of claim 23, wherein the 3D volume comprises a 3D neighborhood.
4. The system of claim 1, wherein the associated pixel data includes 3D volume's pixel-level spectral data.
26. The method of claim 23, wherein the pixel data include pixel-level spectral data corresponding to the 3D volume.
5. The system of claim 1, wherein the associated pixel data includes optical flow information.
27. The method of claim 23, wherein the pixel data further include optical flow information derived from temporal sequences of the 2D image data.
6. The system of claim 1, wherein the processing engine is configured to use the pixel data to determine an estimated velocity for the physical surface.
28. The method of claim 23, further comprising using the pixel data to determine an estimated velocity of a physical surface represented within the 3D volume.
7. The system of claim 1, wherein the pixel data is a subset of pixel data that comprise pixel-level data of a first 3D volume.
29. The method of claim 23, wherein the pixel data comprise a subset of pixel-level data corresponding to a first 3D volume of the real-life scene.
8. The system of claim 1, wherein already calculated likelihood calculations within a cost matrix is used at least in part for a subsequent optimization calculation along at least one additional line across one or more selected images.
30. The method of claim 23, further comprising using already calculated likelihood calculations within a cost matrix at least in part for a subsequent optimization calculation along at least one additional line across one or more selected images.
9. The system of claim 8, wherein the optimization calculation is repeated for a plurality of lines derived from selected image pairs.
31. The method of claim 30, wherein the optimization calculation is repeated for a plurality of lines derived from selected image pairs.
10. The system of claim 9, wherein a portion of the plurality of lines is selected from epipolar lines
32. The method of claim 31, wherein a portion of the plurality of lines is selected from epipolar lines.
11. The system of claim 1, wherein likelihood calculations within a portion of 3D volumes produce numeric results that are independent of an order in which at least a portion of the data from intersection points derived from a set of image pairs is processed.
33. The method of claim 23, wherein likelihood calculations within a portion of 3D volumes produce numeric results that are independent of an order in which at least a portion of the data from intersection points derived from a set of image pairs is processed.
12. The system of claim 11, wherein the data associated with intersection points that are input into the likelihood calculations for one or more 3D neighborhoods that are associated with 3D scene information substantially aligned on at least one reference surface is calculated from the pixel data extracted from at least two rectified images separated by a pixel offset.
34. The method of claim 33, further comprising calculating data associated with intersection points that are input into the likelihood calculations for one or more 3D neighborhoods associated with 3D scene information substantially aligned on at least one reference surface from pixel data extracted from at least two rectified images separated by a pixel offset.
13. The system of claim 12, wherein the pixel offset is constant.
35. The method of claim 34, wherein the pixel offset is constant.
14. The system of claim 1, wherein data collected within at least a portion of a set of 3D volumes is used to determine the likelihood that the physical surface is at least partially contained within the set of 3D volumes.
36. The method of claim 23, further comprising using data collected within at least a portion of a set of 3D volumes to determine the likelihood that a physical surface is at least partially contained within the set of 3D volumes.
15. The system of claim 14, wherein the set of 3D volumes is representative of a line passing through the real-life scene.
37. The method of claim 36, wherein the set of 3D volumes is representative of a line passing through the real-life scene.
16. A system configured to determine a presence of one or more surfaces in a 3D scene by processing multiple 3D volumes using the system of claim 1 to determine the likelihood of a surface within at least one 3D volume, and collect at least a portion of these results into an accumulated dataset.
38. The method of claim 23, further comprising: determining a presence of one or more surfaces in a 3D scene by processing multiple 3D volumes to determine the likelihood of a surface within at least one 3D volume; and collecting at least a portion of these results into an accumulated dataset.
17. The system of claim 16, wherein the multiple 3D volumes do not overlap.
39. The method of claim 38, wherein the multiple 3D volumes do not overlap.
18. The system of claim 16, wherein the multiple 3D volumes do at least partially overlap.
40. The method of claim 38, wherein the multiple 3D volumes do at least partially overlap.
19. The system of claim 16, wherein the multiple 3D volumes do not in aggregate cover the real-life scene in its entirety.
41. The method of claim 38, wherein the multiple 3D volumes do not in aggregate cover the real-life scene in its entirety.
20. The system of claim 16, wherein at least a portion of the multiple 3D volumes are substantially aligned along at least one line projecting into the real-life scene from at least one 3D point relative to one or more of the plurality of cameras.
42. The method of claim 38, wherein at least a portion of the multiple 3D volumes are substantially aligned along at least one line projecting into the real-life scene from at least one 3D point relative to one or more of the plurality of cameras.
21. The system of claim 16, wherein at least a portion of the multiple 3D volumes are substantially aligned along a plurality of lines projecting into the real-life scene from at least one 3D point relative to one or more of the plurality of cameras.
43. The method of claim 38, wherein at least a portion of the multiple 3D volumes are substantially aligned along a plurality of lines projecting into the real-life scene from at least one 3D point relative to one or more of the plurality of cameras.
22. The system of claim 16, wherein one or more of the 3D volumes are one or more 3D neighborhoods, wherein data collected within at least a portion of the multiple 3D volumes is used to determine the likelihood that the physical surface is at least partially contained within the one or more 3D neighborhoods.
44. The method of claim 38, further comprising; determining the likelihood that a physical surface is at least partially contained within one or more 3D neighborhoods, wherein one or more of the 3D volumes are the one or more 3D neighborhoods.
Claim Rejections - 35 USC § 103
6. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
7. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
8. The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
9. Claims 23-26, 29, 36-37 are rejected under 35 U.S.C. 103 as being unpatentable over Colmenares (US 10,949,986 B1) in view of Tay et al. (US 2019/0311546 A1) (IDS provided 02/19/2025).
Regarding claim 23, Colmenares discloses a method for generating three-dimensional information of a real-life scene (Abstract, Col 5 lines 7-32 the depth information and calibration data can be used/combined with the images from the cameras 112 to synthesize the output image as a 3D (or stereoscopic 2D) rendering of the scene 108), comprising: (a) positioning a camera array comprising a plurality of optical cameras that are configured, at least in part, to view a real-life scene the (col 3 lines 47- line 60 teaches camera array 110 includes a plurality of cameras 112 (identified individually as cameras 112a-112n) that are each configured to capture images of a scene 108 from a different perspective; col. 4 line 41-44: one or more cameras 119 (e.g., a pair of the cameras 119) configured to detect the structured light projected onto the scene 108 by the projector 118 to estimate a depth of a surface in the scene 108; col. 10 line 15-21: FIG. 6... a patient 665 is positioned at least partially within the scene 108 below the camera array 110); (b) capturing, via a first optical camera, spectral data from a first field of view to produce a first image; (c) capturing, via a second optical camera, spectral data from a second field of view to produce a second image; (d) capturing, via a third optical camera, spectral data from a third field of view to produce a third image; (e) capturing, via a fourth optical camera, spectral data from a fourth field of view to produce a fourth image (col 4 lines 27- 30 teaches cameras 112 can have different spectral sensitives (e.g., infrared vs. visible wavelength). Col 9 lines 28-39 teaches a field of view of each of the cameras 112 can at least partially overlap the field of view of one or more other ones of the cameras 112, In some embodiments, the field of view of individual ones of the cameras 112 can be selected (e.g., via selection of an attached lens) to vary the effective spatial resolution of the cameras 112. For example, the field of view of the cameras 112 can be made smaller to increase their effective spatial resolution and the resulting accuracy of the system 100. col 10 lines 44-45 teaches the cameras 112 each have a field of view 664 of the scene 108); (f) transmitting the first, second, third, and fourth images to a processing engine (col. 7 line 7- 13: The processing device 102 can process received inputs from the input controller 106 and process the captured images from the camera array 110 to generate output images; see fig. 1; col. 4 line 56-59: the processing device 102 includes an image processing device 107 (e.g., an image processor, an image processing module, an image processing unit, etc.)col lines 60-65 teaches the image processing device 107 is configured to (i) receive images (e.g., light-field images, light field image data, etc.) captured by the cameras 112 of the camera array 110 and (ii) process the images to synthesize an output image corresponding to a selected virtual camera perspective); (g) obtaining, by the processing engine, two-dimensional (2D) image data from the plurality of optical cameras (Col 14 lines 27-30 teaches the 3D volume is then mapped/projected to the 2D images from the cameras 112); projection of a three-dimensional (3D) volume in at least one of the camera images (Fig. 1 items 110 & 102 , 107, Col 4 lines 56-67, col 14 lines 27-30 teaches the 3D volume is then mapped/projected to the 2D images from the cameras 112, col 15 lines 14-25 teaches where the system 100 processes image data from only one of the cameras 112 (e.g., where a ROI is determined in an image from only one of the cameras 112), the 3D position of the tip 103 can be determined by projecting the position of the tip 103 in the 2D image from the camera 112 into a 3D line using the calibration of the camera 112 (block 772)); determine a likelihood that the 3D volume contains a physical surface within the real-life scene (Figs. 7, 9B; Col 2 lines 21-31, Col 5 lines 20-67, col 11 lines 41-67, col 12 lines 1-14, col 14 lines 19-43, col 14 lines 19-30 teaches at block 777, the method 770 includes defining a region of interest (ROI) in an image from one or more of the cameras 112 based on the estimated position of the tip 103 determined at block 776. As shown in FIG. 9B, for example, the system 100 can define a ROI 986 around the estimated 3D tip position 103′. More specifically, the estimated 3D tip position 103′ is used to initialize a 3D volume (e.g., a cube, sphere, rectangular prism, etc.) with a determined critical dimension (e.g., radius, area, diameter, etc.). The 3D volume is then mapped/projected to the 2D images from the cameras 112. ).
Colmenares does not explicitly disclose (h) extracting pixel data associated with the 2D image data; the pixel data being selected from at least a substantial portion of the pixel data based on a projection of a three-dimensional (3D) volume in at least one of the camera images; and (i) using the extracted pixel data to determine a likelihood that the 3D volume contains a physical surface within the real-life scene. However Tay discloses
extracting pixel data associated with the 2D image data (para[0100] teaches 2D color pixels depicting the dynamic object in the concurrent 2D color image)), the pixel data being selected from at least a substantial portion of the pixel data based on a projection of a three-dimensional (3D) volume in at least one of the camera images (para[0100] teaches augment a cluster of points representing a dynamic object in the current 3D point cloud with synthetic color pixels generated from 2D color pixels depicting the dynamic object in the concurrent 2D color image, thereby increasing density of points depicting this dynamic object in the augmented 3D point cloud; para[0123] teaches isolate a cluster of color pixels in the concurrent color image that correspond to a cluster of points representing this particular object in the 3D point cloud; augment this cluster of points with synthetic 3D color points generated based on this cluster of color pixels); and (i) using the extracted pixel data to determine a likelihood that the 3D volume contains a physical surface within the real-life scene paragraph (Para[0060] teaches porting 2D color pixels from the 2D color image onto a 2D (i.e., planar) manifold that (approximately) intersects this cluster of points in the 3D point cloud, thereby preserving 2D color pixel data within a 2D domain; paragraph [0075]: calculate planes that approximately intersect clusters of points that represent faces of other dynamic objects in the field). It would have been obvious to one having ordinary skill in the art before the effective filing date of the invention to use the method of 3D volume is mapped/projected to the 2D images from the cameras of Colmenares with the method of to display color pixels as taught of Tay in order to provide a system to render 2D and 3D data within a 3D image.
Regarding claim 24, Tay further discloses the method of claim 23, further comprising generating a plurality of 3D volumes representative of portions of the real-life scene, wherein one or more of the 3D volumes define respective 3D neighborhoods (para [0100] teaches augment a cluster of points representing a dynamic object in the current 3D point cloud with synthetic color pixels generated from 2D color pixels depicting the dynamic object in the concurrent 2D color image, thereby increasing density of points depicting this dynamic object in the augmented 3D point cloud; para [0123] teaches isolate a cluster of color pixels in the concurrent color image that correspond to a cluster of points representing this particular object in the 3D point cloud; augment this cluster of points with synthetic 3D color points generated based on this cluster of color pixels). Motivation to combine as indicted in claim 23.
Regarding claim 25, Tay discloses the method of claim 23, wherein the 3D volume comprises a 3D neighborhood (para[0060] teaches porting 2D color pixels from the 2D color image onto a 2D (i.e., planar) manifold that (approximately) intersects this cluster of points in the 3D point cloud, thereby preserving 2D color pixel data within a 2D domain; para[0075] teaches calculate planes that approximately intersect clusters of points that represent faces of other dynamic objects in the field). Motivation to combine as indicted in claim 23.
Regarding claim 26, Colmenares the method of claim 23, wherein the pixel data include pixel-level spectral data corresponding to the 3D volume (col. 4, line 31-33 teaches the trackers 114 and the cameras 112 can have different spectral sensitives (e.g., infrared vs. visible wavelength).
Regarding claim 29, Colmenares discloses the method of claim 23, wherein the pixel data comprise a subset of pixel-level data corresponding to a first 3D volume of the real-life scene (col. 10, lines 15-21 & FIG. 6 teaches a patient 665 is positioned at least partially within the scene 108 below the camera array 110. The surgical application can be a procedure to be carried out on a portion of interest of the patient; col. 2 line 36-40: The imaging system can then project the estimated 3D position into two- dimensional (2D) images from the cameras, and define a region of interest (ROI) in each of the images based on the projected position of the tool tip; col. 10 line 44-col 11 line 2: the cameras 112 each have a field of view 664 of the scene 108, and the trackers 114 each have a field of view 666 of the scene 108.. the fields of view 666 of the trackers 114 can at least partially overlap one another (and/or the fields of view 664 of the cameras 112) to together define a tracking volume.. the regions of overlap are tiled such that the resulting imaging volume covered by all the cameras 112 has a selected volume that exists as a subset of the volume covered by the trackers 114).
Regarding claim 36, Tay discloses the method of claim 23, further comprising using data collected within at least a portion of a set of 3D volumes to determine the likelihood that a physical surface is at least partially contained within the set of 3D volumes (para [0060] teaches porting 2D color pixels from the 2D color image onto a 2D (i.e., planar) manifold that (approximately) intersects this cluster of points in the 3D point cloud, thereby preserving 2D color pixel data within a 2D domain; para [0075] teaches calculate planes that approximately intersect clusters of points that represent faces of other dynamic objects in the field). Motivation to combine as indicated in claim 23.
Regarding claim 37, Tay discloses the method of claim 36, wherein the set of 3D volumes is representative of a line passing through the real-life scene (Para[0013] teaches for each camera on the autonomous vehicle, the system can define an image plane that is perpendicular to the ground plane. the system can set this offset distance such that an image plane defined for a 2D image feed recorded by a laterally-facing camera (e.g., a left-forward camera, a right-forward camera) on the autonomous vehicle is approximately centered on an adjacent road lane; see fig. 2)). Motivation to combine as indicted in claim 23.
10. Claims 27-28, 30 are rejected under 35 U.S.C. 103 as being unpatentable over Colmenares (US 10,949,986 B1) in view of Tay et al. (US 2019/0311546 A1) (IDS provided 02/19/2025) and Rowell et al. (US 2020/0342652 A1).
Regarding claim 27, Colmenares in view of Tay discloses the method of claim 23, Colmenares in view of Tay does not explicitly disclose wherein the pixel data further include optical flow information derived from temporal sequences of the 2D image data. However Rowell discloses wherein the pixel data further include optical flow information derived from temporal sequences of the 2D image data (para [0086] teaches optical flow information may be represented in an object direction map including direction vectors for every pixel in an image). It would have been obvious to one having ordinary skill in the art before the effective filing date of the invention to use the method of 3D volume mapped/projected to the 2D images from the cameras of augmenting 3D depth map data with 2D color image data of Colmenares in view of Tay with the method to include optical flow information as taught of Rowell in order to provide a sufficient data for performing tasks without the expense of complex hardware.
Regarding claim 28, Colmenares in view of Tay discloses the method of claim 23, Colmenares in view of Tay does not explicitly disclose, further comprising using the pixel data to determine an estimated velocity of a physical surface represented within the 3D volume. However Rowell discloses, further comprising using the pixel data to determine an estimated velocity of a physical surface represented within the 3D volume (para [0086] teaches the AIDRU 106 may also express optical flow information as an object velocity map including velocity direction vectors for every pixel in an image. Optical flow maps including object flow vectors assembled by combining direction and velocity vectors may also be generated by the AIDRU 106). It would have been obvious to one having ordinary skill in the art before the effective filing date of the invention to use the method of 3D volume mapped/projected to the 2D images from the cameras of augmenting 3D depth map data with 2D color image data of Colmenares in view of Tay with the method of to include optical flow information as taught of Rowell in order to provide a system for sufficient data for performing tasks without the expense of complex hardware.
Regarding claim 30, Colmenares in view of Tay discloses the method of claim 23, Colmenares in view of Tay does not explicitly disclose, further comprising using already calculated likelihood calculations within a cost matrix at least in part for a subsequent optimization calculation along at least one additional line across one or more selected images. However Rowell discloses further comprising using already calculated likelihood calculations within a cost matrix at least in part for a subsequent optimization calculation along at least one additional line across one or more selected images (Para [0150] teaches rectification matrices 636 are generated using the rotation matrix 634 and translation vector 635, wherein the rotation matrix 634 describes a rotational correction aligning images captured by a virtual stereo camera device so that the image planes of the left and right image channels are on the same plane). It would have been obvious to one having ordinary skill in the art before the effective filing date of the invention to use the method of 3D volume mapped/projected to the 2D images from the cameras of augmenting 3D depth map data with 2D color image data of Colmenares in view of Tay with the method of to include optical flow information as taught of Rowell in order to provide a system for sufficient data for performing tasks without the expense of complex hardware.
11. Claims 31-32 are rejected under 35 U.S.C. 103 as being unpatentable over Colmenares (US 10,949,986 B1) in view of Tay et al. (US 2019/0311546 A1) (IDS provided 02/19/2025) and Rowell et al. (US 2020/0342652 A1) in further view of Aratani (US 2020/0104969 A1).
Regarding claim 31, Colmenares in view of Tay and Rowell discloses the method of claim 30, Colmenares in view of Tay and Rowell does not explicitly disclose wherein the optimization calculation is repeated for a plurality of lines derived from selected image pairs. However Aratani discloses wherein the optimization calculation is repeated for a plurality of lines derived from selected image pairs (para [0007] teaches the feature point detected from each key frame is searched on an epipolar line and associated with each other. a map is calculated with high accuracy by nonlinear optimization calculation). It would have been obvious to one having ordinary skill in the art before the effective filing date of the invention to use the method of 3D volume mapped/projected to the 2D images from the cameras of augmenting 3D depth map data with 2D color image data to include optical flow information of Colmenares in view of Tay Rowell with the method of to perform optimization calculation as taught of Aratani, to perform estimation with high accuracy
Regarding claim 32, Aratani further discloses the method of claim 31, wherein a portion of the plurality of lines is selected from epipolar lines (para [0007] teaches the feature point detected from each key frame is searched on an epipolar line and associated with each other... a map is calculated with high accuracy by nonlinear optimization calculation). Motivation to combine as indicated in claim 32.
12. Claims 33 is rejected under 35 U.S.C. 103 as being unpatentable over Colmenares (US 10,949,986 B1) (IDS provided 02/19/2025) in view of Tay et al. (US 2019/0311546 A1) (IDS provided 02/19/2025) and Simon et al. (US 2012/0257792 A1).
Regarding claim 33, Colmenares in view of Tay discloses the method of claim 23,
Colmenares in view of Tay does not explicitly disclose wherein likelihood calculations within a portion of 3D volumes produce numeric results that are independent of an order in which at least a portion of the data from intersection points derived from a set of image pairs is processed. However Simon discloses wherein likelihood calculations within a portion of 3D volumes produce numeric results that are independent of an order in which at least a portion of the data from intersection points derived from a set of image pairs is processed (para[0246] teaches two successive images are mapped together by using homologous primitives (representing the same details of the scene) belonging to the overlaps of the images. a (one-off) intersection of a road portion (linear) or the contour of a field (surface) These homologous primitives are independent of the range-found points). It would have been obvious to one having ordinary skill in the art before the effective filing date of the invention to use the method 3D volume mapped/projected to the 2D images from the cameras of augmenting 3D depth map data with 2D color image data of Colmenares in view of Tay with the method of homologous primitives of Simon in order to provide a system to geo-reference an area
13. Claims 38, 41-44 are rejected under 35 U.S.C. 103 as being unpatentable over Colmenares (US 10,949,986 B1) (IDS provided 02/19/2025) in view of Tay et al. (US 2019/0311546 A1) (IDS provided 02/19/2025) and Owechko et al. (US 20190035148 A1).
Regarding claim 38, Colmenares in view of Tay discloses the method of claim 23, Colmenares in view of Tay further comprising: determining a presence of one or more surfaces in a 3D scene by processing multiple 3D volumes to determine the likelihood of a surface within at least one 3D volume; and collecting at least a portion of these results into an accumulated dataset. However Owechko discloses determining a presence of one or more surfaces in a 3D scene by processing multiple 3D volumes to determine the likelihood of a surface within at least one 3D volume; and collecting at least a portion of these results into an accumulated dataset (para[0006] teaches the point cloud data from each sensor defines a point cloud that represents the object. Each point cloud includes a multiplicity of 3-D points and each 3-D point includes at least location information for the 3-D point on the object. Para[0040] teaches the 3-D point cloud data 114 or point cloud 106 from each sensor 110a-110n is stored in a database 122 or other data storage device). It would have been obvious to one having ordinary skill in the art before the effective filing date of the invention to use the method 3D volume mapped/projected to the 2D images from the cameras of augmenting 3D depth map data with 2D color image data of Colmenares in view of Tay with method are provided for representing and optimally combining 3-D point cloud data generated by a plurality of 3-D scanning sensors of Owechko in order to provide a system that measures and tracks objects in large spatial volumes.
Regarding claim 41, Colmenares further discloses the method of claim 38, wherein the multiple 3D volumes do not in aggregate cover the real-life scene in its entirety (col. 10 line 44-col 11 line 2 teaches the cameras 112 each have a field of view 664 of the scene 108, and the trackers 114 each have a field of view 666 of the scene 108. the fields of view 666 of the trackers 114 can at least partially overlap one another (and/or the fields of view 664 of the cameras 112) to together define a tracking volume the regions of overlap are tiled such that the resulting imaging volume covered by all the cameras 112 has a selected volume that exists as a subset of the volume covered by the trackers 114).
Regarding claim 42, Colmenares further discloses the method of claim 38, wherein at least a portion of the multiple 3D volumes are substantially aligned along at least one line projecting into the real-life scene from at least one 3D point relative to one or more of the plurality of cameras (col 15 lines 10-25 teaches where the system 100 processes image data from only one of the cameras 112 (e.g., where a ROI is determined in an image from only one of the cameras 112), the 3D position of the tip 103 can be determined by projecting the position of the tip 103 in the 2D image from the camera 112 into a 3D line using the calibration of the camera 112 (block 772)).
Regarding claim 43, Tay further discloses the method of claim 38, wherein at least a portion of the multiple 3D volumes are substantially aligned along a plurality of lines projecting into the real-life scene from at least one 3D point relative to one or more of the plurality of cameras (para [0026] teaches the system can align any other horizontal reference point in the image plane with the center of the corresponding lane. For example, for a color camera plane of a left-forward camera, the system can align a pixel one-third of the width of the image plane from the right edge of the image plane with the center of the corresponding lane). Motivation to combine as indicted in claim 38.
Regarding claim 44, Tay discloses the method of claim 38, further comprising; determining the likelihood that a physical surface is at least partially contained within one or more 3D neighborhoods, wherein one or more of the 3D volumes are the one or more 3D neighborhoods (Para[0060] teaches porting 2D color pixels from the 2D color image onto a 2D (i.e., planar) manifold that (approximately) intersects this cluster of points in the 3D point cloud, thereby preserving 2D color pixel data within a 2D domain, Para[0075] teaches calculate planes that approximately intersect clusters of points that represent faces of other dynamic objects in the field. para [0123] teaches isolate a cluster of color pixels in the concurrent color image that correspond to a cluster of points representing this particular object in the 3D point cloud). Motivation to combine as indicted in claim 38.
14. Claims 39-40 are rejected under 35 U.S.C. 103 as being unpatentable over Colmenares (US 10,949,986 B1) (IDS provided 02/19/2025) in view of Tay et al. (US 2019/0311546 A1) (IDS provided 02/19/2025) and Owechko et al. (US 20190035148 A1) and Taieb et al. (WO 2020/243484 A1) (attached).
Regarding claim 39, Colmenares in view of Tay and Owechko discloses the method of claim 38, Colmenares in view of Tay and Owechko does not explicitly disclose
wherein the multiple 3D volumes do not overlap. However Taieb discloses wherein the multiple 3D volumes do not overlap (para[0150] teaches the image capture devices 122, 124, and 126 may capture adjacent FOVs or may have partial overlap in their FOVs. Para[0573] teaches alternatively, the first camera and the second camera have different fields-of-view. Para[0574] teaches the first camera may be configured to capture the first image at the same time as the second camera captures the second image. Alternatively, the first camera may be configured to capture the first image at a different time as the second camera captures the second image. For example, first image 4301 may be captured at 0.3 seconds earlier than the time when second image 4302 is captured). It would have been obvious to one having ordinary skill in the art before the effective filing date of the invention to use the method 3D volume mapped/projected to the 2D images from the cameras of augmenting 3D depth map data with 2D color image data optimally combining 3-D point cloud data generated by a plurality of 3-D scanning sensors of Colmenares in view of Tay and Owechko in the method of performing perform stereo image analysis of Taieb in order to provide a system in which vehicle navigating to confirm the position and/or direction of the mapped arrow based on an arrow detected on the roadway..
Regarding claim 40, Colmenares in view of Tay and Owechko discloses the method of claim 38, Colmenares in view of Tay and Owechko does not explicitly disclose
wherein the multiple 3D volumes do at least partially overlap. However Taieb discloses the method of claim 38, wherein the multiple 3D volumes do at least partially overlap (para[0150] teaches the image capture devices 122, 124, and 126 may capture adjacent FOVs or may have partial overlap in their FOVs. Para[0573] In some embodiments, the first camera and the second camera may have at least partially overlapping fields-of-view. Alternatively, the first camera and the second camera have different fields-of-view. Para[0574] teaches the first camera may be configured to capture the first image at the same time as the second camera captures the second image. Alternatively, the first camera may be configured to capture the first image at a different time as the second camera captures the second image. For example, first image 4301 may be captured at time that is 0.3 seconds earlier than the time when second image 4302 is captured). It would have been obvious to one having ordinary skill in the art before the effective filing date of the invention to use the method 3D volume mapped/projected to the 2D images from the cameras of augmenting 3D depth map data with 2D color image data optimally combining 3-D point cloud data generated by a plurality of 3-D scanning sensors of Colmenares in view of Tay and Owechko in the method of performing perform stereo image analysis of Taieb in order to provide a system in which vehicle navigating to confirm the position and/or direction of the mapped arrow based on an arrow detected on the roadway.
Allowable Subject Matter
15. The following is a statement of reasons for the indication of allowable subject matter:
Claims 34-35 contain allowable subject matter, but are rejected under Double Patenting. If Terminal disclaimer is filed and the limitations are rewritten in independent form including all of the limitations of the base claim and any intervening claims then is in condition for allowance.
The following is a statement of reasons for the indication of allowable subject matter: Claim 34 discloses calculating data associated with intersection points that are input into the likelihood calculations for one or more 3D neighborhoods associated with 3D scene information substantially aligned on at least one reference surface from pixel data extracted from at least two rectified images separated by a pixel offset.
Prior arts cited above combined cannot teach these features perfectly. These limitations when read in light of the rest of the limitations in the claim and the claims to which it depends make the claim allowable subject matter.
Claim 35 depend on claim 34, and will be allowed based on same reason as claim 34.
Conclusion
16. Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROWINA J CATTUNGAL whose telephone number is (571)270-5922. The examiner can normally be reached Monday-Thursday 7:30am-6pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Brian Pendleton can be reached at (571) 272-7527. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ROWINA J CATTUNGAL/Primary Examiner, Art Unit 2425
