CTFR 18/818,662 CTFR 85918 DETAILED ACTION Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Response to Arguments Applicant's arguments filed 4/29/26 have been fully considered. Applicant asserts that support for claim amendments can be found in ¶ 0008 of the disclosure. A review of ¶ 8 shows the statement “The extracted features may be transmitted to a three dimensional convolution network for depth and pose estimation.” Further, ¶ 5 discusses “The neural network may be a combination of feature extraction network and a three dimensional convolution network.” Meanwhile, amended claim 1, recites “calculating a pose and depth estimate from the generated feature maps using at least one convolutional neural network wherein the convolutional neural network comprises a feature- extraction network coupled to a three-dimensional convolutional network”. While the difference is nuanced, the amended claim language infers relationships that are not present in the original disclosure. The original disclosure discusses a neural network which may be a combination of feature extraction network and a three dimensional convolution network. The original disclosure does not support a convolutional neural network which comprises a feature extraction network coupled to three dimensional convolution network. The examiner suggests amending this limitation to read “calculating a pose and depth estimate from the generated feature maps using at least one convolutional neural network wherein the convolutional neural network comprises a feature-extraction network coupled to and a three-dimensional convolutional network”. This proposed amendment would be more consistent with the arrangement described in the original disclosure. Applicant argues (Remarks page 5) that “Neither Herman nor Wang discloses….producing a bowl-shaped composite view for the surround image.” The Applicant asserts that Herman teaches a conventional top-down view stitched onto a flat plane. This argument is not persuasive as Herman explicitly discusses creating a bowl shaped surround view in ¶ 19 & ¶ 30, “The projection surface 202 is a virtual object that is defined to so that the boundary of the projection surface 202 is a distance from the vehicle 100 in a bowl shape.” Applicant argues (Remarks page 5) that Neither Herman nor Wang discloses…that the convolutional neural network comprised a feature-extraction network coupled to a three-dimensional convolutional network.” Applicant asserts that Wang discloses the use of a CNN broadly and differs from the use of a specialized 3D convolutional network. This argument is persuasive but is considered moot because a new ground of rejection is introduced to teach this feature. Claim Objections Claim 4 is objected to because of the following informalities: claim 4 recites “a feature extraction network” which already has antecedent basis in claim 1. The examiner considers this to be a typo and both claims 1 and 4 are referring to the same feature extraction network, which is consistent with the specification. Appropriate correction is required. The examiner suggests replacing “a” with “the” or “said”. Claim Rejections - 35 USC § 112 07-30-01 AIA The following is a quotation of the first paragraph of 35 U.S.C. 112(a): (a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention. The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112: The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention. 07-31-01 Claim 1-8 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention. Amended claim 1 recites amended limitation “calculating a pose and depth estimate from the generated feature maps using at least one convolutional neural network wherein the convolutional neural network comprises a feature- extraction network coupled to a three-dimensional convolutional network ”. While the difference is minor, the amended claim language infers relationships that are not supported by the original disclosure. The original disclosure in ¶ 5 & 8 provide the following “The extracted features may be transmitted to a three dimensional convolution network for depth and pose estimation” and “The neural network may be a combination of feature extraction network and a three dimensional convolution network.” The original disclosure does not discuss the feature-extraction network and 3D CNN being part of an overarching CNN. Instead, the specification indicates they are part of an overarching neural network. As the applicant acknowledges in their arguments, there is a distinct difference in NN, CNN, and 3D CNN. Further, the original disclosure does not discuss the feature-extraction network being coupled to the 3D CNN. Both of the amended features change the scope of the claim in a manner that is not supported by the original disclosure and thus are considered new matter. The examiner suggests amending this limitation to read “calculating a pose and depth estimate from the generated feature maps using at least one convolutional neural network wherein the convolutional neural network comprises a feature-extraction network coupled to and a three-dimensional convolutional network”. This proposed amendment would be more consistent with the arrangement described in the original disclosure. Claims 2-8 are similarly rejected based on their dependence. Claim Rejections - 35 USC § 103 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-21-aia AIA Claim s 1-8 are rejected under 35 U.S.C. 103 as being unpatentable over Herman et al. (US 2019/0349571 A1) in view of in view of Wang et al. (US 2023/0136860 A1) and further in view of Yoo et al. (US 2020/0294257) . Regarding claim 1, Herman discloses a method for generating a three-dimensional reconstruction of an environment around a vehicle (¶19, “Increasingly, vehicles include camera systems that generate a virtual isometric or top-down view of the three-dimensional area around the vehicle.”) comprising: capturing multiple wide-angle images around the vehicle using multiple fisheye lens cameras mounted on the vehicle (¶ 19, “However, these camera systems generate these isometric images based the characteristics of the three-dimensional area around the vehicle using images captures by cameras (e.g., a 360-degree camera system, ultra-wide-angle cameras positioned on the peripheral of the vehicle, etc.), stitching them together, and projecting the stitched image onto a projection surface.”) ; creating a bowl-shaped surround view image around the vehicle from the multiple captured images by generating one or more feature maps (¶ 19, “However, these camera system generate these isometric images based the characteristics of the three-dimensional area around the vehicle using images captures by cameras (e.g., a 360 degree camera system, ultra-wide angle cameras positioned on the peripheral of the vehicle, etc.), stitching them together, and projecting the stitched image onto a projection surface.”; ¶ 19 and 30 discuss a bowl-shaped projection surface) ; and calculating a pose and depth estimate from the generated feature maps using at least one convolutional neural network wherein the convolutional neural network comprises a feature-extraction network coupled to a three-dimensional convolutional network (¶ 29, “In such examples, the image generator 110 retrieves a three-dimensional geometry of recognized object from a database (e.g., a database residing on the external server, a database stored in computer memory, etc.) and inserts the three-dimensional geometry into the depth map based on the pose (e.g., distance, relative angle, etc.) of the detected object”; ¶ 20 discusses a neural network) ; detecting one or more objects in the multiple captured images (¶ 20 discusses detecting objects and structures around the vehicle) ; mapping one or more objects detected around a model of the vehicle to the created surround view image using the calculated pose and depth estimate (¶ 25 discusses generating a depth map; ¶ 29 discusses the depth map includes objects proximate to the vehicle; ¶ 29 further discusses using the pose of the detected object) ; and constructing the three-dimensional reconstruction of the environment, using the surround view image and the mapped objects (Fig 7 illustrates the process of using sensor fusion to reconstruct the environment using the image voxel map and the sensor data voxel map; see ¶ 20 & 40) . Herman does not explicitly teach the ultrawide cameras are “fisheye” or the neural network is a “convolutional neural network”. Further, Herman discusses a point cloud which, when given the broadest reasonable interpretation, could be considered a feature map, but Herman does not use the explicit term “feature map”. Wang discloses fisheye cameras (¶ 52) , the use of feature maps ( ¶ 76-77) , and a convolutional neural network (¶ 134) . Wang is in the same field of endeavor of stitching images of an environment around a vehicle (¶ 59) to create a surrounding view. Herman teaches the use of ultrawide cameras that cause distortion in their captured image. This is a hallmark characteristic of fisheye lenses which are a common and well-known type of ultra-wide camera lens, as evidenced by Wang. It would have been obvious to one having ordinary skill in the art to modify Herman with Wang such that fisheye cameras are used. Such a modification would have been a simple substitution which would have provided the same predictable results. Herman teaches the use of a neural network. A convolutional neural network is a common and well-known type of neural network, as evidenced by Wang. It would have been obvious to one having ordinary skill in to modify Herman such that the neural network is of a convolutional type. Such a modification would have been a simple substitution which would have provided the same predictable results. Herman discusses using generating a point cloud as well as detecting objects/structures from images (¶ 20 & 29 and Fig 7). Herman is largely silent as to how these objects are detected. Wang ¶ 76-77 discusses extracting features and creating a feature map. Feature maps are well known and common in the machine learning art, as evidenced by Wang. As such, it would have been obvious to one having ordinary skill in the art to modify Herman such that Herman’s neural network applies feature maps. Such a modification would have provided predictable results. Herman, as modified by Wang, is silent as to the use of a 3D CNN. Yoo discloses convolutional neural network comprises a feature-extraction network (126, fig 1) coupled to a three-dimensional convolutional network (128, fig 1) (Yoo ¶ 47 & 70 discuss performing a feature extraction and using a 3D CNN to create feature maps) . Yoo is in the same field of endeavor of object detection in autonomous machines. Wang discusses the use of a convolutional neural network, but does not explicitly discuss the use of a 3d CNN. Yoo discusses that the use of a 3D CNN, in conjunction with a 2D CNN, has the improvement of “increasing accuracy of the predictions with respect to using data from a single dimensional space.” Yoo ¶ 52-53 further discuss that the use of 3D voxels in a 3D CNN is more accurate than simply using only 2D points in a 2D CNN. As such, it would have been obvious to one having ordinary skill in the art to modify Herman, as modified, with Yoo such that a neural network using a feature-extraction network coupled to a three-dimensional convolutional network is used. Such a modification would have the benefit of improved accuracy as suggested by Yoo. Regarding claim 2, Herman as modified, discloses the method as claimed in claim 1, wherein capturing the multiple wide-angle images further comprises capturing at least two wide-angle images having overlapping field of view, from the multiple fisheye lens cameras (Wang ¶ 128 discusses overlapping views) ; calculating a disparity between the captured at least two wide angle images (Wang ¶ 56 discusses creating a disparity map) ; and detecting coordinates of an object on the at least two or more captured wide-angle images, and a dimension of the object on the image using the calculated disparity (Wang ¶ 70 discusses 3D world coordinates; Herman fig 3A, 3B, and 5 showcase dimensionality of an object; Herman ¶ 29 discusses object 3D geometry) . Herman teaches the use of multiple ultra-wide-angle cameras, a disparity map (¶ 20) , and finding the location of objects using a point cloud. Herman does not explicitly discuss overlapping field of views (FOV) or detecting coordinates of objects. However, Herman teaches the use of 4 cameras (each ultrawide or 360 degrees; see element 106 in figure 1). The placement and FOV, it would have been obvious to a person having ordinary skill in the art that their views would overlap. Having overlapping FOVs is a well-known feature and is commonly used to stitch together multiple images into a composite image, as evidenced by Wang ¶ 128. Further, it is well known in the art to use coordinates in generating a point cloud. It is certainly obvious, if not inherent. This is evidenced by Wang ¶ 70 which discusses a point cloud using world coordinates. Because overlapping FOVs and using a coordinate system are well-known and common in the art of disparity maps and object detection, it would have been obvious to a person having ordinary skill in the art to modify Herman with Wang. Such a modification would be a simple application of a well-known technique to Herman’s device. Regarding claim 3, Herman as modified, discloses the method as claimed in claim 1, wherein capturing multiple wide-angle images further comprises, undistorting the captured images and correcting geometric alignment in the undistorted images (Herman figures 4 and 5 show “undistorting” of capturing images; ¶ 20) ; generating a point cloud by proving the calculated difference between the captured at least two wide-angle images as input to the convolutional neural network (Wang ¶ 56 & 59) ; and performing simultaneous localization of the objects on the captured at least two wide-angle images image using the generated point cloud and estimating motion of the object (Wang ¶ 91 and 220 discuss localization of objects and discuss multiple neural networks being run simultaneously; ¶ 97, 171, & 202 discusses object tracking and motion estimation) . As discussed above, Herman is largely silent as to how object detecting and tracking is performed. Wang discusses well known methods of using image disparity and SLAM. It would have been obvious to apply Wang’s techniques to Herman as they are well-known in the art and would have the benefit of improving automation of a vehicle. In regards to claim 4, Herman as modified discloses the method as claimed in claim 1, wherein the capturing, creating, calculating, detecting, mapping and constructing are performed more than once and wherein generating the feature map and calculating the pose and depth estimate comprise processing each of the captured wide-angle images by a feature extraction network, and calculating an incremental pose update for each of the captured wide-angle image (Wang ¶ 76-77 discusses a feature extractor using a neural network; ¶ 86 discusses continually updating the world model (i.e., incrementally updating object pose)) . In regards to claim 5, Herman as modified discloses the method as claimed in claim 1, wherein the model of the vehicle is placed at a center of the created surround view image (Herman figures 2B and 4 illustrate the vehicle being in the center; Wang ¶ 92 discusses the origin point being on the vehicle) . In regards to claim 6, Herman as modified discloses the method as claimed in claim 1, comprising continuously performing the method until a driving control unit in the vehicle is turned off. Wang ¶ 86 discusses continually updating the world model. Neither Herman or Wang discuss ceasing the surround view projections when a driving control unit in the vehicle is turned off. However, the examiner takes official notice that it is well known and common in the vehicular art, to cease various computational operations when a vehicle is turned off. As such, it would have been obvious to a person having ordinary skill in the art, to modify Herman as modified with Wang, such that the three-dimensional reconstruction ceases when the vehicle is turned off. Generally, the autonomous driving feature is not necessary when a car is turned off. Further, this has the benefit of saving power by deactivating unnecessary sensors and processors. In regards to claim 7, Herman as modified discloses a device for generating a three-dimensional reconstruction of an environment around a vehicle, comprising: a memory (Herman fig 6, ¶ 22 discusses memory) ; multiple fisheye lens cameras mounted over the vehicle (Herman camera 106, fig 1; Wang ¶ 52 discusses fisheye) ; one or more image processors coupled to the multiple fish eye lens cameras (Wang TPUs ¶ 190) ; one or more graphical processing units coupled to the multiple fish eye lens cameras (Wang GPUs ¶ 190) ; and one or more processing units for performing the method (Wang CPUSs ¶ 190) as claimed in claim 1 (See Wang figure 17C and ¶ 189-190) . Herman is largely silent as to the specifics of the processing hardware. It would have been obvious to one having ordinary skill in the art, to modify Herman with Wang such that Herman utilizes multiple types of processing units to help support the CNN and improve image processing performance. In regards to claim 8, Herman as modified discloses a non-transitory computer-readable storage medium comprising instructions, which when executed by a processor, performs the method of claim 1 (Herman ¶ 36-38) . Conclusion 07-40 AIA Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL . See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action. Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW A FRY whose telephone number is (303)297-4769. The examiner can normally be reached M-F, 9-5 MT. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amanda Lauritzen can be reached at (571) 272-4303. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /MATTHEW A FRY/Primary Examiner, Art Unit 6215 Application/Control Number: 18/818,662 Page 2 Art Unit: 6215 Application/Control Number: 18/818,662 Page 3 Art Unit: 6215 Application/Control Number: 18/818,662 Page 4 Art Unit: 6215 Application/Control Number: 18/818,662 Page 5 Art Unit: 6215 Application/Control Number: 18/818,662 Page 6 Art Unit: 6215 Application/Control Number: 18/818,662 Page 7 Art Unit: 6215 Application/Control Number: 18/818,662 Page 8 Art Unit: 6215 Application/Control Number: 18/818,662 Page 9 Art Unit: 6215 Application/Control Number: 18/818,662 Page 10 Art Unit: 6215 Application/Control Number: 18/818,662 Page 11 Art Unit: 6215 Application/Control Number: 18/818,662 Page 12 Art Unit: 6215
Prosecution Insights