Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims 1-13 are pending.
Claims 1,2,4,12,and 16 are amended.
Claim 13 is added.
Response to Arguments/Remarks
Claim amendments and new claims
Regarding the new dependent claim 13, please see 103 below.
Objections to the Claims
Regarding the objection to claims 7-12, the examiner agrees that the amendments overcome the objection and has therefore been withdrawn.
Prior Art Rejections
Applicant’s arguments with respect to claims 1-12 have been considered but are moot in view of the new ground(s) of rejection as necessitated by applicant's amendments.
Please see 35 U.S.C. 103 rejection below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claims 1,7 are rejected under 35 U.S.C. 103 as being unpatentable over MENASHE (US20150228077) in view of UMEMURA (JP2018206038A).
Regarding claim 1:
MENASHE discloses:
a microprocessor and a memory coupled to the microprocessor; and the microprocessor is configured to perform: (see at least MENASHE, ¶ 0004, “According to another aspect, a system for mapping, localization and pose correction includes, a processor and a visual odometry module that causes the processor to determine a current position of a vehicle along a travel route and a set of currently observable landmarks along the travel route relative to the current position.”)
extracting a plurality of feature points from a camera image acquired by a camera configured to image a space ahead of a subject vehicle; (see at least MENASHE, ¶ 0045, “The imaging device 122 can include one or more cameras or other imaging devices and sensing devices. For example, the imaging device 122 can be one or more stereo cameras, three-dimensional cameras, remote sensing devices (e.g., LIDAR, lasers, sensors), among others. In one embodiment, the imaging device 122 acquires stereo images (e.g., a left image, a right image) of an image scene (e.g., a scene of a travel route, a road, the environment surrounding the vehicle). The network 124 is, for example, a data network, the Internet, a wide area network or a local area network. The network 124 serves as a communication medium to various remote devices (e.g., web servers, remote servers, application servers, intermediary servers, client machines, other portable devices).”; ¶ 0053, "In one embodiment, the local 3D map 206 is built using a volumetric and key-frame based approach. In this embodiment, a volumetric representation of the stereo images 202 is generated. The current frame is designated a key-frame if there is sufficient camera motion between the frame and the previous key-frame. The relative pose between key-frames is estimated. In one embodiment, the relative pose is estimated using KLT trackers to track Harris corner features in the stereo image (e.g., left and right images). The features are matched between the left and right images using Normalized SSD (Sum of Squared Differences) and the 3D positions of the features are computed based on their disparity. ")
selecting feature points for which three-dimensional positions are to be calculated from among the plurality of feature points ; (see at least MENASHE, ¶ 0053; ¶ 0054, “The key-frames and their poses are used to build the local 3D map 206 of the environment surrounding the vehicle. In particular, to generate the local 3D map 206, for each key-frame a dense stereo disparity/depth image is computed using a variant of a Semi-Global Matching Algorithm with SAD Block Matching on a weighted combination of the original stereo images 202 and a Sobel Filtered image. Additionally, Scanline Optimization can be performed, for example, in only two directions as opposed to eight or 16 to save on computation time.”)
calculating a three-dimensional position for each of the feature points selected, which includes estimating a position and posture of the camera such that a same feature point in a plurality of camera images converges to a single point ; and (see at least MENASHE, ¶ 0053; ¶ 0055, “The local 3D map 206, centered on the vehicle, is generated and/or updated at each key-frame by combining the dense depth images over multiple key-frames making use of the key-frame poses to register the depth images into a single coordinate frame. In particular, in one embodiment, a 3D occupancy grid is used to build the local 3D map and remove spurious disparity values in the images. Combining multiple depth images and using a 3D occupancy grid provides a fast and effective way to handle outliers and missing depths thereby generating the local 3D map 206 in an accurate manner. The local 3D map 206 is utilized for landmark segmentation and tracking 208, which will be described in detail below.”; ¶ 0066, “In a one current and many existing equivalence class, several existing landmarks merge into a single current landmark, as can happen when new depth data becomes available. The voxel and point list of the existing landmark with the maximum overlap is updated to the current landmark's voxel list, and its attributes are updated as in the case of a one current and one existing equivalence class, discussed above. The rest of the existing landmarks are deleted.”; ¶ 0080, “In one embodiment, the visual odometry module 116 determines a vehicle relative pose along a travel route and a set of currently observable landmarks along the travel route relative to the vehicle relative pose. For example, as discussed above with FIGS. 1 and 2, stereo images 302 and a vehicle relative pose 304 (e.g., determined by the visual odometry module 116) are used to generate a local 3D map 306. The local 3D map 306 is processed using landmark segmentation 310 (e.g., as discussed with landmark segmentation 210 in FIG. 2), and results in a set of currently observable landmarks 312.”)
generating a map including information of the three-dimensional positions calculated for each of the feature points selected , wherein the microprocessor is configured to (see at least MENASHE, ¶ 0053; ¶ 0072, “To compute the properties, in one embodiment, the landmark is modeled using signed distance fields. A new 3D grid, with a finer resolution than the 3D occupancy grid, is created enclosing the landmark. For each start point associated with the landmark, a ray is cast to its end point. The voxel in which the endpoint falls is assigned a distance value of zero. Voxels along the ray and between the start point and the endpoint voxel are assigned a positive distance, measuring how far the voxel is from the endpoint. Similarly, voxels along the ray that are beyond the endpoint voxel (going away from the start point) are assigned negative distances. Since this process is executed for each start-end point pair, a voxel can be assigned multiple distance values if the voxel happens to fall on rays between the several start-end point pairs.”; ¶ 0080)
MENASHE does not disclose, but UMEMURA teaches:
perform the selecting including, when defining a first distance range as including feature points whose distance in a depth direction from the camera is less than a predetermined value, and defining a second distance range as including feature points whose distance in the depth direction from the camera is equal to or greater than the predetermined value, (see at least UMEMURA, ¶ 0009, "In order to achieve the above object, a point cloud data processing device according to an embodiment of the present technology processes point cloud data acquired by an environmental measurement sensor. The environmental measurement sensor acquires point cloud data in which each measured point on the surface of an object within a measurement range as viewed from a measurement position is regarded as a coordinate point and the position of each coordinate point is expressed in a set coordinate system. The point cloud data processing device includes a data amount reduction unit that reduces the number of coordinate points included in the point cloud data by performing a data amount reduction process on the point cloud data. The point cloud data includes close distance Point cloud data and long-distance point cloud data, and in the set coordinate system, the close-distance point cloud data indicates the position of each coordinate point in a close-distance area on the side of the measurement position, and the long-distance point cloud data indicates the position of each coordinate point in a long-distance area that is farther away from the measurement position than the close-distance area. The data volume reduction process is either: (A) a process of reducing the coordinate point density of the close distance point cloud data to a first threshold or less and the coordinate point density of the long-distance point cloud data to a second threshold or less, where the first threshold is higher than the second threshold, or (B) a process of reducing the number of coordinate points of the close-distance point cloud data so that the coordinate point density in the close distance point cloud data is the first threshold or less, while maintaining the number of coordinate points of the long-distance point cloud data."; ¶ 0014, "The above effect can be achieved by reducing the number of coordinate points in the close distance point cloud data between the close-distance point cloud data and the long-distance point cloud data so that the density of coordinate points in the close-distance point cloud data is below the first threshold through data volume reduction processing. In other words, since the density of coordinate points in close-distance point cloud data tends to be high, if the density of coordinate points in close-distance point cloud data is reduced to below an appropriate threshold (first threshold), there is no need to reduce the number of coordinate points in long-distance point cloud data. This also makes it possible to perform data volume reduction processing to improve the efficiency or speed of integrating point cloud data, while obtaining point cloud data suitable for generating a highly accurate map. ")
selecting the feature points to be used for calculating the three-dimensional positions from among the plurality of feature points extracted so as to reduce a bias in a number of feature points located in the first distance range and a number of feature points located in the second distance range. (see at least UMEMURA, ¶ 0014; ¶ 0021, "The point cloud data acquired by the environment measurement sensor 3 includes shortdistance point cloud data and long-distance point cloud data. The long-distance point cloud data is data of a spatial region that is farther away from the measurement position than the short-distance point cloud data. In other words, in the sensor coordinate system, the close-distance point cloud data indicates the position (coordinates) of each coordinate point in the close-distance area on the side of the measurement position, and the long-distance point cloud data indicates the position (coordinates) of each coordinate point in the long-distance area that is farther away from the measurement position than the close-distance area. For example, a spatial region whose distance from the measurement position where the point cloud data is measured is within a set distance is a short distance region, and a spatial region whose distance from the measurement position exceeds the set distance is a long distance region. Here, the set distance is, for example, a value of 20 m or more and 50 m or less, a value of 50 m or more and 75 m or less, a value of 75 m or more and 100 m or less, a value of 100 m or more and 150 m or less, or a value of 150 m or more and 200 m or less, but is not limited to these. ")
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify, with a reasonable expectation of success, the image and keyframe based point cloud mapping system of MENASHE to include dividing feature points between different predetermined distances and filtering within UNEMURA to yield an efficient point cloud mapping feature that filters out excessive data collected from closer objects with denser feature point distributions (UNEMURA, ¶ 0009).
EXAMINERS NOTE: Even though MENASHE does not explicitly collect “feature points”, it is implied that feature points are collected by referencing feature point collection techniques such as Harris corners (¶ 0053).
Regarding claim 7:
With regards to claim 7, this independent claim is the microcontroller function claim to the microcontroller configured claim 1 and is substantially similar to claim 1 and is therefore rejected using the same references and rationale.
Claims 2-6, 8-12 are rejected under 35 U.S.C. 103 as being unpatentable over MENASHE (US20150228077) in view of UMEMURA (JP2018206038A) in further view of INABA (US20120275711).
Regarding claim 2:
MENASHE in view of UMEMURA discloses the limitations within claim 1 and MENASHE does not teach, but INABA teaches:
the microprocessor is configured to perform the selecting including thinning out feature points from a distance range having feature points whose number is more than an average of the numbers of the feature points in the first distance range and the number of the feature points in the second distance range so as to bring the number of the feature points in the distance range closer to the average to select remaining feature points after the thinning. (see at least INABA, ¶ 0008, "By the way, when ANMS is used, it is possible to remove feature points in a spatially uniform manner while controlling the upper limit of the number of the remaining feature points using a parameter. In this method, it is necessary to, for each feature point, first compute the distance (hereinafter referred to as a radius) between each feature point and a feature point whose coordinates are close among the feature points having higher scores than each feature point. Further, it is also necessary to store the coordinates and the radii of all feature points into memory. The size of the radius is first set to "0," and the number of feature points at that time is counted, and if the counted number of the feature points is higher than the set upper limit, the size of the radius is increased a little. Such a process is repeated, and the removing process terminates when the number of the remaining feature points is within the set upper limit Therefore, a high-speed CPU and high-capacity memory are needed."; ¶ 0015, "According to the present technology described above, feature points are detected from an image. In addition, the reliability of each of the detected feature points is computed. Further, the image is split into a plurality of regions, and some of the detected feature points are removed on the basis of the reliability so that the number of the remaining feature points in each split region is within the restricted number of feature points. Therefore, the feature points can be removed in a spatially uniform manner with a simple configuration at a high speed."; ¶ 0049, "The removal processing unit 22, on the basis of the reliability of each feature point, selects a desired number of feature points in order of decreasing reliability, and removes the non-unselected feature points.")
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify, with a reasonable expectation of success, the image-based multi-distance point cloud mapping system of MENASHE in view of UNEMORA to include the splitting of feature points into regions for selection and filtering to remove excess data within INABA to effectively yield, cleaner, less computationally intensive mapping of feature points by removing excess unnecessary data.
Regarding claim 3:
MENASHE in view of UMEMURA in further view of INABA discloses the limitations within claim 2 and MENASHE does not teach, but INABA teaches:
perform the selecting including dividing the camera image (see at least INABA, ¶ 0043, “Next, the removing process will be described. The removal processing unit 22 splits an image into blocks. FIG. 3 exemplarily shows a case where the block size is represented by a vertical direction "H" and a horizontal direction "W." Note that when an image is split into a plurality of blocks in predetermined size, it is acceptable if a block with a size smaller than the predetermined size is generated at the end of the image.”; ¶ 0044, “FIG. 4 is a diagram illustrating the removing process. Suppose, as shown in (A) and (B) in FIG. 4, that seven feature points are detected in a single split block by a feature point detection process, for example. The removal processing unit 22 ranks the detected feature points on the basis of the reliability (score) of each feature point. Note that in (B) in FIG. 4, the feature points are defined in order of decreasing reliability such that "a1, a2, a3, . . . , a7."”)
into a plurality of groups to thin out the feature points from the distance range having more feature points than the average so that the number of the feature points in the distance range to be thinned out is not biased among the plurality of groups. (see at least INABA, ¶ 0008, "By the way, when ANMS is used, it is possible to remove feature points in a spatially uniform manner while controlling the upper limit of the number of the remaining feature points using a parameter. In this method, it is necessary to, for each feature point, first compute the distance (hereinafter referred to as a radius) between each feature point and a feature point whose coordinates are close among the feature points having higher scores than each feature point. Further, it is also necessary to store the coordinates and the radii of all feature points into memory. The size of the radius is first set to "0," and the number of feature points at that time is counted, and if the counted number of the feature points is higher than the set upper limit, the size of the radius is increased a little. Such a process is repeated, and the removing process terminates when the number of the remaining feature points is within the set upper limit Therefore, a high-speed CPU and high-capacity memory are needed."; ¶ 0054, "Removal of the feature points may also be performed by splitting a three-dimensional region, which is obtained by adding a new dimension to a two-dimensional image, into a plurality of regions and removing some feature points from each split region so that the remaining feature points are uniformly dispersed in the three-dimensional space. For example, in detection of feature points, not only the position (x,y) in the image, but also an image with a different scale (s) is used. In such a case, it is considered that feature points are distributed across the three-dimensional space of (x,y,s). Such a space is split into rectangular parallelepipeds and the upper limit of the number of feature points included in each rectangular parallelepiped is set, and then the feature points are removed in order of increasing reliability so that the number of the remaining feature points in each rectangular parallelepiped is within the upper limit. By performing such a removing process on each rectangular parallelepiped, it becomes possible to remove feature points with a simple configuration at a high speed so that the remaining feature points are dispersed in a spatially uniform manner even in the three-dimensional space. In addition, it is also possible to, in a two-dimensional space on the X-Y plane, remove feature points from a three-dimensional space including a Z-direction that intersects the XY plane at right angles so that the number of the remaining feature points are dispersed in a spatially uniform manner.")
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify, with a reasonable expectation of success, the image-based multi-distance point cloud mapping collection of MENASHE in view of UNEMORA to include the dividing the images into blocks for feature point selection and filtering to remove excess data of INABA to effectively yield, cleaner, less computationally-intensive mapping by removing excess unnecessary data and noise.
Regarding claim 4:
MENASHE in view of UMEMURA discloses the limitations within claim 1 and MENASHE does not teach, but INABA teaches:
wherein the microprocessor is configured to perform the selecting including thinning out the feature points from the first or second distance ranges which has a larger number of feature points than the other so as to bring a number of feature points in the first or second distance ranges which has the larger number of feature points closer to a number of feature points in the first or second distance ranges which has a smaller number of feature points than the other to select the remaining feature points after the thinning. (see at least INABA, ¶ 0008, "By the way, when ANMS is used, it is possible to remove feature points in a spatially uniform manner while controlling the upper limit of the number of the remaining feature points using a parameter. In this method, it is necessary to, for each feature point, first compute the distance (hereinafter referred to as a radius) between each feature point and a feature point whose coordinates are close among the feature points having higher scores than each feature point. Further, it is also necessary to store the coordinates and the radii of all feature points into memory. The size of the radius is first set to "0," and the number of feature points at that time is counted, and if the counted number of the feature points is higher than the set upper limit, the size of the radius is increased a little. Such a process is repeated, and the removing process terminates when the number of the remaining feature points is within the set upper limit Therefore, a high-speed CPU and high-capacity memory are needed."; ¶ 0015, "According to the present technology described above, feature points are detected from an image. In addition, the reliability of each of the detected feature points is computed. Further, the image is split into a plurality of regions, and some of the detected feature points are removed on the basis of the reliability so that the number of the remaining feature points in each split region is within the restricted number of feature points. Therefore, the feature points can be removed in a spatially uniform manner with a simple configuration at a high speed."; ¶ 0049, "The removal processing unit 22, on the basis of the reliability of each feature point, selects a desired number of feature points in order of decreasing reliability, and removes the non-unselected feature points.")
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify, with a reasonable expectation of success, the image-based multi-distance point cloud mapping system of MENASHE in view of UNEMORA to include the splitting of feature points into regions for selection and filtering to remove excess data within INABA to effectively yield, cleaner, less computationally intensive mapping by removing unnecessary excess datapoints.
Regarding claim 5:
MENASHE in view of UMEMURA in further view of INABA discloses the limitations within claim 4 and MENASHE does not teach, but INABA teaches:
wherein the microprocessor is configured to perform the selecting including dividing the camera image into a plurality of groups (see at least INABA, ¶ 0043, “Next, the removing process will be described. The removal processing unit 22 splits an image into blocks. FIG. 3 exemplarily shows a case where the block size is represented by a vertical direction "H" and a horizontal direction "W." Note that when an image is split into a plurality of blocks in predetermined size, it is acceptable if a block with a size smaller than the predetermined size is generated at the end of the image.”; ¶ 0044, “FIG. 4 is a diagram illustrating the removing process. Suppose, as shown in (A) and (B) in FIG. 4, that seven feature points are detected in a single split block by a feature point detection process, for example. The removal processing unit 22 ranks the detected feature points on the basis of the reliability (score) of each feature point. Note that in (B) in FIG. 4, the feature points are defined in order of decreasing reliability such that "a1, a2, a3, . . . , a7."”)
to thin out the feature points from the first or second distance ranges which has the larger number of feature points so that the number of feature points to be thinned out is not biased among the plurality of groups. (see at least INABA, ¶ 0008, "By the way, when ANMS is used, it is possible to remove feature points in a spatially uniform manner while controlling the upper limit of the number of the remaining feature points using a parameter. In this method, it is necessary to, for each feature point, first compute the distance (hereinafter referred to as a radius) between each feature point and a feature point whose coordinates are close among the feature points having higher scores than each feature point. Further, it is also necessary to store the coordinates and the radii of all feature points into memory. The size of the radius is first set to "0," and the number of feature points at that time is counted, and if the counted number of the feature points is higher than the set upper limit, the size of the radius is increased a little. Such a process is repeated, and the removing process terminates when the number of the remaining feature points is within the set upper limit Therefore, a high-speed CPU and high-capacity memory are needed."; ¶ 0054, "Removal of the feature points may also be performed by splitting a three-dimensional region, which is obtained by adding a new dimension to a two-dimensional image, into a plurality of regions and removing some feature points from each split region so that the remaining feature points are uniformly dispersed in the three-dimensional space. For example, in detection of feature points, not only the position (x,y) in the image, but also an image with a different scale (s) is used. In such a case, it is considered that feature points are distributed across the three-dimensional space of (x,y,s). Such a space is split into rectangular parallelepipeds and the upper limit of the number of feature points included in each rectangular parallelepiped is set, and then the feature points are removed in order of increasing reliability so that the number of the remaining feature points in each rectangular parallelepiped is within the upper limit. By performing such a removing process on each rectangular parallelepiped, it becomes possible to remove feature points with a simple configuration at a high speed so that the remaining feature points are dispersed in a spatially uniform manner even in the three-dimensional space. In addition, it is also possible to, in a two-dimensional space on the X-Y plane, remove feature points from a three-dimensional space including a Z-direction that intersects the XY plane at right angles so that the number of the remaining feature points are dispersed in a spatially uniform manner.")
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify, with a reasonable expectation of success, the image-based multi-distance point cloud mapping system of MENASHE in view of UNEMORA to include the dividing the images into blocks for feature point selection and filtering to remove excess data of INABA to effectively yield, cleaner, less computationally intensive mapping by removing unnecessary excess datapoints.
Regarding claim 6:
MENASHE in view of UMEMURA in further view of INABA discloses the limitations within claim 5 and MENASHE does not teach, but INABA teaches:
the plurality of groups are a plurality of groups of rectangular shape obtained by dividing the camera image in a horizontal direction and a vertical direction, and (see at least INABA, ¶ 0043, “Next, the removing process will be described. The removal processing unit 22 splits an image into blocks. FIG. 3 exemplarily shows a case where the block size is represented by a vertical direction "H" and a horizontal direction "W." Note that when an image is split into a plurality of blocks in predetermined size, it is acceptable if a block with a size smaller than the predetermined size is generated at the end of the image.”)
the microprocessor is configured to perform the selecting including performing thinning of the feature points so that the number of feature points to be thinned out is not biased among the plurality of groups of rectangular shape. (see at least INABA, ¶ 0044, “FIG. 4 is a diagram illustrating the removing process. Suppose, as shown in (A) and (B) in FIG. 4, that seven feature points are detected in a single split block by a feature point detection process, for example. The removal processing unit 22 ranks the detected feature points on the basis of the reliability (score) of each feature point. Note that in (B) in FIG. 4, the feature points are defined in order of decreasing reliability such that "a1, a2, a3, . . . , a7.")
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify, with a reasonable expectation of success, the image-based multi-distance point cloud mapping system of MENASHE in view of UNEMORA to include the dividing the images into blocks for feature point selection and filtering to remove excess data of INABA to effectively yield, cleaner, less computationally intensive mapping by removing unnecessary excess datapoints.
Regarding claim 8:
With regards to claim 8, this claim is substantially similar to claim 2 and is therefore rejected using the same references and rationale.
Regarding claim 9:
With regards to claim 9, this claim is substantially similar to claim 3 and is therefore rejected using the same references and rationale.
Regarding claim 10:
With regards to claim 10, this claim is substantially similar to claim 4 and is therefore rejected using the same references and rationale.
Regarding claim 11:
With regards to claim 11, this claim is substantially similar to claim 5 and is therefore rejected using the same references and rationale.
Regarding claim 12:
With regards to claim 12, this claim is substantially similar to claim 11 and is therefore rejected using the same references and rationale.
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over MENASHE (US20150228077) in view of UMEMURA (JP2018206038A) in further view of ISHII (JP2021117130A).
Regarding claim 13:
MENASHE in view of UMEMURA discloses the limitations within claim 1 and MENASHE does not teach, but ISHII teaches:
the microprocessor is configured to perform the estimating including estimating, the distance in the depth direction from the camera to each of the plurality of feature points extracted using machine learning based on a position in the camera image of an object corresponding to each of the plurality of feature points extracted. (see at least ISHII, ¶ 0005, "However, monocular cameras generally have advantages over depth sensors in terms of computational cost, financial cost, and size constraints, but are inferior in terms of accuracy. Furthermore, with a monocular camera, the length W1 on the image at the reference distance Z1 must be known, or the actual dimensions of the object must be known (it cannot be used if the size is unknown), and if the dimensions are not known, a stereo camera or the like must be used. However, even when a stereo camera is used, if the object is three-dimensional, errors may occur due to surface irregularities, rotation of the object, and the like. "; ¶ 0026, "The CNN unit 210 generates two-dimensional coordinates (map) of feature points of an object and provides them to the machine learning unit 230. The machine learning unit 230 uses the teacher label (feature point map with depth information) provided by the teacher label generation unit 220 to adjust the filter coefficients or weighting coefficients of the CNN unit 210 so that the two-dimensional coordinates of the feature points of the object generated by the CNN unit 210 match the feature point map of the teacher label L. The adjusted coefficients are stored in a memory or the like as learning data. Furthermore, the machine learning unit 230 uses the teacher label L to provide depth information to the three-dimensional coordinates of the feature points. In this way, a feature point map with depth information is simultaneously generated as the three-dimensional coordinates of the object, and this is obtained from the output unit 240. "; ¶ 0027, "The teacher label generation unit 220 calculates the distance to the object (i.e., depth information) using the parallax of the object captured in at least two stereo image data captured by the stereo camera 220, and generates a teacher label L, which is a feature point map with depth information. In this way, the learning unit 140 performs machine learning on the CNN unit 210 so that the three-dimensional coordinates of an object can be estimated from the image data P2 of the monocular camera 120.")
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify, with a reasonable expectation of success, the image and keyframe based point cloud mapping system of MENASHE to include a Machine Learning depth processing based on images captured by a monocular camera within ISHII to yield an effective cost-efficient depth detection system for determining distance from the vehicle to objects without the need of extra sensors such as a stereoscopic camera (ISHII, ¶ 0005)
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
ZHANG (US10410405)
Col 8, lines 61-67 and Col 9, lines 0-10, “In some cases, the feature points can be filtered based on a similarity between the feature points and the matching feature points. For example, a feature point eliminating threshold can be determined based on the similarity between a plurality of feature points and a corresponding plurality of matching feature points, such as the similarity S calculated based on the previously shown equation. Individual feature point and its matching feature point can be filtered out if the similarity between them is less than the feature point eliminating threshold. When the similarity between a feature point and its matching feature point is less than the eliminating threshold, it can be inferred that the matching feature point is not be the corresponding point on the second image to the feature point on the first image. Accordingly, it can be determined that the matching of the feature point has failed and the failed feature points are not to be included in subsequent 3D modeling calculation.”
Kiyonari Kishikawa et al. (JP6080640B2)
¶ 0063, “Next, the three-dimensional model generating device 200 performs alignment so that the end points of the side surfaces match the two-dimensional map (step S82). The process is shown in the figure. The dashed line on the left represents the shape of the building frame obtained from the distribution of the 3D point cloud, and the solid line and points on the right represent the building frame and its vertices on the 2D map. In this example, side end points W1, W2, W3, and W4 obtained from two buildings among the three-dimensional point cloud are matched to a two-dimensional map. The 3D model generating device 200 first arbitrarily associates the end points W1 to W4 with vertices on the 2D map.”
M. Brown, R. Szeliski and S. Winder, "Multi-image matching using multi-scale oriented patches," 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), San Diego, CA, USA, 2005, pp. 510-517 vol. 1.
Section 3, “Since the computational cost of matching is superlinear in the number of interest points, it is desirable to restrict the maximum number of interest points extracted from each image. At the same time, it is important that interest points are spatially well distributed over the image, since for image stitching applications, the area of overlap between a pair of images may be small. To satisfy these requirements, we have developed a novel adaptive non-maximal suppression (ANMS) strategy to select a fixed number of interest points from each image. Interest points are suppressed based on the corner strength fHM, and only those that are a maximum in a neighborhood of radius r pixels are retained. Conceptually, we initialise the suppression radius r = 0 and then increase it until the desired number of interest points nip is obtained. In practice, we can perform this operation without search as the set of interest points which are generated in this way form an ordered list.”
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RAFAEL VELASQUEZ VANEGAS whose telephone number is (571)272-6999. The examiner can normally be reached M-F 8 - 4.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, VIVEK KOPPIKAR can be reached at (571) 272-5109. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RAFAEL VELASQUEZ VANEGAS/Patent Examiner, Art Unit 3667
/VIVEK D KOPPIKAR/Supervisory Patent Examiner
Art Unit 3667
October 10, 2025