Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicant's amendment filed on June 10, 2026 has been entered. Claims 1, 2, 4, 5, 7 – 9, 12, 13, 18, 19, 21 and 22 have been amended. Claims 6 and 14 have been canceled. No claims have been added. Claims 1 – 5, 7 – 13 and 15 – 22 are still pending in this application, with claims 1, 9 and 18 being independent.
Response to Arguments
Regarding rejections under 35 USC § 102/103
The Applicant alleges: “
Applicant respectfully submits that the Office has now shown that Yang teaches or suggests, at least, "determining, based at least on one or more points represented by the depth data that are located between the first plane and the second plane of the 3D shape, a planar surface that is located between the first plane and the second plane of the 3D shape and associated with the traffic sign; determining, based at least on the one or more points, an orientation associated with the planar surface; projecting first vertices associated with the bounding shape from the image to the planar surface to determine second vertices on the planar surface; determining a boundary of the traffic sign on the planar surface using the second vertices; [and] determining a 3D location associated with the traffic sign that is defined by the boundary and the orientation of the planar surface," as amended claim 1 recites.
In the rejection of previously presented independent claim 1, the Office cites Yang as allegedly teaching, "determining, based at least on the one or more points, a planar surface that is located within the 3D shape associated with the traffic sign; based at least on the determining the planar surface, projecting first vertices associated with the bounding shape from the image to determine second vertices associated with on the planar surface; determining, based at least on the planar surface and the second vertices, a 3D location associated with the traffic sign." Office Action, pp. 4 and 5. However, Yang states:
…
Yang, para. [0138].
As shown, Yang describes initially determining a bounding box 1220 in image space, where each vertex of the bounding box 1220 establishes a ray from a camera and the rays create a frustum. Id. Yang then describes using a closest LiDAR point to determine a minimum depth plane and a maximum plane within the frustum. Id. Additionally, Yang describes filtering out points that fall outside of the minimum depth plane and the maximum depth plane of the frustum, where the remaining points correspond to the traffic sign. Id. However, the Office has not shown that this paragraph then describes determining another "planar surface" using the filtered points that is located between the minimum depth plane and the maximum depth plane. Rather, the bounding box 1220, the minimum depth plane, and the maximum depth plane are all initially determined before the filtering of the points. For a specific example, the minimum depth plane and/or the maximum depth plane, which the Office cites as teaching the "planar surface" of previously presented independent claim 1, are actually determined using the closest LiDAR point, which is actually located outside of the minimum depth plane and the maximum depth plane within the frustum.
Therefore, the Office further has also not shown that this paragraph of Yang describes "projecting first vertices associated with [a] bounding shape from [an] image to the planar surface to determine second vertices associated with the planar surface," as the Office has not shown that Yang teaches or suggests such as "planar surface." Additionally, and for similar reasons, the Office has not shown that Yang teaches or suggests determining a "boundary" of the traffic sign on such a "planar surface" using the projected vertices. Furthermore, the Office has not shown that Yang teaches or suggests determining an "orientation" of the traffic sign using the filtered points and/or determining a 3D location of the traffic sign using such a "boundary" and an "orientation."”
Examiner’s response:
The Examiner respectfully disagrees.
Yang teaches: “
determining, based at least on one or more points represented by the depth data that are located between the first plane and the second plane of the 3D shape, a planar surface that is located between the first plane and the second plane of the 3D shape and associated with the traffic sign ([0139 - 0140], determine a reduced fitted plane 1430 between 1230 and 1240);
determining, based at least on the one or more points, an orientation associated with the planar surface ([0127 – 0128, 0134], talked about determining an orientation associated with the plane);
projecting first vertices associated with the bounding shape from the image to the planar surface to determine second vertices associated with on the planar surface ([0140], determine second vertices 1410 associated with on the planar surface; [0138] talked about ray projecting);
determining a boundary of the traffic sign on the planar surface using the second vertices ([0140], determine a boundary of 1420);
determining, based at least on the planar surface and the second vertices, a 3D location associated with the traffic sign that is defined by the boundary and the orientation of the planar surface ([0140]; also [0132 – 0133] talked about sign 3D location).
The Applicant further alleges: “
Yang then states:
…
Id., para. [0139].
As shown, Yang then describes determining a minimum depth point from the filtered points and a threshold depth 1320 from the minimum depth point. Id. Additionally, Yang describes selecting the filtered points that are between the minimum depth point and the threshold depth 1320. Id. Again, the Office has not shown that this paragraph of Yang then describes determining a "planar surface" using the selected points that are between the minimum depth plane and the maximum depth plane. Therefore, the Office has also not shown that this paragraph of Yang teaches or suggests the amended features of independent claim 1 described above.”
Examiner’s response:
The Examiner again respectfully disagrees.
Yang teaches: “
determining, based at least on one or more points represented by the depth data that are located between the first plane and the second plane of the 3D shape, a planar surface that is located between the first plane and the second plane of the 3D shape and associated with the traffic sign ([0139 - 0140], determine a reduced fitted plane 1430 between 1230 and 1240).
The Applicant still further alleges: “
Yang also states in cited portions:
…
Id., para. [0140].
As shown, Yang describes that if the sign is considered a plane with some thickness in 3D, then the 3D box is shifted until there is the maximum number of points within the 3D box. Id. However, again, the Office has not shown that Yang then teaches or suggests "projecting ... vertices" to this plane after determining the location of the 3D box. Additionally, and for similar reasons, the Office has not shown that Yang teaches or suggest determining a "boundary" of the Page 18 of sign on the plane using "projected ... vertices." Furthermore, the Office has not shown that Yang teaches or suggests determining an "orientation" of the sign using the points and/or determining a 3D location of the sign using such a "boundary" and an "orientation."
Consequently, the Office has now shown that Yang teaches or suggests "determining, based at least on one or more points represented by the depth data that are located between the first plane and the second plane of the 3D shape, a planar surface that is located between the first plane and the second plane of the 3D shape and associated with the traffic sign; determining, based at least on the one or more points, an orientation associated with the planar surface; projecting first vertices associated with the bounding shape from the image to the planar surface to determine second vertices on the planar surface; determining a boundary of the traffic sign on the planar surface using the second vertices; [and] determining a 3D location associated with the traffic sign that is defined by the boundary and the orientation of the planar surface," as amended claim 1 recites.”
Examiner’s response:
As the Examiner explained above, Yang teaches: “
determining, based at least on the one or more points, an orientation associated with the planar surface ([0127 – 0128, 0134], talked about determining an orientation associated with the plane);
projecting first vertices associated with the bounding shape from the image to the planar surface to determine second vertices associated with on the planar surface ([0140], determine second vertices 1410 associated with on the planar surface; [0138] talked about ray projecting)”.
The Applicant still further alleges: “
Additionally, Applicant respectfully asserts that the Office has now shown that Yang teaches or suggests, "determining one or more intensity values associated with the one or more points, wherein the determining the planar surface is further based at least on the one or more intensity values," as amended claim 8 recites.
For instance, Yang describes intensity values associated with LiDAR data in paragraphs [0182]-[0184]. However, Yang describes that the intensity values are used for generating an occupancy map. Id., paras. [0182]-[0184]. The Office has not shown that Yang teaches or suggests using the intensity values to identify the traffic sign as described in the cited portions above and with regard to independent claim 1. Consequently, the Office has not shown that Yang teaches or suggests "determining one or more intensity values associated with the one or more points, wherein the determining the planar surface is further based at least on the one or more intensity values," as amended claim 8 recites.
Thus, amended independent claim 1 is patentably distinguishable over the cited reference and withdrawal of the rejection is respectfully requested. For at least reasons similar to amended independent claim 1, amended independent claims 9 and 18 are also patentably distinguishable over the cited reference and withdrawal of the rejections is respectfully requested. Thus, amended independent claims 1, 9, and 18, along with each claim depending therefrom rejected under this section, are patentably distinguishable over the cited reference and withdrawal of the rejections is respectfully requested.”
Examiner’s response:
The Examiner respectfully disagrees.
Yang teaches:
determining one or more intensity values associated with the one or more points ([0182 – 0184], LIDAR sensor provides intensity for each point),
wherein the determining the planar surface is further based at least on the one or more intensity values ([0182 – 0184; 0138 - 0140], determining the planar surface is based on each point, thus is further based at least on the one or more intensity values).
Therefore, claims 1, 8, 9 and 18 are still read on by Yang.
Regarding to the rest of claims, the Applicant does not argue about the rest of claims.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1 – 5, 7 – 13, 15 – 20 and 22 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Yang et al. (US Patent Application Publication 2018/0189578, IDS), hereinafter referred as Yang.
Regarding claim 1, Yang discloses a method (abstract) comprising:
receiving image data obtained using one or more image sensors of a machine (Fig. 9, 910, [0112], receiving image data obtained using camera) and depth data obtained using one or more depth sensors of the machine within an environment (Fig. 9, 930, [0114], receiving 930 a depth map including the traffic sign captured by a detection and ranging sensor);
determining, based at least on a bounding shape associated with a traffic sign as depicted by an image represented by the image data, at least a first plane and a second plane corresponding to a three-dimensional (3D) shape associated with the traffic sign within the environment ([0138], determine planes 1230 and 1240);
determining, based at least on one or more points represented by the depth data that are located between the first plane and the second plane of the 3D shape, a planar surface that is located between the first plane and the second plane of the 3D shape and associated with the traffic sign ([0139 - 0140], determine a reduced fitted plane 1430);
determining, based at least on the one or more points, an orientation associated with the planar surface ([0127 – 0128, 0134], talked about determining an orientation associated with the plane);
projecting first vertices associated with the bounding shape from the image to the planar surface to determine second vertices associated with on the planar surface ([0140], determine second vertices 1410 associated with on the planar surface; [0138] talked about ray projecting);
determining a boundary of the traffic sign on the planar surface using the second vertices ([0140], determine a boundary of 1420);
determining, based at least on the planar surface and the second vertices, a 3D location associated with the traffic sign that is defined by the boundary and the orientation of the planar surface ([0140]; also [0132 – 0133] talked about sign 3D location); and
updating a map to indicate at least the 3D location associated with the traffic sign ([0132 – 0133], optimizing sign 3D location in HD map system; (update/optimize) the accuracy of the sign position relative to the OMap (disclosed in [0093 – 0095])).
Regarding claim 2 (depends on claim 1), Yang discloses the method further comprising:
determining, based at least on the bounding shape at least the first plane and the second plane, a frustum associated with the traffic sign, the frustum including the 3D shape ([0138 – 0139]); and
determining that the one or more points correspond to the traffic sign based at least on the one or more points being located within the frustum ([0138 – 0139]).
Regarding claim 3 (depends on claim 1), Yang discloses the method wherein the determining the 3D shape associated with the traffic sign comprises:
the determining the first plane is further based at least on the bounding shape and a set minimum distance ([0138 – 0139]); and
the determining the second plane is further based at least on the bounding shape and a set maximum distance ([0138 – 0139]).
Regarding claim 4 (depends on claim 1), Yang discloses the method wherein the determining the 3D shape associated with the traffic sign comprises further comprising:
determining, based at least on projecting one or more of the first vertices associated with the bounding shape, one or more 3D vertices ([0135 – 0140]);
determining the 3D shape based at least on the one or more 3D vertices, the first plane, and the second plane ([0138 – 0139]); and
determining that the one or more points correspond to the traffic sign based at least on the one or more points being located within the 3D shape ([0135 – 0140]).
Regarding claim 5 (depends on claim 1), Yang discloses the method wherein the determining the planar surface that is located between the first plane and the second plane the 3D shape associated with the traffic sign is based at least on fitting the one or more points to be located on the planar surface ([0140], fitting subset points).
Regarding claim 7 (depends on claim 1), Yang discloses the method further comprising:
determining, based at least on 3D locations associated with points from a point cloud represented by the depth data, that the one or more points are located within between the first plane and the second plane of the 3D shape ([0138 – 0139], determine points within frustum 1250 (between two planes));
and determining that the one or more points are associated with the traffic sign ([0138 – 0140]).
Regarding claim 8 (depends on claim 1), Yang discloses the method further comprising:
determining one or more intensity values associated with the one or more points ([0182 – 0184], LIDAR sensor provides intensity for each point),
wherein the determining the planar surface is further based at least on the one or more intensity values ([0182 – 0184; 0138 - 0140], determining the planar surface is based on each point, thus is further based at least on the one or more intensity values).
Regarding claim 9, Yang discloses a system (Fig. 44) comprising:
one or more processors (Fig. 44, #4402) to:
determine, based at least on a bounding shape associated with a traffic sign as represented by image data obtained using a machine (Fig. 9, 910, [0112], receiving image data obtained using camera), a three-dimensional (3D) shape associated with the traffic sign (Fig. 12), the 3D shape defined using at least a first plane and a second plane located within an environment ([0138], determine planes 1230 and 1240);
determine, based at least on depth data obtained using the machine, that one or more points from a point cloud associated with the depth data are located within the 3D shape (Fig. 9, 930, [0114], receiving 930 a depth map including the traffic sign captured by a detection and ranging sensor);
determine, based at least on the one or more points being located within the 3D shape, an orientation of a planar surface that is located between the first plane and the second plane of the 3D shape and associated with the traffic sign ([0139 - 0140], determine a reduced fitted plane 1430; [0127 – 0128, 0134], talked about determining an orientation associated with the plane);
project first vertices associated with the bounding shape from an image represented by the image data to the planar surface to determine second vertices associated with on the planar surface ([0140], determine second vertices 1410 associated with on the planar surface; [0138] talked about ray projecting);
determine a boundary of the traffic sign on the planar surface as being defined using the second vertices ([0140], determine a boundary of 1420);
determine a 3D location associated with the traffic sign that is defined by the boundary and the orientation ([0140]; also [0132 – 0133] talked about sign 3D location); and
cause a performance of one or more operations based at least on the 3D location associated with the traffic sign ([0132 – 0133], optimizing sign 3D location in HD map system; (update/optimize) the accuracy of the sign position relative to the OMap (disclosed in [0093 – 0095])).
Regarding claims 10 – 13 and 15 – 16, they are corresponding to claims 2 – 5 and 7 – 8, respectively, thus, they are interpreted and rejected for a same reason set forth for claims 2 – 5 and 7 – 8.
Regarding claim 17 (depends on claim 9), Yang discloses the system wherein the system is comprised in at least one of: a control system for an autonomous or semi-autonomous machine ([0066, 0073, 0074, 0083]); a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system implementing one or more large language models (LLMs); a system for performing conversational AI operations; a system for generating synthetic data; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources ([0059]).
Regarding claims 18 – 20, they are corresponding to claims 1, 3 and 17, respectively, thus, they are interpreted and rejected for a same reason set forth for claims 1, 3 and 17.
Regarding claim 22 (depends on claim 1), Yang discloses the method further comprising:
determining a direction of travel of the machine when obtaining at least one of the image data or the depth data ([0065], direction of movement); and
determining a sign orientation associated with the traffic sign based at least on the direction of travel ([0127 – 0128, 0134], talked about determining an orientation associated with the plane),
wherein the 3D location further indicates the sign orientation ([0127 – 0128, 0134]).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yang in view of Agarwal et al. (US Patent Application Publication 2024/0096111), hereinafter referred as Agarwal.
Regarding claim 21 (depends on claim 1), Yang discloses the method wherein the determining the planar surface that is located between the first plane and the second plane of the 3D shape associated with the traffic sign comprises determining the planar surface as being located at substantially between the first plane and the second plane of the 3D shape ([0140]).
However, Yang fails to explicitly disclose the method wherein determining the planar surface as being located at substantially a center of the one or more points.
However, in a similar field of endeavor Agarwal discloses an image processing method (Fig. 5). In addition, Agarwal discloses the method comprising determining the planar surface as being located at substantially a center of the one or more points ([0055], center of bounding box).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Yang, and determining the planar surface as being located at substantially a center of the one or more points. The motivation for doing this is the middle plane of the object can be defined so that location can be more accurate.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to QIAN YANG whose telephone number is (571)270-7239. The examiner can normally be reached on Monday-Thursday 8am-6pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached on 571-270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/QIAN YANG/
Primary Examiner, Art Unit 2677
Prosecution Insights