DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 4/1/2026 has been entered.
Response to Arguments
Applicant’s response to the last Office Action, filed 4/1/2026, has been entered and made of record.
Applicant has amended claims 1, 11, and 21. Claims 1-19, and 21-22 are currently pending.
Applicant’s arguments, filed 2/26/2026, with respect to the rejection of claim 1, 11, and 21 under 35 U.S.C. 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground of rejection is made in view of Kim (U.S. Patent Pub. No. 2023/0222671).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-3 and 11-13, and 21-22 are rejected under 35 U.S.C. 103 as being unpatentable over Liu (U.S. Patent Pub. No. 2019/0258876) in view of Ho (U.S. Patent No. 10650278) in view of Kim (U.S. Patent Pub. No. 2023/0222671).
Regarding claim 1, Liu teaches a three-dimensional target detection method, comprising:
obtaining an image and point cloud data of a target environment (Liu ¶33 The trajectory planning system includes a first camera system for obtaining a stereoscopic image of the environment. The first camera system can produce a three-dimensional point cloud of the surrounding environment;)
obtaining semantic information of the image, wherein the semantic information comprises category information corresponding to pixels in the image; and (¶33 The one or more processors can perform operations on the two-dimensional image to obtain boundary boxes for objects in the environment from the two-dimensional image and semantic segmentation of the two-dimensional image to identify and classify objects)
determining three-dimensional location information of a target in the target environment based on the point cloud data, the image, and the semantic information of the image (¶41 FIG. 5 illustrates schematically a system 500 for combining two-dimensional identification, including bounding boxes and semantic classification, to the three-dimensional point cloud to extend a semantic map of a region into three dimensions; ¶44 Once a cluster of points is identified, a representative coordinate 512 of the cluster can be provided to the second module 504. The second module 504 receives the representative coordinate 512 and provides a bounding box 514 that includes or is associated with the representative coordinate to the first module 504. The first module 502 receives the bounding box 514 and applies the bounding box 514 to the 3D point cloud. As a result, the cluster of points 515 are identified with, associated with, or assigned to a three-dimensional bounding box 517.)
Liu does not explicitly disclose obtaining an image from a first device and obtaining point cloud data of a target environment from a second device. While it is common in the art, Liu does not explicitly disclose wherein the semantic information comprises category information corresponding to each pixel in the image.
Ho is in the same field of art of image analysis. Further, Ho teaches obtaining an image from a first device and obtaining point cloud data of a target environment from a second device (Fig. 1, 104 and 102; Col 3-4 Lines 65-2: The system 100 takes as input a three dimensional point cloud 102 of data based, at least in part on, lidar sensor data reflecting objects in a space (e.g., the vicinity of segment of road); Col 4 Lines 14-19: The system 100 also takes as input a set of two dimensional images 104 (e.g., greyscale images or color images) that include views of objects in the space. For example, the set of images 104 may be captured with one or more cameras or other image sensors (e.g., an array of cameras) operating in the same space as the lidar sensor.)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Liu by using two different devices to capture data that is taught by Ho; thus, one of ordinary skilled in the art would be motivated to combine the references to improve predictions projected back to the point cloud (Ho Col 3).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Kim is in the same field of art of image analysis. Further, Kim explicitly teaches wherein the semantic information comprises category information corresponding to each pixel in the image (¶32 The system for predicting a near future location of an object according to the present disclosure generates a background image consisting of only a static object without a dynamic object by pre-processing an image by using a neural network scheme for assigning semantic attributes to each of pixels within the image; ¶44 FIG. 8 is a block diagram illustrating a structure of a segmentation neural network in the present disclosure, and illustrates a structure of a semantic segmentation fully convolutional network (FCN). The semantic segmentation means a task for segmenting all objects within an image in a semantic unit, and is also referred to as dense prediction that predicts labels of all pixels within an image.)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Liu by explicitly finding semantic information for each pixel that is taught by Kim; thus, one of ordinary skilled in the art would be motivated to combine the references to accurately predict a near future location of an object (Kim ¶5).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 2, Liu in view of Kim teaches the method according to claim 1, wherein the determining three-dimensional location information of a target in the target environment based on the point cloud data, the image, and the semantic information of the image comprises (see rejection of claim 1:)
extracting feature information of the semantic point cloud data to generate (Liu, ¶36 In box 210, a region of the point cloud is selected. In box 212, the processor extracts various features from the selected point cloud)
determining the three-dimensional location information of the target in the semantic point cloud data based on the (Liu, ¶36 In box 216, homography computing is performed in which the identified object can be viewed from any selected perspective. The points in the identify object generally having three associate coordinates (x, y, z).)
Liu does not explicitly disclose projecting the image and the semantic information of the image into the point cloud data to generate semantic point cloud data.
Ho is in the same field of art of image analysis. Further, Ho teaches projecting the image and the semantic information of the image into the point cloud data to generate semantic point cloud data (Col 5 Lines 37-40: The 2D-3D projection & accumulation module 130 maps predictions of the semantic labeled image 122 to respective points of the point cloud 102 to obtain a semantic labeled point cloud 132)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Liu by generate semantic point cloud data that is taught by Ho; thus, one of ordinary skilled in the art would be motivated to combine the references to improve predictions projected back to the point cloud (Ho Col 3).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 3, Liu in view of Ho in view of Kim discloses the method according to claim 2, wherein the semantic point cloud feature information is output by using a semantic point cloud feature recognition network (Ho, Col 6 Lines 12-16: The 3D CNN classification module 150 includes a three dimensional convolutional neural network that takes a three dimensional array of predictions for a cluster (e.g., based on the 3D semantic priors for the cluster) as input and outputs a label prediction for the cluster as a whole,) and the three-dimensional location information is output by using a target detection network (Liu, ¶38 The pixels enclosed by the context region 304 are sent to a convolution neural network 306 that extracts a set of feature maps 308 from the context region 304, for each location of the context region 304. The feature maps 308 can be used to determine a bounding box for the object, at box 310. Each bounding box is parametrized by a 5×1 tensor of (x, y, h, w c) where (x, y) indicate the pixel coordinates of the bounding box, (h, w) indicate the height and width of the bounding box, and c is a confidence score. The confidence score indicates how likely a target vehicle or a portion of a target vehicle is to be found inside the bounding box at box 310.)
Regarding claim 11, claim 11 has been analyzed with regard to claim 1 and is rejected for the same reasons of anticipation as used above as well as in accordance with Liu further teaching on: a three-dimensional target detection apparatus, comprising: at least one processor; and a memory coupled to the at least one processor and storing programming instructions for execution by the at least one processor to perform operations (¶39 The controller 34 includes at least one processor 44 and a computer readable storage device or media 46. The processor 44 can be any custom made or commercially available processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor among several processors associated with the controller 34, a semiconductor based microprocessor (in the form of a microchip or chip set), a macroprocessor, any combination thereof, or generally any device for executing instructions.)
Claim 12 recites limitations similar to claim 2 and is rejected under the same rationale and reasoning.
Claim 13 recites limitations similar to claim 3 and is rejected under the same rationale and reasoning.
Regarding claim 21, claim 11 has been analyzed with regard to claim 1 and is rejected for the same reasons of anticipation as used above as well as in accordance with Liu further teaching on: a computer program product comprising computer-executable instructions stored on a non-transitory computer-readable storage medium (¶29 The computer readable storage device or media 46 may include volatile and nonvolatile storage in read-only memory (ROM), random-access memory (RAM), and keep-alive memory (KAM), for example. KAM is a persistent or non-volatile memory that may be used to store various operating variables while the processor 44 is powered down. The computer-readable storage device or media 46 may be implemented using any of a number of known memory devices such as PROMs (programmable read-only memory), EPROMs (electrically PROM), EEPROMs (electrically erasable PROM), flash memory, or any other electric, magnetic, optical, or combination memory devices capable of storing data, some of which represent executable instructions)
Regarding Claim 22, Liu in view of Ho in view of Kim discloses the method according to claim 1, wherein the first device comprises a camera, the image is a two-dimensional image, and the second device comprises a LIDAR (Ho, Fig. 1, 104 and 102; Col 3-4 Lines 65-2: The system 100 takes as input a three dimensional point cloud 102 of data based, at least in part on, lidar sensor data reflecting objects in a space (e.g., the vicinity of segment of road); Col 4 Lines 14-19: The system 100 also takes as input a set of two dimensional images 104 (e.g., greyscale images or color images) that include views of objects in the space. For example, the set of images 104 may be captured with one or more cameras or other image sensors (e.g., an array of cameras) operating in the same space as the lidar sensor.)
Claims 4 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Liu (U.S. Patent Pub. No. 2019/0258876) in view of Ho (U.S. Patent No. 10650278) in view of Kim (U.S. Patent Pub. No. 2023/0222671) in view of Huang (U.S. Patent Pub No. 2022/0188554).
Regarding Claim 4, Liu in view of Ho in view of Kim teaches the method according to claim 3, wherein:
the semantic point cloud feature recognition network comprises a point cloud feature recognition subnetwork (Ho, Col 6 Lines 12-16: The 3D CNN classification module 150 includes a three dimensional convolutional neural network that takes a three dimensional array of predictions for a cluster (e.g., based on the 3D semantic priors for the cluster) as input and outputs a label prediction for the cluster as a whole) and an image feature recognition subnetwork (Ho, Col 5 Lines 22-26: The 2D CNN semantic segmentation module 120 includes a two dimensional convolutional neural network that is trained to receive an augmented image 112 as input and output label predictions for pixels of the augmented image 112;)
the point cloud feature recognition subnetwork is used to extract point cloud feature information of the point cloud data; and (Ho, Col 6 Lines 12-16: The 3D CNN classification module 150 includes a three dimensional convolutional neural network that takes a three dimensional array of predictions for a cluster (e.g., based on the 3D semantic priors for the cluster) as input and outputs a label prediction for the cluster as a whole)
Liu in view of Ho in view of Kim does not explicitly disclose the following the image feature recognition subnetwork is used to extract image feature information of the image based on the image and the semantic information, and dynamically adjust a network parameter of the point cloud feature recognition subnetwork based on the image feature information.
Huang is in the same field of art of image analysis. Further, Huang teaches the image feature recognition subnetwork is used to extract image feature information of the image based on the image and the semantic information, and dynamically adjust a network parameter of the point cloud feature recognition subnetwork based on the image feature information (¶37 Based on the input of the estimated 2D bounding box and an input of the image class label 312 (e.g., 2D ground truth annotation), the 2D loss module 324 may then compare the estimated 2D bounding box generated by the 2D object detector 322 to the image class label 312 (e.g., 2D ground truth annotation) associated with the 2D image 310. In certain embodiments, the 2D loss module 324 may generate, for example, a regression loss … the 2D regression loss may be then utilized in backpropagation to update parameters of the 2D object detector 322, the neural network 318, and the neural network 320 (point cloud network).)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Liu in view of Ho by extracting image features and adjusting the point cloud network parameters based on the feature information that is taught by Huang; thus, one of ordinary skilled in the art would be motivated to combine the references to accurately perceive any objects that may become apparent within its (autonomous vehicle) drive path (Huang ¶2).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim 14 recites limitations similar to claim 4 and is rejected under the same rationale and reasoning.
Claims 9-10 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Liu (U.S. Patent Pub. No. 2019/0258876) in view of Ho (U.S. Patent No. 10650278) in view of Kim (U.S. Patent Pub. No. 2023/0222671) in view of Du (U.S. Patent Pub. No. 2022/0156968).
Regarding Claim 9, Liu in view of Ho in view of Kim teaches the method according to claim 1.
Liu in view of Ho in view of Kim does not explicitly disclose wherein the obtaining semantic information from the image comprises:
performing panoramic segmentation on the image to generate the semantic information of the image, wherein the semantic information comprises a panoramic segmentation image of the image, and the panoramic segmentation image comprises image regions, obtained through panoramic segmentation, of different objects and category information corresponding to the image regions.
Du is in the same field of art of image analysis. Further, Du teaches wherein the obtaining semantic information from the image comprises:
performing panoramic segmentation on the image to generate the semantic information of the image, wherein the semantic information comprises a panoramic segmentation image of the image (¶14 The database creation image may be a panoramic image, a wide-angle image, or the like; ¶202 As shown in FIG. 4, semantic segmentation is performed on the database creation image,) and the panoramic segmentation image comprises image regions, obtained through panoramic segmentation, of different objects and category information corresponding to the image regions (¶202 As shown in FIG. 4, semantic segmentation is performed on the database creation image, to divide the database creation image into six corresponding regions, and semantics of images in the six regions are a pedestrian, a road, a tree, a building, sky, and glass; ¶205 It should be understood that, in the process shown in FIG. 4, semantic segmentation is directly performed on the database creation image, to finally obtain a semantic category (a specific representation form of the semantic information) of the feature point of the database creation image and a confidence degree of the semantic category of the feature point of the database creation image.)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Liu by implementing a panoramic image and performing image segmentation on the image that is taught by Du; thus, one of ordinary skilled in the art would be motivated to combine the references in order to better perform visual positioning (Du ¶6).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 10, Liu in view of Ho in view of Kim in view of Du discloses the method according to claim 1, wherein the image comprises a panoramic image (Du, ¶14 The database creation image may be a panoramic image, a wide-angle image, or the like; ¶202 As shown in FIG. 4, semantic segmentation is performed on the database creation image.)
Claim 19 recites limitations similar to claim 9 and is rejected under the same rationale and reasoning
Allowable Subject Matter
Claims 5-8 and 15-18 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DUSTIN BILODEAU whose telephone number is (571)272-1032. The examiner can normally be reached 9am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DUSTIN BILODEAU/Examiner, Art Unit 2664
/JENNIFER MEHMOOD/Supervisory Patent Examiner, Art Unit 2664
Prosecution Insights