DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Remarks
This office action is responsive to the preliminary amendment filed on 04/04/2024.
Claim(s) 1-21 is/are pending in the application.
Independent claim(s) 1 was/were amended.
Dependent claim(s) 9 was/were amended.
Applicant's argument(s), regarding the amended portion(s) of “modifying a model orientation and/or a model position and/or a model pose of a respective trailer model from the at least one read-in model dataset by fitting the respective trailer model to the determined feature representation” as recited in independent claim 1, filed 12/23/2025, have/has been fully considered and is/are not persuasive. Upon further consideration, Examiner still views that the prior art(s) of Kroeze, used in the previous rejection of claim(s) 1, can be relied upon for the aforementioned amended portion(s). To note, applicant's amendment necessitated the updated ground(s) of rejection presented in this office action.
In response to applicant’s arguments, as noted above, arguments are not persuasive. As shown previously and in the updated rejection below, Kroeze discloses in paragraph [0058] that the received image is processed, such as using edge detection techniques, to generate a processed image. The processed image is then compared to the hitch assembly model to determine an angular displacement the hitch assembly elements detected in each image/model. The method is then operative to estimate an HAA in response to the angular displacement. Further in paragraph [0046], Kroeze discloses that the HAA may be estimated by edge matching performed between images by comparison against rotated templates and wherein the HAA is determined by similarity match. Examiner believes that this shows the modification of a model orientation and/or a model pose of a respective trailer mode from the at least one read-in model dataset (i.e. the rotated template), by fitting the respective trailer model to the determined feature representation (i.e. the edge detected image is matched/fitted to that of the rotated template).
Claim Rejections - 35 USC § 102
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claim(s) 1, 7-10, 13-21 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Kroeze et al. (US 2021/0179172 A1).
In regards to claim 1, Kroeze teaches a method for localizing a trailer in the surroundings of a towing vehicle, the method comprising:
reading in at least one single image in which the surroundings of the towing vehicle are imaged in two dimensions (e.g. [0057],Fig.4: once the towing operation has been determined, the method is then operative to receive 430 an image from a rear facing camera; the image may be a single image or a frame of a video stream; Examiner’s note: example shown in Fig.2a);
determining a feature representation from the at least one read-in single image using an image processing algorithm, wherein defined features of the trailer to be localized are reproduced in the feature representation (e.g. [0058],Fig.4: method is next operative to perform 435 image processing techniques on the received image to generate a processed image; in one exemplary embodiment, the image processing techniques may be edge detection techniques or other image processing techniques, similar to those used to generate the hitch assembly model; Examiner’s note: detected edges represent the feature representation; example shown in Fig.2d);
reading in at least one model dataset from a trailer database that includes model datasets from a plurality of trailer models, wherein in each model dataset a defined trailer model is simulated by a model (e.g. [0058],Fig.4: processed image is then compared 440 to the hitch assembly model to determine an angular displacement the hitch assembly elements detected in each image/model; the method is then operative to estimate an hitch articulation angle (HAA) in response to the angular displacement; see also [0047]: an initial learning routine may be started to learn an image featured model for each customer’s trailer regardless of additions to trailer; trailer templates of different hitch articulation angles may then be generated by applying view perspective transformation and image rotation; thus, a template matching step to determine a hitch articulation angle for may be performed for each rear view camera video frame; soft edge matching scores may then be calculated by comparing current image edge map with the learned templates; Examiner’s note: where the trailer templates represent the different hitch assembly models used to determine HAA);
modifying a model orientation and/or a model position and/or a model pose of a respective trailer model from the at least one read-in model dataset for by fitting the respective trailer model to the determined feature representation; and determining: the trailer orientation of the trailer to be localized is determined from the modified model orientation and/or the trailer position of the trailer to be localized from the modified model position and/or the trailer pose of the trailer to be localized from the modified model pose that is then (e.g. as above, [0058],Fig.4: processed image is then compared 440 to the hitch assembly model to determine an angular displacement the hitch assembly elements detected in each image/model; the method is then operative to estimate an hitch articulation angle (HAA) in response to the angular displacement; see also [0046]: the HAA may be estimated by edge matching performed between images by comparison against rotated templates and wherein the HAA is determined by similarity match; Examiner’s note: this shows template models are rotated for comparison purposes in order to determine HAA).
In regards to processing unit claim 20, claim(s) 20 recite(s) limitations that is/are similar in scope to the limitations recited in claim 1. Therefore, claim(s) 20 is/are subject to rejections under the same rationale as applied hereinabove for claim 1. To note, paragraph [0048] discloses the use of a processor.
In regards to claim 7, Kroeze teaches a method, wherein an edge representation is determined as the feature representation, wherein at least trailer edges are reproduced in the edge representation as defined features of the trailer to be localized (e.g. as above, [0058],Fig.4: perform 435 image processing techniques on the received image to generate a processed image; the image processing techniques may be edge detection techniques).
In regards to claim 8, Kroeze teaches a method, wherein the trailer edges of the edge representation are determined from at least one read-in single image using an edge algorithm (e.g. as above, [0058],Fig.4: perform 435 image processing techniques on the received image to generate a processed image; the image processing techniques may be edge detection techniques).
In regards to claim 9, Kroeze teaches a method, wherein an edge-model dataset is read in as the model dataset, wherein the respective trailer model is described in the edge-model dataset by model edges which are characteristic for the simulated trailer model (e.g. as above, [0047]: trailer templates of different hitch articulation angles may then be generated by applying view perspective transformation and image rotation; thus, a template matching step to determine a hitch articulation angle for may be performed for each rear view camera video frame; soft edge matching scores may then be calculated by comparing current image edge map with the learned templates).
In regards to claim 10, Kroeze teaches a method, wherein the trailer edges in the determined edge representation and/or the model edges in the read-in edge-model dataset are described two-dimensionally or three-dimensionally (e.g. [0055],Fig.4: generate 420 a model of the trailer hitch assembly; receive an image from a camera and to perform image recognition or image processing techniques to determine a physical model of the trailer hitch assembly; the image processing may include a perspective transformation or edge detection; Examiner’s note: example shown in Fig.2d, viewed as being in two-dimensions; model of trailer hitch assembly corresponding to the trailer templates generated).
In regards to claim 13, Kroeze teaches a method, wherein the simulated trailer model is represented in the respective at least one model dataset in scaled form (e.g. as above, [0047]: trailer templates of different hitch articulation angles may then be generated by applying view perspective transformation and image rotation).
In regards to claim 14, Kroeze teaches a method, wherein the modification of a model orientation and/or a model position and/or a model pose of the respective trailer model is carried out by applying a geometric transformation to a particular model dataset of the at least one model datasets read in (e.g. as above, [0046]: the HAA may be estimated by edge matching performed between images by comparison against rotated templates and wherein the HAA is determined by similarity match).
In regards to claim 15, Kroeze teaches a method, wherein the trailer orientation of the trailer to be localized and/or the trailer position of the trailer to be localized and/or the trailer pose of the trailer to be localized are determined from the geometric transformation, the application of which fits the respective trailer model to the determined feature representation (e.g. as above, [0046]: the HAA may be estimated by edge matching performed between images by comparison against rotated templates and wherein the HAA is determined by similarity match).
In regards to claim 16, Kroeze teaches a method, wherein the fitting of the respective trailer model to the determined feature representation is carried out in a series of iteration steps, wherein the model orientation and/or the model position and/or the model pose of the respective trailer model is iteratively modified in the respective iteration steps (e.g. as above, [0046]: the HAA may be estimated by edge matching performed between images by comparison against rotated templates and wherein the HAA is determined by similarity match; Examiner’s note: where rotation of different templates iteratively performed).
In regards to claim 17, Kroeze teaches a method, wherein the iterative fitting of the respective trailer model to the determined feature representation is terminated when an exit criterion is reached, wherein the exit criterion is satisfied when an iteration number of iteration steps has been reached and/or a mean distance between the trailer model of the respective model dataset and the respectively assigned features in the respective feature representation falls below a limit distance (e.g. as above, [0046]: the HAA may be estimated by edge matching performed between images by comparison against rotated templates and wherein the HAA is determined by similarity match; Examiner’s note: similarity match shows limit distance determination).
In regards to claim 18, Kroeze teaches a method, wherein a chassis and/or a platform and/or a coupling are simulated as a model by means of the trailer model of the particular model dataset read in (e.g. [0055],Fig.4: generate 420 a model of the trailer hitch assembly; Examiner’s note: trailer hitch viewed as coupling).
In regards to claim 19, Kroeze teaches a method, wherein an articulation angle of the trailer relative to a towing vehicle and/or a trajectory for an approach of the towing vehicle to the trailer are determined from the determined trailer orientation and/or trailer position and/or trailer pose of the trailer (e.g. as above, [0058],Fig.4: processed image is then compared 440 to the hitch assembly model to determine an angular displacement the hitch assembly elements detected in each image/model; the method is then operative to estimate an hitch articulation angle (HAA) in response to the angular displacement).
In regards to claim 21, Kroeze teaches a vehicle comprising the processing unit of claim 20 (e.g. [0048],Fig.3: system for determining a trailer hitch articulation angle in a motor vehicle 300; system 300 includes a processor 320, a vehicle controller 345, a trailer interface module 350, video controller 315, and a camera 310).
Claim Rejections - 35 USC § 103
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claim(s) 2-3, 5-6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kroeze as applied to claim 1 above, and further in view of Jin et al. (US 2021/0349217 A1).
In regards to claim 2, Kroeze teaches the method of claim 1 (Examiner’s note: Kroeze discloses in paragraph [0060] that the camera may be a LiDAR system), but does not explicitly teach the method, wherein a point cloud representation is determined as the feature representation, wherein the point cloud representation contains a point cloud of a plurality of object points in the surroundings, wherein at least some of the object points are assigned to the trailer in the surroundings.
However, Jin teaches a method, wherein a point cloud representation is determined as the feature representation, wherein the point cloud representation contains a point cloud of a plurality of object points in the surroundings, wherein at least some of the object points are assigned to the trailer in the surroundings (e.g. [0036]-[0038],Fig.2: at step 302, the multi-line LiDAR provided on each of the two sides of the tractor is controlled to emit laser light, such that a surface of the trailer reflects the laser light emitted by the multi-line LiDAR; at step 303, each of the multi-line LiDARs is controlled to receive a corresponding laser point cloud reflected by the surface of the trailer; at step 304, a trailer angle is calculated based on the corresponding laser point clouds received by the respective multi-line LiDARs and the initial point cloud data using a point cloud matching algorithm).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings/combination of Kroeze to use point cloud as the feature representation, in the same conventional manner as taught by Jin as both deal with determining a trailer angle. The motivation to combine the two would be that it would allow the use of point cloud data for matching in order to determine trailer angle.
In regards to claim 3, the combination of Kroeze and Jin teaches a method, wherein in the point cloud representation at least the three-dimensional trailer shape of the trailer to be localized is reproduced as a defined feature (e.g. Jin as above, [0036]-[0038],Fig.2: at step 302, the multi-line LiDAR provided on each of the two sides of the tractor is controlled to emit laser light, such that a surface of the trailer reflects the laser light emitted by the multi-line LiDAR; at step 303, each of the multi-line LiDARs is controlled to receive a corresponding laser point cloud reflected by the surface of the trailer).
In addition, the same rationale/motivation of claim 2 is used for claim 3.
In regards to claim 5, the combination of Kroeze and Jin teaches a method, wherein a 3D model dataset is read in as the model dataset, wherein the respective trailer model is described in the 3D model dataset in three dimensions by means of model points (e.g. Jin as above, [0036]-[0038],Fig.2: at step 304, a trailer angle is calculated based on the corresponding laser point clouds received by the respective multi-line LiDARs and the initial point cloud data using a point cloud matching algorithm; Examiner’s note: shows matching with initial point cloud data (read-in model dataset)).
In addition, the same rationale/motivation of claim 2 is used for claim 5.
In regards to claim 6, the combination of Kroeze and Jin teaches a method, wherein the model points of the respective 3D model dataset are brought into overlap with the object points in the point cloud representation for fitting the respective trailer model to the determined point cloud representation (e.g. Jin as above, [0036]-[0038],Fig.2: at step 304, a trailer angle is calculated based on the corresponding laser point clouds received by the respective multi-line LiDARs and the initial point cloud data using a point cloud matching algorithm; Examiner’s note: point cloud matching shows determination of comparison/overlap).
In addition, the same rationale/motivation of claim 5 is used for claim 5.
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Kroeze and Jin as applied to claim 2 above, and further in view of Fan et al. (US 2015/0339541 A1).
In regards to claim 4, the combination of Kroeze and Jin teaches the method of claim 2, but does not explicitly teach the method, wherein the point cloud of the point cloud representation is determined using a Structure from Motion (SfM) algorithm, wherein depth information is determined by triangulation for a plurality of object points in the surroundings from at least two read-in single images using the SfM algorithm and the point cloud is generated from the plurality of object points as a function of the respectively determined depth information.
However, Fan teaches a method, wherein the point cloud of the point cloud representation is determined using a Structure from Motion (SfM) algorithm, wherein depth information is determined by triangulation for a plurality of object points in the surroundings from at least two read-in single images using the SfM algorithm and the point cloud is generated from the plurality of object points as a function of the respectively determined depth information (e.g. [0025]: in Structure-From-Motion (SFM), three-dimensional structures are estimated from two-dimensional image sequences, where the observer and/or the objects to be observed move in relation to each other; the obtained geometric models are stored as 3D point cloud; Examiner’s note: the three dimensions, including depth, would be estimated based on the different images).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings/combination of Kroeze and Jin to determine point clouds, in the same conventional manner as taught by Fan as both deal with generating point cloud for point cloud matching. The motivation to combine the two would be that it would allow the generation of point clouds using multiple images.
Claim(s) 11-12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kroeze as applied to claim 1 above, and further in view of Ramamurthi et al. (US 2011/0231162 A1).
In regards to claim 11, Kroeze teaches the method of claim 1, but does not explicitly teach the method, wherein an intensity representation is determined as the feature representation by using an intensity algorithm, wherein in the intensity representation at least trailer intensity values are also reproduced in a spatially resolved manner as defined features of the trailer to be localized.
However, Ramamurthi teaches a method, wherein an intensity representation is determined as the feature representation by using an intensity algorithm, wherein in the intensity representation at least image intensity values are also reproduced in a spatially resolved manner as defined features of the image to be localized (e.g. [0023]: a pixel-by-pixel comparison of the images may be performed to identify differences in pixel intensity value, and a model may be determined to be a best fit match if the differences in pixel intensity value for the image(s) are below a pre-identified threshold).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings/combination of Kroeze to model match, in the same conventional manner as taught by Ramamurthi as both deal with matching models based on feature representations. The motivation to combine the two would be that it would allow the matching of models based on corresponding spatially determined intensity values.
In regards to claim 12, the combination of Kroeze and Ramamurthi teaches a method, wherein an intensity-model dataset is read in as the model dataset, wherein the respective trailer model is described in the intensity-model dataset in a spatially resolved manner by model intensity values which are characteristic for the simulated trailer model (e.g. Ramamurthi as above, [0023]: a pixel-by-pixel comparison of the images may be performed to identify differences in pixel intensity value, and a model may be determined to be a best fit match if the differences in pixel intensity value for the image(s) are below a pre-identified threshold).
In addition, the same rationale/motivation of claim 11 is used for claim 12.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JED-JUSTIN IMPERIAL whose telephone number is (571)270-5807. The examiner can normally be reached Monday to Friday, 9am - 6pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Hajnik can be reached at (571) 272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JED-JUSTIN IMPERIAL/ Examiner, Art Unit 2616
/DANIEL F HAJNIK/ Supervisory Patent Examiner, Art Unit 2616
Prosecution Insights