DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Joint Inventors
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). A certified copy of this document has been placed in the file wrapper. As such, the effective filing date of the instant application is considered 11/25/2022, coinciding with the filing date of the Republic of Korea application to which foreign priority was requested.
Response to Amendment
Claims 1, 2, 6, 7, 10, and 15 have been amended. Claims 9 and 18 have been cancelled and claims 19-22 have been added. The 35 U.S.C. 101 rejection has been withdrawn as a result of amendment, and the prior art rejections have been updated in view of amendment.
Response to Arguments
Applicant's arguments filed 11/25/2025 have been considered but are not persuasive.
Applicant first contends that the prior art does not disclose the limitations in claim 1 as amended. Examiner respectfully disagrees, and points to the updated mapping presented below in view of amendment.
Applicant finally contends with respect claim 9, that the object classification and segmentation in Mehnert does not disclose dynamic ROI determination based on driving situation relevance or conditional processing. Examiner respectfully disagrees, and finds that Applicant is overstating the currently claimed language, which merely includes the determination that the object exists within an ROI, and generating images reflecting dynamic characteristics of the object based on the object being in the ROI. These limitations are fully disclosed in Mehnert in at least col. 2, lines 23-54, which outlines dynamic object processing on a region of interest, as best described in the following excerpt from the citation:
In the actual driving operation, this unit may be driven through different surroundings and may create one or multiple image sequence(s), which may be provided here as an image sequence for the described method. Then, at least one trajectory is determined, which is situatable in the robot surroundings. In this connection, a trajectory may be understood to mean a kind of space curve, i.e., a possible track or path of the robot or of another, in particular, dynamic object in or through the environment or vehicle surroundings present in the image sequence. The trajectory may be considered to be situatable when it may be considered as implementable, e.g., with respect to physical boundaries, by an assigned object, i.e., an object which is to follow the trajectory. In other words, at least one arbitrary trajectory is generated for all arbitrary objects which result from the provided image sequence of the robot surroundings recorded therein.
While Applicant argues in Remarks that the ROI is dynamically set, it is not claimed that narrowly, and the broadest interpretation of the claimed language surrounding the instant applications ROI is the region captured in images along the vehicle trajectory in Mehnert, which are the basis of the then determined dynamic object trajectories in said captured region.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-2, 8, 10-11, 17, 19-20, and 22 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Mehnert (US12019414, referred to as Mehnert).
Regarding claim 1: Mehnert discloses: A vehicle comprising: a plurality of sensors configured to obtain surrounding environment information; and a processor operatively coupled to the plurality of sensors, the processor being configured to: determine whether a surrounding obiect exists within a region of interest based on the surrounding environment information; in response to the surrounding object being within the region of interest, generate a plurality of images reflecting dynamic characteristics according to a type of a surrounding object based on the surrounding environment information, determine a collision avoidance trajectory for avoiding collision with the surrounding object based on the plurality of images reflecting dynamic characteristics according to the type of the surrounding object, and take action so that the vehicle can avoid the collision based on the collision avoidance trajectory. ([col. 2, lines 23-54] Initially, an image sequence is provided in which, in general, surroundings, in particular, robot surroundings, are recorded in images. In other words, the image sequence encompasses images of surroundings or an environment in which the robot may be present, move, etc. The image sequence may be recorded in advance with the aid of a vehicle, e.g., which includes an image recording unit, such as a camera, LIDAR sensors etc. In the actual driving operation, this unit may be driven through different surroundings and may create one or multiple image sequence(s), which may be provided here as an image sequence for the described method. Then, at least one trajectory is determined, which is situatable in the robot surroundings. In this connection, a trajectory may be understood to mean a kind of space curve, i.e., a possible track or path of the robot or of another, in particular, dynamic object in or through the environment or vehicle surroundings present in the image sequence. The trajectory may be considered to be situatable when it may be considered as implementable, e.g., with respect to physical boundaries, by an assigned object, i.e., an object which is to follow the trajectory. In other words, at least one arbitrary trajectory is generated for all arbitrary objects which result from the provided image sequence of the robot surroundings recorded therein. For example, on the one hand, all possible movements ( except for the limitation of the accuracy, a finite number of trajectories, i.e., different trajectory configurations) of dynamic objects may be taken into consideration, but also the movement of the robot itself relative to its surroundings. [col. 7, lines 54-59] It includes one or multiple multilayer ANN(s), for example, which is able to generate an output in the form of 55 signals for a device for controlling an at least semi-autonomous robot. This output may prompt the device for controlling the robot to activate actuators and similar units to automatically carry out computer-controlled movements.)
Regarding claim 2: Mehnert discloses: The vehicle of claim 1,
Mehnert further discloses: wherein the processor is configured to: determine a display form of the surrounding object based on the type of the surrounding object; and perform control such that the surrounding object is displayed according to the determined display form for the generating the plurality of images, wherein the type of the surrounding object includes one of or any combination of another vehicle, a pedestrian, a bicycle, and an electric scooter. ([col. 2, lines 23-54] Initially, an image sequence is provided in which, in general, surroundings, in particular, robot surroundings, are recorded in images. In other words, the image sequence encompasses images of surroundings or an environment in which the robot may be present, move, etc. The image sequence may be recorded in advance with the aid of a vehicle, e.g., which includes an image recording unit, such as a camera, LIDAR sensors etc. In the actual driving operation, this unit may be driven through different surroundings and may create one or multiple image sequence(s), which may be provided here as an image sequence for the described method. Then, at least one trajectory is determined, which is situatable in the robot surroundings. In this connection, a trajectory may be understood to mean a kind of space curve, i.e., a possible track or path of the robot or of another, in particular, dynamic object in or through the environment or vehicle surroundings present in the image sequence. The trajectory may be considered to be situatable when it may be considered as implementable, e.g., with respect to physical boundaries, by an assigned object, i.e., an object which is to follow the trajectory. In other words, at least one arbitrary trajectory is generated for all arbitrary objects which result from the provided image sequence of the robot surroundings recorded therein. For example, on the one hand, all possible movements ( except for the limitation of the accuracy, a finite number of trajectories, i.e., different trajectory configurations) of dynamic objects may be taken into consideration, but also the movement of the robot itself relative to its surroundings. [col. 6, lines 19-30] To be able to classify objects, such as obstacles or other road users, and/or features of the vehicle surroundings
recorded in the image sequence, the prediction may include the generation of a semantic segmentation of at least several individual images of the image sequence. In this way, both
the driving events may be predicted more accurately, and their assessment may be improved. Of course the methods of depth prediction and of semantic segmentation may be combined, so that the prediction accuracy may be improved yet again. The estimation from the optical flow may thus, for
example, only be carried out for certain classes of the semantic segmentation, such as for dynamic objects.)
Regarding claim 8: Mehnert discloses: The vehicle of claim 1,
Mehnert further discloses: wherein the processor is configured to: predict a positional change of the surrounding object based on the surrounding environment information; generate trajectories of the vehicle for avoiding a collision; and generate the plurality of images reflecting dynamic characteristics according to the type of the surrounding object, based on the positional change of the surrounding object and the generated trajectories of the vehicle for avoiding a collision. ([col. 2, lines 23-54] Initially, an image sequence is provided in which, in general, surroundings, in particular, robot surroundings, are recorded in images. In other words, the image sequence encompasses images of surroundings or an environment in which the robot may be present, move, etc. The image sequence may be recorded in advance with the aid of a vehicle, e.g., which includes an image recording unit, such as a camera, LIDAR sensors etc. In the actual driving operation, this unit may be driven through different surroundings and may create one or multiple image sequence(s), which may be provided here as an image sequence for the described method. Then, at least one trajectory is determined, which is situatable in the robot surroundings. In this connection, a trajectory may be understood to mean a kind of space curve, i.e., a possible track or path of the robot or of another, in particular, dynamic object in or through the environment or vehicle surroundings present in the image sequence. The trajectory may be considered to be situatable when it may be considered as implementable, e.g., with respect to physical boundaries, by an assigned object, i.e., an object which is to follow the trajectory. In other words, at least one arbitrary trajectory is generated for all arbitrary objects which result from the provided image sequence of the robot surroundings recorded therein. For example, on the one hand, all possible movements ( except for the limitation of the accuracy, a finite number of trajectories, i.e., different trajectory configurations) of dynamic objects may be taken into consideration, but also the movement of the robot itself relative to its surroundings. [col. 6, lines 19-30] To be able to classify objects, such as obstacles or other road users, and/or features of the vehicle surroundings recorded in the image sequence, the prediction may include the generation of a semantic segmentation of at least several individual images of the image sequence. In this way, both
the driving events may be predicted more accurately, and their assessment may be improved. Of course the methods of depth prediction and of semantic segmentation may be combined, so that the prediction accuracy may be improved yet again. The estimation from the optical flow may thus, for
example, only be carried out for certain classes of the semantic segmentation, such as for dynamic objects.)
Regarding claim 10: Rejected using the same rationale as claim 1.
Regarding claim 11: Rejected using the same rationale as claim 2.
Regarding claim 17: Rejected using the same rationale as claim 8.
Regarding claim 19: Rejected using the same rationale as claim 1, however additionally directed to “a method of autonomously operating a vehicle”, which is further disclosed by Mehnert: a method of autonomously operating a vehicle ([col. 1, lines 36-38] This ANN is trained using training data sets to gradually teach the AI module how to move, e.g., drive, autonomously in a roadworthy manner.)
Regarding claim 20: Rejected using the same rationale as claims 2 and 11.
Regarding claim 22: Rejected using the same rationale as claims 8 and 17.
Claims 3-5 and 12-14 are rejected under 35 U.S.C. 103 as being unpatentable over Mehnert et al. (US12019414, referred to as Mehnert) in view of Nakanishi (JP2020154831A, referred to as Nakanishi).
Regarding claim 3: Mehnert discloses: The vehicle of claim 2,
Mehnert does not explicitly disclose: determine the display form of the surrounding object as an ellipse in response to the surrounding object being the pedestrian, and determine one of or any combination of a heading angle, a length, and a width, of the ellipse based on one of or both of a magnitude of a current movement speed of the pedestrian and a current movement direction of the pedestrian.
Mehnert does not disclose the following limitations, however Nakanishi, in an analogous field of endeavor teaches: wherein the processor is configured to: determine the display form of the surrounding object as an ellipse in response to the surrounding object being the pedestrian, and determine one of or any combination of a heading angle, a length, and a width, of the ellipse based on one of or both of a magnitude of a current movement speed of the pedestrian and a current movement direction of the pedestrian. ([pg. 4, lines 1-20] In the captured image 106 shown in FIG. 4, a moving object and a shadow of the moving object are displayed, and an example of a feature point and an optical flow in the moving object is shown. In the captured image 106, the elliptical portion drawn by the solid line indicates the three-dimensional object 107 which is a moving object, and the elliptical portion drawn by the broken line indicates the shadow 108 of the three-dimensional object 107. The three-dimensional object 107 corresponds to, for example, a human being or an automobile. A plurality of black circles drawn on the contours of the three-dimensional object 107 and the shadow 108 of the three-dimensional object 107 each indicate a feature point 221. As shown in FIG. 4, feature points 221 are extracted from both the three-dimensional object 107 and the shadow 108 of the three-dimensional object 107. The arrows drawn at each feature point 221 indicate the optical flow 222 with respect to the movement of the three-dimensional object 107 for each feature point. The magnitude and direction of the arrows in the optical flow 222 simulate the movement amount (or velocity) and movement direction of the related feature points 221 respectively. As shown in FIG. 4, when a three-dimensional object 107 moves, its shadow 108 also moves at the same speed, so that the feature points 221 move at similar speeds, and the optical flow 222 of the feature points 221 is also similar. The size and orientation to be obtained. In addition, another three-dimensional object 109 is also shown in FIG. The relationship between the shadow 110, the feature point 223, and the optical flow 224 of the three-dimensional object 109 corresponds to the relationship of the shadow 108, the feature point 221 and the optical flow 222 of the three-dimensional object related to the three-dimensional object 107. Even if there are a plurality of moving objects in the captured image in this way, feature points can be extracted for each and the optical flow can be calculated.)
Mehnert and Nakanishi are analogous art to the claimed invention since they are from the similar field of vehicle object detection and collision avoidance. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention, with a reasonable expectation for success, to modify the imaging system of Mehnert to enable the dynamic elliptical pedestrian representation taught in Nakanishi.
The motivation for modification would have been to provide the dynamic representation method disclosed in Mehnert with the further dynamic elliptical pedestrian representation taught in Nakanishi.
Regarding claim 4: Mehnert discloses: The vehicle of claim 3,
Mehnert does not explicitly disclose the following limitations, however Nakanishi further teaches: wherein the processor is configured to: determine a possibility of movement in a direction other than the current movement direction based on posture information of the pedestrian; and change the display form of the ellipse based on the possibility of movement in the other direction. ([pg. 4, lines 1-20] In the captured image 106 shown in FIG. 4, a moving object and a shadow of the moving object are displayed, and an example of a feature point and an optical flow in the moving object is shown. In the captured image 106, the elliptical portion drawn by the solid line indicates the three-dimensional object 107 which is a moving object, and the elliptical portion drawn by the broken line indicates the shadow 108 of the three-dimensional object 107. The three-dimensional object 107 corresponds to, for example, a human being or an automobile. A plurality of black circles drawn on the contours of the three-dimensional object 107 and the shadow 108 of the three-dimensional object 107 each indicate a feature point 221. As shown in FIG. 4, feature points 221 are extracted from both the three-dimensional object 107 and the shadow 108 of the three-dimensional object 107. The arrows drawn at each feature point 221 indicate the optical flow 222 with respect to the movement of the three-dimensional object 107 for each feature point. The magnitude and direction of the arrows in the optical flow 222 simulate the movement amount (or velocity) and movement direction of the related feature points 221 respectively. As shown in FIG. 4, when a three-dimensional object 107 moves, its shadow 108 also moves at the same speed, so that the feature points 221 move at similar speeds, and the optical flow 222 of the feature points 221 is also similar. The size and orientation to be obtained. In addition, another three-dimensional object 109 is also shown in FIG. The relationship between the shadow 110, the feature point 223, and the optical flow 224 of the three-dimensional object 109 corresponds to the relationship of the shadow 108, the feature point 221 and the optical flow 222 of the three-dimensional object related to the three-dimensional object 107. Even if there are a plurality of moving objects in the captured image in this way, feature points can be extracted for each and the optical flow can be calculated.)
As previously stated, Mehnert and Nakanishi are analogous art to the claimed invention since they are from the similar field of vehicle object detection and collision avoidance. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention, with a reasonable expectation for success, to modify the imaging system of Mehnert to enable the dynamic elliptical pedestrian representation taught in Nakanishi.
The motivation for modification would have been to provide the dynamic representation method disclosed in Mehnert with the further dynamic elliptical pedestrian representation taught in Nakanishi.
Regarding claim 5: Mehnert discloses: The vehicle of claim 3,
Mehnert does not explicitly disclose the following limitations, however Nakanishi further teaches: wherein the processor is configured to: determine a possibility of movement in a direction other than the current movement direction based on posture information of the pedestrian; and further display an additional ellipse indicating the possibility of movement in the other direction. ([pg. 4, lines 1-20] In the captured image 106 shown in FIG. 4, a moving object and a shadow of the moving object are displayed, and an example of a feature point and an optical flow in the moving object is shown. In the captured image 106, the elliptical portion drawn by the solid line indicates the three-dimensional object 107 which is a moving object, and the elliptical portion drawn by the broken line indicates the shadow 108 of the three-dimensional object 107. The three-dimensional object 107 corresponds to, for example, a human being or an automobile. A plurality of black circles drawn on the contours of the three-dimensional object 107 and the shadow 108 of the three-dimensional object 107 each indicate a feature point 221. As shown in FIG. 4, feature points 221 are extracted from both the three-dimensional object 107 and the shadow 108 of the three-dimensional object 107. The arrows drawn at each feature point 221 indicate the optical flow 222 with respect to the movement of the three-dimensional object 107 for each feature point. The magnitude and direction of the arrows in the optical flow 222 simulate the movement amount (or velocity) and movement direction of the related feature points 221 respectively. As shown in FIG. 4, when a three-dimensional object 107 moves, its shadow 108 also moves at the same speed, so that the feature points 221 move at similar speeds, and the optical flow 222 of the feature points 221 is also similar. The size and orientation to be obtained. In addition, another three-dimensional object 109 is also shown in FIG. The relationship between the shadow 110, the feature point 223, and the optical flow 224 of the three-dimensional object 109 corresponds to the relationship of the shadow 108, the feature point 221 and the optical flow 222 of the three-dimensional object related to the three-dimensional object 107. Even if there are a plurality of moving objects in the captured image in this way, feature points can be extracted for each and the optical flow can be calculated.)
As previously stated, Mehnert and Nakanishi are analogous art to the claimed invention since they are from the similar field of vehicle object detection and collision avoidance. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention, with a reasonable expectation for success, to modify the imaging system of Mehnert to enable the dynamic elliptical pedestrian representation taught in Nakanishi.
The motivation for modification would have been to provide the dynamic representation method disclosed in Mehnert with the further dynamic elliptical pedestrian representation taught in Nakanishi.
Regarding claim 12: Rejected using the same rationale as claim 3.
Regarding claim 13: Rejected using the same rationale as claim 4.
Regarding claim 14: Rejected using the same rationale as claim 5.
Claims 6-7, 15-16, and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Mehnert et al. (US12019414, referred to as Mehnert) in view of Liu et al. (WO2023221067A1, referred to as Liu).
Regarding claim 6: Mehnert discloses: The vehicle of claim 1,
Mehnert does not explicitly disclose the following limitations, however Liu, from an analogous field of endeavor teaches: wherein the processor is configured to: obtain line information from the surrounding environment information; and perform control such that a line is displayed in a specified first color for the generating the plurality of images. ([pg. 12, lines 7-9] Optionally, when the visual signal indicates the degree of collision risk of the first vehicle through lines, this application does not limit the number of lines. For example, the visual signal can use a single line, a double line (line 03 in Figure 5c) or three lines. The visual element of the line indicates the degree of collision risk. [pg. 12, line 27] The wavelength of light in the color of the signal mark is inversely related to the risk of a vehicle collision;)
As previously stated, Mehnert and Nakanishi are analogous art to the claimed invention since they are from the similar field of vehicle object detection and collision avoidance. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention, with a reasonable expectation for success, to modify the imaging system of Mehnert to enable the visual aid image processing brightness/color gradients taught in Liu.
The motivation for modification would have been to provide the dynamic representation method disclosed in Mehnert with the common visual gradient representations taught in Liu.
Regarding claim 7: Mehnert discloses: The vehicle of claim 1,
Mehnert does not explicitly disclose the following limitations, however Liu further teaches: wherein the processor is configured to: calculate a risk level of collision with the surrounding object based on the surrounding environment information; determine a display brightness of the surrounding object according to the calculated risk level of collision; and perform control such that the surrounding object is displayed according to the determined display brightness for the generating the plurality of images. ([pg. 12, lines 7-9] Optionally, when the visual signal indicates the degree of collision risk of the first vehicle through lines, this application does not limit the number of lines. For example, the visual signal can use a single line, a double line (line 03 in Figure 5c) or three lines. The visual element of the line indicates the degree of collision risk. [pg. 12, line 28] The brightness of the signal sign is positively related to the risk of vehicle collision;)
As previously stated, Mehnert and Nakanishi are analogous art to the claimed invention since they are from the similar field of vehicle object detection and collision avoidance. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention, with a reasonable expectation for success, to modify the imaging system of Mehnert to enable the visual aid image processing brightness/color gradients taught in Liu.
The motivation for modification would have been to provide the dynamic representation method disclosed in Mehnert with the common visual gradient representations taught in Liu.
Regarding claim 15: Rejected using the same rationale as claim 6.
Regarding claim 16: Rejected using the same rationale as claim 7.
Regarding claim 21: Rejected using the same rationale as claims 7 and 16.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ATTICUS A CAMERON whose telephone number is 703-756-4535. The examiner can normally be reached M-F 8:30 am - 4:30 pm. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Thomas Worden can be reached on 571-272-4876. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ATTICUS A CAMERON/ /JASON HOLLOWAY/ Primary Examiner, Art Unit 3658
Examiner, Art Unit 3658A
Prosecution Insights