DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 02/29/2024, 09/15/2025 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are considered by examiner.
Examiner note regarding Drawings
It is the examiner’s opinion any form of photograph image shown in Figures 1, 2, 5, 7 and 8 are necessary and “are the only practicable medium for illustrating the claimed invention” per 37 CFR 1.84(b)(1) because the invention pertains to image-processing technology, and more particularly to “input feature information corresponding to a specific past frame intervals” (specification ¶ [0003]). In each figure, an original image is processed to change details of the image, which may not be captured with sufficient detail in a line drawing to demonstrate the applicant’s invention. Therefore, no drawing objection is raised.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-3, 8-12, 17-18 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claim 1 recites a method of generating pedestrian behavior prediction information (a person can use sensory information to perceive an environment, including pedestrian behavior), the method being performed by a computer (the mind can process environmental information) and comprising:
setting image information from present to a certain time in the past as the target observation image (a person can see and the mind can identify temporal behavior of other people);
extracting multiple visual input feature information and non-visual input feature information from the target observation image (a person can see multiple visual input feature information (environment, persons, objects, time of day) and use additional sensory (smell, sound) data for the mind to analyze and perceive the environment);
grouping the multiple visual input feature information and non-visual input feature information (the mind can identify different data for different groups, such as grouping multiple data to the same object (such as a pedestrian)); and
inputting the grouped feature information to separated processing modules (the mind can simultaneously process different data for multiple environment factors) and
generating pedestrian behavior prediction information by concatenating output results of the processing modules (the mind can predict behavior of a pedestrian based on the compiled environmental sensory data received through sensory systems).
Claim 2 recites the method of claim 1 (as described above), wherein the extracting of the multiple visual input feature information and non-visual input feature information from the target observation image (a person can see multiple visual input feature information (environment, persons, objects, time of day) and use additional sensory (smell, sound) data for the mind to analyze and perceive the environment) comprises:
extracting pose feature information of the pedestrian (the mind can perceive pose information of a person), bounding box information of the pedestrian (the mind can identify the location of a pedestrian (a bounding box is a simple shape that can be identified using paper and pencil, see MPEP §§ 2106.04(a)(2) and 2111), and speed information of the vehicle as the non-visual input feature information (a person can approximate speed of a vehicle through sensory input other than visual, such as sound); and
extracting, as the visual input feature information, local context information (the mind can visually perceive details regarding a single object in the scene, such as the pedestrian), which is an image with a size that is a certain multiple of the bounding box information (the mind can perceive size based on the visualized area of the pedestrian), global context information comprising image segmentation information (the mind can visually perceive multiple objects in a scene simultaneously), scene context information comprising entire area information of the image, local box information obtained from the image information, and local surround information from which the local box information has been removed (the mind can analyze and determine context information of an entire scene, such as an intersection with traffic signals and the associated operations of all relevant objects, including pedestrian, responding to the traffic signals).
Claim 3 recites the method of claim 2 (as described above), wherein the grouping of the multiple visual input feature information and non-visual input feature information (the mind can identify different data for different groups (visual, sound, smell), such as grouping multiple data to the same object (such as a pedestrian)) comprises:
generating the pose feature information of the pedestrian (the mind can perceive pose information of a person), bounding box information of the pedestrian (the mind can identify the location of a pedestrian (a bounding box is a simple shape that can be identified using paper and pencil, see MPEP §§ 2106.04(a)(2) and 2111), and the speed information of the vehicle as first group feature information (a person can approximate speed of a vehicle through sensory input other than visual, such as sound);
generating the local context information that is the image of the bounding box information, which has the predetermined multiple size (the mind can visually perceive details regarding a single object in the scene, and can perceive depth changes, such as the pedestrian while walking), the global context information comprising the image segmentation information (the mind can perceive size based on the visualized area of the pedestrian), and the scene context information comprising the entire area information of the image as second group feature information (the mind can analyze and determine context information of an entire scene, such as an intersection with traffic signals and the associated operations of all relevant objects, including pedestrian, responding to the traffic signals); and
generating the local box information obtained from the image information and the local surround information from which the local box information has been removed as third group feature information (localized data can be analyzed regarding a pedestrian in a region in which the pedestrian is interacting).
Claim 8 recites the method of claim 1 (as described above), further comprising automatically labeling whether the pedestrian is at risk with respect to all frames of the image information based on the generated behavior prediction information of the pedestrian (a person can perceive risk and dangers of a pedestrian in an environment based on the context of the perceived scene and the behavior of the pedestrian relative to other objects).
Claim 9 recites the method of claim 1 (as described above), wherein in the generating pedestrian behavior prediction information, the behavior prediction information of all pedestrians included in the image information is generated in real time (the mind can analyze environmental information in real time to predict behaviors, including risk of a pedestrian in an environment) and simultaneously displayed as video results (considered an insignificant post-solution activity of data output that is standard and routine; see MPEP § 2106.05(g)).
Claim 10 recites a system for generating pedestrian behavior prediction information (a person can use sensory information to perceive an environment, including pedestrian behavior), the system comprising: a communication module configured to receive image information obtained through a camera (the mind can receive environmental imagery perceived by eyes); memory in which a program for generating pedestrian behavior prediction information based on the image information has been stored (the mind understands behavior of other people in given environmental situations and can process environmental information based on customary recognized behaviors); and a processor (the mind can process environmental information) configured to perform steps identical to claim 1 (as described above).
Claim 11 recites the system of claim 10 (as described above), wherein the processor (mind) is configured to perform steps identical to claim 2 (as described above).
Claim 12 recites the system of claim 11 (as described above), wherein the processor (mind) is configured to perform steps identical to claim 3 (as described above).
Claim 17 recites the system of claim 10 (as described above), wherein the processor (mind) is configured to perform steps identical to claim 8 (as described above).
Claim 18 recites the system of claim 10 (as described above), wherein the processor (mind) is configured to perform steps identical to claim 9 (as described above).
The limitations pertaining to perceiving an environment to predict pedestrian behavior are processes that, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. That is, regarding the system, other than reciting generic placeholder-related computer components, such as a computer, processor, memory, or instructions, nothing in the claim elements precludes the steps from practically being performed in the mind. For example in claim 1, language of “setting” refers to recognizing current and previous visual information; “extracting” is to identify multiple pieces of information from a scene; “grouping” is to cluster or organize the multiple pieces of information; “inputting” is to analyze the multiple pieces of information and “generating” is to determine a prediction behavior of the pedestrian. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, these claims each recite an abstract idea.
This judicial exception is not integrated into a practical application. In particular, the system and method claims do not recite any elements which could not be performed in the mind and the claims only recite generic placeholder-related computer components, such as a computer, processor, memory. The computer components are recited at a high-level of generality (i.e., generic computer, processor, memory, for performing a general function of pedestrian behavior prediction, which is described with a high level of generality of automating a manual operation) such that it amounts to no more than mere instructions to apply the exception using a generic computer component. Accordingly, the computer components do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. Therefore, the aforementioned claims are directed to abstract ideas.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element of using a generic placeholder-related computer components, the computer, processor, memory, used to predict pedestrian behavior, amount to no more than mere instructions to apply the exception using generic computer components. Mere instructions to apply an exception using a generic computer component cannot provide an invention concept. The claims are not patent eligible.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1, 10 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Adeli-Mosabbeb et al (US 2021/0103742, hereinafter “Adeli et al” and cited in IDS filed 02/29/2024).
Regarding Claim 1, Adeli et al teach a method of generating pedestrian behavior prediction information (method 400 for predicting a pedestrian’s crossing intent; Fig 4 and ¶ [0050]), the method being performed by a computer (method 400 is executed with prediction circuit 375; Fig 3, 4 and ¶ [0050]) and comprising:
setting image information from present to a certain time in the past as the target observation image (a sequence of images (over time) from plurality of cameras are received from camera and are processed to generate a combined frame, 402; Fig 4 and ¶ [0051]);
extracting multiple visual input feature information and non-visual (non-visual includes pose of pedestrian, bounding box of pedestrian and speed of ego-vehicle, specification ¶ [0054]) input feature information from the target observation image (each frame of the sequence of the images are parsed to identify visual data, such as pedestrian and objects, and non-visual data, such as bounding box for pedestrians, 404; Fig 4 and ¶ [0052]);
grouping the multiple visual input feature information and non-visual input feature information (the parsed data is organized into a pedestrian-centric graph 406; Fig 4 and ¶ [0053]); and
inputting the grouped feature information to separated processing modules (the pedestrian-centric graph are analyzed with a refined pedestrian node feature vector and refined context node feature vector for each frame 406; Fig 4 and ¶ [0054]) and
generating pedestrian behavior prediction information by concatenating output results of the processing modules (the refined pedestrian node feature vector and the refined context node feature vector are concatenated 408; Fig 4 and ¶ [0054]).
Regarding Claim 10, Adeli et al teach a system for generating pedestrian behavior prediction information (prediction circuit 375 executes method 400; Fig 3, 4 and ¶ [0042], [0049]-[0050]), the system comprising: a communication module (“communication module” is element 310, Figure 3 and “receives image information obtained through a camera” and include “a wired” and “a wireless” device, specification ¶ [0036], therefore “communication module” interpreted as the hardware, not signals) configured to receive image information obtained through a camera (communication path 304 receives sensor 320 data, including from camera 210 and may include wireless signals or wired signals; Fig 3 and ¶ [0043], [0046]); memory in which a program for generating pedestrian behavior prediction information based on the image information has been stored (memory 306 may store machine readable instructions to perform pedestrian prediction analysis; Fig 3, 4 and ¶ [0049]); and a processor (prediction circuit 375 and/or processors 305 implement instructions; Fig 3, 4 and ¶ [0048]-[0050]) configured to perform steps identical to claim 1 (as described above).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2, 11 are rejected under 35 U.S.C. 103 as being unpatentable over Adeli et al (US 2021/0103742) in view of Shi et al (US 2023/0150550, cited in IDS filed 09/15/2025) and Li et al (US 2022/0413507).
Regarding Claim 2, Adeli et al teach the method of claim 1 (as described above), including the extracting of the multiple visual input feature information and non-visual input feature information from the target observation image, including bounding box information of the pedestrian (each frame of the sequence of the images are parsed to identify visual data, such as pedestrian and objects, and non-visual data, such as bounding box for pedestrians, 404; Fig 4 and ¶ [0052]).
Adeli et al does not teach extracting pose feature information of the pedestrian, and speed information of the vehicle as the non-visual input feature information; and extracting, as the visual input feature information, local context information, which is an image with a size that is a certain multiple of the bounding box information, global context information comprising image segmentation information, scene context information comprising entire area information of the image, local box information obtained from the image information, and local surround information from which the local box information has been removed.
Shi et al is analogous art pertinent to the technological problem addressed in the current application and teaches extracting pose feature information of the pedestrian (human pose information may be extracted from the image data; ¶ [0015], [0037]), and speed information of the vehicle as the non-visual input feature information (the planning system 160 receives behavior predictions 152 and may generate a new route for the vehicle to reduce vehicle speed (thereby recognizing relative speed information of vehicle and pedestrian) to yield to pedestrian; ¶ [0047]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the current application to combine the teachings of Adeli et al with Shi et al including extracting pose feature information of the pedestrian, and speed information of the vehicle as the non-visual input feature information. By extracting pedestrian data, such as pose, and vehicle data such as speed, behavior prediction is fine-grained in accuracy thereby improving trajectory prediction of the pedestrian and the vehicle to safely and effectively plan a path of travel for the vehicle, as recognized by Shi et al (¶ [0015], [0047]).
Li et al is analogous art pertinent to the technological problem addressed in the current application and teaches extracting, as the visual input feature information (the feature extractor 120 may extract features from image data; Fig 2A and ¶ [0046]-[0048]),
local context information, which is an image with a size that is a certain multiple of the bounding box information (object (local) features 202 from image data include those of the visual features of object within bounding box, where bounding box information includes scale, relative to image; Fig 2A and ¶ [0048]),
global context information comprising image segmentation information (semantic segmentation is performed to the image the scene; Fig 2A and ¶ [0047]),
scene context information comprising entire area information of the image (local context of objects and global context from the whole image are obtained from the visual features; Fig 2A and ¶ [0053]),
local box information obtained from the image information (bounding box information (position, scale) may be determined; Fig 2A and ¶ [0049], [0054]), and
local surround information from which the local box information has been removed (visual features in the scene correspond to detected objects 250b around bounding box object 250a, thereby additional information surrounding a given bounding box; Fig 2A and ¶ [0049]-[0050]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the current application to combine the teachings of Adeli et al with Li et al including extracting, as the visual input feature information, local context information, which is an image with a size that is a certain multiple of the bounding box information, global context information comprising image segmentation information, scene context information comprising entire area information of the image, local box information obtained from the image information, and local surround information from which the local box information has been removed. By analyzing feature data of the image at a global and local context, perception of the environment is determined to detect potential risks, thereby improving motion planning and safety of objects (people and objects in the environment) of an autonomous vehicle, as recognized by Li et al (¶ [0002]).
Claim 11 recites the system of claim 10 (as described above), wherein the processor is configured to perform steps identical to claim 2 (as described above).
Claims 8, 17 are rejected under 35 U.S.C. 103 as being unpatentable over Adeli et al (US 2021/0103742) in view of Hahn et al (US 2022/0242453).
Regarding Claim 8, Adeli et al teach the method of claim 1 (as described above),
Adeli et al does not teach automatically labeling whether the pedestrian is at risk with respect to all frames of the image information based on the generated behavior prediction information of the pedestrian.
Hahn et al is analogous art pertinent to the technological problem addressed in the current application and teaches automatically labeling whether the pedestrian is at risk with respect to all frames of the image information based on the generated behavior prediction information of the pedestrian (labels are generated describing the action (behavior) of the tracked pedestrian using the pedestrian behavior assessment module 130 with tracking module 110 based on action detected over time, which is associated with an awareness (risk) state class; Fig 1 and ¶ [0045]-[0048]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the current application to combine the teachings of Adeli et al with Hahn et al including automatically labeling whether the pedestrian is at risk with respect to all frames of the image information based on the generated behavior prediction information of the pedestrian. By identifying an awareness state of the tracked pedestrian, the safety of the pedestrian with respect to the vehicle is improved in real-time, as recognized by Hahn et al (¶ [0002]-[0003]).
Claim 17 recites the system of claim 10 (as described above), wherein the processor is configured to perform steps identical to claim 8 (as described above).
Claims 9, 18 are rejected under 35 U.S.C. 103 as being unpatentable over Adeli et al (US 2021/0103742) in view of Shi et al (US 2023/0150550, cited in IDS 09/15/2025).
Regarding Claim 9, Adeli et al teach the method of claim 1 (as described above), wherein in the generating pedestrian behavior prediction information (the refined pedestrian node feature vector and the refined context node feature vector are concatenated 408; Fig 4 and ¶ [0054]), the behavior prediction information of all pedestrians included in the image information is generated in real time (the pedestrian-intent prediction modeling is implemented within an ego-centric vehicle 200 and performed during operation (real-time)including detection and analysis for multiple pedestrians in the scene; ¶ [0041], [0050]-[0052]) and simultaneously displayed as video results.
Adeli et al does not explicitly teach the behavior prediction information of all pedestrians included in the image information is generated in real time and simultaneously displayed as video results.
Shi et al is analogous art pertinent to the technological problem addressed in the current application and teaches the behavior prediction information of all pedestrians included in the image information is generated in real time and simultaneously displayed as video results (a user interface system 165 can present information to the driver based on the pedestrian behavior prediction outputs 152, which are displayed on a visual display system of the vehicle 102 to notify the driver; Fig 1 and ¶ [0048]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the current application to combine the teachings of Adeli et al with Shi et al including the behavior prediction information of all pedestrians included in the image information is generated in real time and simultaneously displayed as video results. By displaying pedestrian behavior prediction outputs to the driver in real-time during operation, a driver is alerted to adjust the trajectory of the vehicle, thereby improving safety of the pedestrian(s) and the driver, as recognized by Shi et al (¶ [0047]).
Claim 18 recites the system of claim 10 (as described above), wherein the processor is configured to perform steps identical to claim 9 (as described above).
Allowable Subject Matter
Claims 3-7, 12-16 objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding Claim 3 and claim 12, the following claim limitations, considered in its entirety in combination with all limitations in which it depends, are considered novel over the prior art (citing claim 3 below and recited in parallel in claim 12):
Claim 3. The method of claim 2, wherein the grouping of the multiple visual input feature information and non-visual input feature information comprises:
generating the pose feature information of the pedestrian, the bounding box information of the pedestrian, and the speed information of the vehicle as first group feature information;
generating the local context information that is the image of the bounding box information, which has the predetermined multiple size, the global context information comprising the image segmentation information, and the scene context information comprising the entire area information of the image as second group feature information; and
generating the local box information obtained from the image information and the local surround information from which the local box information has been removed as third group feature information.
Claims 4-7 are dependent on claim 3 and therefore allowable for similar reasons.
Claims 13-16 are dependent on claim 12 and therefore allowable for similar reasons.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Oh et al (US 2025/0037469, application 18/770,900) was identified as a pending application from the same applicant and inventors teaching a pedestrian prediction system and method, similarly to the pending application, but claims focus on generating trajectory prediction and estimating risk of pedestrians, whereas the current application focuses on grouped feature identification from images to predict pedestrian behavior. Both applications will be monitored during prosecution of the current application.
Noy et al (US 2021/0070322) teach a system and method for analyzing human behaviors in the environment of an autonomous vehicle and using the predicted behavior for controlling the vehicle.
Morales Morales et al (US 2021/0192748) teach a system and method for trajectory prediction of dynamic objects in an environment based on bounding box information to identify the object and incorporating road network data and a vehicle data to determine a trajectory of a vehicle with respect to the dynamic object.
Ham et al (CIPF: Crossing Intentional Prediction Network based on Feature Fusion Modules for Improving Pedestrian Safety) was identified as the current application’s associated scientific publication from the same applicant and inventors teaching a pedestrian prediction system and method, with the disclosure made within one year of the application’s priority date.
Singh et al (Multi-Input Fusion for Practical Pedestrian Intention Prediction) teach the analysis of pedestrian intention to predict behavioral action including use of a bounding box to identify the pedestrian and gather both global and local information to determine if the pedestrian will or will not cross the road.
Lorenzo et al (IntFormer: Prediction pedestrian intention with the aid of the Transformer architecture) teach the use of local and global information regarding image and non-image data to determine if a pedestrian will cross the road including detection of the vehicle speed relative to the pedestrian that is identified using a bounding box and keypoints to determine pose and trajectory.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KATHLEEN M BROUGHTON whose telephone number is (571)270-7380. The examiner can normally be reached Monday-Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John Villecco can be reached at (571) 272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KATHLEEN M BROUGHTON/ Primary Examiner, Art Unit 2661