DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 03/06/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Preliminary Amendment
The preliminary amendment filed on 03/06/2024 have been acknowledged.
Claims 1-8 and 10-12 have been currently amended.
Claim 16 has been canceled.
Claims 1-15 are pending.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 2, 7, 13 and 15 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Park et al (Park hereinafter US 20200327317 A1) .
As per claim 1
Park teaches a user detection apparatus comprising at least one memory storing instructions, and at least one processor configured to execute the instructions (Paragraph [0025] “The memory is configured to store the modules and the processor is specifically configured to execute said modules to perform one or more processes which are described further below.”) to detect a predetermined landmark based on an image obtained by photographing a vicinity of a vehicle the predetermined landmark indicating a user and candidate and a getting on place (Paragraph [0036] “The camera 204 may be configured to acquire an image (e.g., still image or video) by photographing a predetermined range in front of the unmanned autonomous vehicle 100. The image acquired through the camera 204 may be analyzed by the controller 202. The controller 202 may be configured to determine whether there are any of the waiting passengers 152 and 154 intending to board at the next stop 150 by analysis of the image. In response to determining that there are the waiting passengers 152 and 154 intending to board…” and Figure 3A-3C and figure 5A ) and determine whether the user candidate is a user of the vehicle or not based on the information on the user candidate and the predetermined land mark detected by the image processing unit (Figure 3A-3C, Paragraph [0041] “ FIG. 3A is a view illustrating a scene of photographing the waiting passenger at the bus stop 150 using the camera 204 of the unmanned autonomous vehicle 100. As shown in FIG. 3A, when the waiting passenger 152 at the next bus stop 150 is photographed using the camera 204 of the unmanned autonomous vehicle 100, the controller 202 may be configured to determine whether the waiting passenger 152 is making a predetermined gesture to indicate the boarding intention by analysis of the photographed image”
As per claim 2
Park teaches an apparatus that detects the state of the user (Fig 5B and Paragraph [0008] “a predetermined indication of the boarding intention is included in a photographed image. The predetermined indication of the boarding intention may include the passenger making a predetermined gesture to indicate the boarding intention; and the unmanned autonomous vehicle may be configured to verify the boarding intention by capturing the gesture of the passenger using the camera.”), and a distance between the user candidate and predetermined landmark “photographing the waiting passenger at the bust stop 150 using camera 204” Since what parameters for “distance” isn’t disclosed in the claim it is reasonable to cite the distance of “at the bus stop” or simply in its general proximity as being satisfactory. You can see this proximity to the landmark in figure 1. This also satisfies the indication if the user is or is not a user candidate for pick up by the bus based on their state and proximity to bus stop.
As per claim 7
Park teaches the vehicle of the apparatus being a bus and the apparatus further configured to detect a bus stop as the predetermined landmark ( Figure 1 ,Figure 2A, 2B 5A-5C)
As per claim 13
Park teaches an onboard device which is configured to photograph an image of a vicinity (Figure 2A label 204) and a server apparatus configured to determine whether a user candidate is a user of the vehicle based on the user candidate and a predetermined landmark indicating a getting on place, the user candidate and the predetermined land mark being detected by image processing on the image (Figure 2A label 202 and paragraph [0026] “Furthermore, control logic of the present invention may be embodied as non-transitory computer readable media on a computer readable medium containing executable program instructions executed by a processor, controller/control unit or the like… The computer readable recording medium can also be distributed in network coupled computer systems so that the computer readable media is stored and executed in a distributed fashion, e.g., by a telematics server or a Controller Area Network (CAN).) Park shows the controller analyzes the acquired image to determine whether waiting passengers intend to board at the next stop and teaches that the executable program instructions performing this control logic can also be distributed in network coupled computer systems so that the computer readable media is stored and executed in a distributed fashion (telematic server). Under broadest reasonable interpretation Park discloses the claimed onboard device and server apparatus performing the determination based on image processed detection of the user candidate and landmark.
As per claim 15
Claim 15 recites a method of having steps that correspond to the same functions and operations recited in claim 1. As discussed in respects to claim 1, Park teaches detecting a predetermined landmark from an image obtained by photographing a vicinity of a vehicle and determining whether a user candidate is a user of the vehicle based on information on the user candidate and the predetermined land mark. The same disclosure of Park relied upon in the rejection of claim 1 applies equally to the method steps of claim 15. The acts performed by Park’s controller when executing stored modules inherently disclose the corresponding method steps recited in claim 15.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 3-6 are rejected under 35 U.S.C. 103 as being unpatentable over Park et al (Park hereinafter US 20200327317 A1) in view of Huang et al (Huang hereinafter US 5953055 A)
As per claim 3
Park teaches all claim limitations rejected in claim 2’s 102 ejection
Park does not teach the distance between the user candidate and the predetermined landmark as being equal to or shorter than a predetermined value
Huang teaches applying a predetermined spatial limit in the image through a defined queue zone and bounding box/ROI which is equivalent to a preset spatial boundary that functions as the “predetermined value”. (Column 3 line 41: “ Bounding Box: The bounding box describes the location of the queue zone 420 within the camera view.” and Column 5 line 33 “The region of interest 440 corresponds to a bounding box that encloses the one or more queue zones 420. The output of the operation is a smaller frame that contains only the region of the image that corresponds to the queue zone 420 being analyzed.” A person is treated as “in the queue” only if they fall within that predetermined bounding box/ROI which is equivalent to a “predetermined value” vicinity constraint relative to the landmark area being analyzed. ) and the user candidate stays in the same location (Column 3 line 61 “Time threshold: Specifies the amount of time a slot must be in a steady state in order for the slot status to change.” Requiring a slot in a steady state for a threshold time before a change of status shows that the user is detected staying in the same location (slot) rather than a transient pass through.
Accordingly, a person of ordinary skill in the art at the time this invention was effectively filed would have found it obvious to modify Park’s camera-based bus stop boarding intention determination to incorporate Huang’s queue analysis mechanisms specifically restricting analysis to a predetermined “queue zone” defined by a bounding box and applying a time threshold “steady state”. Huang expressly teaches these images processed controls to regulate detection and analysis of waiting people and reduce noise through non user transience. This modified workflow between Park and Huang improves user determination at a stop by filtering out non users passing by causing momentary detections while retaining true waiting riders by understanding and detecting the rider is in a fixed location within proximity of an area.
As per claim 4
Park teaches all claim limitations rejected in claim 2’s 102 rejection
Park does not expressly teach a number of candidates to be determined
Huang teaches the distance between the user candidate and the predetermined landmark as being equal to or shorter than a predetermined value (Huang discloses applying a predetermined spatial limit in the image through a defined queue zone and bounding box/ROI which is equivalent to a preset spatial boundary that functions as the “predetermined value”. (Column 3 line 41: “ Bounding Box: The bounding box describes the location of the queue zone 420 within the camera view.” and Column 5 line 33 “The region of interest 440 corresponds to a bounding box that encloses the one or more queue zones 420. The output of the operation is a smaller frame that contains only the region of the image that corresponds to the queue zone 420 being analyzed.” A person is treated as “in the queue” only if they fall within that predetermined bounding box/ROI which is equivalent to a “predetermined value” vicinity constraint relative to the landmark area being analyzed.) and a number of candidates being equal to or more than a predetermined number as the state of the user candidate (Column 1 line 14 “the present invention is able to detect and record the number of people in the queue, the length of time the people have been waiting in the queue, as well as other characteristics of the queue” and (Column 5 line 64 “To determine the number of people in the queue, the queue zone 420 may be subdivided into a set of slots 425…Each slot 425 may be created such that it is approximately the size of a customer waiting in the queue…the system determines the number of slots 425 occupied by a customer. The number of slots 425 occupied by a customer correspond to the number of customers in the queue.” Huang’s system also used for alerts when the queue reaches a threshold for size. Column 14 line 2 “system 130 may also be programmed with sound to perform real-time alerts, such as by sounding an alarm once the queue reaches a certain length” These attributes in Huang’s system shows the system detects users, counts people, compares the count against a predetermined value and triggers a behavior.)
Accordingly, a person of ordinary skill in the art at the time this invention was effectively filed would have found it obvious to enhance the Park/Huang camera-based bus stop user determination workflow with Huang’s idea of queue analysis functionality which includes detecting and recording a number of people in a queue and comparing that number to a predetermined threshold. This is because Huang expressly teaches counting people within a defined region and generating control actions when the number reaches a certain value. This improves Park’s end of the modified workflow by augmenting user determination by enabling the system to distinguish true riders based on a collective waiting behavior of a plurality of users rather than isolated individual detections using routine computer vision counting techniques
As per claim 5
Park teaches all claim limitations rejected in claim 2’s 102 rejection
Huang teaches the distance between the user candidate and the predetermined landmark as being equal to or shorter than a predetermined value (Huang teaches applying a predetermined spatial limit in the image through a defined queue zone and bounding box/ROI which is equivalent to a preset spatial boundary that functions as the “predetermined value”. (Column 3 line 41: “ Bounding Box: The bounding box describes the location of the queue zone 420 within the camera view.” and Column 5 line 33 “The region of interest 440 corresponds to a bounding box that encloses the one or more queue zones 420. The output of the operation is a smaller frame that contains only the region of the image that corresponds to the queue zone 420 being analyzed.” A person is treated as “in the queue” only if they fall within that predetermined bounding box/ROI which is equivalent to a “predetermined value” vicinity constraint relative to the landmark area being analyzed.) and a plurality of the user candidates’ lines up as the state of the user candidate (Column 4 line 66 “In pixel-based queue analysis, it is assumed that customers waiting in a queue form a relatively straight line.” and Column 6 line 1 “If the customers queue in a top-to-bottom direction, the zone 420 is divided into a set of vertical slots 425, as shown in FIG. 4C. If customers queue in a left-to-right direction, the zone 420 is divided into a set of horizontal slots 425, “ The latter quote showing that “lining up” is supported in that the queue is analyzed as an ordered line direction (top to bottom or left to right) via a slot partitioning along the line).
Accordingly, it would have been obvious to a person of ordinary skill at the time this invention was effectively filed to further enhance the Park/Huang workflow to incorporate Huang’s concept of queue line analysis specifically regarding detection of whether multiple candidates are lined up within the monitored areas. Huang expressly shows that the user form relatively straight lines and uses queue direction assumption and slot partitioning image processing structure to characterize a line of users. This modification augments the Park/Huang workflow by distinguishing true riders who would typically form an organized line or grouping from passing by pedestrians near the stop using known computer vision analysis techniques in an expected manner.
As per claim 6
Park teaches all claim limitations rejected in claim 2’s 102 rejection
Huang teaches candidate distance is equal to or shorter than a predetermined value (Huang teaches applying a predetermined spatial limit in the image through a defined queue zone and bounding box/ROI which is equivalent to a preset spatial boundary that functions as the “predetermined value”. (Column 3 line 41: “ Bounding Box: The bounding box describes the location of the queue zone 420 within the camera view.” and Column 5 line 33 “The region of interest 440 corresponds to a bounding box that encloses the one or more queue zones 420. The output of the operation is a smaller frame that contains only the region of the image that corresponds to the queue zone 420 being analyzed.” A person is treated as “in the queue” only if they fall within that predetermined bounding box/ROI which is equivalent to a “predetermined value” vicinity constraint relative to the landmark area being analyzed.)
Park teaches the user candidate is performing a predetermined action as the state of the user candidates (Figure 3C, Paragraph [0008] The predetermined indication of the boarding intention may include the passenger making a predetermined gesture to indicate the boarding intention…” then Park further teaches in the same paragraph verifying the boarding intention by image capture and analysis of that gesture “may be configured to verify the boarding intention by capturing the gesture of the passenger using the camera.”)
Accordingly, a person of ordinary skill in the art at the time this invention was effectively filed would have found it obvious to continue the Park/Huang workflow with Park’s concept of detecting a predetermined gesture from a photographed image in unison with Huang’s predetermined spatial gating (the bonding box corresponding to a relevant zone) in a fashion where the “predetermined action” determination is made for candidates within a predetermined proximity of the relevant landmark. Huang shows restriction image processing to a predefined region of interest to focus analysis on the intended area to reduce interference and applying this to Park’s gesture-based user determination is a way to predictably reduce false positives from non-riders or transient pedestrians outside and around the pickup zone while maintaining the same action-based detection format.
Claims 8 and 9 are rejected under 35 U.S.C. 103 as being unpatentable over Park et al (Park hereinafter US 20200327317 A1) in view of Papineau et al (Papineau hereinafter US 11694464 B2)
As per claim 8
Park teaches all claim limitations previously rejected in claim 1
Park does not teach the vehicle is driven by control of a driver or is configured to inform the driver of the presence of the user in a case where the determination unit determines that the user candidate is the user.
Papineau teaches the vehicle is a vehicle driven by a control of a driver (Column 3 line 63 “vehicle driven by Driver A “), at least one processor is configured to inform the driver of presence of the user in a case where the determination unit determines that the user candidate is the user (Papineau shows confirming and validating presence of occupants via face verification and providing notifications to the driver: Column 5 line 61 “techniques may include facial count and verification where each participant registers their individual face (face print signature) via the camera and is added to the ride. This technique may include a count of human faces “seen” in a prompted (relevant time/location stamped) photo validated by biometric human face determination and/or a count of human faces” In regards to informing the driver of verified users: After “ facial count and verification where each participant registers their individual face (face print signature) via the camera and is added to the ride. “ and “ a count of human faces “seen” in a prompted (relevant time/location stamped) photo validated by biometric human face determination” establishes that the system determines whether a detected person isa verified participant, Papineau teaches that the system communicates that determination via “the server sends determination regarding satisfaction of reward criteria to…the driver” (Column 11 line 24) and that “At 152, if the reward grantor is satisfied that that the occupancy of the vehicle has been properly verified, and that the driver is not “cheating” in some fashion, the reward grantor may then transmit the reward certificate, notification, validation, or permission to the driver of the vehicle.” (Column 13 line 6). This shows that after verification and determination of appropriate user candidate, the system transmits a notification to the driver. The notification follows the user determination and their presence.
Accordingly, a person of ordinary skill at the time this invention was effectively filed would have been motivated to modify Park’s user detection apparatus to inform a human driver of verified user presence using the biometric verification and driver notification techniques taught by Papineau. Incorporating this verification to notification system to Park’s workflow would predictably improve reliability and driver awareness by ensuring that only verified users are communicated to the driver. This reduces ambiguity and false pickup scenarios and driver safety in the modified workflow.
As per claim 9
Park and Papineau teach all claim limitations previously rejected in claim 8’s 103 rejection. See claim 8’s 103 rejection.
Papineau also teaches a number of users is informed of the drive( Papineau shows number determination by facial count verification: Column 13 line 35 “…server determines the number of occupants that are present in the vehicle based upon the count of the number of facial signatures…” and Column 5 line 62 “facial count and verification where each participant registers their individual face “. Then this determination is communicated with the driver via the checkpoint of occupancy confirmation and can be seen in the flow of figure 3 (flow directionality between from label 134, 138, 140 and 136) 3
Accordingly, a person of ordinary skill in the art at the time this invention was effectively filed would have found it obvious to further modify the Park/ Papineau workflow so that the driver is informed not only of presence but also the number of users because Papineau teaches using facial verification to validate the number of occupants and communicating the server’s determination to the driver. The combination enables a clearer driver decision making and compliance using facial count verification techniques improving reliability versus manual counting or guessing.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Park et al (Park hereinafter US 20200327317 A1) in view of Vallespi et al (Vallespi hereinafter US 20190042865 A1)
Park teaches all claim limitations rejected in claim 1’s 102 ejection
Park further teaches the vehicle is driven by a control of the self-driving control system ( Paragraph [0011] “A controller configured to determine whether to stop the unmanned autonomous vehicle at a bus stop based on the acquired boarding intention…”)
Park does not expressly disclose output positional information of the user to a self-driving control system…where…determination unit determines that the user candidate is the user.
teaches determining an estimated location of a detected pedestrian object (positional information) and providing output to a vehicle control system for autonomous vehicle operation
Vallespi teaches output positional information of the user to a self-driving control system in a case where the vehicle is a vehicle driven by a control of the self-driving control system and the determination unit determines the user candidate is the user (Paragraph [0007] “an estimated location of the object of interest within three-dimensional space based at least in part on a known relative location of the camera “ is determined. This is positional information.) In regard to output to a vehicle control system used for autonomous driving: Paragraph [0049] “one or more outputs from the classification model and/or some or all of the image data including one or more image variations can be provided as output data to one or more computing devices in a vehicle control system” and Paragraph [0050] “a vehicle control system configured to analyze image data and/or outputs from a disclosed detection system can be provided as an integrated component in an autonomous vehicle”. Therefore the “estimated location…within three-dimensional space” is positional information and it is output to a vehicle control system.
Note that Park supplies the “determination unit determines user candidate is user” via the boarding intention determination stated previously.
Accordingly, it would have been obvious to modify Park’s unmanned autonomous vehicle pickup intention determination at a bus stop to further output positional information of a determined user to the vehicles self-driving control stack. This is because Vallespi shows determining an object’s estimated location within 3D space and providing detection outputs to a vehicle control system is used in an autonomous vehicle. Incorporating that known localization output function into Parks workflow would predictably improve autonomous pick-up execution such as enabling precise stopping alignment and safer approach relative to the user’s position.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Park et al (Park hereinafter US 20200327317 A1) in view of Huang et al (Huang hereinafter US 5953055 A) in further view of Nobe et al (Nobe hereinafter US 5365448 A) .
Park teaches all claim limitations previously rejected in claim 1
Park does not teach the distance between the user candidate and the predetermined landmark as being equal to or shorter than a predetermined value
Huang teaches applying a predetermined spatial limit in the image through a defined queue zone and bounding box/ROI which is equivalent to a preset spatial boundary that functions as the “predetermined value” or distance. (Column 3 line 41: “ Bounding Box: The bounding box describes the location of the queue zone 420 within the camera view.” and Column 5 line 33 “The region of interest 440 corresponds to a bounding box that encloses the one or more queue zones 420. The output of the operation is a smaller frame that contains only the region of the image that corresponds to the queue zone 420 being analyzed.” A person is treated as “in the queue” only if they fall within that predetermined bounding box/ROI which is equivalent to a “predetermined value” vicinity/ distance constraint relative to the landmark area being analyzed. ) and the user candidate stays in the same location (Column 3 line 61 “Time threshold: Specifies the amount of time a slot must be in a steady state in order for the slot status to change.” Requiring a slot in a steady state for a threshold time before a change of status shows that the user is detected staying in the same location (slot) rather than a transient pass through.
Huang does not teach a distance between the vehicle and the predetermined landmark is estimated based on positional information of the vehicle.
Nobe teaches estimating a distance between a vehicle and a stored location using vehicle positional information (Column 2 line 1 “present invention comprises detection means for detecting present-location coordinate data representing a present location of a vehicle… means for computing a distance from the present location to the destination on the basis of the present-location coordinate data and the destination coordinate data, “
Accordingly, it would have been obvious to modify Park’s image-based landmark and user determination at a bus stop to operate only when the vehicle is within a predetermined distance of the landmark as taught by Nobe and to implement the predetermined-distance condition using a predefined spatial region as taught by Huang . Nobe teaches computing vehicle to location distance from positional data and Huang teaches constraining detection and localization to a predefined spatial extent to improve efficiency. Combining these two techniques with Park’s backbone would predictably reduce unnecessary image processing when the vehicle is far from the landmark while improving accuracy and computational efficiency when the vehicle approaches the relevant pickup location using an exact estimated distance based on the vehicle’s positional information.
Claims 12 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Park et al (Park hereinafter US 20200327317 A1) in view of Park Minwoo et al (Park M hereinafter US 20230214654 A1)
Park teaches all limitations previously rejected in claim 1. See claim 1’s rejection
Park M teaches an AI engine configured to learn an image of the predetermined landmark or an image of something similar to a predetermined landmark. (Park M teaches training a DNN on annotated landmark features in training datasets Paragraph [0006] “To train the DNN(s) for accurate prediction, ground truth data corresponding to control point locations for landmark features may be generated from polyline and/or polygon annotations corresponding to landmark features in training data sets”) and assist detection of the predetermined landmark (Paragraph [0006] “ Once trained and deployed, the DNN(s) may accurately and precisely compute outputs indicating control point locations and semantic information corresponding thereto, and these outputs may be decoded and post-processed to determine curves corresponding to landmark features.”). Park M’s AI engine is also being executed through a processor is (Paragraph [0061] “process 600 of using a DNN for landmark detection…may be carried out by a processor executing instructions stored in memory)
Accordingly, a person of ordinary skill in the art would have found it obvious to modify Park’s camera-based bus stop landmark imaging/detection workflow to incorporate a trained AI landmark detection engine as taught by Park M. Park M expressly teaches training a DNN using annotated landmark training data and then deploying the trained DNN to identify a landmark location and further explains that reducing heavy post processing decreases latency and computing requirements for real time vehicle deployment. This combination improves accuracy of detecting the bus stop landmark under real world variation and supports a real time operation with reduced latency compared to a non-AI approach.
As per claim 14
Park covers all claim limitations of claim 13 in claim 13’s 103 rejection. See claim 13’s 103 rejection
Park has front facing cameras but not explicitly on the left and right sides.
Park M shows a plurality of cameras on the side of his vehicle for detection (Figure 11B)
Accordingly, a person of ordinary skill in the art at the time this invention was effectively filed would have been motivated to further incorporate Park M’s concept of having a left and right-side camera in addition to Parks front facing camera’s because the addition would predictably improve reliable identification and yield a full-scale comprehensive scene coverage.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHANE WRENSFORD CODRINGTON whose telephone number is (571)272-8130. The examiner can normally be reached 8:00am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at (571) 272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHANE WRENSFORD CODRINGTON/Examiner, Art Unit 2667
/TOM Y LU/Primary Examiner, Art Unit 2667