Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims 2, 5-8, 10, 13, 19, and 24-26 have been amended.
Claims 14-17. 20 and 22-23 cancelled or previously cancelled.
Claims 27 and 28 are new
Claims 2-13, 18, 19, 21, and 24-26 are pending.
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Response to Arguments
Applicant’s arguments with respect to claims 2-13, 18, 19, 21, and 24-26 have been considered but are moot in view of the new ground(s) of rejection as necessitated by applicant's amendments.
Claim Interpretation
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. Under a broadest reasonable interpretation (BRI), words of the claim must be given their plain meaning, unless such meaning is inconsistent with the specification. The plain meaning of a term means the ordinary and customary meaning given to the term by those of ordinary skill in the art at the relevant time. The ordinary and customary meaning of a term may be evidenced by a variety of sources, including the words of the claims themselves, the specification, drawings, and prior art. However, the best source for determining the meaning of a claim term is the specification - the greatest clarity is obtained when the specification serves as a glossary for the claim terms. The words of the claim must be given their plain meaning unless the plain meaning is inconsistent with the specification. 2111.01 (I). See also In re Marosi, 710 F.2d 799, 802, 218 USPQ 289, 292 (Fed. Cir. 1983) ("'[C]laims are not to be read in a vacuum, and limitations therein are to be interpreted in light of the specification in giving them their ‘broadest reasonable interpretation.'"2111.01 (II).
Claim Rejections - 35 USC§ 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed
invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
Determining the scope and contents of the prior art.
Ascertaining the differences between the prior art and the claims at issue.
Resolving the level of ordinary skill in the pertinent art.
Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 2-7, 9-13, 19, 21-22, 24, 27, 28 are rejected under 35 U.S.C. 103 (a) as unpatentable over Arndt et al.[US20160012301, now Arndt], in view of Ogale et al. [US20190034794, now Ogale].
In regard to Independent claims 2, 10, and 19 Arndt teaches a method and system and processing unit comprising: receiving first sensor data obtained using one or more external sensors of a machine, the first sensor data representative of at least a gesture being performed by a pedestrian located outside of the machine; [See at least Arndt, ¶ 0030 (“receiving via a camera (sensor) data representative of a pedestrian”); 0049-57 (“pedestrian detection”); 0033-0037; and 0045-0046 (“gestures”)];
determining, using [[the ]]one or more second first neural networks and based at least on the first sensor data representative of the gesture, one or more actions for the machine to perform that are related to the gesture being performed by the pedestrian [See at least Arndt, ¶ 0034 (“the classification module and image analysis module 5 and 6 perform analysis of the pedestrian and based on the recognized gesture cause the car to in some cases perform an action.”); 0037 (“or example by means of audible or visual signals) that the driver should pay attention to a gesture of a person 3.”); 0039 (“One example of this is a motor vehicle 2 that approaches a school where a crossing guard is standing on or by the road and is waving a traffic paddle. One example of this is a motor vehicle 2 that approaches a school where a crossing guard is standing on or by the road and is waving a traffic paddle. One example of this is a motor vehicle 2 that approaches a school where a crossing guard is standing on or by the road and is waving a traffic paddle.”); Note one sees a man waiving his arm while holding a sign and standing in the road which conveys a safety message; 0052 (“e.g. in the form of a neural network, could improve the decision logic for uncertain or inconclusive situations”); 0069-0082 (show more gestures by pedestrians and workmen, including emergency workers)];
Arndt does not specifically disclose but Ogale teaches determining, using [[the ]]one or more second first neural networks and based at least on the first sensor data representative of the gesture, one or more actions for the machine to perform that are related to the gesture being performed by the pedestrian; determining whether the one or more actions would cause a collision with at least the pedestrian; and determining, using one or more second neural networks and based at least on second sensor data obtained using one or more internal sensors of the machine, a gaze direction associated with a driver of the machine; determining, based at least on the gaze direction, that the pedestrian is located outside of a field-of-view (FOV) of the driver when making the gesture; and causing, based at least on determining the one or more actions would not cause the collision with the pedestrian being located outside of the FOV when making the gesture, the machine to navigate according to a trajectory that is related to the one or more actions.
However, Ogale teaches at least one sensor data is sent through a neural network to determine if a collision will happen along the path of a vehicle. [See at least Ogale, ¶ 0009; 0010; 0016-0019; explains process of using more than one neural network].
Ogale is analogous art to Arndt autonomous vehicles and processing sensor data. [¶ 0009). Ogale teaches a trajectory planning neural network that uses a first neural network and a second neural network where the second neural network uses environmental data of the vehicle (e.g. camera data in the vicinity of a vehicle and traffic artifacts about a vehicle; ¶ 0017]. The output of the first and second neural networks can cause a vehicle to take an action or perform an action (para 21, such as braking, accelerating or steering). (Para 22). As stated in Ogale, the second neural network can process object data of objects in the vicinity of the vehicle, camera data of an optical image in vicinity of the vehicle and plan a trajectory (para 28) and identify a driving scenario such as a collision (¶ 0037- 0038).The planned trajectory can be improved to select waypoints to navigate that improves the safety and comfort of the passenger. (Para 49). Ogale teaches the sensor data in the second neural network can include data of pedestrians and indicate these sections to avoid for collision risk purposes (Para 71). (See also Fig. 3). Ogale teaches the planning system can, if a collision is imminent, can override a trajectory (¶ 0079).
Accordingly it would have been obvious to the skilled artisan prior to the effective date of the invention having the teachings of Arndt and Ogale inf rant of them to modify the sensors and neural network of Arndt, with a reasonable degree of success to provide a second neural network that processes trajectory information to avoid collisions with pedestrians. The motivation to combine Arndt with Ogale comes from Ogale which suggests using the second neural network to process environmental data that includes pedestrian information to generate trajectory information that allows the vehicle to cross an intersection in a safe, legal and comfortable manner and avoid trajectories that indicate a collision will occur [¶ 0071-0072,0079).
With respect to dependent claims 3 and 11, Arndt teaches the method and system further comprising: determining, based at least on the gesture, an intent associated with the pedestrian, wherein the determining the trajectory is further based at least on the intent.. [See at least Arndt, ¶ 007, intention 51 and (¶ 0033-0037, 0045-0046). (See e.g. stop vehicle based on policeman gesture or audible gestures or hitchhiker finger or crossing guard causing vehicle to slow or stop or any of the gestures in figure 4a-7 and table 1, ¶ 0068].
With respect to dependent claims 4 and 12, Arndt teaches the method and system wherein the gesture is associated with one or more of: causing the machine to continue navigating; causing the machine to stop; or causing the machine to navigate to a position associated with the pedestrian. [See at least Arndt, ¶ 0007, intention 51; ¶ 0033-0037, 0045-0046]. (See at least Arndt, e.g. stop vehicle based on policeman gesture or audible gestures or hitchhiker finger or crossing guard causing vehicle to slow or stop or any of the gestures in figure 4a-7 and table 1, ¶ 0068].
With respect to dependent claims 5, 13 and 28 Arndt teaches the method and system further comprising: determining, based at least on the first sensor data, that the pedestrian includes personnel corresponding to one or more of law enforcement, fire protection, emergency services, or a crossing guard, wherein the causing the machine to navigate according to the trajectory is further based at least on the pedestrian including the personnel corresponding to the one or more of law enforcement, the fire protection, the emergency services,, or the crossing guard. (See at least Arndt, abstract determining the pedestrian is a policeman or crossing guard [See at least Arndt; ¶ 0003-0004; 0010; 0047-0051; 0060].
With respect to dependent claim 6 Arndt teaches the method further comprising: determining, based at least on the first sensor data, that the pedestrian is associated with a vehicle detected in an environment corresponding to the machine and represented at least partially in the sensor data, wherein the causing the machine to navigate according to the trajectory is further based at least on the pedestrian being associated with the vehicle.. (See at least Arndt, Para 10, customers associated with the taxi [See at least Arndt, ¶ 0007, intention 51; 00 33-0037, 0045-0046). (See e.g. stop vehicle based on policeman gesture or audible gestures or hitchhiker finger or crossing guard causing vehicle to slow or stop or any of the gestures in figure 4a-7 and table 1, ¶ 0068).
With respect to dependent claim 7, Arndt teaches the method and system further comprising causing, using one or more output devices associated with the machine, an alert associated with the gesture being performed by the pedestrian. (See at least Arndt, Para 12, alert, Para 83).
With respect to dependent claim 9 Arndt teaches the method wherein the gesture is associated with at least one of: a motion of a portion of the pedestrian; or a motion of an item that is in possession of the pedestrian. [See at least Arndt, ¶ 0007, intention 51; ¶ 0033-0037, 0045- 0046). (See e.g. stop vehicle based on policeman gesture or audible gestures or hitchhiker finger or crossing guard causing vehicle to slow or stop or any of the gestures in figure 4a-7 and table 1, ¶ 0068].
With respect to dependent claims 18 and 21, Arndt teaches the method and system wherein the system and processing unit is comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing deep learning operations; a system for generating synthetic data; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources. [See at least Arndt, semi- autonomous system using a driver assistance ¶ 0038-0039].
With respect to dependent claim 25, Arndt does not specifically disclose wherein: the action is associated with a trajectory for the machine to navigate; and the machine is caused to navigate along the trajectory. But Arndt does disclose navigating a path [See at least Arndt, ¶ 0007 (intention 51); 0033-0037, 0045-0046). (See e.g. stop vehicle based on policeman gesture or audible gestures or hitchhiker finger or crossing guard causing vehicle to slow or stop or any of the gestures in figure 4a-7 and table 1, ¶ 0068).
However, Ogale more specifically teaches the action is associated with a trajectory for the machine to navigate (See at least Ogale, Abstract; ¶ 0001).
With respect to dependent claim 27, as indicated in the above discussion Arndt in view of Ogale teaches each element of claim 19.
Arndt does not disclose but Ogale teaches the processing circuitry is further to cause, based at least on the pedestrian being located outside of the region associated with the gaze direction, output of an alert associated with the gesture . [See at least Ogale, ¶ 0009; 0010; 0016-0019; explains process of using more than one neural network].
Claim(s) 24 and 26 is rejected under 35 U.S.C. 103 as being unpatentable over Arndt in view of Ogale as applied to claim 1 above, and further in view of Sinha et al. [US20170168586, now Sinha].
With respect to dependent claim 24, as indicated in the above discussion Arndt in view of Ogale teaches each element of claim 2.
However, Arndt in view of Ogale does not disclose generating, using the one or more first neural networks and based at least on the first sensor data, pose data representative of one or more three-dimensional (3D) poses associated with thepedestrian, wherein the determining the one or more actions for the machine is based at least on the pose data
Nonetheless, Sinha is analogous art to Arndt and the present application as Sinha is directed to the problem solving area of hand poses analyzed by a neural network (See at least Sinha, ¶ 0007-0008). Sinha teaches determining the 3D processing of the hand pose [¶ 0004-0005) so as to determine intent. (¶ 0024). Sinha teaches using three dimensional imaging devices and processing the pose of the gestures f ram different angles (¶ 0029-0030, 0037). Sinha teaches recognizing the gesture and by comparing to set of hand pose parameters a best fit is determined. (¶ 0033). Sinha teaches using a combination of neural network outputs for the fingers and wrist and then combines the two to provide a gesture input into the system (¶ 0035,0043). Sinha teaches using a plurality of networks corresponding to each finger on the hand. (¶ 0043). Each finger can represent a different activation of a feature. (¶. 0044-0045). Sinha teaches processing the gesture in a three dimensional software using various poses from various angles corresponding to a pose of the hand (¶ 0050).
Accordingly it would have been obvious to the skilled artisan prior to the effective date of the invention having the teachings of Arndt, Sinha and Ogale inf rant of them to modify the sensors and neural network of Arndt, with a reasonable degree of success to provide a second neural network that processes trajectory information to avoid collisions with pedestrians and a neural network that processes 3D gestures. The motivation to combine Arndt with Ogale comes from Ogale which suggests using the second neural network to process environmental data that includes pedestrian information to generate trajectory information that allows the vehicle to cross an intersection in a safe, legal and comfortable manner and avoid trajectories that indicate a collision will occur (¶ 0071-0072,0079). The motivation to combine Arndt with Sinha comes from Sinha which suggests capturing three dimensional input of a user's hand or gesture without the use of a glove (Para 4) thereby improving the speed and recognition of hand poses in three dimensional spaces [ 0063]
With respect to dependent claim 26, as indicated in the above discussion Arndt in view of Ogale teaches each element of claim 10.
However, Arndt in view of Ogale does not disclose/teach wherein: the action is associated with one or more directions being provided by the pedestrian; and the machine is caused to navigate based at least on the one or more direction. Nonetheless, Sinha is analogous art to Arndt and the present application as Sinha is directed to the problem solving area of hand poses analyzed by a neural network (See at least Sinha, Para 7-8). Sinha teaches determining the 3D processing of the hand pose (Para 4-5) so as to determine intent. (Para 24). Sinha teaches using three dimensional imaging devices and processing the pose of the gestures f ram different angles (¶ 0029-0030, 0037). Sinha teaches recognizing the gesture and by comparing to set of hand pose parameters a best fit is determined. (¶ 0033). Sinha teaches using a combination of neural network outputs for the fingers and wrist and then combines the two to provide a gesture input into the system (¶ 0035,0043). Sinha teaches using a plurality of networks corresponding to each finger on the hand. (¶ 0043). Each finger can represent a different activation of a feature. (¶ 0044-0045). Sinha teaches processing the gesture in a three dimensional software using various poses from various angles corresponding to a pose of the hand(¶ 0050).
Accordingly it would have been obvious to the skilled artisan prior to the effective date of the invention having the teachings of Arndt, Sinha and Ogale inf rant of them to modify the sensors and neural network of Arndt, with a reasonable degree of success to provide a second neural network that processes trajectory information to avoid collisions with pedestrians and a neural network that processes 3D gestures. The motivation to combine Arndt with Ogale comes from Ogale which suggests using the second neural network to process environmental data that includes pedestrian information to generate trajectory information that allows the vehicle to cross an intersection in a safe, legal and comfortable manner and avoid trajectories that indicate a collision will occur (¶ 0071-0072,0079). The motivation to combine Arndt with Sinha comes from Sinha which suggests capturing three dimensional input of a user's hand or gesture without the use of a glove (¶ 0004) thereby improving the speed and recognition of hand poses in three dimensional spaces. (¶ 0063)
Claim(s) 8 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Arndt in view of Ogale in further view of Trinh et al. [US20170334348, now Trinh].
With respect to dependent claims 8 and 16, as indicated in the above discussion Arndt in view of Ogale teaches each element of claim 2, and 10. Arndt teaches tracking head movement or orientation and viewing direction of a person [see at least Arndt, abstract; ¶ 0003, 0050). However, Arndt does not specifically disclose the method
further comprising: determining, based at least on second sensor data obtained using one or
more interior sensors of the machine, a gaze direction associated with ad river of the machine; and determining, based at least on the gaze direction, that the pedestrian is located outside of a field-of-view (FOV) of the driver, wherein the one or more operations of the machine are further performed based at least on the pedestrian being located outside of the FOV of the driver.
However, Trinh is analogous art as being directed to the same problem solving area of detecting a pedestrian outside of a vehicle. Trinh captures the gaze direction of the driver and whether they are gazing at either pedestrian 8 or 9, in this case person 8 is in the field of view (See at least Trinh, ¶ 0023). Trinh teaches the system can either flash a light or project a crosswalk to the pedestrian (¶ 0024-0026). The combination of Arndt head tracking system with Trinh gaze tracking of gestures of the driver would result in both the pedestrian and driver being tracked for gestures to be interpreted by the vehicle.
Accordingly it would have been obvious to the skilled artisan prior to the effective date of the invention having the teachings of Arndt, Ogale and Trinh in front of them to modify the sensors of Arndt, with a reasonable degree of success, with the gaze sensor of Trinh. The motivation to combine Arndt with Trinh comes from Trinh which suggests tracking a gaze direction of ad river to cause a command to be recognized by the vehicle to let the pedestrian know the driver is looking at them and thereby providing a notification externally to the pedestrian (Para 5).
Allowable Subject Matter
Upon receipt of the amendments, Examiner observed that the limitation “comprises determining, using one or more third neural networks, that a trajectory for the machine to navigate that is related to the action would avoid the collision with the pedestrian” in Claim 25 was amended to exclude that feature (third neural network- which supported in the Specification ¶ 0070 (plurality of neural networks), has been deleted from Claim 25 and not included in the independent claims this feature was indicated, in the past, as allowable subject matter if included in the independent claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOAN T GOODBODY whose telephone number is (571) 270-7952. The examiner can normally be reached on M-TH 7-3 (US Eastern time).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at https://www.uspto.gov/patents/uspto-automated-interview-request-air-form.html.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, RACHID BENDIDI can be reached at (571) 272-4896. The Fax phone number for the organization where this application or proceeding is assigned is (571) 273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspot.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at (866) 217-9197 (toll-free). If you would like assistance from the USPTO Customer Serie Representative or access to the automated information system, call (800) 786-9199 (IN USA OR CANADA) or (571) 272-1000.
/JOAN T GOODBODY/
Primary Examiner, Art Unit 3664
(571) 270-7952
Prosecution Insights