DETAILED ACTION
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
2. Claims 1-20 are pending in Instant Application.
Examiner’s Note
3. Examiner has cited particular paragraphs/columns and line numbers or figures in the references as applied to the claims below for the convenience of the applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested from the applicant, in preparing the responses, to fully consider the references in their entirety as potentially teaching all of part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the examiner. Applicant is reminded that the Examiner is entitled to give the broadest reasonable interpretation to the language of the claims. Furthermore, the Examiner is not limited to Applicant’s definition which is not specifically set forth in the claims.
Drawings
4. The drawings are objected to under 37 CFR 1.83(a) because they fail to show the details of the processes/methods illustrated in Figures 3, 4A-4C, 6A-17 and 53 as described in the specification. Any structural detail that is essential for a proper understanding of the disclosed invention should be shown in the drawing. MPEP § 608.02(d). These figures merely show blank boxes. There are no details regarding what each box represents. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Double Patenting
5. The non-statutory obvious double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A non-statutory obvious double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on non-statutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/process/file/efs/guidance/eTD-info-I.jsp.
Claims 1-20 are rejected on the ground of non-statutory double patenting as being unpatentable over claims 1, 6-7, and 9-20 of U.S. Patent No. 20230398994. The claims at issue are identical, they are not patentably distinct from each other, which can be shown below. Bolded portions showcase exact similarities.
Application No. 19/213,384
US Patent 20230398994
1. A system for sensing and control of a vehicle, comprising: one or more first sensors configured to sense an interior of the vehicle, wherein the one or more first sensors are configured to generate first sensor data based on sensing the interior of the vehicle; one or more second sensors configured to sense an exterior of the vehicle, wherein the one or more second sensors are configured to generate second sensor data based on sensing the exterior of the vehicle; a computing device, wherein the computing device is configured to detect a user indication of interest, wherein the computing device is configured to detect the user indication of interest based at least in part on a result of providing the first sensor data as input to an artificial neural network, wherein the computing device is configured to determine an item of interest based at least in part on the detected user indication of interest and the second sensor data; a vehicle controller, wherein the vehicle controller is configured to receive information about the item of interest, wherein the vehicle controller is configured to control operation of the vehicle based on the information about the item of interest.
1. A system for sensing and control of a vehicle, comprising: one or more first sensors configured to sense an interior of the vehicle, wherein the one or more first sensors are configured to generate first sensor data based on sensing the interior of the vehicle; one or more second sensors configured to sense an exterior of the vehicle, wherein the one or more second sensors are configured to generate second sensor data based on sensing the exterior of the vehicle; a computing device, wherein the computing device is configured to detect a user indication of interest, wherein the computing device is configured to detect the user indication of interest based at least in part on a result of providing the first sensor data as input to an artificial neural network, wherein the computing device is configured to determine an item of interest based at least in part on the detected user indication of interest and the second sensor data; a vehicle controller, wherein the vehicle controller is configured to receive information about the item of interest, wherein the vehicle controller is configured to control operation of the vehicle based on the information about the item of interest, wherein the user indication of interest is a gesture, wherein the computing device is configured to determine the item of interest based at least in part on calculation of a pointing vector, wherein the computing device is configured to calculate the pointing vector based at least in part on the first sensor data, wherein the computing device is configured to calculate the pointing vector based at least in part on a plurality of landmarks in the first sensor data, and wherein the computing device is configured to calculate the pointing vector based at least in part on calculating a best-fit vector for the plurality of landmarks.
2. The system of claim 1, wherein the user indication of interest is a gesture.
See above in Claim 1
3. The system of claim 2, wherein the computing device is configured to determine the item of interest based at least in part on calculation of a pointing vector.
See above in Claim 1
4. The system of claim 3, wherein the computing device is configured to calculate the pointing vector based at least in part on the first sensor data.
See above in Claim 1
5. The system of claim 4, wherein the computing device is configured to calculate the pointing vector based at least in part on a plurality of landmarks in the first sensor data.
See above in Claim 1
6. The system of claim 5, wherein the plurality of landmarks includes an indication of the location of an index finger of an occupant of the vehicle.
6. The system of claim 1, wherein the plurality of landmarks includes an indication of the location of an index finger of an occupant of the vehicle.
7. The system of claim 6, wherein the plurality of landmarks includes an indication of the location of a wrist of an occupant of the vehicle.
7. The system of claim 6, wherein the plurality of landmarks includes an indication of the location of a wrist of an occupant of the vehicle.
8. The system of claim 5, wherein the computing device is configured to calculate the pointing vector based at least in part on calculating a best-fit vector for the plurality of landmarks.
See above claim 1
9. The system of claim 8, wherein the computing device is configured to determine the item of interest based at least in part on a geographic location of the vehicle at the time the user indication of interest occurred.
9. The system of claim 1, wherein the computing device is configured to determine the item of interest based at least in part on a geographic location of the vehicle at the time the user indication of interest occurred.
10. The system of claim 4, wherein the computing device is configured to calculate the pointing vector based at least in part on the first sensor data that includes time of flight sensor data.
10. The system of claim 1, wherein the computing device is configured to calculate the pointing vector based at least in part on the first sensor data that includes time of flight sensor data.
11. The system of claim 4, wherein the computing device is configured to calculate the pointing vector based at least in part on the first sensor data that includes image sensor data.
11. The system of claim 1, wherein the computing device is configured to calculate the pointing vector based at least in part on the first sensor data that includes image sensor data.
12. The system of claim 1, wherein the user indication of interest is the utterance of a predefined trigger word.
12. The system of claim 1, wherein the user indication of interest is the utterance of a predefined trigger word.
13. The system of claim 12, wherein the computing device is configured to determine the item of interest based at least in part on calculation of a gazing vector.
13. The system of claim 12, wherein the computing device is configured to determine the item of interest based at least in part on calculation of a gazing vector.
14. The system of claim 13, wherein the computing device is configured to calculate the gazing vector based at least in part on the first sensor data.
14. The system of claim 13, wherein the computing device is configured to calculate the gazing vector based at least in part on the first sensor data.
15. The system of claim 14, wherein the first sensor data includes at least one image that captures the orientation of the head of an occupant of the vehicle.
15. The system of claim 14, wherein the first sensor data includes at least one image that captures the orientation of the head of an occupant of the vehicle.
16. The system of claim 15, wherein the computing device is configured to calculate the gazing vector based at least in part on the orientation of the head of the occupant of the vehicle.
16. The system of claim 15, wherein the computing device is configured to calculate the gazing vector based at least in part on the orientation of the head of the occupant of the vehicle.
17. The system of claim 16, wherein the computing device is configured to calculate the gazing vector based at least in part on determining a roll value of the orientation of the head of the occupant of the vehicle, wherein the computing device is configured to calculate the gazing vector based at least in part on determining a pitch value of the orientation of the head of the occupant of the vehicle, wherein the computing device is configured to calculate the gazing vector based at least in part on determining a yaw value of the orientation of the head of the occupant of the vehicle.
17. The system of claim 16, wherein the computing device is configured to calculate the gazing vector based at least in part on determining a roll value of the orientation of the head of the occupant of the vehicle, wherein the computing device is configured to calculate the gazing vector based at least in part on determining a pitch value of the orientation of the head of the occupant of the vehicle, wherein the computing device is configured to calculate the gazing vector based at least in part on determining a yaw value of the orientation of the head of the occupant of the vehicle.
18. The system of claim 17, wherein the computing device is configured to calculate the item of interest based at least in part on a geographic location of the vehicle at the time the user indication of interest occurred.
18. The system of claim 17, wherein the computing device is configured to calculate the item of interest based at least in part on a geographic location of the vehicle at the time the user indication of interest occurred.
19. The system of claim 14, wherein the computing device is configured to calculate the gazing vector based at least in part on the first sensor data that includes image data.
19. The system of claim 14, wherein the computing device is configured to calculate the gazing vector based at least in part on the first sensor data that includes image data.
20. The system of claim 19, wherein the first sensor data includes audio sensor data.
20. The system of claim 19, wherein the first sensor data includes audio sensor data.
Claim Rejections - 35 USC § 103
6. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
7. Claim 1-16 and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Yamamoto (US Pub. No. 20150362988) in view of Weyers (US Pub. No. 20210081689).
Regarding Claim 1, Yamamoto discloses A system for sensing and control of a vehicle, comprising: (Yamamoto, see at least [0007] wherein a first user indication is sensed and performs an action corresponding to the sensed indication.)
one or more first sensors configured to sense an interior of the vehicle, (Yamamoto, see at least [0017] “visual sensors 104 may be placed within the interior of a vehicle”)
wherein the one or more first sensors are configured to generate first sensor data based on sensing the interior of the vehicle; (Yamamoto, see at least [0017] “visual sensors 104 may be used to detect visual objects both in the inner environment and in the outer environment.”)
one or more second sensors (¶0017, “The visual sensors 104…, multiple visual sensors 104”) configured to sense an exterior of the vehicle (¶0017, “provide a field of view outside of the vehicle (the outer environment in this example) to provide a 360-degree viewing angle of the outer environment, or a partial viewing angle less than 360 degrees.”), wherein the one or more second sensors are configured to generate second sensor data based on sensing the exterior of the vehicle (¶0017, “ The visual sensors 104 may be used to detect visual objects both in the inner environment and in the outer environment.”);
a computing device, wherein the computing device is configured to detect a user indication of interest, (Yamamoto, see at least [0005] “The processing device can be configured to determine an occurrence of a first user indication; determine a plurality of Point of Interest (POI) corresponding to the first user indication”);
wherein the computing device is configured to determine an item of interest based at least in part on the detected user indication of interest and the second sensor data; (Yamamoto, see at least [0005] “The apparatus may further include at least one processing device communicatively coupled to the at least one sensor device. The processing device can be configured to determine an occurrence of a first user indication; determine a plurality of Point of Interest (POI) corresponding to the first user indication;” and [0017] “The user indication recognition system 100 can include multiple sensor devices.”)
a vehicle controller, wherein the vehicle controller is configured to receive information about the item of interest, (Yamamoto, see at least [0051] “the user indication recognition system 100 may be used to control aspects of the larger system (e.g., a vehicle, home-theater, etc.) while the user is in the outer environment 206. For example, a user may be outside of a vehicle and point to the vehicle and state “lock it” or “unlock it.””)
wherein the vehicle controller is configured to control operation of the vehicle based on the information about the item of interest. (Yamamoto, see at least [0030] “Points of Interest (POI) may be present in any given application setting. For example, a POI 210 may be within the inner environment 204 and may relate to a system of a vehicle (such as the infotainment center, navigation, climate control, etc.) with which a user 202 may wish to control, inquire about, or interface with. Additionally, other POIs 212, 214 may exist in the outer environment 206. These may correspond to locations (such as addresses, buildings, parks, rest areas, etc.), businesses (such as stores, gas stations, retail shops, restaurants, hotels, etc.), roads or streets, objects, or any point of interest 212, 214 about which a user may wish to inquire or provide a command.”)
Yamamoto does not explicitly disclose wherein the computing device is configured to detect the user indication of interest based at least in part on a result of providing the first sensor data as input to an artificial neural network,
However, Weyers discloses wherein the computing device is configured to detect the user indication of interest based at least in part on a result of providing the first sensor data as input to an artificial neural network, (Weyers, see at least [0046] “a method for recognizing the current activity of the driver, for example based on conventional (intensity) images and depth images of a time-of-flight (ToF) camera mounted inside the car, may be provided. The method may be based on time of flight image sequences (i.e. sequences of distance information), driver body keypoints and a trained artificial neural network to classify different actions of a driver.”)
Accordingly, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to utilize the teachings as in modified Yamamoto with the teachings of Weyers to include the technique of utilizing neural networks when detecting and classifying actions by a user, as this would allow for gestures to be recognized along with object recognition when sensor data is inputted into a neural network and therefore would improve accuracy. This would further improve the effectiveness of determining the state of the driver of a vehicle.
Regarding Claim 2, Yamamoto in view of Weyers discloses The system of claim 1, (see rejection above)
Yamamoto further discloses wherein the user indication of interest is a gesture. (Yamamoto, see at least [0031] “first user indication can include either or both of an indication gesture (for example, a finger point) or an audible indication (for example, a spoken command).”)
Regarding Claim 3, Yamamoto in view of Herz does not explicitly disclose The system of claim 2, (see rejection above)
Yamamoto does not explicitly disclose wherein the computing device is configured to determine the item of interest based at least in part on calculation of a pointing vector.
However, Weyers discloses wherein the computing device is configured to determine the item of interest based at least in part on calculation of a pointing vector. (Weyers, see at least [0068] “The second set of features 610 may include features related to the occupant's body (for example a segmentation), related to the occupant's hands (for example crops, a segmentation, or a finger point vector, for example a vector indicating a location or direction to which the occupant points with one of his fingers), and/or related to the occupant's face (for example a crop, a segmentation, and/or a direction of view, for example a vector indicating a location or direction to which the occupant directs his face or to which he is looking).”)
Accordingly, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to utilize the teachings as in modified Yamamoto with the teachings of Weyers to include the technique of determining a pointing vector of a gesture because this would allow to enable for the point of interest of a user to be identified in a more accurate manner. This would further improve the effectiveness of sensing of the internal/external environment of a vehicle and vehicle safety actions as a result during hazardous situations.
Regarding Claim 4, Yamamoto in view of Herz discloses The system of claim 3, (see rejection above)
Yamamoto further discloses wherein the computing device is configured to calculate the pointing vector based at least in part on the first sensor data. (Yamamoto, see at least [0017] “visual sensors 104 may be used to detect visual objects both in the inner environment and in the outer environment.”)
Yamamoto does not explicitly disclose wherein the computing device is configured to calculate the pointing vector
However, Weyers discloses wherein the computing device is configured to calculate the pointing vector (Weyers, see at least [0068] “The second set of features 610 may include features related to the occupant's body (for example a segmentation), related to the occupant's hands (for example crops, a segmentation, or a finger point vector, for example a vector indicating a location or direction to which the occupant points with one of his fingers), and/or related to the occupant's face (for example a crop, a segmentation, and/or a direction of view, for example a vector indicating a location or direction to which the occupant directs his face or to which he is looking).”)
Accordingly, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to utilize the teachings as in modified Yamamoto with the teachings of Weyers to include the technique of determining a pointing vector of a gesture because this would allow to enable for the point of interest of a user to be identified in a more accurate manner. This would further improve the effectiveness of sensing of the internal/external environment of a vehicle and vehicle safety actions as a result during hazardous situations.
Regarding Claim 5, Yamamoto in view of Herz discloses The system of Claim 4, (see rejection above)
Yamamoto further discloses based at least in part on a plurality of landmarks in the first sensor data. (Yamamoto, see at least [0017] “a POI 210 may be within the inner environment 204 and may relate to a system of a vehicle (such as the infotainment center, navigation, climate control, etc.) with which a user 202 may wish to control, inquire about, or interface with. Additionally, other POIs 212, 214 may exist in the outer environment 206. These may correspond to locations (such as addresses, buildings, parks, rest areas, etc.), businesses (such as stores, gas stations, retail shops, restaurants, hotels, etc.), roads or streets, objects, or any point of interest 212, 214 about which a user may wish to inquire or provide a command..”)
Yamamoto does not explicitly disclose wherein the computing device is configured to calculate the pointing vector
However, Weyers discloses wherein the computing device is configured to calculate the pointing vector (Weyers, see at least [0068] “The second set of features 610 may include features related to the occupant's body (for example a segmentation), related to the occupant's hands (for example crops, a segmentation, or a finger point vector, for example a vector indicating a location or direction to which the occupant points with one of his fingers), and/or related to the occupant's face (for example a crop, a segmentation, and/or a direction of view, for example a vector indicating a location or direction to which the occupant directs his face or to which he is looking).”)
Accordingly, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to utilize the teachings as in modified Yamamoto with the teachings of Weyers to include the technique of determining a pointing vector of a gesture because this would allow to enable for the point of interest of a user to be identified in a more accurate manner. This would further improve the effectiveness of sensing of the internal/external environment of a vehicle and vehicle safety actions as a result during hazardous situations.
Regarding Claim 6, Yamamoto in view of Herz discloses The system of claim 5, (see rejection above)
Yamamoto further discloses wherein the plurality of landmarks (Yamamoto, see at least [0017] “a POI 210 may be within the inner environment 204 and may relate to a system of a vehicle (such as the infotainment center, navigation, climate control, etc.) with which a user 202 may wish to control, inquire about, or interface with. Additionally, other POIs 212, 214 may exist in the outer environment 206. These may correspond to locations (such as addresses, buildings, parks, rest areas, etc.), businesses (such as stores, gas stations, retail shops, restaurants, hotels, etc.), roads or streets, objects, or any point of interest 212, 214 about which a user may wish to inquire or provide a command..”)
includes an indication of the location of an index finger of an occupant of the vehicle. (Yamamoto, see at least [0017] “the visual sensors 104 provide suitable resolution to capture and allow determination of, for example, a direction in which a user's eyes are directed, head movements, and even fine and/or quick finger movements”)
Regarding Claim 7, Yamamoto in view of Herz discloses The system of claim 6, (see rejection above)
Yamamoto further discloses wherein the plurality of landmarks (Yamamoto, see at least 0037] and [0017] “a POI 210 may be within the inner environment 204 and may relate to a system of a vehicle (such as the infotainment center, navigation, climate control, etc.) with which a user 202 may wish to control, inquire about, or interface with. Additionally, other POIs 212, 214 may exist in the outer environment 206. These may correspond to locations (such as addresses, buildings, parks, rest areas, etc.), businesses (such as stores, gas stations, retail shops, restaurants, hotels, etc.), roads or streets, objects, or any point of interest 212, 214 about which a user may wish to inquire or provide a command...”)
includes an indication of the location of a wrist of an occupant of the vehicle. (Yamamoto, see at least [0035] “For example, a coarse user indication may be a finger point, a hand directional indication, or a head nod. However, a fine user indication may constitute much finer movement such as moving a finger up or down slightly (possibly while keeping the arm steady).”)
Regarding Claim 8, Yamamoto in view of Herz discloses The system of claim 5, (see rejection above)
Yamamoto further discloses wherein the computing device is configured to calculate the pointing vector based at least in part on calculating a best-fit vector for the plurality of landmarks. (Yamamoto, see at least [0035] “For example, a coarse user indication may be a finger point, a hand directional indication, or a head nod. However, a fine user indication may constitute much finer movement such as moving a finger up or down slightly (possibly while keeping the arm steady).” And also see [0037] and [0017] “a POI 210 may be within the inner environment 204 and may relate to a system of a vehicle (such as the infotainment center, navigation, climate control, etc.) with which a user 202 may wish to control, inquire about, or interface with. Additionally, other POIs 212, 214 may exist in the outer environment 206. These may correspond to locations (such as addresses, buildings, parks, rest areas, etc.), businesses (such as stores, gas stations, retail shops, restaurants, hotels, etc.), roads or streets, objects, or any point of interest 212, 214 about which a user may wish to inquire or provide a command...”)
Regarding Claim 9, Yamamoto in view of Herz discloses The system of claim 8, (see rejection above)
Yamamoto further discloses wherein the computing device is configured to determine the item of interest based at least in part on a geographic location of the vehicle at the time the user indication of interest occurred. (Yamamoto, see at least [0019] “The user indication recognition system 100 may also include or be coupled to one or more telemetry sensors 108. Non-limiting examples of telemetry sensors 108 can include weather sensors, traffic condition sensors (in a vehicular setting), accelerometers, compasses, or other sensors configured to measure a configuration, attitude, position, speed, velocity, orientation, steering wheel angle (in a vehicular setting), and the like. In various approaches, a given application setting may already include multiple telemetry sensors of varying types as may be required in that particular setting, which sensors may be made available to the user indication recognition system 100 as needed or requested.”)
Regarding Claim 10, Yamamoto in view of Herz discloses The system of claim 4, (see rejection above)
wherein the computing device is configured to calculate the pointing vector (Weyers, see at least [0068] “The second set of features 610 may include features related to the occupant's body (for example a segmentation), related to the occupant's hands (for example crops, a segmentation, or a finger point vector, for example a vector indicating a location or direction to which the occupant points with one of his fingers), and/or related to the occupant's face (for example a crop, a segmentation, and/or a direction of view, for example a vector indicating a location or direction to which the occupant directs his face or to which he is looking).”) based at least in part on the first sensor data that includes time of flight sensor data. (Weyers, see at least [0049] “The sensor data 102 may include (grayscale) image sequences and corresponding depth information recorded by a time of flight camera.”)
Accordingly, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to utilize the teachings as in modified Yamamoto with the teachings of Weyers to include the technique of determining a pointing vector of a gesture based on collected sensor data because this would allow to enable for the point of interest of a user to be identified in a more accurate manner. This would further improve the effectiveness of sensing of the internal/external environment of a vehicle and vehicle safety actions as a result during hazardous situations.
Regarding Claim 11, Yamamoto in view of Herz discloses The system of claim 4, (see rejection above)
Yamamoto further discloses wherein the computing device is configured to calculate the pointing vector based at least in part on the first sensor data that includes image sensor data. (Yamamoto, see at least [0017] “The visual sensors 104 may be used to detect visual objects both in the inner environment and in the outer environment. As described, the visual sensors 104 may be part of a “machine vision” system that may also include the at least one processing device 102. Typically, the machine vision system will include at least two visual sensors 104 to be able to capture stereoscopic images. With these stereoscopic images, the processing device 102 can implement various algorithms to determine the contents of one or more captured images and be able to relate the determined images to an action, function, or feature. For example, the machine vision system may be able to capture hand gestures, finger movements, the direction in which a user is pointing or looking, or other macro- or micro-details pertaining to the user.”)
Regarding Claim 12, Yamamoto in view of Weyers discloses The system of claim 1, (see rejection above)
Yamamoto further discloses wherein the user indication of interest is the utterance of a predefined trigger word. (Yamamoto, see at least [0039] “After analyzing the data stored within the historical data to make a determination as to whether the results of a particular search can be filtered to generate a narrowed set of search results in this manner, the system may automatically filter the results based upon that prior behavior, or can prompt the user to filter the results in accordance with their prior behavior.”)
Regarding Claim 13, Yamamoto in view of Weyers discloses The system of claim 12, (see rejection above)
Yamamoto further discloses wherein the computing device is configured to determine the item of interest based at least in part on calculation of a gazing vector. (Yamamoto, see at least [0058] “For example, the processing device 102 may refer to the direction in which the user's eyes are looking or were looking immediately before or after the audible indication. As will be understood by those of skill in the art, any number of audible indications can be utilized which the processing device 102 can be configured to use to distinguish between POIs within the inner 204 or outer environment 206.”)
Regarding Claim 14, Yamamoto in view of Weyers discloses The system of claim 13, (see rejection above)
Yamamoto further discloses wherein the computing device is configured to calculate the gazing vector based at least in part on the first sensor data. (Yamamoto, see at least [0058] “For example, the processing device 102 may refer to the direction in which the user's eyes are looking or were looking immediately before or after the audible indication. As will be understood by those of skill in the art, any number of audible indications can be utilized which the processing device 102 can be configured to use to distinguish between POIs within the inner 204 or outer environment 206.” Examiner notes that the audible indication is an example of sensor data collected.)
Regarding Claim 15, Yamamoto in view of Weyers discloses The system of claim 14, (see rejection above)
Yamamoto further discloses wherein the first sensor data includes at least one image that captures the orientation of the head of an occupant of the vehicle. (Yamamoto, see at least [0017] “the machine vision system will include at least two visual sensors 104 to be able to capture stereoscopic images. With these stereoscopic images, the processing device 102 can implement various algorithms to determine the contents of one or more captured images and be able to relate the determined images to an action, function, or feature. For example, the machine vision system may be able to capture hand gestures, finger movements, the direction in which a user is pointing or looking, or other macro- or micro-details pertaining to the user.”)
Regarding Claim 16, Yamamoto in view of Weyers discloses The system of claim 15, (see rejection above)
Yamamoto further discloses wherein the computing device is configured to calculate the gazing vector (Yamamoto, see at least [0058] “For example, the processing device 102 may refer to the direction in which the user's eyes are looking or were looking immediately before or after the audible indication. As will be understood by those of skill in the art, any number of audible indications can be utilized which the processing device 102 can be configured to use to distinguish between POIs within the inner 204 or outer environment 206.” Examiner notes that the audible indication is an example of sensor data collected.)
based at least in part on the orientation of the head of the occupant of the vehicle. (Yamamoto, see at least [0060] “Such a determination may be made simply by determining one or more particular POIs 210 within the inner environment 204 that the user is specifically addressing or that the user is specifically gesturing toward. Further, such a determination may be made by selecting one or more POIs 212 within an outer boundary 216 (discussed further below) if the outer environment 206 is indicated. However, by one approach, if a directional indication is provided (e.g., a finger point, hand gesture, head nod, eye movement, etc.), the processing device 102 may be able to select a particular one POI or a set of POIs within the outer environment 206.”)
Regarding Claim 19, Yamamoto in view of Weyers discloses The system of claim 14, (see rejection above)
Yamamoto further discloses wherein the computing device is configured to calculate the gazing vector based at least in part on the first sensor data that includes image data. (Yamamoto, see at least [0017] “the visual sensors 104 may be part of a “machine vision” system that may also include the at least one processing device 102. Typically, the machine vision system will include at least two visual sensors 104 to be able to capture stereoscopic images. With these stereoscopic images, the processing device 102 can implement various algorithms to determine the contents of one or more captured images and be able to relate the determined images to an action, function, or feature. For example, the machine vision system may be able to capture hand gestures, finger movements, the direction in which a user is pointing or looking, or other macro- or micro-details pertaining to the user.”)
Regarding Claim 20, Yamamoto in view of Weyers discloses The system of claim 19, (see rejection above)
Yamamoto further discloses wherein the first sensor data includes audio sensor data. (Yamamoto, see at least [0018] “The user indication recognition system 100 may also include or be coupled to one or more audio sensors 106.”)
8. Claim 17 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Yamamoto (US Pub. No. 20150362988) in view of Weyers (US Pub. No. 20210081689) in further view of Noble (US Pub. No. 20190122044).
Regarding Claim 17, Yamamoto in view of Herz discloses The system of claim 16, (see rejection above)
wherein the computing device is configured to calculate the gazing vector based at least in part on determining a roll value of the orientation of the head of the occupant of the vehicle, wherein the computing device is configured to calculate the gazing vector based at least in part on determining a pitch value of the orientation of the head of the occupant of the vehicle, wherein the computing device is configured to calculate the gazing vector based at least in part on determining a yaw value of the orientation of the head of the occupant of the vehicle. (Weyers, see at least [0068] “features related to the occupant's body (for example a segmentation), related to the occupant's hands (for example crops, a segmentation, or a finger point vector, for example a vector indicating a location or direction to which the occupant points with one of his fingers), and/or related to the occupant's face (for example a crop, a segmentation, and/or a direction of view, for example a vector indicating a location or direction to which the occupant directs his face or to which he is looking).” ** vector related to direction of view is interested as a gazing vector)
Accordingly, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to utilize the teachings as in modified Yamamoto with the teachings of Weyers to include the technique of determining a gazing vector of a gesture based on collected sensor data because this would allow to enable for the point of interest of a user to be identified in a more accurate manner. This would further improve the effectiveness of sensing of the internal/external environment of a vehicle and vehicle safety actions as a result during hazardous situations.
Modified Yamamoto does not explicitly disclose calculate the gazing vector based at least in part on determining a pitch value of the orientation of the head of the occupant of the vehicle
calculate the gazing vector based at least in part on determining a yaw value of the orientation of the head of the occupant of the vehicle
However, Noble discloses calculate the gazing vector based at least in part on determining a pitch value of the orientation of the head of the occupant of the vehicle
calculate the gazing vector based at least in part on determining a yaw value of the orientation of the head of the occupant of the vehicle (Noble, see at least [0111] wherein the determination of an amount of change in head pitch and head yaw provides a method for inferring a driver’s eye and head gaze during the event.)
Accordingly, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to utilize the teachings as in Yamamoto in view of Weyers with the teachings of Noble to include the technique of using head pitch and head yaw to infer a driver’s eye and head gaze in which can be used by the system of Weyers to further calculate a gazing vector of a user. This would further improve the effectiveness of sensing of the internal/external environment of a vehicle and vehicle safety actions as a result during hazardous situations.
Regarding Claim 18, Yamamoto in view of Herz discloses The system of claim 17, (see rejection above)
Yamamoto further discloses wherein the computing device is configured to calculate the item of interest based at least in part on a geographic location of the vehicle at the time the user indication of interest occurred. (Yamamoto, see at least [0019] “The user indication recognition system 100 may also include or be coupled to one or more telemetry sensors 108. Non-limiting examples of telemetry sensors 108 can include weather sensors, traffic condition sensors (in a vehicular setting), accelerometers, compasses, or other sensors configured to measure a configuration, attitude, position, speed, velocity, orientation, steering wheel angle (in a vehicular setting), and the like. In various approaches, a given application setting may already include multiple telemetry sensors of varying types as may be required in that particular setting, which sensors may be made available to the user indication recognition system 100 as needed or requested.”)
Relevant Art
The prior art made of record and not relied upon are considered pertinent to the applicant’s disclosure:
US8818716 – A user, such as the driver of a vehicle, to retrieve information related to a point of interest (POI) near the vehicle by pointing at the POI or performing some other gesture to identify the POI. Gesture recognition is performed on the gesture to generate a target region that includes the POI that the user identified. After generating the target region, information about the POI can be retrieved by querying a server-based POI service with the target region or by searching in a micromap that is stored locally. The retrieved POI information can then be provided to the user via a display and/or speaker in the vehicle. This process beneficially allows a user to rapidly identify and retrieve information about a POI near the vehicle without having to navigate a user interface by manipulating a touchscreen or physical buttons.
US 20180078444 – This invention described a Spatial Recognition Device (SRD) based on a mounted high precision measurement sensor that enables the user to detect and navigate around objects and obstacles like traditional white cane would, but without contact, and also allow the user to identify the location and shape of objects to enable identifying such obstacles and objects. The device can also provide route guidance through a GPS and collision avoidance through onboard and/or cloud based or hybrid computational systems, accelerometers and other sensors. The device enables sensing ranges from approximately one inch to a maximum range dependent on the high precision measurement sensor used in the device. This device would not be mounted on a “white cane” or similar assistive device.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NADA MAHYOOB ALQADERI whose telephone number is (571) 272-2052. The examiner can normally be reached Monday – Friday, 8AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Rachid Bendidi can be reached on (571) 272-4896. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NADA MAHYOOB ALQADERI/Examiner, Art Unit 3664
/REDHWAN K MAWARI/Primary Examiner, Art Unit 3664
Prosecution Insights