Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
The examiner interprets “A computer readable storage medium, as that term is used in the present disclosure, is not to be construed as storage in the form of transitory signals per se…” to mean that the claimed computer readable storage media recited in claim 10 and 17 are non-transitory computer readable storage media (specification; page 17, para [56]).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-11 and 13-20 are rejected under 35 U.S.C. 103 as being unpatentable over Rowell (US 20190096256 A1; hereinafter Rowell) in view of Toyoda et al. (US 20180322783 A1; hereinafter Toyoda).
Regarding claim 17, Rowell teaches a computer system comprising a processor and one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media, the program instructions executable by the processor to cause the processor to perform operations (“a vehicle system includes one or more sensors, one or more processors communicatively coupled to the one or more sensors, a memory module communicatively coupled to the one or more processors, and machine readable instructions stored in the memory module. The machine readable instructions, when executed by the processor, cause the system to…” (page 1, para [0005]).
“The one or more memory modules 106 may comprise RAM, ROM, flash memories, hard drives, or any device capable of storing machine readable instructions such that the machine readable instructions can be accessed by the one or more processors 102. The one or more memory modules 106 may be non-transient memory modules,” (page 2, para [0025]).
The one or more computer readable store media includes any of RAM, ROM, flash memories, hard drives.) comprising:
establishing an object behavior database, wherein the object behavior database comprises a set of historical object behavior data corresponding to one or more objects (“In some embodiments, the one or more memory modules 106 may include a database that includes object classifications and behavior characteristics for each of the object classifications. For example, the one or more memory modules 106 may include object classifications and behavior characteristics as described in Table 1…,” (pages 2-3, para [0026]; page 3, Table 1).
“…the one or more processors 102 may estimate the probability of each of the trajectories 310, 320, and 330 based on the behavior statistics database for standing people at the location where the object 120 is. The behavior statistics database may be stored in the one or more memory modules 106. The behavior statistics database may store object classification, location of the object, and likelihood of behavior. For example, the behavior statistics database may indicate that out of 100 standing people at the location where the object 120 is, 80 people crossed the crosswalk 340, 15 people moved toward the South direction, and 5 people moved toward the North direction,” (pages 4-5, para [0043]).
“the one or more sensors 110 may keep track of the object 120, and obtain the actual trajectory of the object 120. If the object 120 follows the trajectory 310, then one or more processors 102 update the statistics database for standing people by adding the incident of the object 120 crossing the crosswalk 340. If the object 120 shows a new behavior that is not registered in the statistics database, such as jaywalking the street, the one or more processors 102 update the behavior statistics database by adding the incident of the object 120 jaywalking the street. In this regard, behavior of standing people at the location where the object 120 is may be updated in real time,” (page 5, para [0047]).
An object behavior database includes a behavior statistics database. A set of historical object behavior data includes a likelihood of behavior for an object and / or a behavior characteristic.);
scanning a surrounding environment for one or more objects (“The vehicle 100 includes one or more sensors 110 for detecting and monitoring objects within a certain distance. The one or more sensors 110 may be any sensors configured to detect an object, including, for example, cameras, laser sensors, proximity sensors, LIDAR sensors, ultrasonic sensors, and the like. The one or more sensors 110 may be placed on the front, side, top and/or back of the vehicle 100. In embodiments, the one or more sensors 110 may detect objects within a predetermined range, for example, an object 120, an object 130, an object 140, an object 150, an object 160, an object 170, etc,” (pages 1-2, para [0019]; Fig 1).
“the vehicle system 200 comprises one or more sensors 110 configured to detect and monitor objects within a threshold distance…The one or more sensors 110 may include an omni-directional camera, or a panoramic camera,” (page 3, para [0027]).
“In step 910, the one or more processors 102 of the vehicle 100 detect an object based on one or more signals output by the one or more sensors 110. For example, a camera of the vehicle 100 may capture the objects 120, 130, 140, 150, 160, and 170,” (page 7, para [0073]).);
detecting an object in the surrounding environment (“The one or more processors 102 may implement instructions for comparing the image captured by the one or more sensors 110 with the sample images stored in the one or more memory modules 106 using image recognition processing, and classifying objects in the captured image based on the comparison. For example, if the captured image includes an object that matches with a sample image of a ball based on image recognition processing, that object is classified as a ball,” (page 3, para [0029]).
“In step 910, the one or more processors 102 of the vehicle 100 detect an object based on one or more signals output by the one or more sensors 110. For example, a camera of the vehicle 100 may capture the objects 120, 130, 140, 150, 160, and 170,” (page 7, para [0073]).
“The one or more sensors 110 detect the object 130. The one or more processors 102 of the vehicle system 200 classify the object 130 into an object classification of a pet by comparing the captured image of the object 130 with sample images for pets stored in the one or more memory modules 106,” (pages 5-6, para [0054]).);
retrieving a subset of historical object behavior data corresponding to the object detected in the surrounding environment (“The one or more processors 102 may implement instructions for predicting a trajectory of classified objects based on behavior characteristics of the classified object. For example, if an object within the threshold distance is classified as a pet, behavior characteristic for the pet (e.g., random moving direction, follow person nearby as described in Table 1 above) is retrieved, and the trajectory of the pet is predicted based on the behavior characteristic,” (page 3, para [0031]).
“the one or more processors 102 predict a trajectory of the object based on behavior characteristics of the object determined from a model corresponding to the object classification. For example, if the object is classified as a pet, the one or more processors 102 predict the trajectory of the pet based on the behavior characteristics of the pet such as ‘moving in random directions,’ ‘following a person nearby,’ etc,” (page 7, para [0075]).
“…the one or more processors 102 may estimate the probability of each of the trajectories 310, 320, and 330 based on the behavior statistics database for standing people at the location where the object 120 is. The behavior statistics database may be stored in the one or more memory modules 106. The behavior statistics database may store object classification, location of the object, and likelihood of behavior. For example, the behavior statistics database may indicate that out of 100 standing people at the location where the object 120 is, 80 people crossed the crosswalk 340, 15 people moved toward the South direction, and 5 people moved toward the North direction,” (pages 4-5, para [0043]; Fig 3).
A subset of historical object behavior data includes a behavior characteristic and / or a likelihood of behavior.);
displaying media corresponding to the object based in part on the subset of historical object behavior data and a current contextual scenario (“If the object 160 is likely to move from a non-obstacle positon to an obstacle position, the screen 108 may highlight the object 160 and provide a warning that the object 160 is approaching the vehicle 100. A predicted trajectory of the object 160 may also be displayed on the screen 108. In some embodiments, only one or more objects that are likely to move from a non-obstacle position to an obstacle position may be displayed on the screen 108 in order to effectively inform objects at issue.” (page 4, para [0036]).
“if the predicted trajectory of the object overlaps with the predicted trajectory of the vehicle, it is determined that the detected object is likely to move from a non-obstacle position to an obstacle position,” (page 3, para [0032]).
“the one or more processors 102 may predict the probability that the predicted trajectory of the object 120 overlap with the trajectory of the vehicle 100 based on the probability of the trajectory of the object 120 and the probability of the trajectory of the vehicle 100. For example, if the one or more processors 102 estimate the probability of the trajectory 350 as 50% for the vehicle 100 and the one or more processors 102 estimate the probability of each of the trajectories 310, 320, and 330, as 80%, 5%, and 15%, the one or more processors 102 may predict the probability of overlap as 40%,” (page 5, para [0046]; Fig 3).
“the behavior statistics database may indicate that out of 100 standing people at the location where the object 120 is, 80 people crossed the crosswalk 340, 15 people moved toward the South direction, and 5 people moved toward the North direction. In this regard, the one or more processors 102 may assign 80% likelihood to the trajectory 310, 15% likelihood to the trajectory 330, and 5% likelihood to the trajectory 320,” (pages 4-5, para [0043]; Fig 3).
Media includes the visual elements corresponding to the display of highlighting the object, providing a warning, display of predicted trajectory of the object, and / or display of one or more objects that are likely to move from a non-obstacle position to an obstacle position.
Determining which objects and / or trajectories get displayed on the screen is based on whether the predicted trajectories overlap (are likely to move from a non-obstacle position to an obstacle position). Predicting the trajectories is based on a likelihood of behavior (subset of historical object behavior data). Thus, displaying media corresponding to the object is based in part on the subset of historical object behavior data.
“The one or more processors 102 may predict three trajectories 310, 320, and 330 for the object 120 based on at least one of the behavior characteristic of the object 120, the current speed of the object 120, the current moving direction of the object 120, the current position of the object 120, and road marks or signs proximate to the object 120,” (page 4, para [0042]).
A subset of historical object behavior data may also include a behavior characteristic of the object. A current contextual scenario includes one or more of the current speed of the object 120, the current moving direction of the object 120, the current position of the object 120, and road marks or signs proximate to the object 120. Thus, displaying media corresponding to the object is based in part on a current contextual scenario.
“The speaker 114 may warn the driver by providing audible feedback when an object detected by the one or more sensors 110 is currently in an obstacle position, or is going to be in an obstacle position based on the projected trajectory of the object and the projected trajectory of the vehicle 100. For example, the speaker 114 may provide audible sound ‘Be careful of a bicycle on your right side’ or ‘Be careful of a bicycle at the corner’,” (page 4, para [0038]).
Media includes audible feedback.); and
displaying the media on a display interface (“If the object 160 is likely to move from a non-obstacle positon to an obstacle position, the screen 108 may highlight the object 160 and provide a warning that the object 160 is approaching the vehicle 100. A predicted trajectory of the object 160 may also be displayed on the screen 108. In some embodiments, only one or more objects that are likely to move from a non-obstacle position to an obstacle position may be displayed on the screen 108 in order to effectively inform objects at issue.” (page 4, para [0036]).
“the vehicle system 200 comprises a screen 108 for providing visual output such as, for example, maps, navigation, entertainment, warnings, alerts, or a combination thereof… The screen 108 may provide a warning to the driver when an object detected by the one or more sensors 110 is currently in an obstacle position, or is going to be in an obstacle position based on the projected trajectory of an object and the projected trajectory of the vehicle 100,” (pages 3-4, para [0034]).
A display interface includes a screen.).
Rowell does not explicitly teach generating media. Toyoda explicitly discloses that displaying media includes generating media (“the engagement module 230 generally includes instructions that function to control the processor 110 to render a display scenario that depicts the potential hazards in the surrounding environment… determines graphical elements and/or other aspects associated with rendering the potential hazard and controls the AR system 180 to display the graphical elements for the display scenario. Accordingly, the engagement module 230 can control the AR system 180 to display static graphical elements over locations of the potential hazards, to animate the graphical elements to portray the potential hazards as though the potential hazards were actually occurring even though they are not (e.g., animate an outline of a person stepping into roadway from between parked cars), and so on,” (Toyoda; page 4, para [0035]; page 6, para [0051]).
Generating media includes rendering a display scenario and / or animating graphical elements.)
Before the effective filling date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Toyoda to Rowell. The motivation would have been to “induce the driver to engage with the surrounding environment while also informing the driver of the potential hazards,” (Toyoda; page 1, para [0005]) and / or to “induce a heightened sense of responsibility within the driver by relating the potential hazard to the driver,” (page 6, para [0051]). Additional benefit would have been to reduce a driver’s confusion / uncertainty regarding a potential hazard and / or to improve the clarity of a hazard warning.
Regarding claims 1 and 10, they are rejected using the same citations and rationales described in the rejection of claim 17.
Regarding claim 13, Rowell in view of Toyoda teaches the computer program product of claim 10, wherein generating media corresponding to the object further comprises generating media depicting expected behavior associated with the object (Rowell; “If the object 160 is likely to move from a non-obstacle positon to an obstacle position, the screen 108 may highlight the object 160 and provide a warning that the object 160 is approaching the vehicle 100. A predicted trajectory of the object 160 may also be displayed on the screen 108,” (page 4, para [0036]).
“In step 930, the one or more processors 102 predict a trajectory of the object based on behavior characteristics of the object determined from a model corresponding to the object classification. For example, if the object is classified as a pet, the one or more processors 102 predict the trajectory of the pet based on the behavior characteristics of the pet such as “moving in random directions,” “following a person nearby,” etc. Thus, the one or more processors 102 may predict the trajectory of the pet as a trajectory in random direction or a trajectory toward a person nearby. As another example, if the object is classified as a vehicle, the one or more processors 102 predict the trajectory of the vehicle based on the behavior characteristics of the vehicle such as “following roads,” “making turn at intersection,” “following traffic rules,” etc,” (Rowell; page 7, para [0075]).
Expected behavior associated with the object includes a predicted trajectory of the object.).
Regarding claim 2, it is rejected using the same citations and rationales described in the rejection of claim 13.
Regarding claim 20, Rowell in view of Toyoda teaches the computer system of claim 17, further comprising performing a responsive action based on the object, the subset of historical object behavior data, and the current contextual scenario (Rowell; “The one or more processors 102 may implement instructions for providing a warning when it is determined that the detected object is likely to move from a non-obstacle position to an obstacle position, e.g., by providing a visual, audible, or tactile feedback,” (page 3, para [0033]).
“In step 950, the one or more processors 102 instruct an output device to provide a warning in response to determination that the detected object is likely to move from a non-obstacle position to an obstacle position based on the predicted trajectory of the object and the predicted trajectory of the vehicle 100. For example, if the predicted trajectory of the vehicle 100 is the trajectory 360 in FIG. 7, and the predicted trajectory of the object 160 is the trajectory 710 in FIG. 7, then it is determined that the ball is likely to move from a non-obstacle position to an obstacle position. In response to the determination that the ball is likely to move from a non-obstacle position to an obstacle position, an output device of the vehicle 100 provides a warning to a driver,” (Rowell; page 7, para [0077]; Fig 7).
“The trajectory of the vehicle 100 may be predicted based on various factors including status of left or right turn signals, the current lane on which the vehicle 100 is present, a GPS route, etc,” (Rowell; page 3, para [0030]).
A responsive action includes a warning and / or a visual, audible, or tactile feedback. A current contextual scenario includes the trajectory of the vehicle, status of left or right turn signals, the current lane on which the vehicle 100 is present, and / or a GPS route. A responsive action based on the current contextual scenario includes a warning in response to determination that the detected object is likely to move from a non-obstacle position to an obstacle position based on the predicted trajectory of the vehicle.
“the one or more processors 102 may predict the probability that the predicted trajectory of the object 120 overlap with the trajectory of the vehicle 100 based on the probability of the trajectory of the object 120 and the probability of the trajectory of the vehicle 100. For example, if the one or more processors 102 estimate the probability of the trajectory 350 as 50% for the vehicle 100 and the one or more processors 102 estimate the probability of each of the trajectories 310, 320, and 330, as 80%, 5%, and 15%, the one or more processors 102 may predict the probability of overlap as 40% (50% times 80%) because the trajectory 310 is the only trajectory that overlaps with the trajectory 350 of the vehicle 100, the probability of the trajectory 310 is 80%, and the probability of the trajectory 350 is 50%. The level of feedback may be determined based on the probability of overlap. For example, the level of warning by the speaker 114 may be proportion to the probability of overlap,” (Rowell; page 5, para [0046]).
“if the predicted trajectory of the object overlaps with the predicted trajectory of the vehicle, it is determined that the detected object is likely to move from a non-obstacle position to an obstacle position,” (Rowell; page 3, para [0032]).
“the behavior statistics database may indicate that out of 100 standing people at the location where the object 120 is, 80 people crossed the crosswalk 340, 15 people moved toward the South direction, and 5 people moved toward the North direction. In this regard, the one or more processors 102 may assign 80% likelihood to the trajectory 310, 15% likelihood to the trajectory 330, and 5% likelihood to the trajectory 320,” (Rowell; pages 4-5, para [0043]; Fig 3).
A subset of historical object behavior data includes a behavior characteristic and / or a likelihood of behavior. Performing a responsive action based on the subset of historical object behavior data includes providing a warning in response to determination that the detected object is likely to move from a non-obstacle position to an obstacle position based on the predicted trajectory of the object. Predicting the trajectory of the object is based on a likelihood of behavior and / or a behavior characteristic (subset of historical object behavior data).).
Regarding claim 6, it is rejected using the same citations and rationales described in the rejection of claim 20.
Regarding claim 14, Rowell in view of Toyoda teaches the computer program product of claim 10, wherein generating media corresponding to the object further comprises generating media depicting a potential consequence associated with the object (Toyoda; “the engagement system 170 identifies the blind corners 610 and 620 and renders an outline of a person 630 walking from behind the blind corner 610. Moreover, the engagement system 170 can also illustrate a cartoonish collision symbol 640 to further emphasize the possibility of a collision with a person or object crossing the roadway from the blind corner 610,” (page 7, para [0062] Fig 4; Fig 5; Fig 6).
“the engagement module 230 renders graphical elements as animations of the potential hazards as though the potential hazards are occurring when, in fact, the potential hazards are not occurring. Thus, the portrayal or imitation of the potential hazard by the engagement module 230 through the AR system 180 acts as a warning to the driver about the potential hazard. Moreover, when the engagement module 230 renders the graphical elements in familiar forms such as with shapes of persons including children, families, bouncing balls, animals (e.g., dogs, cats, etc.), etc., the graphical elements induce a heightened sense of responsibility within the driver by relating the potential hazard to the driver. That is, when the driver is aware of a particular nature of the potential hazard such as a child running into the roadway, the driver generally becomes more aware and cautious of the potential hazard since the graphical elements facilitate relating the potential hazard to the driver. Thus, the driver may be self-motivated to engage the driving tasks to avoid the potential hazards,” (Toyoda; page 6, para [0051]).
“By way of example, the engagement module 230 can control the AR system 180 to render animations of objects intersecting with a current/planned/future trajectory of the vehicle 100 from unexpected locations in the surrounding environment,… animations of persons walking/running into a path of the vehicle 100, and so on…,” (Toyoda; page 6, para [0052]).
“Additionally, the engagement system 170 identifies a person 530 walking a dog 540 in a cavalier manner by, for example, not leashing the dog 540 and/or by using a reel-style leash that permits the dog 540 to abruptly run from the person 530. In either case, the engagement system 170 animates a moving arrow 550 indicating that the dog 540 and/or the person 530 may abruptly enter the roadway in front of the vehicle 100,” (Toyoda; page 7, para [0061]; Fig 4; Fig 5; Fig 6).
Media depicting a potential consequence includes a collision symbol, animations of objects intersecting with a current/planned/future trajectory of the vehicle and / or animations of persons walking/running into a path of the vehicle.).
Before the effective filling date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Toyoda to Rowell. The motivation would have been to “induce a heightened sense of responsibility within the driver by relating the potential hazard to the driver,” (page 6, para [0051]) and / or to “emphasize the possibility of a collision… and motivate the driver to engage driving tasks,” (page 7, para [0062]). Additional benefit would have been to reduce a driver’s confusion / uncertainty regarding a potential hazard and / or to improve the clarity of a hazard warning.
Regarding claim 3, it is rejected using the same citations and rationales described in the rejection of claim 14.
Regarding claim 15, Rowell in view of Toyoda teaches the computer program product of claim 10, wherein generating media corresponding to the object further comprises generating media depicting a suggested alteration to a driving pattern to prevent a potential consequence associated with the object (Toyoda; “the engagement module 230 provides graphics that explain hazards (e.g., why something is a hazard), explain control behaviors for avoiding the potential hazards (e.g., suggested speed, suggested distance, suggested steering maneuver, suggested indicator light usage, etc.) and so on…,” (page 6, para [0050]).
“The potential hazards are, for example, objects, or other aspects for which the engagement module 230 is to render a display scenario including one or more visuals within the AR system 180. Thus, display scenarios can include potential hazards (e.g., pedestrians, moving objects, locations of potential jaywalking or unseen pedestrians, etc.)…” (Toyoda; page 5, para [0043]).
“The graphical elements can identify the potential hazards, inform the driver about why the potential hazards are a concern, and can also indicate how the driver should react (e.g., slow down, maintain certain distance, avoid passing, etc.),” (Toyoda; page 4, para [0037]).
“the engagement system 170 can also render graphical elements to inform the driver of how to control the vehicle 100. That is, the engagement module 230 can render graphics indicating to the driver that the vehicle 100 should slow down, merge away from the cars 440, and so on,” (page 7, para [0059]).
A suggested alteration to a driving pattern to prevent a potential consequence includes control behaviors for avoiding the potential hazards.).
Before the effective filling date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Toyoda to Rowell. The motivation would have been to “educate the driver on the potential hazards and how to control the vehicle in relation to the potential hazards,” (page 4, para [0036]) and / or to “induce behaviors (e.g., reflexive behaviors) in the driver,” (page 4, para [0037]). Additional benefit would have been to reduce the driver’s cognitive load and / or reaction time.
Regarding claim 4, it is rejected using the same citations and rationales described in the rejection of claim 15.
Regarding claim 18, Rowell in view of Toyoda teaches the computer system of claim 17, wherein generating media corresponding to the object further comprises generating media depicting expected behavior associated with the object (Rowell; “If the object 160 is likely to move from a non-obstacle positon to an obstacle position, the screen 108 may highlight the object 160 and provide a warning that the object 160 is approaching the vehicle 100. A predicted trajectory of the object 160 may also be displayed on the screen 108,” (page 4, para [0036]).
“In step 930, the one or more processors 102 predict a trajectory of the object based on behavior characteristics of the object determined from a model corresponding to the object classification. For example, if the object is classified as a pet, the one or more processors 102 predict the trajectory of the pet based on the behavior characteristics of the pet such as “moving in random directions,” “following a person nearby,” etc. Thus, the one or more processors 102 may predict the trajectory of the pet as a trajectory in random direction or a trajectory toward a person nearby. As another example, if the object is classified as a vehicle, the one or more processors 102 predict the trajectory of the vehicle based on the behavior characteristics of the vehicle such as “following roads,” “making turn at intersection,” “following traffic rules,” etc,” (Rowell; page 7, para [0075]).
Expected behavior associated with the object includes a predicted trajectory of the object.),
generating media depicting a potential consequence associated with the object (Toyoda; “the engagement system 170 identifies the blind corners 610 and 620 and renders an outline of a person 630 walking from behind the blind corner 610. Moreover, the engagement system 170 can also illustrate a cartoonish collision symbol 640 to further emphasize the possibility of a collision with a person or object crossing the roadway from the blind corner 610,” (page 7, para [0062] Fig 4; Fig 5; Fig 6).
“the engagement module 230 renders graphical elements as animations of the potential hazards as though the potential hazards are occurring when, in fact, the potential hazards are not occurring. Thus, the portrayal or imitation of the potential hazard by the engagement module 230 through the AR system 180 acts as a warning to the driver about the potential hazard. Moreover, when the engagement module 230 renders the graphical elements in familiar forms such as with shapes of persons including children, families, bouncing balls, animals (e.g., dogs, cats, etc.), etc., the graphical elements induce a heightened sense of responsibility within the driver by relating the potential hazard to the driver. That is, when the driver is aware of a particular nature of the potential hazard such as a child running into the roadway, the driver generally becomes more aware and cautious of the potential hazard since the graphical elements facilitate relating the potential hazard to the driver. Thus, the driver may be self-motivated to engage the driving tasks to avoid the potential hazards,” (Toyoda; page 6, para [0051]).
“By way of example, the engagement module 230 can control the AR system 180 to render animations of objects intersecting with a current/planned/future trajectory of the vehicle 100 from unexpected locations in the surrounding environment,… animations of persons walking/running into a path of the vehicle 100, and so on…,” (Toyoda; page 6, para [0052]).
“Additionally, the engagement system 170 identifies a person 530 walking a dog 540 in a cavalier manner by, for example, not leashing the dog 540 and/or by using a reel-style leash that permits the dog 540 to abruptly run from the person 530. In either case, the engagement system 170 animates a moving arrow 550 indicating that the dog 540 and/or the person 530 may abruptly enter the roadway in front of the vehicle 100,” (Toyoda; page 7, para [0061]; Fig 4; Fig 5; Fig 6).
Media depicting a potential consequence includes a collision symbol, animations of objects intersecting with a current/planned/future trajectory of the vehicle and / or animations of persons walking/running into a path of the vehicle.), and
generating media depicting a suggested alteration to a driving pattern to prevent the potential consequence associated with the object (Toyoda; “the engagement module 230 provides graphics that explain hazards (e.g., why something is a hazard), explain control behaviors for avoiding the potential hazards (e.g., suggested speed, suggested distance, suggested steering maneuver, suggested indicator light usage, etc.) and so on…,” (page 6, para [0050]).
“The potential hazards are, for example, objects, or other aspects for which the engagement module 230 is to render a display scenario including one or more visuals within the AR system 180. Thus, display scenarios can include potential hazards (e.g., pedestrians, moving objects, locations of potential jaywalking or unseen pedestrians, etc.)…” (Toyoda; page 5, para [0043]).
“The graphical elements can identify the potential hazards, inform the driver about why the potential hazards are a concern, and can also indicate how the driver should react (e.g., slow down, maintain certain distance, avoid passing, etc.),” (Toyoda; page 4, para [0037]).
“the engagement system 170 can also render graphical elements to inform the driver of how to control the vehicle 100. That is, the engagement module 230 can render graphics indicating to the driver that the vehicle 100 should slow down, merge away from the cars 440, and so on,” (page 7, para [0059]).
A suggested alteration to a driving pattern to prevent a potential consequence includes control behaviors for avoiding the potential hazards.).
Before the effective filling date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Toyoda to Rowell. The motivation would have been to “educate the driver on the potential hazards and how to control the vehicle in relation to the potential hazards,” (Toyoda; page 4, para [0036]) and / or to “emphasize the possibility of a collision… and motivate the driver to engage driving tasks,” (Toyoda; page 7, para [0062]). Additional benefit would have been to reduce the driver’s cognitive load and / or reaction time.
Regarding claim 16, Rowell in view of Toyoda teaches the computer program product of claim 10, wherein generating media corresponding to the object further comprises generating media depicting a safe distance to perform an alteration to a driving pattern to prevent a potential consequence associated with the object (Toyoda; “the engagement module 230 provides graphics that explain hazards (e.g., why something is a hazard), explain control behaviors for avoiding the potential hazards (e.g., suggested speed, suggested distance, suggested steering maneuver, suggested indicator light usage, etc.) and so on…,” (page 6, para [0050]).
“The graphical elements can identify the potential hazards, inform the driver about why the potential hazards are a concern, and can also indicate how the driver should react (e.g., slow down, maintain certain distance, avoid passing, etc.),” (Toyoda; page 4, para [0037]).
“the system 170 controls the AR system 180 to animate a flashing text warning 520 within the roadway for the vehicle 100 to stay back from the truck 510,” (Toyoda; para [0060]; Fig 5).
Generating media depicting a safe distance includes providing graphics that explain control behaviors such as a suggested distance. An alteration to a driving pattern includes a driver maintaining a following distance.).
Before the effective filling date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Toyoda to Rowell. The motivation would have been to “educate the driver on the potential hazards and how to control the vehicle in relation to the potential hazards,” (Toyoda; page 4, para [0036]) and / or to “induce behaviors (e.g., reflexive behaviors) in the driver,” (Toyoda; page 4, para [0037]). Additional benefit would have been to reduce the driver’s cognitive load and / or reaction time.
Regarding claim 5, it is rejected using the same citations and rationales described in the rejection of claim 16.
Regarding claim 19, Rowell in view of Toyoda teaches the computer system of claim 17, wherein the display interface is an augmented reality interface, and wherein displaying the media on the augmented reality interface comprises superimposing the media over a real view of the surrounding environment in real time (Toyoda; “…the AR system 180 can take many different forms but in general functions to augment or otherwise supplement viewing of objects within a real-world environment surrounding the vehicle 100. That is, for example, the AR system 180 can overlay graphics and animations of graphics using one or more AR displays in order to provide for an appearance that the graphics are integrated with the real-world. Thus, the AR system 180 can include displays integrated with a windshield, side windows, rear windows, mirrors and other aspects of the vehicle 100. In further aspects, the AR system 180 can include head-mounted displays such as goggles or glasses,” (page 3, para [0027]; page 7, para [0058]; Fig 4; Fig 5; Fig 6).
“The one or more sensors can be configured to detect, and/or sense in real-time,” (Toyoda; page 8, para [0071]).
“…the monitoring module 220 collects the sensor data every x seconds (e.g., 0.1 s) to maintain an up-to-date view of the surrounding environment and the driver,” (Toyoda; page 4, para [0039]).
“the monitoring module 220 continues to collect sensor data as the graphical elements are rendered within the AR system 180,” (Toyoda; page 6, para [0053]).
“the engagement system 170 through the noted modules 220 and 230, continuously monitors for hazards and renders different graphical elements in displays of the AR system 180 to engage the driver on driving tasks, educate the driver on the potential hazards and how to control the vehicle in relation to the potential hazards, and generally to inform the driver about the surrounding environment,” (Toyoda; page 4, para [0036]).
Superimposing media includes overlaying graphics and animations of graphics.).
Before the effective filling date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Toyoda to Rowell. The motivation would have been “to engage the driver with the vehicle and the surrounding environment of the vehicle,” (Toyoda; page 2, para [0020]) and / or to “induce a heightened sense of responsibility within the driver by relating the potential hazard to the driver,” (page 6, para [0051]). Additional benefit would have been to improve the speed at which a driver’s attention is captured and / or assist the driver to react to a hazard sooner.
Regarding claim 9, it is rejected using the same citations and rationales described in the rejection of claim 19.
Regarding claim 7, Rowell in view of Toyoda teaches the computer-implemented method of claim 6, wherein the responsive action comprises reducing vehicle speed (Toyoda; “The processor(s) 110, the engagement system 170, and/or the autonomous driving module(s) 160 may be operable to control the navigation and/or maneuvering of the vehicle 100 by controlling one or more of the vehicle systems 140 and/or components thereof. For instance, when operating in an autonomous mode, the processor(s) 110, the engagement system 170, and/or the autonomous driving module(s) 160 can control the direction and/or speed of the vehicle 100. The processor(s) 110, the engagement system 170, and/or the autonomous driving module(s) 160 can cause the vehicle 100 to accelerate (e.g., by increasing the supply of fuel provided to the engine), decelerate (e.g., by decreasing the supply of fuel to the engine and/or by applying brakes) and/or change direction…” (page 9, para [0082]).
“The autonomous driving module(s) 160 either independently or in combination with the engagement system 170 can be configured to determine travel path(s), current autonomous driving maneuvers for the vehicle 100, future autonomous driving maneuvers and/or modifications to current autonomous driving maneuvers based on data acquired by the sensor system 120, driving scene models, and/or data from any other suitable source. “Driving maneuver” means one or more actions that affect the movement of a vehicle. Examples of driving maneuvers include: accelerating, decelerating, braking, turning, moving in a lateral direction of the vehicle 100, changing travel lanes, merging into a travel lane, and/or reversing, just to name a few possibilities. The autonomous driving module(s) 160 can be configured can be configured to implement determined driving maneuvers. The autonomous driving module(s) 160 can cause, directly or indirectly, such autonomous driving maneuvers to be implemented,” (Toyoda; page 10, para [0088]).
“the engagement system can collect sensor data from sensors of the vehicle to identify the potential hazards,” (Toyoda; page 2, para [0022]).
Reducing the vehicle speed includes decelerating and / or braking.).
Before the effective filling date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Toyoda to Rowell. The motivation would have been to ensure the vehicle responds to hazards in the case that the driver fails to notice them and/ or reacts slowly. Additional benefit would have been to prevent a crash and / or reduce crash severity.
Regarding claim 8, Rowell in view of Toyoda teaches the computer-implemented method of claim 6, wherein the responsive action comprises breaking (Toyoda; “The processor(s) 110, the engagement system 170, and/or the autonomous driving module(s) 160 may be operable to control the navigation and/or maneuvering of the vehicle 100 by controlling one or more of the vehicle systems 140 and/or components thereof. For instance, when operating in an autonomous mode, the processor(s) 110, the engagement system 170, and/or the autonomous driving module(s) 160 can control the direction and/or speed of the vehicle 100. The processor(s) 110, the engagement system 170, and/or the autonomous driving module(s) 160 can cause the vehicle 100 to accelerate (e.g., by increasing the supply of fuel provided to the engine), decelerate (e.g., by decreasing the supply of fuel to the engine and/or by applying brakes) and/or change direction…” (page 9, para [0082]).
“The autonomous driving module(s) 160 either independently or in combination with the engagement system 170 can be configured to determine travel path(s), current autonomous driving maneuvers for the vehicle 100, future autonomous driving maneuvers and/or modifications to current autonomous driving maneuvers based on data acquired by the sensor system 120, driving scene models, and/or data from any other suitable source. “Driving maneuver” means one or more actions that affect the movement of a vehicle. Examples of driving maneuvers include: accelerating, decelerating, braking, turning, moving in a lateral direction of the vehicle 100, changing travel lanes, merging into a travel lane, and/or reversing, just to name a few possibilities. The autonomous driving module(s) 160 can be configured can be configured to implement determined driving maneuvers. The autonomous driving module(s) 160 can cause, directly or indirectly, such autonomous driving maneuvers to be implemented,” (Toyoda; page 10, para [0088]).
“the engagement system can collect sensor data from sensors of the vehicle to identify the potential hazards,” (Toyoda; page 2, para [0022]).).
Before the effective filling date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Toyoda to Rowell. The motivation would have been to ensure the vehicle responds to hazards in the case that the driver fails to notice them and/ or reacts slowly. Additional benefit would have been to prevent a crash and / or reduce crash severity.
Regarding claim 11, Rowell in view of Toyoda teaches the computer program product of claim 10, wherein the stored program instructions are stored in a computer readable storage device in a data processing system, and wherein the stored program instructions are transferred over a network from a remote data processing system (Toyoda; “Program code embodied on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber, cable, RF, etc., or any suitable combination of the foregoing. Computer program code for carrying out operations for aspects of the present arrangements may be written in any combination of one or more programming languages, including an object-oriented programming language such as Java™ Smalltalk, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider),” (pages 10-11, para [0093]).
“the engagement system 170 includes a memory 210 that stores a monitoring module 220 and an engagement module 230. The memory 210 is a random-access memory (RAM), read-only memory (ROM), a hard-disk drive, a flash memory, or other suitable memory for storing the modules 220 and 230. The modules 220 and 230 are, for example, computer-readable instructions that when executed by the processor 110 cause the processor 110 to perform the various functions disclosed herein,” (Toyoda; page 3, para [0028]).
A remote data processing system includes a remote computer and / or server.).
Before the effective filling date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Toyoda to Rowell. The motivation would have been to enable and / or improve the efficiency of software maintenance and / or updates.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Rowell in view of Toyoda in further view of Katsuki (US 11225259 B1; hereinafter Katsuki).
Regarding claim 12, Rowell in view of Toyoda is not relied upon teaching but Katsuki teaches the computer program product of claim 10, wherein the stored program instructions are stored in a computer readable storage device in a server data processing system (“the process software can also be automatically or semi-automatically deployed into a computer system by sending the process software to a central server or a group of central servers. The process software is then downloaded into the client computers that will execute the process software,” (col 19, lines 25-49).), and
wherein the stored program instructions are downloaded in response to a request over a network to a remote data processing system for use in a computer readable storage device associated with the remote data processing system (“Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device,” (col 17, lines 63-67; col 18, lines 1-9).
“computer 600 receives instructions related to the aforementioned methods and functionalities by downloading processor-executable instructions from a remote data processing system via network 650,” (col 13, lines 56-67; col 14, lines 1-7).
“operation 302 includes downloading the anomaly detection system 112 from a remote data processing system to a device such as a computer, a server, a vehicle 100, a system 200, or another device,” (col 11, lines 32-56).
“a cloud consumer can unilaterally provision computing capabilities, such as server time and network storage, as needed automatically without requiring human interaction with the service's provider,” (col 15, lines 1-15).
A computer that obtains instructions by downloading them has requested them. ), further comprising:
program instructions to meter use of the program instructions associated with the request (“the method is performed by one or more computers according to software that is downloaded to the one or more computers from a remote data processing system. Optionally, the method further comprises: metering a usage of the software; and generating an invoice based on metering the usage.” (col 22, lines 50-57).
“These embodiments can also include analyzing the client's operations, creating recommendations responsive to the analysis, building systems that implement subsets of the recommendations, integrating the systems into existing processes and infrastructure, metering use of the systems, allocating expenses to users of the systems, and billing, invoicing (e.g., generating an invoice), or otherwise receiving payment for use of the systems,” (col 19, lines 50-64).); and
program instructions to generate an invoice based on the metered use (“the method is performed by one or more computers according to software that is downloaded to the one or more computers from a remote data processing system. Optionally, the method further comprises: metering a usage of the software; and generating an invoice based on metering the usage.” (col 22, lines 50-57).
“These embodiments can also include analyzing the client's operations, creating recommendations responsive to the analysis, building systems that implement subsets of the recommendations, integrating the systems into existing processes and infrastructure, metering use of the systems, allocating expenses to users of the systems, and billing, invoicing (e.g., generating an invoice), or otherwise receiving payment for use of the systems,” (col 19, lines 50-64).).
Before the effective filling date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Katsuki to Rowell in view of Toyoda. The motivation would have been to enable monetization and / or receiving payment for the use of the systems / software. Additional benefit would have been to enable providing transparency to consumers regarding bills / costs associated with software use.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ERICA G THERKORN whose telephone number is (571)272-2939. The examiner can normally be reached Monday - Friday 9:00am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona Faulk can be reached at 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ERICA G THERKORN/Examiner, Art Unit 2618
/DEVONA E FAULK/Supervisory Patent Examiner, Art Unit 2618
Prosecution Insights