Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 19 and 20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claims do not fall within at least one of the four categories of patent eligible subject matter because they claim a computer program product comprising “a computer readable storage media”. Neither the specifications or claims state that this is a “non-transitory” computer readable storage medium and the claims therefore are attempting to claim a transitory signal, which does not fall within one of the four categories of acceptable subject matter.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 5, 13, 14, 16, 19, are rejected under 35 U.S.C. 103 as being unpatentable over Kirschner (Expectable Motion Unit: Avoiding Hazards From Human Involuntary Motions in Human-Robot Interaction) in view of Garnavi (US 10762460 B2).
Regarding claims 16, 1, 19,
Kirschner teaches:
A computer system comprising: a memory; and one or more processors in communication with the memory, (Kirschner 4B “The experimental setup introduced in [32] is used. The set-up is depicted in Fig. 3, consisting of a robot manipulator that is mounted on a table, a PC, a camera, which captures the human upper body and face, a tablet placed on a mounting at 0.44m distance from the robot base in y-direction” Note: Kirschner teaches its experimental set up includes a PC and a robot manipulator, two devices that implicitly require a processor and memory in communication to operate.) wherein the computer system is configured to perform a method, said method comprising: monitoring, by one or more processors, one or more individuals within the physical space;(Kirschner Abstract “In robotics, many control and planning schemes have been developed that ensure the human physical safety in human-robot interaction. The human psychological state and expectation towards the robot, however, are typically neglected. Even if the robot behaviour is regarded as biomechanically safe, humans may still react with rapid involuntary motion (IM) caused by startle or surprise. Obviously, such sudden, uncontrolled motions can jeopardize safety and should be prevented by any means. In this paper, we propose the Expectable Motion Unit (EMU) concept which ensures that a certain probability of IM occurrence is not exceeded in a typical HRI setting.” Kirschner 4B “Experimental procedure and design The experimental setup introduced in [32] is used. The set-up is depicted in Fig. 3, consisting of a robot manipulator that is mounted on a table, a PC, a camera, which captures the human upper body and face,”
PNG
media_image1.png
350
1152
media_image1.png
Greyscale
Note: Kirschner teaches a system for monitoring humans that attempts to recognize rapid involuntary motion caused by a startle or surprise in an environment where humans interact with robots.) detecting, by the one or more processors, a stimulus within the physical space;(Kirschner 1 “As proximity is an essential part of smooth human-robot interaction, collisions and contact (desired, undesired, or even unforeseen) may occur. In robotics, many pre- and post-collision strategies have been introduced to ensure the human physical integrity, e.g., collision detection and reaction [1], collision avoidance [2], [3], and real time model-, metrics-, or injury data-based control … An important factor that should be considered in HRI is the human expectation [10]. If the expectation is violated, then the human can react with startle and surprise [11]. This includes rapid involuntary human motions (IM), which may jeopardize safety [12].” Note: The specifications define a “stimulus” as ¶4 “A reflexive action can be a response of the human body to a stimulus or to a sudden change in the individual's environment … In a reflex action the signals do not route to the brain and are instead directed to the spinal cord and therefore, the reaction of the individual to the environmental stimulus is almost instantaneous. Reactions to stimulus by the human body are instantaneous and can include pulling away of a hand or the jerking aware of a knee.” Thus, a stimulus can be understood to simply be a sudden change in environment that can cause a reflex action in individuals in the environment. Kirschner teaches this, as it details a sudden or unexpected motion by a robot working near humans can elicit a reflex, or involuntary motion rection by a human.) determining, by the one or more processors, based on applying a model, that the stimulus will result in a hazardous condition in the physical space, (Kirschner 6 “Therefore, to deploy the EMU approach in real application scenarios, we need to 1) identify and monitor the human condition using a human profiler (eye-tracking can be used to determine the human’s level of awareness, for example), 2) select the risk matrix based on a scenario and human condition, 3) define the desired IMO probability threshold and the respective expectation curve. To fulfill human expectation context-dependent and individualized may also require learning algorithms which can be deployed on top of the general models.” 1 “An important factor that should be considered in HRI is the human expectation [10]. If the expectation is violated, then the human can react with startle and surprise [11]. This includes rapid involuntary human motions (IM), which may jeopardize safety [12]. To avoid possibly hazardous contacts even in case of IM, several authors assume the worst case human motion range and dynamics in their control and planning schemes” 3 “In this work, we propose a cognitive-grounded safety concept based on the human expectation fulfillment the so called Expectable Motion Unit approach. The EMU aims to ensure a robot performs motions which are expected by the human and thus avoids human involuntary motions in HRI by velocity scaling based on a model of human IMO … we conduct an experiment where the human reaction is analysed in a common HRI scenario, where the robot approaches the human workspace with variable motion parameters, e.g., speed, acceleration, or direction. The human reaction is recorded and classified via social cue analysis. From the experiments, we derive the relative frequency of IMO depending on the robot velocity and distance between human and robot, which are two relevant parameters that influence IMO (cf. IV-A). We call this mapping a risk matrix for IMO; see 2 c). For a certain scenario and human condition, we can then define a threshold in terms of IMO probability that shall not be exceeded. In the risk matrix, this threshold can be represented by a so-called expectation curve, which relates the current human-robot distance to an expected velocity; see Fig. 2 d). The expectation curve is integrated into the robot motion generation as EMU, which limits the robot speed to a value which is considered expectable by the human if necessary. Finally, the EMU is combined with the Safe Motion Unit (SMU), that provides a safe velocity, which is based on injury data from biomechanics collision experiments.”
PNG
media_image2.png
262
1046
media_image2.png
Greyscale
Note: Kirschner a unit that will determine that the stimulus, a change or movement by a robot in the environment, will create a hazardous condition for the nearby individuals. To do this experimental data was collected to understand the relationship between involuntary movement occurrences, or reflexive movements, and the distance and velocity of the moving robot. Thus, Kirschner teaches that from the distance and velocity of movements it can be determined that an individual is likely to make an involuntary motion.) the determining comprising: predicting, by the one or more processors, based on the model, that the stimulus will trigger a reflexive movement of at least one individual of the one or more individuals in the physical space;(Kirschner 3 and Fig. 2, cited above, teaches that from collected historical data it can be determined, or predicted, that a nearby human is most likely going to make an involuntary motion in response to a robot’s motion based on its velocity and distance to the human. Thus, Kirschner teaches the ability to predict that a stimulus will trigger a reflexive movement, aka an involuntary motion.) and determining, by the one or more processors, based on the model and based on the predicted reflexive movement that the reflexive movement will result in a hazardous condition in the physical space;(Kirschner 7 “In this paper, we proposed and validated the Expectable Motion Unit (EMU) concept, which aims at avoiding possibly hazardous human involuntary motions (IM) in human-robot interaction.” Kirschner 1, cited above, teaches when humans are working in an environment with robots contact or collisions may occur. Kirschner notes that these hazardous conditions, contacts or unsafe interactions with nearby robots often occurs from involuntary human motions, aka reflexive movement. Kirschner 3, also cited above, teaches that the distance to a human and velocity of a robot’s motion are used to determine the probability that the nearby human(s) will produce an involuntary motion, aka reflexive movement, which would result in a hazardous condition. Thus, the involuntary motion reflexive movement itself is treated as the hazardous condition in the physical space that the EMU attempts to avoid by limiting fast movement when close to individuals to prevent them from having an IM that would result in a collision or contact due to the proximity.) and initiating, by the one or more processors, a remedial action to mitigate the hazardous condition. (Kirschner Abstract “the mapping between robot velocity, robot-human distance, and the relative frequency of IM occurrence is established. This mapping is processed towards a real-time capable robot motion generator, which limits the robot velocity during task execution if necessary. The EMU is combined with the well-established Safe Motion Unit in order to integrate both physical and psychological safety knowledge and data into a holistic safety framework. In a validation experiment, it was shown that the EMU successfully avoids human IM in five out of six cases.” 3 “or a certain scenario and human condition, we can then define a threshold in terms of IMO probability that shall not be exceeded. In the risk matrix, this threshold can be represented by a so-called expectation curve, which relates the current human-robot distance to an expected velocity; see Fig. 2 d). The expectation curve is integrated into the robot motion generation as EMU, which limits the robot speed to a value which is considered expectable by the human if necessary. Finally, the EMU is combined with the Safe Motion Unit (SMU), that provides a safe velocity, which is based on injury data from biomechanics collision experiments. The combination of the two control laws improves safety and trustworthiness of an autonomous system by ensuring that both the human expectation towards its motion is fulfilled and injury is avoided.” Note: Kirschner teaches that based on the prediction that an IM is likely to occur that the expectable motion unit, EMU, can limit the robot’s movement to avoid the predicted IM from occurring and the SMU, safe motion unit, can similarly limit/control the velocity of the motion to a safe movement speed. Thus, Kirschner teaches that a remedial action can be done to mitigate a hazardous condition, where the hazardous condition would be created by a reflexive action/involuntary motion, by limiting the robot’s motion to decrease the chance of a user making a reflex reaction with the EMU, and further by the SMU which will can scale down the speed of the robot’s action to mitigate damage caused by potential contact.)
While Kirschner teaches remedial actions to mitigate a hazardous condition created by a reflexive reaction to a stimulus that a model attempts to predict, Kirschner accomplished this with a unit, the expectable motion unit, that makes models reflective actions/involuntary motions to a stimulus’s velocity and distance to a user to understand the relationship between the two. Kirschner simply models a relationship between an outcome and environment conditions and does not teach a machine learning model. A trained machine learning model that can determine or predict a hazardous condition is taught by Garnavi which teaches determining, by the one or more processors, based on applying a trained machine learning model, a hazardous condition in the physical space;(Garnavi Col. 3 Line 22 “The remote server 110 performs analytics on the information acquired by the user device 102 and workplace monitoring 106. User states (including, e.g., their physical, cognitive, and emotional state) are determined based on the collected biometric information and the workplace monitoring information and categorized using unsupervised learning. Upon the occurrence of, for example, an industrial hygiene or injury event, the states are further categorized using supervised learning to identify state sequences that precede or do not precede these events. The sequences of user states that predict events (for example, a decreased heart-rate that might indicate drowsiness) are compiled as a cognitive suite of workplace hygiene and injury predictors (abbreviated herein as cognitive WHIPs).”
Col. 4 Line 21 “Block 308 then learns state sequences, based on the data gathered about the users, that precede a hazard event. From these sequences, block 310 generates a cognitive WHIP that adaptively predicts whether a given state sequence correlates to a high risk. Block 312 uses the cognitive WHIP to generate a risk heat map 202 of the workplace. Block 314 then provides alerts to users who move into high-risk areas, including for example providing a visual or auditory alert, providing a textual description of the risk and any ameliorative or mitigating action that can be taken, notifying management of a high-risk situation, and triggering any automatic safety measures that are appropriate” Note: Garnavi teaches a trained machine learning model that will monitor individuals in a work place and make a prediction on if a workplace environment is at “high risk” of an injury. While Garnavi does not teach the reflexive movement aspect, this specific portion of the claim that a hazardous condition can be determined by a trained machine learning model is taught by Garnavi.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Kirschner with Garnavi where a model that predicts if a stimulus will trigger a hazardous environment by predicting if a reflexive movement will occur in response to the stimulus that will create the hazardous environment is a trained machine learning model.
There are several reasons that would motivate one to do so, while Kirschner is able to accurately predict if an involuntary motion, or reflexive movement, will occur within the scope of its environment (an individual interacting with a single robot) Kirschner’s simple modelling of recorded reactions given inputs would likely be too simple in a complex environment with many individuals and moving parts. Thus, if one wished for the model’s scope to be increased or for it to be more versatile a machine learning model could be used to more accurately predict outputs with more complex inputs.
Regarding claim 5,
Kirschner teaches:
The computer-implemented method of claim 1, wherein the remedial action
While Kirschner teaches that a remedial action can be taken it does not teach that it can include alerting the individuals. This is taught by Garnavi which teaches comprises alerting the at least one individual of the hazardous condition via a wearable device worn by the at least one individual. (“A method for predicting injury risk includes generating state sequences that precede a hazard event based on information regarding a user's state including user biometric information from a device worn by the user and a user's location from one or more workplace monitoring devices. A cognitive suite of workplace hygiene and injury predictors (WHIP) is generated based on the state sequences” “In addition, the remote server 110 may communicate information back to the user device 102 to provide one or more alerts to the user in the event that the user enters a high-risk state. For example, an alert 105 in the user device 102 may include an audio (e.g., an alarm or spoken warning) or visual (e.g, a flashing light or textual message) indicator.” Note: Garnavi teaches that hazardous conditions, or high-risk states, in a work environment are predicted and users with a wearable device have an alert of the predicted hazard sent to their wearable device.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Kirschner with Garnavi where the remedial action to mitigate a hazardous condition involves alerting an individual via a wearable device.
There are several reasons that would motivate one to do so, to ensure safety of workers it is important that alerts of hazardous conditions are communicated as fast as possible. If one wished to maximize safety by sending out fast, efficient alerts they could ensure that the alert goes specifically to a device being worn by the individual, removing the need to retrieve a device like a phone or laptop with the alert.
Regarding claim 13,
Kirschner teaches:
The computer-implemented method of claim 1, wherein the determining that the reflexive movement will result in the hazardous condition in the physical space is also based on at least one element of the physical space proximate to the least one individual. (Kirschner 3, cited previously, teaches that determining if a reflexive movement will result in the hazardous condition is based on an element in physical space proximate to the individual working. Specifically, Kirschner teaches a robot arm with a known distance to an individual and uses the distance to determine the probability a user will act reflexively resulting in a crash or collision, aka making a hazardous condition.)
Regarding claim 14,
Kirschner teaches:
The computer-implemented method of claim 1, wherein the remedial action is selected from the group consisting of: automatically moving at least one object in the physical space and recommending a physical change to the physical space. (Kirschner 1, and Fig. 2, cited previously, teach that the velocity of a robot arm can be limited if the probability a user will act reflexively in response to the robot arm’s motion is above an acceptable threshold. As the robot arm is a large, moving part of the physical space when Kirschner changes how the robot arm moves and operates, Kirschner is making a physical change to a physical space.)
Claims 2, 3, 4, 15, 17, 18, 20, are rejected under 35 U.S.C. 103 as being unpatentable over Kirschner (Expectable Motion Unit: Avoiding Hazards From Human Involuntary Motions in Human-Robot Interaction) in view of Garnavi (US 10762460 B2), further in view of Egashira (Near-future perception system: Previewed Reality), and further in view of Nie (Safety envelope of pedestrians upon motor vehicle conflicts identified via active avoidance behaviour)
Regarding claim 17, 2, 20,
Kirschner teaches:
The computer system of claim 16, wherein initiating the remedial action comprises:
While Kirschner has been shown to teach predicting reflexive movements and hazardous conditions it does not teach creating a visualization of hazardous conditions or that these visualizations can be virtually overlayed to a portion of the physical space. This is found in Egashira which teaches generating, by the one or more processors, a visualization of the predicted hazardous condition;(Egashira 3 “Therefore, we can forecast possible subsequent events using a dynamics simulator such as Gazebo and synthesize virtual images from the viewpoint of the user, which the user will actually see in the near future. It is also possible to show the robot motion to the user before starting the planned motion … However, by showing the planned robot motion to the user in advance by the virtual robot using VR techniques, the user can directly recognize the robot motion and can identify hazardous situations that will occur after a few seconds.” 5.1 “Next, we examine collision avoidance, which is the same as that provided by Previewed Reality 1.0. As shown in Figure 13, the user first approaches the wine bottles on the desk (Figure 13 1 ⃝). The robot then starts to move toward the user ( 2 ⃝). When the virtual robot approaches the user too closely, the color of the virtual robot becomes red ( 3 ⃝). The user notices the danger of collision and steps back ( 4 ⃝). If the virtual robot leaves a certain distance (1.6 m), the color becomes blue ( 5 ⃝).In this experiment, the safe distance is determined according to the length of the arm (0.8 m).”
PNG
media_image3.png
830
494
media_image3.png
Greyscale
Note: Egashira teaches that when a human and robot share an environment hazardous conditions can occur. Egashira teaches that these hazardous conditions can be predicted by forecasting possible subsequent events that will occur to see if there will be hazards. An example is provided above where the robot’s planned motion to move forward is visualized and shown to the user in an augmented reality environment, seen above in Fig. 13. If the planned movement of the robot will brings it too close to a user a hazardous condition will be created where the risk of a hazard can occur, in this case a collision. As seen in images 3 and 4 of Fig. 13 when the collision hazardous condition is predicted to occur a virtual visualization of the robot overlapping/colliding with the user or being in extremely close proximity is displayed, the robot is also colored in red in these visualizations to further indicate that a collision hazard is being shown.) and projecting, by the one or more processors, as a virtual overlay to at least a portion of the physical space, the visualization. (
PNG
media_image4.png
664
544
media_image4.png
Greyscale
4.1 “Figure 6 shows the concept of Previewed Reality 2.0 and an example of images seen by the user. This system adopts an AR display (HoloLens) instead of an immersive VR display (Oculus Rift DK2) and a stereo camera in Pre viewed Reality 1.0. Since a user can see a real scene through a transparent AR display, the time delay caused by image capture and transfer is drastically reduced.” Note: Egashira Fig. 6 clarifies that the aforementioned visualizations overlayed to parts of the physical environment to show users a predicted hazardous condition are virtual overlays of an augmented reality environment.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Kirschner with Egashira where the prediction of a potential hazardous condition can be used to generate a visualization of the predicted hazard to project as a virtual overlay on parts of the physical environment for the user.
There are several reasons that would motivate one to do so, it is possible that after an individual has a reflex action, or involuntary motion, taken in response to a sudden stimulus that they may still have time to make a voluntary motion, or conscious non reflexive action, to avoid a hazard. The present invention and Kirschner aim to protect individuals from hazards, if one wished to further protect individuals by helping them react properly to predicted hazards a visualization could be shown to them to allow proper time to avoid the hazard.
Kirschner does not however teach that a visualization of the reflexive action itself is made. This idea, to visualize a predicted reflexive movement is found in Nie which teaches generating, by the one or more processors, a visualization of the predicted reflexive movement (Nie page 1 “Existing safety-focused strategies for pedestrians are mostly focusing on the available motion information to predict subject trajectories, project future traffic interactions and compute time-to-collision (TTC) … In recent perspectives 24, researchers from academia and industry call for efforts from multiple disciplines to develop models and algorithms to integrate the human reaction mechanisms into the development of effective safety systems”
Nie Page 2 Categories of the natural pedestrian reactions “Simultaneous reaction behaviour in 40 experimental cases performed by 22 subjects were recorded. Pedestrians exhibited a typical “perception-decision-execution” sequence along the timeline in two given typical traffic conflicts (i.e., crossing an urban road with and without visual obstacles, labelled as traffic scene A and B) … The action categories were divided based on the pedestrian perception of the “accident vehicle” and the relative motion direction of the pedestrian body: (1) backward avoidance (BA) (13 cases, 33%) (i.e., pedestrians noticed the “bullet vehicle” and chose to move backwards for avoiding purpose; for the pedestrians who had not entered the vehicle lane, they would choose to stop); (2) forward avoidance (running) (FA) (9 cases, 23%) (i.e., rushing forwards); (3) oblique stepping (OS) (2 cases, 6%); (4) no avoidance reaction (NAR) (16 cases, 35%) (i.e., normal walking without noticing the upcoming vehicle; labelled as “collision occurred”). Pedestrians in the BA and FA categories noticed the coming vehicle, exhibited collision avoidance capability (85%, 67%) and sustained significant kinematic and posture change. The OS behaviour was essentially a startle response, where the pedestrians generally became overwhelmed and cannot avoid the collision”
Nie Page 6 Discussion “Yet, braking only is not always the best action for vehicle given the active movement of the pedestrian itself. Proper steering with motion prediction of the pedestrian reduces the hazard zones (Fig. 6) and is potential to avoid real collisions especially for highly automated transportation tools capable of sensing pedestrian information ahead of time32. Towards such applications, our results indicated a time gap of about 0.17–0.41 s (i.e., t[ps, pa]) from human perceiving signal to execution (Fig. 4a), which is comparable to the previous studies13. The activation of the pedestrian “execution” can be identified and predicted via kinematic feature (e.g., velocity change). As velocity is one of the most significant influencing factors on injury probability and severity in motor vehicle collisions7, the subsequent collision consequences can be more precisely predicted and better handled by incorporating realistic pedestrian response.”
PNG
media_image5.png
504
1006
media_image5.png
Greyscale
PNG
media_image6.png
728
898
media_image6.png
Greyscale
Note: Nie teaches that individuals in an environment where hazardous conditions may occur, pedestrians crossing the street, have their actions predicted. Nie page 6 teaches that this prediction is made to aid in a steering decision to reduce hazard zones to hopefully avoid a collision. Of the possible predicted reactions a pedestrian may take, Nie teaches that one of these is specifically a “startle reaction”, aka a reflexive movement. As seen above in Fig. 1 when the training data is being developed in the “experiment process” portion of the figure a virtual visualization of the user reacting to the stimulus, a car approaching them is made, shown in the “pedestrian reaction” box. As these reactions have a virtual visualization of them made and at least one of these reactions is a reflex reaction specifically Nie teaches making a visualization of an individual’s reflexive action.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Kirschner with Nie where: generating a visualization of hazardous condition to virtually overlay onto the physical environment for a user to see also generates a visualization of a reflexive movement to overlay.
There are several reasons that would motivate one to do so, as pointed out by Kirschner and the present invention it is often times an individual’s reaction that causes injury or a hazard rather than what they are reacting to. Thus, the real hazard or danger in some contexts may be how an individual reflexively reacts. As reflexive movements are non-voluntary the only way a user could avoid them is to avoid being in that situation to begin with. Thus, rather than simply showing a user predicted hazards safety could be further enhanced by showing them predicted reflexive movements as well.
Regarding claim 18, 3,
Kirschner teaches:
The computer system of claim 17,
Kirschner does not however teach the use of an augmented reality device to output images in real time, this is taught by Egashira which teaches wherein the projecting comprises utilizing an augmented reality output device to project a real-time image of the portion of the physical space and the virtual overlay.(
PNG
media_image3.png
830
494
media_image3.png
Greyscale
PNG
media_image4.png
664
544
media_image4.png
Greyscale
Note: The above figures and Egashira 3 and 5.1, cited previously, teach an augment reality device worn by a user that displays real time predictions of where a robot will move in the future overlayed on the physical space the user and robot exist in.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Kirschner with Egashira where image data depicting predicted reflexive movements and hazardous conditions is displayed to a user via an augmented reality device which will overlay the content on the physical environment.
There are several reasons that would motivate one to do so, many workplace hazards are by nature spontaneous and hard to predict and thus require quick action to avoid. To give workers ample time to respond to danger either from a predicted hazard or predicted reflexive movement that may cause injury the worker should be informed as soon as possible. An augment reality device allows this information to be immediately displayed to the user’s vision with no need to notify the user to use another device to view the warning.
Regarding claim 4,
Kirschner teaches:
The computer-implemented method of claim 3,
Kirschner does not however detail using augment reality glasses worn by at least one individual, this is found in Egashira which teaches: wherein the augmented reality output device comprises augmented reality glasses worn by the at least one individual. (
PNG
media_image4.png
664
544
media_image4.png
Greyscale
Note: While having a unique appearance Egashira’s chosen AR device, the Microsoft HoloLens, uses a see-through piece of glass as its display allowing virtual content to be overlayed on the user’s view of the physical environment. As the device itself allows users to peer through its glass as opposed to capturing camera footage to provide to a VR display Egashira teaches that its device can be defined as AR “glasses”.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Kirschner with Egashira where the AR device leveraged to display predicted hazards and reflexive movements to a user are AR glasses.
There are several reasons that would motivate one to do so, not all AR devices are AR glasses, some are similar to VR head mounted displays and provide a view of the user’s surroundings via a camera. Devices like these will always have an implicit lag time when sending captured camera footage to the displays, and damage to the displays or camera could prevent the user from seeing his surroundings. To ensure the AR device being used does not decrease a user’s safety by obstructing their vision AR glasses could be used which will not inhibit a user’s vision if they malfunction.
Regarding claim 15,
Kirschner teaches:
The computer-implemented method of claim 2,
Kirschner does not however detail using an AR device to project actions in the physical space with the virtual overlay, this is found in Egashira which teaches wherein the projecting comprises utilizing the augmented reality output device to project a real-time image of the portion of the physical space and the virtual overlay,(
PNG
media_image3.png
830
494
media_image3.png
Greyscale
Note: Egashira Fig. 13 shows a projection of a hazardous condition, a robot moving too close to a user which would cause a collision, to a user in AR. Egashira 5.1 and 3, cited previously in the rejection of claim 2, teach that the robot’s projected future movements are shown dynamically to the user as they interact with it wearing the AR device. Thus, the projections being shown to the user are updated regularly and shown in real time.) wherein the projecting comprises generating and projecting a visual simulation (Egashira Fig. 13, cited above, shows an example of the visual simulation of the hazardous condition, the robot moving too close to the user causing a collision, projected onto the AR view. As a 3D visual simulation is being displayed dynamically to the user it is implicit that the 3D model must have been generated by some means.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Kirschner with Egashira where an AR device is leveraged to display images projected onto the virtual overlay over the portion of physical space, and the projections are visual simulations.
There are several reasons that would motivate one to do so, when communicating information about a potential hazard that poses risk to an individual it is important that the warning is communicated clearly, and quickly. One way this could be accomplished is by leveraging an AR device, allowing the user’s vision to be immediately supplied with the warning. Similarly, the visual simulation of the hazard allows for the user to visually identify the situation immediately, without having to interpret a written warning, or other type of alert.
While Egashira teaches an AR device that shows an individual a projection of a predicted hazardous condition in the virtual overlay on the physical environment so they can avoid the hazardous condition, in this case a collision, does not project the remedial action itself. Generating a visual simulation of the remedial action is taught by Nie which teaches generating a visual simulation of the remedial action. (
PNG
media_image6.png
728
898
media_image6.png
Greyscale
Note: Nie Page 2 Categories of the natural pedestrian reactions, cited previously in the rejection of claim 2, teaches that a visual simulation of a hazardous condition is created, in this case a pedestrian crossing the road in front of a vehicle. Nie Page 2 Categories of the natural pedestrian reactions specifically refers to some reactions as reflexive responses, and some as “avoidance” responses. The “remedial action” that would mitigate the hazardous condition in this case is the pedestrian’s reaction of avoiding the vehicle as a response. While not all pedestrians take an “avoidance” response, as taught by Nie Page 2, all pedestrian reactions are made into a visual simulation as seen in Fig. 1 above. Thus, Nie teaches generating a visual simulation of a remedial action to a hazardous condition.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Kirschner with Nie where a visual simulation projected to an AR device on a virtual overlay over the physical space projects a visual simulation of the remedial action.
There are several reasons that would motivate one to do so, workplace hazards often occur suddenly with little time available to react safely. One way an individual can be helped to react with the proper remedial action in a timely manner is to visually show them the remedial action they should take. As opposed to a written warning or instructions on what to do, which takes time to read and comprehend, a visual display of the remedial action communicates the information much more quickly.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Kirschner (Expectable Motion Unit: Avoiding Hazards From Human Involuntary Motions in Human-Robot Interaction) in view of Garnavi (US 10762460 B2) and further in view of Egashira (Near-future perception system: Previewed Reality).
Regarding claim 6,
Kirschner teaches:
The computer-implemented method of claim 1, wherein monitoring the one or more individuals within the physical space comprises: identifying, by the one or more processors, one or more computing resources;(Kirschner 1 “First, we investigate the influence of robot motion parameters on the probability of human IMO in a common use case via an exploratory study involving 29 participants. The collected data and knowledge are then processed towards a human aware, real-time capable motion generator that limits the robot speed so that a certain IMO probability is not exceeded.” 4B “The experimental setup introduced in [32] is used. The set-up is depicted in Fig. 3, consisting of a robot manipulator that is mounted on a table, a PC, a camera, which captures the human upper body and face, a tablet placed on a mounting at 0.44m distance from the robot base in y-direction and a standing-chair”
PNG
media_image7.png
292
974
media_image7.png
Greyscale
Note: Here Kirschner describes its system used for monitoring and predicting hazardous conditions and reflexive movements, specifying that a PC is used. ) and utilizing, by the one or more processors, the one or more computing resources to track movements of the user to determine if the movements comprise reflexive movements based on stimuli in the environment.( 4C “The experiment requires a reliable evaluation whether the human movement can be classified as IM. Based on the assumption that IM can be measured by social cues of startle and surprise (S-S) as suggested in [32] we conduct a multi modal video analysis (facial displays, gaze, gestures/postures)” Note: From the video data recorder Kirschner teaches that is tracked to determine which, if any, movements recorded can be determined to be involuntary motion, aka reflexive movements. As clarified previously by Kirschner, the robot arm device that moves near the individual is the simulus that would initiate.)
While Kirschner teaches the use of a computing device to determine if monitored movements by individuals comprise reflexive movements based on stimuli it does not specify that the computing device referenced, a PC, is placed near or in a proximity to the individual. This is taught by Egashira which teaches
by the one or more processors, one or more computing resources proximate to the one or more individuals; (Egashira 4.1 “Since the Unity and the PhysX are both executed on the HoloLens, the time delay due to the image synthesis and the image transfer can be greatly reduced. In addition, the HoloLens is a self-contained holographic computer including a built in CPU and GPU and most of the processes required for the Previewed Reality, other than the database in the ROS-TMS, can be executed as a standalone hardware.” Note: Egashira teaches that the device worn by the user contains processors, a GPU and CPU, on the device. Thus, Egashira teaches computing resources proximate to an individual.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Kirschner with Egashira where computing resources that track movements to determine if they comprise reflexive movements uses computing resources proximate to the one or more individuals.
There are several reasons that would motivate one to do so, Kirschner and the present invention acknowledge that it is often the involuntary response or reflexive movements that individuals have that causes injury in a workplace rather than a stimulus itself. Deploying a remedial safety action in response to a reflexive movement can be difficult, as the reflective movement itself may be extremely quick. Placing the computing device in proximity to the individuals ensures that there is not a significant amount of valuable response time wasted in data transfer to and from the computing device.
Claims 7, 8 are rejected under 35 U.S.C. 103 as being unpatentable over Kirschner (Expectable Motion Unit: Avoiding Hazards From Human Involuntary Motions in Human-Robot Interaction) in view of Garnavi (US 10762460 B2), further in view of Egashira (Near-future perception system: Previewed Reality, and further in view of Moolman (WO 2022013738 A1)
Regarding claim 7,
Kirschner teaches:
The computer-implemented method of claim 6,
Kirschner does not directly specify that one of its devices used is an Internet of Things device. This is taught by wherein at least one computing resource of the one or more computing resources comprises an Internet of Things device. (Moolman Abstract “There is provided a worker health and safety system and associated method. A server is provided for receiving sensor data and health data of a plurality of workers registered at the server. A plurality of sensors are in data communication with the server and capable of sensing sensor data and communicating the sensor data to the server over a data communications network.” Page 8 Line 31 “There is provided a system and method for monitoring employee health and safety … A plurality of sensing devices may form part of an Internet of Things (loT) network. These devices may also be referred to as edge devices or endpoints, and these devices may be interconnected.” Note: Moolman teaches a similar system to Kirschner where sensing/computing devices that can monitor individuals at a workplace’s safety. Moolman teaches that one of these sensing devices is an Internet of Things device, it is known that this ‘sensor’ can also be considered a computing device/resource as to communicate with the internet by sending/receiving data some level of computing is required.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Kirschner with Moolman where one or more computing resources is an Internet of Things device.
There are several reasons that would motivate one to do so, an IoT device is built for network communication, the sending information between sensors and computing devices is a core part of the present invention and Kirschner needed to ensure safety by deploying remedial actions to mitigate detected hazards. As the sending of data over the network is a critical part of the system using a device tailored specifically for network communication, like an IoT device, provides a reliable method to do so.
Regarding claim 8,
The computer-implemented method of claim 7, wherein identifying the one or more resources comprises obtaining, by the one or more processors,
While Kirschner teaches that from recorded video data individuals can be identified, and their movements themselves can be identified to determine whether they are involuntary motions or not. Kirschner does not however teach a registration process. a registration, an individual of the one or more individuals, via the given computing device, of the one or more computing resources. (Moolman Page 10 Line 6 “The server (12) may be arranged to associate the sensor data with one of the registered workers (18). The sensor data may be generated by one or more of the sensors” Page 15 line 1 “The image capturing device(s) (28) may detect or identify whether any one of the registered workers is wearing a facemask or not. The system (10) may also be arranged to monitor other personnel or humans, such as visitors, contractors or the like, and these individuals may preferably be required to register at the server, whereafter such an individual may be treated as a worker for the purposes of data analytics.” Note: Previously Moolman abstract established that the described system is a worker health and safety system that monitors individuals on a worksite. Here, Moolman teaches that the workers on site are registered by the server for purposes of data analytics. This shows Moolman teaches a registration, as existing workers are registered for data analytics and newly detected individuals can be similarly register as taught by Moolman Page 15 line 1.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Kirschner with Moolman where the identification process of individuals includes a registration.
There are several reasons that would motivate one to do so, one aim of the present invention and Kirschner is to provide remedial actions to avoid or reduce injury from a hazard. Different individuals at different areas of a worksite may need different alerts or other types of remedial actions to address the hazard specific to their area. Registering users allows for them to be uniquely identified, which could help differentiate users for purposes of sending urgent alerts/warning, and deploying other remedial actions.
Claims 9, 10, are rejected under 35 U.S.C. 103 as being unpatentable over Kirschner (Expectable Motion Unit: Avoiding Hazards From Human Involuntary Motions in Human-Robot Interaction) in view of Garnavi (US 10762460 B2), further in view of Egashira (Near-future perception system: Previewed Reality, and further in view of Chowdhary (US 20200229710 A1)
Regarding claim 9,
Kirschner teaches:
The computer-implemented method of claim 1, wherein monitoring the one or more individuals within the physical space comprises: utilizing, by the one or more processors, the one or more computing resources to monitor physical activities of the one or more individuals, comprising movements of the one or more individuals; (Kirschner Abstract “In robotics, many control and planning schemes have been developed that ensure the human physical safety in human-robot interaction. The human psychological state and expectation towards the robot, however, are typically neglected. Even if the robot behaviour is regarded as biomechanically safe, humans may still react with rapid involuntary motion (IM) caused by startle or surprise. Obviously, such sudden, uncontrolled motions can jeopardize safety and should be prevented by any means. In this paper, we propose the Expectable Motion Unit (EMU) concept which ensures that a certain probability of IM occurrence is not exceeded in a typical HRI setting.” Kirschner 4B “Experimental procedure and design The experimental setup introduced in [32] is used. The set-up is depicted in Fig. 3, consisting of a robot manipulator that is mounted on a table, a PC, a camera, which captures the human upper body and face,” 1 “As proximity is an essential part of smooth human-robot interaction, collisions and contact (desired, undesired, or even unforeseen) may occur. In robotics, many pre- and post-collision strategies have been introduced to ensure the human physical integrity, e.g., collision detection and reaction [1], collision avoidance [2], [3], and real time model-, metrics-, or injury data-based control … An important factor that should be considered in HRI is the human expectation [10]. If the expectation is violated, then the human can react with startle and surprise [11]. This includes rapid involuntary human motions (IM), which may jeopardize safety [12].”
PNG
media_image1.png
350
1152
media_image1.png
Greyscale
Note: Kirschner teaches a system for monitoring humans that attempts to recognize rapid involuntary motion caused by a startle or surprise in an environment where humans interact with robots.) and generating, by the one or more processors, a reflexive movement profile for at least one of the one or more individuals, based on the monitoring,(Kirschner 3 “In this work, we propose a cognitive-grounded safety concept based on the human expectation fulfillment the so called Expectable Motion Unit approach. The EMU aims to ensure a robot performs motions which are expected by the human and thus avoids human involuntary motions in HRI by velocity scaling based on a model of human IMO … The first step in the derivation of the EMU concept is to understand under which circumstances IM occur in HRI. For this, we conduct an experiment where the human reaction is analysed in a common HRI scenario, where the robot approaches the human workspace with variable motion parameters, e.g., speed, acceleration, or direction. The human reaction is recorded and classified via social cue analysis. From the experiments, we derive the relative frequency of IMO depending on the robot velocity and distance between human and robot, which are two relevant parameters that influence IMO (cf. IV-A). We call this mapping a risk matrix for IMO; see 2 c). For a certain scenario and human condition, we can then define a threshold in terms of IMO probability that shall not be exceeded”
PNG
media_image2.png
262
1046
media_image2.png
Greyscale
PNG
media_image8.png
354
442
media_image8.png
Greyscale
Note: Kirschner teaches that multiple individuals are monitored to determine whether the movements they make can be determined to be reflexive movements, aka involuntary motions. As seen from table 3 above specific motions of users monitored are classified as indicating reflexive movements such as evasive head and trunk movements, rapid eyeblinks, delayed felt smile, etc… From this monitoring and analysis of movements Kirschner teaches the reflexive movement data can be mapped and modelled in relation to other pieces of data. Data profiling involves a process of analysis or review where useful info on our data can be obtained, like its relationship to other data, internal structures, patterns, etc… Kirschner teaches that a profile for reflexive movement is made by analyzing its relationship to other relevant data, in this case the velocity and distance of an individual to a robot arm. As seen above in Fig. 2 and Kirschner 3, the reflexive movement data is analyzed and mapped, aka profiled, in order to develop a relationship between reflexive movement and the stimuli that caused it.) wherein the movement profile comprises a measure indicating a probability of a movement pattern(Kirschner 3, cited above, teaches that the probability of a movement pattern occurring, specifically where the movement pattern is one of an involuntary/reflexive movement, can be predicted based on the distance and velocity of a stimulus, a robot arm, to the user.) of the movement patterns creating a hazardous condition in the physical space,(Kirschner 1 “As proximity is an essential part of smooth human-robot interaction, collisions and contact (desired, undesired, or even unforeseen) may occur. In robotics, many pre- and post-collision strategies have been introduced to ensure the human physical integrity, e.g., collision detection and reaction [1], collision avoidance [2], [3], and real time model-, metrics-, or injury data-based control … An important factor that should be considered in HRI is the human expectation [10]. If the expectation is violated, then the human can react with startle and surprise [11]. This includes rapid involuntary human motions (IM), which may jeopardize safety [12].” Note: Kirschner 1 clarifies that it the human reaction of rapid involuntary motions in response to a stimulus that causes injury such as collisions or contact with a robot in a human-robot interaction. Thus, the hazardous condition that Kirschner aims to predict and avoid is the occurrence of a reflexive movement. Kirschner 3, cited previously, teaches that the probability of creating a hazardous condition, that is the probability of a reflexive movement, in the space is determined by the movement profile. The movement profile, as clarified above, relates the velocity and distance of a robot arm to an individual to find the probability that the individual will have a reflexive action in response to the stimulus.)
While Kirschner teaches a reflexive movement profile it does not teach that the profile comprises machine learned movement patterns for specific individuals. Similarly, Kirschner teaches that the probability a hazardous condition in the physical space is determined. Kirshner does not however teach that the specific movement patterns themselves are measured to find the probability that they will create a hazardous condition. These ideas are found in Chowdhary which teaches: a movement profile for at least one of the one or more individuals, (Chowdhary Abstract “Motion activity data is collected from at least one sensor. An initial motion activity classifier function is applied to the motion activity data to produce an initial motion activity posteriorgram. Pre-processing and segmenting the motion activity data into windows produces segmented motion activity data from which sensor specific features are extracted. An updated motion activity classifier function is generated from the extracted sensor specific features. Subsequent motion activity data is also collected from the at least one sensor, and the updated motion activity classifier function is applied to the subsequent motion activity data to produce an updated motion activity posteriorgram.” ¶43 “The pattern of motion activities as recorded on sensor(s) are generally person specific as these depend on height, weight, gender, age of the person, in addition to personal style of performing the activities. The common motion activities that have personalized characteristics are walking, cycling, going upstairs, going downstairs, and jogging. Also, as mobile and wearable devices are personal devices of a user, the option of designing and using a personalized classier is available. As a result, a person dependent classifier trained on the target user's motion activity patterns will detect motion activities more accurately.” Note: Chowdhary teaches that motion data is recorded for an individual and a classifier is used to determine what motion activity, aka motion pattern, is being performed by the user. As Chowdhary teaches monitoring users and classifying their motion into specific categories for specific individual users, Chowdhary teaches the ability to make movement profiles for individuals.) wherein the movement profile comprises machine learned movement patterns for the one individual, (Chowdhary ¶53 “With reference to FIG. 3, generation of a personalized classifier using unsupervised learning is now described. After initialization of the sensors 62, 63, 65, 66, 67, 68 (Block 102), the control circuit 64 uses one or more of the sensors 62, 63, 65, 66, 67, 68 to collect a frame of motion activity data (Block 104). The control circuit 64 pre-processes the frame of motion activity data (i.e. performs noise filtering, segmenting, and windowing, further details of which will be given below), and then computes the posterior probabilities of the motion activities of that frame (Block 106) using a factory-set generalized classifier (Block 116). The set of these probabilities is called a “posteriorgram” Note: Chowdhary teaches that movement patterns like walking, jogging, etc… for an individual are classified by an unsupervised machine learning model. This shows that Chowdhary’s movement patterns identified as part of an individual’s movement profile are machine learned movement patterns.) wherein the movement profile comprises a portion of the model. (Chowdhary Abstract teaches user specific data classifying monitored motion data into different movement patterns, analogous to the claim’s movement profile. Part of the profile includes classifying the different recorded motion, which is handled by the unsupervised model as specified by Chowdhary ¶53, teaching that part of the movement profile comprises an unsupervised machine learning model.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Kirschner with Chowdhary where a reflexive movement profile can be determined for a single individual and can use a machine learning model to learn the movement patterns.
There are several reasons that would motivate one to do so, developing a reflexive movement profile for each individual, where the movement profile classifies different movement patterns requires a substantial amount of individual movement data to be processed. Machine learning models excel at processing input data to learn patterns that help make classifications, if one wished for the accuracy of movement pattern classification of reflexive movement profiles to be enhanced a machine learning model could be leveraged.
Kirschner does not however specify that the computing resources it uses are proximate to the one or more individuals. This is taught by Egashira which teaches identifying, by the one or more processors, one or more computing resources proximate to the one or more individuals; (Egashira 4.1 “Since the Unity and the PhysX are both executed on the HoloLens, the time delay due to the image synthesis and the image transfer can be greatly reduced. In addition, the HoloLens is a self-contained holographic computer including a built in CPU and GPU and most of the processes required for the Previewed Reality, other than the database in the ROS-TMS, can be executed as a standalone hardware.” Note: Egashira teaches that the device worn by the user contains processors, a GPU and CPU, on the device. Thus, Egashira teaches computing resources proximate to an individual.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Kirschner with Egashira where computing resources that track movements to determine if they comprise reflexive movements uses computing resources proximate to the one or more individuals.
There are several reasons that would motivate one to do so, Kirschner and the present invention acknowledge that it is often the involuntary response or reflexive movements that individuals have that causes injury in a workplace rather than a stimulus itself. Deploying a remedial safety action in response to a reflexive movement can be difficult, as the reflective movement itself may be extremely quick. Placing the computing device in proximity to the individuals ensures that there is not a significant amount of valuable response time wasted in data transfer to and from the computing device.
Regarding claim 10,
Kirschner teaches:
The computer-implemented method of claim 9, further comprising: identifying, based on the monitoring, the at least one of the one or more individuals performing a given movement pattern with the measure indicating the probability the given movement pattern creating the hazardous condition(Kirschner 3, cited below, teaches that each individual movement pattern, or occurrence of movement, has the probability it will create a hazardous condition predicted to determine if different movements may result in a hazardous condition.) of above a predefined threshold value that the given movement pattern indicates the hazardous condition. (Kirschner 3 “In this work, we propose a cognitive-grounded safety concept based on the human expectation fulfillment the so called Expectable Motion Unit approach. The EMU aims to ensure a robot performs motions which are expected by the human and thus avoids human involuntary motions in HRI by velocity scaling based on a model of human IMO … we conduct an experiment where the human reaction is analysed in a common HRI scenario, where the robot approaches the human workspace with variable motion parameters, e.g., speed, acceleration, or direction. The human reaction is recorded and classified via social cue analysis. From the experiments, we derive the relative frequency of IMO depending on the robot velocity and distance between human and robot, which are two relevant parameters that influence IMO (cf. IV-A). We call this mapping a risk matrix for IMO; see 2 c). For a certain scenario and human condition, we can then define a threshold in terms of IMO probability that shall not be exceeded. In the risk matrix, this threshold can be represented by a so-called expectation curve, which relates the current human-robot distance to an expected velocity; see Fig. 2 d). The expectation curve is integrated into the robot motion generation as EMU, which limits the robot speed to a value which is considered expectable by the human if necessary. Finally, the EMU is combined with the Safe Motion Unit (SMU), that provides a safe velocity, which is based on injury data from biomechanics collision experiments.”
PNG
media_image2.png
262
1046
media_image2.png
Greyscale
Note: Kirschner teaches that given a specific movement pattern the probability of a hazardous condition (reflexive movement) is determined. While the type of the given movement pattern is not considered (arm movement, head movement, etc…), the specific conditions of the given movement pattern, such as the distance to the robot arm while moving and the robot arm’s velocity, are checked to determine the chance a user will react reflexively, resulting in a hazardous condition. If the probability of a hazardous condition is too likely remedial actions should be taken to reduce the chance an individual acts reflexively. Kirschner 3 teaches that whether or not the chance is too high is determined by a predetermined threshold value. More specifically, Kirshcener 3 teaches “For a certain scenario and human condition, we can then define a threshold in terms of IMO probability that shall not be exceeded. In the risk matrix, this threshold can be represented by a so-called expectation curve, which relates the current human-robot distance to an expected velocity; see Fig. 2 d). The expectation curve is integrated into the robot motion generation as EMU, which limits the robot speed to a value which is considered expectable by the human if necessary.”)
Claims 11, 12, are rejected under 35 U.S.C. 103 as being unpatentable over Kirschner (Expectable Motion Unit: Avoiding Hazards From Human Involuntary Motions in Human-Robot Interaction) in view of Garnavi (US 10762460 B2), and further in view of Chowdhary (US 20200229710 A1)
Regarding claim 11,
Kirschner teaches:
The computer-implemented method of claim 1, further comprising: data obtained based on the monitoring. (Kirschner 6 and 3, cited previously, teach that 29 individuals are monitored to obtain data on reflexive movements.)
While Kirschner teaches a dataset obtained from monitoring individuals it does not teach that the data will be used to train a machine learning model. This is taught by Chowdhary which teaches training, by the one or more processors, the machine learning model with data comprising a corpus, wherein the data comprises: data obtained based on the monitoring. (Chowdhary Abstract, cited in the rejection of claim 9 above, teaches that individuals are monitored via at least one sensor to record their movements. Chowdhary ¶7 “The method may include generating a data selection confidence measure after generating the initial probabilistic context. The collected motion activity data may be stored in the training data set if the data selection confidence measure is greater than a lower threshold, and the collection motion activity data may not be stored in the training data set if the data selection confidence measure is less than the lower threshold.” Note: Chowdhary teaches that from raw motion data obtained from sensors monitoring individuals’ data with a good confidence measure can be made into a training data set to train an unsupervised machine learning model.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Kirschner with Chowdhary where data obtained based on monitoring individuals is used in a dataset to train a machine learning model.
There are several reasons that would motivate one to do so, while a dataset of recorded data may contain valuable insights depending on the size of the dataset it may be unrealistic for a human observer to label, identify, or otherwise extrapolate the useful information from the all the raw data. If one wished to make more efficient use of available data a machine learning model could be employed to accept the data as input to learn from.
Regarding claim 12,
Kirschner teaches:
The computer-implemented method of claim 11, wherein the data further comprises information selected from the group consisting of: historical activities in the physical space, historical hazardous conditions in the physical space, (Kirschner 1 “First, we investigate the influence of robot motion parameters on the probability of human IMO in a common use case via an exploratory study involving 29 participants. The collected data and knowledge are then processed towards a human aware, real-time capable motion generator that limits the robot speed so that a certain IMO probability is not exceeded. This safety tool is called the Expectable Motion Unit (EMU). ” Kirschner 3 “We assume that the human has a certain task-dependent expectation towards the robot’s behaviour when both are working in close proximity. TABLE II For example, the human may expect that the robot moves slowly inside the human’s workspace. Our goal is to ensure that the human expectation is fulfilled, which then leads to controlled, intended human behavior instead of possibly hazardous startle and surprise reactions.” 4B “The experimental setup introduced in [32] is used. The set-up is depicted in Fig. 3, consisting of a robot manipulator that is mounted on a table, a PC, a camera, which captures the human upper body and face, a tablet placed on a mounting at 0.44m distance from the robot base in y-direction and a standing-chair, which ensures that all participants’ heads are positioned at approximately the same height in relation to the robot start configuration; see Fig. 3a).”
PNG
media_image7.png
292
974
media_image7.png
Greyscale
Kirschner 5C “To validate whether the EMU concept reduces IMO in practice, we repeat the experimental procedure proposed in Sec. IV-B using the EMU velocity shaping. The experimental setup and and velocity profiles are depicted in Fig. 7. The desired, nominal velocity profile is shown in blue, the velocity that was shaped by the combination of EMU and SMU in green. A group of eleven participants is part of the validation experiment with an average age of 28 ± 4.4 years including six males (54,5 %) and five females (45,5 %). We expect the relative frequency of IMO observed in this experiment to be less or equal to the desired threshold of qr = 0.15.”
PNG
media_image9.png
394
484
media_image9.png
Greyscale
Note: Kirschner, when describing its data collection from monitoring, teaches specific historical actives, this being a user performing a task at a desk near a robot arm. The user is monitored and the stimulus’s impact on triggering an involuntary motion, the hazardous condition, is recoded. It is known that the recorded historical data and hazardous conditions are in the same physical space where the hazard prediction and prevention will actually take place as Kirschner 5C and Fig. 7 teach that the same set up, of an individual at a desk with the same robot arm, is used.) historical activities in spaces similar to the physical space, motion of objects in the physical space, and prevention methods to address various historical hazardous conditions in the physical space. As Kirschner has been shown to teach a data set consisting of one of the above listed conditions, Kirschner has been shown to teach this claim.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALAN GREGORY HAKALA whose telephone number is (571)272-7863. The examiner can normally be reached 8:00am-5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at (571) 270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALAN GREGORY HAKALA/ Examiner, Art Unit 2617
/KING Y POON/Supervisory Patent Examiner, Art Unit 2617
Prosecution Insights