Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant Amendments and Remarks filed on 07/15/2026 in response to the Non-Final office action mailed on 06/03/2026 have been fully considered and are addressed as follows:
Regarding the Claim Objections: The objections of claims are withdrawn, as the amended claims have properly addressed the objections recited in the Non-Final office action.
Regarding the Claim Rejections under 35 USC § 112(b): The rejections of claims for being indefinite are withdrawn, as the amended claims have properly addressed the rejections recited in the Non-Final office action.
Regarding the Claim Rejections under 35 USC § 103: With respect to the previous claim rejections under 35 U.S.C. § 103, Applicant has amended the independent claims and these amendments have changed the scope of the original application. Therefore, the Office has supplied new grounds of rejection attached below in the FINAL office action and therefore the prior arguments are considered moot.
Regarding claim 1, Applicant alleges that:
“First, Chen discloses no audio signal and cannot teach representing any quantity by a tune or an average pitch of an audio signal.” (pg. 10, Applicant Amendments and Remarks filed on 07/15/2026)
“Second, Loeillet ties its audio to the identity and range of an external object, not to mass or physical dimensions.” (pg. 10, Applicant Amendments and Remarks filed on 07/15/2026)
“Loeillet never recites mass or physical dimensions, and never correlates a tune or a pitch to either. The Office bridges this gap only by observing that a dog, a pedestrian, a cyclist, and a vehicle happen to have different masses, and then treating Loeillet's identity-based signatures as a proxy for mass. That inference is supplied by the claims themselves, not by Loeillet. Moreover, Loeillet's audio represents an object external to the host vehicle, not the device whose planned movement path is visualized to the user, as recited.” (pg. 11, Applicant Amendments and Remarks filed on 07/15/2026)
“Third, Szafir supplies no audio representation of mass or physical dimensions, and the combination therefore fails to reach the claimed limitation.” (pg. 11, Applicant Amendments and Remarks filed on 07/15/2026)
Examiner disagrees.
Initially, Szafir and Chen are not cited to teach the limitation of “mass or physical dimensions being represented by a tune or an average pitch of an audio signal accompanying the visualization.” Szafir in view of Chen teaches the limitation of “a visualization of the at least one planned movement path relative to the user position, wherein the visualization of the at least one planned movement path has an appearance that differs for different values of at least one quantity” of “a mass or physical dimensions of the robotic device.”
Loeillet teaches the limitation of “the mass or physical dimensions being represented by a tune or an average pitch of an audio signal accompanying the visualization.” It is noted that the claimed limitation does not require the mass or physical dimensions of the object to be correlated to a tune or a pitch. The claimed limitation requires the mass or physical dimensions being “represented by” a tune or an average pitch of an audio signal. The meaning of “represent” is “to give a clear understanding or impression of: to describe or present clearly” or “to serve as a sign or symbol of.” (“Represent.” Merriam-Webster.com Dictionary, Merriam-Webster, https://www.merriam-webster.com/dictionary/represent. Accessed 17 Sep. 2026.)
Loeillet teaches that the audio signature can be defined to facilitate determination of the nature of the identified object (Loeillet at para. [0248]). The audio signature provides clear understanding or impression of mass or physical dimensions, such as shape or size of the object by generating different tunes including a vehicle horn representing a vehicle, a bicycle bell representing a cyclist or a bike, a voice representing a pedestrian, and a dog barking representing a dog. Loeillet also teaches generating visual and audio alerts to provide multimodal warnings (Loeillet at para. [0263]).
Therefore, Loeillet, as being combined with the visualization of the planned movement path as taught by Szafir in view of Chen, teaches the limitation of “mass or physical dimensions being represented by a tune or an average pitch of an audio signal accompanying the visualization.”
For at least the foregoing reasons, and the rejections outlined below, the prior art rejections are maintained.
FINAL OFFICE ACTION
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-4, 9, and 13-21 are rejected under 35 U.S.C. 103 as being unpatentable over Szafir et al. (US 2021/0094180 A1, hereinafter “Szafir”) in view of Chen et al. (US 2020/0039427 A1, hereinafter “Chen”) further in view of Loeillet et al. (US 2018/0005528 A1, hereinafter “Loeillet”).
Regarding claim 1, Szafir discloses a method of indicating a condition of at least one robotic device to a user, comprising the steps of:
obtaining, via an augmented-reality (AR) interface associated with the user, at least one planned movement path of the robotic device from a server that schedules, controls, monitors, and/or coordinates movement of the at least one robotic device (Szafir at para. [0044]: “As illustrated in FIG. 1, communications environment 100 may include one or more mobile robots 110, user 120 with a controller to teleoperate the robot 110 and an augmented reality system (e.g., head-mounted display 140) to provide contextual information to the user about the operations or intended operations of the robot”; para. [0059]: “The robot can collect and then report state and environmental data back to the monitoring service 520. The monitoring service can store the data. The head mounted display can then connect to the monitoring service and based its current location request data collected by the robot. This data can then be transmitted back to the head-mounted display where an augmented reality visualization can be generated with the data. For example, this may be useful to replay robot paths or data collection activities”; para. [0151]: “Those skilled in the relevant art(s) will recognize that portions of the monitoring service may reside on a server computer”);
obtaining, via positioning equipment in the AR interface or a communication interface of the AR interface in communication with an external positioning service, a position of the user (Szafir at para. [0056]: “Calibration operation 420 can then be initiated to ensure the proper display of the physical and virtual objects” “some augmented reality interfaces include cameras to recreate a line of sight (e.g., for closed systems) and cameras to project or render contextual data onto the display. In either case, the augmented reality interface may need to identify the position and orientation of the camera in order to properly provide a perspective to the user”; Cameras (i.e., “positioning equipment”)); and
displaying in real-time, via the AR interface, a visualization of the at least one planned movement path relative to the user position (Szafir at para. [0036]: “virtual imagery is displayed as an overlay directly in front of the user to provide an interface to the physical world, inspired by "window-on-the-world" AR applications and heads-up display technologies used for pilots” “user interface augmentations might include spatial mini-maps that provide information on the position or planned route of robots relative to the user”; para. [0046]: “At time T3, display 142 can be updated to show a real-time virtual surrogate (RVS) showing where and how the robot would fly”; para. [0079]: “FIG. 8 shows some examples of such approaches to using augmented reality to mediate collocated human-robot interactions by visually conveying robot motion intent”; para. [0081]: “The NavPoints design (illustrated in FIG. 8A) is an example of augmenting the environment. This design has a spatial focus as it provides virtual imagery that displays the robot's planned flight path as a series of X lines and navigation waypoints”).
However, Szafir does not explicitly state:
wherein the visualization of the at least one planned movement path has an appearance that differs for different values of at least one quantity, the at least one quantity being
a mass or physical dimensions of the robotic device, the mass or physical dimensions being represented by a tune or an average pitch of an audio signal accompanying the visualization.
Nevertheless, Szafir at least suggests the idea of differing the visualization of the movement path based on the direction and velocity of the robot (see Szafir at FIG. 8A and para. [0081]).
In the same field of endeavor, Chen teaches:
wherein the visualization of the at least one planned movement path has an appearance that differs for different values of at least one quantity, the at least one quantity being
a mass or physical dimensions of the robotic device, (Chen at para. [0097]: “the mobile robot 10 moves along the east direction, and a pedestrian 43 moves along the west direction. When the mobile robot 10 obtains sensing data that includes a first location of the pedestrian 43, the mobile robot 10 determines path prompt information 44 according to the sensing data, and projects and displays the path prompt information 44 on the floor in a form of an animation guide arrow by using the width of the mobile robot 10 as a projection boundary, to prompt a planned path of the mobile robot 10 to the pedestrian 43”; para. [0107]: “the width of the projection area is greater than or equal to the width of the mobile robot 10”; The width of the projection area (i.e., “an appearance that differs for different values of at least one quantity”) differs based on the width of the robot (i.e., “at least one quantity being” “physical dimensions of the robotic device”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Szafir by adding visualization of Chen with a reasonable expectation of success. Specifically, Chen teaches how to adapt the visualization of the planned movement path based on the physical dimensions of the robotic device, such as the width of the robotic device. Modifying the visualization of Szafir to have the appearance that differs based on the physical dimensions of the robotic device as taught by Chen would have been obvious to one of ordinary skill in the art. The motivation to modify the method of Szafir in view of Chen is to provide precise indication information of a planned path of a mobile robot.
However, Szafir in view of Chen does not explicitly state:
the mass or physical dimensions being represented by a tune or an average pitch of an audio signal accompanying the visualization.
In the same field of endeavor, Loeillet teaches:
the mass or physical dimensions being represented by a tune or an average pitch of an audio signal accompanying the visualization (Loeillet at para. [0222]: “The alert in the present embodiment comprises three modalities”; para. [0223]: “1. Vision-patterns using colours, pulse and motion are displayed in the visual structure of the occupant compartment”; para. [0224]: “2. Sound-directional object-based sound associated with the nature of the identified object”; para. [0240]: “The display position of the visual pattern P can change in a horizontal direction to indicate changes in the relative angular position of the identified object 6. The size and/or illumination level of the visual pattern P could also be controlled, for example to indicate a determined range to the identified object 6 and/or a determined size of the identified object 6”; para. [0248]: “The audio signature signal S7 can be generated as a function of the determined nature of the identified object 6. For example, a first audio signature can be output if the identified object 6 is identified as another vehicle (such as the sound of a vehicle horn); a second audio signature (such as the ringing of a bicycle bell) can be output if the identified object 6 is identified as a cyclist; a third audio signature (such as the sound of voices) can be output if the identified object 6 is identified as a pedestrian; and a fourth audio signature ( such as the sound of a dog barking) can be output if the identified object 6 is identified as an animal”; Dogs, pedestrians, cyclists, and vehicles have different mass or physical dimensions. Physical dimensions include a shape or size. Therefore, the mass or physical dimensions of the identified object is represented by different sounds including sounds of vehicle horn, bicycle bell, voice, and dog barking (i.e., “a tune or an average pitch of an audio signal”); para .[0263]: “The vehicle interface device has been described herein with reference to implementing three modalities, namely visual, audio and haptic feedback. It will be appreciated that the vehicle interface device could be implemented with only one of said modalities or two of said modalities”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Szafir in view of Chen by adding the audio signal of Loeillet with a reasonable expectation of success. Specifically, modifying the method of Szafir in view of Chen where the AR interface provides visualization of the planned movement path of the robotic device by adding the audio signal representing the physical dimensions of an approaching object of Loeillet would have been obvious to one of ordinary skill in the art. The motivation to modify the method of Szafir in view of Chen further in view of Loeillet is to provide enhanced perceptibility of the audio signal to avoid collision hazard.
Office Note: The meaning of the word “represent” in the limitation “being represented by” is defined as “to give a clear understanding or impression of: to describe or present clearly” or “to serve as a sign or symbol of.” (“Represent.” Merriam-Webster.com Dictionary, Merriam-Webster, https://www.merriam-webster.com/dictionary/represent. Accessed 17 Sep. 2026.)
Regarding claim 2, Szafir in view of Chen further in view of Loeillet teaches the method of claim 1.
Chen further teaches further comprising obtaining said at least one quantity (Chen at para. [0097]: “projects and displays the path prompt information 44 on the floor in a form of an animation guide arrow by using the width of the mobile robot 10 as a projection boundary, to prompt a planned path of the mobile robot 10 to the pedestrian 43”; para. [0107]: “the width of the projection area is greater than or equal to the width of the mobile robot 10”; the width of the mobile robot (i.e., “at least one quantity”) must be obtained to further utilize the width to change the projection boundary).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Szafir in view of Chen further in view of Loeillet by adding obtaining the quantity of Chen with a reasonable expectation of success. The motivation to modify the method of Szafir in view of Chen further in view of Zollner and Kuffner is to provide precise indication information to pedestrians.
Regarding claim 3, Szafir in view of Chen further in view of Loeillet teaches the method of claim 1.
Chen further teaches wherein the at least one planned movement path is a data structure which represents locations of the at least one robotic device at different points in time and which optionally includes specifics of the robotic device, such that said at least one quantity is derivable from the data structure (Chen at para. [0052]: “The sensing data includes: data related to the mobile robot 10”; para. [0054]: “the camera is configured to measure a travel location of the mobile robot 10” “The ultrasonic sensor is usually disposed on a lateral side of the mobile robot 10, and is configured to measure a distance between the mobile robot 10 and an ambient object by using ultrasonic waves”; para. [0097]: “When the mobile robot 10 obtains sensing data that includes a first location of the pedestrian 43, the mobile robot 10 determines path prompt information 44 according to the sensing data, and projects and displays the path prompt information 44 on the floor in a form of an animation guide arrow by using the width of the mobile robot 10 as a projection boundary, to prompt a planned path of the mobile robot 10 to the pedestrian 43”; The width of the robot is possibly calculated from (i.e., “derivable from”) the sensing data including the distance between the robot and an ambient object).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Szafir in view of Chen further in view of Zollner and Kuffner by adding the data structure of Chen with a reasonable expectation of success. The motivation to modify the method of Szafir in view of Chen further in view of Loeillet is to provide precise indication information to pedestrians.
Office Note: The limitation “derivable” merely indicates a possibility of being derived.
Regarding claim 4, Szafir in view of Chen further in view of Loeillet teaches the method of claim 1.
Szafir further discloses wherein the AR interface associated with the user is worn by the user (Szafir at para. [0044]: “an augmented reality system (e.g., head-mounted display 140) to provide contextual information to the user about the operations or intended operations of the robot”).
Regarding claim 9, Szafir in view of Chen further in view of Loeillet teaches the method of claim 1.
Loeillet further teaches wherein a direction of the robotic device relative to the user is represented by an imaginary point of origin of the audio signal accompanying the visualization (Loeillet at para. [0244]: “The perceived spatial location of the audio object 29 conveys information relating to the position of the identified object 6 in relation to the vehicle 2”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Szafir in view of Chen further in view of Zollner and Kuffner by adding the audio signal accompanying the visualization of Kuffner with a reasonable expectation of success. Specifically, modifying the method of Szafir in view of Chen where the AR interface provides visualization of the planned movement path of the robotic device by adding the audio signal representing the direction of an approaching object of Loeillet would have been obvious to one of ordinary skill in the art. The motivation to modify the method of Szafir in view of Chen further in view of Loeillet is to provide enhanced perceptibility of the audio signal to avoid collision hazard.
Regarding claim 13, Szafir in view of Chen further in view of Loeillet teaches the method of claim 1.
Chen further teaches wherein the visualization includes an indication of a risk of the robotic device colliding with the user (Chen at para. [0168]: “when the mobile robot 10 passes through some particular intersections, that is, when there is a large blind area between the mobile robot and a pedestrian due to blocking of the static obstacle, by means of projection and display on the target projection plane, the mobile robot 10 can provide an information prompt for the pedestrian who cannot observe the mobile robot, thus greatly reducing a collision risk”; The target projection plane (i.e., “visualization”) provides an information prompt for the pedestrian to reduce a collision risk (i.e., “an indication of a risk”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Szafir in view of Chen further in view of Loeillet by adding the indication of the risk of Chen with a reasonable expectation of success. The motivation to modify the method of Szafir in view of Chen further in view of Loeillet is to provide precise indication information to people nearby.
Regarding claim 14, Szafir in view of Chen further in view of Loeillet teaches the method of claim 13.
Chen further teaches wherein the risk of colliding with the user that exceeds a predetermined threshold is represented by any of:
a local deviation from the at least one planned movement path of particles of a particle flow,
a shift of animated pointing elements (Chen at para. [0096]: “The animation guide arrow is used for indicating a movement direction of the mobile robot 10”; para. [0097]: “the mobile robot 10 moves along the east direction, and a pedestrian 43 moves along the west direction. When the mobile robot 10 obtains sensing data that includes a first location of the pedestrian 43, the mobile robot 10 determines path prompt information 44 according to the sensing data, and projects and displays the path prompt information 44 on the floor in a form of an animation guide arrow by using the width of the mobile robot 10 as a projection boundary, to prompt a planned path of the mobile robot 10 to the pedestrian 43”; [0100]: “Because the path prompt information usually changes dynamically, the mobile robot 10 may project and display a dynamic video on the target projection plane. The dynamic video is used for indicating the planned path of the mobile robot 10”; The planned path dynamically changes based on the location of the pedestrian).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Szafir in view of Chen further in view of Loeillet by adding the shift of animated pointing elements of Chen with a reasonable expectation of success. The motivation to modify the method of Szafir in view of Chen further in view of Loeillet is to provide precise indication information to people nearby.
Regarding claim 15, Szafir discloses an information system configured to indicate a condition of one or more robotic devices to a user, the information system comprising:
a communication interface (Szafir at para. [0044]: “the robots 110, augmented reality systems, and controllers may include network communication components that enable these devices to communicate with remote servers or other portable electronic devices by transmitting and receiving wireless signals”) configured to:
obtain at least one planned movement path of the robotic devices from a server that schedules, controls, monitors, and/or coordinates movement of the at least one robotic device (Szafir at para. [0044]: “As illustrated in FIG. 1, communications environment 100 may include one or more mobile robots 110, user 120 with a controller to teleoperate the robot 110 and an augmented reality system (e.g., head-mounted display 140) to provide contextual information to the user about the operations or intended operations of the robot”; para. [0059]: “The robot can collect and then report state and environmental data back to the monitoring service 520. The monitoring service can store the data. The head mounted display can then connect to the monitoring service and based its current location request data collected by the robot. This data can then be transmitted back to the head-mounted display where an augmented reality visualization can be generated with the data. For example, this may be useful to replay robot paths or data collection activities”; para. [0151]: “Those skilled in the relevant art(s) will recognize that portions of the monitoring service may reside on a server computer”), and
obtain a position of the user (Szafir at para. [0056]: “Calibration operation 420 can then be initiated to ensure the proper display of the physical and virtual objects” “some augmented reality interfaces include cameras to recreate a line of sight (e.g., for closed systems) and cameras to project or render contextual data onto the display. In either case, the augmented reality interface may need to identify the position and orientation of the camera in order to properly provide a perspective to the user”);
an augmented-reality, AR, interface associated with the user, the AR interface having positioning equipment or being in communication with an external positioning service for obtaining the position of the user (Szafir at para. [0056]: “Calibration operation 420 can then be initiated to ensure the proper display of the physical and virtual objects” “some augmented reality interfaces include cameras to recreate a line of sight (e.g., for closed systems) and cameras to project or render contextual data onto the display. In either case, the augmented reality interface may need to identify the position and orientation of the camera in order to properly provide a perspective to the user”; Cameras (i.e., “positioning equipment”)); and
processing circuitry configured to display in real-time, by means of the AR interface, a visualization of the at least one planned movement path relative to the user position (Szafir at para. [0036]: “virtual imagery is displayed as an overlay directly in front of the user to provide an interface to the physical world, inspired by "window-on-the-world" AR applications and heads-up display technologies used for pilots” “user interface augmentations might include spatial mini-maps that provide information on the position or planned route of robots relative to the user”; para. [0046]: “At time T3, display 142 can be updated to show a real-time virtual surrogate (RVS) showing where and how the robot would fly”; para. [0079]: “FIG. 8 shows some examples of such approaches to using augmented reality to mediate collocated human-robot interactions by visually conveying robot motion intent”; para. [0081]: “The NavPoints design (illustrated in FIG. 8A) is an example of augmenting the environment. This design has a spatial focus as it provides virtual imagery that displays the robot's planned flight path as a series of X lines and navigation waypoints”).
However, Szafir does not explicitly state:
wherein the visualization of the at least one planned movement path has an appearance that differs for different values of at least one quantity, the at least one quantity being
a mass or physical dimensions of the robotic device, the mass or physical dimensions being represented by a tune or an average pitch of an audio signal accompanying the visualization.
Nevertheless, Szafir at least suggests the idea of differing the visualization of the movement path based on the direction and velocity of the robot (see Szafir at FIG. 8A and para. [0081]).
In the same field of endeavor, Chen teaches:
wherein the visualization of the at least one planned movement path has an appearance that differs for different values of at least one quantity, the at least one quantity being
a mass or physical dimensions of the robotic device, (Chen at para. [0097]: “the mobile robot 10 moves along the east direction, and a pedestrian 43 moves along the west direction. When the mobile robot 10 obtains sensing data that includes a first location of the pedestrian 43, the mobile robot 10 determines path prompt information 44 according to the sensing data, and projects and displays the path prompt information 44 on the floor in a form of an animation guide arrow by using the width of the mobile robot 10 as a projection boundary, to prompt a planned path of the mobile robot 10 to the pedestrian 43”; para. [0107]: “the width of the projection area is greater than or equal to the width of the mobile robot 10”; The width of the projection area (i.e., “an appearance that differs for different values of at least one quantity”) differs based on the width of the robot (i.e., “at least one quantity being” “physical dimensions of the robotic device”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Szafir by adding visualization of Chen with a reasonable expectation of success. Specifically, Chen teaches how to adapt the visualization of the planned movement path based on the physical dimensions of the robotic device, such as the width of the robotic device. Modifying the visualization of Szafir to have the appearance that differs based on the physical dimensions of the robotic device as taught by Chen would have been obvious to one of ordinary skill in the art. The motivation to modify the method of Szafir in view of Chen is to provide precise indication information of a planned path of a mobile robot.
However, Szafir in view of Chen does not explicitly state:
the mass or physical dimensions being represented by a tune or an average pitch of an audio signal accompanying the visualization.
In the same field of endeavor, Loeillet teaches:
the mass or physical dimensions being represented by a tune or an average pitch of an audio signal (Loeillet at para. [0222]: “The alert in the present embodiment comprises three modalities”; para. [0223]: “1. Vision-patterns using colours, pulse and motion are displayed in the visual structure of the occupant compartment”; para. [0224]: “2. Sound-directional object-based sound associated with the nature of the identified object”; para. [0240]: “The display position of the visual pattern P can change in a horizontal direction to indicate changes in the relative angular position of the identified object 6. The size and/or illumination level of the visual pattern P could also be controlled, for example to indicate a determined range to the identified object 6 and/or a determined size of the identified object 6”; para. [0248]: “The audio signature signal S7 can be generated as a function of the determined nature of the identified object 6. For example, a first audio signature can be output if the identified object 6 is identified as another vehicle (such as the sound of a vehicle horn); a second audio signature (such as the ringing of a bicycle bell) can be output if the identified object 6 is identified as a cyclist; a third audio signature (such as the sound of voices) can be output if the identified object 6 is identified as a pedestrian; and a fourth audio signature ( such as the sound of a dog barking) can be output if the identified object 6 is identified as an animal”; Dogs, pedestrians, cyclists, and vehicles have different mass or physical dimensions. Physical dimensions include a shape or size. Therefore, the mass or physical dimensions of the identified object is represented by different sounds including sounds of vehicle horn, bicycle bell, voice, and dog barking (i.e., “a tune or an average pitch of an audio signal”); para .[0263]: “The vehicle interface device has been described herein with reference to implementing three modalities, namely visual, audio and haptic feedback. It will be appreciated that the vehicle interface device could be implemented with only one of said modalities or two of said modalities”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Szafir in view of Chen by adding the audio signal of Loeillet with a reasonable expectation of success. Specifically, modifying the method of Szafir in view of Chen where the AR interface provides visualization of the planned movement path of the robotic device by adding the audio signal representing the physical dimensions of an approaching object of Loeillet would have been obvious to one of ordinary skill in the art. The motivation to modify the method of Szafir in view of Chen further in view of Loeillet is to provide enhanced perceptibility of the audio signal to avoid collision hazard.
Regarding claim 16, Szafir discloses a computer program product comprising a non-transitory computer readable medium having instructions to cause an information system configured to indicate a condition of one or more robotic devices to a user, the information system having:
a communication interface (Szafir at para. [0044]: “the robots 110, augmented reality systems, and controllers may include network communication components that enable these devices to communicate with remote servers or other portable electronic devices by transmitting and receiving wireless signals”) configured to:
obtain at least one planned movement path of the robotic devices from a server that schedules, controls, monitors, and/or coordinates movement of the at least one robotic device (Szafir at para. [0044]: “As illustrated in FIG. 1, communications environment 100 may include one or more mobile robots 110, user 120 with a controller to teleoperate the robot 110 and an augmented reality system (e.g., head-mounted display 140) to provide contextual information to the user about the operations or intended operations of the robot”; para. [0059]: “The robot can collect and then report state and environmental data back to the monitoring service 520. The monitoring service can store the data. The head mounted display can then connect to the monitoring service and based its current location request data collected by the robot. This data can then be transmitted back to the head-mounted display where an augmented reality visualization can be generated with the data. For example, this may be useful to replay robot paths or data collection activities”; para. [0151]: “Those skilled in the relevant art(s) will recognize that portions of the monitoring service may reside on a server computer”), and
obtain a position of the user (Szafir at para. [0056]: “Calibration operation 420 can then be initiated to ensure the proper display of the physical and virtual objects” “some augmented reality interfaces include cameras to recreate a line of sight (e.g., for closed systems) and cameras to project or render contextual data onto the display. In either case, the augmented reality interface may need to identify the position and orientation of the camera in order to properly provide a perspective to the user”);
an augmented-reality, AR, interface associated with the user, the AR interface having positioning equipment or being in communication with an external positioning service for obtaining the position of the user (Szafir at para. [0056]: “Calibration operation 420 can then be initiated to ensure the proper display of the physical and virtual objects” “some augmented reality interfaces include cameras to recreate a line of sight (e.g., for closed systems) and cameras to project or render contextual data onto the display. In either case, the augmented reality interface may need to identify the position and orientation of the camera in order to properly provide a perspective to the user”; Cameras (i.e., “positioning equipment”)); and
processing circuitry configured to display in real-time, by means of the AR interface, a visualization of the at least one planned movement path relative to the user position (Szafir at para. [0036]: “virtual imagery is displayed as an overlay directly in front of the user to provide an interface to the physical world, inspired by "window-on-the-world" AR applications and heads-up display technologies used for pilots” “user interface augmentations might include spatial mini-maps that provide information on the position or planned route of robots relative to the user”; para. [0046]: “At time T3, display 142 can be updated to show a real-time virtual surrogate (RVS) showing where and how the robot would fly”; para. [0079]: “FIG. 8 shows some examples of such approaches to using augmented reality to mediate collocated human-robot interactions by visually conveying robot motion intent”; para. [0081]: “The NavPoints design (illustrated in FIG. 8A) is an example of augmenting the environment. This design has a spatial focus as it provides virtual imagery that displays the robot's planned flight path as a series of X lines and navigation waypoints”).
However, Szafir does not explicitly state:
wherein the visualization of the at least one planned movement path has an appearance that differs for different values of at least one quantity, the at least one quantity being
a mass or physical dimensions of the robotic device, the mass or physical dimensions being represented by a tune or an average pitch of an audio signal accompanying the visualization.
Nevertheless, Szafir at least suggests the idea of differing the visualization of the movement path based on the direction and velocity of the robot (see Szafir at FIG. 8A and para. [0081]).
In the same field of endeavor, Chen teaches:
wherein the visualization of the at least one planned movement path has an appearance that differs for different values of at least one quantity, the at least one quantity being
a mass or physical dimensions of the robotic device, (Chen at para. [0097]: “the mobile robot 10 moves along the east direction, and a pedestrian 43 moves along the west direction. When the mobile robot 10 obtains sensing data that includes a first location of the pedestrian 43, the mobile robot 10 determines path prompt information 44 according to the sensing data, and projects and displays the path prompt information 44 on the floor in a form of an animation guide arrow by using the width of the mobile robot 10 as a projection boundary, to prompt a planned path of the mobile robot 10 to the pedestrian 43”; para. [0107]: “the width of the projection area is greater than or equal to the width of the mobile robot 10”; The width of the projection area (i.e., “an appearance that differs for different values of at least one quantity”) differs based on the width of the robot (i.e., “at least one quantity being” “physical dimensions of the robotic device”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Szafir by adding visualization of Chen with a reasonable expectation of success. The motivation to modify the method of Szafir in view of Chen is to provide precise indication information of a planned path of a mobile robot (see Chen at para. [0020]).
However, Szafir in view of Chen does not explicitly state:
the mass or physical dimensions being represented by a tune or an average pitch of an audio signal accompanying the visualization.
In the same field of endeavor, Loeillet teaches:
the mass or physical dimensions being represented by a tune or an average pitch of an audio signal (Loeillet at para. [0222]: “The alert in the present embodiment comprises three modalities”; para. [0223]: “1. Vision-patterns using colours, pulse and motion are displayed in the visual structure of the occupant compartment”; para. [0224]: “2. Sound-directional object-based sound associated with the nature of the identified object”; para. [0240]: “The display position of the visual pattern P can change in a horizontal direction to indicate changes in the relative angular position of the identified object 6. The size and/or illumination level of the visual pattern P could also be controlled, for example to indicate a determined range to the identified object 6 and/or a determined size of the identified object 6”; para. [0248]: “The audio signature signal S7 can be generated as a function of the determined nature of the identified object 6. For example, a first audio signature can be output if the identified object 6 is identified as another vehicle (such as the sound of a vehicle horn); a second audio signature (such as the ringing of a bicycle bell) can be output if the identified object 6 is identified as a cyclist; a third audio signature (such as the sound of voices) can be output if the identified object 6 is identified as a pedestrian; and a fourth audio signature ( such as the sound of a dog barking) can be output if the identified object 6 is identified as an animal”; Dogs, pedestrians, cyclists, and vehicles have different mass or physical dimensions. Physical dimensions include a shape or size. Therefore, the mass or physical dimensions of the identified object is represented by different sounds including sounds of vehicle horn, bicycle bell, voice, and dog barking (i.e., “a tune or an average pitch of an audio signal”); para .[0263]: “The vehicle interface device has been described herein with reference to implementing three modalities, namely visual, audio and haptic feedback. It will be appreciated that the vehicle interface device could be implemented with only one of said modalities or two of said modalities”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Szafir in view of Chen by adding the audio signal of Loeillet with a reasonable expectation of success. Specifically, modifying the method of Szafir in view of Chen where the AR interface provides visualization of the planned movement path of the robotic device by adding the audio signal representing the physical dimensions of an approaching object of Loeillet would have been obvious to one of ordinary skill in the art. The motivation to modify the method of Szafir in view of Chen further in view of Loeillet is to provide enhanced perceptibility of the audio signal to avoid collision hazard.
Regarding claim 17, Szafir discloses a non-transitory data carrier, comprising:
a non-transitory computer readable medium having stored thereon a computer program comprising instructions which, when executed by an information system, cause the information system to indicate a condition of one or more robotic devices to a user (Szafir at para. [0143]: “Computers may employ central processing unit (CPU) or processor to process information”), the information system having:
a communication interface (Szafir at para. [0044]: “the robots 110, augmented reality systems, and controllers may include network communication components that enable these devices to communicate with remote servers or other portable electronic devices by transmitting and receiving wireless signals”) configured to:
obtain at least one planned movement path of the robotic devices from a server that schedules, controls, monitors, and/or coordinates movement of the at least one robotic device (Szafir at para. [0044]: “As illustrated in FIG. 1, communications environment 100 may include one or more mobile robots 110, user 120 with a controller to teleoperate the robot 110 and an augmented reality system (e.g., head-mounted display 140) to provide contextual information to the user about the operations or intended operations of the robot”; para. [0059]: “The robot can collect and then report state and environmental data back to the monitoring service 520. The monitoring service can store the data. The head mounted display can then connect to the monitoring service and based its current location request data collected by the robot. This data can then be transmitted back to the head-mounted display where an augmented reality visualization can be generated with the data. For example, this may be useful to replay robot paths or data collection activities”; para. [0151]: “Those skilled in the relevant art(s) will recognize that portions of the monitoring service may reside on a server computer”), and
obtain a position of the user (Szafir at para. [0056]: “Calibration operation 420 can then be initiated to ensure the proper display of the physical and virtual objects” “some augmented reality interfaces include cameras to recreate a line of sight (e.g., for closed systems) and cameras to project or render contextual data onto the display. In either case, the augmented reality interface may need to identify the position and orientation of the camera in order to properly provide a perspective to the user”);
an augmented-reality, AR, interface associated with the user, the AR interface having positioning equipment or being in communication with an external positioning service for obtaining the position of the user (Szafir at para. [0056]: “Calibration operation 420 can then be initiated to ensure the proper display of the physical and virtual objects” “some augmented reality interfaces include cameras to recreate a line of sight (e.g., for closed systems) and cameras to project or render contextual data onto the display. In either case, the augmented reality interface may need to identify the position and orientation of the camera in order to properly provide a perspective to the user”; Cameras (i.e., “positioning equipment”)); and
processing circuitry configured to display in real-time, by means of the AR interface, a visualization of the at least one planned movement path relative to the user position (Szafir at para. [0036]: “virtual imagery is displayed as an overlay directly in front of the user to provide an interface to the physical world, inspired by "window-on-the-world" AR applications and heads-up display technologies used for pilots” “user interface augmentations might include spatial mini-maps that provide information on the position or planned route of robots relative to the user”; para. [0046]: “At time T3, display 142 can be updated to show a real-time virtual surrogate (RVS) showing where and how the robot would fly”; para. [0079]: “FIG. 8 shows some examples of such approaches to using augmented reality to mediate collocated human-robot interactions by visually conveying robot motion intent”; para. [0081]: “The NavPoints design (illustrated in FIG. 8A) is an example of augmenting the environment. This design has a spatial focus as it provides virtual imagery that displays the robot's planned flight path as a series of X lines and navigation waypoints”).
However, Szafir does not explicitly state:
wherein the visualization of the at least one planned movement path has an appearance that differs for different values of at least one quantity, the at least one quantity being
a mass or physical dimensions of the robotic device, the mass or physical dimensions being represented by a tune or an average pitch of an audio signal accompanying the visualization.
Nevertheless, Szafir at least suggests the idea of differing the visualization of the movement path based on the direction and velocity of the robot (see Szafir at FIG. 8A and para. [0081]).
In the same field of endeavor, Chen teaches:
wherein the visualization of the at least one planned movement path has an appearance that differs for different values of at least one quantity, the at least one quantity being
a mass or physical dimensions of the robotic device, (Chen at para. [0097]: “the mobile robot 10 moves along the east direction, and a pedestrian 43 moves along the west direction. When the mobile robot 10 obtains sensing data that includes a first location of the pedestrian 43, the mobile robot 10 determines path prompt information 44 according to the sensing data, and projects and displays the path prompt information 44 on the floor in a form of an animation guide arrow by using the width of the mobile robot 10 as a projection boundary, to prompt a planned path of the mobile robot 10 to the pedestrian 43”; para. [0107]: “the width of the projection area is greater than or equal to the width of the mobile robot 10”; The width of the projection area (i.e., “an appearance that differs for different values of at least one quantity”) differs based on the width of the robot (i.e., “at least one quantity being” “physical dimensions of the robotic device”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Szafir by adding visualization of Chen with a reasonable expectation of success. Specifically, Chen teaches how to adapt the visualization of the planned movement path based on the physical dimensions of the robotic device, such as the width of the robotic device. Modifying the visualization of Szafir to have the appearance that differs based on the physical dimensions of the robotic device as taught by Chen would have been obvious to one of ordinary skill in the art. The motivation to modify the method of Szafir in view of Chen is to provide precise indication information of a planned path of a mobile robot (see Chen at para. [0020]).
However, Schwer in view of Chen does not explicitly state:
the mass or physical dimensions being represented by a tune or an average pitch of an audio signal accompanying the visualization.
In the same field of endeavor, Loeillet teaches:
the mass or physical dimensions being represented by a tune or an average pitch of an audio signal (Loeillet at para. [0222]: “The alert in the present embodiment comprises three modalities”; para. [0223]: “1. Vision-patterns using colours, pulse and motion are displayed in the visual structure of the occupant compartment”; para. [0224]: “2. Sound-directional object-based sound associated with the nature of the identified object”; para. [0240]: “The display position of the visual pattern P can change in a horizontal direction to indicate changes in the relative angular position of the identified object 6. The size and/or illumination level of the visual pattern P could also be controlled, for example to indicate a determined range to the identified object 6 and/or a determined size of the identified object 6”; para. [0248]: “The audio signature signal S7 can be generated as a function of the determined nature of the identified object 6. For example, a first audio signature can be output if the identified object 6 is identified as another vehicle (such as the sound of a vehicle horn); a second audio signature (such as the ringing of a bicycle bell) can be output if the identified object 6 is identified as a cyclist; a third audio signature (such as the sound of voices) can be output if the identified object 6 is identified as a pedestrian; and a fourth audio signature ( such as the sound of a dog barking) can be output if the identified object 6 is identified as an animal”; Dogs, pedestrians, cyclists, and vehicles have different mass or physical dimensions. Physical dimensions include a shape or size. Therefore, the mass or physical dimensions of the identified object is represented by different sounds including sounds of vehicle horn, bicycle bell, voice, and dog barking (i.e., “a tune or an average pitch of an audio signal”); para .[0263]: “The vehicle interface device has been described herein with reference to implementing three modalities, namely visual, audio and haptic feedback. It will be appreciated that the vehicle interface device could be implemented with only one of said modalities or two of said modalities”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Szafir in view of Chen by adding the audio signal of Loeillet with a reasonable expectation of success. Specifically, modifying the method of Szafir in view of Chen where the AR interface provides visualization of the planned movement path of the robotic device by adding the audio signal representing the physical dimensions of an approaching object of Loeillet would have been obvious to one of ordinary skill in the art. The motivation to modify the method of Szafir in view of Chen further in view of Loeillet is to provide enhanced perceptibility of the audio signal to avoid collision hazard.
Regarding claim 18, Szafir in view of Chen further in view of Loeillet teaches the method of claim 1.
Chen further teaches wherein the appearance of the visualization of the at least one planned movement path differs for different values of at least one additional quantity selected from the group consisting of:
an identity of the robotic device,
an activity or task of the robotic device,
a velocity of the robotic device (Chen at para. [0150]: “The projection length is used for indicating the movement speed of the mobile robot 10. A longer projection length indicates a higher movement speed of the mobile robot 10”), and
a proximity of the robotic device to the user position.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Szafir in view of Chen further in view of Loeillet by adding visualization of Chen with a reasonable expectation of success. The motivation to modify the method of Szafir in view of Chen further in view of Loeillet to provide precise indication information to people nearby.
Regarding claim 19, Szafir in view of Chen further in view of Loeillet teaches the information system of claim 15.
Chen further teaches wherein the appearance of the visualization of the at least one planned movement path differs for different values of at least one additional quantity selected from the group consisting of:
an identity of the robotic device,
an activity or task of the robotic device,
a velocity of the robotic device (Chen at para. [0150]: “The projection length is used for indicating the movement speed of the mobile robot 10. A longer projection length indicates a higher movement speed of the mobile robot 10”), and
a proximity of the robotic device to the user position.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Szafir in view of Chen further in view of Loeillet by adding visualization of Chen with a reasonable expectation of success. The motivation to modify the system of Szafir in view of Chen further in view of Loeillet is to provide precise indication information to people nearby.
Regarding claim 20, Szafir in view of Chen further in view of Loeillet teaches the computer program product of claim 16.
Chen further teaches wherein the appearance of the visualization of the at least one planned movement path differs for different values of at least one additional quantity selected from the group consisting of:
an identity of the robotic device,
an activity or task of the robotic device,
a velocity of the robotic device (Chen at para. [0150]: “The projection length is used for indicating the movement speed of the mobile robot 10. A longer projection length indicates a higher movement speed of the mobile robot 10”), and
a proximity of the robotic device to the user position.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the product of Szafir in view of Chen further in view of Loeillet by adding visualization of Chen with a reasonable expectation of success. The motivation to modify the product of Szafir in view of Chen further in view of Loeillet is to provide precise indication information to pedestrians.
Regarding claim 21, Szafir in view of Chen further in view of Loeillet teaches the non-transitory data carrier of claim 17.
Chen further teaches wherein the appearance of the visualization of the at least one planned movement path differs for different values of at least one additional quantity selected from the group consisting of:
an identity of the robotic device,
an activity or task of the robotic device,
a velocity of the robotic device (Chen at para. [0150]: “The projection length is used for indicating the movement speed of the mobile robot 10. A longer projection length indicates a higher movement speed of the mobile robot 10”), and
a proximity of the robotic device to the user position.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the non-transitory data carrier of Szafir in view of Chen further in view of Loeillet by adding visualization of Chen with a reasonable expectation of success. The motivation to modify the non-transitory data carrier of Szafir in view of Chen further in view of Loeillet is to provide precise indication information to people nearby.
Claims 5 and 8 are rejected under 35 U.S.C. 103 as being unpatentable over Szafir in view of Chen further in view of Loeillet and Kuffner (US 2016/0055677 A1).
Regarding claim 5, Szafir in view of Chen further in view of Loeillet teaches the method of claim 1.
However, Szafir in view of Chen further in view of Loeillet does not explicitly state:
wherein an identity, an activity, or a task of the robotic device is represented by any of:
a hue of particles of a particle flow,
a hue of animated pointing elements.
In the same field of endeavor, Kuffner teaches:
wherein an identity, an activity, or a task of the robotic device is represented by any of:
a hue of particles of a particle flow,
a hue of animated pointing elements (Kuffner at para. [0102]: “the method 500 may optionally include animating the virtual representation on the augmented reality interface to illustrate the robotic device performing the task according to the planned trajectory”; para. [0128]: “a virtual representation of the action or the intent, and the virtual representation includes an indication of at least a portion of the planned trajectory of the robotic device or highlighting the object to be handled by the robotic device”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Szafir in view of Chen further in view of Loeillet by adding the hue of animated pointing elements as taught by Kuffner with a reasonable expectation of success. The motivation to modify the method of Szafir in view of Chen further in view of Loeillet and Kuffner is to inform humans nearby of an action or an intent of the robotic device.
Regarding claim 8, Szafir in view of Chen further in view of Loeillet teaches the method of claim 1.
However, Szafir in view of Chen further in view of Loeillet does not explicitly state:
wherein the appearance of the visualization is based on a sense of the at least one planned movement path.
In the same field of endeavor, Kuffner teaches:
wherein the appearance of the visualization is based on a sense of the at least one planned movement path (Kuffner at para. [0117]: “virtual representation 1010 with annotations overlaid thereon including arrows 1012 indicating that the robotic device 1002 intends to move toward the pitcher 1004”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Szafir in view of Chen further in view of Loeillet by adding the sense of the at least one planned movement path as taught by Kuffner with a reasonable expectation of success. The motivation to modify the method of Szafir in view of Chen further in view of Loeillet and Kuffner is to inform humans nearby of an action or an intent of the robotic device.
Office Note: “sense” is interpreted as “one of two opposite directions especially of motion (as of a point, line, or surface)” (“sense,” Merriam-Webster.com Dictionary, https://www.merriam-webster.com/dictionary/sense. Accessed 4/22/2025).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JISUN CHOI whose telephone number is (571)270-0710. The examiner can normally be reached Mon-Fri, 9:00 AM - 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Scott Browne can be reached on (571)270-0151. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JISUN CHOI/Examiner, Art Unit 3666
/SCOTT A BROWNE/Supervisory Patent Examiner, Art Unit 3666
Prosecution Insights