DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 02/26/2026 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 2, 4, and 6-18 are rejected under 35 U.S.C. 103 as being unpatentable over Chao (US 10058997 B1) in view of Hickman et al. (US 10391633 B1) (Hereinafter Hickman).
Regarding Claim 1, Chao teaches a method of controlling a robot, the method comprising, at an electronic user device (See at least Fig 1 section 140, 160, Fig 2, Col 7 Lines 8-14, “…A robot control system 250 in the form of a smartphone or tablet includes a touch screen display 240 that renders a graphical user interface (not specifically referenced in FIG. 2) that includes a rendition of visual data received, for instance, from a front facing camera (not visible in FIG. 2, see 164 in FIG. 1) of control system 250…”):
receiving, from the robot, data representative of an environment of the robot, the received data indicating a location of at least one moveable object in the environment (See at least Fig 1, Col 6 Lines 49-52 “… In other implementations, one or more sensors 108 that is integral with robot 100 may provide the image feed that is rendered on graphical user interface 160 …”, Col 3 Lines 11-14 “In various implementations, detecting the one or edges may include performing edge detection on visual data sensed by one or more vision sensors…”, Col 7 Lines 47-52 “…In some implementations, records or models of tables or other environmental features stored in index 153 may have been, for instance, previously observed by one or robots while performing simultaneous localization and mapping (“SLAM”) in the environment…”, Col 8 Lines 12-17 “Once the top surface of table 270 is extrapolated, object attribute engine 156 may identify, e.g., based on SLAM data previously gathered by one or more robots and/or based on image data received from vision sensor(s) 164/108, a plurality of objects that are co-present within the extrapolated space”), the received data being generated by the robot using one or more sensors (See at least Fig 1, Col 6 Lines 49-52 “… In other implementations, one or more sensors 108 that is integral with robot 100 may provide the image feed that is rendered on graphical user interface 160 …”, Col 3 Lines 11-14 “In various implementations, detecting the one or edges may include performing edge detection on visual data sensed by one or more vision sensors…”);
in response to receipt of the representative data, displaying a representation of the environment of the robot on a graphical display of the electronic user device (See at least Fig 2, Col 7 Lines 8-14, “…A robot control system 250 in the form of a smartphone or tablet includes a touch screen display 240 that renders a graphical user interface (not specifically referenced in FIG. 2) that includes a rendition of visual data received, for instance, from a front facing camera (not visible in FIG. 2, see 164 in FIG. 1) of control system 250…”, Col 8 Lines 12-17 “Once the top surface of table 270 is extrapolated, object attribute engine 156 may identify, e.g., based on SLAM data previously gathered by one or more robots and/or based on image data received from vision sensor(s) 164/108, a plurality of objects that are co-present within the extrapolated space”);
receiving input from a user of the electronic user device indicating a desired location for the at least one moveable object in the environment of the robot (See at least Fig 4 section 404, 406, Col 10 Lines 58-61 “Referring now to FIG. 4, an example method 400 of enabling users to roughly identify a space within an environment inhabited by a plurality of objects that user wishes for robot 100 to act upon is described.”); and
in response to receipt of the user input, transmitting control data to the robot, the control data being operable to cause the robot to move the at least one object to the desired location in the environment of the robot (See at least Fig 4 section 410 operate robot to manipulate identified plurality of objects pursuant to one or more user commands).
However, Chao does not explicitly spell out …
receiving, from the robot, confirmation data confirming that the at least one moveable object has been moved to the desired location, the confirmation data including updated data representative of the environment generated by the robot using the one or more sensors; and
in response to receipt of the confirmation data, displaying an updated environment of the robot on the graphical display based on the updated data representative of the environment, wherein the updated environment indicates the location of the at least one moveable object according to the received updated data after being moved by the robot.
Hickman teaches receiving, from the robot, confirmation data confirming that the at least one moveable object has been moved to the desired location (See at least Col 31 Lines 58-60 “… The new version of the first map may include those objects that have moved locations since the last update. The server and/or robot 1400 may use the new version of the first map as an update or supplement to the original first map…”, discloses object representation through maps which is construed as confirming that an object has been moved, Para Col 28 Lines 10-18 “The server may compare the first map to the second map to determine if there are differences between the first map and the second map. If differences exist, the server may identify the differences and store information associated with the differences in the database. The server may use the differences to inventory the objects in the area. The inventory may include the movement of an object from a first location to a second location, the addition or removal of an object, etc.”), the confirmation data including updated data representative of the environment generated by the robot using the one or more sensors (See at least Col 21 Lines 61-67 “In further embodiments, the identification data may be associated with RGB-D data. RGB-D data may be used to build a 3D image or map of the object, an area, etc. In an example, the robot may obtain RGB-D data of the object using a depth camera sensor or any other sensor that may be capable of combining visual and/or 3D shape information. The robot may send the RGB-D data to the server …”, Col 22 Lines 3-9 “In some embodiments, the identification data may be associated with metadata derived from RGB-D data and/or image or range data associated with the object. Image or range data may be associated with a 3D image or map of the object, an area, etc., and may be generated from any number of sensors including stereo camera sensors, depth camera sensors, ranging sensors, sonic sensors, etc…”, Col 31 Lines 58-60 “…The new version of the first map may include those objects that have moved locations since the last update. The server and/or robot 1400 may use the new version of the first map as an update or supplement to the original first map...”)
in response to receipt of the confirmation data displaying an updated environment of the robot on the graphical display based on the updated data representative of the environment, wherein the updated environment indicates the location of the at least one moveable object according to the received updated data after being moved by the robot (See at least Col 31 Lines 42-62 “In an example, the server may update the first map associated with the coffee bar. The update may include identifying the current location of each object in the coffee bar and updating the first map to include the current location of each object in the coffee bar, regardless of whether the current location of an object has changed since the last update. Optionally, the update may include identifying the current location of objects in a subset of the coffee bar and updating the current location of the objects within the subset of the coffee bar, regardless of whether the current location of an object has changed since the last update. An example subset may include the section of the coffee bar with the coffee 1404 , 1406 , 1408 . In another example, the server may update the first map by creating a new version of the first map (or a subset thereof). The new version of the first map may include those objects that have moved locations since the last update. The server and/or robot 1400 may use the new version of the first map as an update or supplement to the original first map. In this way, the server and/or robot 1400 may limit the number of objects that may be updated at one time, for example.”, Col 11 Lines 18-22 “Similarly, the robot 402 could update the map created by the robot 408 with new information about the area (e.g., the hallway now has boxes or other obstacles), or with new information collected from sensors that the robot 408 may not have had”).
Therefore, it would have been obvious to one of the ordinary skill in the art before the
effective filing date of the claimed invention to modify the system of Chao with the teachings of Hickman and include the feature of robot sending confirmation data about moving the objects and in response to receipt of the confirmation data representative of the environment, displaying an updated environment of the robot on the graphical display which indicates the location of the at least one moveable object according to the received updated data after being moved by the robot, thereby representing the updated map information to the user through a display for easy and efficient human robot interaction.
Applicant argues on page 12 of the Applicant’s Remarks that “It is submitted, however, that the
cited art in combination fails to suggest the interconnected features set forth in claim 1. The cited art, in combination, does not suggest the robot, in response to completing the instructed action, sending a confirmation with newly acquired data of the environment to the user device and the user device, in response to the user device receiving the confirmation and updated environment data, updating the display. The cited art, in combination, does not suggest actions triggered by a robot carrying out the command.”. The Examiner respectfully disagrees. Chao teaches “receiving data representative of an environment of the robot (from the robot)” (See at least Fig 1, Col 6 Lines 49-52, Col 3 Lines 11-14, Col 7 Lines 47-52, Col 8 Lines 12-17). Chao further teaches in response to receiving this data, a representation of the environment is displayed (See at least See at least Fig 2, Col 7 Lines 8-14, Col 8 Lines 12-17). In addition, Chao teaches receiving input from a user indicating a desired location for a moveable
object in the environment (See at least Fig 4 section 404, 406, Col 10 Lines 58-61) and transmitting control data to the robot based on the user input (See at least Fig 4 section 410). Hence, Chao already teaches robot interaction with an electronic user device for the robot to perform operations according to the user input. Chao depends on Hickman in the rejection for the teachings of receiving confirmation data (from the robot) when the object has been moved - the confirmation data includes updated data representative of the environment (See at least Col 21 Lines 61-67, Col 22 Lines 3-9, Col 31 Lines 58-60, Col 28 Lines 10-18) and in response to receiving the confirmation data, display an updated environment based on the updated data in the confirmation data (See at least Col 31 Lines 42-62, Col 11 Lines 18-22). Therefore, the combination anticipates the claim limitations.
Regarding Claim 2, modified Chao teaches all the elements of claim 1. Chao further teaches the method according to claim 1, wherein the user input is received via the display of the electronic user device (See at least Fig 2, Col 6 Lines 23-26 “For example, an operator may provide, e.g., via graphical user interface 160, a rough approximation of a table top that contains a plurality of objects to be acted upon (e.g., manipulated, moved, etc.) by robot 100…”).
Regarding Claim 4, modified Chao teaches all the elements of claim 1. Chao further teaches the method according to claim 1, wherein the user input is received via a microphone, and wherein the input comprises an audible indication of the desired location for the at least one moveable object (See at least Col 12 Lines 4-8 “User interface input devices 522 may include a keyboard, pointing devices such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touch screen incorporated into the display, audio input devices such as voice recognition systems, microphones, and/or other types of input devices.”).
Regarding Claim 6, modified Chao teaches all the elements of claim 1.
However, Chao does not explicitly spell out the method according to claim 1 comprising: receiving, from the robot, request data requesting the user to provide an identifier for one or
more objects in the environment;
receiving input from a user of the electronic user device indicating a desired identifier for the at
least one object in the environment of the robot; and
transmitting, response data to the robot, the response data including the desired identifier.
Hickman teaches the method according to claim1 comprising: receiving, from the robot, request
data requesting the user to provide an identifier for one or more objects in the environment (See at least Col 23 Lines 16-19 “if the robot is not able to identify an object or its location, then the robot may prompt the user to enter the information and use entered information to reinforce one or more learning algorithms.”);
receiving input from a user of the electronic user device indicating a desired identifier for the at
least one object in the environment of the robot (See at least Col 23 Lines 16-19 “if the robot is not able to identify an object or its location, then the robot may prompt the user to enter the information and use entered information to reinforce one or more learning algorithms.”); and
transmitting, response data to the robot, the response data including the desired identifier (See
at least Col 23 Lines 16-19 “if the robot is not able to identify an object or its location, then the robot may prompt the user to enter the information and use entered information to reinforce one or more learning algorithms.”).
Therefore, it would have been obvious to one of the ordinary skill in the art before the
effective filing date of the claimed invention to modify the teachings of Chao with the teachings of Hickman and include the feature of robot requesting the user for an identifier for one or more objects in the environment and receiving the desired identifier from the user, thereby collecting data for unknown objects which will be used to manipulate the movement of objects in future.
Regarding Claim 7, Chao teaches an apparatus for use in controlling a robot at an electronic user device (See at least Fig 1 section 140, 160, Fig 2, Col 7 Lines 8-14, “…A robot control system 250 in the form of a smartphone or tablet includes a touch screen display 240 that renders a graphical user interface (not specifically referenced in FIG. 2) that includes a rendition of visual data received, for instance, from a front facing camera (not visible in FIG. 2, see 164 in FIG. 1) of control system 250…”), the apparatus being configured to:
receive, from the robot, data representative of an environment of the robot, the received data indicating a location of at least one moveable object in the environment (See at least Fig 1, Col 6 Lines 49-52 “… In other implementations, one or more sensors 108 that is integral with robot 100 may provide the image feed that is rendered on graphical user interface 160 …”, Col 3 Lines 11-14 “In various implementations, detecting the one or edges may include performing edge detection on visual data sensed by one or more vision sensors…”, Col 7 Lines 47-52 “…In some implementations, records or models of tables or other environmental features stored in index 153 may have been, for instance, previously observed by one or robots while performing simultaneous localization and mapping (“SLAM”) in the environment…”, Col 8 Lines 12-17 “Once the top surface of table 270 is extrapolated, object attribute engine 156 may identify, e.g., based on SLAM data previously gathered by one or more robots and/or based on image data received from vision sensor(s) 164/108, a plurality of objects that are co-present within the extrapolated space”), the received data being generated by the robot using one or more sensors (See at least Fig 1, Col 6 Lines 49-52 “… In other implementations, one or more sensors 108 that is integral with robot 100 may provide the image feed that is rendered on graphical user interface 160 …”, Col 3 Lines 11-14 “In various implementations, detecting the one or edges may include performing edge detection on visual data sensed by one or more vision sensors…”);
in response to receipt of the representative data, display a representation of the environment of the robot on a graphical display of the electronic user device (See at least Fig 2, Col 7 Lines 8-14, “…A robot control system 250 in the form of a smartphone or tablet includes a touch screen display 240 that renders a graphical user interface (not specifically referenced in FIG. 2) that includes a rendition of visual data received, for instance, from a front facing camera (not visible in FIG. 2, see 164 in FIG. 1) of control system 250…”, Col 8 Lines 12-17 “Once the top surface of table 270 is extrapolated, object attribute engine 156 may identify, e.g., based on SLAM data previously gathered by one or more robots and/or based on image data received from vision sensor(s) 164/108, a plurality of objects that are co-present within the extrapolated space”);
receive input from a user of the electronic user device indicating a desired location for the at least one moveable object in the environment of the robot (See at least 4 section 404, 406 Col 10 Lines 58-61 “Referring now to FIG. 4, an example method 400 of enabling users to roughly identify a space within an environment inhabited by a plurality of objects that user wishes for robot 100 to act upon is described.”); and
in response to receipt of the user input, transmit control data to the robot, the control data being operable to cause the robot to move the at least one moveable object to the desired location in the environment of the robot (See at least Fig 4 section 410 operate robot to manipulate identified plurality of objects pursuant to one or more user commands).
However, Chao does not explicitly spell out …
receive, from the robot, confirmation data confirming that the at least one moveable object has been moved to the desired location, the confirmation data including updated data representative of the environment; and
in response to receipt of the confirmation data, displaying an updated environment of the robot on the graphical display based on the updated data representative of the environment, wherein the updated environment indicates the location of the at least one moveable object according to the received updated data after being moved by the robot.
Hickman teaches receive, from the robot, confirmation data confirming that the at least one moveable object has been moved to the desired location (See at least Col 31 Lines 58-60 “… The new version of the first map may include those objects that have moved locations since the last update. The server and/or robot 1400 may use the new version of the first map as an update or supplement to the original first map…”, discloses object representation through maps which is construed as confirming that an object has been moved, Col 28 Lines 10-18 “The server may compare the first map to the second map to determine if there are differences between the first map and the second map. If differences exist, the server may identify the differences and store information associated with the differences in the database. The server may use the differences to inventory the objects in the area. The inventory may include the movement of an object from a first location to a second location, the addition or removal of an object, etc.”), the confirmation data including updated data representative of the environment generated by the robot using the one or more sensors (See at least Col 21 Lines 61-67 “In further embodiments, the identification data may be associated with RGB-D data. RGB-D data may be used to build a 3D image or map of the object, an area, etc. In an example, the robot may obtain RGB-D data of the object using a depth camera sensor or any other sensor that may be capable of combining visual and/or 3D shape information. The robot may send the RGB-D data to the server …”, Col 22 Lines 3-9 “In some embodiments, the identification data may be associated with metadata derived from RGB-D data and/or image or range data associated with the object. Image or range data may be associated with a 3D image or map of the object, an area, etc., and may be generated from any number of sensors including stereo camera sensors, depth camera sensors, ranging sensors, sonic sensors, etc…”, Col 31 Lines 58-60 “…The new version of the first map may include those objects that have moved locations since the last update. The server and/or robot 1400 may use the new version of the first map as an update or supplement to the original first map...”)
in response to receipt of the confirmation data displaying an updated environment of the robot on the graphical display based on the updated data representative of the environment, wherein the updated environment indicates the location of the at least one moveable object according to the received updated data after being moved by the robot (See at least Col 31 Lines 42-62 “In an example, the server may update the first map associated with the coffee bar. The update may include identifying the current location of each object in the coffee bar and updating the first map to include the current location of each object in the coffee bar, regardless of whether the current location of an object has changed since the last update. Optionally, the update may include identifying the current location of objects in a subset of the coffee bar and updating the current location of the objects within the subset of the coffee bar, regardless of whether the current location of an object has changed since the last update. An example subset may include the section of the coffee bar with the coffee 1404 , 1406 , 1408 . In another example, the server may update the first map by creating a new version of the first map (or a subset thereof). The new version of the first map may include those objects that have moved locations since the last update. The server and/or robot 1400 may use the new version of the first map as an update or supplement to the original first map. In this way, the server and/or robot 1400 may limit the number of objects that may be updated at one time, for example.”, Col 11 Lines 18-22 “Similarly, the robot 402 could update the map created by the robot 408 with new information about the area (e.g., the hallway now has boxes or other obstacles), or with new information collected from sensors that the robot 408 may not have had”).
Therefore, it would have been obvious to one of the ordinary skill in the art before the
effective filing date of the claimed invention to modify the system of Chao with the teachings of Hickman and include the feature of robot sending confirmation data about moving the objects and in response to receipt of the confirmation data representative of the environment, displaying an updated environment of the robot on the graphical display which indicates the location of the at least one moveable object according to the received updated data after being moved by the robot, thereby representing the updated map information to the user through a display for easy and efficient human robot interaction.
Applicant argues on page 12 of the Applicant’s Remarks that “It is submitted, however, that the
cited art in combination fails to suggest the interconnected features set forth in claim 1. The cited art, in combination, does not suggest the robot, in response to completing the instructed action, sending a confirmation with newly acquired data of the environment to the user device and the user device, in response to the user device receiving the confirmation and updated environment data, updating the display. The cited art, in combination, does not suggest actions triggered by a robot carrying out the command.”. The Examiner respectfully disagrees. Chao teaches “receiving data representative of an environment of the robot (from the robot)” (See at least Fig 1, Col 6 Lines 49-52, Col 3 Lines 11-14, Col 7 Lines 47-52, Col 8 Lines 12-17). Chao further teaches in response to receiving this data, a representation of the environment is displayed (See at least See at least Fig 2, Col 7 Lines 8-14, Col 8 Lines 12-17). In addition, Chao teaches receiving input from a user indicating a desired location for a moveable
object in the environment (See at least Fig 4 section 404, 406, Col 10 Lines 58-61) and transmitting control data to the robot based on the user input (See at least Fig 4 section 410). Hence, Chao already teaches robot interaction with an electronic user device for the robot to perform operations according to the user input. Chao depends on Hickman in the rejection for the teachings of receiving confirmation data (from the robot) when the object has been moved - the confirmation data includes updated data representative of the environment (See at least Col 21 Lines 61-67, Col 22 Lines 3-9, Col 31 Lines 58-60, Col 28 Lines 10-18) and in response to receiving the confirmation data, display an updated environment based on the updated data in the confirmation data (See at least Col 31 Lines 42-62, Col 11 Lines 18-22). Therefore, the combination anticipates the claim limitations.
Regarding Claim 8, Chao teaches a computer program product comprising a set of instructions, which, when executed by a computerized device, cause the computerized device to perform a method of controlling a robot via a network, the method comprising, at an electronic user device (See at least Fig 1 section 140, 160, Fig 2, Col 7 Lines 8-14, “…A robot control system 250 in the form of a smartphone or tablet includes a touch screen display 240 that renders a graphical user interface (not specifically referenced in FIG. 2) that includes a rendition of visual data received, for instance, from a front facing camera (not visible in FIG. 2, see 164 in FIG. 1) of control system 250…”):
receiving, from the robot, data representative of an environment of the robot, the received data indicating a location of at least one moveable object in the environment (See at least Fig 1, Col 6 Lines 49-52 “… In other implementations, one or more sensors 108 that is integral with robot 100 may provide the image feed that is rendered on graphical user interface 160 …”, Col 3 Lines 11-14 “In various implementations, detecting the one or edges may include performing edge detection on visual data sensed by one or more vision sensors…”, Col 7 Lines 47-52 “…In some implementations, records or models of tables or other environmental features stored in index 153 may have been, for instance, previously observed by one or robots while performing simultaneous localization and mapping (“SLAM”) in the environment…”, Col 8 Lines 12-17 “Once the top surface of table 270 is extrapolated, object attribute engine 156 may identify, e.g., based on SLAM data previously gathered by one or more robots and/or based on image data received from vision sensor(s) 164/108, a plurality of objects that are co-present within the extrapolated space”), the received data being generated by the robot using one or more sensors (See at least Fig 1, Col 6 Lines 49-52 “… In other implementations, one or more sensors 108 that is integral with robot 100 may provide the image feed that is rendered on graphical user interface 160 …”, Col 3 Lines 11-14 “In various implementations, detecting the one or edges may include performing edge detection on visual data sensed by one or more vision sensors…”);
in response to receipt of the representative data, displaying a representation of the environment of the robot on a graphical display of the electronic user device (See at least Fig 2, Col 7 Lines 8-14, “…A robot control system 250 in the form of a smartphone or tablet includes a touch screen display 240 that renders a graphical user interface (not specifically referenced in FIG. 2) that includes a rendition of visual data received, for instance, from a front facing camera (not visible in FIG. 2, see 164 in FIG. 1) of control system 250…”, Col 8 Lines 12-17 “Once the top surface of table 270 is extrapolated, object attribute engine 156 may identify, e.g., based on SLAM data previously gathered by one or more robots and/or based on image data received from vision sensor(s) 164/108, a plurality of objects that are co-present within the extrapolated space”);
receiving input from a user of the electronic user device indicating a desired location for the at least one moveable object in the environment of the robot (See at least 4 section 404, 406 Col 10 Lines 58-61 “Referring now to FIG. 4, an example method 400 of enabling users to roughly identify a space within an environment inhabited by a plurality of objects that user wishes for robot 100 to act upon is described.”);
in response to receipt of the user input, transmitting control data to the robot, the control data being operable to cause the robot to move the at least one object to the desired location in the environment of the robot (See at least Fig 4 section 410 operate robot to manipulate identified plurality of objects pursuant to one or more user commands).
However, Chao does not explicitly spell out receiving, from the robot, confirmation data confirming that the at least one moveable object has been moved to the desired location, the confirmation data including updated data representative of the environment; and
in response to receipt of the confirmation data, displaying an updated environment of the robot on the graphical display based on the updated data representative of the environment, wherein the updated environment indicates the location of the at least one moveable object according to the received updated data after being moved by the robot.
Hickman teaches receiving, from the robot, confirmation data confirming that the at least one moveable object has been moved to the desired location (See at least Col 31 Lines 58-60 “… The new version of the first map may include those objects that have moved locations since the last update. The server and/or robot 1400 may use the new version of the first map as an update or supplement to the original first map…”, discloses object representation through maps which is construed as confirming that an object has been moved, Col 28 Lines 10-18 “The server may compare the first map to the second map to determine if there are differences between the first map and the second map. If differences exist, the server may identify the differences and store information associated with the differences in the database. The server may use the differences to inventory the objects in the area. The inventory may include the movement of an object from a first location to a second location, the addition or removal of an object, etc.”), the confirmation data including updated data representative of the environment generated by the robot using the one or more sensors (See at least Col 21 Lines 61-67 “In further embodiments, the identification data may be associated with RGB-D data. RGB-D data may be used to build a 3D image or map of the object, an area, etc. In an example, the robot may obtain RGB-D data of the object using a depth camera sensor or any other sensor that may be capable of combining visual and/or 3D shape information. The robot may send the RGB-D data to the server …”, Col 22 Lines 3-9 “In some embodiments, the identification data may be associated with metadata derived from RGB-D data and/or image or range data associated with the object. Image or range data may be associated with a 3D image or map of the object, an area, etc., and may be generated from any number of sensors including stereo camera sensors, depth camera sensors, ranging sensors, sonic sensors, etc…”, Col 31 Lines 58-60 “…The new version of the first map may include those objects that have moved locations since the last update. The server and/or robot 1400 may use the new version of the first map as an update or supplement to the original first map...”)
in response to receipt of the confirmation data displaying an updated environment of the robot on the graphical display based on the updated data representative of the environment, wherein the updated environment indicates the location of the at least one moveable object according to the received updated data after being moved by the robot (See at least Col 31 Lines 42-62 “In an example, the server may update the first map associated with the coffee bar. The update may include identifying the current location of each object in the coffee bar and updating the first map to include the current location of each object in the coffee bar, regardless of whether the current location of an object has changed since the last update. Optionally, the update may include identifying the current location of objects in a subset of the coffee bar and updating the current location of the objects within the subset of the coffee bar, regardless of whether the current location of an object has changed since the last update. An example subset may include the section of the coffee bar with the coffee 1404 , 1406 , 1408 . In another example, the server may update the first map by creating a new version of the first map (or a subset thereof). The new version of the first map may include those objects that have moved locations since the last update. The server and/or robot 1400 may use the new version of the first map as an update or supplement to the original first map. In this way, the server and/or robot 1400 may limit the number of objects that may be updated at one time, for example.”, Col 11 Lines 18-22 “Similarly, the robot 402 could update the map created by the robot 408 with new information about the area (e.g., the hallway now has boxes or other obstacles), or with new information collected from sensors that the robot 408 may not have had”).
Therefore, it would have been obvious to one of the ordinary skill in the art before the
effective filing date of the claimed invention to modify the system of Chao with the teachings of Hickman and include the feature of robot sending confirmation data about moving the objects and in response to receipt of the confirmation data representative of the environment, displaying an updated environment of the robot on the graphical display which indicates the location of the at least one moveable object according to the received updated data after being moved by the robot, thereby representing the updated map information to the user through a display for easy and efficient human robot interaction.
Applicant argues on page 12 of the Applicant’s Remarks that “It is submitted, however, that the
cited art in combination fails to suggest the interconnected features set forth in claim 1. The cited art, in combination, does not suggest the robot, in response to completing the instructed action, sending a confirmation with newly acquired data of the environment to the user device and the user device, in response to the user device receiving the confirmation and updated environment data, updating the display. The cited art, in combination, does not suggest actions triggered by a robot carrying out the command.”. The Examiner respectfully disagrees. Chao teaches “receiving data representative of an environment of the robot (from the robot)” (See at least Fig 1, Col 6 Lines 49-52, Col 3 Lines 11-14, Col 7 Lines 47-52, Col 8 Lines 12-17). Chao further teaches in response to receiving this data, a representation of the environment is displayed (See at least See at least Fig 2, Col 7 Lines 8-14, Col 8 Lines 12-17). In addition, Chao teaches receiving input from a user indicating a desired location for a moveable
object in the environment (See at least Fig 4 section 404, 406, Col 10 Lines 58-61) and transmitting control data to the robot based on the user input (See at least Fig 4 section 410). Hence, Chao already teaches robot interaction with an electronic user device for the robot to perform operations according to the user input. Chao depends on Hickman in the rejection for the teachings of receiving confirmation data (from the robot) when the object has been moved - the confirmation data includes updated data representative of the environment (See at least Col 21 Lines 61-67, Col 22 Lines 3-9, Col 31 Lines 58-60, Col 28 Lines 10-18) and in response to receiving the confirmation data, display an updated environment based on the updated data in the confirmation data (See at least Col 31 Lines 42-62, Col 11 Lines 18-22). Therefore, the combination anticipates the claim limitations.
Regarding Claim 9, Chao teaches a method of operating a robot, the robot having one or more sensors (See at least Fig 1 item 108), the method comprising:
generating a representation of an environment of the robot by operating the one or more sensors to sense a set of parameters representative of the environment of the robot, wherein the representation comprises a location for at least one moveable object in the environment (See at least Fig 1, Col 6 Lines 49-52 “… In other implementations, one or more sensors 108 that is integral with robot 100 may provide the image feed that is rendered on graphical user interface 160 …”, Col 3 Lines 11-14 “In various implementations, detecting the one or edges may include performing edge detection on visual data sensed by one or more vision sensors…”, Col 7 Lines 47-52 “…In some implementations, records or models of tables or other environmental features stored in index 153 may have been, for instance, previously observed by one or robots while performing simultaneous localization and mapping (“SLAM”) in the environment…”, Col 8 Lines 12-17 “Once the top surface of table 270 is extrapolated, object attribute engine 156 may identify, e.g., based on SLAM data previously gathered by one or more robots and/or based on image data received from vision sensor(s) 164/108, a plurality of objects that are co-present within the extrapolated space”);
transmitting, to an electronic user device, data representative of the environment of the robot
(See at least Fig 2, Col 7 Lines 8-14, “…A robot control system 250 in the form of a smartphone or tablet includes a touch screen display 240 that renders a graphical user interface (not specifically referenced in FIG. 2) that includes a rendition of visual data received, for instance, from a front facing camera (not visible in FIG. 2, see 164 in FIG. 1) of control system 250…”, Col 8 Lines 12-17 “Once the top surface of table 270 is extrapolated, object attribute engine 156 may identify, e.g., based on SLAM data previously gathered by one or more robots and/or based on image data received from vision sensor(s) 164/108, a plurality of objects that are co-present within the extrapolated space”);
receiving control data from the electronic user device, the control data indicating a desired
location for the at least one moveable object in the environment of the robot (See at least 4 section 404, 406 Col 10 Lines 58-61 “Referring now to FIG. 4, an example method 400 of enabling users to roughly identify a space within an environment inhabited by a plurality of objects that user wishes for robot 100 to act upon is described.”); and
in response to receipt of the control data, operating the robot to move the at least one object to the desired location in the environment of the robot (See at least Fig 4 section 410 operate robot to manipulate identified plurality of objects pursuant to one or more user commands); and
However, Chao does not explicitly spell out sending confirmation data confirming that the at least one moveable object has been moved to the desired location, the confirmation data including updated data representative of the environment generated using the one or more sensors, the updated data representative of the environment indicates the location of the at least one moveable object sensed using the one or more sensors after being moved by the robot.
Hickman teaches sending confirmation data confirming that the at least one moveable object has been moved to the desired location, the confirmation data including updated data representative of the environment generated using the one or more sensors, the updated data representative of the environment indicates the location of the at least one moveable object sensed using the one or more sensors after being moved by the robot (See at least Col 31 Lines 58-60 “… The new version of the first map may include those objects that have moved locations since the last update. The server and/or robot 1400 may use the new version of the first map as an update or supplement to the original first map…”, discloses object representation through maps which is construed as confirming that an object has been moved, Col 31 Lines 58-60 “… The new version of the first map may include those objects that have moved locations since the last update. The server and/or robot 1400 may use the new version of the first map as an update or supplement to the original first map…”, discloses object representation through maps which is construed as confirming that an object has been moved, Col 31 Lines 42-62 “In an example, the server may update the first map associated with the coffee bar. The update may include identifying the current location of each object in the coffee bar and updating the first map to include the current location of each object in the coffee bar, regardless of whether the current location of an object has changed since the last update. Optionally, the update may include identifying the current location of objects in a subset of the coffee bar and updating the current location of the objects within the subset of the coffee bar, regardless of whether the current location of an object has changed since the last update. An example subset may include the section of the coffee bar with the coffee 1404 , 1406 , 1408 . In another example, the server may update the first map by creating a new version of the first map (or a subset thereof). The new version of the first map may include those objects that have moved locations since the last update. The server and/or robot 1400 may use the new version of the first map as an update or supplement to the original first map. In this way, the server and/or robot 1400 may limit the number of objects that may be updated at one time, for example.”, Col 11 Lines 18-22 “Similarly, the robot 402 could update the map created by the robot 408 with new information about the area (e.g., the hallway now has boxes or other obstacles), or with new information collected from sensors that the robot 408 may not have had”, Col 21 Lines 61-67 “In further embodiments, the identification data may be associated with RGB-D data. RGB-D data may be used to build a 3D image or map of the object, an area, etc. In an example, the robot may obtain RGB-D data of the object using a depth camera sensor or any other sensor that may be capable of combining visual and/or 3D shape information. The robot may send the RGB-D data to the server …”, Col 22 Lines 3-9 “In some embodiments, the identification data may be associated with metadata derived from RGB-D data and/or image or range data associated with the object. Image or range data may be associated with a 3D image or map of the object, an area, etc., and may be generated from any number of sensors including stereo camera sensors, depth camera sensors, ranging sensors, sonic sensors, etc…”).
Therefore, it would have been obvious to one of the ordinary skill in the art before the
effective filing date of the claimed invention to modify the system of Chao with the teachings of Hickman and include the feature of robot sending confirmation data about moving the objects and in response to receipt of the confirmation data representative of the environment, displaying an updated environment of the robot on the graphical display which indicates the location of the at least one moveable object according to the received updated data after being moved by the robot, thereby representing the updated map information to the user through a display for easy and efficient human robot interaction.
Applicant argues on page 12 of the Applicant’s Remarks that “It is submitted, however, that the
cited art in combination fails to suggest the interconnected features set forth in claim 1. The cited art, in combination, does not suggest the robot, in response to completing the instructed action, sending a confirmation with newly acquired data of the environment to the user device and the user device, in response to the user device receiving the confirmation and updated environment data, updating the display. The cited art, in combination, does not suggest actions triggered by a robot carrying out the command.”. The Examiner respectfully disagrees. Chao teaches “receiving data representative of an environment of the robot (from the robot)” (See at least Fig 1, Col 6 Lines 49-52, Col 3 Lines 11-14, Col 7 Lines 47-52, Col 8 Lines 12-17). Chao further teaches in response to receiving this data, a representation of the environment is displayed (See at least See at least Fig 2, Col 7 Lines 8-14, Col 8 Lines 12-17). In addition, Chao teaches receiving input from a user indicating a desired location for a moveable
object in the environment (See at least Fig 4 section 404, 406, Col 10 Lines 58-61) and transmitting control data to the robot based on the user input (See at least Fig 4 section 410). Hence, Chao already teaches robot interaction with an electronic user device for the robot to perform operations according to the user input. Chao depends on Hickman in the rejection for the teachings of receiving confirmation data (from the robot) when the object has been moved - the confirmation data includes updated data representative of the environment (See at least Col 21 Lines 61-67, Col 22 Lines 3-9, Col 31 Lines 58-60, Col 28 Lines 10-18) and in response to receiving the confirmation data, display an updated environment based on the updated data in the confirmation data (See at least Col 31 Lines 42-62, Col 11 Lines 18-22). Therefore, the combination anticipates the claim limitations.
Regarding Claim 10, modified Chao teaches all the elements of claim 9.
However, Chao does not explicitly spell out the method according to claim 9, wherein the
generating comprises:
generating a list of known objects and associated identifiers;
storing a list of known objects and identifiers for each object in the list;
identifying an unknown object not in the list;
in response to the identification, transmitting to the electronic user device a request to identify the unknown object;
receiving from the electronic user device, data indicating an identifier for the unknown object; and
in response to receipt of the data indicating the identifier, updating the list to associate the identifier with the unknown object.
Hickman teaches the method according to claim 9, wherein the generating comprises:
generating a list of known objects and associated identifiers (See at least Lines Col 33 27-30 “the optimal inventory may be created by a user and may be in the form of a list or map of an optimal or ideal amount and type of objects within an area”);
storing a list of known objects and identifiers for each object in the list (See at least Col 11 Lines 57-67 Col 12 Lines 1-4 “…Each object may further include information in the database in an ordered list, for example. In further examples, the database 412 may include a global unique identifier (GUID) for objects identified in the database 412 (e.g., to enable distinguishing between specific objects), and the GUID may be associated with any characteristics or information describing the object. Thus, a robot may be configured to access the database 412 to receive information generally distinguishing objects (e.g., a baseball vs. a computer), and to receive information that may distinguish between specific objects (e.g., two different computers)”);
identifying an unknown object not in the list (See at least Col 23 Lines 16-19 “if the robot is not able to identify an object or its location, then the robot may prompt the user to enter the information and use entered information to reinforce one or more learning algorithms.”);
in response to the identification, transmitting to the electronic user device a request to identify the unknown object (See at least Col 23 Lines 16-19 “if the robot is not able to identify an object or its location, then the robot may prompt the user to enter the information and use entered information to reinforce one or more learning algorithms.”);
receiving from the electronic user device, data indicating an identifier for the unknown object (See at least Col 23 Lines 16-19 “if the robot is not able to identify an object or its location, then the robot may prompt the user to enter the information and use entered information to reinforce one or more learning algorithms.”); and
in response to receipt of the data indicating the identifier, updating the list to associate the identifier with the unknown object (See at least Col 31 Lines 34-41“The database and/or the first map may be updated at various intervals. The intervals may be related to the frequency at which the robot scans for objects, when there is a change in location data associated with the first map, upon a predetermined schedule, etc. The process of updating the first map may include updating the entire first map, updating a subset of the first map, or creating a new version of the first map or a subset thereof.”).
Therefore, it would have been obvious to one of the ordinary skill in the art before the
effective filing date of the claimed invention to modify the teachings of Chao with the teachings of Hickman and include the feature of generating and storing a list of known objects and associated identifiers, identifying an unknown object not in the list and transmitting to the electronic user device a request to identify the unknown object, receiving data indicating an identifier for the unknown object, updating the list to associate the identifier with the unknown object, thereby providing organized data management.
Regarding Claim 11, modified Chao teaches all the elements of claim 9.
However, Chao does not explicitly spell out the method according to claim 9, comprising
maintaining the generated representation at the robot or an external device, by one or more of:
periodically updating the generated representation
updating the representation in response to operation of the one or more sensors indicating a change in one or more of the parameters in the set.
Hickman teaches the method according to claim 9, comprising maintaining the generated
representation at the robot or an external device, by one or more of:
periodically updating the generated representation (See at least Col 24 Lines 39-53 “In some embodiments, the user, robot, server, etc., may determine a default or preferred map for the server to compare to the first map. The default or preferred map may be a map associated with a specific time period or a past location of the object. For example, the user may identify a second map having past location data associated with the last time the area was cleaned. The first map (e.g., with the current location of the objects) may be compared to the second map (e.g., of the clean area). In another example, the first map may include an inventory of what objects are currently in a store and the second map may include an inventory of what items were in the store one or two weeks prior. The first map (e.g., with the current objects in the store) may be compared to the second map (e.g., of the objects in the store one or two weeks prior.)”, Col 25 Lines 55-61 “The inventory process may be periodic or event driven. Periodic inventorying may occur on an hourly, daily, weekly, monthly, quarterly, yearly, etc., basis. Event driven inventorying may occur when a user requests an inventory, when a searcher instructs the robot to search for an object, etc. In embodiments, the inventory of objects may be stored on the server or sent to the robot or user for storage.”), and
updating the representation in response to operation of the one or more sensors indicating a change in one or more of the parameters in the set (See at least “The database and/or the first map may be updated at various intervals. The intervals may be related to the frequency at which the robot scans for objects, when there is a change in location data associated with the first map, upon a predetermined schedule, etc. The process of updating the first map may include updating the entire first map, updating a subset of the first map, or creating a new version of the first map or a subset thereof.”).
Therefore, it would have been obvious to one of the ordinary skill in the art before the
effective filing date of the claimed invention to modify the teachings of Chao with the teachings of Hickman and include the feature of periodically updating the representation in response to operation of the one or more sensors indicating a change in one or more of the parameters in the set in order to update the map when change is detected in the environment of the robot, thereby keeping the map up to date.
Regarding Claim 12, modified Chao teaches all the elements of claim 10.
However, Chao does not explicitly spell out the method according to claim 10, wherein the list
comprises a home location for at least one object in the list.
Hickman teaches the method according to claim 10, wherein the list comprises a home location
for at least one object in the list (See at least Col 25 Lines 46-51 “the server may obtain or maintain a list of objects or identifiers that should be present in the storeroom, for example. The identifiers associated with the objects that are currently in the storeroom may be compared to the identifiers on the list of objects or identifiers that should be present in the storeroom.”, Col 16 Lines 37-59 “FIGS. 9A-9B are example interfaces illustrating a map of an area, and objects in the map. For example, in FIG. 9A, a home has been outlined (e.g., in a blueprint format) to show different rooms in the home. Items in the home may also have been mapped. FIG. 9B illustrates an example interface illustrating mapping of objects in a room. In some examples, the interfaces in FIGS. 9A-9B, and methods described herein, may enable a user to determine configurations of rooms, and objects in the rooms. As one example, a user may be in a store and identify a few television stands that the user would like to purchase; however, the user would like to see how the television stands look/fit into the room configuration. The user may capture some images of the television stands using a device, cause the device to perform object recognition (e.g., using the method 500 in FIG. 5), access the interface in FIG. 9A to select a room in which to place the television stand, and access the interface in FIG. 9B to insert the new television stand in place of an old television stand (e.g., swap out the old television stand with the new television stand and place the television and peripherals into a desired configuration). The example interfaces in FIGS. 9A-9B may be used to maneuver computer generated objects in a room, for example.”).
Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Chao with the teachings of Hickman and include the feature of a list having a home location for at least one object in the list, thereby enabling object manipulation inside a home environment.
Therefore, it would have been obvious to one of the ordinary skill in the art before the
effective filing date of the claimed invention to modify the teachings of Chao with the teachings of Hickman and include the feature of a list having a home location for at least one object in the list, thereby enabling object manipulation inside a home environment.
Regarding Claim 13, modified Chao teaches all the elements of claim 12.
However, Chao does not explicitly spell out the method according to claim 12, wherein the list
comprises a plurality of objects having the same identifier, and
wherein objects in the plurality have the same home location.
Hickman teaches the method according to claim 12, wherein the list
comprises a plurality of objects having the same identifier (See at least “The counter may be represented in any number of bases. For example, assuming that the same types of objects share the same identifier, the counter may be used to determine how many of the objects having the same identifier are in an area. Thus, the counter may be used to determine how many cans of “soda x” having identifier “abc” are located within the area”), and
wherein objects in the plurality have the same home location (See at least “The counter may be represented in any number of bases. For example, assuming that the same types of objects share the same identifier, the counter may be used to determine how many of the objects having the same identifier are in an area. Thus, the counter may be used to determine how many cans of “soda x” having identifier “abc” are located within the area”).
Therefore, it would have been obvious to one of the ordinary skill in the art before the effective
filing date of the claimed invention to modify the teachings of Chao with the teachings of Hickman and include the feature of list with a plurality of objects having the same identifier and have the same home location, thereby performing plurality of object manipulation inside a home environment.
Regarding Claim 14, modified Chao teaches all the elements of claim 12.
However, Chao does not explicitly spell out the method according to claim 12, wherein the transmitted request further comprises a request to specify a home location for the unknown object,
wherein the received data comprises data specifying the home location for the unknown object, and
wherein updating the list comprises updating the list to include the specified home location for the unknown object.
Hickman teaches the method according to claim 12, wherein the transmitted request further comprises a request to specify a home location for the unknown object (See at least Col 23 Lines 16-19 “if the robot is not able to identify an object or its location, then the robot may prompt the user to enter the information and use entered information to reinforce one or more learning algorithms.”),
wherein the received data comprises data specifying the home location for the unknown object (See at least Col 23 Lines 16-19 “if the robot is not able to identify an object or its location, then the robot may prompt the user to enter the information and use entered information to reinforce one or more learning algorithms.”), and
wherein updating the list comprises updating the list to include the specified home location for the unknown object (See at least Col 23 Lines 16-19 “if the robot is not able to identify an object or its location, then the robot may prompt the user to enter the information and use entered information to reinforce one or more learning algorithms.”).
Therefore, it would have been obvious to one of the ordinary skill in the art before the effective
filing date of the claimed invention to modify the teachings of Chao with the teachings of Hickman and include the feature of transmitted request includes a home location for the unknown object, the received data includes data specifying the home location for the unknown object, and updating the list to include the specified home location for the unknown object, thereby keeping the list up to date.
Regarding Claim 15, modified Chao teaches all the elements of claim 12.
However, Chao does not explicitly spell out the method according to claim 12, wherein operating the robot to move the at least one object to the desired location in the environment of the robot comprises operating the robot to move the at least one object to its home location.
Hickman teaches the method according to claim 12, wherein operating the robot to move the at least one object to the desired location in the environment of the robot comprises operating the robot to move the at least one object to its home location (See at least Fig 9A, 9B “The inventory may include the movement of an object from a first location to a second location, the addition or removal of an object, etc.”, Fig 15).
Therefore, it would have been obvious to one of the ordinary skill in the art before the effective
filing date of the claimed invention to modify the teachings of Chao with the teachings of Hickman and include the feature of moving an object to a desired home location, thereby providing object manipulation in a home environment.
Regarding Claim 16, Chao teaches an apparatus for operating a robot, the robot having one or more sensors (See at least Fig 1 item 108), the apparatus being configured to, at the robot:
generate a representation of an environment of the robot by operating the one or more sensors to sense a set of parameters representative of the environment of the robot, wherein the representation includes a location for at least one moveable object in the environment (See at least Fig 1, Col 6 Lines 49-52 “… In other implementations, one or more sensors 108 that is integral with robot 100 may provide the image feed that is rendered on graphical user interface 160 …”, Col 3 Lines 11-14 “In various implementations, detecting the one or edges may include performing edge detection on visual data sensed by one or more vision sensors…”, Col 7 Lines 47-52 “…In some implementations, records or models of tables or other environmental features stored in index 153 may have been, for instance, previously observed by one or robots while performing simultaneous localization and mapping (“SLAM”) in the environment…”, Col 8 Lines 12-17 “Once the top surface of table 270 is extrapolated, object attribute engine 156 may identify, e.g., based on SLAM data previously gathered by one or more robots and/or based on image data received from vision sensor(s) 164/108, a plurality of objects that are co-present within the extrapolated space”);
transmit, to an electronic user device, data representative of the environment of the robot (See
at least Fig 2, Col 7 Lines 8-14, “…A robot control system 250 in the form of a smartphone or tablet includes a touch screen display 240 that renders a graphical user interface (not specifically referenced in FIG. 2) that includes a rendition of visual data received, for instance, from a front facing camera (not visible in FIG. 2, see 164 in FIG. 1) of control system 250…”, Col 8 Lines 12-17 “Once the top surface of table 270 is extrapolated, object attribute engine 156 may identify, e.g., based on SLAM data previously gathered by one or more robots and/or based on image data received from vision sensor(s) 164/108, a plurality of objects that are co-present within the extrapolated space”);
receive control data from the electronic user device, the control data indicating a desired location for the at least one moveable object in the environment of the robot (See at least 4 section 404, 406 Col 10 Lines 58-61 “Referring now to FIG. 4, an example method 400 of enabling users to roughly identify a space within an environment inhabited by a plurality of objects that user wishes for robot 100 to act upon is described.”); and
in response to receipt of the control data, operate the robot to move the at least one object to the desired location in the environment of the robot (See at least Fig 4 section 410 operate robot to manipulate identified plurality of objects pursuant to one or more user commands); and
However, Chao does not explicitly spell out send confirmation data confirming that the at least one moveable object has been moved to the desired location, the confirmation data including updated data representative of the environment generated using the one or more sensors, the updated data representative of the environment indicates the location of the at least one moveable object sensed using the one or more sensors after being moved by the robot.
Hickman teaches send confirmation data confirming that the at least one moveable object has been moved to the desired location, the confirmation data including updated data representative of the environment generated using the one or more sensors, the updated data representative of the environment indicates the location of the at least one moveable object sensed using the one or more sensors after being moved by the robot (See at least Col 31 Lines 58-60 “… The new version of the first map may include those objects that have moved locations since the last update. The server and/or robot 1400 may use the new version of the first map as an update or supplement to the original first map…”, discloses object representation through maps which is construed as confirming that an object has been moved, Col 31 Lines 42-62 “In an example, the server may update the first map associated with the coffee bar. The update may include identifying the current location of each object in the coffee bar and updating the first map to include the current location of each object in the coffee bar, regardless of whether the current location of an object has changed since the last update. Optionally, the update may include identifying the current location of objects in a subset of the coffee bar and updating the current location of the objects within the subset of the coffee bar, regardless of whether the current location of an object has changed since the last update. An example subset may include the section of the coffee bar with the coffee 1404 , 1406 , 1408 . In another example, the server may update the first map by creating a new version of the first map (or a subset thereof). The new version of the first map may include those objects that have moved locations since the last update. The server and/or robot 1400 may use the new version of the first map as an update or supplement to the original first map. In this way, the server and/or robot 1400 may limit the number of objects that may be updated at one time, for example.”, Col 11 Lines 18-22 “Similarly, the robot 402 could update the map created by the robot 408 with new information about the area (e.g., the hallway now has boxes or other obstacles), or with new information collected from sensors that the robot 408 may not have had”, Col 21 Lines 61-67 “In further embodiments, the identification data may be associated with RGB-D data. RGB-D data may be used to build a 3D image or map of the object, an area, etc. In an example, the robot may obtain RGB-D data of the object using a depth camera sensor or any other sensor that may be capable of combining visual and/or 3D shape information. The robot may send the RGB-D data to the server …”, Col 22 Lines 3-9 “In some embodiments, the identification data may be associated with metadata derived from RGB-D data and/or image or range data associated with the object. Image or range data may be associated with a 3D image or map of the object, an area, etc., and may be generated from any number of sensors including stereo camera sensors, depth camera sensors, ranging sensors, sonic sensors, etc…”).
Therefore, it would have been obvious to one of the ordinary skill in the art before the
effective filing date of the claimed invention to modify the system of Chao with the teachings of Hickman and include the feature of robot sending confirmation data about moving the objects and in response to receipt of the confirmation data representative of the environment, displaying an updated environment of the robot on the graphical display which indicates the location of the at least one moveable object according to the received updated data after being moved by the robot, thereby representing the updated map information to the user through a display for easy and efficient human robot interaction.
Applicant argues on page 12 of the Applicant’s Remarks that “It is submitted, however, that the
cited art in combination fails to suggest the interconnected features set forth in claim 1. The cited art, in combination, does not suggest the robot, in response to completing the instructed action, sending a confirmation with newly acquired data of the environment to the user device and the user device, in response to the user device receiving the confirmation and updated environment data, updating the display. The cited art, in combination, does not suggest actions triggered by a robot carrying out the command.”. The Examiner respectfully disagrees. Chao teaches “receiving data representative of an environment of the robot (from the robot)” (See at least Fig 1, Col 6 Lines 49-52, Col 3 Lines 11-14, Col 7 Lines 47-52, Col 8 Lines 12-17). Chao further teaches in response to receiving this data, a representation of the environment is displayed (See at least See at least Fig 2, Col 7 Lines 8-14, Col 8 Lines 12-17). In addition, Chao teaches receiving input from a user indicating a desired location for a moveable
object in the environment (See at least Fig 4 section 404, 406, Col 10 Lines 58-61) and transmitting control data to the robot based on the user input (See at least Fig 4 section 410). Hence, Chao already teaches robot interaction with an electronic user device for the robot to perform operations according to the user input. Chao depends on Hickman in the rejection for the teachings of receiving confirmation data (from the robot) when the object has been moved - the confirmation data includes updated data representative of the environment (See at least Col 21 Lines 61-67, Col 22 Lines 3-9, Col 31 Lines 58-60, Col 28 Lines 10-18) and in response to receiving the confirmation data, display an updated environment based on the updated data in the confirmation data (See at least Col 31 Lines 42-62, Col 11 Lines 18-22). Therefore, the combination anticipates the claim limitations.
Regarding Claim 17, Chao teaches a computer program product comprising a set of instructions, which, when executed by a computerized device, cause the computerized device to perform a method of operating a robot, the robot having one or more sensors (See at least Fig 1 section 140, 160, Fig 2, Col 7 Lines 8-14, “…A robot control system 250 in the form of a smartphone or tablet includes a touch screen display 240 that renders a graphical user interface (not specifically referenced in FIG. 2) that includes a rendition of visual data received, for instance, from a front facing camera (not visible in FIG. 2, see 164 in FIG. 1) of control system 250…”, Fig 1 item 108), the method comprising, at the robot:
generating a representation of an environment of the robot by operating the one or more sensors to sense a set of parameters representative of the environment of the robot, wherein the representation includes a location for at least one moveable object in the environment (See at least Fig 1, Col 6 Lines 49-52 “… In other implementations, one or more sensors 108 that is integral with robot 100 may provide the image feed that is rendered on graphical user interface 160 …”, Col 3 Lines 11-14 “In various implementations, detecting the one or edges may include performing edge detection on visual data sensed by one or more vision sensors…”, Col 7 Lines 47-52 “…In some implementations, records or models of tables or other environmental features stored in index 153 may have been, for instance, previously observed by one or robots while performing simultaneous localization and mapping (“SLAM”) in the environment…”, Col 8 Lines 12-17 “Once the top surface of table 270 is extrapolated, object attribute engine 156 may identify, e.g., based on SLAM data previously gathered by one or more robots and/or based on image data received from vision sensor(s) 164/108, a plurality of objects that are co-present within the extrapolated space”);
transmitting, to an electronic user device, data representative of the environment of the robot
(See at least Fig 2, Col 7 Lines 8-14, “…A robot control system 250 in the form of a smartphone or tablet includes a touch screen display 240 that renders a graphical user interface (not specifically referenced in FIG. 2) that includes a rendition of visual data received, for instance, from a front facing camera (not visible in FIG. 2, see 164 in FIG. 1) of control system 250…”, Col 8 Lines 12-17 “Once the top surface of table 270 is extrapolated, object attribute engine 156 may identify, e.g., based on SLAM data previously gathered by one or more robots and/or based on image data received from vision sensor(s) 164/108, a plurality of objects that are co-present within the extrapolated space”);
receiving control data from the electronic user device, the control data indicating a desired
location for the at least one moveable object in the environment of the robot (See at least 4 section 404, 406 Col 10 Lines 58-61 “Referring now to FIG. 4, an example method 400 of enabling users to roughly identify a space within an environment inhabited by a plurality of objects that user wishes for robot 100 to act upon is described.”); and
in response to receipt of the control data, operating the robot to move the at least one object to the desired location in the environment of the robot (See at least Fig 4 section 410 operate robot to manipulate identified plurality of objects pursuant to one or more user commands); and
However, Chao does not explicitly spell out sending confirmation data confirming that the at least one moveable object has been moved to the desired location, the confirmation data including updated data representative of the environment generated using the one or more sensors, the updated data representative of the environment indicates the location of the at least one moveable object sensed using the one or more sensors after being moved by the robot.
Hickman teaches sending confirmation data confirming that the at least one moveable object has been moved to the desired location, the confirmation data including updated data representative of the environment generated using the one or more sensors, the updated data representative of the environment indicates the location of the at least one moveable object sensed using the one or more sensors after being moved by the robot (See at least Col 31 Lines 58-60 “… The new version of the first map may include those objects that have moved locations since the last update. The server and/or robot 1400 may use the new version of the first map as an update or supplement to the original first map…”, discloses object representation through maps which is construed as confirming that an object has been moved, Col 31 Lines 42-62 “In an example, the server may update the first map associated with the coffee bar. The update may include identifying the current location of each object in the coffee bar and updating the first map to include the current location of each object in the coffee bar, regardless of whether the current location of an object has changed since the last update. Optionally, the update may include identifying the current location of objects in a subset of the coffee bar and updating the current location of the objects within the subset of the coffee bar, regardless of whether the current location of an object has changed since the last update. An example subset may include the section of the coffee bar with the coffee 1404 , 1406 , 1408 . In another example, the server may update the first map by creating a new version of the first map (or a subset thereof). The new version of the first map may include those objects that have moved locations since the last update. The server and/or robot 1400 may use the new version of the first map as an update or supplement to the original first map. In this way, the server and/or robot 1400 may limit the number of objects that may be updated at one time, for example.”, Col 11 Lines 18-22 “Similarly, the robot 402 could update the map created by the robot 408 with new information about the area (e.g., the hallway now has boxes or other obstacles), or with new information collected from sensors that the robot 408 may not have had”, Col 21 Lines 61-67 “In further embodiments, the identification data may be associated with RGB-D data. RGB-D data may be used to build a 3D image or map of the object, an area, etc. In an example, the robot may obtain RGB-D data of the object using a depth camera sensor or any other sensor that may be capable of combining visual and/or 3D shape information. The robot may send the RGB-D data to the server …”, Col 22 Lines 3-9 “In some embodiments, the identification data may be associated with metadata derived from RGB-D data and/or image or range data associated with the object. Image or range data may be associated with a 3D image or map of the object, an area, etc., and may be generated from any number of sensors including stereo camera sensors, depth camera sensors, ranging sensors, sonic sensors, etc…”).
Therefore, it would have been obvious to one of the ordinary skill in the art before the
effective filing date of the claimed invention to modify the system of Chao with the teachings of Hickman and include the feature of robot sending confirmation data about moving the objects and in response to receipt of the confirmation data representative of the environment, displaying an updated environment of the robot on the graphical display which indicates the location of the at least one moveable object according to the received updated data after being moved by the robot, thereby representing the updated map information to the user through a display for easy and efficient human robot interaction.
Applicant argues on page 12 of the Applicant’s Remarks that “It is submitted, however, that the
cited art in combination fails to suggest the interconnected features set forth in claim 1. The cited art, in combination, does not suggest the robot, in response to completing the instructed action, sending a confirmation with newly acquired data of the environment to the user device and the user device, in response to the user device receiving the confirmation and updated environment data, updating the display. The cited art, in combination, does not suggest actions triggered by a robot carrying out the command.”. The Examiner respectfully disagrees. Chao teaches “receiving data representative of an environment of the robot (from the robot)” (See at least Fig 1, Col 6 Lines 49-52, Col 3 Lines 11-14, Col 7 Lines 47-52, Col 8 Lines 12-17). Chao further teaches in response to receiving this data, a representation of the environment is displayed (See at least See at least Fig 2, Col 7 Lines 8-14, Col 8 Lines 12-17). In addition, Chao teaches receiving input from a user indicating a desired location for a moveable
object in the environment (See at least Fig 4 section 404, 406, Col 10 Lines 58-61) and transmitting control data to the robot based on the user input (See at least Fig 4 section 410). Hence, Chao already teaches robot interaction with an electronic user device for the robot to perform operations according to the user input. Chao depends on Hickman in the rejection for the teachings of receiving confirmation data (from the robot) when the object has been moved - the confirmation data includes updated data representative of the environment (See at least Col 21 Lines 61-67, Col 22 Lines 3-9, Col 31 Lines 58-60, Col 28 Lines 10-18) and in response to receiving the confirmation data, display an updated environment based on the updated data in the confirmation data (See at least Col 31 Lines 42-62, Col 11 Lines 18-22). Therefore, the combination anticipates the claim limitations.
Regarding Claim 18, Chao teaches a robot having one or more sensors (See at least Fig 1 item 108), the robot being configured to:
generate a representation of an environment of the robot by operating the one or more sensors to sense a set of parameters representative of the environment of the robot, wherein the representation includes a location for at least one moveable object in the environment (See at least Fig 1, Col 6 Lines 49-52 “… In other implementations, one or more sensors 108 that is integral with robot 100 may provide the image feed that is rendered on graphical user interface 160 …”, Col 3 Lines 11-14 “In various implementations, detecting the one or edges may include performing edge detection on visual data sensed by one or more vision sensors…”, Col 7 Lines 47-52 “…In some implementations, records or models of tables or other environmental features stored in index 153 may have been, for instance, previously observed by one or robots while performing simultaneous localization and mapping (“SLAM”) in the environment…”, Col 8 Lines 12-17 “Once the top surface of table 270 is extrapolated, object attribute engine 156 may identify, e.g., based on SLAM data previously gathered by one or more robots and/or based on image data received from vision sensor(s) 164/108, a plurality of objects that are co-present within the extrapolated space”);
transmit, to an electronic user device, data representative of the environment of the robot(See at least Fig 2, Col 7 Lines 8-14, “…A robot control system 250 in the form of a smartphone or tablet includes a touch screen display 240 that renders a graphical user interface (not specifically referenced in FIG. 2) that includes a rendition of visual data received, for instance, from a front facing camera (not visible in FIG. 2, see 164 in FIG. 1) of control system 250…”, Col 8 Lines 12-17 “Once the top surface of table 270 is extrapolated, object attribute engine 156 may identify, e.g., based on SLAM data previously gathered by one or more robots and/or based on image data received from vision sensor(s) 164/108, a plurality of objects that are co-present within the extrapolated space”);
receive control data from the electronic user device, the control data indicating a desired
location for the at least one moveable object in the environment of the robot (See at least 4 section 404, 406 Col 10 Lines 58-61 “Referring now to FIG. 4, an example method 400 of enabling users to roughly identify a space within an environment inhabited by a plurality of objects that user wishes for robot 100 to act upon is described.”); and
in response to receipt of the control data, move the at least one object to the desired location in the environment of the robot (See at least Fig 4 section 410 operate robot to manipulate identified plurality of objects pursuant to one or more user commands).
However, Chao does not explicitly spell out send confirmation data confirming that the at least one moveable object has been moved to the desired location, the confirmation data including updated data representative of the environment generated using the one or more sensors, the updated data representative of the environment indicates the location of the at least one moveable object sensed using the one or more sensors after being moved by the robot.
Hickman teaches send confirmation data confirming that the at least one moveable object has been moved to the desired location, the confirmation data including updated data representative of the environment generated using the one or more sensors, the updated data representative of the environment indicates the location of the at least one moveable object sensed using the one or more sensors after being moved by the robot (See at least Col 31 Lines 58-60 “… The new version of the first map may include those objects that have moved locations since the last update. The server and/or robot 1400 may use the new version of the first map as an update or supplement to the original first map…”, discloses object representation through maps which is construed as confirming that an object has been moved, Col 31 Lines 42-62 “In an example, the server may update the first map associated with the coffee bar. The update may include identifying the current location of each object in the coffee bar and updating the first map to include the current location of each object in the coffee bar, regardless of whether the current location of an object has changed since the last update. Optionally, the update may include identifying the current location of objects in a subset of the coffee bar and updating the current location of the objects within the subset of the coffee bar, regardless of whether the current location of an object has changed since the last update. An example subset may include the section of the coffee bar with the coffee 1404 , 1406 , 1408 . In another example, the server may update the first map by creating a new version of the first map (or a subset thereof). The new version of the first map may include those objects that have moved locations since the last update. The server and/or robot 1400 may use the new version of the first map as an update or supplement to the original first map. In this way, the server and/or robot 1400 may limit the number of objects that may be updated at one time, for example.”, Col 11 Lines 18-22 “Similarly, the robot 402 could update the map created by the robot 408 with new information about the area (e.g., the hallway now has boxes or other obstacles), or with new information collected from sensors that the robot 408 may not have had”, Col 21 Lines 61-67 “In further embodiments, the identification data may be associated with RGB-D data. RGB-D data may be used to build a 3D image or map of the object, an area, etc. In an example, the robot may obtain RGB-D data of the object using a depth camera sensor or any other sensor that may be capable of combining visual and/or 3D shape information. The robot may send the RGB-D data to the server …”, Col 22 Lines 3-9 “In some embodiments, the identification data may be associated with metadata derived from RGB-D data and/or image or range data associated with the object. Image or range data may be associated with a 3D image or map of the object, an area, etc., and may be generated from any number of sensors including stereo camera sensors, depth camera sensors, ranging sensors, sonic sensors, etc…”).
Therefore, it would have been obvious to one of the ordinary skill in the art before the
effective filing date of the claimed invention to modify the system of Chao with the teachings of Hickman and include the feature of robot sending confirmation data about moving the objects and in response to receipt of the confirmation data representative of the environment, displaying an updated environment of the robot on the graphical display which indicates the location of the at least one moveable object according to the received updated data after being moved by the robot, thereby representing the updated map information to the user through a display for easy and efficient human robot interaction.
Applicant argues on page 12 of the Applicant’s Remarks that “It is submitted, however, that the
cited art in combination fails to suggest the interconnected features set forth in claim 1. The cited art, in combination, does not suggest the robot, in response to completing the instructed action, sending a confirmation with newly acquired data of the environment to the user device and the user device, in response to the user device receiving the confirmation and updated environment data, updating the display. The cited art, in combination, does not suggest actions triggered by a robot carrying out the command.”. The Examiner respectfully disagrees. Chao teaches “receiving data representative of an environment of the robot (from the robot)” (See at least Fig 1, Col 6 Lines 49-52, Col 3 Lines 11-14, Col 7 Lines 47-52, Col 8 Lines 12-17). Chao further teaches in response to receiving this data, a representation of the environment is displayed (See at least See at least Fig 2, Col 7 Lines 8-14, Col 8 Lines 12-17). In addition, Chao teaches receiving input from a user indicating a desired location for a moveable
object in the environment (See at least Fig 4 section 404, 406, Col 10 Lines 58-61) and transmitting control data to the robot based on the user input (See at least Fig 4 section 410). Hence, Chao already teaches robot interaction with an electronic user device for the robot to perform operations according to the user input. Chao depends on Hickman in the rejection for the teachings of receiving confirmation data (from the robot) when the object has been moved - the confirmation data includes updated data representative of the environment (See at least Col 21 Lines 61-67, Col 22 Lines 3-9, Col 31 Lines 58-60, Col 28 Lines 10-18) and in response to receiving the confirmation data, display an updated environment based on the updated data in the confirmation data (See at least Col 31 Lines 42-62, Col 11 Lines 18-22). Therefore, the combination anticipates the claim limitations.
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Chao (US 10058997 B1) in view of Hickman et al. (US 10391633 B1) (Hereinafter Hickman), and further in view of Matsukawa et al. (JP2004268148A) (Hereinafter Matsukawa).
Regarding Claim 3, modified Chao teaches all the elements of claim 2.
However, Chao does not explicitly spell out the method according to claim 2, wherein the user
input comprises a drag and drop action from a current location of the at least one moveable object to the desired location.
Matsukawa teaches the method according to claim 2, wherein the user input comprises a drag
and drop action from a current location of the at least one moveable object to the desired location (See at least Page 11 Lines 47-49 “Correspondingly, an article moving operation is designated by an intuitive operation of dragging and dropping an article icon arranged in the virtual space to a desired destination in the virtual space.”).
Therefore, it would have been obvious to one of the ordinary skill in the art before the
effective filing date of the claimed invention to modify the system of Chao with the teachings of Matsukawa and include the drag and drop action feature in the display of the electronic user device, thereby providing user friendly action to perform movement of objects to desired location.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Rajkumar et al. (US 2018/0018518 A1) teaches determining information such as object type and/or pose by the object recognition client
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHAHEDA HOQUE whose telephone number is (571)270-5310. The examiner can normally be reached Monday-Friday 8:00 am- 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ramon Mercado can be reached on 571-270-5744. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHAHEDA HOQUE/Examiner, Art Unit 3658
/Ramon A. Mercado/Supervisory Patent Examiner, Art Unit 3658
Prosecution Insights