DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of the Claims
This action is in response to the Applicant’s filing on February 13, 2026. Claims 1-30 are pending and examined below.
Response to Arguments
The previous rejections of claims 1-30 under 35 U.S.C. 103 are withdrawn in consideration of amended independent claims 1, 19, and 25. However, new rejections of claims 1-30 under 35 U.S.C. 103 are set forth below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 5, 8-9, 11-14, 16-17, 19-20, and 24 are rejected under 35 U.S.C. 103 as unpatentable over U.S. Patent Application Publication No. US 2024/0077882 by Degirmenci et al. (herein after “Degirmenci”), in view of U.S. Patent Application Publication No. US 2024/0118103 by Shin et al. (herein after “Shin”) and U.S. Patent No. US 12,619,240 by Aggarwal et al. (herein after “Aggarwal”).
Note: Text written in bold typeface is claim language from the instant application. Text written in normal typeface are comments made by the Examiner and/or passages from the prior art reference(s).
Regarding claim 1, Degirmenci discloses a computer-implemented method comprising: obtaining, by data processing hardware, from a first data source, a site model associated with a site (Degirmenci ¶ [0091]: Block 402 includes the controller 118 learning a site map within the environment. To learn a map, controller 118 may be navigated through its environment to collect data from sensor units 114. The data may indicate the presence/location of various objects in the environment which may be localized onto the site map);
obtaining, by the data processing hardware, from at least one sensor of a robot (Degirmenci ¶ [0060]: sensor units 114 may comprise systems and/or methods that may detect characteristics within and/or around robot 102), sensor data captured from the site (Degirmenci ¶ [0089]: a local route or local scanning route includes a route for a robot 102 to navigate, wherein the robot 102 scans for features in acquired sensor data during execution of the local route; Degirmenci ¶ [0130]: the controller 118 may collect data useful for identifying features, such as images, videos, LiDAR/point cloud data, thermal data, and/or any other data collected by sensor units 114),
generating, by the data processing hardware, a virtual representation of the sensor data (Degirmenci ¶ [0095]: During navigation of each of these local routes, controller 118 may produce a corresponding local route map. Local route maps may include various objects sensed and localized by the sensor units 114 of the robot 102);
identifying, by the data processing hardware, a first association between at least a portion of the sensor data and at least a portion of the site model (Degirmenci ¶ [0093]: The locations of the various objects on the site map may be defined with respect to an origin. In some embodiments, the origin may comprise the start of the route. In some embodiments, the origin may be an arbitrary point within the environment. The robot 102 may recognize its initial position (e.g., upon turning on) based on detecting one or more recognizable features, such as landmarks (e.g., objects sensed in the past), computer-readable codes (e.g., quick-response codes, barcodes, etc.), markers, beacons, and the like. Such markers may indicate the origin point, or may be at a known distance from the origin with respect to which the robot 102 may localize itself)
transforming, by the data processing hardware, the virtual representation of the sensor data based on the first association to generate transformed data (Degirmenci ¶ [0094]: the controller 118 learning at least one local scanning route within the environment, wherein each of the at least one local scanning routes corresponds to a respective local route map; Degirmenci ¶ [0097]: Block 406 includes the controller 118 aligning the at least one local route map to the site map. To align the maps, the controller 118 may determine a transform (i.e., translation and/or rotation) between the origin of the site map and the origin(s) of each of the at least one local route map such that both the site map and the at least one local map align. Controller 118 may utilize iterative closest point (“ICP”) algorithms, or similar nearest neighboring alignment algorithms, to determine the transform, as shown in FIGS. 7A-B below. Once the maps align with minimal error, the controller 118 may define an origin of the local route map with respect to the origin of the site map);
instructing, by the data processing hardware, display of a user interface, wherein the user interface reflects the transformed data overlaid on the site model (Degirmenci ¶ [0127]: FIG. 8A may illustrate a display on a user interface unit 112 of the robot 102 or user interface coupled to the robot 102 (e.g., a personal computer or device 208 coupled to a server 202). According to at least one non-limiting exemplary embodiment, the aligned site map 510 and local scanning route maps 604 may be communicated to a server 202, wherein a device 208 coupled to the server 202 may be configured to receive the annotations. In some embodiments, the one or more local scanning route maps 604 may be overlaid on top of the site map 510 to display all of the objects 502 in the environment, even if some objects 502 are only partially sensed on the site map 510); and
instructing, by the data processing hardware, the robot to traverse at least a portion of the site based on an input received via the user interface (Degirmenci ¶ [0141]: FIG. 9A(i-ii) illustrate edits performed to a route 902 in accordance with block 410 of method 400, according to an exemplary embodiment. Two types of route edits may be performed: (a) manual edits based on user inputs to user interface units 112, and (b) automatic edits. First, FIG. 9A(i) shows an exemplary scenario where a robot 102, in learning a local scanning route 604, encounters an object 902. The object 902 may be a temporary object which does not persist within the environment. For example, the object 902 may be a human, a shopping cart, a puddle of water, and so forth. The human operator, during training/demonstration of the local route 604, may navigate around the temporary object 902 and produce a curve in route 604 around the object 902. The curve route 604 was only executed to avoid collision and is not an indented behavior of the robot 102 to be learned. If the robot 102 is to scan for features of objects 502 (e.g., objects on a shelf) the curve may be undesirable as the robot 102 may capture sensor data of the object 502 at poor angles and from farther distances. Accordingly, the human operator may be prompted to perform edits to the route 604 after completing the training to enhance the feature scanning. The human operator may utilize the user interface 112 of the robot 102 or a separate user interface, such as a user interface on a device 208 coupled to a server 202 or directly coupled to the robot 102).
It is noted Degirmenci discloses a site map based on prior sensor readings of a robot and a local map based on current sensor readings of a robot and identifying associated features between the site map and local map but fails to explicitly disclose obtaining, by data processing hardware, from a first data source, a site model associated with a site;
obtaining, by the data processing hardware, from at least one sensor of a robot, sensor data captured from the site, wherein the first data source is distinct and separate from the at least one sensor, and wherein the site model and the sensor data correspond to different types of data; and
identifying, by the data processing hardware, a first association between at least a portion of the sensor data and at least a portion of the site model based on determining that the sensor data and the site model are associated with the same site.
However, Shin, in the same field of endeavor, teaches obtaining, by data processing hardware, from a first data source (Shin ¶ [0100]: the server 2000 obtains a pre-generated first spatial map including first map features representing spatial information about a space. The server 2000 may obtain a pre-generated first spatial map from a map database stored on the server 2000), a site model associated with a site (Shin ¶ [0070]: a first spatial map 210 may be map data including various pieces of information related to a space. The first spatial map 210 may include, for example, a two-dimensional (2D)/3D spatial map, a 2D/3D floorplan, and a blueprint, but is not limited thereto, and the first spatial map 210 may be various forms of map data including 2D/3D spatial information);
obtaining, by the data processing hardware, from at least one sensor of a robot, sensor data captured from the site (Shin ¶ [0084]: the second spatial map 220 may be a spatial map generated based on sensor data obtained by measuring the space. The sensor data may be obtained using one or more sensors capable of measuring the space. The one or more sensors may be, for example, but are not limited to, a red, green, blue (RGB) sensor, an RGB-Depth (RGB-D) sensor, a time-of-flight (ToF) sensor, a light detection and ranging (lidar) sensor, a radio detection and ranging (radar) sensor, etc. In addition, the one or more sensors may be mounted on an indoor robot, a robot cleaner, a user's smartphone, etc), wherein the first data source is distinct and separate from the at least one sensor (Shin ¶ [0086]: The server 2000 may obtain the second spatial map 220 generated by an electronic device including one or more sensors), and wherein the site model and the sensor data correspond to different types of data (Shin ¶ [0130]: based on a spatial dimension of the first spatial map being different from that of the second spatial map, the server 2000 projects a higher dimensional spatial map among the first spatial map and the second spatial map to correspond to a spatial dimension of the lower dimensional spatial map).
Further, Aggarwal, in the same field of endeavor, teaches obtaining, by data processing hardware, from a first data source, a site model associated with a site (Aggarwal col. 26 lines 35-38: At 1010, the process 1000 may include determining a layout of the environment 102. For example, the management system 110 may access the layout data 122 for determining the layout of the environment 102);
obtaining, by the data processing hardware, from at least one sensor of a robot, sensor data captured from the site (Aggarwal col. 26 lines 9-10: At 1006, the process 1000 may include receiving LiDAR data from the reality capture robot 100; Aggarwal col. 26 lines 19-21: At 1008, the process 1000 may include determining, based at least in part on the LiDAR data, a 3D point cloud of the environment);
identifying, by the data processing hardware, a first association between at least a portion of the sensor data and at least a portion of the site model (Aggarwal col. 26 lines 46-51: At 1012, the process 1000 may include aligning the layout and the 3D point cloud. For example, the management system 110 may align the layout and the 3D point cloud. In some instances, origins or reference points in the layout and the 3D point cloud such that like objects are compared within one another across the layout and the 3D point cloud. For example, certain walls, pillars, or a corner of the environment 102 may be aligned) based on determining that the sensor data and the site model are associated with the same site (Aggarwal col. 26 lines 56-62: In instances where the reality capture robot 100 scans a portion of the environment 102, a corresponding portion of the layout may be determined for aligning the 3D point cloud thereto. For example, in instances where reality capture robot 100 scans a portion of the environment 102, the management system 110 may determine corresponding portion of the layout for aligning with the 3D point cloud).
Therefore, given the teachings as whole, it would have been prima facie obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the feature scanning for localization and navigation display of Degirmenci to include the first spatial map from a first data source separate from a sensor and the second spatial map based on sensor data corresponding to a different data type than the first spatial map of Shin and the alignment of sensor data with layout data based on determining the sensor data and layout data are associated with the same site of Aggarwal with a reasonable expectation of success. A person of ordinary skill in the art would be motivated to make these modifications in order to reflect personal spatial information such as an actual layout of a user’s space and real-world objects when generating and updating spatial maps (Shin ¶ [0005]) and to compare respective locations of objects within an environment allowing for inaccuracies in layout data to be corrected while generating a static map (Aggarwal col. 15 lines 1-10).
Regarding claim 5, Degirmenci discloses wherein the sensor data comprises point cloud data (Degirmenci ¶ [0130]: the controller 118 may collect data useful for identifying features, such as images, videos, LiDAR/point cloud data, thermal data, and/or any other data collected by sensor units 114), wherein the first association is between a portion of the point cloud data and one or more corresponding features of the site model (Degirmenci ¶ [0093]: The locations of the various objects on the site map may be defined with respect to an origin. In some embodiments, the origin may comprise the start of the route. In some embodiments, the origin may be an arbitrary point within the environment. The robot 102 may recognize its initial position (e.g., upon turning on) based on detecting one or more recognizable features, such as landmarks (e.g., objects sensed in the past), computer-readable codes (e.g., quick-response codes, barcodes, etc.), markers, beacons, and the like. Such markers may indicate the origin point, or may be at a known distance from the origin with respect to which the robot 102 may localize itself).
Regarding claim 8, Degirmenci discloses wherein the transformed data comprises a transformed virtual representation of at least one of: the sensor data; or route data (Degirmenci ¶ [0094]: the controller 118 learning at least one local scanning route within the environment, wherein each of the at least one local scanning routes corresponds to a respective local route map; Degirmenci ¶ [0097]: Block 406 includes the controller 118 aligning the at least one local route map to the site map. To align the maps, the controller 118 may determine a transform (i.e., translation and/or rotation) between the origin of the site map and the origin(s) of each of the at least one local route map such that both the site map and the at least one local map align. Controller 118 may utilize iterative closest point (“ICP”) algorithms, or similar nearest neighboring alignment algorithms, to determine the transform, as shown in FIGS. 7A-B below. Once the maps align with minimal error, the controller 118 may define an origin of the local route map with respect to the origin of the site map).
Regarding claim 9, Degirmenci discloses wherein transforming the virtual representation of the sensor data comprises at least one of:
moving one or more portions of the virtual representation of the sensor data relative to the site model; scaling one or more portions of the virtual representation of the sensor data relative to the site model;
turning one or more portions of the virtual representation of the sensor data relative to the site model;
rotating one or more portions of the virtual representation of the sensor data relative to the site model;
translating one or more portions of the virtual representation of the sensor data relative to the site model; or
warping one or more portions of the virtual representation of the sensor data relative to the site model (Degirmenci ¶ [0094]: the controller 118 learning at least one local scanning route within the environment, wherein each of the at least one local scanning routes corresponds to a respective local route map; Degirmenci ¶ [0097]: Block 406 includes the controller 118 aligning the at least one local route map to the site map. To align the maps, the controller 118 may determine a transform (i.e., translation and/or rotation) between the origin of the site map and the origin(s) of each of the at least one local route map such that both the site map and the at least one local map align. Controller 118 may utilize iterative closest point (“ICP”) algorithms, or similar nearest neighboring alignment algorithms, to determine the transform, as shown in FIGS. 7A-B below. Once the maps align with minimal error, the controller 118 may define an origin of the local route map with respect to the origin of the site map).
Regarding claim 11, Degirmenci discloses wherein the sensor data comprises at least one of: odometry data; point cloud data; fiducial data; orientation data; position data; height data; a serial number; or time data (Degirmenci ¶ [0130]: the controller 118 may collect data useful for identifying features, such as images, videos, LiDAR/point cloud data, thermal data, and/or any other data collected by sensor units 114).
Regarding claim 12, Degirmenci discloses wherein the at least one sensor comprises a stereo camera, a scanning light-detection and ranging sensor, or a scanning laser-detection and ranging sensor (Degirmenci ¶ [0107]: feature identification is performed using range data or point cloud data from the robot 102 (e.g., from LiDAR sensors and/or depth cameras)).
Regarding claim 13, Degirmenci discloses further comprising instructing display of a second user interface on a user computing device, wherein the second user interface reflects the virtual representation of the sensor data overlaid on the site model, wherein identifying the first association comprises obtaining, from the user computing device, data identifying the first association (Degirmenci ¶ [0141]: FIG. 9A(i-ii) illustrate edits performed to a route 902 in accordance with block 410 of method 400, according to an exemplary embodiment. Two types of route edits may be performed: (a) manual edits based on user inputs to user interface units 112, and (b) automatic edits. First, FIG. 9A(i) shows an exemplary scenario where a robot 102, in learning a local scanning route 604, encounters an object 902. The object 902 may be a temporary object which does not persist within the environment. For example, the object 902 may be a human, a shopping cart, a puddle of water, and so forth. The human operator, during training/demonstration of the local route 604, may navigate around the temporary object 902 and produce a curve in route 604 around the object 902. The curve route 604 was only executed to avoid collision and is not an indented behavior of the robot 102 to be learned. If the robot 102 is to scan for features of objects 502 (e.g., objects on a shelf) the curve may be undesirable as the robot 102 may capture sensor data of the object 502 at poor angles and from farther distances. Accordingly, the human operator may be prompted to perform edits to the route 604 after completing the training to enhance the feature scanning. The human operator may utilize the user interface 112 of the robot 102 or a separate user interface, such as a user interface on a device 208 coupled to a server 202 or directly coupled to the robot 102). The examiner interprets a virtual representation of sensor data to include a local scanning route. An operator may be prompted to edit the route in order to establish an association between the virtual representation and a site map.
Regarding claim 14, Degirmenci discloses further comprising identifying a second association between the virtual representation of the sensor data and the site model, wherein transforming the virtual representation of the sensor data is further based on the second association (Degirmenci ¶ [0093]: The locations of the various objects on the site map may be defined with respect to an origin. In some embodiments, the origin may comprise the start of the route. In some embodiments, the origin may be an arbitrary point within the environment. The robot 102 may recognize its initial position (e.g., upon turning on) based on detecting one or more recognizable features, such as landmarks (e.g., objects sensed in the past), computer-readable codes (e.g., quick-response codes, barcodes, etc.), markers, beacons, and the like. Such markers may indicate the origin point, or may be at a known distance from the origin with respect to which the robot 102 may localize itself). The examiner interprets the one or more recognizable features such as landmarks to be associations between a virtual representation and a site model.
Regarding claim 16, Degirmenci fails to particularly disclose wherein the site model comprises a virtual representation of one or more of a blueprint, a map, a computer-aided design (“CAD”) model, a floor plan, a facilities representation, a geo-spatial map, or a graph.
However, Shin, in the same field of endeavor, teaches wherein the site model comprises a virtual representation of one or more of a blueprint, a map, a computer-aided design (“CAD”) model, a floor plan, a facilities representation, a geo-spatial map, or a graph (Shin ¶ [0070]: a first spatial map 210 may be map data including various pieces of information related to a space. The first spatial map 210 may include, for example, a two-dimensional (2D)/3D spatial map, a 2D/3D floorplan, and a blueprint, but is not limited thereto, and the first spatial map 210 may be various forms of map data including 2D/3D spatial information).
Therefore, given the teachings as whole, it would have been prima facie obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the feature scanning for localization and navigation display of Degirmenci modified by the first spatial map from a first data source separate from a sensor and the second spatial map based on sensor data corresponding to a different data type than the first spatial map of Shin and the alignment of sensor data with layout data based on determining the sensor data and layout data are associated with the same site of Aggarwal to further include the 2D/3D spatial map, blueprint, or floor plan of Shin with a reasonable expectation of success. A person of ordinary skill in the art would be motivated to make this modification in order to reflect personal spatial information such as an actual layout of a user’s space and real-world objects when generating and updating spatial maps (Shin ¶ [0005]).
Regarding claim 17, Degirmenci discloses wherein identifying the first association comprises:
determining that the site model corresponds to a particular pixel characteristic (Degirmenci ¶ [0037]: a feature may comprise one or more numeric values (e.g., floating point, decimal, a tensor of values, etc.) characterizing an input from a sensor unit of a robot, described in FIG. 1A below, including, but not limited to, detection of an object, parameters of the object (e.g., size, shape, color, orientation, edges, etc.), the object itself, color values of pixels of an image, depth values of pixels of a depth image, brightness of an image, the image as a whole, changes of features over time (e.g., velocity, trajectory, etc. of an object), sounds, spectral energy of a spectrum bandwidth, motor feedback (i.e., encoder values), sensor values (e.g., gyroscope, accelerometer, GPS, magnetometer, etc. readings), a binary categorical variable, an enumerated type, a character/string, or any other characteristic of a sensory input); and
automatically identifying the first association based on determining that the site model corresponds to the particular pixel characteristic (Degirmenci ¶ [0093]: The locations of the various objects on the site map may be defined with respect to an origin. In some embodiments, the origin may comprise the start of the route. In some embodiments, the origin may be an arbitrary point within the environment. The robot 102 may recognize its initial position (e.g., upon turning on) based on detecting one or more recognizable features, such as landmarks (e.g., objects sensed in the past), computer-readable codes (e.g., quick-response codes, barcodes, etc.), markers, beacons, and the like. Such markers may indicate the origin point, or may be at a known distance from the origin with respect to which the robot 102 may localize itself; Degirmenci ¶ [0125]: FIG. 7B shows the local scanning map 604-B having been rotated by the controller 118).
Regarding claim 19, Degirmenci discloses a system comprising:
data processing hardware (Degirmenci ¶ [0049]: As illustrated in FIG. 1A, robot 102 may include controller 118, memory 120); and
memory in communication with the data processing hardware, the memory storing instructions that when executed on the data processing hardware cause the data processing hardware to (Degirmenci ¶ [0052]: Memory 120 may provide instructions and data to controller 118. For example, memory 120 may be a non-transitory, computer-readable storage apparatus and/or medium having a plurality of instructions stored thereon, the instructions being executable by a processing apparatus (e.g., controller 118) to operate robot 102):
obtain, from a first data source, a site model associated with a site (Degirmenci ¶ [0091]: Block 402 includes the controller 118 learning a site map within the environment. To learn a map, controller 118 may be navigated through its environment to collect data from sensor units 114. The data may indicate the presence/location of various objects in the environment which may be localized onto the site map);
obtain, from at least one sensor of a robot (Degirmenci ¶ [0060]: sensor units 114 may comprise systems and/or methods that may detect characteristics within and/or around robot 102), sensor data captured from the site (Degirmenci ¶ [0089]: a local route or local scanning route includes a route for a robot 102 to navigate, wherein the robot 102 scans for features in acquired sensor data during execution of the local route; Degirmenci ¶ [0130]: the controller 118 may collect data useful for identifying features, such as images, videos, LiDAR/point cloud data, thermal data, and/or any other data collected by sensor units 114)
generate a virtual representation of the sensor data (Degirmenci ¶ [0095]: During navigation of each of these local routes, controller 118 may produce a corresponding local route map. Local route maps may include various objects sensed and localized by the sensor units 114 of the robot 102);
identify a first association between at least a portion of the sensor data and at least a portion of the site model (Degirmenci ¶ [0093]: The locations of the various objects on the site map may be defined with respect to an origin. In some embodiments, the origin may comprise the start of the route. In some embodiments, the origin may be an arbitrary point within the environment. The robot 102 may recognize its initial position (e.g., upon turning on) based on detecting one or more recognizable features, such as landmarks (e.g., objects sensed in the past), computer-readable codes (e.g., quick-response codes, barcodes, etc.), markers, beacons, and the like. Such markers may indicate the origin point, or may be at a known distance from the origin with respect to which the robot 102 may localize itself)
transform the virtual representation of the sensor data based on the first association to generate transformed data (Degirmenci ¶ [0094]: the controller 118 learning at least one local scanning route within the environment, wherein each of the at least one local scanning routes corresponds to a respective local route map; Degirmenci ¶ [0097]: Block 406 includes the controller 118 aligning the at least one local route map to the site map. To align the maps, the controller 118 may determine a transform (i.e., translation and/or rotation) between the origin of the site map and the origin(s) of each of the at least one local route map such that both the site map and the at least one local map align. Controller 118 may utilize iterative closest point (“ICP”) algorithms, or similar nearest neighboring alignment algorithms, to determine the transform, as shown in FIGS. 7A-B below. Once the maps align with minimal error, the controller 118 may define an origin of the local route map with respect to the origin of the site map);
instruct display of a user interface, wherein the user interface reflects the transformed data overlaid on the site model (Degirmenci ¶ [0127]: FIG. 8A may illustrate a display on a user interface unit 112 of the robot 102 or user interface coupled to the robot 102 (e.g., a personal computer or device 208 coupled to a server 202). According to at least one non-limiting exemplary embodiment, the aligned site map 510 and local scanning route maps 604 may be communicated to a server 202, wherein a device 208 coupled to the server 202 may be configured to receive the annotations. In some embodiments, the one or more local scanning route maps 604 may be overlaid on top of the site map 510 to display all of the objects 502 in the environment, even if some objects 502 are only partially sensed on the site map 510); and
instruct the robot to traverse at least a portion of the site based on an input received via the user interface (Degirmenci ¶ [0141]: FIG. 9A(i-ii) illustrate edits performed to a route 902 in accordance with block 410 of method 400, according to an exemplary embodiment. Two types of route edits may be performed: (a) manual edits based on user inputs to user interface units 112, and (b) automatic edits. First, FIG. 9A(i) shows an exemplary scenario where a robot 102, in learning a local scanning route 604, encounters an object 902. The object 902 may be a temporary object which does not persist within the environment. For example, the object 902 may be a human, a shopping cart, a puddle of water, and so forth. The human operator, during training/demonstration of the local route 604, may navigate around the temporary object 902 and produce a curve in route 604 around the object 902. The curve route 604 was only executed to avoid collision and is not an indented behavior of the robot 102 to be learned. If the robot 102 is to scan for features of objects 502 (e.g., objects on a shelf) the curve may be undesirable as the robot 102 may capture sensor data of the object 502 at poor angles and from farther distances. Accordingly, the human operator may be prompted to perform edits to the route 604 after completing the training to enhance the feature scanning. The human operator may utilize the user interface 112 of the robot 102 or a separate user interface, such as a user interface on a device 208 coupled to a server 202 or directly coupled to the robot 102).
It is noted Degirmenci discloses a site map based on prior sensor readings of a robot and a local map based on current sensor readings of a robot and identifying associated features between the site map and local map but fails to explicitly disclose
obtaining, from a first data source, a site model associated with a site;
obtaining, from at least one sensor of a robot, sensor data captured from the site, wherein the first data source is distinct and separate from the at least one sensor, and wherein the site model and the sensor data correspond to different types of data; and
identify a first association between at least a portion of the sensor data and at least a portion of the site model based on determining that the sensor data and the site model are associated with the same site.
However, Shin, in the same field of endeavor, teaches obtain, from a first data source (Shin ¶ [0100]: the server 2000 obtains a pre-generated first spatial map including first map features representing spatial information about a space. The server 2000 may obtain a pre-generated first spatial map from a map database stored on the server 2000), a site model associated with a site (Shin ¶ [0070]: a first spatial map 210 may be map data including various pieces of information related to a space. The first spatial map 210 may include, for example, a two-dimensional (2D)/3D spatial map, a 2D/3D floorplan, and a blueprint, but is not limited thereto, and the first spatial map 210 may be various forms of map data including 2D/3D spatial information);
obtain, from at least one sensor of a robot, sensor data captured from the site (Shin ¶ [0084]: the second spatial map 220 may be a spatial map generated based on sensor data obtained by measuring the space. The sensor data may be obtained using one or more sensors capable of measuring the space. The one or more sensors may be, for example, but are not limited to, a red, green, blue (RGB) sensor, an RGB-Depth (RGB-D) sensor, a time-of-flight (ToF) sensor, a light detection and ranging (lidar) sensor, a radio detection and ranging (radar) sensor, etc. In addition, the one or more sensors may be mounted on an indoor robot, a robot cleaner, a user's smartphone, etc), wherein the first data source is distinct and separate from the at least one sensor (Shin ¶ [0086]: The server 2000 may obtain the second spatial map 220 generated by an electronic device including one or more sensors), and wherein the site model and the sensor data correspond to different types of data (Shin ¶ [0130]: based on a spatial dimension of the first spatial map being different from that of the second spatial map, the server 2000 projects a higher dimensional spatial map among the first spatial map and the second spatial map to correspond to a spatial dimension of the lower dimensional spatial map).
Further, Aggarwal, in the same field of endeavor, teaches obtaining, from a first data source, a site model associated with a site (Aggarwal col. 26 lines 35-38: At 1010, the process 1000 may include determining a layout of the environment 102. For example, the management system 110 may access the layout data 122 for determining the layout of the environment 102);
obtaining, from at least one sensor of a robot, sensor data captured from the site (Aggarwal col. 26 lines 9-10: At 1006, the process 1000 may include receiving LiDAR data from the reality capture robot 100; Aggarwal col. 26 lines 19-21: At 1008, the process 1000 may include determining, based at least in part on the LiDAR data, a 3D point cloud of the environment);
identify a first association between at least a portion of the sensor data and at least a portion of the site model (Aggarwal col. 26 lines 46-51: At 1012, the process 1000 may include aligning the layout and the 3D point cloud. For example, the management system 110 may align the layout and the 3D point cloud. In some instances, origins or reference points in the layout and the 3D point cloud such that like objects are compared within one another across the layout and the 3D point cloud. For example, certain walls, pillars, or a corner of the environment 102 may be aligned) based on determining that the sensor data and the site model are associated with the same site (Aggarwal col. 26 lines 56-62: In instances where the reality capture robot 100 scans a portion of the environment 102, a corresponding portion of the layout may be determined for aligning the 3D point cloud thereto. For example, in instances where reality capture robot 100 scans a portion of the environment 102, the management system 110 may determine corresponding portion of the layout for aligning with the 3D point cloud).
Therefore, given the teachings as whole, it would have been prima facie obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the feature scanning for localization and navigation display of Degirmenci to include the first spatial map from a first data source separate from a sensor and the second spatial map based on sensor data corresponding to a different data type than the first spatial map of Shin and the alignment of sensor data with layout data based on determining the sensor data and layout data are associated with the same site of Aggarwal with a reasonable expectation of success. A person of ordinary skill in the art would be motivated to make these modifications in order to reflect personal spatial information such as an actual layout of a user’s space and real-world objects when generating and updating spatial maps (Shin ¶ [0005]) and to compare respective locations of objects within an environment allowing for inaccuracies in layout data to be corrected while generating a static map (Aggarwal col. 15 lines 1-10).
Regarding claim 20, Degirmenci discloses wherein the input indicates a selection of at least a portion of the transformed data (Degirmenci ¶ [0141]: FIG. 9A(i-ii) illustrate edits performed to a route 902 in accordance with block 410 of method 400, according to an exemplary embodiment. Two types of route edits may be performed: (a) manual edits based on user inputs to user interface units 112, and (b) automatic edits. First, FIG. 9A(i) shows an exemplary scenario where a robot 102, in learning a local scanning route 604, encounters an object 902. The object 902 may be a temporary object which does not persist within the environment. For example, the object 902 may be a human, a shopping cart, a puddle of water, and so forth. The human operator, during training/demonstration of the local route 604, may navigate around the temporary object 902 and produce a curve in route 604 around the object 902. The curve route 604 was only executed to avoid collision and is not an indented behavior of the robot 102 to be learned. If the robot 102 is to scan for features of objects 502 (e.g., objects on a shelf) the curve may be undesirable as the robot 102 may capture sensor data of the object 502 at poor angles and from farther distances. Accordingly, the human operator may be prompted to perform edits to the route 604 after completing the training to enhance the feature scanning. The human operator may utilize the user interface 112 of the robot 102 or a separate user interface, such as a user interface on a device 208 coupled to a server 202 or directly coupled to the robot 102) overlaid on the site model (Degirmenci ¶ [0127]: FIG. 8A may illustrate a display on a user interface unit 112 of the robot 102 or user interface coupled to the robot 102 (e.g., a personal computer or device 208 coupled to a server 202). According to at least one non-limiting exemplary embodiment, the aligned site map 510 and local scanning route maps 604 may be communicated to a server 202, wherein a device 208 coupled to the server 202 may be configured to receive the annotations. In some embodiments, the one or more local scanning route maps 604 may be overlaid on top of the site map 510 to display all of the objects 502 in the environment, even if some objects 502 are only partially sensed on the site map 510). The examiner interprets a virtual representation of sensor data to include a local scanning route which is overlaid on a site model. An operator may be prompted to edit the route in order to establish an association between the virtual representation and a site map.
Regarding claim 24, Degirmenci discloses wherein execution of the instructions on the data processing hardware further causes the data processing hardware to: identify a set of associations between the virtual representation of the sensor data and the site model (Degirmenci ¶ [0093]: The locations of the various objects on the site map may be defined with respect to an origin. In some embodiments, the origin may comprise the start of the route. In some embodiments, the origin may be an arbitrary point within the environment. The robot 102 may recognize its initial position (e.g., upon turning on) based on detecting one or more recognizable features, such as landmarks (e.g., objects sensed in the past), computer-readable codes (e.g., quick-response codes, barcodes, etc.), markers, beacons, and the like. Such markers may indicate the origin point, or may be at a known distance from the origin with respect to which the robot 102 may localize itself), wherein transforming the virtual representation of the sensor data is further based on the set of associations (Degirmenci ¶ [0094]: the controller 118 learning at least one local scanning route within the environment, wherein each of the at least one local scanning routes corresponds to a respective local route map; Degirmenci ¶ [0097]: Block 406 includes the controller 118 aligning the at least one local route map to the site map. To align the maps, the controller 118 may determine a transform (i.e., translation and/or rotation) between the origin of the site map and the origin(s) of each of the at least one local route map such that both the site map and the at least one local map align. Controller 118 may utilize iterative closest point (“ICP”) algorithms, or similar nearest neighboring alignment algorithms, to determine the transform, as shown in FIGS. 7A-B below. Once the maps align with minimal error, the controller 118 may define an origin of the local route map with respect to the origin of the site map).
Claims 2 and 23 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication No. US 2024/0077882 by Degirmenci et al. (herein after “Degirmenci”), in view of U.S. Patent Application Publication No. US 2024/0118103 by Shin et al. (herein after “Shin”) and U.S. Patent No. US 12,619,240 by Aggarwal et al. (herein after “Aggarwal”), further in view of O. Mendez, S. Hadfield, N. Pugeault and R. Bowden, "SeDAR - Semantic Detection and Ranging: Humans can Localise without LiDAR, can Robots?," 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, QLD, Australia, 2018, pp. 6053-6060 (herein after “Mendez”) and U.S. Patent Application Publication No. US 2019/0033459 by Tisdale et al. (herein after “Tisdale”).
Note: Text written in bold typeface is claim language from the instant application. Text written in normal typeface are comments made by the Examiner and/or passages from the prior art reference(s).
Regarding claim 2, Degirmenci discloses wherein identifying the first association comprises:
generating an estimation of the first association based on the sensor data and the point cloud (Degirmenci ¶ [0093]: The locations of the various objects on the site map may be defined with respect to an origin. In some embodiments, the origin may comprise the start of the route. In some embodiments, the origin may be an arbitrary point within the environment. The robot 102 may recognize its initial position (e.g., upon turning on) based on detecting one or more recognizable features, such as landmarks (e.g., objects sensed in the past), computer-readable codes (e.g., quick-response codes, barcodes, etc.), markers, beacons, and the like. Such markers may indicate the origin point, or may be at a known distance from the origin with respect to which the robot 102 may localize itself);
instructing display of a second user interface on a user computing device, wherein the second user interface reflects the refined estimation of the first association, wherein identifying the first association comprises obtaining, from the user computing device, data corresponding to an acceptance, a rejection, or a modification of the refined estimation of the first association (Degirmenci ¶ [0141]: FIG. 9A(i-ii) illustrate edits performed to a route 902 in accordance with block 410 of method 400, according to an exemplary embodiment. Two types of route edits may be performed: (a) manual edits based on user inputs to user interface units 112, and (b) automatic edits. First, FIG. 9A(i) shows an exemplary scenario where a robot 102, in learning a local scanning route 604, encounters an object 902. The object 902 may be a temporary object which does not persist within the environment. For example, the object 902 may be a human, a shopping cart, a puddle of water, and so forth. The human operator, during training/demonstration of the local route 604, may navigate around the temporary object 902 and produce a curve in route 604 around the object 902. The curve route 604 was only executed to avoid collision and is not an indented behavior of the robot 102 to be learned. If the robot 102 is to scan for features of objects 502 (e.g., objects on a shelf) the curve may be undesirable as the robot 102 may capture sensor data of the object 502 at poor angles and from farther distances. Accordingly, the human operator may be prompted to perform edits to the route 604 after completing the training to enhance the feature scanning. The human operator may utilize the user interface 112 of the robot 102 or a separate user interface, such as a user interface on a device 208 coupled to a server 202 or directly coupled to the robot 102). The examiner interprets a virtual representation of sensor data to include a local scanning route. An operator may be prompted to modify the route in order to establish a new association between the virtual representation and a site map.
It is noted that Degirmenci fails to explicitly disclose wherein identifying the first association comprises:
converting the site model into a point cloud;
flattening the sensor data relative to a plane of the site model to generate flattened sensor data;
refining the estimation of the first association based on the flattened sensor data to generate a refined estimation of the first association.
However, Mendez, in the same field of endeavor, teaches wherein identifying the first association comprises: converting the site model into a point cloud (Mendez pg. 6055 ¶ 3: we augment the localisation with semantic labels extracted from an existing floorplan. In our experiments we limit these labels to walls, doors and windows (see figure 2), which are easy to automatically extract from a floorplan, and are also salient for human localisation. In order to make a labelled floorplan readable by the robot, it must first be converted into an occupancy grid. An occupancy grid is a 2D representation of the world, in which each cell in the grid has an occupancy probability, determined by it’s normalized greyscale value; Fig. 2);
flattening the sensor data relative to a plane of the site model to generate flattened sensor data (Mendez pg. 6055 ¶ 5: the centre scanline is assumed to be parallel with the ground plane and is therefore used to collapse the 3D information of the RGB-D image into the 2D floorplan);
estimating the first association based on the flattened sensor data to generate a refined estimation of the first association (Mendez pg. 6053 ¶ 7: We then collapse all semantic information into 2D in order to reduce the assumptions about the environment. We then use these labels, image geometry and (optionally) depth along with a semantically labelled floorplan to create a state-of-the-art sensing and localisation framework).
It is further noted that Mendez fails to explicitly teach refining the estimation of the first association based on the flattened sensor data to generate a refined estimation of the first association.
However, Tisdale, in the same field of endeavor, teaches wherein identifying the first association comprises:
flattening the sensor data relative to a plane of the site model to generate flattened sensor data (Tisdale ¶ [0068]: the controller 150 may receive point cloud data from the LIDAR 112 and transform the point cloud data to provide a top-down image. In an example embodiment, the top-down image may include a flattened two-dimensional image of the environment around the vehicle 110; Tisdale ¶ [0074]: The controller 150 may transform the camera image data to provide a top-down image. The top-down image includes a flattened two-dimensional image of the environment around the vehicle. In some embodiments, the transformation of the camera image data providing the top-down image may include a geometric perspective transform algorithm or another type of image processing);
refining the estimation of the first association based on the flattened sensor data to generate a refined estimation of the first association (Tisdale ¶ [0084]: While embodiments described herein may relate to heading (yaw), it is understood that other refinements to estimated pose are possible by comparing various reference data and sensor data. For example, an estimated pitch or roll of a vehicle may be refined; Tisdale ¶ [0106]: Block 604 includes transforming the point cloud data to provide a top-down image. In such a scenario, the top-down image includes a flattened two-dimensional image of the environment around the vehicle. For example, the transforming could include a scale-invariant feature transform, image flattening, image perspective rotation, and/or a 3D-to-2D image transform. Other ways to represent three-dimensional object information as a two-dimensional, top-down image are contemplated herein; Tisdale ¶ [0069]-[0070]: The comparison of the top-down image to the reference image may be performed in various ways. For example, the comparison may be performed with one or more of: normalized cross-correlation, grayscale matching, gradient matching, histogram matching, edge detection (e.g., Canny edge detection), scale-invariant feature transform (SIFT), and/or speeded-up robust features (SURF). Additionally or alternatively, in some embodiments, the comparison may include various image registration or morphing algorithms).
Therefore, given the teachings as whole, it would have been prima facie obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the feature scanning for localization and navigation display of Degirmenci modified by the first spatial map from a first data source separate from a sensor and the second spatial map based on sensor data corresponding to a different data type than the first spatial map of Shin and the alignment of sensor data with layout data based on determining the sensor data and layout data are associated with the same site of Aggarwal to further include the floorplan point cloud conversion and sensor data flattening of Mendez and the sensor data flattening to refine an association between sensor data and a reference of Tisdale with a reasonable expectation of success. A person of ordinary skill in the art would be motivated to make these modifications in order to facilitate global localization relying on semantic labels present in a floorplan and extracted from RGB images (Mendez Abstract) and to determine a pose of a vehicle based on various combinations of map data and sensor data received from light detection and ranging (LIDAR) devices and/or camera devices (Tisdale ¶ [0004]).
Regarding claim 23, Degirmenci discloses wherein to identify the first association, execution of the instructions on the data processing hardware further causes the data processing hardware to:
generate an estimation of the first association based on the sensor data and the point cloud (Degirmenci ¶ [0093]: The locations of the various objects on the site map may be defined with respect to an origin. In some embodiments, the origin may comprise the start of the route. In some embodiments, the origin may be an arbitrary point within the environment. The robot 102 may recognize its initial position (e.g., upon turning on) based on detecting one or more recognizable features, such as landmarks (e.g., objects sensed in the past), computer-readable codes (e.g., quick-response codes, barcodes, etc.), markers, beacons, and the like. Such markers may indicate the origin point, or may be at a known distance from the origin with respect to which the robot 102 may localize itself);
.
It is noted that Degirmenci fails to explicitly disclose wherein to identify the first association, execution of the instructions on the data processing hardware further causes the data processing hardware to:
convert the site model into a point cloud;
flatten the sensor data relative to a plane of the site model to generate flattened sensor data; and
refine the estimation of the first association based on the flattened sensor data to generate a refined estimation of the first association.
However, Mendez, in the same field of endeavor, teaches wherein to identify the first association, execution of the instructions on the data processing hardware further causes the data processing hardware to: convert the site model into a point cloud (Mendez pg. 6055 ¶ 3: we augment the localisation with semantic labels extracted from an existing floorplan. In our experiments we limit these labels to walls, doors and windows (see figure 2), which are easy to automatically extract from a floorplan, and are also salient for human localisation. In order to make a labelled floorplan readable by the robot, it must first be converted into an occupancy grid. An occupancy grid is a 2D representation of the world, in which each cell in the grid has an occupancy probability, determined by it’s normalized greyscale value; Fig. 2);
flatten the sensor data relative to a plane of the site model to generate flattened sensor data (Mendez pg. 6055 ¶ 5: the centre scanline is assumed to be parallel with the ground plane and is therefore used to collapse the 3D information of the RGB-D image into the 2D floorplan); and
estimating the first association based on the flattened sensor data to generate a refined estimation of the first association (Mendez pg. 6053 ¶ 7: We then collapse all semantic information into 2D in order to reduce the assumptions about the environment. We then use these labels, image geometry and (optionally) depth along with a semantically labelled floorplan to create a state-of-the-art sensing and localisation framework).
It is further noted that Mendez fails to explicitly teach refine the estimation of the first association based on the flattened sensor data to generate a refined estimation of the first association.
However, Tisdale, in the same field of endeavor, teaches flatten the sensor data relative to a plane of the site model to generate flattened sensor data (Tisdale ¶ [0068]: the controller 150 may receive point cloud data from the LIDAR 112 and transform the point cloud data to provide a top-down image. In an example embodiment, the top-down image may include a flattened two-dimensional image of the environment around the vehicle 110; Tisdale ¶ [0074]: The controller 150 may transform the camera image data to provide a top-down image. The top-down image includes a flattened two-dimensional image of the environment around the vehicle. In some embodiments, the transformation of the camera image data providing the top-down image may include a geometric perspective transform algorithm or another type of image processing); and
refine the estimation of the first association based on the flattened sensor data to generate a refined estimation of the first association (Tisdale ¶ [0084]: While embodiments described herein may relate to heading (yaw), it is understood that other refinements to estimated pose are possible by comparing various reference data and sensor data. For example, an estimated pitch or roll of a vehicle may be refined; Tisdale ¶ [0106]: Block 604 includes transforming the point cloud data to provide a top-down image. In such a scenario, the top-down image includes a flattened two-dimensional image of the environment around the vehicle. For example, the transforming could include a scale-invariant feature transform, image flattening, image perspective rotation, and/or a 3D-to-2D image transform. Other ways to represent three-dimensional object information as a two-dimensional, top-down image are contemplated herein; Tisdale ¶ [0069]-[0070]: The comparison of the top-down image to the reference image may be performed in various ways. For example, the comparison may be performed with one or more of: normalized cross-correlation, grayscale matching, gradient matching, histogram matching, edge detection (e.g., Canny edge detection), scale-invariant feature transform (SIFT), and/or speeded-up robust features (SURF). Additionally or alternatively, in some embodiments, the comparison may include various image registration or morphing algorithms).
Therefore, given the teachings as whole, it would have been prima facie obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the feature scanning for localization and navigation display of Degirmenci modified by the first spatial map from a first data source separate from a sensor and the second spatial map based on sensor data corresponding to a different data type than the first spatial map of Shin and the alignment of sensor data with layout data based on determining the sensor data and layout data are associated with the same site of Aggarwal to further include the floorplan point cloud conversion and sensor data flattening of Mendez and the sensor data flattening to refine an association between sensor data and a reference of Tisdale with a reasonable expectation of success. A person of ordinary skill in the art would be motivated to make these modifications in order to facilitate global localization relying on semantic labels present in a floorplan and extracted from RGB images (Mendez Abstract) and to determine a pose of a vehicle based on various combinations of map data and sensor data received from light detection and ranging (LIDAR) devices and/or camera devices (Tisdale ¶ [0004]).
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication No. US 2024/0077882 by Degirmenci et al. (herein after “Degirmenci”), in view of U.S. Patent Application Publication No. US 2024/0118103 by Shin et al. (herein after “Shin”) and U.S. Patent No. US 12,619,240 by Aggarwal et al. (herein after “Aggarwal”), further in view of U.S. Patent Application Publication No. US 2022/0382287 by Van De Velde et al. (herein after “Van De Velde”).
Note: Text written in bold typeface is claim language from the instant application. Text written in normal typeface are comments made by the Examiner and/or passages from the prior art reference(s).
Regarding claim 3, Degirmenci discloses wherein the transformed data overlaid on the site model includes a route for the robot (Degirmenci ¶ [0127]: the one or more local scanning route maps 604 may be overlaid on top of the site map 510 to display all of the objects 502 in the environment, even if some objects 502 are only partially sensed on the site map 510; Figs. 8A-B, 9A-B, and 10) .
It is noted Degirmenci fails to explicitly disclose wherein the transformed data overlaid on the site model includes a route for the robot represented by a set of route waypoints and at least one route edge.
However, Van De Velde, in the same field of endeavor, teaches wherein the transformed data overlaid on the site model includes a route for the robot represented by a set of route waypoints and at least one route edge (Van De Velde ¶ [0113]: An exemplary user interface is shown with various user interactive portions. As illustrated a VOP position 201 is shown adjacent to a staging area 208. The VOP (not illustrated in FIG. 2) will travel along the VAP 202 to reach a destination 207. The VAP 202 circumvents one or more obstacles 209 that prevent the VOP from traveling in a direct line path 210 to reach the destination 207).
Therefore, given the teachings as whole, it would have been prima facie obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the feature scanning for localization and navigation display of Degirmenci modified by the first spatial map from a first data source separate from a sensor and the second spatial map based on sensor data corresponding to a different data type than the first spatial map of Shin and the alignment of sensor data with layout data based on determining the sensor data and layout data are associated with the same site of Aggarwal to further include the route with waypoints of Van De Velde with a reasonable expectation of success. A person of ordinary skill in the art would be motivated to make this modification in order to control a sequence of origination positions and destination positions such that the operation and travel of an autonomous vehicle or a person through a sequence of positions and actions is safe and efficient (Van De Velde ¶ [0002]).
Claims 4 and 6 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication No. US 2024/0077882 by Degirmenci et al. (herein after “Degirmenci”), in view of U.S. Patent Application Publication No. US 2024/0118103 by Shin et al. (herein after “Shin”) and U.S. Patent No. US 12,619,240 by Aggarwal et al. (herein after “Aggarwal”), further in view of U.S. Patent Application Publication No. US 2019/0355173 by Gao (hereafter “Gao”).
Note: Text written in bold typeface is claim language from the instant application. Text written in normal typeface are comments made by the Examiner and/or passages from the prior art reference(s).
Regarding claim 4, Degirmenci discloses wherein obtaining the sensor data comprises obtaining, by the data processing hardware, a first set of sensor data obtained by a first robot and a second set of sensor data obtained by a second robot (Degirmenci ¶ [0076]: One skilled in the art may appreciate that any determination or calculation described herein may comprise one or more processors of the server 202, edge devices 208, and/or robots 102 of networks 210 performing the determination or calculation by executing computer-readable instructions. The instructions may be executed by a processor of the server 202 and/or may be communicated to robot networks 210 and/or edge devices 208 for execution on their respective controllers/processors in part or in entirety (e.g., a robot 102 may calculate a coverage map using measurements 308 collected by itself or another robot 102)).
It is noted Degirmenci fails to explicitly disclose wherein obtaining the sensor data comprises merging, by the data processing hardware, a first set of sensor data obtained by a first robot with a second set of sensor data obtained by a second robot.
However, Gao, in the same field of endeavor, teaches wherein obtaining the sensor data comprises merging, by the data processing hardware, a first set of sensor data obtained by a first robot with a second set of sensor data obtained by a second robot (Gao ¶ [0036]: In 204, the system may merge multiple 3D point cloud maps. As described, the system may receive information from multiple devices at the same time, and accordingly, multiple maps may be generated simultaneously (or in parallel, concurrently, etc.) from information received from various locations. For instance, a first robot may traverse a first room while a second robot traverses a second room at the same time. Accordingly, a merged (or global) map may be generated by the system when it is determined that the paths of the first and second robots have crossed. Accordingly, upon such a determination, the system may merge portions (or segments, fragments, etc.) of the map automatically).
Therefore, given the teachings as whole, it would have been prima facie obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the feature scanning for localization and navigation display of Degirmenci modified by the first spatial map from a first data source separate from a sensor and the second spatial map based on sensor data corresponding to a different data type than the first spatial map of Shin and the alignment of sensor data with layout data based on determining the sensor data and layout data are associated with the same site of Aggarwal to further include merging the sensor data of multiple robots taught by Gao with a reasonable expectation of success. A person of ordinary skill in the art would be motivated to make this modification in order to improve mapping and navigation within a spatial environment (Gao ¶ [0002]).
Regarding claim 6, Degirmenci fails to explicitly disclose wherein the first association comprises an anchoring of a waypoint associated with the sensor data to a corresponding feature of the site model.
However, Gao, in the same field of endeavor, teaches wherein the first association comprises an anchoring of a waypoint associated with the sensor data to a corresponding feature of the site model (Gao ¶ [0022]: the system may perform an alignment between the real world environment and a generated 3D point map. In another aspect, the system provides the ability to identify virtual landmarks within the real world and associate (e.g. label) these virtual landmarks within the 3D point cloud map. These virtual landmarks may be used to reference and index objects within the 3D point cloud map. Accordingly, these virtual landmarks and objects may then be used for mapping and navigation. For example, the virtual landmarks may act as navigation waypoints that may be shared across various devices. In addition, the virtual landmarks may be referenced by various devices in real time for re-localization).
Therefore, given the teachings as whole, it would have been prima facie obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the feature scanning for localization and navigation display of Degirmenci modified by the first spatial map from a first data source separate from a sensor and the second spatial map based on sensor data corresponding to a different data type than the first spatial map of Shin and the alignment of sensor data with layout data based on determining the sensor data and layout data are associated with the same site of Aggarwal to further include the waypoint anchoring of Gao with a reasonable expectation of success. A person of ordinary skill in the art would be motivated to make this modification in order to improve mapping and navigation within a spatial environment (Gao ¶ [0002]).
Claims 7, 15, 18, and 21 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication No. US 2024/0077882 by Degirmenci et al. (herein after “Degirmenci”), in view of U.S. Patent Application Publication No. US 2024/0118103 by Shin et al. (herein after “Shin”) and U.S. Patent No. US 12,619,240 by Aggarwal et al. (herein after “Aggarwal”), further in view of U.S. Patent Application Publication No. US 2020/0258400 by Yuan et al. (herein after “Yuan”).
Note: Text written in bold typeface is claim language from the instant application. Text written in normal typeface are comments made by the Examiner and/or passages from the prior art reference(s).
Regarding claim 7, Degirmenci discloses wherein transforming the virtual representation of the sensor data comprises mapping a set of points of the virtual representation of the sensor data to a set of corresponding features of the site model (Degirmenci ¶ [0093]: The locations of the various objects on the site map may be defined with respect to an origin. In some embodiments, the origin may comprise the start of the route. In some embodiments, the origin may be an arbitrary point within the environment. The robot 102 may recognize its initial position (e.g., upon turning on) based on detecting one or more recognizable features, such as landmarks (e.g., objects sensed in the past), computer-readable codes (e.g., quick-response codes, barcodes, etc.), markers, beacons, and the like. Such markers may indicate the origin point, or may be at a known distance from the origin with respect to which the robot 102 may localize itself).
It is noted Degirmenci fails to explicitly disclose wherein transforming the virtual representation of the sensor data comprises mapping a set of points of the virtual representation of the sensor data to a set of corresponding features of the site model, and applying a non-linear transformation to a portion of the virtual representation of the sensor data between the set of points.
However, Yuan, in the same field of endeavor, teaches wherein transforming the virtual representation of the sensor data comprises mapping a set of points of the virtual representation of the sensor data to a set of corresponding features of the site model (Yuan ¶ [0071]: As a result of localization 401 and perception 402, ground events (both ground objects and ground conditions) are identified. In the localization step 401, the localization of the UAV identifies its coordinates and places it within the semantic map 311. Ground events in the semantic map 311 that are near the coordinates of the UAV are identified. Real-time ground events are also identified by the perception step 402), and applying a non-linear transformation to a portion of the virtual representation of the sensor data between the set of points (Yuan ¶ [0083]: For one or more ground events 603 that are not impassable obstacles, the ground events 603 are passed to cost function 604. The cost function 604 may accept as input one or more ground events and output a weight for each ground event. The weight may represent the cost of the UAV flying over the ground event or in the vicinity of the ground event. A high weight may disincentivize a flight path and a low weight may incentivize the flight path; Yuan ¶ [0084]: The cost function may be implemented in a variety of ways such as a hash table or lookup table mapping types of ground events to weights, a linear combination that weights one or more aspects of the ground event and sums the weighted values, a non-linear function that computes the weight based on one or more aspects of the ground event, and so on). The examiner interprets the virtual representation of sensor data to include the flight path. An initial flight path may be transformed to an updated flight path based on a non-linear cost function.
Therefore, given the teachings as whole, it would have been prima facie obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the feature scanning for localization and navigation display of Degirmenci modified by the first spatial map from a first data source separate from a sensor and the second spatial map based on sensor data corresponding to a different data type than the first spatial map of Shin and the alignment of sensor data with layout data based on determining the sensor data and layout data are associated with the same site of Aggarwal to further include the non-linear transformation of Yuan with a reasonable expectation of success. A person of ordinary skill in the art would be motivated to make this modification in order to improve navigation by computing lowest cost routes utilizing ground objects and events that have been weighted based on a non-linear cost function and semantic map (Yuan ¶ [0028]).
Regarding claim 15, Degirmenci discloses wherein transforming the virtual representation of the sensor data comprises at least one of:
automatically transforming the virtual representation of the sensor data based on identifying the first association (Degirmenci ¶ [0123]: Starting with the local map 604-B in its illustrated position/orientation, controller 118 may, in executing ICP alignment algorithms, determine corresponding points on the objects 502 for both maps 510, 604-B. To determine a transform which causes alignment of the local scanning map 604-B to the site map 510; Fig. 7A).
It is noted Degirmenci fails to particularly disclose wherein transforming the virtual representation of the sensor data comprises at least one of: performing a non-linear transformation of the sensor data relative to the site model.
However, Yuan, in the same field of endeavor, teaches wherein transforming the virtual representation of the sensor data comprises at least one of:
performing a non-linear transformation of the sensor data relative to the site model (Yuan ¶ [0071]: As a result of localization 401 and perception 402, ground events (both ground objects and ground conditions) are identified. In the localization step 401, the localization of the UAV identifies its coordinates and places it within the semantic map 311. Ground events in the semantic map 311 that are near the coordinates of the UAV are identified. Real-time ground events are also identified by the perception step 402; Yuan ¶ [0083]: For one or more ground events 603 that are not impassable obstacles, the ground events 603 are passed to cost function 604. The cost function 604 may accept as input one or more ground events and output a weight for each ground event. The weight may represent the cost of the UAV flying over the ground event or in the vicinity of the ground event. A high weight may disincentivize a flight path and a low weight may incentivize the flight path; Yuan ¶ [0084]: The cost function may be implemented in a variety of ways such as a hash table or lookup table mapping types of ground events to weights, a linear combination that weights one or more aspects of the ground event and sums the weighted values, a non-linear function that computes the weight based on one or more aspects of the ground event, and so on). The examiner interprets the virtual representation of sensor data to include the flight path. An initial flight path may be transformed to an updated flight path based on a non-linear cost function.
Therefore, given the teachings as whole, it would have been prima facie obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the feature scanning for localization and navigation display of Degirmenci modified by the first spatial map from a first data source separate from a sensor and the second spatial map based on sensor data corresponding to a different data type than the first spatial map of Shin and the alignment of sensor data with layout data based on determining the sensor data and layout data are associated with the same site of Aggarwal to further include the non-linear transformation of Yuan with a reasonable expectation of success. A person of ordinary skill in the art would be motivated to make this modification in order to improve navigation by computing lowest cost routes utilizing ground objects and events that have been weighted based on a non-linear cost function and semantic map (Yuan ¶ [0028]).
Regarding claim 18, Degirmenci discloses further comprising: identifying a second association between the sensor data and the site model;
(Degirmenci ¶ [0093]: The locations of the various objects on the site map may be defined with respect to an origin. In some embodiments, the origin may comprise the start of the route. In some embodiments, the origin may be an arbitrary point within the environment. The robot 102 may recognize its initial position (e.g., upon turning on) based on detecting one or more recognizable features, such as landmarks (e.g., objects sensed in the past), computer-readable codes (e.g., quick-response codes, barcodes, etc.), markers, beacons, and the like. Such markers may indicate the origin point, or may be at a known distance from the origin with respect to which the robot 102 may localize itself).
It is noted Degirmenci fails to explicitly disclose further comprising: assigning a first weight to the first association; assigning a second weight to the second association, wherein transforming the virtual representation of the sensor data is further based on the second association, the first weight, and the second weight.
However, Yuan, in the same field of endeavor, teaches further comprising: identifying a second association between the sensor data and the site model (Yuan ¶ [0025]: The localization system precisely localizes the UAV in the environment and may use any one or combination of global navigation satellite system (GNSS), global positioning system (GPS), inertial measurement unit (IMU), perception-based localization systems (e.g., image-based, LIDAR based, depth-sensor based, and so on), and sensor fusion of the aforementioned approaches. After the precise localization of the UAV, the UAV is aware of the ground objects and conditions in its vicinity and relation to the UAV based on the detailed semantic map);
assigning a first weight to the first association; and assigning a second weight to the second association, wherein transforming the virtual representation of the sensor data is further based on the second association, the first weight, and the second weight (Yuan ¶ [0071]: As a result of localization 401 and perception 402, ground events (both ground objects and ground conditions) are identified. In the localization step 401, the localization of the UAV identifies its coordinates and places it within the semantic map 311. Ground events in the semantic map 311 that are near the coordinates of the UAV are identified. Real-time ground events are also identified by the perception step 402; Yuan ¶ [0083]: For one or more ground events 603 that are not impassable obstacles, the ground events 603 are passed to cost function 604. The cost function 604 may accept as input one or more ground events and output a weight for each ground event. The weight may represent the cost of the UAV flying over the ground event or in the vicinity of the ground event. A high weight may disincentivize a flight path and a low weight may incentivize the flight path; Yuan ¶ [0084]: The cost function may be implemented in a variety of ways such as a hash table or lookup table mapping types of ground events to weights, a linear combination that weights one or more aspects of the ground event and sums the weighted values, a non-linear function that computes the weight based on one or more aspects of the ground event, and so on). The examiner interprets the virtual representation of sensor data to include the flight path. An initial flight path may be transformed to an updated flight path based on a cost function that weights ground events detected by sensors, wherein the ground events may be used to localize the UAV within a semantic map using perception-based localization.
Therefore, given the teachings as whole, it would have been prima facie obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the feature scanning for localization and navigation display of Degirmenci modified by the first spatial map from a first data source separate from a sensor and the second spatial map based on sensor data corresponding to a different data type than the first spatial map of Shin and the alignment of sensor data with layout data based on determining the sensor data and layout data are associated with the same site of Aggarwal to further include the weighted associations and path transformation of Yuan with a reasonable expectation of success. A person of ordinary skill in the art would be motivated to make this modification in order to improve navigation by computing lowest cost routes utilizing ground objects and events that have been weighted based on a cost function and semantic map (Yuan ¶ [0028]).
Regarding claim 21, Degirmenci fails to explicitly disclose wherein execution of the instructions on the data processing hardware further causes the data processing hardware to: assign a weight to the first association, wherein transforming the virtual representation of the sensor data is further based on the weight.
However, Yuan, in the same field of endeavor, teaches wherein execution of the instructions on the data processing hardware further causes the data processing hardware to: assign a weight to the first association, wherein transforming the virtual representation of the sensor data is further based on the weight (Yuan ¶ [0071]: As a result of localization 401 and perception 402, ground events (both ground objects and ground conditions) are identified. In the localization step 401, the localization of the UAV identifies its coordinates and places it within the semantic map 311. Ground events in the semantic map 311 that are near the coordinates of the UAV are identified. Real-time ground events are also identified by the perception step 402; Yuan ¶ [0083]: For one or more ground events 603 that are not impassable obstacles, the ground events 603 are passed to cost function 604. The cost function 604 may accept as input one or more ground events and output a weight for each ground event. The weight may represent the cost of the UAV flying over the ground event or in the vicinity of the ground event. A high weight may disincentivize a flight path and a low weight may incentivize the flight path; Yuan ¶ [0084]: The cost function may be implemented in a variety of ways such as a hash table or lookup table mapping types of ground events to weights, a linear combination that weights one or more aspects of the ground event and sums the weighted values, a non-linear function that computes the weight based on one or more aspects of the ground event, and so on). The examiner interprets the virtual representation of sensor data to include the flight path. An initial flight path may be transformed to an updated flight path based on a cost function that weights ground events, wherein the ground events may be used to localize the UAV within a semantic map based on perception-based localization.
Therefore, given the teachings as whole, it would have been prima facie obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the feature scanning for localization and navigation display of Degirmenci modified by the first spatial map from a first data source separate from a sensor and the second spatial map based on sensor data corresponding to a different data type than the first spatial map of Shin and the alignment of sensor data with layout data based on determining the sensor data and layout data are associated with the same site of Aggarwal to further include the weighted associations and path transformation of Yuan with a reasonable expectation of success. A person of ordinary skill in the art would be motivated to make this modification in order to improve navigation by computing lowest cost routes utilizing ground objects and events that have been weighted based on a cost function and semantic map (Yuan ¶ [0028]).
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication No. US 2024/0077882 by Degirmenci et al. (herein after “Degirmenci”), in view of U.S. Patent Application Publication No. US 2024/0118103 by Shin et al. (herein after “Shin”) and U.S. Patent No. US 12,619,240 by Aggarwal et al. (herein after “Aggarwal”), further in view of U.S. Patent Application Publication No. US 2012/0197464 by Wang et al. (herein after “Wang”) and U.S. Patent Application Publication No. US 2013/0002822 by Yu et al. (herein after “Yu”).
Note: Text written in bold typeface is claim language from the instant application. Text written in normal typeface are comments made by the Examiner and/or passages from the prior art reference(s).
Regarding claim 10, Degirmenci fails to explicitly disclose further comprising:
identifying a first scale associated with the site model; identifying a second scale associated with the sensor data; determining a ratio of the site model to the sensor data based on the first scale and the second scale; and at least one of: adjusting one or more of the first scale, the second scale, or the ratio based on the first association; or instructing display of the virtual representation of the sensor data overlaid on the site model based on the ratio.
However, Wang, in the same field of endeavor, teaches further comprising:
identifying a first scale associated with the site model; identifying a second scale associated with the sensor data; and at least one of: adjusting one or more of the first scale, the second scale, or the ratio based on the first association; or instructing display of the virtual representation of the sensor data overlaid on the site model based on the ratio (Wang ¶ [0262]-[0263]: the robot 100 computes a scaling size, origin mapping, and rotation between the plan view map 810 and the robot map 820 using existing tagged locations, and then applies the computed parameters to determine the robot map location (e.g., using an affine transformation or coordinates). The robot map 820 may not be the same orientation and scale as the plan view map 810. Moreover, the layout map may not be to scale and may have distortions that vary by map area. For example, a plan view map 810 created by scanning a fire evacuation map typically seen in hotels, offices, and hospitals is usually not drawn to scale and can even have different scales in different regions of the map).
It is noted Wang fails to particularly teach determining a ratio of the site model to the sensor data based on the first scale and the second scale.
However, Yu, in the same field of endeavor, teaches determining a ratio of the site model to the sensor data based on the first scale and the second scale (Yu ¶ [0015]: The processing unit 20 further generates a scaled down 3D model of the user according to the ratio of the life size data Y to a scale of the selected product model).
Therefore, given the teachings as whole, it would have been prima facie obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the feature scanning for localization and navigation display of Degirmenci modified by the first spatial map from a first data source separate from a sensor and the second spatial map based on sensor data corresponding to a different data type than the first spatial map of Shin and the alignment of sensor data with layout data based on determining the sensor data and layout data are associated with the same site of Aggarwal to further include the scaling adjustment of Wang and the ratio determination of Yu with a reasonable expectation of success. A person of ordinary skill in the art would be motivated to make this modification in order to improve robot localization within a map when a local robot map and global plan view map have different orientations and scale (Wang ¶ [0263]) and to improve alignment of overlaid images.
Claim 22 is rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication No. US 2024/0077882 by Degirmenci et al. (herein after “Degirmenci”), in view of U.S. Patent Application Publication No. US 2024/0118103 by Shin et al. (herein after “Shin”) and U.S. Patent No. US 12,619,240 by Aggarwal et al. (herein after “Aggarwal”), further in view of O. Mendez, S. Hadfield, N. Pugeault and R. Bowden, "SeDAR - Semantic Detection and Ranging: Humans can Localise without LiDAR, can Robots?," 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, QLD, Australia, 2018, pp. 6053-6060 (herein after “Mendez”).
Note: Text written in bold typeface is claim language from the instant application. Text written in normal typeface are comments made by the Examiner and/or passages from the prior art reference(s).
Regarding claim 22, Degirmenci discloses wherein to identify the first association, execution of the instructions on the data processing hardware further causes the data processing hardware to:
generate an estimation of the first association based on the sensor data and the point cloud (Degirmenci ¶ [0093]: The locations of the various objects on the site map may be defined with respect to an origin. In some embodiments, the origin may comprise the start of the route. In some embodiments, the origin may be an arbitrary point within the environment. The robot 102 may recognize its initial position (e.g., upon turning on) based on detecting one or more recognizable features, such as landmarks (e.g., objects sensed in the past), computer-readable codes (e.g., quick-response codes, barcodes, etc.), markers, beacons, and the like. Such markers may indicate the origin point, or may be at a known distance from the origin with respect to which the robot 102 may localize itself).
It is noted Degirmenci fails to explicitly disclose wherein to identify the first association, execution of the instructions on the data processing hardware further causes the data processing hardware to: convert the site model into a point cloud.
However, Mendez, in the same field of endeavor, teaches wherein to identify the first association, execution of the instructions on the data processing hardware further causes the data processing hardware to: convert the site model into a point cloud (Mendez pg. 6055 ¶ 3: we augment the localisation with semantic labels extracted from an existing floorplan. In our experiments we limit these labels to walls, doors and windows (see figure 2), which are easy to automatically extract from a floorplan, and are also salient for human localisation. In order to make a labelled floorplan readable by the robot, it must first be converted into an occupancy grid. An occupancy grid is a 2D representation of the world, in which each cell in the grid has an occupancy probability, determined by it’s normalized greyscale value; Fig. 2); and
generate an estimation of the first association based on the sensor data and the point cloud (Mendez pg. 6053 ¶ 7: We then collapse all semantic information into 2D in order to reduce the assumptions about the environment. We then use these labels, image geometry and (optionally) depth along with a semantically labelled floorplan to create a state-of-the-art sensing and localisation framework).
Therefore, given the teachings as whole, it would have been prima facie obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the feature scanning for localization and navigation display of Degirmenci modified by the first spatial map from a first data source separate from a sensor and the second spatial map based on sensor data corresponding to a different data type than the first spatial map of Shin and the alignment of sensor data with layout data based on determining the sensor data and layout data are associated with the same site of Aggarwal to further include the floorplan point cloud conversion of Mendez with a reasonable expectation of success. A person of ordinary skill in the art would be motivated to make these modifications in order to facilitate global localization relying on semantic labels present in a floorplan and extracted from RGB images (Mendez Abstract).
Claims 25 and 27-30 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication No. US 2024/0077882 by Degirmenci et al. (herein after “Degirmenci”), in view of U.S. Patent Application Publication No. US 2024/0118103 by Shin et al. (herein after “Shin”) and U.S. Patent No. US 12,619,240 by Aggarwal et al. (herein after “Aggarwal”) and U.S. Patent Application Publication No. US 2023/0215092 by Kim et al. (herein after “Kim”).
Note: Text written in bold typeface is claim language from the instant application. Text written in normal typeface are comments made by the Examiner and/or passages from the prior art reference(s).
Regarding claim 25, Degirmenci discloses a robot comprising: at least one sensor; data processing hardware in communication with the at least one sensor (Degirmenci ¶ [0060]: sensor units 114 may comprise systems and/or methods that may detect characteristics within and/or around robot 102); and memory in communication with the data processing hardware (Degirmenci ¶ [0049]: As illustrated in FIG. 1A, robot 102 may include controller 118, memory 120, user interface unit 112, sensor units 114, navigation units 106, actuator unit 108, and communications unit 116), the memory storing instructions that when executed on the data processing hardware cause the data processing hardware to (Degirmenci ¶ [0049]: As illustrated in FIG. 1A, robot 102 may include controller 118, memory 120; Degirmenci ¶ [0052]: Memory 120 may provide instructions and data to controller 118. For example, memory 120 may be a non-transitory, computer-readable storage apparatus and/or medium having a plurality of instructions stored thereon, the instructions being executable by a processing apparatus (e.g., controller 118) to operate robot 102):
obtain, from the at least one sensor, sensor data captured from a site (Degirmenci ¶ [0089]: a local route or local scanning route includes a route for a robot 102 to navigate, wherein the robot 102 scans for features in acquired sensor data during execution of the local route; Degirmenci ¶ [0130]: the controller 118 may collect data useful for identifying features, such as images, videos, LiDAR/point cloud data, thermal data, and/or any other data collected by sensor units 114), wherein the site is associated with a site model (Degirmenci ¶ [0091]: Block 402 includes the controller 118 learning a site map within the environment. To learn a map, controller 118 may be navigated through its environment to collect data from sensor units 114. The data may indicate the presence/location of various objects in the environment which may be localized onto the site map)
provide the sensor data to a computing system for generation of a virtual representation of the sensor data (Degirmenci ¶ [0095]: During navigation of each of these local routes, controller 118 may produce a corresponding local route map. Local route maps may include various objects sensed and localized by the sensor units 114 of the robot 102),
wherein at least a portion of the sensor data is associated with at least a portion of the site model via a first association (Degirmenci ¶ [0093]: The locations of the various objects on the site map may be defined with respect to an origin. In some embodiments, the origin may comprise the start of the route. In some embodiments, the origin may be an arbitrary point within the environment. The robot 102 may recognize its initial position (e.g., upon turning on) based on detecting one or more recognizable features, such as landmarks (e.g., objects sensed in the past), computer-readable codes (e.g., quick-response codes, barcodes, etc.), markers, beacons, and the like. Such markers may indicate the origin point, or may be at a known distance from the origin with respect to which the robot 102 may localize itself)
wherein the virtual representation of the sensor data is transformed based on the first association to generate transformed data (Degirmenci ¶ [0094]: the controller 118 learning at least one local scanning route within the environment, wherein each of the at least one local scanning routes corresponds to a respective local route map; Degirmenci ¶ [0097]: Block 406 includes the controller 118 aligning the at least one local route map to the site map. To align the maps, the controller 118 may determine a transform (i.e., translation and/or rotation) between the origin of the site map and the origin(s) of each of the at least one local route map such that both the site map and the at least one local map align. Controller 118 may utilize iterative closest point (“ICP”) algorithms, or similar nearest neighboring alignment algorithms, to determine the transform, as shown in FIGS. 7A-B below. Once the maps align with minimal error, the controller 118 may define an origin of the local route map with respect to the origin of the site map),
wherein a user interface reflects the transformed data overlaid on the site model (Degirmenci ¶ [0127]: FIG. 8A may illustrate a display on a user interface unit 112 of the robot 102 or user interface coupled to the robot 102 (e.g., a personal computer or device 208 coupled to a server 202). According to at least one non-limiting exemplary embodiment, the aligned site map 510 and local scanning route maps 604 may be communicated to a server 202, wherein a device 208 coupled to the server 202 may be configured to receive the annotations. In some embodiments, the one or more local scanning route maps 604 may be overlaid on top of the site map 510 to display all of the objects 502 in the environment, even if some objects 502 are only partially sensed on the site map 510);
obtain one or more instructions to traverse the site based on the user interface; and instruct traversal of the site (Degirmenci ¶ [0141]: FIG. 9A(i-ii) illustrate edits performed to a route 902 in accordance with block 410 of method 400, according to an exemplary embodiment. Two types of route edits may be performed: (a) manual edits based on user inputs to user interface units 112, and (b) automatic edits. First, FIG. 9A(i) shows an exemplary scenario where a robot 102, in learning a local scanning route 604, encounters an object 902. The object 902 may be a temporary object which does not persist within the environment. For example, the object 902 may be a human, a shopping cart, a puddle of water, and so forth. The human operator, during training/demonstration of the local route 604, may navigate around the temporary object 902 and produce a curve in route 604 around the object 902. The curve route 604 was only executed to avoid collision and is not an indented behavior of the robot 102 to be learned. If the robot 102 is to scan for features of objects 502 (e.g., objects on a shelf) the curve may be undesirable as the robot 102 may capture sensor data of the object 502 at poor angles and from farther distances. Accordingly, the human operator may be prompted to perform edits to the route 604 after completing the training to enhance the feature scanning. The human operator may utilize the user interface 112 of the robot 102 or a separate user interface, such as a user interface on a device 208 coupled to a server 202 or directly coupled to the robot 102).
It is noted Degirmenci fails to explicitly disclose a robot comprising: at least two legs;
obtain, from the at least one sensor, sensor data captured from a site by the at least one sensor, wherein the site is associated with a site model, wherein a data source of the site model is distinct and separate from the at least one sensor, and wherein the site model and the sensor data correspond to different types of data;
wherein at least a portion of the sensor data is associated with at least a portion of the site model via a first association based on determining that the sensor data and the site model are associated with the same site; and
instruct traversal of the site using the at least two legs based on the one or more instructions.
However, Shin, in the same field of endeavor, teaches obtain, from the at least one sensor, sensor data captured from a site (Shin ¶ [0084]: the second spatial map 220 may be a spatial map generated based on sensor data obtained by measuring the space. The sensor data may be obtained using one or more sensors capable of measuring the space. The one or more sensors may be, for example, but are not limited to, a red, green, blue (RGB) sensor, an RGB-Depth (RGB-D) sensor, a time-of-flight (ToF) sensor, a light detection and ranging (lidar) sensor, a radio detection and ranging (radar) sensor, etc. In addition, the one or more sensors may be mounted on an indoor robot, a robot cleaner, a user's smartphone, etc), wherein the site is associated with a site model (Shin ¶ [0070]: a first spatial map 210 may be map data including various pieces of information related to a space. The first spatial map 210 may include, for example, a two-dimensional (2D)/3D spatial map, a 2D/3D floorplan, and a blueprint, but is not limited thereto, and the first spatial map 210 may be various forms of map data including 2D/3D spatial information), wherein a data source of the site model is distinct and separate from the at least one sensor (Shin ¶ [0100]: the server 2000 obtains a pre-generated first spatial map including first map features representing spatial information about a space. The server 2000 may obtain a pre-generated first spatial map from a map database stored on the server 2000; Shin ¶ [0086]: The server 2000 may obtain the second spatial map 220 generated by an electronic device including one or more sensors), and wherein the site model and the sensor data correspond to different types of data (Shin ¶ [0130]: based on a spatial dimension of the first spatial map being different from that of the second spatial map, the server 2000 projects a higher dimensional spatial map among the first spatial map and the second spatial map to correspond to a spatial dimension of the lower dimensional spatial map).
Further, Aggarwal, in the same field of endeavor, teaches obtain, from the at least one sensor, sensor data captured from a site by the at least one sensor (Aggarwal col. 26 lines 9-10: At 1006, the process 1000 may include receiving LiDAR data from the reality capture robot 100; Aggarwal col. 26 lines 19-21: At 1008, the process 1000 may include determining, based at least in part on the LiDAR data, a 3D point cloud of the environment), wherein the site is associated with a site model (Aggarwal col. 26 lines 35-38: At 1010, the process 1000 may include determining a layout of the environment 102. For example, the management system 110 may access the layout data 122 for determining the layout of the environment 102);
wherein at least a portion of the sensor data is associated with at least a portion of the site model via a first association (Aggarwal col. 26 lines 46-51: At 1012, the process 1000 may include aligning the layout and the 3D point cloud. For example, the management system 110 may align the layout and the 3D point cloud. In some instances, origins or reference points in the layout and the 3D point cloud such that like objects are compared within one another across the layout and the 3D point cloud. For example, certain walls, pillars, or a corner of the environment 102 may be aligned) based on determining that the sensor data and the site model are associated with the same site (Aggarwal col. 26 lines 56-62: In instances where the reality capture robot 100 scans a portion of the environment 102, a corresponding portion of the layout may be determined for aligning the 3D point cloud thereto. For example, in instances where reality capture robot 100 scans a portion of the environment 102, the management system 110 may determine corresponding portion of the layout for aligning with the 3D point cloud).
Finally, Kim, in the same field of endeavor, teaches a robot comprising: at least two legs (Kim ¶ [0175]: the walking-type moving device may include at least two leg-type supports for movement of the robot 500); and
obtain one or more instructions to traverse the site based on the user interface; and instruct traversal of the site using the at least two legs based on the one or more instructions (Kim ¶ [0172]: the moving unit 540 of the robot 500 may allow the robot 500 to move to a position under the control of a user or a processor; Kim ¶ [0251]: the application 111 may acquire a user's input for setting an execution function such as outputting predetermined augmented reality content or designating a predetermined waypoint with respect to the target object).
Therefore, given the teachings as whole, it would have been prima facie obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the robot including feature scanning for localization and navigation display of Degirmenci to include the first spatial map from a first data source separate from a sensor and the second spatial map based on sensor data corresponding to a different data type than the first spatial map of Shin, the alignment of sensor data with layout data based on determining the sensor data and layout data are associated with the same site of Aggarwal and the at least 2 legs and traversal instructions of Kim with a reasonable expectation of success. A person of ordinary skill in the art would be motivated to make these modifications in order to reflect personal spatial information such as an actual layout of a user’s space and real-world objects when generating and updating spatial maps (Shin ¶ [0005]), to compare respective locations of objects within an environment allowing for inaccuracies in layout data to be corrected while generating a static map (Aggarwal col. 15 lines 1-10) and to generate a 3D map and set object-based function operations using classified objects displayed in the map (Kim ¶ [0015]).
Regarding claim 27, Degirmenci discloses wherein to obtain the one or more instructions, execution of the instructions on the data processing hardware further causes the data processing hardware to: obtain the one or more instructions from a user computing device (Degirmenci ¶ [0141]: FIG. 9A(i-ii) illustrate edits performed to a route 902 in accordance with block 410 of method 400, according to an exemplary embodiment. Two types of route edits may be performed: (a) manual edits based on user inputs to user interface units 112, and (b) automatic edits. First, FIG. 9A(i) shows an exemplary scenario where a robot 102, in learning a local scanning route 604, encounters an object 902. The object 902 may be a temporary object which does not persist within the environment. For example, the object 902 may be a human, a shopping cart, a puddle of water, and so forth. The human operator, during training/demonstration of the local route 604, may navigate around the temporary object 902 and produce a curve in route 604 around the object 902. The curve route 604 was only executed to avoid collision and is not an indented behavior of the robot 102 to be learned. If the robot 102 is to scan for features of objects 502 (e.g., objects on a shelf) the curve may be undesirable as the robot 102 may capture sensor data of the object 502 at poor angles and from farther distances. Accordingly, the human operator may be prompted to perform edits to the route 604 after completing the training to enhance the feature scanning. The human operator may utilize the user interface 112 of the robot 102 or a separate user interface, such as a user interface on a device 208 coupled to a server 202 or directly coupled to the robot 102; Degirmenci ¶ [0072]: Devices 208 may comprise any device configured to perform a task at an edge of the server 202. These devices may include, without limitation, internet of things (IoT) devices (e.g., stationary CCTV cameras, smart locks, smart thermostats, etc.), external processors (e.g., external CPUs or GPUs), and/or external memories configured to receive and execute a sequence of computer-readable instructions, which may be provided at least in part by the server 202, and/or store large amounts of data).
Regarding claim 28, Degirmenci discloses wherein the user interface comprises a user interface of a user computing device (Degirmenci ¶ [0141]: The human operator may utilize the user interface 112 of the robot 102 or a separate user interface, such as a user interface on a device 208; Degirmenci ¶ [0072]: Devices 208 may comprise any device configured to perform a task at an edge of the server 202. These devices may include, without limitation, internet of things (IoT) devices (e.g., stationary CCTV cameras, smart locks, smart thermostats, etc.), external processors (e.g., external CPUs or GPUs), and/or external memories configured to receive and execute a sequence of computer-readable instructions, which may be provided at least in part by the server 202, and/or store large amounts of data), wherein to obtain the one or more instructions, execution of the instructions on the data processing hardware further causes the data processing hardware to: obtain the one or more instructions from the user computing device (Degirmenci ¶ [0141]: FIG. 9A(i-ii) illustrate edits performed to a route 902 in accordance with block 410 of method 400, according to an exemplary embodiment. Two types of route edits may be performed: (a) manual edits based on user inputs to user interface units 112, and (b) automatic edits. First, FIG. 9A(i) shows an exemplary scenario where a robot 102, in learning a local scanning route 604, encounters an object 902. The object 902 may be a temporary object which does not persist within the environment. For example, the object 902 may be a human, a shopping cart, a puddle of water, and so forth. The human operator, during training/demonstration of the local route 604, may navigate around the temporary object 902 and produce a curve in route 604 around the object 902. The curve route 604 was only executed to avoid collision and is not an indented behavior of the robot 102 to be learned. If the robot 102 is to scan for features of objects 502 (e.g., objects on a shelf) the curve may be undesirable as the robot 102 may capture sensor data of the object 502 at poor angles and from farther distances. Accordingly, the human operator may be prompted to perform edits to the route 604 after completing the training to enhance the feature scanning. The human operator may utilize the user interface 112 of the robot 102 or a separate user interface, such as a user interface on a device 208 coupled to a server 202 or directly coupled to the robot 102).
Regarding claim 29, Degirmenci discloses wherein the first association is maintained within the transformed data (Degirmenci ¶ [0093]: The locations of the various objects on the site map may be defined with respect to an origin. In some embodiments, the origin may comprise the start of the route. In some embodiments, the origin may be an arbitrary point within the environment. The robot 102 may recognize its initial position (e.g., upon turning on) based on detecting one or more recognizable features, such as landmarks (e.g., objects sensed in the past), computer-readable codes (e.g., quick-response codes, barcodes, etc.), markers, beacons, and the like. Such markers may indicate the origin point, or may be at a known distance from the origin with respect to which the robot 102 may localize itself).
Regarding claim 30, Degirmenci discloses wherein the transformed data is associated with the site model via the first association and a second association (Degirmenci ¶ [0093]: The locations of the various objects on the site map may be defined with respect to an origin. In some embodiments, the origin may comprise the start of the route. In some embodiments, the origin may be an arbitrary point within the environment. The robot 102 may recognize its initial position (e.g., upon turning on) based on detecting one or more recognizable features, such as landmarks (e.g., objects sensed in the past), computer-readable codes (e.g., quick-response codes, barcodes, etc.), markers, beacons, and the like. Such markers may indicate the origin point, or may be at a known distance from the origin with respect to which the robot 102 may localize itself). The examiner interprets the one or more recognizable features such as landmarks to be associations between a virtual representation and a site model.
Claim 26 is rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication No. US 2024/0077882 by Degirmenci et al. (herein after “Degirmenci”), in view of U.S. Patent Application Publication No. US 2024/0118103 by Shin et al. (herein after “Shin”) and U.S. Patent No. US 12,619,240 by Aggarwal et al. (herein after “Aggarwal”) and U.S. Patent Application Publication No.US 2023/0215092 by Kim et al. (herein after “Kim”), and further in view of U.S. Patent Application Publication No. US 2019/0355173 by Gao (herein after “Gao”).
Note: Text written in bold typeface is claim language from the instant application. Text written in normal typeface are comments made by the Examiner and/or passages from the prior art reference(s).
Regarding claim 26, Degirmenci fails to particularly disclose wherein the sensor data is captured by a set of sensors from two or more robots.
However, Gao, in the same field of endeavor, teaches wherein the sensor data is captured by a set of sensors from two or more robots (Gao ¶ [0036]: In 204, the system may merge multiple 3D point cloud maps. As described, the system may receive information from multiple devices at the same time, and accordingly, multiple maps may be generated simultaneously (or in parallel, concurrently, etc.) from information received from various locations. For instance, a first robot may traverse a first room while a second robot traverses a second room at the same time. Accordingly, a merged (or global) map may be generated by the system when it is determined that the paths of the first and second robots have crossed. Accordingly, upon such a determination, the system may merge portions (or segments, fragments, etc.) of the map automatically).
Therefore, given the teachings as whole, it would have been prima facie obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the robot including feature scanning for localization and navigation display of Degirmenci the first spatial map from a first data source separate from a sensor and the second spatial map based on sensor data corresponding to a different data type than the first spatial map of Shin, the alignment of sensor data with layout data based on determining the sensor data and layout data are associated with the same site of Aggarwal and the at least 2 legs and traversal instructions of Kim to further include merging of the sensor data of multiple robots taught by Gao. A person of ordinary skill in the art would be motivated to make this modification in order to improve mapping and navigation within a spatial environment (Gao ¶ [0002]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to the applicant’s disclosure:
US 20210138659 by Arora et al. discloses a method that includes presenting, on a user interface of a user computing device, a map of an environment and a first indicator of a recommended behavior control zone overlaid on the map, the recommended behavior control zone being based on sensor data collected by an autonomous mobile robot
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NICHOLAS P LANGHORNE whose telephone number is (571)272-5670. The examiner can normally be reached M-F 8:30-5:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Anne Antonucci can be reached at (313) 446-6519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/N.P.L./Examiner, Art Unit 3666
/ANNE MARIE ANTONUCCI/Supervisory Patent Examiner, Art Unit 3666
Prosecution Insights