DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendment to claim 1 has overcome the objection due to minor informality. In light of the amendments, the objection to claim 1 has been withdrawn.
Response to Arguments
Applicant's arguments, see the section titled “B. Claim Rejections under 35 U.S.C. 103” starting on page 11 of the reply filed 10/29/2025, have been fully considered but they are not persuasive.
For example, in the final paragraph beginning on page 13 of the reply filed 10/29/2025, and similarly in the second to last paragraph beginning on page 17, Applicant argues that “the solution of claim 1 is now clarified to utilize only point locations of one or two reference points predetermined in the worksite coordinate system by specifying that in the data acquired by the single tracking apparatus the tracked one to two reference point locations with respect to the single tracking apparatus consist of direction and distance from the single tracking apparatus to each of the tracked one to two reference point locations”; however, the Examiner disagrees. The Examiner understands that the claim language does not limit the utilization of only point locations of one to two reference points. For example, the claim states “when the single tracking apparatus is set external from the machine, the single tracking apparatus acquires data by tracking both the one to two reference point locations and the locations of the at least three marker points” where the Examiner opines that the limitation merely requires that a set of one to two reference point locations are tracked. The claim now further recites “wherein in the data acquired by the single tracking apparatus the tracked locations with respect to the single tracking apparatus consist of direction and distance from the single tracking apparatus to each of the tracked locations”. The Examiner notes that the phrase “consist of” is generally understood to exclude any element, step, or ingredient not specified in the claim (See MPEP 2111.03 II), however, the claim now merely requires that the tracked locations with respect to the single tracking apparatus consist of direction and distance from the single tracking apparatus to each of the tracked locations and does not preclude the number of tracked locations, for example. In paragraphs [0023-0025], Friend discloses that perception device 120 and processor 130 identifies the presence of at least one fiducial 190a in the field of view and determines a relative distance and orientation to the fiducial 190a (marker point) and/or loading machine 180 (machine). Therefore, the Examiner maintains the previous grounds of rejection. See the rejections under 35 U.S.C. 103 below.
Furthermore, Applicant argues in the first paragraph beginning on page 19 that “The skilled person would not turn to Yao to look for a solution in which sensor data is obtained simultaneously or to Clark to look for a solution in which scanners are mounted on a machine for measuring distances to a number of points in the ground, … since Friend has already provided a solution to determine location and orientation of a machine in a worksite by tracking the three-dimensional reference fiducial.” However, the Examiner disagrees. It is the Examiner’s opinion that a person having ordinary skill in the art before the effective filing date of the claimed invention to implement the teachings of Yao and Clark in that obtaining the sensor data at the same time instant as taught by Yao is advantageous in that it ensures the sensor data being used to determine the position and orientation are obtained simultaneously, maintaining the relative accuracy of the measurements to each other, for example, thereby increasing the accuracy of the determinations which utilize the measurements, and the ability to provide real-time data of the machine gives a redundant check and validation of three dimensional machine control system performance, and where using this data in real time gives the machine control system the ability to fine tune the automatic controls to achieve required tolerances in the generated surface, as suggested by Clark in paragraph [0018], which may advantageously increase the contextual accuracy of operation of the machine, for example, as expressed by Examiner in the rejections under 35 U.S.C. 103 below.
In the first paragraph beginning on page 20 Applicant further argues that neither Friend, Yao, nor Clark could perform the concepts of the presented claims. In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986). It is the Examiner’s opinion that the combination of Friend, Yao, and Clark does teach the subject matter of claim 1 as is detailed in the rejections under 35 U.S.C. 103 below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 3-6, 8-9, 11-14, and 17-19 are rejected under 35 U.S.C. 103 as being unpatentable over Friend (US 2014/0146167 A1), in view of Yao (US 2019/0102668 A1) and Clark (US 2010/0046800 A1).
Regarding claim 1, Friend discloses a method for determining situational awareness in a worksite, the method comprising:
setting a single tracking apparatus at least one of: external from the machine, when the single tracking apparatus has a location in the worksite coordinate system, for at least tracking both at least one reference point and at least three marker points, wherein the at least one reference point is in the worksite and has a location determined in the worksite coordinate system and the at least three marker points are attached to the machine and have locations determined in the machine coordinate system (In paragraph [0021], Friend discloses that a location may be a coordinate such as distances from a reference point that specifies where a machine is or may be situated at a mining site; the examiner understands the machine coordinate system to be at least a coordinate system for providing the relative locations of the haul machine 110 and loading machine 180; in paragraph [0022], Friend discloses fiducials 190a and 190b (marker points) on loading machine 180 (machine); in paragraph [0023], Friend discloses a haul machine 110 (tracking apparatus) including a perception device 120 and processor 130; in paragraph [0046], Friend discloses fiducials 631a and 631b (reference points) indicating a reference location 630 in a mining site 600 (worksite); see also paragraphs [0039-0044] where Friend discloses examples of loading machines, such as loading machines 180 and 181 (machine), including one or more fiducials mounted on excavator 400, such as fiducials 430a, 430b, 430c, 430d, 430e, 430f, and 430g, or one or more fiducials mounted on wheel loader 550, such as fiducials 580a, 580b, 580c, and 580d);
acquiring data by the single tracking apparatus by tracking reference point and marker point locations with respect to the single tracking apparatus (In paragraphs [0023-0025], Friend discloses that perception device 120 and processor 130 identifies the presence of at least one fiducial 190a in the field of view and determines a relative distance and orientation to the fiducial 190a (marker point) and/or loading machine 180 (machine); in paragraph [0046], Friend discloses that processor 130 may be configured to obtain information from fiducial 631a (reference point) via perception device 120 for determining a position of mining truck 300 (tracking apparatus)), wherein
when the single tracking apparatus is set external from the machine, the single tracking apparatus acquires data by tracking both the one to two reference point locations and the locations of the at least three marker points (In paragraphs [0023-0025], Friend discloses that perception device 120 and processor 130 identifies the presence of at least one fiducial 190a in the field of view and determines a relative distance and orientation to the fiducial 190a (marker point) and/or loading machine 180 (machine); in paragraph [0046], Friend discloses that processor 130 may be configured to obtain information from fiducial 631a (reference point) via perception device 120 for determining a position of mining truck 300 (tracking apparatus); see also paragraphs [0039-0044] where Friend discloses examples of loading machines, such as loading machines 180 and 181 (machine), including one or more fiducials mounted on excavator 400, such as fiducials 430a, 430b, 430c, 430d, 430e, 430f, and 430g, or one or more fiducials mounted on wheel loader 550, such as fiducials 580a, 580b, 580c, and 580d);
and wherein in the data acquired by the single tracking apparatus the tracked locations with respect to the single tracking apparatus consist of direction and distance from the single tracking apparatus to each of the tracked locations (In paragraphs [0023-0025], Friend discloses that perception device 120 and processor 130 identifies the presence of at least one fiducial 190a in the field of view and determines a relative distance and orientation to the fiducial 190a (marker point) and/or loading machine 180 (machine));
acquiring data by at least one sensor (In paragraph [0033], Friend discloses that haul machine 110 may include an inertial measurement unit (IMU) 170, which typically comprises three orthogonally oriented accelerometers for detecting changes in position, three orthogonally oriented gyroscopes for detecting changes in orientation, and, in some cases, three orthogonally oriented magnetometers serving as a three-dimensional electronic compass), wherein
when the single tracking apparatus tracks one reference point location with respect to the single tracking apparatus, the at least one sensor acquires inclination and heading data of one of: the tracking apparatus or the machine (In paragraphs [0023-0025], Friend discloses that perception device 120 and processor 130 identifies the presence of at least one fiducial 190a in the field of view and determines a relative distance and orientation to the fiducial 190a (marker point) and/or loading machine 180 (machine); in paragraph [0046], Friend discloses that processor 130 may be configured to obtain information from fiducial 631a (reference point) via perception device 120 for determining a position of mining truck 300 (tracking apparatus); in paragraph [0033], Friend discloses that processor 130 may be configured to use information obtained from IMU 170 for determining changes in position and orientation of haul machine 110 between determinations of position and orientation using information obtained from perception device 120, which may be able at a much lower rate than the IMU information; see also paragraph [0033] where Friend discloses that the inertial measurement unit (IMU) 170 typically comprises three orthogonally oriented accelerometers for detecting changes in position, three orthogonally oriented gyroscopes for detecting changes in orientation, and, in some cases, three orthogonally oriented magnetometers serving as a three-dimensional electronic compass),
when the single tracking apparatus tracks two reference point locations with respect to the single tracking apparatus, the at least one sensor acquires at least one of: inclination or heading of one of: the tracking apparatus or the machine (In paragraphs [0023-0025], Friend discloses that perception device 120 and processor 130 identifies the presence of at least one fiducial 190a in the field of view and determines a relative distance and orientation to the fiducial 190a (marker point) and/or loading machine 180 (machine); in paragraph [0046], Friend discloses that processor 130 may be configured to obtain information from fiducial 631a (reference point) via perception device 120 for determining a position of mining truck 300 (tracking apparatus); in paragraph [0033], Friend discloses that processor 130 may be configured to use information obtained from IMU 170 for determining changes in position and orientation of haul machine 110 between determinations of position and orientation using information obtained from perception device 120, which may be able at a much lower rate than the IMU information; see also paragraph [0033] where Friend discloses that the inertial measurement unit (IMU) 170 typically comprises three orthogonally oriented accelerometers for detecting changes in position, three orthogonally oriented gyroscopes for detecting changes in orientation, and, in some cases, three orthogonally oriented magnetometers serving as a three-dimensional electronic compass),
receiving by at least one position determination unit
data acquired by the at least one sensor (In paragraph [0033], Friend discloses that processor 130 may be configured to use information obtained from IMU 170 for determining changes in position and orientation of haul machine 110 between determinations of position and orientation using information obtained from perception device 120, which may be able at a much lower rate than the IMU information),
data related to the single tracking apparatus comprising at least the data acquired by the single tracking apparatus, wherein in the data acquired by the single tracking apparatus the tracked locations with respect to the single tracking apparatus consist of direction and distance from the single tracking apparatus to each of the tracked locations (In paragraphs [0023-0025], Friend discloses that perception device 120 and processor 130 identifies the presence of at least one fiducial 190a in the field of view and determines a relative distance and orientation to the fiducial 190a (marker point) and/or loading machine 180 (machine), where processor 130 (position determination unit) determines the location or position of a fiducial based on data from perception device 120; the examiner understands that the acquired data from the perception device 120 (of tracking apparatus) must at least be received by the processor 130 (position determination unit) to be used in the determination made by the processor 130); and
determining by the at least one position determination unit, based on the received data, the location and orientation of both the machine and a working tool of the machine in the worksite coordinate system (In paragraphs [0023-0025], Friend discloses that perception device 120 and processor 130 identifies the presence of at least one fiducial 190a in the field of view and determines a relative distance and orientation to the fiducial 190a (marker point) and/or loading machine 180 (machine); in paragraph [0046], Friend discloses that processor 130 may be configured to obtain information from fiducial 631a (reference point) via perception device 120 for determining a position of mining truck 300 (tracking apparatus); see also paragraph [0021], where Friend defines “position” to be a location also having a respective orientation for a machine);
executing designed operations by the working tool of the machine in the worksite based on the determined location and orientation of both the machine and the working tool of the machine in the worksite coordinate system (In paragraph [0040], Friend discloses that a common use pattern for excavator 400 is to place excavator 400 at a fixed position for a period of time while performing multiple sequences of digging with and filling bucket 420 and depositing the contents of the bucket 420 into a haul machine at a destination location for receiving materials from excavator 400; in paragraph [0056], Friend discloses, for example, that processor 130 assists with or automatically guides haul machine 110 toward the desired destination location or position; see also paragraphs [0059] and [0061], where Friend discloses that the desired destination position or location may be automatically identified or where processor 130 may be configured to automatically stop haul machine 110 if, via the information obtained from nearby fiducials or other sources, processor 130 anticipates haul machine 110 may collide with something, such as a loading machine associated with the desired destination location).
Friend does not explicitly disclose setting an environment modelling apparatus at least one of: on a machine, when the environment modelling apparatus has a location in a machine coordinate system, or external from the machine, when the environment modelling apparatus has a location in a worksite coordinate system, for at least detecting surrounding areas as spatial data of or relating to the worksite, wherein the spatial data is at least surface profile of a ground;
tracking at same time instants both at least one reference point and at least three marker points;
wherein the acquiring data by the single tracking apparatus of instants reference point and marker point locations with respect to the single tracking apparatus is at same time instants;
acquiring data by the environment modelling apparatus by tracking with respect to the environment modelling apparatus spatial data of or relating to the worksite, wherein the spatial data comprises a surface profile of a ground;
wherein the acquiring by the at least one sensor of inclination and heading data of one of: the tracking apparatus or the machine is at same time instant;
receiving by the at least one position determination unit data related to the environment modelling apparatus comprising at least the data acquired by the environment modelling apparatus; and
providing the tracked surface profile of the ground as as-built data as determined situational awareness in the worksite.
However, Yao teaches tracking at same time instants both at least one reference point and at least three marker points (In paragraph [0047], Yao discloses that a current state, s, includes not only a vehicle's [tracking apparatus] state but also includes the environment's state (e.g., measurement of the vehicle 105 with respective to the environment) at the same time, t, and for example the state, s, at time, t, includes: sensor data including current camera views, including: current images of all the cameras 112 installed on the vehicle 105 and other sensory measurements such as current GNSS data from the satellite receiver 132, current compass reading, current IMU reading, etc.; data derived from current and/or past sensor data including: current distance to other environmental references, etc.);
wherein the acquiring data by the single tracking apparatus of instants reference point and marker point locations with respect to the single tracking apparatus is at same time instants (In paragraph [0047], Yao discloses that a current state, s, includes not only a vehicle's [tracking apparatus] state but also includes the environment's state (e.g., measurement of the vehicle 105 with respective to the environment) at the same time, t, and for example the state, s, at time, t, includes: sensor data including current camera views, including: current images of all the cameras 112 installed on the vehicle 105 and other sensory measurements such as current GNSS data from the satellite receiver 132, current compass reading, current IMU reading, etc.; data derived from current and/or past sensor data including: current distance to other environmental references, etc.); and
wherein the acquiring by the at least one sensor of inclination and heading data of one of: the tracking apparatus or the machine is at same time instant (In paragraph [0047], Yao discloses that a current state, s, includes not only a vehicle's [tracking apparatus] state but also includes the environment's state (e.g., measurement of the vehicle 105 with respective to the environment) at the same time, t, and for example the state, s, at time, t, includes: sensor data including current camera views, including: current images of all the cameras 112 installed on the vehicle 105 and other sensory measurements such as current GNSS data from the satellite receiver 132, current compass reading, current IMU reading, etc.; data derived from current and/or past sensor data including: current distance to other environmental references, etc.).
Yao is considered to be analogous to the claimed invention in that they both pertain to obtaining sensor data at a same time instant to predict the state of an apparatus and its environment at the same time instant. It would be obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to implement the teachings of Yao with the method as disclosed by Friend, wherein obtaining the sensor data at the same time instant as taught by Yao is advantageous in that it ensures the sensor data being used to determine the position and orientation are obtained simultaneously, maintaining the relative accuracy of the measurements to each other, for example, thereby increasing the accuracy of the determinations which utilize the measurements.
The combination of Friend and Yao does not explicitly disclose setting an environment modelling apparatus at least one of: on a machine, when the environment modelling apparatus has a location in a machine coordinate system, or external from the machine, when the environment modelling apparatus has a location in a worksite coordinate system, for at least detecting surrounding areas as spatial data of or relating to the worksite, wherein the spatial data is at least surface profile of a ground;
acquiring data by the environment modelling apparatus by tracking with respect to the environment modelling apparatus spatial data of or relating to the worksite, wherein the spatial data comprises a surface profile of a ground;
receiving by the at least one position determination unit data related to the environment modelling apparatus comprising at least the data acquired by the environment modelling apparatus; and
providing the tracked surface profile of the ground as as-built data as determined situational awareness in the worksite.
However, Clark teaches setting an environment modelling apparatus at least one of: on a machine, when the environment modelling apparatus has a location in a machine coordinate system, or external from the machine, when the environment modelling apparatus has a location in a worksite coordinate system, for at least detecting surrounding areas as spatial data of or relating to the worksite, wherein the spatial data is at least surface profile of a ground (In paragraphs [0018-0019], Clark teaches update and mapping a three dimensional worksite surface dynamically from the earthmoving machine while it is being operated over a worksite, wherein to derive a dynamic, three-dimensional terrain model, the system includes a pair of scanners 12 and 14 that are mounted on the machine 16);
acquiring data by the environment modelling apparatus by tracking with respect to the environment modelling apparatus spatial data of or relating to the worksite, wherein the spatial data comprises a surface profile of a ground (In paragraphs [0018-0019], Clark teaches update and mapping a three dimensional worksite surface dynamically from the earthmoving machine while it is being operated over a worksite, wherein to derive a dynamic, three-dimensional terrain model, the system includes a pair of scanners 12 and 14 that are mounted on the machine 16);
receiving by the at least one position determination unit data related to the environment modelling apparatus comprising at least the data acquired by the environment modelling apparatus (In paragraphs [0018-0019], Clark teaches update and mapping a three dimensional worksite surface dynamically from the earthmoving machine while it is being operated over a worksite, wherein to derive a dynamic, three-dimensional terrain model, the system includes a pair of scanners 12 and 14 that are mounted on the machine 16); and
providing the tracked surface profile of the ground as as-built data as determined situational awareness in the worksite (In paragraphs [0024-0025], Clark teaches that a worksite surface behind the machine is scanned at 52 and the three-dimensional data is stored at 56; see also paragraphs [0021-0022] where Clark teaches that the terrain model may also be displayed to the operator of the machine 16 on display 30).
Clark is considered to be analogous to the claimed invention in that they both pertain to environment modelling including obtaining spatial data that is a surface profile of the ground. It would be obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to implement the teachings of Clark with the method as disclosed by the combination of Friend and Yao, where the ability to provide real-time data of the machine gives a redundant check and validation of three dimensional machine control system performance, and where using this data in real time gives the machine control system the ability to fine tune the automatic controls to achieve required tolerances in the generated surface, as suggested by Clark in paragraph [0018], which may advantageously increase the contextual accuracy of operation of the machine, for example.
Regarding claim 3, Friend further discloses determining at least one of: accuracy level or validity of the determined location and orientation of the machine in the worksite (In paragraph [0032], Friend discloses that processor 130 may be configured to base its location determination on location information received from geolocation unit 160, although processor 130 may be configured to not use location information received from geolocation unit 160 if its precision or accuracy fail to meet predetermined requirements).
Regarding claim 4, Friend further discloses wherein the data related to the single tracking apparatus further comprises at least one of: inclination angle of the tracking apparatus, heading of the tracking apparatus, us, location and orientation of the tracking apparatus in at least one of: a machine coordinate system, the worksite coordinate system or a world coordinate system or at least one of: accuracy level or validity of at least one of the previous (In paragraph [0021], Friend discloses that a "position," in this disclosure, is a location also having a respective orientation for a machine; in paragraph [0032], Friend discloses that processor 130 may be configured to base its location determination on location information received from geolocation unit 160, although processor 130 may be configured to not use location information received from geolocation unit 160 if its precision or accuracy fail to meet predetermined requirements).
Regarding claim 5, Clark further teaches wherein the data related to the at least one environment modelling apparatus is further at least one of: inclination angle of the environment modelling apparatus, heading of the environment modelling apparatus, location and orientation of the environment modelling apparatus in at least one of: a machine coordinate system, the worksite coordinate system or a world coordinate system (In paragraph [0009], Clark teaches that The scanning arrangement may include one or more GNSS receivers carried on machine for determining the position and orientation of the machine, a laser transmitter which projects a reference beam of laser light, and one or more laser light detectors on the machine to detect the reference beam of light wherein the position of the one or more laser light detectors may be determined with respect to the laser transmitter and the position and orientation of the pair of scanners may be determined, an inertial guidance system for determining the position and orientation of the machine including the position and orientation of the pair of scanners, or one or more inclinometers for determining the orientation of the machine).
Regarding claim 6, Friend further discloses wherein the spatial data comprises at least one of: pictorial data, point cloud data or data with implicit or explicit reference to a location relative to at least one of: the worksite or the Earth (In paragraphs [0023-0025], Friend discloses that perception device 120 and processor 130 (of haul machine 110) identifies the presence of at least one fiducial 190a in the field of view and determines a relative distance and orientation to the fiducial 190a (marker point) and/or loading machine 180 (machine); in paragraph [0046], Friend discloses fiducials 631a and 631b (reference points) indicating a reference location 630 in a mining site 600 (worksite); see also paragraph [0032] where Friend discloses a geolocation unit 160, from which processor 130 may obtain an absolute position, typically including latitude, longitude, and altitude, of haul machine 110 on the surface of the Earth).
Regarding claim 8, Clark further teaches receiving by the at least one environment modelling unit
one or more indications relating to at least one of: work phase or work stage of respective areas (In paragraphs [0024-0025], Clark teaches that a worksite surface behind the machine is scanned at 52 and the three-dimensional data is stored at 56, wherein a desired surface contour is compared at 62 with the actual worksite surface contour behind the machine, and those areas of the worksite surface that require further work are noted and stored at 64), and
data related to the at least one environment modelling apparatus (In paragraphs [0024-0025], Clark teaches that a worksite surface behind the machine is scanned at 52 and the three-dimensional data is stored at 56), and wherein
by taking into account the one or more indications
it is derived from the data related to the at least one environment modelling apparatus a georeferenced spatial data of respective areas (In paragraphs [0024-0025], Clark teaches that a worksite surface behind the machine is scanned at 52 and the three-dimensional data is stored at 56, wherein a desired surface contour is compared at 62 with the actual worksite surface contour behind the machine, and those areas of the worksite surface that require further work are noted and stored at 64; see also paragraph [0026] where Clark teaches that GPS/GNSS receivers may be used on the machine 16 to determine machine position and orientation); and
saved at least in part the georeferenced spatial data (In paragraphs [0024-0025], Clark teaches that a worksite surface behind the machine is scanned at 52 and the three-dimensional data is stored at 56, wherein a desired surface contour is compared at 62 with the actual worksite surface contour behind the machine, and those areas of the worksite surface that require further work are noted and stored at 64).
Regarding claim 9, Clark further teaches wherein the step of saving at least in part the georeferenced spatial data further comprises:
determining, based at least in part on the data received from the at least one environment modelling apparatus, the areas the georeferenced spatial data of which is to be saved (In paragraphs [0024-0025], Clark teaches that a worksite surface behind the machine is scanned at 52 and the three-dimensional data is stored at 56, wherein a desired surface contour is compared at 62 with the actual worksite surface contour behind the machine, and those areas of the worksite surface that require further work are noted and stored at 64); and
saving the georeferenced spatial data of the areas determined to be saved (In paragraphs [0024-0025], Clark teaches that a worksite surface behind the machine is scanned at 52 and the three-dimensional data is stored at 56, wherein a desired surface contour is compared at 62 with the actual worksite surface contour behind the machine, and those areas of the worksite surface that require further work are noted and stored at 64).
Regarding claim 11, Friend further discloses resolving, by the at least one position determination unit which determined the location and orientation of the machine and the working tool of the machine in the worksite coordinate system, data regarding at least one of: a tracking apparatus, an environment modelling apparatus, an object or another machine (In paragraph [0025], Friend discloses that processor 130 (position determination unit) determines the location or position of a fiducial based on data from perception device 120; the examiner understands that the acquired data from the perception device 120 (of tracking apparatus) must at least be received by the processor 130 (position determination unit) to be used in the determination made by the processor 130); and
transmitting the data resolved at least one of:
as part of data related to the respective tracking apparatus, environment modelling apparatus, object or another machine (In paragraph [0022], Friend discloses fiducials 190a and 190b (marker points) on loading machine 180 (machine); in paragraph [0023], Friend discloses a haul machine 110 (tracking apparatus) including a perception device 120 and processor 130; in paragraph [0046], Friend discloses fiducials 631a and 631b (reference points) indicating a reference location 630 in a mining site 600 (worksite)); or
as data receivable by at least one position determination unit (In paragraph [0025], Friend discloses that processor 130 (position determination unit) determines the location or position of a fiducial based on data from perception device 120; the examiner understands that the acquired data from the perception device 120 (of tracking apparatus) must at least be received by the processor 130 (position determination unit) to be used in the determination made by the processor 130).
Regarding claim 12, Friend further discloses initializing the single tracking apparatus by
determining location and orientation of the tracking apparatus in at least one of: a machine coordinate system or the worksite coordinate system if the tracking apparatus is set on at least one of: another machine (In paragraphs [0023-0025], Friend discloses that perception device 120 and processor 130 (of haul machine 110) identifies the presence of at least one fiducial 190a in the field of view and determines a relative distance and orientation to the fiducial 190a (marker point) and/or loading machine 180 (machine)); and
determining location and orientation of the tracking apparatus in the worksite coordinate system if the tracking apparatus is set external from the machine (In paragraphs [0023-0025], Friend discloses that perception device 120 and processor 130 (of haul machine 110) identifies the presence of at least one fiducial 190a in the field of view and determines a relative distance and orientation to the fiducial 190a (marker point) and/or loading machine 180 (machine)).
Regarding claim 13, Clark further teaches initializing the environment modelling apparatus by
determining location and orientation of the environment modelling apparatus in at least one of: a machine coordinate system or the worksite coordinate system if the environment modelling apparatus is set on at least one of: the machine or another machine (In paragraph [0009], Clark teaches that The scanning arrangement may include one or more GNSS receivers carried on machine for determining the position and orientation of the machine, a laser transmitter which projects a reference beam of laser light, and one or more laser light detectors on the machine to detect the reference beam of light wherein the position of the one or more laser light detectors may be determined with respect to the laser transmitter and the position and orientation of the pair of scanners may be determined, an inertial guidance system for determining the position and orientation of the machine including the position and orientation of the pair of scanners, or one or more inclinometers for determining the orientation of the machine), and
determining the location and orientation of the environment modelling apparatus in the worksite coordinate system if the environment modelling apparatus is set external from the machine or another machine (In paragraph [0009], Clark teaches that The scanning arrangement may include one or more GNSS receivers carried on machine for determining the position and orientation of the machine, a laser transmitter which projects a reference beam of laser light, and one or more laser light detectors on the machine to detect the reference beam of light wherein the position of the one or more laser light detectors may be determined with respect to the laser transmitter and the position and orientation of the pair of scanners may be determined, an inertial guidance system for determining the position and orientation of the machine including the position and orientation of the pair of scanners, or one or more inclinometers for determining the orientation of the machine).
Regarding claim 14, Clark further teaches wherein the determined situational awareness is further at least one of spatial data (In paragraphs [0018-0019], Clark teaches update and mapping a three dimensional worksite surface dynamically from the earthmoving machine while it is being operated over a worksite, wherein to derive a dynamic, three-dimensional terrain model, the system includes a pair of scanners 12 and 14 that are mounted on the machine 16).
Regarding claim 17, Friend further discloses wherein the data related to at least one of: the single tracking apparatus further comprises at least one of: data from sensors installed on the respective apparatus, data from sensors installed on an attachment point of the apparatus, data resolved by at least one of: any position determination unit or any apparatus at least one of: by tracking the respective apparatus or as a result of any calculations relating to the respective apparatus or at least one of: accuracy level or validity of at least one of the previous (In paragraphs [0023-0025], Friend discloses that perception device 120 such as a camera, LIDAR, or hybrid system, and processor 130 (of haul machine 110) identifies the presence of at least one fiducial 190a in the field of view and determines a relative distance and orientation to the fiducial 190a (marker point) and/or loading machine 180 (machine); in paragraph [0032], Friend discloses that processor 130 may be configured to base its location determination on location information received from geolocation unit 160, although processor 130 may be configured to not use location information received from geolocation unit 160 if its precision or accuracy fail to meet predetermined requirements).
Regarding claim 18, Friend further discloses wherein
if the single tracking apparatus is set on the worksite and if it comprises a tracking device for tracking the location of the tracking apparatus with one or more GNSS antennas (In paragraph [0032], Friend discloses a geolocation unit 160, from which processor 130 may obtain an absolute position, typically including latitude, longitude, and altitude, of haul machine 110 on the surface of the Earth such as GPS),
the single tracking apparatus further comprises at least one of: a camera, a stereocamera, or a lidar as a tracking device (In paragraphs [0023-0025], Friend discloses that perception device 120 such as a camera, LIDAR, or hybrid system).
Regarding claim 19, Friend further discloses wherein determination, by the at least one position determination unit, of the location and orientation of the machine in the worksite is additionally based at least in part on data received from one or more sensors installed on at least one of: another machine; wherein the sensors comprise at least one of: position, orientation, inclination, heading or distance travelled of at least one of: the machine or another machine (In paragraph [0032], Friend discloses a geolocation unit 160, from which processor 130 may obtain an absolute position, typically including latitude, longitude, and altitude, of haul machine 110 on the surface of the Earth such as GPS; in paragraphs [0023-0025], Friend discloses that perception device 120 such as a camera, LIDAR, or hybrid system).
Claims 2 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Friend (US 2014/0146167 A1), Yao (US 2019/0102668 A1), and Clark (US 2010/0046800 A1), in view of Kamat (US 2015/0168136 A1).
Regarding claim 2, although in paragraph [0021], Friend discloses that a "position," in this disclosure, is a location also having a respective orientation for a machine, the combination of Friend, Yao, and Clark does not explicitly disclose determining by the at least one position determination unit, based at least in part on the received data, at least one of: direction of travel or alternative direction of travel of the machine in the worksite.
However, Kamat teaches determining by the at least one position determination unit, based at least in part on the received data, at least one of: direction of travel or alternative direction of travel of the machine in the worksite (In paragraph [0039], Kamat teaches mathematical representations of 3D poses of the excavator's 10 different articulated components including matrices for position, yaw, pitch, and roll for the crawlers 19 of the excavator 10; in paragraph [0025], Kamat teaches that the base 12 of the excavator 10 moves forward and backward via its crawlers 19).
Kamat is considered to be analogous to the claimed invention in that they both pertain to determining a direction of travel for the machine. It would be obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to implement the teachings of Kamat with the method as disclosed by the combination of Friend, Yao, and Clark, where doing so may be advantageous in that the 3D pose of the particular articulated components of the machine can be understood relative to each other as well as to the coordinate system, improving the contextual accuracy of the pose determination for example.
Regarding claim 20, Friend further discloses wherein the machine is an excavator (In paragraph [0022], Friend discloses that examples of loading machine 180 include, but are not limited to, excavators).
The combination of Friend, Yao, and Clark does not explicitly disclose wherein determination, by the at least one position determination unit, of the location and orientation of the machine and the working tool of the machine in the worksite coordinate system is additionally based at least in part on data received from one or more sensors installed on the upper carriage of at least one of: the machine or another machine, wherein
the sensors comprise at least one of: position, orientation, inclination or heading of the upper carriage of at least one of: the machine or another machine.
However, Kamat teaches wherein the machine is an excavator (In paragraphs [0024-0025], Kamat teaches estimating a 3D pose of an articulated machine such as an excavator 10); and wherein
determination, by the at least one position determination unit, of the location and orientation of the machine in the worksite is additionally based at least in part on data received from one or more sensors installed on the upper carriage of at least one of: the machine or another machine (In paragraphs [0028-0030], Kamat teaches estimating 3D pose of a marker 22 attached to a planar surface of the excavator 10 by determining the 3D pose of a camera 26 mounted to a roof of a cabin 20 of the excavator 10), wherein
the sensors comprise at least one of: position, orientation, inclination or heading of the upper carriage of at least one of: the machine or another machine (In paragraph [0028], Kamat teaches that the camera 26 is aimed at the marker 22 so that the camera can take images of the marker as the marker moves up and down and fore and aft with the stick 16 and bucket 18 relative to the cabin 20; see also paragraph [0023], where Kamat teaches that the 3D pose comprises a three-dimensional (3D) position and orientation of articulated components of the articulated machine).
Kamat is considered to be analogous to the claimed invention in that they both pertain to determining the pose of an upper component of an excavator by a sensor installed on the upper component. It would be obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to implement the teachings of Kamat with the method as disclosed by the combination of Friend, Yao, and Clark, where in paragraph [0022], Friend discloses that examples of loading machine 180 include, but are not limited to, excavators. Implementing the camera mounted on a roof of a cabin of an excavator as taught by Kamat may be advantageous in that the 3D pose of the particular articulated components of the machine can be understood relative to each other as well as to the coordinate system, improving the contextual accuracy of the pose determination for example.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Friend (US 2014/0146167 A1), Yao (US 2019/0102668 A1), and Clark (US 2010/0046800 A1), in view of Berry (US 2018/0142441 A1).
The combination of Friend, Yao, and Clark does not explicitly disclose receiving by at least one environment modelling unit
an indication of a material delivery,
material delivery base determined by first data related to at least one environment modelling apparatus covering an area of material to be placed, and
material delivery complete determined by second data related to at least one environment modelling apparatus covering the area of material to be placed; and
saving by the at least one environment modelling unit at least in part the data regarding the indication of the material delivery, the material delivery base and the material delivery complete as a material delivered.
However, Berry teaches receiving by at least one environment modelling unit (In paragraphs [0029-0030], Berry teaches a perception system 120 mounted on the machine 102, or alternatively mounted remotely from the machine 102 but in data communication with the machine 102, configured to determine the locations of one or more points on the machine 102 relative to points on the material receptacle 106, angular alignment of a plane defined by the machine 102 relative to a plane defined by the material receptacle 106, or combinations thereof)
an indication of a material delivery (Starting in paragraph [0052], Berry teaches a method 400 or transferring a material 108 from a material source to a material receptacle 106 using a material-handling machine 102),
material delivery base determined by first data related to at least one environment modelling apparatus covering an area of material to be placed (In paragraphs [0053-0055], Berry teaches that the machine 102 is located in a staging area in preparation for unloading material 108 from its implement 130 into a material receptacle 106, wherein the staging area may be an area that locates the material receptacle 106 within the zone of detection 208 of the perception system 120, and wherein the controller 196 may proceed to locate features of the material receptacle 106 relative to the implement 130 or other portion of the machine 102 based on input from the perception system 120), and
material delivery complete determined by second data related to at least one environment modelling apparatus covering the area of material to be placed (In paragraph [0059], Berry teaches wherein the controller 196 determines whether the material receptacle 106 is full); and
saving by the at least one environment modelling unit at least in part the data regarding the indication of the material delivery, the material delivery base and the material delivery complete as a material delivered (In paragraph [0036], Berry teaches that the controller 196 may include power electronics, preprogrammed logic circuits, data processing circuits, volatile memory, non-volatile memory, software, firmware, input/output processing circuits, combinations thereof, or any other controller structures known in the art; the Examiner understands that the data utilized by the controller 196 as expressed above must at least be “saved” under its broadest reasonable interpretation to a memory of the controller in order to be utilized for the determinations of the controller as disclosed).
Berry is considered to be analogous to the claimed invention in that they both pertain to determining a material delivery area and completion in order to control a machine. It would be obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to implement the teachings of Berry with the method as disclosed by the combination of Friend, Yao, and Clark, where doing so may advantageously improve coordination of contextual movement of the machine and machine implement relative to the worksite for example, as suggested by Berry in paragraph [0004].
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Friend (US 2014/0146167 A1), Yao (US 2019/0102668 A1), and Clark (US 2010/0046800 A1), in view of Moroniti (US 2019/0332114 A1) and Moteki (US 2019/0257659 A1).
The combination of Friend, Yao, and Clark does not explicitly disclose wherein the step of determining, based at least in part on the data received from the environment modelling apparatus, the areas the georeferenced spatial data of which are to be saved, further comprises
detecting the areas where the georeferenced spatial data of the environment modelling apparatus is obstacle-free and regarding the obstacle-free areas, the method further comprising:
comparing current accuracy of the determined location and orientation of the machine in the worksite with the accuracy of the determined location and orientation of the machine in the worksite in time of the previously saved georeferenced spatial data; and
whether the current accuracy is above a threshold, updating the saved georeferenced spatial data in the obstacle-free areas.
However, Moroniti teaches wherein the step of determining, based at least in part on the data received from the at least one environment modelling apparatus, the areas the georeferenced spatial data of which are to be saved, further comprises
detecting the areas where the georeferenced spatial data of the at least one environment modelling apparatus is obstacle-free and regarding the obstacle-free areas (In paragraph [0065], Moroniti teaches that the contextual layer of a map may be updated, wherein the robot can determine that portions of the area designated as keepout regions are in fact obstacle-free, and can update the map or can inform a remote computing system (such as a building server) that the portions should be re-classified; see also paragraph [0067] where Moroniti teaches that the robot utilizes high-accuracy sensors like a LIDAR sensor), the method further comprising:
updating the saved georeferenced spatial data in the obstacle-free areas (In paragraph [0065], Moroniti teaches that the contextual layer of a map may be updated, wherein the robot can determine that portions of the area designated as keepout regions are in fact obstacle-free, and can update the map or can inform a remote computing system (such as a building server) that the portions should be re-classified; see also paragraph [0067] where Moroniti teaches that the robot utilizes high-accuracy sensors like a LIDAR sensor).
Moroniti is considered to be analogous to the claimed invention in that they both pertain to updating data representing the environment of a machine by detecting areas that are obstacle free. It would be obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to implement the teachings of Morniti with the method as disclosed by the combination of Friend, Yao, and Clark, where doing so may provide the other robots with the most up-to-date contextual maps available as suggested by Moroniti in paragraph [0067] advantageously improving accuracy of operation of the machines, for example.
The combination of Friend, Yao, Clark, and Morotini does not explicitly disclose the method further comprising:
comparing current accuracy of the determined location and orientation of the machine in the worksite with the accuracy of the determined location and orientation of the machine in the worksite in time of the previously saved georeferenced spatial data; and
whether the current accuracy is above a threshold, updating the saved georeferenced spatial data in the obstacle-free areas.
However, Moteki teaches the method further comprising:
comparing current accuracy of the determined location and orientation of the machine in the worksite with the accuracy of the determined location and orientation of the machine in the worksite in time of the previously saved georeferenced spatial data (In paragraphs [0131-0147], Moteki teaches reidentifying each location of a vehicle represented in the local map according to a location of the vehicle when a key frame image of a local map transmitted from the in-vehicle device 10 is captured, determining whether or not the reliability of the local map calculated is larger than a threshold, wherein if the reliability of the local map is larger than the threshold, the correction unit 36 corrects the location and posture of the vehicle in the local map based on the local map output and each location of the vehicle reidentified wherein the data management device 230 updates the overall map by using the local map according to the reliability, and if the reliability of the local map is equal to or less than the threshold, the processing ends); and
whether the current accuracy is above a threshold, updating the saved georeferenced spatial data in the obstacle-free areas (In paragraphs [0131-0147], Moteki teaches reidentifying each location of a vehicle represented in the local map according to a location of the vehicle when a key frame image of a local map transmitted from the in-vehicle device 10 is captured, determining whether or not the reliability of the local map calculated is larger than a threshold, wherein if the reliability of the local map is larger than the threshold, the correction unit 36 corrects the location and posture of the vehicle in the local map based on the local map output and each location of the vehicle reidentified wherein the data management device 230 updates the overall map by using the local map according to the reliability, and if the reliability of the local map is equal to or less than the threshold, the processing ends).
Moteki is considered to be analogous to the claimed invention in that they both pertain to utilizing data describing the environment of a vehicle based on a comparison of the accuracy of the localization of the vehicle. It would be obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to implement the teachings of Moteki with the method as disclosed by the combination of Friend, Yao, Clark, and Morotini where accuracy of the data may be advantageously curated by ensuring a certain level of accuracy is achieved, for example.
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable Friend (US 2014/0146167 A1), Yao (US 2019/0102668 A1), and Clark (US 2010/0046800 A1), in view of Fasola (US 2020/0408557 A1).
The combination of Friend, Yao, and Clark does not explicitly disclose determining a minimum level of accuracy of the determined location and orientation of the machine in the worksite; and
determining a threshold level above the minimum level of accuracy, and
disabling the controls for moving an under carriage of the machine if the level of accuracy falls below the threshold level and if the work task in progress may be carried out without moving the under carriage.
However, Fasola teaches determining a minimum level of accuracy of the determined location and orientation of the machine in the worksite (In paragraph [0080] for example, Fasola teaches where if there is a jump (512) in localization over a threshold value, indicating that the location estimate has shifted quickly, then the vehicle can be brought to a safe stop (514); the Examiner understands the minimum level of accuracy to correspond to a jump in localization less than the threshold value, where the accuracy is lower than the threshold accuracy when a jump in localization is at the threshold value);
determining a threshold level above the minimum level of accuracy (In paragraph [0080] for example, Fasola teaches where if there is a jump (512) in localization over a threshold value, indicating that the location estimate has shifted quickly, then the vehicle can be brought to a safe stop (514); the Examiner understands the threshold level of accuracy to correspond to a jump in localization at the threshold value); and
disabling controls for moving an under carriage of the machine if the level of accuracy falls below the threshold level and if the work task in progress may be carried out without moving the under carriage (In paragraph [0080] for example, Fasola teaches where if there is a jump (512) in localization over a threshold value, indicating that the location estimate has shifted quickly, then the vehicle can be brought to a safe stop (514); the Examiner understands that if the level of accuracy falls below the threshold level, bringing the vehicle to a safe stop becomes the work task in progress under its broadest reasonable interpretation, which may be carried out without moving the under carriage).
Fasola is considered to be analogous to the claimed invention in that they both pertain to stopping control of the motive element of a vehicle when an accuracy of localization of the vehicle falls below a threshold level. It would be obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to implement the teachings of Fasola with the method as disclosed by the combination of Friend, Yao, and Clark, where stopping control of the motive element of a vehicle when an accuracy of localization of the vehicle falls below a threshold level may advantageously increase safety, for example, by ensuring that the motive element is only controlled when a certain level of accuracy can be achieved.
Allowable Subject Matter
Claim 15 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Harrison Heflin whose telephone number is (571)272-5629. The examiner can normally be reached Monday - Friday, 1:00PM - 10:00PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hunter Lonsberry can be reached at 571-272-7298. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HARRISON HEFLIN/Examiner, Art Unit 3665
/HUNTER B LONSBERRY/Supervisory Patent Examiner, Art Unit 3665
Prosecution Insights