Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 7, 9, 11-13, 17, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Ellaboudy, et al., hereinafter Ellaboudy (U.S. Patent Application Pub. No. 2021/0000006) in view of Davis (U.S. Patent Application Pub. No. 2024/0142986).
Regarding Claim 1, Ellaboudy teaches: A method (Ellaboudy, Abstract and Para. 0084 – “systems and methods for agricultural lane following” including a process for “automatically controlling a vehicle”) comprising:
receiving map data about a feature of an operational environment in which an autonomous vehicle operates (Ellaboudy, Para. 0058, 0084-0087, and 0107 – where a process for “automatically controlling a vehicle” includes accessing “a map data structure storing a map representing locations of physical objects in a geographic area”; where the map may include “permanent obstacles (e.g. barns, houses, or fences)”, or features);
obtaining sensor data about a recurrent object in the operational environment (Ellaboudy, Fig. 32 and Para. 0105-0108, 0118, 0228 – detecting “environmental landmarks for vehicle navigation” by “a lidar sensor” used to capture “point cloud” data, where for example the landmarks are a “crop row” made of “trees along traveling direction”, such that the crop/trees are recurrent; Fig. 32 shows a visual representation of crop rows made of recurring crops);
“a map data structure storing a map representing locations of physical objects in a geographic area” including “permanent obstacles (e.g. barns, houses, or fences)” and “environmental landmarks”, such as a “crop row” made of “trees along traveling direction”, such that the crop/trees are recurrent);
and causing the autonomous vehicle to navigate the operational environment using the map (Ellaboudy, Para. 0081 and 0100-0103 – generating a “path data structure” based on the “map” and “boundary” data, where boundary data specifies “an area within the map”; where the vehicle may execute a “path following mode” to steer to waypoints of a path “autonomously”).
PNG
media_image1.png
1088
1083
media_image1.png
Greyscale
Ellaboudy, Fig. 32
While Ellaboudy teaches comparing map data using the sensor data to be about the feature of the operational environment and the recurrent object in the operational environment (Ellaboudy, Para. 0109-0115 – collecting “current point cloud data captured using a distance sensor connected to a vehicle”, accessing “a map data structure storing a map representing locations of physical objects in a geographic area”, and “matching” and comparing the point cloud data with the map data to determine “a difference between the current point cloud and the expected point cloud of a candidate state”), Ellaboudy does not teach augmenting the map data to be about the feature of the operational environment and the recurrent object in the operational environment by adding at least a portion of the sensor data to the map data, nor does Ellaboudy teach generating, based on the augmented map data, a map indicating a location of the feature and the recurrent object.
However, Davis teaches augmenting the map data to be about the feature of the operational environment and the recurrent object in the operational environment by adding at least a portion of the sensor data to the map data (Davis, Para. 0051, 0061, 0064 – “map generator 312 can dynamically update the worksite map based on additional sensor data that is collected, such as additional sensor data that is collected as the mobile machine 100 travels in non-plants of interest areas and/or additional sensor data that is collected as the mobile machine travels or operates in the plants of interest area”; where “sensor data” may indicate “the size of the plants of interest area, locations of plants of interest, locations of plants of interest rows” , or recurrent objects, and “the locations of non-plant objects (e.g., fences or fence posts, telephone or power line poles,… etc.”), and generating, based on the augmented map data, a map indicating a location of the feature and the recurrent object (Davis, Para. 0051, 0068-0070, 0094-0095 – “map generator 312 can generate an updated (or revised) worksite map (e.g., 470) as the machine travels or operates at the worksite”, the “updated (or revised) worksite map” including an updated “plant map layer 442, non-plant map layer 444, and plant and non-plant map layer 446”, the plant map layer indicating “the location of the plants of interest area”, the non-plant map layer indicating “non-plant object(s)”, the “plant and non-plant map layer” indicating “locations of plants of interest rows at the worksite, locations of the spaces between the plants of interest rows at the worksite”, etc.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Ellaboudy to include augmenting the map data to be about the feature of the operational environment and the recurrent object in the operational environment by adding at least a portion of the sensor data to the map data, and generating, based on the augmented map data, a map indicating a location of the feature and the recurrent object, as taught by Davis, in order to provide updated maps to a vehicle to improve routing and obstacle avoidance.
In regards to Claim 2, Ellaboudy in view of Davis teaches the method of Claim 1, and Ellaboudy further teaches wherein the map comprises at least one of a three-dimensional view of the recurrent object or a bird’s eye view of the operational environment (Ellaboudy, Para. 0087, 0097, 0111 – presenting a map “as an image reflecting a two-dimensional projection or slice (e.g., a birds-eye-view) of a three-dimensional map (e.g., a map including point cloud data)”, where the three-dimensional map includes “point cloud data representing the positions of objects (e.g., trees or other plants, furrows, buildings, fences, and/or shelves) located in the geographic area”, where the plants/trees are part of a “crop row”, for example “a row of trees”, “row of vines”, etc., such that the plants/trees are recurrent objects).
In regards to Claim 3, Ellaboudy in view of Davis teaches the method of Claim 1, and Ellaboudy further teaches further comprising obtaining navigational data about navigational factors of the autonomous vehicle within the operational environment (Ellaboudy, Para. 0139-0149 – the “autonomous vehicle” having “sensors” to “provide feedback about the vehicle state for use by the control system”, including “one or more control feedback sensors” for sensing “vehicle speed, engine speed, fuel levels, and engine health”, etc., “one or more orientation sensors” such as “Global Positioning System (GPS) sensors”, “accelerometers”, “gyroscopes”, “magnetometers”, “an inertial measurement unit”, etc. which sense navigational factors when operating in “agricultural environments”).
In regards to Claim 7, Ellaboudy in view of Davis teaches the method of Claim 1, and Ellaboudy further teaches wherein the recurrent object comprises a first recurrent object (Ellaboudy, Fig. 32 and Para. 0105-0108, 0118, 0195, 0228 – detecting landmarks such as a “crop row” made of “trees along traveling direction”, such that the crop/trees are recurrent; Fig. 32 shows a visual representation of crop rows made of recurring crops having a “left crop row” and a “right crop row”) and the causing the autonomous vehicle to navigate the operational environment using the map comprises: determining a path for the autonomous vehicle through the operational environment using the map (Ellaboudy, Para. 0081 and 0100-0103 – generating a “path data structure” based on the “map” and “boundary” data, where boundary data specifies “an area within the map”; where the vehicle may execute a “path following mode” to steer to waypoints of a path “autonomously”); determining a first distance between the autonomous vehicle and the first recurrent object and a second distance between the autonomous vehicle and a second recurrent object within the operational environment (Ellaboudy, Para. 0222 and 0228-0231 – fitting a line along a “left crop row” of first objects and a “right crop row” of second objects, and “determining the yaw and the lateral position based on the first line and the second line”); and responsive to the first distance differing from the second distance by a threshold distance, adjusting the path of the autonomous vehicle to center the autonomous vehicle between the first recurrent object and the second recurrent object (Ellaboudy, Para. 0220, 0222, and 0228-0231 – determining a “composite line 3254 (e.g., a center line for the lane) is determined based on the line 3250 and the line 3252”, corresponding to the left crop row and right crop row respectively, and determining “a yaw 3270 and a lateral position 3280 in relation to lane based on the composite line 3254”, or center line, where the “center line of the lane may be defined to facilitate maintaining a consistent distance between the vehicle and the nearest portion of the crop row to facilitate processing of the crop row with an agricultural implement”, such that when the vehicle is not centered, the distances are different).
PNG
media_image2.png
625
612
media_image2.png
Greyscale
Ellaboudy, Annotated Fig. 32
In regards to Claim 9, Ellaboudy in view of Davis teaches the method of Claim 1, and Ellaboudy further teaches wherein:
the recurrent object forms part of a plurality of recurrent objects (Ellaboudy, Fig. 32 and Para. 0228 – where crops, or the recurrent objects, form “crop rows”, such that there are a plurality of recurring crops);
each of the recurrent objects of the plurality of recurrent objects comprises a common feature (Ellaboudy, Fig. 32 and Para. 0228-0230 – where the vehicle is configured to input “sensor data” in a “neural network” to identify a specific crop; for example when detecting the “right crop row” in Fig. 32, the “neural network” may be trained to detect “significant features of the plants” such as “the trunks of mature almond trees”, where the trunks are the common feature);
and the plurality of recurrent objects are correlated with a crop growing in the operational environment (Ellaboudy, Para. 0059 and 0228-0230 – “crop rows”, which for example may be plants or trees such as “raspberry bushes” or “almond trees” in a “geographic area”).
PNG
media_image1.png
1088
1083
media_image1.png
Greyscale
Ellaboudy, Annotated Fig. 32
Regarding Claim 11, Ellaboudy teaches: One or more computer readable mediums configured to store instructions that when executed perform operations (Ellaboudy, Abstract and Para. 0009, 0084 – “non-transitory computer-readable storage medium storing executable instructions” that when executed perform “methods for agricultural lane following” including a process for “automatically controlling a vehicle”), the operations comprising:
receiving map data about a feature of an operational environment in which an autonomous vehicle operates (Ellaboudy, Para. 0058, 0084-0087, and 0107 – where a process for “automatically controlling a vehicle” includes accessing “a map data structure storing a map representing locations of physical objects in a geographic area”; where the map may include “permanent obstacles (e.g. barns, houses, or fences)”, or features);
obtaining sensor data about a recurrent object in the operational environment (Ellaboudy, Fig. 32 and Para. 0105-0108, 0118, 0228 – detecting “environmental landmarks for vehicle navigation” by “a lidar sensor” used to capture “point cloud” data, where for example the landmarks are a “crop row” made of “trees along traveling direction”, such that the crop/trees are recurrent; Fig. 32 shows a visual representation of crop rows made of recurring crops);
“a map data structure storing a map representing locations of physical objects in a geographic area” including “permanent obstacles (e.g. barns, houses, or fences)” and “environmental landmarks”, such as a “crop row” made of “trees along traveling direction”, such that the crop/trees are recurrent);
and causing the autonomous vehicle to navigate the operational environment using the map (Ellaboudy, Para. 0081 and 0100-0103 – generating a “path data structure” based on the “map” and “boundary” data, where boundary data specifies “an area within the map”; where the vehicle may execute a “path following mode” to steer to waypoints of a path “autonomously”).
PNG
media_image1.png
1088
1083
media_image1.png
Greyscale
Ellaboudy, Fig. 32
While Ellaboudy teaches comparing map data using the sensor data to be about the feature of the operational environment and the recurrent object in the operational environment (Ellaboudy, Para. 0109-0115 – collecting “current point cloud data captured using a distance sensor connected to a vehicle”, accessing “a map data structure storing a map representing locations of physical objects in a geographic area”, and “matching” and comparing the point cloud data with the map data to determine “a difference between the current point cloud and the expected point cloud of a candidate state”), Ellaboudy does not teach augmenting the map data to be about the feature of the operational environment and the recurrent object in the operational environment by adding at least a portion of the sensor data to the map data, nor does Ellaboudy teach generating, based on the augmented map data, a map indicating a location of the feature and the recurrent object.
However, Davis teaches augmenting the map data to be about the feature of the operational environment and the recurrent object in the operational environment by adding at least a portion of the sensor data to the map data (Davis, Para. 0051, 0061, 0064 – “map generator 312 can dynamically update the worksite map based on additional sensor data that is collected, such as additional sensor data that is collected as the mobile machine 100 travels in non-plants of interest areas and/or additional sensor data that is collected as the mobile machine travels or operates in the plants of interest area”; where “sensor data” may indicate “the size of the plants of interest area, locations of plants of interest, locations of plants of interest rows” , or recurrent objects, and “the locations of non-plant objects (e.g., fences or fence posts, telephone or power line poles,… etc.”), and generating, based on the augmented map data, a map indicating a location of the feature and the recurrent object (Davis, Para. 0051, 0068-0070, 0094-0095 – “map generator 312 can generate an updated (or revised) worksite map (e.g., 470) as the machine travels or operates at the worksite”, the “updated (or revised) worksite map” including an updated “plant map layer 442, non-plant map layer 444, and plant and non-plant map layer 446”, the plant map layer indicating “the location of the plants of interest area”, the non-plant map layer indicating “non-plant object(s)”, the “plant and non-plant map layer” indicating “locations of plants of interest rows at the worksite, locations of the spaces between the plants of interest rows at the worksite”, etc.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Ellaboudy to include augmenting the map data to be about the feature of the operational environment and the recurrent object in the operational environment by adding at least a portion of the sensor data to the map data, and generating, based on the augmented map data, a map indicating a location of the feature and the recurrent object, as taught by Davis, in order to provide updated maps to a vehicle to improve routing and obstacle avoidance.
In regards to Claim 12, Ellaboudy in view of Davis teaches the one or more computer readable mediums of Claim 11, and Ellaboudy further teaches wherein the map comprises at least one of a three-dimensional view of the recurrent object or a bird’s eye view of the operational environment (Ellaboudy, Para. 0087, 0097, 0111 – presenting a map “as an image reflecting a two-dimensional projection or slice (e.g., a birds-eye-view) of a three-dimensional map (e.g., a map including point cloud data)”, where the three-dimensional map includes “point cloud data representing the positions of objects (e.g., trees or other plants, furrows, buildings, fences, and/or shelves) located in the geographic area”, where the plants/trees are part of a “crop row”, for example “a row of trees”, “row of vines”, etc., such that the plants/trees are recurrent objects).
In regards to Claim 13, Ellaboudy in view of Davis teaches the method of Claim 11, and Ellaboudy further teaches the operations further comprise obtaining navigational data about navigational factors of the autonomous vehicle within the operational environment (Ellaboudy, Para. 0139-0149 – the “autonomous vehicle” having “sensors” to “provide feedback about the vehicle state for use by the control system”, including “one or more control feedback sensors” for sensing “vehicle speed, engine speed, fuel levels, and engine health”, etc., “one or more orientation sensors” such as “Global Positioning System (GPS) sensors”, “accelerometers”, “gyroscopes”, “magnetometers”, “an inertial measurement unit”, etc. which sense navigational factors when operating in “agricultural environments”).
In regards to Claim 17, Ellaboudy in view of Davis teaches the one or more computer readable mediums of Claim 11, and Ellaboudy further teaches wherein the recurrent object comprises a first recurrent object (Ellaboudy, Fig. 32 and Para. 0105-0108, 0118, 0195, 0228 – detecting landmarks such as a “crop row” made of “trees along traveling direction”, such that the crop/trees are recurrent; Fig. 32 shows a visual representation of crop rows made of recurring crops having a “left crop row” and a “right crop row”) and the causing the autonomous vehicle to navigate the operational environment using the map comprises: determining a path for the autonomous vehicle through the operational environment using the map (Ellaboudy, Para. 0081 and 0100-0103 – generating a “path data structure” based on the “map” and “boundary” data, where boundary data specifies “an area within the map”; where the vehicle may execute a “path following mode” to steer to waypoints of a path “autonomously”); determining a first distance between the autonomous vehicle and the first recurrent object and a second distance between the autonomous vehicle and a second recurrent object within the operational environment (Ellaboudy, Para. 0222 and 0228-0231 – fitting a line along a “left crop row” of first objects and a “right crop row” of second objects, and “determining the yaw and the lateral position based on the first line and the second line”); and responsive to the first distance differing from the second distance by a threshold distance, adjusting the path of the autonomous vehicle to center the autonomous vehicle between the first recurrent object and the second recurrent object (Ellaboudy, Para. 0220, 0222, and 0228-0231 – determining a “composite line 3254 (e.g., a center line for the lane) is determined based on the line 3250 and the line 3252”, corresponding to the left crop row and right crop row respectively, and determining “a yaw 3270 and a lateral position 3280 in relation to lane based on the composite line 3254”, or center line, where the “center line of the lane may be defined to facilitate maintaining a consistent distance between the vehicle and the nearest portion of the crop row to facilitate processing of the crop row with an agricultural implement”, such that when the vehicle is not centered, the distances are different).
PNG
media_image2.png
625
612
media_image2.png
Greyscale
Ellaboudy, Annotated Fig. 32
In regards to Claim 19, Ellaboudy in view of Davis teaches the one or more computer readable mediums of Claim 11, and Ellaboudy further teaches wherein:
the recurrent object forms part of a plurality of recurrent objects (Ellaboudy, Fig. 32 and Para. 0228 – where crops, or the recurrent objects, form “crop rows”, such that there are a plurality of recurring crops);
each of the recurrent objects of the plurality of recurrent objects comprises a common feature (Ellaboudy, Fig. 32 and Para. 0228-0230 – where the vehicle is configured to input “sensor data” in a “neural network” to identify a specific crop; for example when detecting the “right crop row” in Fig. 32, the “neural network” may be trained to detect “significant features of the plants” such as “the trunks of mature almond trees”, where the trunks are the common feature);
and the plurality of recurrent objects are correlated with a crop growing in the operational environment (Ellaboudy, Para. 0059 and 0228-0230 – “crop rows”, which for example may be plants or trees such as “raspberry bushes” or “almond trees” in a “geographic area”).
PNG
media_image1.png
1088
1083
media_image1.png
Greyscale
Ellaboudy, Annotated Fig. 32
Claim(s) 4, 8, 14, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Ellaboudy in view of Davis, and further in view of Sibley, et al., hereinafter Sibley (U.S. Patent Application Pub. No. 2022/0183208).
In regards to Claim 4, Ellaboudy in view of Davis teaches the method of Claim 1, and Ellaboudy teaches wherein: the obtaining the sensor data (Ellaboudy, Para. 0105-0108, 0118, 0228 – detecting “environmental landmarks for vehicle navigation” by “a lidar sensor”) comprises: obtaining first sensor data about the recurrent object relative to a first location within the operational environment; and obtaining second sensor data about the recurrent object relative to a second location within the operational environment; and the method further comprises generating a three-dimensional view of the recurrent object using the first sensor data and the second sensor data for viewing in association with the map (Ellaboudy, Para. 0097 – “presenting 410 the map to a user in a user interface” where the map may be presented as a “slice (e.g., a birds-eye-view) of a three-dimensional map (e.g., a map including point cloud data)”), but Ellaboudy does not teach obtaining second sensor data about the recurrent object relative to a second location within the operational environment, the recurrent object being partially obscured relative to the second location; and the method further comprises generating a three-dimensional view of the recurrent object using the first sensor data and the second sensor data.
However, Sibley teaches obtaining second sensor data about the recurrent object relative to a second location within the operational environment, the recurrent object being partially obscured relative to the second location (Sibley, Para. 0105 and 0338 – collecting images of objects, where “object A” is “occluded behind another object B” in different image frames, where it may be determined “whether the object A re-appeared in subsequent frames 4 or later when camera angle relative to the object A changes”); and the method further comprises generating a three-dimensional view of the recurrent object using the first sensor data and the second sensor data (Sibley, Para. 0099 and 0105 – generating a “virtual scene” which can be a “3D map” of the “agricultural scene surveyed, observed, treated, logged, or a combination thereof”, where some “objects may be occluded such that an image sensor travelling along a path guided by a vehicle may not capture the entire view of the object detected”, and when generating a virtual image of the environment, the system compensates “for the occluded portions of the object”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified the method including the above limitations of Ellaboudy in view of Davis to include obtaining second sensor data about the recurrent object relative to a second location within the operational environment, the recurrent object being partially obscured relative to the second location; and the method further comprises generating a three-dimensional view of the recurrent object using the first sensor data and the second sensor data, as taught by Sibley, in order to account of objects that may be hidden or otherwise obscured from the vehicle’s sensors to improve routing and obstacle avoidance.
In regards to Claim 8, Ellaboudy in view of Davis teaches the method of Claim 1, but Ellaboudy in view of Davis does not teach further comprising: determining a first metric associated with the recurrent object at a first time; determining a second metric associated with the recurrent object at a second time; comparing the first metric to the second metric; determining, based on the comparison of the first metric to the second metric, a mission to be performed relative to the recurrent object or the operational environment; and causing the autonomous vehicle to perform the mission.
However, Sibley teaches further comprising: determining a first metric associated with the recurrent object at a first time (Sibley, Para. 0096 – where, for example, the vehicle “sensing system senses a potential agricultural object” and can identify “varying characteristics detected such as stage of growth, size, color, health, density, etc.”, or metric); determining a second metric associated with the recurrent object at a second time (Sibley, Para. 0096 – when identifying the “potential agricultural object”, the “sensing system” can determine whether the object is “a previously identified, tagged, and stored object detected again”, such that it is detected at both a previous time and a second, current, time, and determining whether the object “state or stage of growth in its phenological cycle” has changed, based on “varying characteristics detected such as stage of growth, size, color, health, density, etc.”, or metric); comparing the first metric to the second metric (Sibley, Para. 0096 – determining whether a potential agricultural object has “changed its state or stage of growth in its phenological cycle, a previously identified object that has moved or changed in anatomy, or other objects with varying characteristics detected such as stage of growth, size, color, health, density, etc.”, or determining whether a metric has changed from a previous time at the current time); determining, based on the comparison of the first metric to the second metric, a mission to be performed relative to the recurrent object or the operational environment; and causing the autonomous vehicle to perform the mission (Sibley, Para. 0096 – “the treatment system 311 can determine, based on a combination of determining the agricultural object's identity, phenotype, stage of growth, and treatment history, if any, whether to perform a unique action”, or mission, “onto the agricultural object 302 identified”, for example, “an interaction between a treat unit of the treatment system 311 that can interact with a target, including preparing a chemical fluid projectile emitted from a device or treatment unit as part of the treatment 311 directly onto a portion of a surface of the agricultural object 302”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified the method including the above limitations of Ellaboudy in view of Davis to include determining a first metric associated with the recurrent object at a first time; determining a second metric associated with the recurrent object at a second time; comparing the first metric to the second metric; determining, based on the comparison of the first metric to the second metric, a mission to be performed relative to the recurrent object or the operational environment; and causing the autonomous vehicle to perform the mission, as taught by Sibley, in order to improve accuracy when determining an action to be taken by the autonomous vehicle based on the state of the recurrent object by comparing the object at different times.
In regards to Claim 14, Ellaboudy in view of Davis teaches the one or more computer readable mediums of Claim 11, and Ellaboudy teaches wherein: the obtaining the sensor data (Ellaboudy, Para. 0105-0108, 0118, 0228 – detecting “environmental landmarks for vehicle navigation” by “a lidar sensor”) comprises: obtaining first sensor data about the recurrent object relative to a first location within the operational environment; and obtaining second sensor data about the recurrent object relative to a second location within the operational environment; and the method further comprises generating a three-dimensional view of the recurrent object using the first sensor data and the second sensor data for viewing in association with the map (Ellaboudy, Para. 0097 – “presenting 410 the map to a user in a user interface” where the map may be presented as a “slice (e.g., a birds-eye-view) of a three-dimensional map (e.g., a map including point cloud data)”), but Ellaboudy does not teach obtaining second sensor data about the recurrent object relative to a second location within the operational environment, the recurrent object being partially obscured relative to the second location; and the method further comprises generating a three-dimensional view of the recurrent object using the first sensor data and the second sensor data.
However, Sibley teaches obtaining second sensor data about the recurrent object relative to a second location within the operational environment, the recurrent object being partially obscured relative to the second location (Sibley, Para. 0105 and 0338 – collecting images of objects, where “object A” is “occluded behind another object B” in different image frames, where it may be determined “whether the object A re-appeared in subsequent frames 4 or later when camera angle relative to the object A changes”); and the method further comprises generating a three-dimensional view of the recurrent object using the first sensor data and the second sensor data (Sibley, Para. 0099 and 0105 – generating a “virtual scene” which can be a “3D map” of the “agricultural scene surveyed, observed, treated, logged, or a combination thereof”, where some “objects may be occluded such that an image sensor travelling along a path guided by a vehicle may not capture the entire view of the object detected”, and when generating a virtual image of the environment, the system compensates “for the occluded portions of the object”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified the one or more computer readable mediums including the above limitations of Ellaboudy in view of Davis to include obtaining second sensor data about the recurrent object relative to a second location within the operational environment, the recurrent object being partially obscured relative to the second location; and the method further comprises generating a three-dimensional view of the recurrent object using the first sensor data and the second sensor data, as taught by Sibley, in order to account of objects that may be hidden or otherwise obscured from the vehicle’s sensors to improve routing and obstacle avoidance.
In regards to Claim 18, Ellaboudy in view of Davis teaches the one or more computer readable mediums of Claim 1, but Ellaboudy in view of Davis does not teach the operations further comprising: determining a first metric associated with the recurrent object at a first time; determining a second metric associated with the recurrent object at a second time; comparing the first metric to the second metric; determining, based on the comparison of the first metric to the second metric, a mission to be performed relative to the recurrent object or the operational environment; and causing the autonomous vehicle to perform the mission.
However, Sibley teaches the operations further comprising: determining a first metric associated with the recurrent object at a first time (Sibley, Para. 0096 – where, for example, the vehicle “sensing system senses a potential agricultural object” and can identify “varying characteristics detected such as stage of growth, size, color, health, density, etc.”, or metric); determining a second metric associated with the recurrent object at a second time (Sibley, Para. 0096 – when identifying the “potential agricultural object”, the “sensing system” can determine whether the object is “a previously identified, tagged, and stored object detected again”, such that it is detected at both a previous time and a second, current, time, and determining whether the object “state or stage of growth in its phenological cycle” has changed, based on “varying characteristics detected such as stage of growth, size, color, health, density, etc.”, or metric); comparing the first metric to the second metric (Sibley, Para. 0096 – determining whether a potential agricultural object has “changed its state or stage of growth in its phenological cycle, a previously identified object that has moved or changed in anatomy, or other objects with varying characteristics detected such as stage of growth, size, color, health, density, etc.”, or determining whether a metric has changed from a previous time at the current time); determining, based on the comparison of the first metric to the second metric, a mission to be performed relative to the recurrent object or the operational environment; and causing the autonomous vehicle to perform the mission (Sibley, Para. 0096 – “the treatment system 311 can determine, based on a combination of determining the agricultural object's identity, phenotype, stage of growth, and treatment history, if any, whether to perform a unique action”, or mission, “onto the agricultural object 302 identified”, for example, “an interaction between a treat unit of the treatment system 311 that can interact with a target, including preparing a chemical fluid projectile emitted from a device or treatment unit as part of the treatment 311 directly onto a portion of a surface of the agricultural object 302”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method including the above limitations of Ellaboudy in view of Sibley to further include determining a first metric associated with the recurrent object at a first time; determining a second metric associated with the recurrent object at a second time; comparing the first metric to the second metric; determining, based on the comparison of the first metric to the second metric, a mission to be performed relative to the recurrent object or the operational environment; and causing the autonomous vehicle to perform the mission, as taught by Sibley, in order to improve accuracy when determining an action to be taken by the autonomous vehicle based on the state of the recurrent object by comparing the object at different times.
Claim(s) 5 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Ellaboudy in view of Davis, and further in view of Iwase, et al., hereinafter Iwase (U.S. Patent Application Pub. No. 2022/0091271).
In regards to Claim 5, Ellaboudy in view of Davis teaches the method of Claim 1, and Ellaboudy further teaches wherein the causing the autonomous vehicle to navigate the operational environment using the map (Ellaboudy, Para. 0081 and 0100-0103 – generating a “path data structure” based on the “map” and “boundary” data, where boundary data specifies “an area within the map”; where the vehicle may execute a “path following mode” to steer to waypoints of a path “autonomously”) comprises: determining a path for the autonomous vehicle through the operational environment using the map (Ellaboudy, Para. 0081 and 0100-0103 – generating a “path data structure” based on the “map” and “boundary” data, where boundary data specifies “an area within the map”); determining a distance between the autonomous vehicle and the recurrent object along the path (Ellaboudy, Para. 0082 and 0194 – “the distance sensor may be configured to output range data reflecting distances of objects with respect to the vehicle”; for example while “traveling, the vehicle may detect objects on its way using both three-dimensional and two-dimensional sensors”); “obstacles are detected, and notification is relayed via video feed to a user interface”); and responsive to the distance being less than or equal to where if row width “is not sufficient for the vehicle to go through, the vehicle may stop to wait for human help where collision alarm is off”, such that the vehicle pauses autonomous movement and awaits manual human operation; where for example “detect whether there is an obstacle on the way and act safely as it takes a certain amount of time for the vehicle to fully stop.”).
Ellaboudy does not teach responsive to the distance being less than or equal to a first threshold distance but greater than a second threshold distance, providing an alert; and responsive to the distance being less than or equal to the second threshold distance, causing the autonomous vehicle to stop navigating the operational environment.
However, Iwase teaches responsive to the distance being less than or equal to a first threshold distance but greater than a second threshold distance, providing an alert (Iwase, Fig. 4 and Para. 0124-0125 – acquiring a “distance to the obstacle”, where if the obstacle is located in “the notification control range Rnc”, “execute first notification control for notifying of the presence of the obstacle in the notification control range Rnc” on a “display”); and responsive to the distance being less than or equal to the second threshold distance, causing the autonomous vehicle to stop navigating the operational environment (Iwase, Fig. 4 and Para. 0124-0127 – acquiring a “distance to the obstacle”, where if the obstacle is located in “the stop control range Rsc”, execute “automatic stop control”; where as seen on Fig. 4 of Iwase, the range Rnc, or first distance, is further from the work machine than the range Rsc, or second distance).
PNG
media_image3.png
368
844
media_image3.png
Greyscale
Iwase, Fig. 4
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified the method including the above limitations of Ellaboudy in view of Davis to include responsive to the distance being less than or equal to a first threshold distance but greater than a second threshold distance, providing an alert; and responsive to the distance being less than or equal to the second threshold distance, causing the autonomous vehicle to stop navigating the operational environment, as taught by Iwase, in order to warn the user of an impending collision to allow the user to manually stop the vehicle and prevent sudden braking to improve the operating experience.
In regards to Claim 15, Ellaboudy in view of Davis teaches the one or more computer readable mediums of Claim 11, and Ellaboudy further teaches wherein the operation causing the autonomous vehicle to navigate the operational environment using the map (Ellaboudy, Para. 0081 and 0100-0103 – generating a “path data structure” based on the “map” and “boundary” data, where boundary data specifies “an area within the map”; where the vehicle may execute a “path following mode” to steer to waypoints of a path “autonomously”) comprises: determining a path for the autonomous vehicle through the operational environment using the map (Ellaboudy, Para. 0081 and 0100-0103 – generating a “path data structure” based on the “map” and “boundary” data, where boundary data specifies “an area within the map”); determining a distance between the autonomous vehicle and the recurrent object along the path (Ellaboudy, Para. 0082 and 0194 – “the distance sensor may be configured to output range data reflecting distances of objects with respect to the vehicle”; for example while “traveling, the vehicle may detect objects on its way using both three-dimensional and two-dimensional sensors”); vehicle (Ellaboudy, Para. 0105 – “obstacles are detected, and notification is relayed via video feed to a user interface”); and responsive to the distance being less than or equal to where if row width “is not sufficient for the vehicle to go through, the vehicle may stop to wait for human help where collision alarm is off”, such that the vehicle pauses autonomous movement and awaits manual human operation; where for example “detect whether there is an obstacle on the way and act safely as it takes a certain amount of time for the vehicle to fully stop.”).
Ellaboudy does not teach responsive to the distance being less than or equal to a first threshold distance but greater than a second threshold distance, providing an alert; and responsive to the distance being less than or equal to the second threshold distance, causing the autonomous vehicle to stop navigating the operational environment.
However, Iwase teaches responsive to the distance being less than or equal to a first threshold distance but greater than a second threshold distance, providing an alert (Iwase, Fig. 4 and Para. 0124-0125 – acquiring a “distance to the obstacle”, where if the obstacle is located in “the notification control range Rnc”, “execute first notification control for notifying of the presence of the obstacle in the notification control range Rnc” on a “display”); and responsive to the distance being less than or equal to the second threshold distance, causing the autonomous vehicle to stop navigating the operational environment (Iwase, Fig. 4 and Para. 0124-0127 – acquiring a “distance to the obstacle”, where if the obstacle is located in “the stop control range Rsc”, execute “automatic stop control”; where as seen on Fig. 4 of Iwase, the range Rnc, or first distance, is further from the work machine than the range Rsc, or second distance).
PNG
media_image3.png
368
844
media_image3.png
Greyscale
Iwase, Fig. 4
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified the one or more computer readable mediums including the above limitations of Ellaboudy in view of Davis to include responsive to the distance being less than or equal to a first threshold distance but greater than a second threshold distance, providing an alert; and responsive to the distance being less than or equal to the second threshold distance, causing the autonomous vehicle to stop navigating the operational environment, as taught by Iwase, in order to warn the user of an impending collision to allow the user to manually stop the vehicle and prevent sudden braking to improve the operating experience.
Claim(s) 6 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Ellaboudy in view of Davis, and further in view of Friedlein, et al., hereinafter Friedlein (U.S. Patent Application Pub. No. 2023/0189690) and Gornik, et al., hereinafter Gornik (U.S. Patent Application Pub. No. 2021/0019903).
In regards to Claim 6, Ellaboudy in view of Davis teaches the method of Claim 1, but Ellaboudy does not teach wherein: the obtaining the sensor data comprises: obtaining first sensor data about the recurrent object at a first time; and obtaining second sensor data about the recurrent object at a second time; and the method further comprises: comparing the first sensor data to the second sensor data; determining whether the recurrent object is detected at the second time based on the comparison; and responsive to the recurrent object not being detected at the second time, providing an alert on a display associated with the autonomous vehicle.
However, Friedlein teaches wherein: the obtaining the sensor data comprises: obtaining first sensor data about the recurrent object at a first time; and obtaining second sensor data about the recurrent object at a second time; and the method further comprises: comparing the first sensor data to the second sensor data (Friedlein, Para. 0044 and 0097-0103 – “sensor assemblies” for obtaining “harvest data inputs, such as stalk count, stalk size”, etc., and collecting “data or map layers” when a stalk row is planted, “previous pass”, and when a row is harvested, “harvest pass”, and “comparing that planted amount to the number of plants per acre that emerged”); determining whether the recurrent object is detected at the second time based on the comparison (Friedlein, Para. 0097-0103 – “comparing that planted amount to the number of plants per acre that emerged to determine a percentage of plants lost”, where the plants are the recurrent objects, where non-emerged plants are “missing”, or not detected); and responsive to the recurrent object not being detected at the second time, providing an alert (Friedlein, Para. 0097-0105 – providing a “metric” on “value per acre that is lost” for comparing results)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified the method including the above limitations of Ellaboudy in view of Davis to include wherein: the obtaining the sensor data comprises: obtaining first sensor data about the recurrent object at a first time; and obtaining second sensor data about the recurrent object at a second time; and the method further comprises: comparing the first sensor data to the second sensor data; determining whether the recurrent object is detected at the second time based on the comparison; and responsive to the recurrent object not being detected at the second time, providing an alert, as taught by Friedlein, in order to allow users to identify potential issues and “explore the impact, if any, of various treatments” on harvest (Friedlein, Para. 0105).
Ellaboudy in view of Davis and Friedlein does not teach responsive to the recurrent object not being detected, providing an alert on a display associated with the autonomous vehicle.
However, Gornik teaches responsive to the recurrent object not being detected, providing an alert on a display associated with the autonomous vehicle (Gornik, Para. 0089 – “if control unit 205 identifies that, with respect to where a plant 240 or row 285 are expected, distance measurements indicate no plants 240 or row 285 exist”, “control unit 205 may inform or alert a user, e.g., by presenting a notification on a monitor of a computer or sending a message”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified the method including the above limitations of Ellaboudy in view of Davis and Friedlein to include responsive to the recurrent object not being detected, providing an alert on a display associated with the autonomous vehicle, as taught by Gornik, in order to notify a user of missing recurrent object to allow the user to identify the cause.
In regards to Claim 16, Ellaboudy in view of Davis teaches the one or more computer readable mediums of Claim 11, but Ellaboudy does not teach wherein: the operation obtaining the sensor data comprises: obtaining first sensor data about the recurrent object at a first time; and obtaining second sensor data about the recurrent object at a second time; and the operations further comprise: comparing the first sensor data to the second sensor data; determining whether the recurrent object is detected at the second time based on the comparison; and responsive to the recurrent object not being detected at the second time, providing an alert on a display associated with the autonomous vehicle.
However, Friedlein teaches wherein: the operation obtaining the sensor data comprises: obtaining first sensor data about the recurrent object at a first time; and obtaining second sensor data about the recurrent object at a second time; and the operations further comprise: comparing the first sensor data to the second sensor data (Friedlein, Para. 0044 and 0097-0103 – “sensor assemblies” for obtaining “harvest data inputs, such as stalk count, stalk size”, etc., and collecting “data or map layers” when a stalk row is planted, “previous pass”, and when a row is harvested, “harvest pass”, and “comparing that planted amount to the number of plants per acre that emerged”); determining whether the recurrent object is detected at the second time based on the comparison (Friedlein, Para. 0097-0103 – “comparing that planted amount to the number of plants per acre that emerged to determine a percentage of plants lost”, where the plants are the recurrent objects, where non-emerged plants are “missing”, or not detected); ; and responsive to the recurrent object not being detected at the second time, providing an alert (Friedlein, Para. 0097-0105 – providing a “metric” on “value per acre that is lost” for comparing results)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified the one or more computer readable mediums including the above limitations of Ellaboudy in view of Davis to include wherein: the operation obtaining the sensor data comprises: obtaining first sensor data about the recurrent object at a first time; and obtaining second sensor data about the recurrent object at a second time; and the operations further comprise: comparing the first sensor data to the second sensor data; determining whether the recurrent object is detected at the second time based on the comparison; and responsive to the recurrent object not being detected at the second time, providing an alert, as taught by Friedlein, in order to allow users to identify potential issues and “explore the impact, if any, of various treatments” on harvest (Friedlein, Para. 0105).
Ellaboudy in view of Davis and Friedlein does not teach responsive to the recurrent object not being detected, providing an alert on a display associated with the autonomous vehicle.
However, Gornik teaches responsive to the recurrent object not being detected, providing an alert on a display associated with the autonomous vehicle (Gornik, Para. 0089 – “if control unit 205 identifies that, with respect to where a plant 240 or row 285 are expected, distance measurements indicate no plants 240 or row 285 exist”, “control unit 205 may inform or alert a user, e.g., by presenting a notification on a monitor of a computer or sending a message”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified the one or more computer readable mediums including the above limitations of Ellaboudy in view of Davis and Friedlein to include responsive to the recurrent object not being detected, providing an alert on a display associated with the autonomous vehicle, as taught by Gornik, in order to notify a user of missing recurrent object to allow the user to identify the cause.
Claim(s) 10 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Ellaboudy in view of Davis, and further in view of Schroeder (U.S. Patent No. 10,703,277).
In regards to Claim 10, Ellaboudy in view of Sibley teaches the method of Claim 1, and Ellaboudy in view of Sibley teaches augmented map data (Sibley, Para. 0113-0119 – “the system can update the same map), but Ellaboudy in view of Sibley does not teach wherein the recurrent object comprises a first recurrent object and the method further comprises predicting a location of a second recurrent object within the operational environment based on map data.
However, Schroeder teaches wherein the recurrent object comprises a first recurrent object and the method further comprises predicting a location of a second recurrent object within the operational environment based on the map data (Schroeder, Col. 6 Line 40-61 – utilizing “a map of the crop field” to “identify the physical location of the next crop entry point” by providing a “reference point” of a next crop “calculated using conventional geometric formulae” based on the location of the current “crop entry point”, such that the “reference point” is a prediction of a next crop entry point).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified the method including the above limitations of Ellaboudy in view of Davis to include wherein the recurrent object comprises a first recurrent object and the method further comprises predicting a location of a second recurrent object within the operational environment based on the map data, as taught by Schroeder, in order to determine how a vehicle should move to reach a next recurrent object when path planning.
In regards to Claim 20, Ellaboudy in view of Davis teaches the one or more computer readable mediums of Claim 11, and Ellaboudy in view of Davis teaches augmented map data (Davis, Para. 0051, 0068-0070, 0094-0095 – “map generator 312 can generate an updated (or revised) worksite map (e.g., 470) as the machine travels or operates at the worksite”), but Ellaboudy in view of Davis does not teach wherein the recurrent object comprises a first recurrent object and the operations further comprise predicting a location of a second recurrent object within the operational environment based on map data.
However, Schroeder teaches wherein the recurrent object comprises a first recurrent object and the operations further comprise predicting a location of a second recurrent object within the operational environment based on the map data (Schroeder, Col. 6 Line 40-61 – utilizing “a map of the crop field” to “identify the physical location of the next crop entry point” by providing a “reference point” of a next crop “calculated using conventional geometric formulae” based on the location of the current “crop entry point”, such that the “reference point” is a prediction of a next crop entry point).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified the method including the above limitations of Ellaboudy in view of Davis to include wherein the recurrent object comprises a first recurrent object and the operations further comprise predicting a location of a second recurrent object within the operational environment based on the map data, as taught by Schroeder, in order to determine how a vehicle should move to reach a next recurrent object when path planning.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Sneyders, et al. (U.S. Patent Application Pub. No. 2022/0189063) teaches systems and techniques for calibrating a crop row computer vision system, including searching models of a field to find a model that best fits a field represented in an image set.
Chowdhary, et al. (U.S. Patent Application Pub. No. 2022/0317702) teaches a method, system and non-transitory computer readable medium for obtaining an electronic map of an agricultural field, including adjusting or providing an updated designation in a electronic field map.
Madsen, et al. (U.S. Patent Application Pub. No. 2019/0129435) teaches generating and utilizing three-dimensional terrain maps for vehicular control in agricultural/farming applications, where generation of three-dimensional terrain maps may include determining a height of a portion of the vegetation (e.g., crops) on the terrain above the ground surface to help determine whether a crop is ready for harvesting, where a vehicle drives between rows of crops.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HELEN LI whose telephone number is (703)756-4719. The examiner can normally be reached Monday through Friday, from 9am to 5pm eastern.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hunter Lonsberry can be reached at (571) 272-7298. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/H.L./Examiner, Art Unit 3665
/HUNTER B LONSBERRY/Supervisory Patent Examiner, Art Unit 3665
Prosecution Insights