DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claims 1 – 5, 7 – 19 are rejected under 35 U.S.C. 103 as being unpatentable over Sibley et al. (US 20220183208 A1; hereafter referred to as Sibley) in view of Fu et al. (US 20210090274 A1; hereafter referred to as Fu).
Regarding Claim 1, Sibley teaches:
A method for visually determining a position within a global coordinate system for a feature of interest in an agricultural field (Sibley, [0102] “a visual representation of each individual and unique agricultural object on the geographic boundary, and can be identified and indexed, associated with its position data”), the method comprising steps of:
determining a position of an agricultural machine within the global coordinate system (Sibley, [0089] “, a positional data module 108 for capturing, fusing, and transmitting sensor data related to position, localization, pose, velocity, and other position related signals to the rest of the system 100”; Sibley, [0101] “as the vehicle 310 passes by a particular agricultural object in the real world”);
determining an orientation of the agricultural machine (Sibley, [0102] “. The object 320 may have an associated geo-graphic data associated to the object, including position data, orientation and pose estimation relative to the geographic boundary view or relative to physical components of the agricultural treatment system 400, including image sensors, or treatment engines, or relative to other agricultural objects”);
determining a camera position for a camera operatively connected to the agricultural machine (Sibley, [0159] “ a pair or more than one pair of image sensors 313, in stereo, can be mounted on the vehicle such that the sensors are pointed out to the real world to observe a global scene,… Additionally, Lidars, radar, sonar, and other sensors can be mounted in a similar location as that of the cameras to register a global scene”) and a camera orientation for the camera based on position offset from the position of the agricultural machine within the global coordinate system and orientation offset from the orientation of the agricultural machine (Sibley, [0271] “The treatment unit process may then determine its pose by using the pose of the vehicle and may offset the pose of the vehicle based on a distance value from where a respective treatment unit is positioned relative to the point of where the pose is determined for the vehicle”);
acquiring imagery from the camera, the imagery including the feature of interest (Sibley, [0081] “The system utilizes one or more image sensors, including stereoscopic cameras to obtain digital imagery, including 3D imagery of an agricultural scene such as a tree in an orchard or a row of plants on a farm while the system moves along a path near the crops”; Sibley, [0092] “a plant such as a tree 303 can include a trunk, root, branch, stems, leaves, pedals, flowers, plant pistils and stigma, buds, fruitlets, fruits, and many other portions of a plant that make up the plant's anatomy, all of which can be agricultural objects of interest for observation and treatment”); and
While Sibley teaches using a plurality of different coordinate systems and origin points to locate relative positions and orientations of objects relative to other objects, (Sibley, [0106], [0156] and [0171] “While the compute unit and sensors associated with the navigation unit performs slam and accounts for a vehicle pose (the vehicles components' own SLAM), for navigation purposes, it can also combine to individually map every single plant, because the plant will have a known location due to its relative location to the component treatment modules' sensors, for example stereo image sensors. The stereo image sensors will know where it is relative to a global scene to do the mapping of a local scene and the real time localization of the vehicle itself in the global scene”), it does not explicitly recite:
at a processor, combining depth data associated with the imagery, pixel location of the feature of interest from the imagery, the camera position, the camera orientation, the position of the agricultural machine within the global coordinate system, and the orientation of the agricultural machine to locate the feature of interest by providing visually derived coordinates within the global coordinate system.
In the same field of endeavor, Fu teaches:
at a processor, combining depth data associated with the imagery, pixel location of the feature of interest from the imagery, the camera position, the camera orientation, the position of the agricultural machine within the global coordinate system, and the orientation of the agricultural machine to locate the feature of interest by providing visually derived coordinates within the global coordinate system (Fu, [0079] “This technique may utilize accurate georeferencing of image coordinates to geographic coordinates”; Fu, [0113] “the control system 130 determines depth information from accessed images that are captured by a single image sensor and contain visual information representing depth. … determine depth information from other images, combination of images, etc. including visual information representing depth. For example, can determine depth from a combination of stereo images”; [0159] “the control system 130 accesses labelled depth information in the labelled depth map 760 and applies a coordinate system. The applied coordinate system may be based on any of the positions of the farming machine 100 in the field, the location of the camera on the farming machine 100, and the field of view of the camera. By applying the coordinate system, every pixel in a labelled depth map 460 is assigned a coordinate that corresponds to a real-world location in a field where the farming machine 100 is operating”; Fu, [0160] “The control system 130 combines the coordinates of pixels in the labelled depth map 460 with their corresponding depth information to generate a point cloud. That is, given extrinsic and intrinsic calibration parameters of the camera(s) that acquired the image (e.g., accessed image 800), it is possible to convert the 2D representation of the depth map (e.g., depth map 840) to a 3D representation of a point cloud.. More specifically, to generate the point cloud the control system 130 projects a ray from a model camera location into a representation of the real-world space in the accessed image”; Fu, [0171] “each of the compute units of each component treatment modules can associate local and global pose. While the compute unit and sensors associated with the navigation unit performs slam and accounts for a vehicle pose (the vehicles components' own SLAM), for navigation purposes”; Fu, [0110] describes that camera position and orientation are used to capture images of a plant as farming machine travels; see Fu, [0023], [0076] and [0105] Fu describes that the control system uses orientation of the farming machine and sensors as feature information for treatment mechanism).
Sibley and Fu are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing for agricultural activities. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Sibley with the invention of Fu to make the invention that combines depth data associated with the imagery, pixel location of the feature of interest from the imagery, the camera position, the camera orientation, the position of the agricultural machine within the global coordinate system, and the orientation of the agricultural machine to locate the feature of interest by providing visually derived coordinates within the global coordinate system; doing so can generate a global map with every single unique and individual agricultural object sensed, and indexed with high precision in the global map and provide a precise trajectory for the treatment to be deposited on a surface of the agricultural object (Fu, [0171] and [0192]); thus one of the ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 2, Sibley in view of Fu teaches the method of claim 1, wherein the feature of interest is a plant (Sibley, [0081] “the systems, robots, computer software and systems, applications using computer vision and automation, or a combination thereof, can be configured observe a geographic boundary having one or more plants growing agricultural objects identified as potential crops, detect specific agricultural objects to each individual plant and portions of the plant”; Sibley, [0098] “the agricultural scene can be that of a farm, orchard, or any kind of ground terrain that does not yet have any trees or crops, but only of dirt and soil”).
Regarding Claim 3, Sibley in view of Fu teaches the method of claim 2, further comprising performing a crop row following operation within the agricultural field wherein the visually derived coordinates for the plant are used in performing the crop row following within the agricultural field (Sibley, [0106] “The coordinate system chosen is just an example wherein a plurality of different coordinate systems and origin points can be used to locate relative positions and orientations of objects relative to other objects”; Sibley, [0098] In one example, the agricultural scene and geographic region can be a farm where the ground or terrain is partitioned into a plurality of rows with row crops for planting, growing, and harvesting and the plants themselves are harvested, unlike that of orchards where agricultural objects are harvested from permanent plants”).
Regarding Claim 4, Sibley in view of Fu teaches the method of claim 1, further comprising performing an agricultural operation within the agricultural field using the visually derived coordinates within the global coordinate system for the feature of interest (Sibley, [0098] “The observation and treatment can be observed and performed on the crops themselves, or of other plants of interest. For example, weeds can grow in the same agricultural scene as that of a crop of interest such that the observation and treatment performed by treatment system 311 can be that of both the crop and the one or more different types of weeds, or just the weeds”; Sibley, [0102] “The portion of the full frame image can be extracted as a patch such that each individual patch, which itself is a portion of the full frame image, is a visual representation of each individual and unique agricultural object on the geographic boundary, and can be identified and indexed, associated with its position data, any and all treatment history if any of the objects detected”; Sibley, [0156] “the agricultural observation and treatment system 311 can perform mapping of a scene and localizing the treatment system in the scene. This can include mapping a scene with a global frame of reference or point of origin in a given coordinate system, and determining its location relative to a point in the mapped scene, particularly a point of origin in the scene”).
Regarding Claim 5, Sibley in view of Fu teaches the method of claim 1, further comprising repeating the steps a plurality of times and for each time storing the visually derived coordinates for use in a subsequent operation (Sibley, [0114] “each virtual map of the same real world geographic coordinates, or predetermined geofenced location, can be associated with each other such that an interactive changing map can be displayed where one map changing or updating to another map represents the changing state of the geographical scene changing across a grow season”; Sibley, [0196] “the agricultural treatment system records positional and sensor information and associates this information for each of the obtained images. Some of this information may include geo-spatial location data (e.g., GPS coordinates)”; Sibley, [0232] “The system can repeat steps 1760 to detect a second object detected in the first region and prepare treatment actions associated with the second object. Once all objects of interest are accounted for in the first region of interest, the system can detect objects in a second region of interest for treatment”; See, Fu [0089], [0160] and [0161]).
Regarding Claim 7, Sibley in view of Fu teaches the method of claim 1, wherein the position and the orientation of the agricultural machine are determined based on a combination of GPS location data associated with a GPS antenna associated with the agricultural machine and inertial data associated with an inertial measurement unit (IMU) associated with the agricultural machine (Sibley, [0130] The sensing signals from the sensors 432 can also include depth signals from depth sensing cameras including RGB-D cameras or infrared cameras, or calculated with stereo vision mounted sensors such as stereo vision cameras, as well as other signals from radar, radio, sonar signals, photoelectric and photo optic signals, as well as location sensing signals, from having a global positioning system (GPS) unit, encoders for wheel odometry, IMU's, speedometers, etc”).
Regarding Claim 8, Sibley in view of Fu teaches the method of claim 1, wherein the agricultural machine comprises at least one of an agricultural vehicle and an agricultural implement (Sibley, [0154] The agricultural observation and treatment system can be a modular system similar to that of agricultural observation and treatment system 311 supported by vehicle 310. The system 311 can also include various sensors such as imaged based sensor 313, lidar based sensors 314, or a plurality of non-vision-based sensors as described previously…, the treatment system depicted in diagram 700 can use sensors, such sensor 313 and 314 to perform global registry of the agricultural environment as well as perform localization and pose estimation of the vehicle or portions of the vehicle in a global scene or from a global point of reference”).
Regarding Claim 9, Sibley in view of Fu teaches the method of claim 1, wherein the depth data is determined with a depth sensor associated with the camera (Sibley, [0130] “The sensing signals from the sensors 432 can also include depth signals from depth sensing cameras including RGB-D cameras or infrared cameras, or calculated with stereo vision mounted sensors such as stereo vision cameras”).
Regarding Claim 10, Sibley in view of Fu teaches the method of claim 1, wherein the depth data is acquired through at least one of time-of-flight sensing, and lidar sensing (Sibley, [0134] “ cameras of the camera module 450 and can include depth sensing cameras, LiDAR's, radar, electrooptical sensors, lasers, etc”; Fu, [0099] “depth sensing systems may include devices such as proximity sensors, stereo cameras, time-of-flight cameras, structured light sensors (e.g., emitting patterned light and sensing changes to the pattern), radar, etc”).
Regarding Claim 11, Sibley in view of Fu teaches the method of claim 1, wherein the pixel location is at a center of the feature of interest (Fu, [0115] “ a pixel in the depth map 460 represents, for example, a distance between a center point of the lens of the sensor 310a and the object it represents in the captured image 400, and, in some cases, the group of similarly distanced pixels that the pixel belongs to within the depth map 460”).
The reasons for combining Sibley and Fu are similar to that stated above for claim 1.
Regarding Claim 12, Sibley teaches:
A system configured for visually determining a position within a global coordinate system for a feature of interest, the system comprising (Sibley, [0102] “a visual representation of each individual and unique agricultural object on the geographic boundary, and can be identified and indexed, associated with its position data”), the method comprising steps of:
a camera for operatively connecting to an object (Sibley, [0159] “a pair or more than one pair of image sensors 313, in stereo, can be mounted on the vehicle such that the sensors are pointed out to the real world to observe a global scene, as opposed to down on the ground to observe individual plants or right in front of (a few meters or fewer away) a tree to observe individual plants, crops, or other agricultural objects. Additionally, Lidars, radar, sonar, and other sensors can be mounted in a similar location as that of the cameras to register a global scene”);
a location determining device for determining a position of the object within the global coordinate system and orientation of the object (Sibley, [0089] “, a positional data module 108 for capturing, fusing, and transmitting sensor data related to position, localization, pose, velocity, and other position related signals to the rest of the system 100”; Sibley, [0101] “as the vehicle 310 passes by a particular agricultural object in the real world”);
at least one processor executing instructions to:
determine a camera position for the camera and a camera orientation for the camera based on position offset from the position of the object within the global coordinate system and orientation offset from the orientation of the object (Sibley, [0159] “ a pair or more than one pair of image sensors 313, in stereo, can be mounted on the vehicle such that the sensors are pointed out to the real world to observe a global scene,… Additionally, Lidars, radar, sonar, and other sensors can be mounted in a similar location as that of the cameras to register a global scene”; Sibley, [0271] “The treatment unit process may then determine its pose by using the pose of the vehicle and may offset the pose of the vehicle based on a distance value from where a respective treatment unit is positioned relative to the point of where the pose is determined for the vehicle”);
acquiring imagery from the camera, the imagery including the feature of interest (Sibley, [0081] “The system utilizes one or more image sensors, including stereoscopic cameras to obtain digital imagery, including 3D imagery of an agricultural scene such as a tree in an orchard or a row of plants on a farm while the system moves along a path near the crops”; Sibley, [0092] “a plant such as a tree 303 can include a trunk, root, branch, stems, leaves, pedals, flowers, plant pistils and stigma, buds, fruitlets, fruits, and many other portions of a plant that make up the plant's anatomy, all of which can be agricultural objects of interest for observation and treatment”); and
While Sibley teaches using a plurality of different coordinate systems and origin points to locate relative positions and orientations of objects relative to other objects, (Sibley, [0106], [0156] and [0171] “While the compute unit and sensors associated with the navigation unit performs slam and accounts for a vehicle pose (the vehicles components' own SLAM), for navigation purposes, it can also combine to individually map every single plant, because the plant will have a known location due to its relative location to the component treatment modules' sensors, for example stereo image sensors. The stereo image sensors will know where it is relative to a global scene to do the mapping of a local scene and the real time localization of the vehicle itself in the global scene”), it does not explicitly recite:
combine depth data associated with the imagery, pixel location of the feature of interest from the imagery, the camera position, the camera orientation, the position of the agricultural machine within the global coordinate system, and the orientation of the agricultural machine to locate the feature of interest by providing visually derived coordinates within the global coordinate system.
In the same field of endeavor, Fu teaches:
combine depth data associated with the imagery, pixel location of the feature of interest from the imagery, the camera position, the camera orientation, the position of the agricultural machine within the global coordinate system, and the orientation of the agricultural machine to locate the feature of interest by providing visually derived coordinates within the global coordinate system (Fu, [0079] “This technique may utilize accurate georeferencing of image coordinates to geographic coordinates”; Fu, [0113] “the control system 130 determines depth information from accessed images that are captured by a single image sensor and contain visual information representing depth. … determine depth information from other images, combination of images, etc. including visual information representing depth. For example, can determine depth from a combination of stereo images”; [0159] “the control system 130 accesses labelled depth information in the labelled depth map 760 and applies a coordinate system. The applied coordinate system may be based on any of the positions of the farming machine 100 in the field, the location of the camera on the farming machine 100, and the field of view of the camera. By applying the coordinate system, every pixel in a labelled depth map 460 is assigned a coordinate that corresponds to a real-world location in a field where the farming machine 100 is operating”; Fu, [0160] “The control system 130 combines the coordinates of pixels in the labelled depth map 460 with their corresponding depth information to generate a point cloud. That is, given extrinsic and intrinsic calibration parameters of the camera(s) that acquired the image (e.g., accessed image 800), it is possible to convert the 2D representation of the depth map (e.g., depth map 840) to a 3D representation of a point cloud.. More specifically, to generate the point cloud the control system 130 projects a ray from a model camera location into a representation of the real-world space in the accessed image”; Fu, [0171] “each of the compute units of each component treatment modules can associate local and global pose. While the compute unit and sensors associated with the navigation unit performs slam and accounts for a vehicle pose (the vehicles components' own SLAM), for navigation purposes”; Fu, [0110] describes that camera position and orientation are used to capture images of a plant as farming machine travels; see Fu, [0023], [0076] and [0105] Fu describes that the control system uses orientation of the farming machine and sensors as feature information for treatment mechanism).
Sibley and Fu are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing for agricultural activities. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Sibley with the invention of Fu to make the invention that combines depth data associated with the imagery, pixel location of the feature of interest from the imagery, the camera position, the camera orientation, the position of the agricultural machine within the global coordinate system, and the orientation of the agricultural machine to locate the feature of interest by providing visually derived coordinates within the global coordinate system; doing so can generate a global map with every single unique and individual agricultural object sensed, and indexed with high precision in the global map and provide a precise trajectory for the treatment to be deposited on a surface of the agricultural object (Fu, [0171] and [0192]); thus one of the ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 13, Sibley in view of Fu teaches the system of claim 12, further comprising an agricultural machine wherein the camera is rigidly connected to the agricultural machine (Sibley, [0156] “As the vehicle navigates in the environment, the system can determine where the vehicle and each of the components of the system, due to the components being fixed relative to the vehicle, has moved across time… the image capture devices, optoelectrical sensors, ultrasonic sensors, radar, sonar, and other image capture devices such as NIR cameras, RGB-D cameras, multispectral cameras, etc.,… the components being rigidly connected or supported by the vehicle….each of these cameras can be rigidly connected to the sensors configured to perform global registry of the environment”).
Regarding Claim 14, Sibley in view of Fu teaches the system of claim 13, wherein the camera is a 3D camera (Sibley, [0081] “The system utilizes one or more image sensors, including stereoscopic cameras to obtain digital imagery, including 3D imagery of an agricultural scene”; Sibley, [0102] “ The full frames can be 2D or 3D image showing the images captured directly by one or more cameras and/or rendered by the agricultural treatment system 311”).
Regarding Claim 15, Sibley teaches:
A method for performing field operations within an agricultural field, the method comprising steps of (Sibley, [0079] – [0081], “performing methods and steps described…. can be configured observe a geographic boundary having one or more plants growing agricultural objects identified as potential crops, detect specific agricultural objects to each individual plant and portions of the plant, determine that one or more specific individual agricultural object in the real world geographic boundary requires a treatment based on its growth stage and treatment history from previous observations and treatment, and to deliver a specific treatment to each of the desired agricultural objects, among other objects”; [0095] “the agricultural observation and treatment system 311 can determine the best method of performing treatment actions, based on a variety of parameters that can be adjusted and customized, and apply the same method of treatment actions on the particular zone 304 that yielded the best crop, for other crops in a new or subsequent crop cycle.), the method comprising steps of:
visually determining a position within a global coordinate system for a feature of interest within the agricultural field by determining a position of an agricultural machine within the global coordinate system (Sibley, [0089] “a positional data module 108 for capturing, fusing, and transmitting sensor data related to position, localization, pose, velocity, and other position related signals to the rest of the system 100”; Sibley, [0101] “as the vehicle 310 passes by a particular agricultural object in the real world”; Sibley, [0102] “The object 320 may have an associated geo-graphic data associated to the object, including position data, orientation and pose estimation relative to the geographic boundary view or relative to physical components of the agricultural treatment system 400, including image sensors, or treatment engines, or relative to other agricultural objects….a visual representation of each individual and unique agricultural object on the geographic boundary, and can be identified and indexed, associated with its position data”;),
determining an orientation of the agricultural machine (Sibley, [0102] “The object 320 may have an associated geo-graphic data associated to the object, including position data, orientation and pose estimation relative to the geographic boundary view or relative to physical components of the agricultural treatment system 400, including image sensors, or treatment engines, or relative to other agricultural objects”),
determining a camera position for a camera secured to the agricultural machine based on position offset from the position of the agricultural machine (Sibley, [0159] “ a pair or more than one pair of image sensors 313, in stereo, can be mounted on the vehicle such that the sensors are pointed out to the real world to observe a global scene,… Additionally, Lidars, radar, sonar, and other sensors can be mounted in a similar location as that of the cameras to register a global scene”; Sibley, [0271] “The treatment unit process may then determine its pose by using the pose of the vehicle and may offset the pose of the vehicle based on a distance value from where a respective treatment unit is positioned relative to the point of where the pose is determined for the vehicle”);
acquiring imagery from the camera, the imagery including the feature of interest (Sibley, [0081] “The system utilizes one or more image sensors, including stereoscopic cameras to obtain digital imagery, including 3D imagery of an agricultural scene such as a tree in an orchard or a row of plants on a farm while the system moves along a path near the crops”; Sibley, [0092] “a plant such as a tree 303 can include a trunk, root, branch, stems, leaves, pedals, flowers, plant pistils and stigma, buds, fruitlets, fruits, and many other portions of a plant that make up the plant's anatomy, all of which can be agricultural objects of interest for observation and treatment”); and
performing a field operation within the agricultural field based on the position within the global coordinate system for the feature of interest (Sibley, [0141] “the compute module 424 can determine a position and orientation for the gimbal assembly to position and orient the treatment head 472 in real time and make adjustments in the position and orientation of the treatment head 472 as the agricultural treatment system 400 is moving relative to any target plants or agricultural objects of interest on the ground either in a fixed position on the ground, or is also moving The agricultural treatment system 400 can lock the treatment unit 470, at the treatment head 472, onto the target plant, or other agricultural object of interest through instructions received and controls performed by the control module of the treatment unit 470”; Sibley, [0269] “the agricultural treatment system determines multiple vehicle and/or treatment unit poses. The system evaluates a first pose, and then periodically determine subsequent poses. The system may calculate the difference or changes of the coordinate values from the first pose and a subsequently obtained posed. In other words, the system may calculate the movement of the vehicle and/or treatment unit. The calculated difference or changes then may be translated to a desired pose for the spraying head”).
While Sibley teaches using a plurality of different coordinate systems and origin points to locate relative positions and orientations of objects relative to other objects, (Sibley, [0106], [0156] and [0171] “While the compute unit and sensors associated with the navigation unit performs slam and accounts for a vehicle pose (the vehicles components' own SLAM), for navigation purposes, it can also combine to individually map every single plant, because the plant will have a known location due to its relative location to the component treatment modules' sensors, for example stereo image sensors. The stereo image sensors will know where it is relative to a global scene to do the mapping of a local scene and the real time localization of the vehicle itself in the global scene”), it does not explicitly recite:
combine depth data associated with the imagery, pixel location of the feature of interest from the imagery, the camera position, the camera orientation, the position of the agricultural machine within the global coordinate system, and the orientation of the agricultural machine to locate the feature of interest by providing visually derived coordinates within the global coordinate system.
In the same field of endeavor, Fu teaches:
combine depth data associated with the imagery, pixel location of the feature of interest from the imagery, the camera position, the camera orientation, the position of the agricultural machine within the global coordinate system, and the orientation of the agricultural machine to locate the feature of interest by providing visually derived coordinates within the global coordinate system (Fu, [0079] “This technique may utilize accurate georeferencing of image coordinates to geographic coordinates”; Fu, [0113] “the control system 130 determines depth information from accessed images that are captured by a single image sensor and contain visual information representing depth. … determine depth information from other images, combination of images, etc. including visual information representing depth. For example, can determine depth from a combination of stereo images”; [0159] “the control system 130 accesses labelled depth information in the labelled depth map 760 and applies a coordinate system. The applied coordinate system may be based on any of the positions of the farming machine 100 in the field, the location of the camera on the farming machine 100, and the field of view of the camera. By applying the coordinate system, every pixel in a labelled depth map 460 is assigned a coordinate that corresponds to a real-world location in a field where the farming machine 100 is operating”; Fu, [0160] “The control system 130 combines the coordinates of pixels in the labelled depth map 460 with their corresponding depth information to generate a point cloud. That is, given extrinsic and intrinsic calibration parameters of the camera(s) that acquired the image (e.g., accessed image 800), it is possible to convert the 2D representation of the depth map (e.g., depth map 840) to a 3D representation of a point cloud.. More specifically, to generate the point cloud the control system 130 projects a ray from a model camera location into a representation of the real-world space in the accessed image”; Fu, [0171] “each of the compute units of each component treatment modules can associate local and global pose. While the compute unit and sensors associated with the navigation unit performs slam and accounts for a vehicle pose (the vehicles components' own SLAM), for navigation purposes”; Fu, [0110] describes that camera position and orientation are used to capture images of a plant as farming machine travels; see Fu, [0023], [0076] and [0105] Fu describes that the control system uses orientation of the farming machine and sensors as feature information for treatment mechanism); and
performing a field operation within the agricultural field based on the position within the global coordinate system for the feature of interest (Fu, [0085] “FIGS. 1A-1B, the farming machine 100a includes a single treatment mechanism, wherein the treatment mechanism 120 is actuated or the farming machine 100a moved to align the treatment mechanism 120 active area 122 with the targeted plant 102”; Fu, [0186] “he control system 130 can determine a data structure representing an orientation of the farming machine in the field (e.g., a vector, a path of coordinates, etc.). …the control system 130 may determine the orientation of the farming machine by comparing the direction of a field row to a known orientation of the camera. The control system 130 can determine other characteristics about the farming machine based on points in the point cloud. Some examples include, the angle of the ground plane compared to a known orientation of the camera, the height of a farming machine part relative the ground (e.g., a boom, or sprayer), the angle of a farming implement (e.g., a sprayer) relative to the ground”).
Sibley and Fu are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing for agricultural activities. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Sibley with the invention of Fu to make the invention that combines depth data associated with the imagery, pixel location of the feature of interest from the imagery, the camera position, the camera orientation, the position of the agricultural machine within the global coordinate system, and the orientation of the agricultural machine to locate the feature of interest by providing visually derived coordinates within the global coordinate system; doing so can generate a global map with every single unique and individual agricultural object sensed, and indexed with high precision in the global map and provide a precise trajectory for the treatment to be deposited on a surface of the agricultural object (Fu, [0171] and [0192]); thus one of the ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 16, Sibley in view of Fu teaches the method of claim 15, wherein the field operation is performed by the agricultural machine (Sibley, [0093] The agricultural scene can also include an agricultural observation and treatment system 311, supported by an example vehicle 310, performing observations and actions in the agricultural scene. In one example, the vehicle 310 can travel inside an orchard along a path 312 such that the agricultural observation and treatment system 311 can sense, identify, perform actions on specific agricultural objects 302 in real time”; Sibley, [0151] “vehicle 610 may include functionalities and/or structures of a truck, such as a pick-up truck (or any other truck), an all-terrain vehicle (“ATV”), a utility task vehicle (“UTV”), or any multipurpose off-highway vehicle, including any agricultural vehicle, including tractors or the like. The treatment systems 612 that may be powered or pulled separately by a vehicle, which may navigate paths autonomously in the geographic boundary”).
Regarding Claim 17, Sibley in view of Fu teaches the method of claim 15, wherein the field operation is performed by a machine different from the agricultural machine (Sibley, [0151] “vehicle 610 may include functionalities and/or structures of a truck, such as a pick-up truck (or any other truck), an all-terrain vehicle (“ATV”), a utility task vehicle (“UTV”), or any multipurpose off-highway vehicle, including any agricultural vehicle, including tractors or the like. The treatment systems 612 that may be powered or pulled separately by a vehicle, which may navigate paths autonomously in the geographic boundary”).
Regarding Claim 18, Sibley in view of Fu teaches the method of claim 15, wherein the agricultural machine comprises at least one of an agricultural vehicle and an agricultural implement and wherein the feature of interest comprises a plant (Sibley, [0092] “a plant such as a tree 303 can include a trunk, root, branch, stems, leaves, pedals, flowers, plant pistils and stigma, buds, fruitlets, fruits, and many other portions of a plant that make up the plant's anatomy, all of which can be agricultural objects of interest for observation and treatment”; Sibley, [0151] “vehicle 610 may include functionalities and/or structures of a truck, such as a pick-up truck (or any other truck), an all-terrain vehicle (“ATV”), a utility task vehicle (“UTV”), or any multipurpose off-highway vehicle, including any agricultural vehicle, including tractors or the like. The treatment systems 612 that may be powered or pulled separately by a vehicle, which may navigate paths autonomously in the geographic boundary”).
Claims 6 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Sibley et al. (US 20220183208 A1; hereafter referred to as Sibley) in view of Fu et al. (US 20210090274 A1; hereafter referred to as Fu) further in view of Anagnostopoulos (US 11315258 B1; hereafter referred to as Anagnostopoulos).
Regarding Claim 6, Sibley in view of Fu teaches the method of claim 1 and identifying the objects of interest with markers (Sibley, [0102] “The images can include images captured a few meters away from the physical surface and position of agricultural objects in the geographic boundary, which can include images of a plurality of individual agricultural objects, that are potential crops, as well as landmarks including objects or scenery, or other objects of interest including calibration targets and markers or other farming equipment, devices, structures, or machinery typically found on a farm that can be detected for localization of the treatment system and tracking objects in real time”; [0194] “The features of interest in the bigger image captured by image sensors can include agricultural objects, landmarks, scenes or other objects of interest to be identified, labelled, and assigned a unique identifier or marker to be indexed”), but does not explicitly recite:
wherein the feature of interest is a known in-field fiducial and wherein the method further comprises comparing the visually derived coordinates of the known in-field fiducial to surveyed coordinates for the known in-field fiducial and correcting the visually derived coordinates within the global coordinate system.
In the same field of endeavor, Anagnostopoulos teaches:
wherein the feature of interest is a known in-field fiducial and wherein the method further comprises comparing the visually derived coordinates of the known in-field fiducial to surveyed coordinates for the known in-field fiducial and correcting the visually derived coordinates within the global coordinate system (Anagnostopoulos, col.14, lines 29–- 37,“" comparison of imaged fiducial(s) relative to reference position(s), lines, axes, or other references in camera space; to calibration or registration between camera space and physical space to allow indication or estimation of position of the fiducial(s) or landmark(s) in physical space with their representations in camera space. In one example, the calibration uses a camera space reference position or coordinate system that is correlated to physical space”; Anagnostopoulos, col. 33, lines 53 – 63, “This could include calibration of the captured visual fiducials 30A′ and B′ in camera space 21 into physical space coordinates can allows quantifying the misalignment of implement 12 relative to tractor 14 in physical space with enough accuracy to be used as inputs to, for example, a precision ag auto steeling system. This can allow automatic compensation for undesirable implement misalignment by compensation through corrective auto steering action, both tractor (passive steering of the implement); and implement steering, which is active implement steering”; Anagnostopoulos, col.22, lines 44 – 54, “the imaged fiducials are presented to an observer on a display of the camera space in a manner that allows human observation of the imaged fiducials/landmarks relative to camera space reference(s) or coordinates. This, in turn, provides the observer an immediate understanding of position of those fiducials/landmarks, and thus the object(s) on which the fiducials/landmarks are mounted, in physical space. In another example, by image recognition and processing, imaged fiducials in camera space can be translated to physical space coordinates”).
Sibley, Fu and Anagnostopoulos are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing for agricultural activities. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Sibley in view of Fu with the invention of Anagnostopoulos to make the invention that compares the visually derived coordinates of the known in-field fiducial to surveyed coordinates for the known in-field fiducial and correcting the visually derived coordinates within the global coordinate system; doing so allows quantifying the misalignment of implement relative to tractor in physical space with enough accuracy and can allow automatic compensation for undesirable implement misalignment by compensation through corrective auto steering action and provides the observer an immediate understanding of position of those fiducials/landmarks, and thus the object(s) on which the fiducials/landmarks are mounted, in physical space (Anagnostopoulos, col.22, lines 44 – 54); thus one of the ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 20, Sibley in view of Fu teaches the method of claim 15 and identifying the objects of interest with markers (Sibley, [0102] “The images can include images captured a few meters away from the physical surface and position of agricultural objects in the geographic boundary, which can include images of a plurality of individual agricultural objects, that are potential crops, as well as landmarks including objects or scenery, or other objects of interest including calibration targets and markers or other farming equipment, devices, structures, or machinery typically found on a farm that can be detected for localization of the treatment system and tracking objects in real time”; [0194] “The features of interest in the bigger image captured by image sensors can include agricultural objects, landmarks, scenes or other objects of interest to be identified, labelled, and assigned a unique identifier or marker to be indexed”), but does not explicitly recite:
wherein the feature of interest is a known in-field fiducial and wherein the method further comprises comparing the visually derived coordinates of the known in-field fiducial to surveyed coordinates for the known in-field fiducial and correcting the visually derived coordinates within the global coordinate system.
In the same field of endeavor, Anagnostopoulos teaches:
wherein the feature of interest is a known in-field fiducial and wherein the method further comprises comparing the visually derived coordinates of the known in-field fiducial to surveyed coordinates for the known in-field fiducial and correcting the visually derived coordinates within the global coordinate system (Anagnostopoulos, col.14, lines 29–- 37,“" comparison of imaged fiducial(s) relative to reference position(s), lines, axes, or other references in camera space; to calibration or registration between camera space and physical space to allow indication or estimation of position of the fiducial(s) or landmark(s) in physical space with their representations in camera space. In one example, the calibration uses a camera space reference position or coordinate system that is correlated to physical space”; Anagnostopoulos, col. 33, lines 53 – 63, “This could include calibration of the captured visual fiducials 30A′ and B′ in camera space 21 into physical space coordinates can allows quantifying the misalignment of implement 12 relative to tractor 14 in physical space with enough accuracy to be used as inputs to, for example, a precision ag auto steeling system. This can allow automatic compensation for undesirable implement misalignment by compensation through corrective auto steering action, both tractor (passive steering of the implement); and implement steering, which is active implement steering”; Anagnostopoulos, col.22, lines 44 – 54, “the imaged fiducials are presented to an observer on a display of the camera space in a manner that allows human observation of the imaged fiducials/landmarks relative to camera space reference(s) or coordinates. This, in turn, provides the observer an immediate understanding of position of those fiducials/landmarks, and thus the object(s) on which the fiducials/landmarks are mounted, in physical space. In another example, by image recognition and processing, imaged fiducials in camera space can be translated to physical space coordinates”).
Sibley, Fu and Anagnostopoulos are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing for agricultural activities. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Sibley in view of Fu with the invention of Anagnostopoulos to make the invention that compares the visually derived coordinates of the known in-field fiducial to surveyed coordinates for the known in-field fiducial and correcting the visually derived coordinates within the global coordinate system; doing so allows quantifying the misalignment of implement relative to tractor in physical space with enough accuracy and can allow automatic compensation for undesirable implement misalignment by compensation through corrective auto steering action and provides the observer an immediate understanding of position of those fiducials/landmarks, and thus the object(s) on which the fiducials/landmarks are mounted, in physical space (Anagnostopoulos, col.22, lines 44 – 54); thus one of the ordinary skill in the art would have been motivated to combine the references.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 20240065131 A1 Agricultural Lane Following Systems and methods for agricultural lane following are described. For example, a method includes accessing range data captured using a distance sensor connected to a vehicle and/or image data captured using an image sensor connected to a vehicle; detecting a crop row based on the range data and/or the image data to obtain position data for the crop row; determining, based on the position data for the crop row, a yaw and a lateral position of the vehicle with respect to a lane bounded by the crop row; and based on the yaw and the lateral position, controlling the vehicle to move along a length of the lane bounded by the crop row.
US 12231616 B2 Sparse And/or Dense Depth Estimation From Stereoscopic Imaging Implementations are described herein for performing depth estimation in the agricultural domain, including generating synthetic training data. In various implementations, one or more three-dimensional synthetic plants may be generated in in a three-dimensional space, wherein the one or more three-dimensional synthetic plants include homogenous and densely-distributed synthetic plant parts. The plurality of three-dimensional synthetic plants may be projected onto two-dimensional planes from first and second perspectives in the three-dimensional space to form a pair of synthetic stereoscopic images.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VAISALI RAO KOPPOLU whose telephone number is (571)270-0273. The examiner can normally be reached Monday - Friday 8:30 - 5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format.
For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
VAISALI RAO. KOPPOLU
Examiner
Art Unit 2664
/VAISALI RAO KOPPOLU/Examiner of Art Unit 2664
