DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Summary
The Amendment filed on 12 March 2026 has been acknowledged.
Claims 1, 8 and 15 have been amended.
Currently, claims 1 – 20 are pending and considered as set forth.
Response to Arguments
Applicant’s arguments with respect to claims 1 – 20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1 – 6, 8 – 13 and 15 – 19 are rejected under 35 U.S.C. 103 as being unpatentable over Koivisto et al. (Hereinafter Koivisto) (US 2019/0258878 A1) in view of Sibley et al. (Hereinafter Sibley) (US 2022/0118555A1)
As per claim 1, Koivisto teaches limitations of:
generating one or more data points from one or more sensors coupled to a vehicle, wherein:
the one or more sensors (See at least paragraph 2; To do so, an object detector may be used to accurately detect objects depicted in an image(s) in real-time (e.g., images captured using one or more sensors mounted on the autonomous vehicle).) comprise:
a Light Detection and Ranging (LiDAR) sensor (See at least paragraph 41; This may include features within the object bounding box, such as a histogram of oriented gradients. Other examples of factors include an Internal Measurement Unit (IMU) output(s) that corresponds to an orientation of the vehicle, and distance or Three-Dimensional (3D) location estimates of the object, which may be determined by the CNN and/or other sensors, such as LIDAR or RADAR.); and
a camera (See at least paragraph 57; sensor data received by the communications manager 104 may be generated using any combination of the sensors 1480 of FIG. 14. For example, the sensor data may include image data representing an image(s), image data representing a video (e.g., snapshots of video), and/or sensor data representing fields of view of sensors (e.g., LIDAR data from LIDAR sensor(s) 1564, RADAR data from RADAR sensor(s) 1560, image data from a camera(s) of FIG. 15B, etc.).), and
the one or more data points comprise:
a LiDAR point cloud generated by the LiDAR sensor (See at least paragraph 41; Example factors include parameters such as a number of frames in which the object has been detected, covariance of the tracked object state estimate, statistics on feature points tracked within an object bounding box, optical flow estimates within an object bounding box, correlation with a model of object appearance, estimates of object kinematics, and/or an object trajectory. Other factors include features computed directly from the image data (e.g., from points or pixels) or from the features of one or more primary CNN layers of a CNN used to provide the detected objects, such as the last layer before a layer that predicts detected object locations (e.g., a gridbox output).; and
an image captured by the camera (See at least paragraph 57 and 83; As an example, the object tracker 114 may use the confidence scores and/or aggregated object data to fuse or merge detections of objects (e.g., using the process of FIG. 1B) derived from one or more sensors and/or sensor types (e.g., a camera or array of cameras producing image data) with detections of objects (e.g., using the process of FIG. 1B) determined from one or more other sensors and/or sensor types (e.g., LIDAR and/or RADAR sensors).), and
using a processor:
detecting one or more obstacles within the LiDAR point cloud (See at least paragraph 41);
generating a patch for each of the one or more obstacles (See at least paragraph 41);;
projecting the LiDAR point cloud into the image, wherein each patch represents a region of the image for each of the one or more obstacles (See at least paragraph 41 and 83; the object tracker 114 may be used to track objects and/or detected objects (e.g., aggregated detected objects) across frames (e.g., video frames) and/or images, such as in a time-domain. For example, the object tracker 114 may determine at least a first detected object and a second detected object are the same object depicted across sequential frames (e.g., consecutive frames in time) represented by the sensor data and/or image data. This may be based, at least in part, on the confidence scores associated with the aggregated detected objects. As another example, the object tracker 114 may determine at least a first detected object and a second detected object are the same object captured by different sensors (e.g., of the vehicle 1500) and/or in different images (e.g., and merge or fuse the detections) represented by the sensor data and/or image data. As an example, the object tracker 114 may use the confidence scores and/or aggregated object data to fuse or merge detections of objects (e.g., using the process of FIG. 1B) derived from one or more sensors and/or sensor types (e.g., a camera or array of cameras producing image data) with detections of objects (e.g., using the process of FIG. 1B) determined from one or more other sensors and/or sensor types (e.g., LIDAR and/or RADAR sensors));;
for each of the one or more obstacles, performing a query on the image (See at least paragraph 117; One such example of a statistic that may be computed for a detected object region includes a HOG. This may be computed (e.g., directly) from corresponding elements of the last CNN before the output layer(s) 330 and/or from corresponding input pixels to the object detector 106. Statistics and/or features (e.g. non-statistical) for detected object regions (or aggregated detected object regions) may correspond to color, chroma, luma, pixel, and/or other data values for those regions.);
wherein, for each obstacle, performing the factor query comprises:
performing a shape query to determine one or more shape features of the obstacle (See at least abstract and paragraph 9; detected object data representative of locations of detected objects in a field of view may be determined. One or more clusters of the detected objects may be generated based at least in part on the locations and features of the cluster may be determined for use as inputs to a machine learning model(s). A confidence score, computed by the machine learning model(s) based at least in part on the inputs, may be received, where the confidence score may be representative of a probability that the cluster corresponds to an object depicted at least partially in the field of view. Further examples provide approaches for determining ground truth data for training object detectors, such as for determining coverage values for ground truth objects using associated shapes, and for determining soft coverage values for ground truth objects. … a size of a shape that is at least partially within an object region (e.g., ground truth bounding box region) is computed using a dimension of the object region. The shape may then be used to determine coverage values for spatial element regions of the training image used to train the CNN (e.g., by drawing at least some of the shape in a ground truth tensor). Provided approaches may further allow for determining a coverage value for a spatial element region when the spatial element region corresponds to multiple object regions (indicating different objects). In further respects, a dead-zone area may be used to spatially separate coverage values that correspond to different object regions (indicating different objects). This may train the CNN to better distinguish between adjacent objects. For example, the CNN trained in this manner may provide lower coverage values for areas between adjacent objects.): and
performing a color query to determine one or more color features of the obstacle (See at least paragraph 117; One such example of a statistic that may be computed for a detected object region includes a HOG. This may be computed (e.g., directly) from corresponding elements of the last CNN before the output layer(s) 330 and/or from corresponding input pixels to the object detector 106. Statistics and/or features (e.g. non-statistical) for detected object regions (or aggregated detected object regions) may correspond to color, chroma, luma, pixel, and/or other data values for those regions.);
for each of the one or more obstacles, based on the query, determining a label for the obstacle (See at least paragraph 117 - 127; The object detector 106 (or the object detector 306) may be trained using various possible approaches. In some examples, the object detector 106 may be trained in a fully supervised manner Training images together with their labels may be grouped in minibatches, where the size of the minibatches may be a tunable hyperparameter, in some examples.); and
for each of the one or more obstacles, labeling the obstacle with the label (See at least paragraph 59, 75, 117 and 127; the image 204 depicts regions of the environment 202, where the regions may include any number of objects, examples of which include objects 248A, 248B, 248C, and 248D, which are labeled in FIG. 2B. The objects may comprise any combination of vehicles, people (e.g., pedestrians), motorcycles, bicyclists, trees, animals, buildings, signs, structures, and/or other objects within the environment 202. … Outputs of the clustering algorithm may include labels of the locations of the detected objects, such that all locations that are clustered together may have the same label.).
Koivisto teaches labeling for each of the one or more obstacles, determining shape features and color features of the obstacles (See rejections above) but does not explicitly teach the limitation of:
wherein, during the factor query, the one or more shape features and the one or more color features are queried for each of one or more points of the image;
based in whole or in part on results of the factor query, labeling each of the one or more points;
for each of the one or more obstacles, based on the labels of each of the one or more points, determining a label for the obstacle; and
for each of the one or more obstacles, labeling the obstacle with the label,
wherein the labeling comprises, for each of the one or more obstacles, determining, based on the one or more shape features and the one or more color features, whether the obstacle is a piece of vegetation.
Sibley teaches the limitations of:
wherein, during the factor query, the one or more shape features and the one or more color features are queried for each of one or more points of the image (See at least paragraph 105; . The image 352 can also include the specific image patch extracted from the images captured by the treatment system, instead of a constructed image depicting the shape, size, color, or other unique attributes of the object 353-a at the time the observation and treatment system last observed the object 353-a . In one example, the image 352 can be a pixelated 2d or 3d image that represents a model of the specific object 353-a in the real world at a stage of growth, or state, detected by the treatment system. The 2d or 3d model can be generated by using various computer vision techniques by associating multiple views of the object along with depth and/or motion data of the image capture device. Some objects may be occluded such that an image sensor travelling along a path guided by a vehicle may not capture the entire view of the object detected.);
based in whole or in part on results of the factor query, labeling each of the one or more points;
for each of the one or more obstacles, based on the labels of each of the one or more points, determining a label for the obstacle; and
for each of the one or more obstacles, labeling the obstacle with the label (See at least paragraph 107; the agricultural observation and treatment system 311 may have logged information related to object 353-a before it was labelled as a fruitlet in its most recent timed log. The system may have detected the same object 353-a , at the same or near location, when it was detected with an identifier of a bud associated with the object. And then at a later trial detecting the same object 353-a again, but before it was detected and labelled with the identifier of fruitlet but after it was detected and labelled as a bud, another detection with the identifier, for example, of a flower/blossom. Each of these detections can have location position of, or near location position of (x.sub.1, y.sub.1, z.sub.1). In this case, the system can associate the different identifications of the same object, based on the objects state changes, or stage of growth or phenological changes, and display, via a series of views across time, the state change in sequence in the user interface 350. Identifying, storing and indexing, and associating portions of images and patches and other sensor readings of objects of the same type with near or the same locations of the same objects identified throughout time from different trials and identifying with different states of the same object in the geographic scene can be performed using various techniques including machine learning feature extraction, detection, and or classification to detect and identify objects in a given image frame as well as generating keyframes based on the objects and landmarks detected. The keyframes can be determined to more efficiently identify and index objects in a frame while reducing redundancy, for example, by identifying common and/or the same landmarks across multiple frames. The machine learning and other various computer vision algorithms can be configured to draw bounding boxes to label portions of images with objects of interest and background, masking functions to separate background and regions of interest or objects of interest, perform semantic segmentation to all pixels or a region of pixels of an given image frame to classify each pixel as part of one or more different target objects, other objects of interest, or background and associate its specific location in space relative to the a component of the treatment system and the vehicle supporting the treatment system),
wherein the labeling comprises, for each of the one or more obstacles, determining, based on the one or more shape features and the one or more color features, whether the obstacle is a piece of vegetation (See at least paragraph 108; the agricultural observation and treatment system 311 can perform functions to associate portions of image, for example image patches of objects, image frames, key frames, or a combination thereof from different trials where the agricultural observation and treatment system 311 observed, identified, labelled, and stored information about the same object across multiple states and phenological stages. Additionally, the association of the frames or portion of the frames can be packaged into a series of image frames that can be displayed in sequence as a video displaying the growth, or backwards growth depending on the direction of displaying the images, of the specific object. For example, the series of indexed images or patches of images associated with each other throughout time can be displayed in the user interface 350 in the video or visual time lapse history 356. In one example, the functions can be performed by various computer vision and machine learning techniques including image to image correspondence including template matching and outlier rejection, performed by various techniques including RANSAC, k-means clustering, or other feature-based object detection techniques for analyzing a series of images frames, or a combination thereof. In one example, the above techniques can also be used to generate key frames of a subsequent trial by comparing frames from the subsequent trial with keyframes of one or more prior trials, depending on how many prior trials there are. Additionally, the comparison and the candidate frames or keyframes from a previous trial that may be accessed by the agricultural observation and treatment system 311, or at a server offline, to be used to perform comparisons to identify state and phenological stage change of a same object, such as object 353-a , can be narrowed down for selection based on location data logged at the time of capture, pose data logged at the time of capture, or a combination thereof associated with each of the keyframes, or objects detected in each keyframe. These accessed and selected frames or key frames in the prior trials, having been selected based on its location data associated with the frames or objects detected in the frames, can be used to compare with currently captured frames, or subsequently captured frames from the prior frames, having similar location data associated with the selected frames for key frames to match objects that may have different labels, since different states or phenological stages will have different labels due to the states having different shape, color, size, density, etc. If there is a match, or if there is a threshold reached based on the comparison of the accessed frame or keyframe against one or more frames in a subsequently captured series of frames, the agricultural observation and treatment system 311 can determine that the two, or more, objects of different types and identifiers associated with each of the objects, are the same object and that one, with a first phenological stage, changed into the other having a second label or identifier of a second phenological stage.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to include wherein, during the factor query, the one or more shape features and the one or more color features are queried for each of one or more points of the image; based in whole or in part on results of the factor query, labeling each of the one or more points; for each of the one or more obstacles, based on the labels of each of the one or more points, determining a label for the obstacle; and for each of the one or more obstacles, labeling the obstacle with the label, wherein the labeling comprises, for each of the one or more obstacles, determining, based on the one or more shape features and the one or more color features, whether the obstacle is a piece of vegetation as taught by Sibley in the system of Koivisto, since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
As per claim 2, Koivisto teaches limitations of:
wherein the label comprises one or more of: a piece of vegetation; a pedestrian; and a vehicle (See at least paragraph 59).
As per claim 3, Koivisto teaches limitations of:
using the processor:
for each of the one or more obstacles, based on the label of the obstacle, determining one or more vehicle actions for the vehicle to perform; and causing the vehicle to perform the one or more actions (See at least paragraph 285 and 297).
As per claim 4, Koivisto teaches limitations of:
wherein the one or more actions comprises one or more of:
increasing a speed of the vehicle;
decreasing a speed of the vehicle; stopping the vehicle; and
adjusting a trajectory of the vehicle (See at least paragraph 285 and 297).
As per claim 5, Koivisto teaches limitations of:
wherein each patch forms a bounding box on the image (See at least paragraph 37), and further comprising:
cropping the region of the image within the bounding box, forming a cropped image (See at least paragraph 37 and 157).
As per claim 6, Koivisto teaches limitations of:
resizing the cropped image, forming a resized image, wherein performing the color query comprises performing the color query on the resized image (See at least paragraph 117 and 157) .
Regarding claims 8 – 13 and 15 – 19:
Claims 8 – 13 and 15 – 19 are rejected using the same rationale, mutatis mutandis, applied to claims 1 – 6 above, respectively.
Claims 7, 14 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Koivisto and Sibley in view of Lee et al. (Hereinafter Lee) (US 2021/0370928 A1).
As per claim 7, the combination of Koivisto and Sibley teaches all the limitations of the claimed invention but does not explicitly teach limitations of:
for each of the one or more obstacles, based on the label of the obstacle, determining:
whether the obstacle is an obstacle that the vehicle can hit; and
whether the obstacle is not an obstacle that the vehicle that the vehicle cannot hit.
Lee teaches limitations of:
for each of the one or more obstacles, based on the label of the obstacle, determining:
whether the obstacle is an obstacle that the vehicle can hit; and
whether the obstacle is not an obstacle that the vehicle that the vehicle cannot hit (See at least paragraph 120 – 122).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to include determining: whether the obstacle is an obstacle that the vehicle can hit; and whether the obstacle is not an obstacle that the vehicle that the vehicle cannot hit as taught by Lee in the system of Koivisto and Sibley, since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Regarding claims 14 and 20:
Claims 14 and 20 are rejected using the same rationale, mutatis mutandis, applied to claim 7 above, respectively.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to IG T AN whose telephone number is (571)270-5110. The examiner can normally be reached M - F: 10:00AM- 4:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Aniss Chad can be reached at (571) 270-3832. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
IG T AN
Primary Examiner
Art Unit 3662
/IG T AN/Primary Examiner, Art Unit 3662