DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1 – 5, and 9 - 20 are rejected under 35 U.S.C. 103 as being unpatentable over Dal Mutto et al. (Publication: US 2019/0108396 A1) in view of Ostrowski et al. (Publication: US 2023/0215048 A1).
Regarding claim 1, Dal Mutto discloses an apparatus comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to perform ([0088] - The depth camera system 100 shown in FIG. 2 includes a first camera 102, a second camera 104, a projection source 106 (or illumination source or active projection system), and a host processor 108 and memory 110 with software, wherein the host processor may be, a graphics processing unit (GPU), perform the following: ):
process a first image to determine a first set of image coordinates corresponding to one or more first features of one or more first objects ([0214] In operation 2008, the 3-D model analysis module 350 detects locations of features in the regions of the regions of the 3-D multi-view model. The features may be pre-defined by the operator as items of interest within the shape data (e.g., three dimensional coordinates) and texture data (e.g., surface color information) of the 3-D multi-view model and the reference model. aspects of the features may relate to geometric shape, geometric dimensions and sizes, surface texture and color. One example of a feature is a logo on the side of a shoe. The logo may have a particular size, geometric shape, surface texture, (although there might be some variation between different instances of the same item or good), surface or shape is boundaries.);
process a second image to determine a second set of image coordinates corresponding to one or more second features of one or more second objects ([0214] In operation 2008, the 3-D model analysis module 350 detects locations of features in the regions of the regions of the 3-D multi-view model. The features may be pre-defined by the operator as items of interest within the shape data (e.g., three dimensional coordinates) and texture data (e.g., surface color information) of the 3-D multi-view model and the reference model. aspects of the features may relate to geometric shape, geometric dimensions and sizes, surface texture and color. One example of a feature is a logo on the side of a shoe. The logo may have a particular size, geometric shape, surface texture, (although there might be some variation between different instances of the same item or good), surface or shape is boundaries.);
determine an object size, an object pose, or a combination thereof based on the first set of image coordinates ([0214] In operation 2008, the 3-D model analysis module 350 detects locations of features in the regions of the regions of the 3-D multi-view model. The features may be pre-defined by the operator as items of interest within the shape data (e.g., three dimensional coordinates) and texture data (e.g., surface color information) of the 3-D multi-view model and the reference model. aspects of the features may relate to geometric shape, geometric dimensions and sizes, surface texture and color.) ,
the second set of image coordinates ([0117] If depth images are captured by the depth cameras 100 at different poses (e.g., different locations with respect to the target object 10), then it is possible to acquire data regarding the shape of a larger portion of the surface of the target object 10 than could be acquired by a single depth camera through a point cloud merging module 210 (see FIG. 7) of a 3-D model generation module 200 that merges the separate depth images (represented as point clouds) 14 into a merged point cloud 220. For example, opposite surfaces of an object (e.g., the medial and lateral sides of the boot shown in FIG. 7) can both be acquired, whereas a single camera at a single pose could only acquire a depth image of one side of the target object at a time. The multiple depth images can be captured by moving a single depth camera over multiple different poses or by using multiple depth cameras located at different positions.); and
provide the object size, the object pose, or a combination thereof as an output ( [0114] the extrinsic parameters of the depth cameras 100 (e.g., relative poses) are estimated through another calibration step, in which a calibration target (e.g., an object of known size with identifiable and precisely detectable features, such as a black-and-white 2-D checkerboard) is acquired by all the depth cameras.).
Dal Mutto does not however Ostrowski discloses
semantic features([0118] In comparison, FIG. 6C illustrates an example of an image captured by a second camera mounted on the farming machine in an incorrect orientation, in accordance with an example embodiment. The second camera is positioned adjacent to the first camera described with reference to FIG. 6B such that the object 630 is also in the field of view of the second camera. Thus, the second camera image 650 includes visual information that also identifies the object 630. However, in the image 650, the object 630 is not oriented in the proper upright position illustrated in the image 610. Thus, the position and orientation characteristics extracted from the second camera image 650 will indicate that the second camera is also not oriented in the same upright position as the first camera. Based on the differing orientations of the object 630 in the image 610 and the image 650, the relative pose between the first camera and the second camera can be determined, the calibration error between the first camera and the second camera can be identified.
Semantic meaning in computer graphics refers to the high-level interpretation and context of visual data, focusing on what objects in a scene represent as described above. );
determine an object size, an object pose, or a combination thereof based on the first set of image coordinates, the second set of image coordinates, and a camera pose change between the first image and the second image ([0118] FIG. 6C illustrates an example of an image captured by a second camera mounted on the farming machine in an incorrect orientation. The second camera is positioned adjacent to the first camera described with reference to FIG. 6B such that the object 630 is also in the field of view of the second camera. Thus, the second camera image 650 includes visual information that also identifies the object 630. However, in the image 650, the object 630 is not oriented in the proper upright position illustrated in the image 610. Thus, the position and orientation characteristics extracted from the second camera image 650 will indicate that the second camera is also not oriented in the same upright position as the first camera. Based on the differing orientations of the object 630 in the image 610 and the image 650, the relative pose between the first camera and the second camera can be determined, the calibration error between the first camera and the second camera can be identified, “determine an object size, an object pose, or a combination thereof based on the first set of image coordinates, the second set of image coordinates, and a camera pose change between the first image and the second image”.
[0001] - to identifying a calibration error between a pair of cameras on the farming machine based on a relative pose between the pair of cameras.
[0056] - calibration error(s) may be identified by applying the techniques to compare the relative poses of one or more pairs of image sensors. The relative poses of each pair of sensors may be chained and the calibration error(s) for the array of sensors may be determined by applying regression or finite element techniques to the change of poses.
).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Dal Mutto with semantic features; determine an object size, an object pose, or a combination thereof based on the first set of image coordinates, the second set of image coordinates, and a camera pose change between the first image and the second image as taught by Ostrowski. The motivation for doing is to remedy the calibration error so the cameras can obtain an accurate image as taught by Ostrowski.
Regarding claim 2, Dal Mutto in view of Ostrowski disclose all the limitation of claim 1.
Dal Mutto discloses determine a consistency of the object size, the object pose, or a combination thereof with one or more geometric constraints ([0083] circumstances in which the items to be identified may be characterized by their surface colors and geometry, including the size of the object (although there might be some variation between different instances of the same item or good). this type color and shape of information can be used to automate the identification of different items (e.g., identifying retail goods during checkout). One component of automated object identification systems is a 3-D scanning system that is able to acquire geometry and color information. Because of the volumetric nature of common goods.); and
determine whether the one or more first objects and the one or more second objects are a same object or different objects based on the consistency ([0083] circumstances in which the items to be identified may be characterized by their surface colors and geometry, including the size of the object (although there might be some variation between different instances of the same item or good).).
Regarding claim 3, Dal Mutto in view of Ostrowski disclose all the limitation of claim 2.
Dal Mutto discloses wherein the one or more geometric constraints include a maximum object size, a maximum distance from a camera location, or a combination thereof ([0096] - To detect the depth of a feature in a scene imaged by the cameras, the depth camera system determines the pixel location of the feature in each of the images captured by the cameras. The distance between the features in the two images is referred to as the disparity, which is inversely related to the distance or depth of the object. (This is the effect when comparing how much an object “shifts” when viewing the object with one eye at a time—the size of the shift depends on how far the object is from the viewer's eyes.
[0097] - The magnitude of the disparity between the master and slave cameras depends on physical characteristics of the depth camera system, distance between the cameras and the fields of view of the cameras.
[0104] - depth measurements can be made of distant objects (e.g., to overcome the diminishing of the optical illumination over the distance to the object, by a factor proportional to the inverse square of the distance)).
Regarding claim 4, Dal Mutto in view of Ostrowski disclose all the limitation of claim 1.
Dal Mutto discloses generate a three-dimensional map based on the output ([0114] the extrinsic parameters of the depth cameras 100 (e.g., relative poses) are estimated through another calibration step, in which a calibration target (e.g., an object of known size with identifiable and precisely detectable features, such as a black-and-white 2-D checkerboard) is acquired by all the depth cameras, in order to detect the relative rotation and translation between each of the scanner composing the 3-D modeling system. Accordingly, the extrinsic parameters can be used to compute or to estimate the transformations that may be applied to the separate depth maps (e.g., 3-D point clouds) captured by the different depth cameras in order to merge the depth maps to generate the captured 3-D model of the object.).
Regarding claim 5, Dal Mutto in view of Ostrowski disclose all the limitation of claim 1.
Dal Mutto discloses wherein the one or more first features, the one or more second features, or a combination thereof include one or more boundaries of the one or more first objects or the one or more second objects ([0214] In operation 2008, the 3-D model analysis module 350 detects locations of features in the regions of the regions of the 3-D multi-view model. The features may be pre-defined by the operator as items of interest within the shape data (e.g., three dimensional coordinates) and texture data (e.g., surface color information) of the 3-D multi-view model and the reference model. aspects of the features may relate to geometric shape, geometric dimensions and sizes, surface texture and color. One example of a feature is a logo on the side of a shoe. The logo may have a particular size, geometric shape, surface texture, (although there might be some variation between different instances of the same item or good), surface or shape is boundaries.).
Regarding claim 9, Dal Mutto in view of Ostrowski disclose all the limitation of claim 1.
Dal Mutto discloses detect one or more known objects in the first image, the second image, or a combination thereof; and
determine a first camera pose of the first image, a second camera pose of the second image, or a combination thereof by using semantic visual localization based on the one or more detected known objects ([0118] FIG. 6C illustrates an example of an image captured by a second camera mounted on the farming machine in an incorrect orientation. The second camera is positioned adjacent to the first camera described with reference to FIG. 6B such that the object 630 is also in the field of view of the second camera. Thus, the second camera image 650 includes visual information that also identifies the object 630. However, in the image 650, the object 630 is not oriented in the proper upright position illustrated in the image 610. Thus, the position and orientation characteristics extracted from the second camera image 650 will indicate that the second camera is also not oriented in the same upright position as the first camera. Based on the differing orientations of the object 630 in the image 610 and the image 650, the relative pose between the first camera and the second camera can be determined, the calibration error between the first camera and the second camera can be identified. Semantic meaning in computer graphics refers to the high-level interpretation and context of visual data, focusing on what objects in a scene represent as described above.),
wherein the camera pose change is based on a first camera pose, the second camera pose, or a combination thereof (
[0056], [0118] the relative poses of each pair of sensors may be chained and the calibration error(s), based on the position of each camera , for the array of sensors may be determined by applying regression or finite element techniques to the change of poses. The relative pose between the first camera and the second camera can be determined, the calibration error between the first camera and the second camera can be identified.
[0102] The calibration error module 356 may adjust the orientation of each camera of the array 310 to align the first and second cameras within a minimal error of each other.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Dal Mutto with
detect one or more known objects in the first image, the second image, or a combination thereof; and determine a first camera pose of the first image, a second camera pose of the second image, or a combination thereof by using semantic visual localization based on the one or more detected known objects; wherein the camera pose change is based on a first camera pose, the second camera pose, or a combination thereof as taught by Ostrowski. The motivation for doing is to remedy the calibration error so the cameras can obtain an accurate image as taught by Ostrowski.
Regarding claim 10, Dal Mutto in view of Ostrowski disclose all the limitation of claim 1.
Dal Mutto discloses wherein the output is enhanced with additional object metadata, visual data, or a combination thereof ([0266] - retrieved metadata about expected characteristics of the class of the object; physical measurements of the object (e.g., dimensions, locations of surface features of the object); and one or more result 3-D models that depict the locations of detected features (e.g., each 3-D model may depict a different type of feature or a 3-D model may depict multiple different types of features).).
Regarding claim 11, Dal Mutto in view of Ostrowski disclose all the limitation of claim 1.
Dal Mutto discloses wherein the additional object metadata, the visual data, or a combination thereof is used to render a representation of the one or more first objects, the one or more second objects, or a combination thereof ([0266] - retrieved metadata about expected characteristics of the class of the object; physical measurements of the object (e.g., dimensions, locations of surface features of the object); and one or more result 3-D models that depict the locations of detected features (e.g., each 3-D model may depict a different type of feature or a 3-D model may depict multiple different types of features).).
Regarding claim 12, see rejection on claim 1.
Regarding claim 13, see rejection on claim 2.
Regarding claim 14, see rejection on claim 3.
Regarding claim 15, see rejection on claim 4.
Regarding claim 16, see rejection on claim 5.
Regarding claim 17, see rejection on claim 1.
Regarding claim 18, see rejection on claim 2.
Regarding claim 19, see rejection on claim 3.
Regarding claim 20, see rejection on claim 4.
Claims 6, 7, and 8 are rejected under 35 U.S.C. 103 as being unpatentable over Dal Mutto et al. (Publication: US 2019/0108396 A1) in view of Zhang et al. (Publication: US 2023/0192121 A1).
Regarding claim 6, Dal Mutto in view of Ostrowski disclose all the limitation of claim 1.
Dal Mutto discloses the one or more second objects, or a combination thereof are one or more [[polyhedrons]] ([0214] In operation 2008, the 3-D model analysis module 350 detects locations of features in the regions of the regions of the 3-D multi-view model. The features may be pre-defined by the operator as items of interest within the shape data (e.g., three dimensional coordinates) and texture data (e.g., surface color information) of the 3-D multi-view model and the reference model. aspects of the features may relate to geometric shape, geometric dimensions and sizes, surface texture and color. One example of a feature is a logo on the side of a shoe. The logo may have a particular size, geometric shape, surface texture, (although there might be some variation between different instances of the same item or good), surface or shape is boundaries.); and
wherein the one or more first features, the one or more second features, or a combination thereof include one or more corners of one or more faces of the one or more [[polyhedrons]] ([0214] In operation 2008, the 3-D model analysis module 350 detects locations of features in the regions of the regions of the 3-D multi-view model. The features may be pre-defined by the operator as items of interest within the shape data (e.g., three dimensional coordinates) and texture data (e.g., surface color information) of the 3-D multi-view model and the reference model. aspects of the features may relate to geometric shape, geometric dimensions and sizes, surface texture and color. One example of a feature is a logo on the side of a shoe. The logo may have a particular size, geometric shape, surface texture, (although there might be some variation between different instances of the same item or good), surface or shape is faces.).
Zhang discloses polyhedrons ([0059] – polyhedrons.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Dal Mutto in view of Ostrowski with polyhedrons as taught by Zhang. The motivation for doing is to improve efficiency as taught by Zhang.
Regarding claim 7, Dal Mutto in view of Ostrowski disclose all the limitation of claim 1.
Dal Mutto discloses wherein the first image, the second image, or a combination thereof is processed using image segmentation that is trained to segment a first object
class ([0214] - the features are detected using a convolutional neural network (CNN) that is trained to detect a particular set of features that are expected to be encountered in the context of the product (e.g., logos, blemishes, stitching, shapes of various parts of the object, and the like).
[0156] - pixel-wise segmentation and classification are performed jointly on the objects in the scene (e.g., in view of the scanning system) at the same time. The term “semantic segmentation” may be used to refer to assigning an object class (or type) to each pixel in a scene. For example, in a scene that includes different types of fruit, some pixels may be labeled with an “apple” class, where those pixels are determined to represent a part of an apple, while other pixels that are determined to be a part of an orange may be labeled with an “orange” class. pixel-accurate image segmentation and semantic classification are performed based on a framework based on Fully Convolutional Neural Networks (FCNNs) and described in Song, S., S. P. Lichtenberg, and J. Xiao. Sun RGB-D: A RGB-D scene understanding benchmark suite. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015. In more detail, a training set is generated by randomly placing a set of 3-D models on a surface (e.g., using physics modeling engines to make sure that they respect fundamental physical properties such as gravity and to ensure that the objects do not interpenetrate or “clip” through each other), and then rendering such 3D models from multiple points of view, saving which class (e.g., which object or the type of the object) each pixel in the rendered image belongs to.) .
Dal Mutto in view of Ostrowski do not however Zhang discloses wherein the output is used to generate an initial map of objects in the first object class ([0065] The mapping engine 345 includes and/or uses one or more trained ML models 347 of one or more ML systems to identify the positions and/or poses of the objects bounded b the boundaries along the map of the environment.
[0066] In some examples, the mapping engine 345 outputs the map data 350. The map data 350 includes, positioned on the map, the objects defined by the boundaries in the bounded point data 340, by the clusters in the clustered point data 330, and/or by the classifications in the categorized point data 320.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Dal Mutto in view of Ostrowski with wherein the output is used to generate an initial map of objects in the first object class as taught by Zhang. The motivation for doing is to improve efficiency as taught by Zhang.
Regarding claim 8, Dal Mutto in view of Ostrowski, Zhang disclose all the limitation of claim 7.
Dal Mutto discloses objects in the first object class to semantically localize other objects in a second object class segmented by the image segmentation ([0030] At least one of the one or more 3-D models may include at least two objects, and the method may further include: segmenting the at least one 3-D model into a plurality of segments; and computing a descriptor for each of the segments.
[0156] - pixel-wise segmentation and classification are performed jointly on the objects in the scene (e.g., in view of the scanning system) at the same time. The term “semantic segmentation” may be used to refer to assigning an object class (or type) to each pixel in a scene. For example, in a scene that includes different types of fruit, some pixels may be labeled with an “apple” class, where those pixels are determined to represent a part of an apple, while other pixels that are determined to be a part of an orange may be labeled with an “orange” class. pixel-accurate image segmentation and semantic classification are performed based on a framework based on Fully Convolutional Neural Networks (FCNNs) and described in Song, S., S. P. Lichtenberg, and J. Xiao. Sun RGB-D: A RGB-D scene understanding benchmark suite. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015. In more detail, a training set is generated by randomly placing a set of 3-D models on a surface (e.g., using physics modeling engines to make sure that they respect fundamental physical properties such as gravity and to ensure that the objects do not interpenetrate or “clip” through each other), and then rendering such 3D models from multiple points of view, saving which class (e.g., which object or the type of the object) each pixel in the rendered image belongs to.)
Zhang discloses use the initial map of objects in the first object class to perform ([0064] The mapping engine 345 combines the bounded point data 340, the clustered point data 330, and/or the categorized point data 320 with a map of the environment to identify positions and/or poses (e.g., location and/or orientation) of the objects.);
update the initial map to generate an enhanced map including the other objects in the second object class ([0065] The mapping engine 345 includes and/or uses one or more trained ML models 347 of one or more ML systems to identify the positions and/or poses of the objects bounded b the boundaries along the map of the environment. The trained ML model(s) 347 receive the bounded point data 340, the clustered point data 330, and/or the categorized point data 320 as input(s). the trained ML model(s) 347 also receive the image data 312 and/or other sensor data from other sensor(s) 305 as input(s).
[0075] provides the feedback to one or more ML systems of the depth data processing system 300 as training data to update the one or more ML systems of the depth data processing system 300. For instance, the feedback engine 370 can provide the feedback as training data to the ML system(s) associated with the trained ML model(s) 317 of the semantic segmentation engine 315, the trained ML model(s) 327 of the clustering engine 325, the trained ML model(s) 337 of the boundary engine 335, the trained ML model(s) 347 of the mapping engine 345 thus “update the initial map” .).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Dal Mutto in view of Ostrowski, Zhang with use the initial map of objects in the first object class to perform; update the initial map to generate an enhanced map including the other objects in the second object class as taught by Zhang. The motivation for doing is to improve efficiency as taught by Zhang.
Response to Arguments
In view of the Pre-Brief Appeal Conference issued on 01/26/2026, PROSECUTION IS HEREBY REOPENED.
Claim Rejection Under 35 U.S.C. 103
Applicant’s argument have been rendered moot, as the reject is now based on newly cited reference that address the claimed subject matter.
Regarding claims 2 – 11, 13 – 16, and 18 – 20, the Applicant asserts that they are not obvious over based on their dependency from independent claims 1, 12, and 17 respectively. The examiner cannot concur with the Applicant respectfully from same reason noted in the examiner’s response to argument asserted from claims 1, 12, and 17 respectively.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MING WU whose telephone number is (571)270-0724. The examiner can normally be reached on Monday - Thursday and alternate Fridays: 9:30am - 6:00pm EST .
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona Faulk can be reached on 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MING WU/
Primary Examiner, Art Unit 2618
Prosecution Insights