DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendment filed 03/11/2026 has been entered. Claims 1, 3-16, 19, and 21-24 remain pending in the application, with claims 2, 17, 18, and 20 having been cancelled.
Response to Arguments
In light of the amendments to the claims, the double patenting rejection has been updated below.
Applicant’s arguments have been considered but are moot because the new ground of rejection does not rely on any combination of references applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Instant claims 1, 14, and 19 are rejected on the ground of nonstatutory double patenting as being unpatentable over patent claim 4 of U.S. Patent No. 11,846,091 B2 in view of Vesanen et al. (U.S. Patent No. 2020/0340214), hereinafter Vesanen.
Regarding instant claim 1, patent claim 4 discloses acquiring from a stereo camera an image of a marker on an object (Patent claim 1, which claim 4 is dependent on, teaches components of an implement, which is an object; see combination with Vesanen below wherein the components are markers; use of a stereo camera is an obvious variant known in the art for a 3D camera, also taught by Vesanen below); determining a position of the marker with respect to a local coordinate system of the stereo camera based on the image (patent claims 1 and 4); and determining a position of the object with respect to the local coordinate system of the stereo camera based on the position of the marker (Patent claim 1 teaches determining the position of the implement based on the position of the components; It is not specified that the position is in the local coordinate system, see combination below). However, patent claim 4 fails to disclose wherein the components are markers; wherein the markers comprise encoded information identifying the object and that the determined position of the object is with respect to the local coordinate system; and identifying a plurality of vertices of a 2D projection of the marker separately for left and right frames of the stereo camera and calculating 3D coordinates of the vertices using a triangulation method based on the plurality of vertices from both frames and stereo camera characteristics.
In the related art of marker detection, Vesanen discloses markers that are used to determine positions of objects (Vesanen, tag on an excavator, para 52: “The stereo camera arrangement 22 comprises at least two cameras 23 that are able to determine a 3D position of the tags”; para 49: “Alternatively a tag, a prism, or any other detectable item suitable to be fixed at a specific point in the excavator 1”; see also para 53); wherein the markers comprise encoded information identifying the object that the marker is on (Vesanen, para 57-58: “Both the aruco marker and QR-code comprises a pattern by means of which the marker or the code may be identified…The different shapes may be used to identify the specific tag, and thereby the specific spot in the excavator 1”); determining a position of the marker with respect to a local coordinate system of a stereo camera based on the image (Vesanen, para 55: “Each camera 23 determines the direction to each of the reflector balls it discovers or sees, and by using triangulation are the coordinates of these tags 21 in the coordinate system of the stereo camera arrangement 22 determined”); and identifying a plurality of vertices of a 2D projection of the marker (Vesanen, corners of aruco marker, para 57: “each tag 21 may be an aruco marker or a QR-code the positions of the one or more corners of which, for example, are to be determined”) separately for left and right frames of the stereo camera and calculating 3D coordinates of the vertices using a triangulation method based on the plurality of vertices from both frames and stereo camera characteristics (Vesanen, para 52: “The stereo camera arrangement 22 comprises at least two cameras 23 that are able to determine a 3D position of the tags to be monitored by triangulation when the location and orientation of the at least two cameras 23 in relation to each other are known”; since corners are used to determine the markers, determining 3D position must include calculating that of the corners, see para 57 citation above). It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have combined the encoded markers and determination of their 3D coordinates using a stereo camera, as taught by Vesanen, with the patent claim 4 in order to allow the system to determine which object is in view of the camera based on the detected marker, and share information with regard to that specific object (Para 57-58 of Vesanen). Vesanen teaches a known technique of using a triangulation method with a stereo camera to determine the 3D position of points. A person having ordinary skill in the art, before the effective filing date of the claimed invention, could have applied the known technique, as taught by Vesanen, in the same way to the method of patent claim 4 and achieved predictable results of inferring the 3D position of the marker using a stereo camera.
Regarding instant claim 14, patent claim 4 discloses acquiring an image of a marker located on an object in an environment in which the machine is located from a stereo camera attached to a machine (Patent claim 1, which claim 4 is dependent on, teaches components of an implement, which are in the same environment; see combination with Vesanen below wherein the components are markers; use of a stereo camera is an obvious variant known in the art for a 3D camera, also taught by Vesanen below); determining a position of the marker with respect to a local coordinate system of the stereo camera based on the image (patent claims 1 and 4); and determining a location of the machine with respect to a global coordinate system based on the position of the marker with respect to a local coordinate system of the stereo camera (patent claim 1). However, patent claim 4 fails to disclose wherein the components are markers and wherein the markers comprise encoded information identifying the object; wherein determining a location of the machine with respect to a global coordinate system is based on the encoded information; and identifying a plurality of vertices of a 2D projection of the marker separately for left and right frames of the stereo camera and calculating 3D coordinates of the vertices using a triangulation method based on the plurality of vertices from both frames and stereo camera characteristics.
In the related art of marker detection, Vesanen discloses markers that are used to determine positions of objects (Vesanen, tag on an excavator, para 52: “The stereo camera arrangement 22 comprises at least two cameras 23 that are able to determine a 3D position of the tags”; para 49: “Alternatively a tag, a prism, or any other detectable item suitable to be fixed at a specific point in the excavator 1”; see also para 53); wherein the markers comprise encoded information identifying the object that the marker is on (Vesanen, para 57-58: “Both the aruco marker and QR-code comprises a pattern by means of which the marker or the code may be identified…The different shapes may be used to identify the specific tag, and thereby the specific spot in the excavator 1”). Vesanen further teaches determining a location of the machine with respect to a global coordinate system is based on the encoded information (Vesanen, excavator position is determined based on the marker position, thus it is based on the encoded information of the marker, see para 53). It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have combined the encoded markers and determining the location of the machine using the encoded information, as taught by Vesanen, with the patent claim 4 in order to allow the system to determine which object is in view of the camera based on the detected marker, and share information with regard to that specific object to locate the machine (Para 57-58 of Vesanen). Vesanen also teaches a known technique of using a triangulation method with a stereo camera to determine the 3D position of points. A person having ordinary skill in the art, before the effective filing date of the claimed invention, could have applied the known technique, as taught by Vesanen, in the same way to the method of patent claim 4 and achieved predictable results of inferring the 3D position of the marker using a stereo camera.
Regarding instant claim 19, the claimed limitations are taught by patent claim 4 in combination with Vesanen in the same way as instant claim 1 above.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3, 9-11, 19, and 21-22 are rejected under 35 U.S.C. 103 as being unpatentable over Morin (U.S. Patent No. 2014/0270380 A1) in view of Vesanen.
Regarding claim 1, Morin teaches a method comprising:
acquiring from a camera an image of a marker on an object (Morin, para 64: “An image acquisition device, such as camera 110, obtains one or more images of a target 130 as the target moves through the fixed field of view 120 of the image acquisition device”; target is a marker on an object, see 130 in Fig. 2, para 65: “As the target 130 is operatively connected to the working edge 105 in a known location”);
determining a position of the marker with respect to a local coordinate system of the camera based on the image (Morin, para 64: “The images are conveyed to the vision processor 345 that executes software (not shown) for determining the location of the target in the vision system coordinate system”); and
determining a position of the object with respect to the local coordinate system of the camera based on the position of the marker with respect to the local coordinate system of the camera (Morin, para 65: “As the target 130 is operatively connected to the working edge 105 in a known location, by determining the location of the target, the vision system identifies the location of the working edge within the vision system coordinate system. Thus, the vision system determines the position of, e.g., the bottom of the working edge based on the predefined spatial relationship between the target and a known location of the working edge”).
Morin fails to teach wherein the camera is a stereo camera and wherein the marker comprises encoded information identifying the object. Morin further fails to explicitly teach the determining comprising identifying a plurality of vertices of a 2D projection of the marker separately for left and right frames of the stereo camera and calculating 3D coordinates of the vertices using a triangulation method based on the plurality of vertices from both frames and stereo camera characteristics.
However, Vesanen teaches a similar method (Vesanen, para 9-10) comprising acquiring from a stereo camera an image of a marker on an object (Vesanen, tag on an excavator, para 52: “The stereo camera arrangement 22 comprises at least two cameras 23 that are able to determine a 3D position of the tags”; para 49: “Alternatively a tag, a prism, or any other detectable item suitable to be fixed at a specific point in the excavator 1”), wherein the marker comprises encoded information identifying the object (Vesanen, para 57-58: “Both the aruco marker and QR-code comprises a pattern by means of which the marker or the code may be identified…The different shapes may be used to identify the specific tag, and thereby the specific spot in the excavator 1”); and identifying a plurality of vertices of a 2D projection of the marker (Vesanen, corners of aruco marker, para 57: “each tag 21 may be an aruco marker or a QR-code the positions of the one or more corners of which, for example, are to be determined”) separately for left and right frames of the stereo camera and calculating 3D coordinates of the vertices using a triangulation method based on the plurality of vertices from both frames and stereo camera characteristics (Vesanen, para 52: “The stereo camera arrangement 22 comprises at least two cameras 23 that are able to determine a 3D position of the tags to be monitored by triangulation when the location and orientation of the at least two cameras 23 in relation to each other are known”; since corners are used to determine the markers, determining 3D position must include calculating that of the corners, see para 57 citation above).
Morin discloses detecting a marker using a camera, but does not disclose using a stereo camera to acquire the image. As described above, Vesanen discloses acquiring an image of a marker using a stereo camera. Thus, Morin and Vesanen each disclose using a camera to acquire an image of a marker. A person of ordinary skill in the art, before the effective filing date of the claimed invention, would have recognized that the camera of Morin could have been substituted for the stereo camera of Vesanen because both serve the purpose of capturing an image to be used to detect a marker on an object. Furthermore, a person of ordinary skill in the art would have been able to carry out the substitution. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to substitute the stereo camera of Vesanen for the camera of Morin according to known methods to yield the predictable result of inferring the 3D position of the marker using image data.
Furthermore, Morin discloses a base method for determining the location of the target/marker (Example of using a time of flight camera in para 21), but does not specify using a triangulation method, which is disclosed by Vesanen. Vesanen teaches a known technique of using a triangulation method with a stereo camera to determine the 3D position of points. A person having ordinary skill in the art, before the effective filing date of the claimed invention, could have applied the known technique, as taught by Vesanen, in the same way to the method of Morin and achieved predictable results of inferring the 3D position of the marker using a stereo camera.
Lastly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have combined the marker encoded information of Vesanen with the method of Morin in order to allow the system to determine which object is in view of the camera based on the detected marker, and share information with regard to that specific object (Para 57-58 of Vesanen, cited above).
Regarding claim 3 (dependent on claim 1), Morin in view of Vesanen teaches wherein the encoded information identifies where the marker is located on the object (Vesanen, identify the specific spot on the object/excavator, para 58: “The different shapes may be used to identify the specific tag, and thereby the specific spot in the excavator 1”).
Regarding claim 9 (dependent on claim 3), Morin in view of Vesanen teaches wherein the determining the position of the object with respect to the local coordinate system of the stereo camera is further based on where the marker is located on the object (Morin, locating the object, or working edge, is based on its known location, para 65: “As the target 130 is operatively connected to the working edge 105 in a known location, by determining the location of the target, the vision system identifies the location of the working edge within the vision system coordinate system”; Vesanen teaches that this location is encoded, see claim 3 rejection).
Regarding claim 10 (dependent on claim 1), Morin in view of Vesanen teaches wherein the stereo camera is attached to a machine (Morin, See Figure 1, para 21: “FIG. 1 is a side view of an exemplary heavy equipment vehicle 100, e.g., a bulldozer”; see combination with Vesanen in claim 1 wherein the camera is a stereo camera) and the object is an implement attached to the machine (Morin, working edge, para 21: “The bulldozer 100 illustratively comprises a working edge, e.g., blade 105”).
Regarding claim 11 (dependent on claim 10), Morin in view of Vesanen teaches wherein the machine is a bulldozer (Morin, para 21: “FIG. 1 is a side view of an exemplary heavy equipment vehicle 100, e.g., a bulldozer”; FIG. 1 attached below) and the implement is a blade (Morin, para 21: “The bulldozer 100 illustratively comprises a working edge, e.g., blade 105”).
PNG
media_image1.png
650
831
media_image1.png
Greyscale
Regarding claim 19, Morin teaches an apparatus comprising: a camera (Morin, para 64: “camera 110”); a controller in communication with the camera, the controller configured to perform operations (Morin, para 64: “The images are conveyed to the vision processor 345 that executes software (not shown) for determining the location of the target in the vision system coordinate system”). Claim 19 is the corresponding apparatus (comprising a camera and controller) configured to perform operations equivalent to the method claimed in claim 1. Therefore, further limitations of claim 19 are met and rendered obvious by the combination of Morin in view of Vesanen, demonstrated in the rejection of claim 1 above.
Regarding claim 21 (dependent on claim 19), all claim limitations are met and rendered obvious by Morin in view of Vesanen because the method steps of claim 3 are the same as claim 21.
Regarding claim 22 (dependent on claim 21), all claim limitations are met and rendered obvious by Morin in view of Vesanen because the method steps of claim 9 are the same as claim 22.
Claims 4-5 and 23 are rejected under 35 U.S.C. 103 as being unpatentable over Morin in view of Vesanen, in further view of Qi et al. (U.S. Patent No. 2019/0329407 A1), hereinafter Qi.
Regarding claim 4 (dependent on claim 1), Morin in view of Vesanen teaches further comprising: determining a location of the stereo camera in a global coordinate system (Morin, para 71: “When the vision system is first installed, a calibration procedure occurs that identifies the location of the origin of the vision system's coordinate system at a specific point within the navigation system's coordinate system”; navigation system coordinate system is a global coordinate system, para 70: “navigation system coordinate system that corresponds to the real world”); and
determining a location of the object (Morin, working edge location in subsequent citation) in the global coordinate system based on the position of the object with respect to the local coordinate system of the stereo camera (Morin, para 67: “The vision system illustratively outputs a set of working edge location information. When combined with the navigation information provided by the navigation system, a working edge control system 350 may first transform the working edge location from the vision system coordinate system into the navigation system's coordinate system”) and the location of the stereo camera in the global coordinate system (Morin, continuation of para 71 citation above, para 71: “Using this known location, a transformation may be calculated to translate the origin point of the vision system's coordinate system to the navigation system's coordinate system”).
Morin in view of Vesanen fails to explicitly teach determining an orientation of the stereo camera in a global coordinate system. However, Qi teaches determining an orientation of the stereo camera in a global coordinate system (Qi, roll, yaw, and pitch of the camera, para 140: “mounting position and orientation of the first visual sensor 150 relative to the main body of the mapping device 100, which may be defined as the camera's (x, y, z, roll, yaw, pitch) relative to the mapping device 100.”; global coordinate system, para 134: “Kindly note x, y, z for the mapping device 100, the feature points and the tag(s) are their corresponding location in the global frame”). It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have combined the determination of the stereo camera orientation of Qi with the method of Morin in view of Vesanen in order to consider the pose of the camera and vehicle it is mounted to when determining the position/pose of detected markers/objects (Qi, para 150: “the camera pose in the robot frame (x, y, z, roll, yaw, pitch) from the system file 240, is configured to use the data to optimize the robot pose in the global frame (x, y, yaw)”).
Regarding claim 5 (dependent on claim 1), Morin in view of Vesanen fails to explicitly teach further comprising: determining a center of the marker based on the image, wherein the determining the position of the marker with respect to the local coordinate system of the stereo camera is further based on the center of the marker.
However, Qi teaches a similar system/method for identifying fiducial tags (Qi, abstract) comprising determining a center of the marker based on the image (Qi, para 133: “The tag detector 128 is configured to recognize the fiducial markers 180 from the current image captured by the first visual sensor 150”; para 133: “The tag detector 128 is configured to recognize the fiducial markers 180 from the current image captured by the first visual sensor 150…In certain embodiments, the location of a tag (u, v) may correspond to a specific corner or the center of the tag”). The combination of Morin in view of Vesanen and Qi teaches wherein the determining the position of the marker with respect to the local coordinate system of the stereo camera is further based on the center of the marker (Morin in view of Vesanen teaches determining the position of the marker with respect to the local coordinate system of the stereo camera, while Qi teaches specifically utilizing the center of the marker to detect its position.).
Morin in view of Vesanen teaches recognizing markers, but fails to teach determining the center of the marker; however, Qi teaches utilizing the center of the marker to recognize it. One of ordinary skill in the art, before the effective filing date of the claimed invention, could have combined the determining method taught by Qi above with the method of Morin in view of Vesanen, using known methods. In doing so, each element merely would have performed the same functions as it did separately and would achieve the predictable results of utilizing an identifying feature of each marker to identify different markers in the environment.
Regarding claim 23 (dependent on claim 19), Morin in view of Vesanen renders obvious all claim limitations of claim 19, and further limitations are met by the combination with Qi, as demonstrated in claim 4 above.
Claims 6-8 and 24 are rejected under 35 U.S.C. 103 as being unpatentable over Morin in view of Vesanen, in further view of Chakravarty et al. (U.S. Patent No. 2021/0158564 A1), hereinafter Chakravarty.
Regarding claim 6 (dependent on claim 1), Morin in view of Vesanen fails to explicitly teach further comprising: determining a roll of the marker; determining a heading of the marker; determining a pitch of the marker; and determining an attitude of the marker based on the pitch, the roll, and the heading of the marker.
However, Chakravarty teaches a similar method (Chakravarty, abstract; para 26: “detect a marker 106 displayed on one or more digital displays 145 on the vehicle 105”) comprising: determining a roll of the marker; determining a heading of the marker; determining a pitch of the marker (Chakravarty, para 53: “A first orientation of a marker 106 according to the coordinate system can also be determined from a marker 106 image…These angles can be used to give pitch, yaw, and roll, i.e., orientation”; yaw is equivalent to heading, as defined in the specification: “Heading is defined as the clockwise rotation about axis Y”, see para 17 and 27 in the specification of the claimed invention); and determining an attitude of the marker based on the pitch, the roll, and the heading of the marker (Chakravarty, orientation of the marker in the last citation - orientation based on pitch, roll, and yaw is the marker’s attitude). It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have combined the determining an attitude of the marker based on the pitch, the roll, and the heading of the marker with the method of Morin in view of Vesanen in order to determine the orientation of the object using the orientation of the marker (Chakravarty, para 55: “For example, the computer 150 can receive data describing the dimensions of the vehicle 105 and the marker 106 location and orientation on the vehicle 105 from the marker 106 data record retrieved upon identifying the marker 106 in the image data, as described above”). Furthermore, a person having ordinary skill in the art could determine that determining the attitude of a marker on an object would aid in the determining of the 3D position of the object.
Regarding claim 7 (dependent on claim 6), Morin in view of Vesanen and Chakravarty teaches further comprising: determining an attitude of the object based on the attitude of the marker (Chakravarty, determine vehicle orientation using marker orientation, para 55: “based on the data describing the dimensions of the vehicle 105 and the marker 106 location and orientation on the vehicle 105, the computer 150 can determine a vehicle 105 location and an orientation with respect to the marker 106. The computer 150 can then determine the second location and the second orientation of the vehicle 105 with respect to the coordinate system from the first location and the first orientation of the marker 106 with respect to the coordinate system based on the vehicle 105 location and orientation with respect to the marker 106”; para 53: “An orientation of an object such as a marker 106 or vehicle 105 can be specified according to a pitch, yaw, and roll”).
Regarding claim 8 (dependent on claim 6), Morin in view of Vesanen and Chakravarty teaches further comprising: determining a plurality of vertices about a center of the marker (Taught by Vesanen in claim 1; corners of an aruco marker are about a center of the marker), wherein the determining the attitude of the marker is further based on the plurality of vertices (Vesanen teaches locating the marker based on the vertices, while the attitude of the marker is determined after the marker itself is located, taught by Chakravarty. Therefore, determining the attitude is based on the vertices and is taught by the combination, see also claim 6.).
Regarding claim 24 (dependent on claim 19), all claim limitations are met and rendered obvious by Morin in view of Vesanen and Chakravarty because the method steps of claim 6 are the same as claim 24.
Claims 12-13 are rejected under 35 U.S.C. 103 as being unpatentable over Morin in view of Vesanen, in further view of Anagnostopoulos (U.S. Patent No. 11, 315, 258 B1).
Regarding claim 12 (dependent on claim 10), Morin in view of Vesanen fails to explicitly teach wherein the machine is a tractor and the object is a towed machine. However, Anagnostopoulos teaches a similar system (Anagnostopoulos, abstract and Figure 21A) wherein the machine is a tractor and the object is a towed machine (Anagnostopoulos, col 10, ln 20-26: “FIG. 21A is a duplicate of FIG. 6A. It is a simple, specific, non-limiting example of one exemplary embodiment according to the invention applied to tracking position of a tractor-towed implement, using a camera as optical imaging sensor and a pair of infrared LEDs as fiducials on the implement that, when imaged, can be displayed to indicate when the implement is in an aligned heading with the tractor”). It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have combined the tractor and towed machine embodiment, taught by Anagnostopoulos, with the method of Morin in view of Vesanen in order to track a towed implement when the terrain is variable (Anagnostopoulos, col 9, ln 59-64: “FIG. 16 is intended to illustrate some of the variability of terrain in a first pass across a crop field and how they might affect tractor and towed implement offsets from an intended travel path through a field, and that using the present invention allows tracking and logging of implement actual position during such a first pass”).
Regarding claim 13 (dependent on claim 1), Morin in view of Vesanen teaches wherein the stereo camera is attached to a first vehicle (Morin teaches a camera attached to a first vehicle, a heavy equipment vehicle, but fails to teach a second vehicle, see para 22; see claim 1 rejection wherein the camera is a stereo camera), but fails to teach wherein the object is a second vehicle. However, Anagnostopoulos teaches a similar system (Anagnostopoulos, abstract and Figure 26A) wherein the camera is attached to a first vehicle and the object is a second vehicle (Anagnostopoulos, col 10, ln 42-50: “FIGS. 26A and B relate to monitoring position of a follow along truck/grain cart to a combine. Another example of use of the paradigm of FIGS. 26A and B (and other Figures and examples herein) could be monitoring or correcting for application offsets (such as monitoring position between tractor and where agricultural product is applied on or in the ground instead of simply position of a towed implement or following tractor/truck)”; camera is attached to a first vehicle, while the follow along truck is a monitored object, col 14, ln 27-42: “A process to correlate imaged fiducial(s) in camera space into estimation of position in physical space… The known geometric relationship between the sensor and acquired geospatial information about object 1, and the known geometric relationship between the fiducial(s) and object 2, allows the camera space translation to be combined with the object 1 geospatial information to make the estimate of geospatial position of object 2”). It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have combined the first and second vehicle embodiment of Anagnostopoulos with the method of Morin in view of Vesanen in order to monitor the performance of an attached vehicle (Anagnostopoulos, col 10, ln 49-54: “This would allow being able to monitor whether application of product (e.g. planted seed, herbicide, insecticide, fertilizer) or performance of operations in the ground or on plants (e.g. planting, cultivating, harvesting), and then being able to know those application or operations positions are correct at the ground level”).
Claims 14-15 are rejected under 35 U.S.C. 103 as being unpatentable over Kojima in view of Vesanen.
Regarding claim 14, Kojima teaches a method comprising:
acquiring an image of a marker located on an object in an environment in which a machine is located (Kojima, see marker 4 in Fig. 7, para 67: “FIG. 7 is a diagram showing an example of the image 40 captured by the image capturing apparatus 11 of FIG. 1”; see machine with image capturing apparatus in FIG. 1) from a stereo camera attached to the machine (Kojima, see camera 11 attached to vehicle 1 in Fig. 1, para 67: “image 40 captured by the image capturing apparatus 11 of FIG. 1…when the vehicle 1 passes near any one of the markers 4, the image 40 includes the corresponding marker 4”; stereo camera, para 159: “the image capturing apparatus 11 may be a stereo camera including two cameras disposed apart from each other by a certain distance”) and comprising encoded information (Kojima, para 64: “the marker 4 has a visually distinguishable pattern, into which an identifier of the marker 4 itself is encoded”);
determining a position of the marker with respect to a local coordinate system of the stereo camera based on the image (Kojima, para 102: “In step S22 of FIG. 11, the absolute position calculator 34 calculates the position and the attitude of the marker 4 in a three-dimensional coordinate system whose origin is located at the image capturing apparatus 11 (camera coordinate system)”); and
determining a location of the machine with respect to a global coordinate system based on the position of the marker with respect to a local coordinate system of the stereo camera and the encoded information (Kojima, step 27, is based on step 23, which is based on a translation of the position determined in step 22, see para 103, therefore global location of the machine is based on the position of the marker in the camera coordinate system, see also Fig. 11, para 106: “In step S27, the absolute position calculator 34 calculates the position and the attitude of the vehicle 1 in the world coordinate system (i.e., the absolute position and the absolute attitude of the vehicle 1), based on the position and the attitude of the image capturing apparatus 11 in the marker coordinate system calculated in step S23, and based on the position and the attitude of the marker 4 in the world coordinate system read out in step S26”; encoded information includes the position and attitude of the marker, see claim 15 citation below).
Kojima fails to explicitly teach comprising encoded information identifying the object (emphasis added); and the determining comprising identifying a plurality of vertices of a 2D projection of the marker separately for left and right frames of the stereo camera and calculating 3D coordinates of the vertices using a triangulation method based on the plurality of vertices from both frames and stereo camera characteristics.
However, Vesanen teaches a similar method (Vesanen, para 9-10) comprising acquiring an image of a marker located on an object (Vesanen, tag on an excavator, para 52: “The stereo camera arrangement 22 comprises at least two cameras 23 that are able to determine a 3D position of the tags”; para 49: “Alternatively a tag, a prism, or any other detectable item suitable to be fixed at a specific point in the excavator 1”), wherein the marker comprises encoded information identifying the object (Vesanen, para 57-58: “Both the aruco marker and QR-code comprises a pattern by means of which the marker or the code may be identified…The different shapes may be used to identify the specific tag, and thereby the specific spot in the excavator 1”); and identifying a plurality of vertices of a 2D projection of the marker (Vesanen, corners of aruco marker, para 57: “each tag 21 may be an aruco marker or a QR-code the positions of the one or more corners of which, for example, are to be determined”) separately for left and right frames of the stereo camera and calculating 3D coordinates of the vertices using a triangulation method based on the plurality of vertices from both frames and stereo camera characteristics (Vesanen, para 52: “The stereo camera arrangement 22 comprises at least two cameras 23 that are able to determine a 3D position of the tags to be monitored by triangulation when the location and orientation of the at least two cameras 23 in relation to each other are known”; since corners are used to determine the markers, determining 3D position must include calculating that of the corners, see para 57 citation above).
Kojima discloses a base method for determining the location of the marker (Position and attitude of the marker), but does not specify a triangulation method, which is disclosed by Vesanen. Vesanen teaches a known technique of using a triangulation method with a stereo camera to determine the 3D position of points. A person having ordinary skill in the art, before the effective filing date of the claimed invention, could have applied the known technique, as taught by Vesanen, in the same way to the method of Kojima and achieved predictable results of inferring the 3D position of the marker using a stereo camera.
Lastly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have combined the marker encoded information of Vesanen with the method of Kojima in order to allow the system to determine which object is in view of the camera based on the detected marker, and share information with regard to that specific object (Para 57-58 of Vesanen, cited above).
Regarding claim 15 (dependent on claim 14), Kojima in view of Vesanen teaches wherein the encoded information identifies a location of the marker in the environment with respect to the global coordinate system (Vesanen, marker location on vehicle/excavator, para 58: “The different shapes may be used to identify the specific tag, and thereby the specific spot in the excavator 1”).
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Kojima in view of Vesanen and Oetken (U.S. Patent No. 2020/0117201 A1).
Regarding claim 16 (dependent on claim 15), Kojima in view of Vesanen fails to explicitly teach wherein the machine is an asphalt paver. Oetken teaches an optical surveying system (Oetken, abstract) wherein the machine is an asphalt paver (Oetken, para 15: “paving machine 12”). It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have combined the asphalt paver of Oetken with the method of Kojima in view of Vesanen in order to determine the position of an asphalt paver during operation (Oetken, para 3: “Operators of the machines typically actively drive the machines to maintain the vehicles safely within the work area to, for example, eliminate driving on potentially hazardous terrain and avoiding other potential no-go areas where people or property can become injured or damaged”). Further, Kojima and Oetken each disclose methods for tracking the position of machines. A person of ordinary skill in the art, before the effective filing date of the claimed invention, would have recognized that the machine of Kojima in view of Vesanen could have been substituted by the asphalt paving machine of Oetken. A person of ordinary skill in the art would have been able to carry out the substitution by attaching the stereo camera to the asphalt paver and achieve the predictable result of tracking an asphalt paver during paving.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Marker triangulation:
U.S. Patent No. 2016/0093058 A1
Tracking machine implement:
U.S. Patent No. 2020/0346581 A1
U.S. Patent No. 9,971,943 B2
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to EMMA E DRYDEN whose telephone number is (571)272-1179. The examiner can normally be reached M-F 9-5 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ANDREW BEE can be reached at (571) 270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/EMMA E DRYDEN/Examiner, Art Unit 2677
/ANDREW W BEE/Supervisory Patent Examiner, Art Unit 2677
Prosecution Insights