Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Status
The status of claims 1-20 is:
Claims 1-20 are pending.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 11/05/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3-12, 15-17, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Tang et al. (Tang, J., Wang, M., Luo, H., Wong, P. K. Y., Zhang, X., Chen, W., & Cheng, J. C. (2023). Full-body pose estimation for excavators based on data fusion of multiple onboard sensors. Automation in Construction, 147, 104694., hereinafter “Tang”) in view of Li et al. (U.S. Patent Publication No 2019/0066323, hereinafter “Li”).
Regarding claim 1, Tang discloses a system comprising:
a chassis of a vehicle (Tang Page 3: “This study therefore proposes a full-body pose estimation framework based on data fusion of multiple on-board sensors for excavators, which is illustrated in Fig. 1.”; Tang Fig. 2: shows excavator with chassis);
a movable member movably coupled to the chassis (Tang Page 3: “This study therefore proposes a full-body pose estimation framework based on data fusion of multiple on-board sensors for excavators, which is illustrated in Fig. 1.”; Tang Fig. 2: shows excavator with arm connected to chassis);
a feature board attached to the vehicle (Tang Page 6: “A marker is attached to the lower part of the arm to mimic the focus of the operator's eyes to facilitate estimating poses. The marker should be always in the view of cameras. Fig. 5 shows the actual operator's view and the view obtained by the stereo vision module, as well as the attached marker”; Tang Page 6: “Although various types of markers are available in this method, in order to overcome the changing background on construction sites and enhance the stability of detection, binary square fiducial markers with their pre-defined libraries, such as ArUco [35], are selected in this study”);
a visual inertial measurement unit (IMU) mounted on the vehicle, the visual IMU including a camera and a first set of inertial sensors (Tang Page 4: “9-axis-IMUs are attached to the surface of every movable component for the target excavator (i.e., cabin, boom, arm, and bucket), in order to collect three types of inertial data: (1) acceleration captured from the sensor's accelerometer, (2) angular velocity obtained from the gyroscope, and (3) magnetic flux collected from its magnetometer”; Tang Page 5: “In addition to angular data obtained from the IMUs, cameras are used as another data source for data fusion to collect visual information and track keypoints' positions of the target excavator”), the camera configured to have a field of view that includes the feature board (Tang Page 6: “Hence, two cameras, which provide RGB and geometric information simultaneously, are used to build a stereo vision module in the proposed independent onboard method, which is mounted at the front of the cabin to simulate the operator's eyes. A marker is attached to the lower part of the arm to mimic the focus of the operator's eyes to facilitate estimating poses. The marker should be always in the view of cameras”);
an auxiliary IMU mounted on the vehicle, the auxiliary IMU including a second set of inertial sensors (Tang Page 4: “9-axis-IMUs are attached to the surface of every movable component for the target excavator (i.e., cabin, boom, arm, and bucket), in order to collect three types of inertial data: (1) acceleration captured from the sensor's accelerometer, (2) angular velocity obtained from the gyroscope, and (3) magnetic flux collected from its magnetometer”); and
at least one processor (Tang Page 16: “Additionally, considering the requirement of operational safety monitoring on response time in practice, the proposed framework was conducted a timing-test on the laptop (model name: Lenovo Legion Y7000P2021, CPU: i7-11800H, GPU: GeForce RTX 3050Ti)”);
wherein the at least one processor hosts a program module including a visual-inertial algorithm operative to perform a process that comprises:
receive visual information from the camera (Tang Fig. 2: image-based onboard motion tracking of excavators), the visual information including features detected in images of the feature board captured by the camera (Tang Fig. 4: single feature point tracking and image-based keypoints estimation);
receive inertial measurements from the visual IMU and the auxiliary IMU, the inertial measurements including acceleration data and angular rate data (Tang Fig. 3: data collection & processing); and
estimate orientation of the movable member with respect to the chassis by integrating the visual information with the inertial measurements (Tang Fig 1: estimated full-body pose; Tang Page 8: “A multiple keypoints localization algorithm is developed to combine the IMU and camera measurements competitively and find optimal estimations of the locations of the keypoints in the camera reference frame”).
Tang does not explicitly disclose at least one processor onboard the vehicle, the at least one processor in operative communication with the visual IMU and the auxiliary IMU.
However, Li teaches at least one processor onboard the vehicle, the at least one processor in operative communication with the visual IMU and the auxiliary IMU (Li [0009]: “Merely by way of example, a system in accordance with one set of embodiments might comprise a mobile communication device or any other apparatus comprising an image sensor, an accelerometer, a gyroscope, a display, and one or more processors in communication with the image sensor, accelerometer, gyroscope, and display, and/or a computer readable medium in communication with the processor”).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the integrated and onboard processor as taught by Li with the system of Tang because it would improve the invention by allowing the entire process to be done on the excavator instead of having a computer elsewhere doing the processing. This motivation for the combination of Tang and Li is supported by KSR exemplary rationale (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results.
Regarding claim 3, Tang discloses the system, wherein:
the feature board is attached to the movable member (Tang Page 6: “A marker is attached to the lower part of the arm to mimic the focus of the operator's eyes to facilitate estimating poses”);
the visual IMU is mounted on the chassis (Tang Page 4: “9-axis-IMUs are attached to the surface of every movable component for the target excavator (i.e., cabin, boom, arm, and bucket), in order to collect three types of inertial data: (1) acceleration captured from the sensor's accelerometer, (2) angular velocity obtained from the gyroscope, and (3) magnetic flux collected from its magnetometer”; Tang Page 6: “Hence, two cameras, which provide RGB and geometric information simultaneously, are used to build a stereo vision module in the proposed independent onboard method, which is mounted at the front of the cabin to simulate the operator's eyes. A marker is attached to the lower part of the arm to mimic the focus of the operator's eyes to facilitate estimating poses. The marker should be always in the view of cameras”); and
the auxiliary IMU is mounted on the movable member (Tang Page 4: “9-axis-IMUs are attached to the surface of every movable component for the target excavator (i.e., cabin, boom, arm, and bucket), in order to collect three types of inertial data: (1) acceleration captured from the sensor's accelerometer, (2) angular velocity obtained from the gyroscope, and (3) magnetic flux collected from its magnetometer”).
Regarding claim 4, Tang discloses the system, wherein the vehicle is a construction machine vehicle comprising a dozer or an excavator (Tang Page 2: “in our study, an excavator — is developed to track the machine's motion and collect data regarding its poses”).
Regarding claim 5, Tang discloses the system, wherein the movable member comprises a blade or a bucket (Tang Page 2: “An excavator has four movable components (i.e., a cabin, a boom, an arm, and a bucket)”), movably coupled to the chassis through a mechanical arm structure (Tang Page 2: “An excavator has four movable components (i.e., a cabin, a boom, an arm, and a bucket)”).
Regarding claim 6, Tang does not explicitly disclose the system, wherein the feature board has a flat surface including a chessboard or checkerboard pattern, a ChArUco pattern (although Tang does disclose using a ArUco pattern (Page 6: “Although various types of markers are available in this method, in order to overcome the changing background on construction sites and enhance the stability of detection, binary square fiducial markers with their pre-defined libraries, such as ArUco [35], are selected in this study”)), or a circular grid.
However, Li teaches the system, wherein the feature board has a flat surface including a chessboard or checkerboard pattern (Li [0037]: “In some cases, a target object might be an optical target 130 that is placed on the stick 120 or bucket 125 of the excavator 110. Examples of optical targets, some of which are well known, are prisms, discs, spheres, flags, and/or the like, which can be placed in a known position relative to a target point and may be designed to be relatively easy to acquire (e.g., visually and/or electronically) for position measurement purposes. The optical targets or other reference features may be placed in a readily identifiable pattern such as a checkerboard, a blob, and/or the like”), a ChArUco pattern, or a circular grid.
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the checkerboard pattern as taught by Li with the system of Tang because it is a simple substitution of one type of feature board for another. This motivation for the combination of Tang and Li is supported by KSR exemplary rationale (B) Simple substitution of one known element for another to obtain predictable results.
Regarding claim 7, Tang does not explicitly disclose the system, further comprising:
a global navigation satellite system (GNSS) receiver onboard the vehicle and communicatively coupled with the at least one processor;
wherein the GNSS receiver is configured to provide position and heading information for the vehicle.
However, Li teaches the system, further comprising:
a global navigation satellite system (GNSS) receiver onboard the vehicle and communicatively coupled with the at least one processor (Li [0069]: “The communication device 100 may further include a position sensor 515, which might be a global navigation satellite system (“GNSS”) sensor, such as a global positioning system (“GPS”) sensor or the like”);
wherein the GNSS receiver is configured to provide position and heading information for the vehicle (Li [0069]: “The position sensor 515 can be used to determine a position of the device 100 according to a global or local coordinate system (e.g., latitude/longitude, GPS coordinates, etc.), which can then be used to derive a position of the stick/bucket relative to the same coordinate system (e.g., by performing vector algebra between the position of the device 100 and the position of the stick/bucket relative the device)”).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the checkerboard pattern as taught by Li with the system of Tang because it would improve the system by allowing it to derive a position of the bucket relative to chassis using a global or local coordinate system (Li [0069]). This motivation for the combination of Tang and Li is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention and rationale (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results.
Regarding claim 8, Tang discloses the system, further comprising:
wherein the at least one processor is operative to compute and output relative heading/attitude information, between the visual IMU and the auxiliary IMU (Tang Fig. 1: estimated full-body pose includes the heading/attitude between the visual IMU and the auxiliary IMU).
Tang does not explicitly disclose the system, further comprising:
a computer or controller onboard the vehicle and in operative communication with the at least one processor.
However, Li teaches the system, further comprising:
a computer or controller onboard the vehicle and in operative communication with the at least one processor (Li [0070]: “The communication device 100 may also have a communication interface 520 and a data storage device 525. The communication interface 520 may enable a user to interact with the position and motion tracking system on the communication device 100”).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the computer or controller onboard the vehicle as taught by Li with the system of Tang because it would improve the invention by allowing the entire process to be done on the excavator instead of having a computer elsewhere doing the processing. This motivation for the combination of Tang and Li is supported by KSR exemplary rationale (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results.
Regarding claim 9, Tang discloses the system, wherein:
the first set of inertial sensors of the visual IMU comprises one or more gyroscopes and one or more accelerometers (Tang Page 4: “9-axis-IMUs are attached to the surface of every movable component for the target excavator (i.e., cabin, boom, arm, and bucket), in order to collect three types of inertial data: (1) acceleration captured from the sensor's accelerometer, (2) angular velocity obtained from the gyroscope, and (3) magnetic flux collected from its magnetometer”); and
the second set of inertial sensors of the auxiliary IMU comprises one or more gyroscopes and one or more accelerometers (Tang Page 4: “9-axis-IMUs are attached to the surface of every movable component for the target excavator (i.e., cabin, boom, arm, and bucket), in order to collect three types of inertial data: (1) acceleration captured from the sensor's accelerometer, (2) angular velocity obtained from the gyroscope, and (3) magnetic flux collected from its magnetometer”).
Regarding claim 10, Tang discloses the system, wherein the visual-inertial algorithm is performed by a tightly coupled process in which the visual information and the inertial measurements are combined and processed together in a single optimization filter to estimate the orientation of the movable member with respect to the chassis (Tang Page 8: “An EKF (Extended Kalman Filter), a classical approach for non-linear stochastic system [37], is utilized in this study for competitive data fusion”; Tang Fig. 1: EKF-based multiple keypoints localization algorithm for excavators based on competitive fusion).
Regarding claim 11, Tang does not explicitly disclose the system, wherein the visual-inertial algorithm is performed by a loosely coupled process in which the visual information and the inertial measurements are processed independently to estimate the orientation of the movable member with respect to the chassis before these estimates are fused together.
However, Li teaches the system, wherein the visual-inertial algorithm is performed by a loosely coupled process in which the visual information and the inertial measurements are processed independently to estimate the orientation of the movable member with respect to the chassis before these estimates are fused together (Li [0011]: “The mobile communication device may also capture input from the accelerometer of the mobile communication device to determine an orientation of the mobile communication device relative to the ground. Once the reference image of the one or more reference features and the input from the accelerometer is captured, the mobile communication device may be calibrated based on the orientations of the reference features in the reference image and based on the orientation of the mobile communication device. Alternatively and/or additionally, the communication device may comprise a gyroscope. The gyroscope may be communicatively coupled to the communication device and allow the communication device to determine an orientation of the mobile communication device relative to an initial reference (e.g., the ground, the floor of the excavator cabin, the excavator stick, the excavator bucket, etc.). The communication device may further determine, based on data received from the accelerometer and/or gyroscope, whether the orientation of the communication device has shifted as the excavator traverses the work site and take into account the shifts in orientation when calculating the position of the excavator, excavator stick, and/or excavator bucket”; Li [0012]: “After the mobile communication device is calibrated, the image sensor may capture at least one additional image of the at least one reference feature on the stick of the excavator”).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the loosely coupled process as taught by Li with the system of Tang because it would improve the method because it would allow for the estimations of each sensor to be compared instead of using all sensor data to make an initial estimation. This motivation for the combination of Tang and Li is supported by KSR exemplary rationale (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results.
Regarding claim 12, Tang does not explicitly disclose the system, wherein the visual information including features detected in images of the feature board are processed using an algorithm comprising a histogram of oriented gradients, or a scale-invariant feature transform.
However, Li teaches the system, wherein the visual information including features detected in images of the feature board are processed using an algorithm comprising a histogram of oriented gradients, or a scale-invariant feature transform (Li [0049]: “Given a set of checkerboard images and the known pattern of the checkerboard plane and assuming the corresponding image measurements are corrupted by independent and identically distributed noises, the maximum likelihood estimate can be obtained by minimizing the aforementioned sum of squared differences between the target image coordinates and the computed coordinates from the projective transformation. With the help of the Levenberg-Marquardt algorithm, the camera's intrinsic and extrinsic parameters are iteratively optimized”).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to incorporate transform as taught by Li with the system of Tang because it would allow for the processing of the visual information to be conducted without having to worry about the scale of the information. This motivation for the combination of Tang and Li is supported by KSR exemplary rationale (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results.
Regarding claim 15, Tang discloses a method comprising:
providing a relative attitude measuring system comprising:
a feature board attached to a movable member or a chassis of a vehicle (Tang Page 6: “A marker is attached to the lower part of the arm to mimic the focus of the operator's eyes to facilitate estimating poses. The marker should be always in the view of cameras. Fig. 5 shows the actual operator's view and the view obtained by the stereo vision module, as well as the attached marker”; Tang Page 6: “Although various types of markers are available in this method, in order to overcome the changing background on construction sites and enhance the stability of detection, binary square fiducial markers with their pre-defined libraries, such as ArUco [35], are selected in this study”);
a visual inertial measurement unit (IMU) mounted on the vehicle, the visual IMU including a camera and a plurality of inertial sensors (Tang Page 4: “9-axis-IMUs are attached to the surface of every movable component for the target excavator (i.e., cabin, boom, arm, and bucket), in order to collect three types of inertial data: (1) acceleration captured from the sensor's accelerometer, (2) angular velocity obtained from the gyroscope, and (3) magnetic flux collected from its magnetometer”; Tang Page 5: “In addition to angular data obtained from the IMUs, cameras are used as another data source for data fusion to collect visual information and track keypoints' positions of the target excavator”), the camera having a field of view that includes the feature board (Tang Page 6: “Hence, two cameras, which provide RGB and geometric information simultaneously, are used to build a stereo vision module in the proposed independent onboard method, which is mounted at the front of the cabin to simulate the operator's eyes. A marker is attached to the lower part of the arm to mimic the focus of the operator's eyes to facilitate estimating poses. The marker should be always in the view of cameras”);
an auxiliary IMU mounted on the vehicle and including a plurality of inertial sensors (Tang Page 4: “9-axis-IMUs are attached to the surface of every movable component for the target excavator (i.e., cabin, boom, arm, and bucket), in order to collect three types of inertial data: (1) acceleration captured from the sensor's accelerometer, (2) angular velocity obtained from the gyroscope, and (3) magnetic flux collected from its magnetometer”); and
a processing unit (Tang Page 16: “Additionally, considering the requirement of operational safety monitoring on response time in practice, the proposed framework was conducted a timing-test on the laptop (model name: Lenovo Legion Y7000P2021, CPU: i7-11800H, GPU: GeForce RTX 3050Ti)”) that hosts a visual-inertial algorithm that performs a tight coupling function or a loose coupling function (Tang Page 8: “An EKF (Extended Kalman Filter), a classical approach for non-linear stochastic system [37], is utilized in this study for competitive data fusion”; Tang Fig. 1: EKF-based multiple keypoints localization algorithm for excavators based on competitive fusion);
performing a real-time data collection process, comprising:
determining whether an image is received by the camera of the visual IMU (Tang Fig. 3: data collection & processing);
in response to determining that an image is received by the camera, sending image data corresponding to the received image to the processing unit (Tang Fig. 2: image-based onboard motion tracking of excavators);
determining whether inertial data is received by the inertial sensors of the visual IMU (Tang Fig. 3: data collection & processing);
in response to determining that inertial data is received by the inertial sensors of the visual IMU, sending the inertial data from the visual IMU to the processing unit (Tang Fig. 3: data collection & processing);
determining whether inertial data is received by the inertial sensors of the auxiliary IMU (Tang Fig. 3: data collection & processing); and
in response to determining that inertial data is received by the inertial sensors of the auxiliary IMU, sending the inertial data from the auxiliary IMU to the processing unit (Tang Fig. 3: data collection & processing); and
wherein the data sent to the processing unit is analyzed and processed by the tight coupling function or the loose coupling function to compute an enhanced relative heading/attitude estimate (Tang Fig 1: estimated full-body pose; Tang Page 8: “A multiple keypoints localization algorithm is developed to combine the IMU and camera measurements competitively and find optimal estimations of the locations of the keypoints in the camera reference frame”).
Tang does not explicitly disclose the method comprising:
an onboard processing unit; and
sending the enhanced relative heading/attitude estimate from the processing unit to a control system for the vehicle.
However, Li teaches the method comprising:
an onboard processing unit (Li [0009]: “Merely by way of example, a system in accordance with one set of embodiments might comprise a mobile communication device or any other apparatus comprising an image sensor, an accelerometer, a gyroscope, a display, and one or more processors in communication with the image sensor, accelerometer, gyroscope, and display, and/or a computer readable medium in communication with the processor”); and
sending the enhanced relative heading/attitude estimate from the processing unit to a control system for the vehicle (Li [0064]: “By using the communication device 100 to determine a position (including height, reach, and azimuth position) of an excavator stick 120 and excavator bucket 125, an excavator 110 may be automatically controlled by the communication device 100”).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the integrated and onboard processor and controlling the vehicle as taught by Li with the system of Tang because it would improve the invention by allowing the entire process to be done on the excavator instead of having a computer elsewhere doing the processing and allow the machine to be controlled without an on-board operator. This motivation for the combination of Tang and Li is supported by KSR exemplary rationale (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results.
Regarding claim 16, Tang does not explicitly disclose the method, wherein:
the relative attitude measuring system further comprises a global navigation satellite system (GNSS) receiver onboard the vehicle; and
the real-time data collection process further comprises: determining whether position/heading data is received by the GNSS receiver; and
in response to determining that position/heading data is received by the GNSS receiver, sending the received position/heading data from the GNSS receiver to the processing unit.
However, Li teaches the method, wherein:
the relative attitude measuring system further comprises a global navigation satellite system (GNSS) receiver onboard the vehicle (Li [0069]: “The communication device 100 may further include a position sensor 515, which might be a global navigation satellite system (“GNSS”) sensor, such as a global positioning system (“GPS”) sensor or the like”); and
the real-time data collection process further comprises: determining whether position/heading data is received by the GNSS receiver (Li [0069]: “The position sensor 515 can be used to determine a position of the device 100 according to a global or local coordinate system (e.g., latitude/longitude, GPS coordinates, etc.), which can then be used to derive a position of the stick/bucket relative to the same coordinate system (e.g., by performing vector algebra between the position of the device 100 and the position of the stick/bucket relative the device)”); and
in response to determining that position/heading data is received by the GNSS receiver, sending the received position/heading data from the GNSS receiver to the processing unit (Li [0069]: “The communication device 100 may further include a position sensor 515, which might be a global navigation satellite system (“GNSS”) sensor, such as a global positioning system (“GPS”) sensor or the like”).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the checkerboard pattern as taught by Li with the system of Tang because it would improve the method by allowing it to derive a position of the bucket relative to chassis using a global or local coordinate system (Li [0069]). This motivation for the combination of Tang and Li is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention and rationale (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results.
Regarding claim 17, Tang discloses the method, wherein the tight coupling function comprises:
a feature tracking module that receives and processes the image data from the visual IMU to produce two-dimensional (2D) features from the image data (Tang Page 4: “the projected 2D frame(x,y), which is attached to the camera with the x-axis pointing to the optical axis, the y-axis pointing up, and the origin being the optical center of the stereo vision module”; Tang Page 9: “In the projected 2D plane, the boom joint (K2) is a fixed point, which can be determined by the physical parameters of the excavator. The arm joint (K3) and the bucket joint (K4) are moving according to the movement of different components”); and
a tightly coupled module that receives the inertial data from the visual IMU and the auxiliary IMU, and the 2D features from the feature tracking module (Tang Fig. 1: EKF-based multiple keypoints localization algorithm for excavators based on competitive fusion; Tang Page 8: “An EKF (Extended Kalman Filter), a classical approach for non-linear stochastic system [37], is utilized in this study for competitive data fusion”);
wherein the tightly coupled module processes the 2D features and the inertial data together in a single optimization filter to compute the enhanced relative heading/attitude estimate (Tang Fig. 1: EKF-based multiple keypoints localization algorithm for excavators based on competitive fusion and estimated full-body pose).
Regarding claim 19, Tang discloses the method, wherein the vehicle is a construction machine vehicle comprising a dozer or an excavator (Tang Page 2: “in our study, an excavator — is developed to track the machine's motion and collect data regarding its poses”).
Regarding claim 20, Tang discloses the method, wherein the movable member comprises a blade or a bucket, movably coupled to the chassis through a mechanical arm structure (Tang Page 2: “An excavator has four movable components (i.e., a cabin, a boom, an arm, and a bucket)”).
Claim(s) 13-14 are rejected under 35 U.S.C. 103 as being unpatentable over the Tang and Li combination in view of Li et al. (Li, J., Liu, Y., Wang, L., & Sun, Y. (2024). A vision-based end pose estimation method for excavator manipulator. Multimedia Tools and Applications, 83(26), 68723-68741., hereinafter “Li 2”).
Regarding claim 13, the Tang and Li combination does not explicitly disclose the system, wherein the visual information including features detected in images of the feature board are processed using one or more deep neural networks.
However, Li 2 teaches the system, wherein the visual information including features detected in images of the feature board are processed using one or more deep neural networks (Li 2 Page 68728: “The excavator manipulator image collected by the vision sensor is used as the input of the network. In the encoder part, the input image is first sent to the Xception module with Atrous Convolution to extract image features, obtaining high-level semantic features and low-level semantic features. DeepLabv3 + applies several parallel Atrous Convolutions with different rates (ASPP). High-level semantic features enter ASPP, undergoing convolution and pooling with four Atrous Convolutions and one pooling layer respectively to capture the spatial information of features. After obtaining five feature maps, concatenation is used to fuse the features, followed by a 1 × 1 convolution to compress the channel. In the decoder part, firstly, the extracted multi-scale feature information is upsampled by 4 times using bilinear interpolation to obtain a high-level semantic feature map. In the Xception module of the deep convolution network layer, a low-level semantic feature map with the same resolution as the high-level semantic feature map is found, and the number of channels is reduced by 1 × 1 convolution to match the channel proportion of the high level semantic feature map, making it convenient for model learning. Then, the low-level semantic feature map and high-level semantic feature map are concatenated to obtain a new feature map, which is refined by a 3 × 3 convolution and upsampled by 4 times using bilinear interpolation to get the predicted segmentation result. The final result of semantic segmentation will be a binary graph with only two classes, including the target class and the background class”).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the neural network taught by Li 2 with the system of Tang and Li because it would improve the system by incorporating a neural network that can be trained and because CNNs offer high flexibility for computer vision tasks (Li 2 Page 68724). This motivation for the combination of Tang, Li, and Li 2 is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention and rationale (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results.
Regarding claim 14, the Tang and Li combination does not explicitly disclose the system, wherein the one or more deep neural networks include a convolutional neural network, or a transformer-based network.
However, Li 2 teaches the system, wherein the one or more deep neural networks include a convolutional neural network, or a transformer-based network (Li 2 Page 68727: “It uses a combination of multi-scale convolution layers and an encoder-decoder structure to improve segmentation accuracy. DeepLabv3 + adds a decoder module based on DeepLabv3 to obtain clearer segmentation, in which the convolution operation uses Atrous Convolution, as shown in Fig. 2”).
It would have been obvious to combine Tang, Li, and Li 2 for the same reasons as for claim 13 above.
Allowable Subject Matter
Claims 2 and 18 objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AIDAN KEUP whose telephone number is (703)756-4578. The examiner can normally be reached Monday - Friday 8:00-4:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached at (571) 270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AIDAN KEUP/ Examiner, Art Unit 2666 /Molly Wilburn/Primary Examiner, Art Unit 2666
Prosecution Insights