DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendment filed 6 March, 2026 has been entered. Claims 1-20 remain pending in the application.
Response to Arguments
Applicant’s arguments, see Applicant’s Remarks, filed 6 March, 2026, with respect to the objection to claim 9 have been fully considered and are persuasive. The claim has been amended to correct the error. The objection to claim 9 has been withdrawn.
Applicant’s arguments with respect to the rejection of claim 8 under 35 USC 112(b) have been fully considered and are persuasive. The claim has been amended to correct the antecedent basis of “a headset”. The rejection of claim 8 under 35 USC 112(b) has been withdrawn. However, the rejection of claim 9 is maintained as the claim has not been amended to correct the antecedent basis of “the headset”
Applicant's arguments with respect to the rejection of claims 17-19 under 35 USC 102 have been fully considered but they are not persuasive. Specifically, Applicant argues:
Here, the Office attempts to cite one element of the cited prior art as disclosing multiple distinct elements of claim 17. Namely, the Office recites: "To facilitate remote control, the machine 110 may have a camera 242 mounted on it that records the immediate environment and transmits the image, using the machine transmitter/receiver 114, to a remote location" as teaching both the "one or more sensors with input parameters and on-board inertial measurement units" and the "camera system including one or more cameras" of claim 17. However, the proposed mapping is improper because claim 17 teaches the "one or more sensors" and "camera system" as two separate elements, and thus claim 17 recites at least two sensors, one of which is a camera. Friend is silent as to any sensors on the vehicle(s) in addition to the singular camera 242 identified by the Office and thus fails to disclose, teach, or suggest each and every limitation of claim 17.
The Examiner has carefully considered the argument, however it is not persuasive. Friend does teach separate elements for the “one or more sensors” (See Friend Paragraph 0018 In the present embodiment, the machine transmitter/receivers 114 and personnel transmitter/receivers 116 can be configured to also receive the positioning signals from the positioning transmitters 154 but in other embodiments, a specific and dedicated GPS receiver may be utilized.) and the “camera system” (Paragraph 0027 To facilitate remote control, the machine 110 may have a camera 242 mounted on it that records the immediate environment and transmits the image, using the machine transmitter/receiver 114, to a remote location.) recited in claim 17.
Applicant's arguments with respect to the rejection of claims 1-16 under 35 USC 103 have been fully considered but they are not persuasive. Specifically, Applicant argues:
However, none of the above examples of assistance data provided by Rust include generating time-sequenced data, providing a historical composite of the data, or creating a composite image based on the historical composite data.
The Examiner has carefully considered the argument, however it is not persuasive. The assistance data taught by Rust does teach the broadest reasonable interpretation of generating time-sequenced data (See Rust Paragraph 0060 The assistance request generated and transmitted in S220 preferably includes a set of assistance data to enable an intelligent response to the assistance request.), combining the time-sequenced data to provide a historical composite of the data (See Rust Paragraph 0061 In such case, the autonomous vehicle may send assistance data (pre-assistance data) in advance of the occurrence of the scenario in which an assistance request may be necessary. In this way, the pre-assistance data may be evaluated at the human expert interface 120 and/or the AI expert 140 prior and/or during the scenario in which assistance would be required thereby providing an opportunity for the human expert interface 120 and/or the AI expert 140 to provide preventative assistance in advance of the scenario and/or prepare an assistance response in advance such that the assistance response may be dispatch upon occurrence of the assistance-required scenario.), and creating a composite image based on the historical composite data (See Rust Paragraph 0062 Assistance data may include… processed sensor data (e.g. a camera view overlaid with object identification indicators placed by the autonomous vehicle's on-board computer…)).
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claim 1-12, 14, 17, and 20 rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-4, 6, and 7 of U.S. Patent No. 11,599,125 and U.S. Patent No. 12,085,958. Although the claims at issue are not identical, they are not patentably distinct from each other because claims 1-12, 14, 17, and 20 are anticipated by claims 1-4, 6, and 7 of the reference patents as set forth in the table below.
Claim
Instant Application 18/822,719
Claim
Reference Patent 11,599,125
Reference Patents 12,085,958
1
A material handling vehicle tracking and monitoring system comprising:
1
An autonomous vehicle tracking system comprising
A vehicle tracking system comprising
1
a remote monitoring station including a display interface and a server;
1
(ii) a centralized monitoring station including a computer in communication with the controller of the at least one autonomous vehicle
(ii) a centralized monitoring station including a computer in communication with the controller of the at least one vehicle
1
a material handling vehicle in communication with the remote monitoring station, the material handling vehicle comprising:
1
(i) at least one autonomous vehicle
(i) at least one vehicle
1
one or more sensors with input parameters;
1
a plurality of sensors providing signals
a plurality of sensors providing signals
1
a drive system including a drive and a drive controller, wherein the drive receives control signals from the drive controller;
1
the control system including a drive system providing power for the autonomous vehicle
the control system including a drive system providing power for the vehicle
1
a vehicle controller with a processor and a memory device with programmable instructions executable by the processor that when executed cause the vehicle controller to:
1
including a control system
including a control system
1
receive data from the one or more sensors of the material handling vehicle; process the data and store the data in the memory device of the vehicle controller;
1
aggregate the signals to create an array of data that is a composite of the sensor signals and the camera signals that represents a time-sequenced composite image file that superimposes data derived from the sensor signals onto the images from the camera system
aggregate the signals to create an array of data that is a synchronized composite of the sensor signals and the camera signals that represents a time-sequenced composite image file that superimposes data derived from the sensor signals onto the images from the camera system
1
generate time-sequenced data based on the input parameters of the one or more sensors of the material handling vehicle;
1
aggregate the signals to create an array of data that is a composite of the sensor signals and the camera signals that represents a time-sequenced composite image file that superimposes data derived from the sensor signals onto the images from the camera system
aggregate the signals to create an array of data that is a synchronized composite of the sensor signals and the camera signals that represents a time-sequenced composite image file that superimposes data derived from the sensor signals onto the images from the camera system
1
combine the time-sequenced data to provide a historical composite of the data;
1
aggregate the signals to create an array of data that is a composite of the sensor signals and the camera signals that represents a time-sequenced composite image file that superimposes data derived from the sensor signals onto the images from the camera system
aggregate the signals to create an array of data that is a synchronized composite of the sensor signals and the camera signals that represents a time-sequenced composite image file that superimposes data derived from the sensor signals onto the images from the camera system
1
create a composite image based on the historical composite data; and
1
aggregate the signals to create an array of data that is a composite of the sensor signals and the camera signals that represents a time-sequenced composite image file that superimposes data derived from the sensor signals onto the images from the camera system
aggregate the signals to create an array of data that is a synchronized composite of the sensor signals and the camera signals that represents a time-sequenced composite image file that superimposes data derived from the sensor signals onto the images from the camera system
1
transmit the composite image to the display interface of the remote monitoring station.
1
the transmitted image file being received by the computer and displayed by the display device
the transmitted image file being received by the computer and displayed by the display device
2
The material handling vehicle tracking and monitoring system of claim 1, further comprising: a camera system designed to generate a signal representative of images of an environment surrounding the material handling vehicle.
1
a camera system providing a signal representative of images of the environment surrounding the autonomous vehicle
a camera system providing a signal representative of images of an environment surrounding the vehicle
3
The material handling vehicle tracking and monitoring system of claim 2, wherein the memory devices include instructions that, when executed by the processor, cause the processor to: receive sensor signals from the one or more sensors of the material handling vehicle;
1
cause the processor to receive signals from the sensors and the camera system
cause the first processor to receive signals from the sensors and the camera system
3
receive camera signals from the camera system;
1
cause the processor to receive signals from the sensors and the camera system
cause the first processor to receive signals from the sensors and the camera system
3
aggregate the sensor signals and the camera signals; create a data array that is provided in a form of a synchronized composite of the sensor signals and the camera signals, which represents a time-sequenced composite image file that superimposes data derived from the sensor signals onto the images from the camera system; and
1
aggregate the signals to create an array of data that is a composite of the sensor signals and the camera signals that represents a time-sequenced composite image file that superimposes data derived from the sensor signals onto the images from the camera system
aggregate the signals to create an array of data that is a synchronized composite of the sensor signals and the camera signals that represents a time-sequenced composite image file that superimposes data derived from the sensor signals onto the images from the camera system
3
transmit the time-sequenced composite image file in real-time to the display interface of the remote monitoring station.
1
the transmitted image file being received by the computer and displayed by the display device
the transmitted image file being received by the computer and displayed by the display device
4
The material handling vehicle tracking and monitoring system of claim 1, wherein the remote monitoring station transmits a command signal to the material handling vehicle and a camera associated with the material handling vehicle to simultaneously change a point of time that the time-sequenced data is presented.
1
wherein the computer is operable to transmit a signal to each of the autonomous vehicles and cameras to simultaneously change the point of time that the time-sequenced images are presented
wherein the computer is operable to transmit a signal to each of the vehicles and cameras to simultaneously change a point of time that the time-sequenced images are presented
5
The material handling vehicle tracking and monitoring system of claim 1, wherein the remote monitoring station includes a headset, and
1
wherein the display device comprises a head set
wherein the display device comprises a head set
5
a position of the headset is calibrated from data in an environment surrounding the material handling vehicle.
2
wherein the position of the head set is calibrated from a particular datum in the environment of the plurality of autonomous vehicles.
wherein the position of the head set is calibrated from a particular datum in the environment surrounding the plurality of vehicles.
6
The material handling vehicle tracking and monitoring system of claim 5, wherein the position of the headset is calibrated from a neutral position relative to a particular camera, a second material handling vehicle, or a combination thereof.
3
wherein the position of the head set is calibrated from a neutral position relative to the particular camera or autonomous vehicle.
wherein the position of the head set is calibrated from a neutral position relative to the particular camera or vehicle.
7
The material handling vehicle tracking and monitoring system of claim 1, wherein a user input device of the remote monitoring station includes an input for varying a point in time which corresponds to a portion of an image file associated with the composite image being transmitted, such that a user may choose to view the portion of the image file as it existed a different time from current real-time.
1
wherein the computer is operable to transmit a signal to each of the autonomous vehicles and cameras to simultaneously change the point of time that the time-sequenced images are presented
wherein the computer is operable to transmit a signal to each of the vehicles and cameras to simultaneously change a point of time that the time-sequenced images are presented
8
The material handling vehicle tracking and monitoring system of claim 1, wherein the portion of the image file transmitted is responsive to the position of the headset, such that the user may vary the field of view at the different time to view the surroundings of the vehicle at that point in time.
6
wherein the portion of the image file transmitted is responsive to the position of the head set, such that the user may vary the field of view at the different time to view the surroundings of the autonomous vehicle at that point in time.
wherein the portion of the image file transmitted is responsive to the position of the head set, such that the user may vary the field of view at the different time to view the surroundings of the vehicle at that point in time.
9
The material handling vehicle tracking and monitoring system of claim 1, wherein the time sequence may be paused such that a user may look around the vehicle at a single point in time by moving the head set to change the field of view.
7
wherein the time sequence may be paused such that a user may look around the vehicle at a single point in time by moving the head set to change the field of view.
wherein the time sequence may be paused such that a user may look around the vehicle at a single point in time by moving the head set to change the field of view.
10
The material handling vehicle tracking and monitoring system of claim 1, wherein the material handling vehicle is designed to transmit an alert condition to the remote monitoring station associated with a time of the alert condition, such that a user can select the time of the alert condition to view images from a camera system or the data from the one or more sensors of the material handling vehicle.
4
wherein each of the plurality of autonomous vehicles is operable to transmit an alert condition to the centralized monitoring station, the alert condition prompting the alert to be logged to the particular real time of the alert, and wherein the computer is operable to mark the point in time such that a user may choose the time of the alert to view the images from the cameras or autonomous vehicles.
wherein each of the plurality of vehicles is operable to transmit an alert condition to the centralized monitoring station, the alert condition prompting an alert to be logged to a particular real time of the alert, the computer is operable to mark the point in time such that a user may choose the time of the alert to view the images of the cameras or vehicle.
11
A method for monitoring material handling vehicles, comprising:
1
An autonomous vehicle tracking system comprising
A vehicle tracking system comprising
11
providing a remote monitoring station including a display interface and a server;
1
(ii) a centralized monitoring station including a computer in communication with the controller of the at least one autonomous vehicle
(ii) a centralized monitoring station including a computer in communication with the controller of the at least one vehicle
11
providing a material handling vehicle in communication with the remote monitoring station;
1
(i) at least one autonomous vehicle
(i) at least one vehicle
11
receiving data from one or more sensors on the material handling vehicle via a processor of a vehicle controller of the material handling vehicle;
1
a plurality of sensors providing signals
a plurality of sensors providing signals
11
processing the data using the processor;
1
including a control system
including a control system
11
storing the data in the memory device of the vehicle controller;
1
aggregate the signals to create an array of data that is a composite of the sensor signals and the camera signals that represents a time-sequenced composite image file that superimposes data derived from the sensor signals onto the images from the camera system
aggregate the signals to create an array of data that is a synchronized composite of the sensor signals and the camera signals that represents a time-sequenced composite image file that superimposes data derived from the sensor signals onto the images from the camera system
11
generating time-sequencing data to provide a historical composite of the data;
1
aggregate the signals to create an array of data that is a composite of the sensor signals and the camera signals that represents a time-sequenced composite image file that superimposes data derived from the sensor signals onto the images from the camera system
aggregate the signals to create an array of data that is a synchronized composite of the sensor signals and the camera signals that represents a time-sequenced composite image file that superimposes data derived from the sensor signals onto the images from the camera system
11
creating a composite image based on the historical composite data; and
1
aggregate the signals to create an array of data that is a composite of the sensor signals and the camera signals that represents a time-sequenced composite image file that superimposes data derived from the sensor signals onto the images from the camera system
aggregate the signals to create an array of data that is a synchronized composite of the sensor signals and the camera signals that represents a time-sequenced composite image file that superimposes data derived from the sensor signals onto the images from the camera system
11
transmitting the composite image to the display interface of the remote monitoring station.
1
the transmitted image file being received by the computer and displayed by the display device
the transmitted image file being received by the computer and displayed by the display device
12
The method for monitoring material handling vehicles according to claim 11, further comprising: monitoring updated real-time location and status indicators of the material handling vehicle using the remote monitoring station.
1
the centralized monitoring system including a monitor that provides current status of each of the plurality of autonomous vehicles
the centralized monitoring system station including a monitor that provides current status of each of the plurality of vehicles
14
The method for monitoring material handling vehicles according to claim 11, further comprising: detecting a critical event from one or more sensors on the material handling vehicle using the remote monitoring station.
4
wherein each of the plurality of autonomous vehicles is operable to transmit an alert condition to the centralized monitoring station, the alert condition prompting the alert to be logged to the particular real time of the alert, and wherein the computer is operable to mark the point in time such that a user may choose the time of the alert to view the images from the cameras or autonomous vehicles.
wherein each of the plurality of vehicles is operable to transmit an alert condition to the centralized monitoring station, the alert condition prompting an alert to be logged to a particular real time of the alert, the computer is operable to mark the point in time such that a user may choose the time of the alert to view the images of the cameras or vehicle.
17
A material handling vehicle tracking and monitoring system comprising:
1
An autonomous vehicle tracking system comprising
A vehicle tracking system comprising
17
a remote monitoring system including a display interface and a server;
1
(ii) a centralized monitoring station including a computer in communication with the controller of the at least one autonomous vehicle
(ii) a centralized monitoring station including a computer in communication with the controller of the at least one vehicle
17
a material handling vehicle in communication with the remote monitoring station, the material handling vehicle comprising:
1
(i) at least one autonomous vehicle
(i) at least one vehicle
17
one or more sensors with input parameters and on-board inertial measurement units;
1
a plurality of sensors providing signals
a plurality of sensors providing signals
17
a drive system including a drive and a drive controller, wherein the drive receives control signals from the drive controller;
1
the control system including a drive system providing power for the autonomous vehicle
the control system including a drive system providing power for the vehicle
17
a camera system including one or more cameras;
1
a camera system providing a signal representative of images of the environment surrounding the autonomous vehicle
a camera system providing a signal representative of images of an environment surrounding the vehicle
17
a vehicle controller with a processor and a memory device with programmable instructions executable by the processor that when executed cause the vehicle controller to:
1
including a control system
including a control system
17
receive first signals from the one or more sensors of the material handling vehicle;
1
aggregate the signals to create an array of data that is a composite of the sensor signals and the camera signals that represents a time-sequenced composite image file that superimposes data derived from the sensor signals onto the images from the camera system
aggregate the signals to create an array of data that is a synchronized composite of the sensor signals and the camera signals that represents a time-sequenced composite image file that superimposes data derived from the sensor signals onto the images from the camera system
17
receive second signals from the camera system;
1
aggregate the signals to create an array of data that is a composite of the sensor signals and the camera signals that represents a time-sequenced composite image file that superimposes data derived from the sensor signals onto the images from the camera system
aggregate the signals to create an array of data that is a synchronized composite of the sensor signals and the camera signals that represents a time-sequenced composite image file that superimposes data derived from the sensor signals onto the images from the camera system
17
aggregate the first signals and the second signals received from the one or more sensors and the camera system;
1
aggregate the signals to create an array of data that is a composite of the sensor signals and the camera signals that represents a time-sequenced composite image file that superimposes data derived from the sensor signals onto the images from the camera system
aggregate the signals to create an array of data that is a synchronized composite of the sensor signals and the camera signals that represents a time-sequenced composite image file that superimposes data derived from the sensor signals onto the images from the camera system
17
create visual data using the aggregated signals to generate a time-sequenced composite image file; and
1
aggregate the signals to create an array of data that is a composite of the sensor signals and the camera signals that represents a time-sequenced composite image file that superimposes data derived from the sensor signals onto the images from the camera system
aggregate the signals to create an array of data that is a synchronized composite of the sensor signals and the camera signals that represents a time-sequenced composite image file that superimposes data derived from the sensor signals onto the images from the camera system
17
transmit a time-sequenced composite image file in real-time to the display interface of the remote monitoring station.
1
the transmitted image file being received by the computer and displayed by the display device
the transmitted image file being received by the computer and displayed by the display device
20
The material handling vehicle tracking and monitoring system of claim 17, wherein the one or more sensors include an array of cameras, wherein an operator may toggle between views of one or more material handling vehicles and each camera of the array of cameras at a coordinated point in time using a headset worn by the operator.
1
wherein the computer is operable to transmit a signal to each of the autonomous vehicles and cameras to simultaneously change a point of time that the time-sequenced images are presented such that an operator may toggle between views of each of the autonomous vehicles and each of the cameras at a coordinated point in time, the portion of the image file being transmitted by each autonomous vehicle being responsive to a position of the head set of the user.
wherein the computer is operable to transmit a signal to each of the vehicles and cameras to simultaneously change a point of time that the time-sequenced images are presented such that an operator may toggle between views of each of the vehicles and each of the cameras at a coordinated point in time, the portion of the image file being transmitted by each vehicle being responsive to a position of the head set of the user.
Claim Rejections - 35 USC § 112
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claim 9 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 9 recites the limitation "the headset". There is insufficient antecedent basis for this limitation in this claim.
Claim Rejections - 35 USC § 102
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Regarding claim 17, Friend teaches a material handling vehicle tracking and monitoring system comprising: a remote monitoring system including a display interface and a server (Paragraph 0017 To coordinate and control the activities and movement of the machines 110 and the personnel 112 about the physical worksite 100, a computerized or electronically implemented management system 130 can be based out of a fixed location or an immobile workstation 132.); a material handling vehicle in communication with the remote monitoring station (Paragraph 0015 For example, the machine may be an earth-moving machine, such as a wheel loader, dump truck, backhoe, material handler or the machine may be another type of working machine such as compactors, pavers, etc, or may be an over-the-road hauling type of machine.), the material handling vehicle comprising: one or more sensors with input parameters and on-board inertial measurement units (Paragraph 0018 In the present embodiment, the machine transmitter/receivers 114 and personnel transmitter/receivers 116 can be configured to also receive the positioning signals from the positioning transmitters 154 but in other embodiments, a specific and dedicated GPS receiver may be utilized.); a drive system including a drive and a drive controller, wherein the drive receives control signals from the drive controller (Paragraph 0015 For example, the machine may be an earth-moving machine, such as a wheel loader, dump truck, backhoe, material handler or the machine may be another type of working machine such as compactors, pavers, etc, or may be an over-the-road hauling type of machine. [such vehicles inherently comprise drive systems]); a camera system including one or more cameras (Paragraph 0027 To facilitate remote control, the machine 110 may have a camera 242 mounted on it that records the immediate environment and transmits the image, using the machine transmitter/receiver 114, to a remote location.); a vehicle controller with a processor and a memory device (Paragraph 0023 Those signals may be processed through an onboard controller 210 disposed on the machine 110 and communicating with the machine transmitter/receiver 114.) with programmable instructions executable by the processor that when executed cause the vehicle controller to: receive first signals from the one or more sensors of the material handling vehicle; receive second signals from the camera system; aggregate the first signals and the second signals received from the one or more sensors and the camera system; create visual data using the aggregated signals to generate a time-sequenced composite image file (Paragraph 0027 The captured image thus assists the operator in directing the machine as if the operator were onboard the machine. Augmentation information associated with the captured image of the worksite on the display 250 can be superimposed on the display 250 to augment the image.); and transmit a time-sequenced composite image file in real-time to the display interface of the remote monitoring station (Paragraph 0027 The view captured by the camera 242 can be displayed on a display 250, such as a liquid crystal display (LCD) or cathode ray tube (CRT), at a remote location from where the machine 110 is controlled.).
Regarding claim 18, Friend teaches the system of claim 17 as set forth above. Friend further teaches wherein the material handling vehicle includes a communication circuitry to communicate with the remote monitoring station (Paragraph 0023 Those signals may be processed through an onboard controller 210 disposed on the machine 110 and communicating with the machine transmitter/receiver 114.).
Regarding claim 19, Friend teaches the system of claim 18 as set forth above. Friend further teaches wherein the communication circuitry includes a wireless communication protocol to share data between the material handling vehicle and the remote monitoring station in real-time (Paragraph 0023 Those signals may be processed through an onboard controller 210 disposed on the machine 110 and communicating with the machine transmitter/receiver 114.).
Claim Rejections - 35 USC § 103
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claim(s) 1-3, 5, 6, and 11-13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Friend in view of Rust (U.S. Patent application Publication 2017/0192423).
Regarding claim 1, Friend teaches a material handling vehicle tracking and monitoring system comprising: a remote monitoring station including a display interface and a server (Paragraph 0017 To coordinate and control the activities and movement of the machines 110 and the personnel 112 about the physical worksite 100, a computerized or electronically implemented management system 130 can be based out of a fixed location or an immobile workstation 132.); a material handling vehicle in communication with the remote monitoring station (Paragraph 0015 For example, the machine may be an earth-moving machine, such as a wheel loader, dump truck, backhoe, material handler or the machine may be another type of working machine such as compactors, pavers, etc, or may be an over-the-road hauling type of machine.), the material handling vehicle comprising: one or more sensors with input parameters (Paragraph 0018 In the present embodiment, the machine transmitter/receivers 114 and personnel transmitter/receivers 116 can be configured to also receive the positioning signals from the positioning transmitters 154 but in other embodiments, a specific and dedicated GPS receiver may be utilized.); a drive system including a drive and a drive controller, wherein the drive receives control signals from the drive controller (Paragraph 0015 For example, the machine may be an earth-moving machine, such as a wheel loader, dump truck, backhoe, material handler or the machine may be another type of working machine such as compactors, pavers, etc, or may be an over-the-road hauling type of machine. [such vehicles inherently comprise drive systems]); a vehicle controller with a processor and a memory device (Paragraph 0023 Those signals may be processed through an onboard controller 210 disposed on the machine 110 and communicating with the machine transmitter/receiver 114.) with programmable instructions executable by the processor that when executed cause the vehicle controller to: receive data from the one or more sensors of the material handling vehicle; process the data and store the data in the memory device of the vehicle controller (Paragraph 0023 Those signals may be processed through an onboard controller 210 disposed on the machine 110 and communicating with the machine transmitter/receiver 114.); create a composite image based on the data (Paragraph 0027 The captured image thus assists the operator in directing the machine as if the operator were onboard the machine. Augmentation information associated with the captured image of the worksite on the display 250 can be superimposed on the display 250 to augment the image.); and transmit the composite image to the display interface of the remote monitoring station (Paragraph 0027 The view captured by the camera 242 can be displayed on a display 250, such as a liquid crystal display (LCD) or cathode ray tube (CRT), at a remote location from where the machine 110 is controlled.).
However, Friend does not teach that the vehicle controller is caused to generate time-sequenced data based on the input parameters of the one or more sensors of the material handling vehicle; and combine the time-sequenced data to provide a historical composite of the data.
Rust, in the same field of endeavor, teaches a remote vehicle operating system wherein the vehicle captures images and processes the images with overlaid time-sequenced sensor data on-board the vehicle before transmitting them to a central control station (Paragraph 0059 The assistance request… transmitted in S220 preferably includes a set of assistance data; Paragraph 0062 Assistance data may include… processed sensor data (e.g. a camera view overlaid with object identification indicators placed by the autonomous vehicle's on-board computer…)).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to have modified Friend with the teachings of Rust which teaches capturing images and processing the images with overlaid time-sequenced sensor data on-board the vehicle before transmitting them to a central control station in order to allow the user at the monitoring station to fully evaluate the situation at the vehicle (See Rust Paragraph 0060 The assistance request generated and transmitted in S220 preferably includes a set of assistance data to enable an intelligent response to the assistance request.).
Regarding claim 2, Friend in view of Rust teaches the system of claim 1 as set forth above. Friend further teaches a camera system designed to generate a signal representative of images of an environment surrounding the material handling vehicle (Paragraph 0027 To facilitate remote control, the machine 110 may have a camera 242 mounted on it that records the immediate environment and transmits the image, using the machine transmitter/receiver 114, to a remote location.).
Regarding claim 3, Friend in view of Rust teaches the system of claim 2 as set forth above. Friend further teaches wherein the memory devices include instructions that, when executed by the processor, cause the processor to: receive sensor signals from the one or more sensors of the material handling vehicle; receive camera signals from the camera system (Paragraph 0027 To facilitate remote control, the machine 110 may have a camera 242 mounted on it that records the immediate environment and transmits the image, using the machine transmitter/receiver 114, to a remote location.); aggregate the sensor signals and the camera signals (Paragraph 0027 The captured image thus assists the operator in directing the machine as if the operator were onboard the machine. Augmentation information associated with the captured image of the worksite on the display 250 can be superimposed on the display 250 to augment the image.); and transmit the composite image file in real-time to the display interface of the remote monitoring station (Paragraph 0027 The view captured by the camera 242 can be displayed on a display 250, such as a liquid crystal display (LCD) or cathode ray tube (CRT), at a remote location from where the machine 110 is controlled.).
However, Friend does not teach that the processor is caused to create a data array that is provided in a form of a synchronized composite of the sensor signals and the camera signals, which represents a time-sequenced composite image file that superimposes data derived from the sensor signals onto the images from the camera system.
Rust, in the same field of endeavor, teaches a remote vehicle operating system wherein the vehicle captures images and processes the images with overlaid time-sequenced sensor data on-board the vehicle before transmitting them to a central control station (Paragraph 0059 The assistance request… transmitted in S220 preferably includes a set of assistance data; Paragraph 0062 Assistance data may include… processed sensor data (e.g. a camera view overlaid with object identification indicators placed by the autonomous vehicle's on-board computer…)).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to have modified Friend with the teachings of Rust which teaches capturing images and processing the images with overlaid time-sequenced sensor data on-board the vehicle before transmitting them to a central control station in order to allow the user at the monitoring station to fully evaluate the situation at the vehicle (See Rust Paragraph 0060 The assistance request generated and transmitted in S220 preferably includes a set of assistance data to enable an intelligent response to the assistance request.).
Regarding claim 5, Friend in view of Rust teaches the system of claim 1 as set forth above. Friend further teaches wherein the remote monitoring station includes a headset, and a position of the headset is calibrated from data in an environment surrounding the material handling vehicle (Paragraph 0026 In a possible further feature, the HMD system 220 can be configured to receive augmented content not only specific to the location of the person 112, but specific to the person's line of view. For example, a plurality of sensors 234 can be disposed about the harness 222 to determine the orientation of the head of the wearer. For example, the sensors 234 may be Hall effect sensors that utilize the variable relative positions of a transducer and a magnetic field to deduce the direction, pitch, yaw and roll of an individual's head. The data obtained by these sensors 234 can be transmitted to the management system along with the more general positional data so that the management system may know the orientation of the person's field of view.).
Regarding claim 6, Friend in view of Rust teaches the system of claim 5 as set forth above. Friend further teaches wherein the position of the headset is calibrated from a neutral position relative to a particular camera, a second material handling vehicle, or a combination thereof (Paragraph 0026 In a possible further feature, the HMD system 220 can be configured to receive augmented content not only specific to the location of the person 112, but specific to the person's line of view. For example, a plurality of sensors 234 can be disposed about the harness 222 to determine the orientation of the head of the wearer. For example, the sensors 234 may be Hall effect sensors that utilize the variable relative positions of a transducer and a magnetic field to deduce the direction, pitch, yaw and roll of an individual's head. The data obtained by these sensors 234 can be transmitted to the management system along with the more general positional data so that the management system may know the orientation of the person's field of view.).
Regarding claim 11, the claim is commensurate in scope with claim 1 with the exception that claim 11 is directed to a method. Therefore, the same prior art can be applied to claim 11 as was applied to claim 1.
Regarding claim 12, Friend in view of Rust teaches the method of claim 11 as set forth above. Friend further teaches monitoring updated real-time location and status indicators of the material handling vehicle using the remote monitoring station (Paragraph 0028 The augmentation content generator 302 can receive inputs and process, develop or generate outputs that are communicated to the operator display devices. Inputs may include positional information 310 such as machine positions 312, personnel positions 314 and, where appropriate, orientation 316 of the direction of view.).
Regarding claim 13, Friend in view of Rust teaches the method of claim 11 as set forth above. Friend further teaches transmitting to the display a color based on an error condition and urgency for the material handling vehicle using the remote monitoring station (Paragraph 0035 An operator display device within or near the blasting operation 108 may superimpose the exclusion zone 400 over the view of physical worksite 100, for example, as hatching or an outline delineation in an appropriate color, thereby indicating to nearby personnel 112 that the otherwise indistinguishable exclusion zone should be vacated and avoided.).
Claim(s) 4, 7-10, and 14-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Friend in view of Rust and Donahoe (U.S. Patent Application Publication 2019/0250601).
Regarding claim 4, Friend in view of Rust teaches the system of claim 1 as set forth above. However, Friend in view of Rust does not teach wherein the remote monitoring station transmits a command signal to the material handling vehicle and a camera associated with the material handling vehicle to simultaneously change a point of time that the time-sequenced data is presented.
Donahoe, in the same field of endeavor, teaches a system for monitoring a vehicle. The system records images from a camera of the vehicle and displays the images in either real-time or a past time (Paragraph 0030 UAV can also capture images… that can be displayed in real time and/or recorded for later display at other devices (e.g. mobile device 104)…).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to have modified Friend in view of Rust with the teachings of Donahoe which teaches recording images from a camera of the vehicle and displaying the images in either real-time or a past time in order to allow the user monitoring the vehicle to review relevant data and images from the past (See Donahoe Paragraph 0143 For example, if a user controlling the UAV 100 notices a tracked subject performing an interesting activity, the user can provide an input to, in effect, tag that portion of the captured media as relevant. Notably, the user does not need to start and stop recording of any media in order to mark the event as relevant.).
Regarding claim 7, Friend in view of Rust teaches the system of claim 1 as set forth above. However, Friend in view of Rust does not teach wherein a user input device of the remote monitoring station includes an input for varying a point in time which corresponds to a portion of an image file associated with the composite image being transmitted such that a user may choose to view the portion of the image file as it existed a different time from current real-time.
Donahoe, in the same field of endeavor, teaches a system for monitoring a vehicle. The system records images from a camera of the vehicle and displays the images in either real-time or a past time (Paragraph 0030 UAV can also capture images… that can be displayed in real time and/or recorded for later display at other devices (e.g. mobile device 104)…).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to have modified Friend in view of Rust with the teachings of Donahoe which teaches recording images from a camera of the vehicle and displaying the images in either real-time or a past time in order to allow the user monitoring the vehicle to review relevant data and images from the past (See Donahoe Paragraph 0143 For example, if a user controlling the UAV 100 notices a tracked subject performing an interesting activity, the user can provide an input to, in effect, tag that portion of the captured media as relevant. Notably, the user does not need to start and stop recording of any media in order to mark the event as relevant.).
Regarding claim 8, Friend in view of Rust and Donahoe teaches the system of claim 7 as set forth above. Friend further teaches wherein the portion of the image file transmitted is responsive to the position of the headset, such that the user may vary the field of view at the different time to view the surroundings of the vehicle at that point in time (Paragraph 0026 In a possible further feature, the HMD system 220 can be configured to receive augmented content not only specific to the location of the person 112, but specific to the person's line of view. For example, a plurality of sensors 234 can be disposed about the harness 222 to determine the orientation of the head of the wearer. For example, the sensors 234 may be Hall effect sensors that utilize the variable relative positions of a transducer and a magnetic field to deduce the direction, pitch, yaw and roll of an individual's head.).
Regarding claim 9, Friend in view of Rust teaches the system of claim 1 as set forth above. However, Friend in view of Rust does not teach wherein the time sequence may be paused such that a user may look around the vehicle at a single point in time by moving the headset to change the field of view.
Donahoe, in the same field of endeavor, teaches a system for monitoring a vehicle. The system records images from a camera of the vehicle and displays the images in either real-time or a past time, including a still frame (Paragraph 0143 This mark can annotate a still photo…).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to have modified Friend in view of Rust with the teachings of Donahoe which teaches recording images from a camera of the vehicle and displaying the images in either real-time or a past time, including a still frame, in order to allow the user monitoring the vehicle to review relevant data and images from the past (See Donahoe Paragraph 0143 For example, if a user controlling the UAV 100 notices a tracked subject performing an interesting activity, the user can provide an input to, in effect, tag that portion of the captured media as relevant. Notably, the user does not need to start and stop recording of any media in order to mark the event as relevant.).
Regarding claim 10, Friend in view of Rust teaches the system of claim 1 as set forth above. However, Friend in view of Rust does not teach wherein the material handling vehicle is designed to transmit an alert condition to the remote monitoring station associated with a time of the alert condition such that a user can select the time of the alert condition to view images from a camera system or the data from the one or more sensors of the material handling vehicle.
Donahoe, in the same field of endeavor, teaches a system for monitoring a vehicle. The system tags the recorded images when an event occurs and displays the recorded images at the time of the tagged event (Paragraph 0143 The tag may correspond to an instantaneous point in time or may correspond with a period of time. For example, when recording video, the user may press the interactive tagging element once to mark a beginning point of the relevant period of time and then press the interactive tagging element a second time to mark the end of the relevant period of time. This mark can annotate a still photo, the start or end of a video, or any other meaning the user wishes.).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to have modified Friend in view of Rust with the teachings of Donahoe which teaches tagging the recorded images when an event occurs and displaying the recorded images at the time of the tagged event in order to allow the user monitoring the vehicle to review relevant data and images from the past (See Donahoe Paragraph 0143 For example, if a user controlling the UAV 100 notices a tracked subject performing an interesting activity, the user can provide an input to, in effect, tag that portion of the captured media as relevant. Notably, the user does not need to start and stop recording of any media in order to mark the event as relevant.).
Regarding claim 14, Friend in view of Rust teaches the method of claim 11 as set forth above. However, Friend in view of Rust does not teach detecting a critical event from one or more sensors on the material handling vehicle using the remote monitoring station.
Donahoe, in the same field of endeavor, teaches a system for monitoring a vehicle. The system tags the recorded images when an event occurs and displays the recorded images at the time of the tagged event (Paragraph 0143 The tag may correspond to an instantaneous point in time or may correspond with a period of time. For example, when recording video, the user may press the interactive tagging element once to mark a beginning point of the relevant period of time and then press the interactive tagging element a second time to mark the end of the relevant period of time. This mark can annotate a still photo, the start or end of a video, or any other meaning the user wishes.).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to have modified Friend in view of Rust with the teachings of Donahoe which teaches tagging the recorded images when an event occurs and displaying the recorded images at the time of the tagged event in order to allow the user monitoring the vehicle to review relevant data and images from the past (See Donahoe Paragraph 0143 For example, if a user controlling the UAV 100 notices a tracked subject performing an interesting activity, the user can provide an input to, in effect, tag that portion of the captured media as relevant. Notably, the user does not need to start and stop recording of any media in order to mark the event as relevant.).
Regarding claim 15, Friend in view of Rust teaches the method of claim 11 as set forth above. However, Friend in view of Rust does not teach generating a point cloud from three-dimensional data obtained from one or more sensors using the remote monitoring station.
Donahoe, in the same field of endeavor, teaches a system for monitoring a vehicle. The system collects depth information combined with camera images (Paragraph 0215 In some embodiments, the cameras of an image capture device 4134 may be arranged such that at least two cameras are provided with overlapping FOV at multiple angles around the UAV 100, thereby allowing for stereoscopic (i.e., 3D) image/video capture and depth recovery (e.g., through computer vision algorithms) at multiple angles around UAV 100.).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to have modified Friend in view of Rust with the teachings of Donahoe which teaches collecting depth information combined with camera images in order to create a stereoscopic 3D image (See Donahoe Paragraph 0215 For example, UAV 100 may include four sets of two cameras each positioned so as to provide a stereoscopic view at multiple angles around the UAV 100.)
Regarding claim 16, Friend in view of Rust teaches the method of claim 11 as set forth above. However, Friend in view of Rust does not teach presenting the point cloud in an image plane using the remote monitoring station.
Donahoe, in the same field of endeavor, teaches a system for monitoring a vehicle. The system collects depth information combined with camera images (Paragraph 0215 In some embodiments, the cameras of an image capture device 4134 may be arranged such that at least two cameras are provided with overlapping FOV at multiple angles around the UAV 100, thereby allowing for stereoscopic (i.e., 3D) image/video capture and depth recovery (e.g., through computer vision algorithms) at multiple angles around UAV 100.).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to have modified Friend in view of Rust with the teachings of Donahoe which teaches collecting depth information combined with camera images in order to create a stereoscopic 3D image (See Donahoe Paragraph 0215 For example, UAV 100 may include four sets of two cameras each positioned so as to provide a stereoscopic view at multiple angles around the UAV 100.)
Allowable Subject Matter
Claim 20 objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims and if the nonstatutory double patenting rejection is overcome.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PATRICK D MOHL whose telephone number is (571)272-8987. The examiner can normally be reached M-Th 6:00AM-4:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Anne Antonucci can be reached at (313) 446-6519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PATRICK DANIEL MOHL/Examiner, Art Unit 3666
/ANNE MARIE ANTONUCCI/Supervisory Patent Examiner, Art Unit 3666
Prosecution Insights