DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This office action is in response to applicant’s RCE with amendment/remarks filed 06/22/2026. Claims 1, 5, 10, 13, 15-16 and 19 have been amended. Claims 6, 14, and 20 have been cancelled and claims 21-23 have been newly added. Accordingly, claims 1-5, 7-13, 15-19 and 21-23 are pending.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 06/22/2026 has been entered.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 06/10/2026 have been fully considered by the examiner.
Response to Arguments
Applicant's arguments, see page 8 filed 06/05/2026, with respect to the 35 U.S.C. 103 rejection have been fully considered but they are not persuasive. The applicant discloses that the primary reference Rahim US5,155,683A does not disclose the limitation “displaying an overlay in a video stream based on the pixel coordinates, the overlay including a visualization of a projected path of the vehicle”, and that moving a cursor on a screen is not the same as generating an overlay based on pixel coordinates. The examiner respectfully disagrees. The abstract of Rahim discloses “The path appears as a computer-generated line superimposed on the image of the vehicle's environment, like a stripe painted on the ground. The operator can change or advance the path on the screen with a cursor control”. A computer generated line superimposed on the image that depicts a path is a clear overlay that is generated. The operator’s pointer inputs sets these coordinates and claim 1 of Rahim discloses “a cursor control for the operator to move said cursor on said screen to determine placement of said screen path on said screen”. A cursor position on a screen is a pixel coordinate. Therefore, this claim limitation remains rejected under Rahim.
Applicant’s arguments, see page 9 filed 06/05/2026, with respect to the 35 U.S.C. 103 rejection have been fully considered and are persuasive, since the prior art on record does not disclose the newly added limitation “wherein displaying the overlay is based on determining that a path with a fixed curvature can be generated from a current position of the vehicle to a location in the video stream corresponding to the pixel coordinates.” Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made with Rahim US5,155,683A, Rankawat et al. US20190286153A1, Caldwell et al. US20200409368A1, and Moore et al. US20190155295A1 (henceforth Moore). See the new 35 U.S.C. 103 rejection below.
Applicant's arguments, see page 9 filed 06/05/2026 have been fully considered but they are not persuasive. The applicant traverses the rejection of claims 2-3, 11-12, and 17-18 by disclosing that “the office does not demonstrate that Kumavat teaches an overlap of the pixel coordinates”. However, this limitation is not claimed (i.e. overlapping of pixel coordinates). Therefore, the claims as recited above continue to be rejected under the prior art on record.
Applicant's arguments, see pages 9-10 filed 06/05/2026 have been fully considered but they are not persuasive. The applicant traverses the rejection of claims 4, 13, 19 and claims 6-8 but does not disclose why. Therefore, the claims as recited above continue to be rejected under the prior art on record.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claim 22 is rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Newly added claim 22 recites, “wherein determining the three dimensional spatial coordinates comprises converting the two dimensional pixel coordinates to film coordinates using the transformation component, and converting the film coordinates to the three dimensional spatial coordinates using camera intrinsic parameters of the camera”. The newly added limitation appears to be silent in the specifications filed 03/31/2023. The specifications does not describe this specific limitation in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor had possession of the claimed invention. Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 1, 9-10, 15-16, and 21-22 are rejected under 35 U.S.C. 103 as being unpatentable over Rahim US5,155,683A in view of Rankawat et al. US20190286153A1 (henceforth Rankawat) and Caldwell et al. US20200409368A1 (henceforth Caldwell) further in view of Moore et al. US20190155295A1 (henceforth Moore).
Regarding claim 1,
Rahim discloses:
A method, comprising: receiving a video stream from a camera of a vehicle in a transportation network; (See at least Column 5, lines 19-23, wherein a video stream (i.e. a low data rate stream) is received from a camera of a vehicle. Further see Fig. 6, “camera-video data”.)
Wherein the video stream comprises coordinates that correspond to coordinates in a focal plane of the camera; Converting the coordinates into pixel coordinates with a transformation component. (See at least Fig. 2, which shows a focal plane of the camera, and at least Fig. 6, “camera-video data”. Further see Column 5, lines 12-18, “the transform which maps the screen path onto the ground path uses simple trigonometric formulas and perhaps coordinate transformations. The transform and parameters depend on the camera orientation and lens. The transform parameters can be continuously adjusted if the camera zooms, pans or tilts.” The video stream comprises coordinates that corresponds to coordinates in a focal plane of the camera, and the video stream comprises coordinates since the coordinates are transformed onto the ground path via coordinate transformations.)
receiving an input indicating the pixel coordinates in the video stream;
(See at least Column 6, lines 66-67, “The operator O, seen at the control station in FIG. 1, uses a cursor control joystick 10 to extend or modify the screen path line 12 as seen on the screen 14. The operator O traces out an apparent path for the vehicle V with the aid of a cursor 18 which is superimposed on the viewing image at the end of the line 12. The operator O can move the cursor 18 about at will with the joystick 10.” The operator uses a control joystick that indicates pixel coordinates in the video stream (see Fig. 1 and Fig. 2) to control the vehicle V.)
displaying an overlay in the video stream based on the pixel coordinates, the overlay including a visualization of a projected path of the vehicle; determining, from the pixel coordinates, spatial coordinates of the projected path of the vehicle in the transportation network, wherein the spatial coordinates are relative to at least one of the transportation network or the camera;
(See at least Fig. 1 and Column 8 lines 11-22, “the cursor 18 sweeps out the line 12 on the screen, just as a marker leaves a line on paper, to denote the screen path. As an alternative, the cursor 18 might be used to set the screen positions of the individual ground waypoints 20 through which the vehicle V would pass, like dots on paper. The screen points could be clicked on with a button and appear on the screen as dots or cursor shapes. The computer 16 would then transform the coordinates of the screen path points into ground waypoint coordinates for radio uplink transmission to the vehicle's guidance system.“ As shown in Fig. 1, the overlay is displayed in the video stream based on the pixel coordinates, wherein those coordinates are transferred to the vehicle’s guidance system. Since the video stream is based on the camera on the vehicle, then the spatial coordinates are relative to the camera.)
causing the vehicle based on a selection of the overlay, the information configured to cause the vehicle to follow the projected path according to the spatial coordinates based upon the information. (See at least Fig. 1, and Column 8 lines 16-22, “The computer 16 would then transform the coordinates of the screen path points into ground waypoint coordinates for radio uplink transmission to the vehicle's guidance system.” The information is transmitted to the vehicle based on the selection of the overlay (i.e. from the operator input) to follow the projected path according to the spatial coordinates.)
Rahim does not specifically state “converting the film coordinates into pixel coordinates”. However, Rankawat teaches:
converting the film coordinates into pixel coordinates(See at least Para. 0067, “film coordinates may be converted to pixel coordinates. Once the pixel coordinates are known, and the boundary is determined (e.g., the boundary points 106 are determined), the location of the boundary in the real-world coordinates may be determined using the known mapping from the boundary points 106 to the real-world coordinate”. The film coordinates are converted into pixel coordinates.)
It would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to have modified Rahim to incorporate the teachings of Rankawat to include “converting the film coordinates into pixel coordinates” in order to better determine the location of the boundary in the real-world coordinate (Para. 0067, Rankawat). Furthermore, converting between coordinates would create a more robust system for projecting coordinates on a remote operator’s screen, and to better display a visualization of a projected path of the vehicle. Additionally, a person having ordinary skill in the art would have a reasonable expectation of success in combining the teachings of Rahim and Rankawat. The claimed invention is merely a combination of known elements and in combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the results of the combination would have been predictable.
Rahim does not specifically state wherein displaying the overlay is based on determining that a path can be generated from a current position of the vehicle to a location in the video stream corresponding to the pixel coordinates.
However, Caldwell teaches:
wherein displaying the overlay is based on determining that a path can be generated from a current position of the vehicle to a location in the video stream corresponding to the pixel coordinates. (See at least Para. 0067, “The operator may assess each of the views to determine a path 218 for the vehicle to transit though the remote guidance scenario 214. The path 218 may represent a straight-line distance between an initial position 222 of the vehicle 210 and a first waypoint 206(1)” and Para. 0044, “the GUI may not permit waypoints 112 to be input outside of the drive corridor 110.” Further see Para. 0045, “In some examples, the GUI may not permit waypoints 112 to be input more than the distance D away from one another. In some examples, the GUI may activate an area with a radius of D around the vehicle 102 in which the operator may input a waypoint 112”. If the waypoint is not valid, then the GUI does not permit the waypoint to be input by the operator. Therefore, if the waypoint is not valid, the overlay that represents a visualization of the projected path cannot be displayed.)
It would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to have modified Rahim to incorporate the teachings of Caldwell to include “wherein displaying the overlay is based on determining that a path can be generated from a current position of the vehicle to a location in the video stream corresponding to the pixel coordinates” for “for remotely providing incremental guidance to a vehicle operating in an environment that has encountered a scenario that is difficult to navigate“ and “the techniques described herein improve the safety of the vehicle operating in the environment” (Para. 0011, Caldwell). This would create a more robust remote operation assistant system since displaying a path overlay upon determining it is kinematically feasible before acting on it improves safety for the vehicle. Additionally, a person having ordinary skill in the art would have a reasonable expectation of success in combining the teachings of Rahim and Caldwell. The claimed invention is merely a combination of known elements and in combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the results of the combination would have been predictable.
Rahim and Caldwell does not specifically state that the path is with a fixed curvature.
However, Moore teaches:
A path with a fixed curvature. (See at least Fig. 31 and Para. 0181, “FIG. 31 shows robot 18 advanced along path 772 (for clarify shown as an exaggerated variation from arc path 762) to pose x′.sub.R, y′.sub.R, θ′.sub.R, which may result in a small angular error Φ between the tangent line 766 of circle 764, at location x′.sub.R, y′.sub.R, and trajectory 768 extended in the direction of orientation θ′.sub.R. At each iteration and incremental movement, the radius r should not change. That radius r does not change after each incremental movement implies that robot 18 remains substantially on the arc path 762”. The path between the robot and a waypoint is a fixed curvature (i.e. radius r does not change).)
It would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to have modified Rahim and Caldwell to incorporate the teachings of Moore to include “A path with a fixed curvature” since a “radius r does not change after each incremental movement implies that robot 18 remains substantially on the arc path 762”, which would create a more robust system for a robot/vehicle traveling from one point to another. Additionally, a person having ordinary skill in the art would have a reasonable expectation of success in combining the teachings of Rahim, Caldwell, and Moore. The claimed invention is merely a combination of known elements and in combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the results of the combination would have been predictable.
Regarding claim 9,
Rahim discloses:
receiving a second input indicating second pixel coordinates in the video stream while the vehicle is in motion in the transportation network; and displaying a second overlay in the video stream based on the second input, the second overlay including a visualization of a second projected path of the vehicle
(See at least Column 8 lines 45-50, “Since it may be necessary to readjust the projected path of the vehicle in the face of emergencies or miscalculations, the cursor control should have the capability of erasing the end of the screen path, that is, "back-tracking". It may be helpful to have a separate cursor reverse control which would erase the screen path line 12 from the end of the line back toward the vehicle.” A separate cursor reverse control is a second input that is a second overlay that includes a visualization of a second projected path of the vehicle (i.e. erasing the end of a first projected path involves creating a second projected path).)
Regarding claim 10,
Rahim, Rankawat, Caldwell and Moore discloses the same limitations as recited in claim 1 above, and is therefore rejected under the same rejection and obviousness rational. Rahim further discloses:
display an overlay in the video stream based on the two dimensional pixel coordinates
(See Fig. 2 and Fig. 3, wherein an overlay of the path based on the two dimensional pixel coordinates from the user’s cursor is displayed.)
wherein the pixel coordinates are two dimensional pixel coordinates; (See at least Fig. 2, wherein the coordinates are two-dimensional pixel coordinate.)
determine, from the two dimensional pixel coordinates, three dimensional spatial coordinates of the projected path of the vehicle in the transportation network, wherein the three dimensional spatial coordinates are relative to the transportation network. (See at least Fig. 1 and 3, and Column 3 lines 33-35, “turns them into Cartesian space coordinates x, y, z at the vehicle location”, wherein the 3D coordinates of the projected path of the vehicle is determined based on the location of the cursor.)
transmit information to the vehicle based on a selection of the overlay, the information configured to cause the vehicle to follow the projected path according to the three dimensional spatial coordinates.(See at least Column 3 lines 33-35, “turns them into Cartesian space coordinates x, y, z at the vehicle location” and Column 3 lines 46-52, “The station computer then takes the control readings recorded from the waypoints, transforms them into the appropriate commands (vehicle angles, segment lengths, compass headings), and relays these commands to the vehicle. The received commands tell the vehicle's guidance system how to proceed. The vehicle automatically responds to the commands by moving to the next waypoint; eventually it reaches the final point.” The information is transmitted to the vehicle such that it follows the projected path according to the three dimensional spatial coordinates.)
Regarding claim 15,
Rahim discloses:
wherein the input is received through a graphical user interface (GUI) by a detecting a controller pointing to a location in the video stream.
(See at least Column 3, lines 28-32, “The control drives a cursor which is superimposed on the picture which the operator sees, and which appears to move about in space in response to the operator's motion of the three-dimensional control.” Additionally see Fig. 1, wherein the input is received through a GUI by detecting a controlling pointing to a location in the video stream.)
Regarding claim 16,
Rahim, Rankawat, Caldwell and Moore discloses the same limitations as recited in claim 1 above, and is therefore rejected under the same rejection and obviousness rational.
Rahim further discloses:
receiving a second input indicating a selection of the overlay
(See at least column 3 lines 41-45, “The operator, by depressing a button, can denote any cursor position as a waypoint. He or she denotes a series of waypoints to define points of a path in the space seen in the viewer, over which the operator wants the vehicle to travel.” The operator can input a second input indicating a selection of the overlay.)
Regarding claim 21,
Rahim does not specifically state wherein the transformation component converts the film coordinates to the pixel coordinates using camera intrinsic parameters of the camera including at least one of a focal length of the camera or a pixel density of the camera. However, Rankawat teaches:
wherein the transformation component converts the film coordinates to the pixel coordinates using camera intrinsic parameters of the camera including at least one of a focal length of the camera or a pixel density of the camera.(See at least Para. 0067, “film coordinates may be converted to pixel coordinates. Once the pixel coordinates are known, and the boundary is determined (e.g., the boundary points 106 are determined), the location of the boundary in the real-world coordinates may be determined using the known mapping from the boundary points 106 to the real-world coordinate”. The film coordinates are converted into pixel coordinates. Further see Para. 0034 and 0061, wherein the Focal length (i.e. intrinsic camera parameters) is used.)
It would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to have modified Rahim to incorporate the teachings of Rankawat to include “wherein the transformation component converts the film coordinates to the pixel coordinates using camera intrinsic parameters of the camera including at least one of a focal length of the camera or a pixel density of the camera” in order to better determine the location of the boundary in the real-world coordinate (Para. 0067, Rankawat). Furthermore, converting between coordinates would create a more robust system for projecting coordinates on a remote operator’s screen, and to better display a visualization of a projected path of the vehicle. Additionally, a person having ordinary skill in the art would have a reasonable expectation of success in combining the teachings of Rahim and Rankawat. The claimed invention is merely a combination of known elements and in combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the results of the combination would have been predictable.
Regarding claim 22,
Rahim, Rankawat, Caldwell and Moore discloses the limitations as recited in claim 10 above. Rahim further discloses two-dimensional pixel coordinates (See Fig. 2) and three-dimensional coordinates (see Column 3 lines 33-35).
Rahim does not specifically state wherein determining the three dimensional spatial coordinates comprises converting the pixel coordinates to film coordinates using the transformation component and converting the film coordinates to the three dimensional spatial coordinates using camera intrinsic parameters of the camera.
However, Rankwat teaches:
wherein determining the three dimensional spatial coordinates comprises converting the pixel coordinates to film coordinates using the transformation component and converting the film coordinates to the three dimensional spatial coordinates using camera intrinsic parameters of the camera. (See at least Para. 0034 and Para. 0067).
It would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to have modified Rahim to incorporate the teachings of Rankawat to include the limitations as recited above in order to better determine the location of the boundary in the real-world coordinate (Para. 0067, Rankawat). Furthermore, converting between coordinates would create a more robust system for projecting coordinates on a remote operator’s screen, and to better display a visualization of a projected path of the vehicle. Additionally, a person having ordinary skill in the art would have a reasonable expectation of success in combining the teachings of Rahim and Rankawat. The claimed invention is merely a combination of known elements and in combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the results of the combination would have been predictable.
Claims 2-3, 11-12, 17-18, and 23 are rejected under 35 U.S.C. 103 as being unpatentable over Rahim, Rankawat, Caldwell and Moore further in view of Kumavat et al. US20230176573A1 (henceforth Kumavat).
Regarding claim 2,
Rahim, Rankawat, Caldwell and Moore discloses the limitations as recited in claim 1 above. Rahim does not specifically state the limitation “determining, before displaying the overlay, that the pixel coordinates correspond to a location in the video stream that is valid to generate the projected path”.
However, Kumavat teaches:
determining, before displaying the overlay, that the pixel coordinates correspond to a location in the video stream that is valid to generate the projected path
(See at least Para. 0022, “the technology confers the benefit of validating inputs from remote operators prior to implementation, which functions to maintain safety standards and implement satisfactory actions based on numerous sources of information” and Para. 0023, “In a first example, the technology further confers the benefit of validating portions of remote operator inputs (e.g., in a batched fashion), such as initial waypoints in a series of waypoints entered by the remote operator, such that in an event that the initial waypoints do not satisfy the set of safety constraints and/or other satisfaction criteria, the remote operator can be notified and/or the additional waypoints can be not received and/or not processed”. Since the remote operator inputs are validated prior to implementation, then that includes determining that the pixel coordinates corresponds to a location in the video stream that is valid to generate the projected path before displaying (i.e. prior to implementation).)
It would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to have modified Rahim to incorporate the teachings of Kumavat to include “determining, before displaying the overlay, that the pixel coordinates correspond to a location in the video stream that is valid to generate the projected path” in order to “prevent the operator from providing feedback which is not able to be performed (and/or would be unsafe to be performed) by the AV” (Para. 0024, Kumavat), which would create a more robust vehicle remote operation system. Additionally, a person having ordinary skill in the art would have a reasonable expectation of success in combining the teachings of Rahim and Kumavat. The claimed invention is merely a combination of known elements and in combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the results of the combination would have been predictable.
Regarding claim 3,
Rahim, Rankawat, Caldwell and Moore discloses the limitations as recited in claim 1 above. Rahim further discloses:
receiving a second input indicating second pixel coordinates in the video stream
(See column 3 lines 41-45, “The operator, by depressing a button, can denote any cursor position as a waypoint. He or she denotes a series of waypoints to define points of a path in the space seen in the viewer, over which the operator wants the vehicle to travel.” The operator can input a second input indicating second pixel coordinates to denote a waypoint in the video stream.)
Rahim does not specifically state the limitation “determining that the second pixel coordinates correspond to a location in the video stream that is invalid; and preventing display of a second overlay based on the location being invalid”.
However, Kumavat teaches:
determining that the second pixel coordinates correspond to a location in the video stream that is invalid; and preventing display of a second overlay based on the location being invalid
(See at least Para. 0022, “the technology confers the benefit of validating inputs from remote operators prior to implementation, which functions to maintain safety standards and implement satisfactory actions based on numerous sources of information” and Para. 0023, “In a first example, the technology further confers the benefit of validating portions of remote operator inputs (e.g., in a batched fashion), such as initial waypoints in a series of waypoints entered by the remote operator, such that in an event that the initial waypoints do not satisfy the set of safety constraints and/or other satisfaction criteria, the remote operator can be notified and/or the additional waypoints can be not received and/or not processed”. Since the remote operator inputs are validated/invalidated prior to implementation, then that includes preventing a display of any overlay based on the location of the waypoint being invalid.)
It would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to have modified Rahim to incorporate the teachings of Kumavat to include “determining that the second pixel coordinates correspond to a location in the video stream that is invalid; and preventing display of a second overlay based on the location being invalid” in order to “prevent the operator from providing feedback which is not able to be performed (and/or would be unsafe to be performed) by the AV” (Para. 0024, Kumavat), which would create a more robust vehicle remote operation system. Additionally, a person having ordinary skill in the art would have a reasonable expectation of success in combining the teachings of Rahim and Kumavat. The claimed invention is merely a combination of known elements and in combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the results of the combination would have been predictable.
Regarding claim 11,
Rahim, Rankawat, Caldwell, Moore and Kumavat discloses the same limitations as recited in claim 2 above, and is therefore rejected under the same rejection and obviousness rational.
Rahim further discloses:
a two-dimensional coordinate correspond to a location in the video stream
(Fig. 1 and 3 shows a two-dimensional screen wherein the cursor pointer corresponds to a two-dimensional coordinate location in the video stream.)
Regarding claim 12,
Rahim, Rankawat, Caldwell, Moore and Kumavat discloses the same limitations as recited in claim 2 above, and is therefore rejected under the same rejection and obviousness rational.
Rahim further discloses:
a two-dimensional coordinate in the video stream
(Fig. 1 and 3 shows a two-dimensional screen wherein the cursor pointer corresponds to a two-dimensional coordinate location in the video stream.)
Regarding claim 17,
Rahim, Rankawat, Caldwell, Moore and Kumavat discloses the same limitations as recited in claim 2 above, and is therefore rejected under the same rejection and obviousness rational.
Rahim further discloses:
a two-dimensional coordinate correspond to a location in the video stream
(Fig. 1 and 3 shows a two-dimensional screen wherein the cursor pointer corresponds to a two-dimensional coordinate location in the video stream.)
Regarding claim 18,
Rahim, Rankawat, Caldwell, Moore and Kumavat discloses the same limitations as recited in claim 2 above, and is therefore rejected under the same rejection and obviousness rational.
Rahim further discloses:
a third input indicating second pixel coordinates in the video stream
(See column 3 lines 41-45, “The operator, by depressing a button, can denote any cursor position as a waypoint. He or she denotes a series of waypoints to define points of a path in the space seen in the viewer, over which the operator wants the vehicle to travel.” The operator can input a third input indicating second pixel coordinates to denote a waypoint in the video stream.)
Regarding claim 23,
Rahim, Rankawat, Caldwell, Moore and Kumavat discloses the same limitations as recited in claim 2 above, and is therefore rejected under the same rejection and obviousness rational.
Rahim further discloses:
the coordinates are two-dimensional pixel coordinate (See at least Fig. 2).
Claims 4, 13, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Rahim, Rankawat, Caldwell and Moore further in view of Simpson US12,029,156B1.
Regarding claim 4,
Rahim, Rankawat, Caldwell and Moore discloses the limitations as recited in claim 1 above. Rahim does not specifically state “predicting a position of the vehicle in the video stream based on a latency associated with the video stream; and including an indication of the position in the overlay.” However, Simpson teaches:
predicting a position of the vehicle in the video stream based on a latency associated with the video stream; and including an indication of the position in the overlay
(See at least Column 40, lines 4-13, “The spatial information captured at time t.sub.0 may be used to predict the position of the lawn maintenance machine and other objects in the environment at a time in the future, such as time t.sub.1, to produce a predicted environment that is displayed to a user. As described herein, the difference between to and t.sub.1 may correspond to a predictive offset that is based on the communication latency and/or other processing delays between the time when the spatial information is captured and when the 3-D environment is ultimately displayed to a remote operator.“ The position of the vehicle is predicted based on a latency associated with the video stream.)
It would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to have modified Rahim to incorporate the teachings of Simpson to include “predicting a position of the vehicle in the video stream based on a latency associated with the video stream; and including an indication of the position in the overlay” such that “control of a lawn maintenance machine may be based on predicted real-world, live conditions, rather than outdated conditions” (Column 13, lines 1-3, Simpson). This would create a more robust remote control system. Additionally, a person having ordinary skill in the art would have a reasonable expectation of success in combining the teachings of Rahim and Simpson. The claimed invention is merely a combination of known elements and in combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the results of the combination would have been predictable.
Regarding claim 13,
Rahim, Rankawat, Caldwell and Moore discloses the limitations as recited in claim 10 above. Rahim further discloses:
start the projected path from a point in the overlay corresponding to a predicted position of the vehicle. (See at least Column 9, line 65 to Column 10 line 9, wherein starting the projected path from a point in the overlay corresponds to a predicted position of the vehicle.)
Rahim does not specifically state “predicting a position of the vehicle in the video stream based on a latency associated with the video stream; and including an indication of the position in the overlay.” However, Simpson teaches:
predicting a position of the vehicle in the video stream based on a latency associated with the video stream; and including an indication of the position in the overlay
(See at least Column 40, lines 4-13, “The spatial information captured at time t.sub.0 may be used to predict the position of the lawn maintenance machine and other objects in the environment at a time in the future, such as time t.sub.1, to produce a predicted environment that is displayed to a user. As described herein, the difference between to and t.sub.1 may correspond to a predictive offset that is based on the communication latency and/or other processing delays between the time when the spatial information is captured and when the 3-D environment is ultimately displayed to a remote operator.“ The position of the vehicle is predicted based on a latency associated with the video stream.)
It would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to have modified Rahim to incorporate the teachings of Simpson to include “predicting a position of the vehicle in the video stream based on a latency associated with the video stream; and including an indication of the position in the overlay” such that “control of a lawn maintenance machine may be based on predicted real-world, live conditions, rather than outdated conditions” (Column 13, lines 1-3, Simpson). This would create a more robust remote control system. Additionally, a person having ordinary skill in the art would have a reasonable expectation of success in combining the teachings of Rahim and Simpson. The claimed invention is merely a combination of known elements and in combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the results of the combination would have been predictable.
Regarding claim 19,
Rahim, Rankawat, Caldwell, Moore and Simpson discloses the same limitations as recited in claim 13 above, and are therefore rejected under the same rejection and obviousness rational.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Rahim, Rankawat, Caldwell, and Moore further in view of Mortazavi et al. US20200269877A1 (henceforth Mortazavi).
Regarding claim 5,
Rahim, Rankawat, Caldwell, and Moore discloses the limitations as recited in claim 1 above. Rahim further discloses:
starting the projected path from a point in the overlay corresponding to a predicted position of the vehicle.
(See at least Column 9, line 65 to Column 10 line 9, wherein starting the projected path from a point in the overlay corresponds to a predicted position of the vehicle.)
Rahim does not specifically state “wherein the overlay is represented by lines corresponding to a lane of travel”.However, Mortazavi teaches:
wherein the overlay is represented by lines corresponding to a lane of travel.
(See at least Fig. 3 and 4, wherein the overlay is represented by lines corresponding to a lane of travel.)
It would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to have modified Rahim to incorporate the teachings of Mortazavi to include “wherein the overlay is represented by lines corresponding to a lane of travel” in order to “ help to distinguish lanes for turning versus lanes for proceeding” (Para. 0087, Mortazavi). Furthermore, this would create a more robust remote operating overlay system. Additionally, a person having ordinary skill in the art would have a reasonable expectation of success in combining the teachings of Rahim and Mortazavi. The claimed invention is merely a combination of known elements and in combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the results of the combination would have been predictable.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Rahim, Rankawat, Caldwell, and Moore further in view of Houshmand et al. US20220194419A1 (henceforth Houshmand).
Regarding claim 7,
Rahim, Rankawat, Caldwell, and Moore discloses the limitations as recited in claim 1 above. Rahim does not specifically state “wherein the input is received through a graphical user interface (GUI) by a detecting a mouse hovering over a location in the video stream, and wherein the selection is received by detecting clicking a button of the mouse.” However, Houshmand teaches:
wherein the input is received through a graphical user interface (GUI) by a detecting a mouse hovering over a location in the video stream, and wherein the selection is received by detecting clicking a button of the mouse.
(See at least Para. 0075, “At operation 410, example process 400 may comprise receiving, based at least in part on the presentation, input identifying a proposed trajectory 412 from among the candidate trajectory, according to any of the techniques discussed herein. In examples where the teleoperator is a human user, the input may comprise input via an input device, such as a touch screen, mouse, keyboard, microphone, or the like that identifies the trajectory 412.” An input from a mouse includes a mouse clicking, and the input is received through a GUI.)
It would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to have modified Rahim to incorporate the teachings of Houshmand to include “wherein the input is received through a graphical user interface (GUI) by a detecting a mouse hovering over a location in the video stream, and wherein the selection is received by detecting a button of the mouse clicking”, since clicking via a mouse is more efficient than using another type of controller. This would create a more robust vehicle remote operation system for controlling a vehicle remotely. Additionally, a person having ordinary skill in the art would have a reasonable expectation of success in combining the teachings of Rahim and Houshmand. The claimed invention is merely a combination of known elements and in combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the results of the combination would have been predictable.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Rahim, Rankawat, Caldwell, and Moore further in view of Rosenzweig et al. US20200349844A1 (henceforth Rosenzweig).
Regarding claim 8,
Rahim, Rankawat, Caldwell, and Moore discloses the limitations as recited in claim 1 above. Rahim does not specifically state “wherein determining the spatial coordinates includes applying Ackermann steering geometry to determine a radius of the projected path.”
However, Rosenzweig teaches:
wherein determining the spatial coordinates includes applying Ackermann steering geometry to determine a radius of the projected path.
(See at least Para. 0051 “Equation 2 is used to calculate the second stopping distance. In Equation 2, t.sub.2 is the outgoing latency time (i.e., the amount of time it takes a single frame to reach the vehicle computing unit when sent from the remote operator device) and r is the reaction time of the operator. The reaction time of the operator may be a predetermined value, and may be a default value (i.e., the same for all operators) or may be a value determined based on past reactions by the operator (e.g., as an average of previous reaction times of that operator)” and Para. 0052, “The result of each of Equations 1 and 2 is a vector line representing the stopping distance in a three-dimensional (3D) environment. In an example implementation, two radii of movement of the vehicle are determined using the Ackermann model for steering geometry based on the results of Equations 1 and 2, respectively.” The spatial coordinates includes applying Ackermann steering geometry to determine a radius of the projected path.)
It would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to have modified Rahim to incorporate the teachings of Rosenzweig to include “wherein determining the spatial coordinates includes applying Ackermann steering geometry to determine a radius of the projected path” in order to calculate dynamic trajectories of the vehicle (see Para. 0052, Rosenzweig), which would create a more robust remote vehicle controlling system. Additionally, a person having ordinary skill in the art would have a reasonable expectation of success in combining the teachings of Rahim and Rosenzweig. The claimed invention is merely a combination of known elements and in combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the results of the combination would have been predictable.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Mizoguchi US20230093047A1 discloses the remote control apparatus includes a remote control value generating unit that repeatedly generates the remote control value for traveling control of each vehicle, on the basis of detection information detected by each vehicle. The traveling control unit is provided in each vehicle and executes the traveling control on the basis of the remote control value repeatedly received from the remote control apparatus. The remote control value generating unit generates the remote control value by a process that varies depending on a communication delay of vehicle information including the detection information received from each vehicle. (See abstract)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GABRIEL J LAMBERT whose telephone number is (571)272-4334. The examiner can normally be reached M-F 10:00 am- 6:00 pm MDT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Erin Piateski can be reached at (571) 270-7429. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Erin M Piateski/Supervisory Patent Examiner, Art Unit 3669
/G.J.L./
Examiner
Art Unit 3669
Prosecution Insights