DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-3, 7-8, 17 and 18 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Bhanushali et al. US 2020/0348665.
In regarding to claim 1 Bhanushali teaches:
1. (Currently Amended) A remote machine manipulation system comprising: a camera to shoot an image of a remote machine that is remotely manipulated;
[0019] Vehicle system 102 may comprise any number of components that control various aspects of a vehicle such as a car, truck, boat, plane, etc. In the example of FIG. 1, vehicle system 102 includes a teleoperation system 104, electronic control units (ECUs) 112, and a plurality of sensors 114. The vehicle system 102 is located on the vehicle and is configured to control the vehicle based in part on driving instructions from the remote computer system 106.
[0021] Sensors 114 may comprise any number of devices that capture information about the vehicle and/or information about an environment external to the vehicle. Examples of sensors 114 may include, without limitation, a camera, a microphone, a Radio Detection and Ranging (radar) sensor, an ultrasonic sensor, a Light Detection and Ranging (LIDAR) sensor, a global positioning system (GPS) sensor, a steering angle sensor, and/or a motion sensor (e.g., an accelerometer and/or gyroscope). An example sensor configuration is shown in FIG. 2.
Bhanushali, 0019, 0021, 0028, emphasis added.
and a visual presentation apparatus to receive the image shot by the camera, and present an operator who manipulates the remote machine with the image by displaying the received image,
[0040] Image reconstruction logic 126 can be implemented using software instructions stored on computer readable media. Processing described with respect to the various components of the image reconstruction logic 126 can also be implemented in hardware or a combination of hardware and software. The image reconstruction logic 126 can be configured to reconstruct images from image data sent by the vehicle system 102. Reconstruction can include decoding and/or decompressing visual representations for display at the remote control system 140. In some embodiments, the image reconstruction logic 126 may receive image data in one or more video streams. The video streams can include a stream of data corresponding to an aggregated camera view. Alternatively, the video stream can include separate streams for different cameras, in which case the image reconstruction logic 126 can stitch the streams together to form an aggregated camera view, or send the streams for output on different displays of the remote control system 140.
[0082] At step 714, the vehicle system transmits the visual representation to the remote computer system for display to the remote operator. The visual representation can be transmitted over the connection for which the quality was determined in step 708. If the transmission of the visual representation follows a negative determination in step 708, then the visual representation may be transmitted uncompressed or uniformly compressed (e.g. at a compression ratio that is lower than that used for regions of interest). Upon receipt of the compressed visual representation, the remote computer system may decompress the visual representation for display.
Bhanushali, 0028, 0040, 0082 emphasis added.
wherein the visual presentation apparatus includes: a processor to execute a program; and a memory to store the program which, when executed by the program, performs:
[0018] FIG. 1 illustrates an example computing environment 100 in which one or more embodiments can be implemented. The computing environment 100 includes a vehicle system 102 and a remote computer system 106 communicatively coupled by a network 108. In certain embodiments, the vehicle system 102 and the remote computer system 106 may each comprise one or more processors and one or more non-transitory computer-readable storage media storing instructions that are executable by the one or more processors to implement the embodiments described herein.
Bhanushali, 0018, emphasis added.
a communication situation monitoring process of detecting a communication situation in the reception of the image;
[0079] At step 708, the vehicle system determines, based on the quality of the connection to the remote computer system, whether selective compression of the visual representation generated in step 706 is required. The connection can be a wireless link as shown in FIG. 1, and can be the same connection through which the information was received in step 704, or a different connection (e.g., a wireless link dedicated to video streaming). The quality of the connection can be determined, for example, based on measurements of bandwidth, latency, and/or throughput. In some embodiments, the measurements are performed by the vehicle system 102. Alternatively, the measurements can be performed by the remote computer system, which may send the measurements or a request for selective compression to the vehicle system. The vehicle system or the remote computer system may determine that selective compression is required based on the quality falling below a certain threshold, for example, a certain amount of data per second. If selective compression is required, the method proceeds to step 710. Otherwise, the method proceeds to step 714.
Bhanushali, 0077-0079, emphasis added.
and a presented image control process of determining a central field-of- view region and a peripheral field-of-view region in a display region in which the image is displayed
[0077] At step 704, the vehicle system receives, from the remote computer system, information indicating a direction or area of focus of the remote operator. As explained earlier, the direction can be a direction in which the remote operator is gazing or facing with respect to an image output on a display device, where the direction is determined based on one or more sensors located at the remote computer system. An area of focus can correspond to, for example, a specific display device among a set of display devices, a display region extrapolated from the remote operator's gaze or face direction, or a specific position on a display screen (e.g., the coordinates of the cursor 448 in FIG. 4). Thus, the information received in step 704 can be formatted in various ways, including as display coordinates, an angular value, or using an identifier of a display device (e.g., “display number 1”).
Bhanushali, 0077-0079, emphasis added.
and determining, on the basis of the communication situation detected by the communication situation monitoring process and an operating situation of the remote machine,
[0063] FIG. 4 also includes a representation of a gaze or facing direction of a remote operator relative to the images being displayed on the display devices 410A-C. As shown in the figure, an eye 440 of the remote operator can move between different display devices. For example, the remote operator might be gazing in a direction 442 toward the display device 410A during a majority of the time that the vehicle is being remotely operated. Further, the remote operator may occasionally look away from the display device 410A, for example, in a direction 444 toward display 410B in preparation for executing a lane change maneuver across the lane 406.
[0064] Assuming that selective compression has been activated (e.g., based on a measured bandwidth falling below a certain threshold), the elements displayed by the display devices 410 can be compressed in different ways. First, the vehicle system (e.g., vehicle system 102 in FIG. 1) may define various regions of interest, including a first region 450 around the boundaries 402, 404 of the trajectory, a second region 452 around the lane 406, and a third region 454 around the vehicle 430. In some embodiments, portions corresponding to the vehicle being remotely operated (e.g., the front 412) are eliminated from consideration as regions of interest. As shown in FIG. 4, the regions of interest can be defined to encompass a certain amount of image area beyond the boundaries of the relevant feature. The vehicle system may determine the degree to which the regions of interest extend beyond the feature boundaries in order to meet a margin of error for safety. Alternatively, in some embodiments, the vehicle system may define the region of interest to approximate the boundaries of the feature as closely as possible.
Bhanushali, 0063-0067, emphasis added.
an image parameter related to a transmission rate of a central field-of-view image displayed in the central field-of-view region and a peripheral field-of-view image displayed in the peripheral field-of-view region.
[0079] At step 708, the vehicle system determines, based on the quality of the connection to the remote computer system, whether selective compression of the visual representation generated in step 706 is required. The connection can be a wireless link as shown in FIG. 1, and can be the same connection through which the information was received in step 704, or a different connection (e.g., a wireless link dedicated to video streaming). The quality of the connection can be determined, for example, based on measurements of bandwidth, latency, and/or throughput. In some embodiments, the measurements are performed by the vehicle system 102. Alternatively, the measurements can be performed by the remote computer system, which may send the measurements or a request for selective compression to the vehicle system. The vehicle system or the remote computer system may determine that selective compression is required based on the quality falling below a certain threshold, for example, a certain amount of data per second. If selective compression is required, the method proceeds to step 710. Otherwise, the method proceeds to step 714.
Bhanushali, 0079-0081, emphasis added.
In regarding to claim 2 Bhanushali teaches:
2. (Currently Amended) The remote machine manipulation system according to claim 1, comprising wherein the program when executed by the processor performs: a gaze point setting process of setting a gaze point of the operator, wherein the presented image control process determines the central field-of-view region on the basis of the gaze point set by the gaze point setting process.
Bhanushali, 0063-0067
In regarding to claim 3 Bhanushali teaches:
3. (Currently Amended) The remote machine manipulation system according to claim 2, wherein the gaze point setting process device is a line-of-sight measurement process device that measures a line of sight of the operator.
Bhanushali, 0063-0067
In regarding to claim 7 Bhanushali teaches:
7. (Currently Amended) The remote machine manipulation system according to claim 1, wherein the image parameter includes at least one of a resolution of the peripheral field-of-view image, a frame rate of the peripheral field-of-view image, a color of the peripheral field-of-view region, or a size of the central field-of-view region.
Bhanushali, 0027, 0066, 0072
In regarding to claim 8 Bhanushali teaches:
8. (Currently Amended) The remote machine manipulation system according to claim 1, comprising claim 1, wherein the program when executed by the processor performs an automatic tracking process of controlling a shooting direction of the camera such that a specified target is included in the central field-of-view image.
Bhanushali, 0028, 0044, 0058
Claims 17-18 list all similar elements of claim 1, but in apparatus and method form rather than system form. Therefore, the supporting rationale of the rejection to claim 1 applies equally as well to claims 17-18. Furthermore, Bhanushali teaches a system, apparatus and method processing the claimed invention.
Bhanushali, Fig. 1 vehicle system 102 and remote computer system 106.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 9, 11-15 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Bhanushali et al. US 2020/0348665 as applied to claims 1-8 above, and further in view of Takhirov et al. US 2020/0172014.
In regarding to claim 9 Bhanushali teaches:
9. (Currently Amended) The remote machine manipulation system according to claim 1, however, Bhanushali fails to explicitly teach, but Takhirov teaches: wherein the program when executed by the processor performs a zoom control process of controlling zoom-in and zoom-out of the image shot by the camera.
[0047] In step S504, controller 150 determines the focused distance of each camera 130. Parameters and settings of cameras 130 may be pre-stored in controller 150 or provided by cameras 130 along with the image data. Camera parameters and settings may include, among other things, local length, angle of view, aperture, shutter speed, white balance, metering, and filters, etc. The focal length is usually determined by the type of lens used (normal, long focus, wide angle, telephoto, macro, fisheye, or zoom). Camera parameters may also include, e.g., a distance v between the camera's lens plane (e.g., 410 in FIG. 4) and the image plane (e.g., 430 in FIG. 4). Controller 150 determines the focused distance u for each camera based on its focal length f and the distance v, e.g., according to Equation (2).
Takhirov, 0019, 0047, emphasis added
Accordingly, it would have been obvious to one ordinary skill in the art before the effective filing date to combine the teaching of Takhirov with the system of Bhanushanli in order wherein the program when executed by the processor performs a zoom control process of controlling zoom-in and zoom-out of the image shot by the camera, as such, the target event can be clearly identified.
Note: The motivation that was applied to claim 9 above, applies equally as well to claims 11-15 and 16 as presented blow.
In regarding to claim 11 Bhanushali and Takhirov teaches:
11. (Currently Amended) The remote machine manipulation system according to claim 9, furthermore, Takhirov teaches: wherein the program when executed by the processor performs a gain adjustment process of adjusting a gain of manipulation for an action of the remote machine in accordance with a magnification set by the zoom control process.
Takhirov, 0019, 0047
In regarding to claim 12 Bhanushali teaches:
12. (Currently Amended) The remote machine manipulation system according to claim 1, furthermore, Takhirov teaches: wherein the camera is a compound-eye camera, and the image parameter includes information indicating whether or not each of the central field-of-view image and the peripheral field-of-view image is a stereoscopic image.
Takhirov, 0038, 0046, 0049
In regarding to claim 13 Bhanushali and Takhirov teaches:
13. (Currently Amended) The remote machine manipulation system according to claim 9, furthermore, Takhirov teaches: wherein the camera is a compound-eye camera, and the image parameter includes information indicating whether or not each of the central field-of-view image and the peripheral field-of-view image is a stereoscopic image.
Takhirov, 0038, 0046, 0049
In regarding to claim 14 Bhanushali and Takhirov teaches:
14. (Currently Amended) The remote machine manipulation system according to claim 13, furthermore, Takhirov teaches: wherein the program when executed by the processor performs a camera drive process of controlling least one of a direction of shooting by the compound-eye camera, a distance between cameras of the compound-eye camera, or an angle of convergence of the compound-eye camera.
Takhirov, 0019, 0038, 0047-0048
In regarding to claim 15 Bhanushali and Takhirov teaches:
15. (Currently Amended) The remote machine manipulation system according to claim 13, furthermore, Takhirov teaches: wherein the zoom control process controls the camera in conjunction with zoom-in and zoom-out of the image.
Takhirov, 0019, 0047
In regarding to claim 16 Bhanushali and Takhirov teaches:
16. (Currently Amended) The remote machine manipulation system according to claim 13, furthermore, Takhirov teaches: wherein the program when executed by the processor performs an apparatus peripheral situation monitoring process of detecting whether or not an object exists in a field of view of the operator corresponding to the central field-of-view image, wherein in a case where the central field-of-view image is a stereoscopic image,
Takhirov, 0038, 0049
Furthermore, Bhanushali teaches: the presented image control process reduces a resolution of the peripheral field-of-view image when the presence of the object is detected by the apparatus peripheral situation monitoring process.
Bhanushali, 0002, 0006, 0031
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL T TEKLE whose telephone number is (571)270-1117. The examiner can normally be reached Monday-Friday 8:00-4:30 ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, William Vaughn can be reached at 571-272-3922. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL T TEKLE/Primary Examiner, Art Unit 2481
Prosecution Insights