DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments with respect to claims 1-3, 7-9, 11-17-18 and 20-22 have been considered but are moot because the new ground of rejection does not rely on a newly discover art of Kim applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-3, 7-8, 17-18 and 20-22 are rejected under 35 U.S.C. 103 as being unpatentable over Bhanushali et al. US 2020/0348665 further in view of Kim et al. US 2019/0379837.
In regarding to claim 1 Bhanushali teaches:
1. (Currently Amended) A remote machine manipulation system comprising: a camera to shoot an image of a remote machine that is remotely manipulated;
[0019] Vehicle system 102 may comprise any number of components that control various aspects of a vehicle such as a car, truck, boat, plane, etc. In the example of FIG. 1, vehicle system 102 includes a teleoperation system 104, electronic control units (ECUs) 112, and a plurality of sensors 114. The vehicle system 102 is located on the vehicle and is configured to control the vehicle based in part on driving instructions from the remote computer system 106.
[0021] Sensors 114 may comprise any number of devices that capture information about the vehicle and/or information about an environment external to the vehicle. Examples of sensors 114 may include, without limitation, a camera, a microphone, a Radio Detection and Ranging (radar) sensor, an ultrasonic sensor, a Light Detection and Ranging (LIDAR) sensor, a global positioning system (GPS) sensor, a steering angle sensor, and/or a motion sensor (e.g., an accelerometer and/or gyroscope). An example sensor configuration is shown in FIG. 2.
Bhanushali, 0019, 0021, 0028, emphasis added.
and a visual presentation apparatus to receive the image shot by the camera, and present an operator who manipulates the remote machine with the image by displaying the received image,
[0040] Image reconstruction logic 126 can be implemented using software instructions stored on computer readable media. Processing described with respect to the various components of the image reconstruction logic 126 can also be implemented in hardware or a combination of hardware and software. The image reconstruction logic 126 can be configured to reconstruct images from image data sent by the vehicle system 102. Reconstruction can include decoding and/or decompressing visual representations for display at the remote control system 140. In some embodiments, the image reconstruction logic 126 may receive image data in one or more video streams. The video streams can include a stream of data corresponding to an aggregated camera view. Alternatively, the video stream can include separate streams for different cameras, in which case the image reconstruction logic 126 can stitch the streams together to form an aggregated camera view, or send the streams for output on different displays of the remote control system 140.
[0082] At step 714, the vehicle system transmits the visual representation to the remote computer system for display to the remote operator. The visual representation can be transmitted over the connection for which the quality was determined in step 708. If the transmission of the visual representation follows a negative determination in step 708, then the visual representation may be transmitted uncompressed or uniformly compressed (e.g. at a compression ratio that is lower than that used for regions of interest). Upon receipt of the compressed visual representation, the remote computer system may decompress the visual representation for display.
Bhanushali, 0028, 0040, 0082 emphasis added.
wherein the visual presentation apparatus includes: a processor to execute a program; and a memory to store the program which, when executed by the program, performs:
[0018] FIG. 1 illustrates an example computing environment 100 in which one or more embodiments can be implemented. The computing environment 100 includes a vehicle system 102 and a remote computer system 106 communicatively coupled by a network 108. In certain embodiments, the vehicle system 102 and the remote computer system 106 may each comprise one or more processors and one or more non-transitory computer-readable storage media storing instructions that are executable by the one or more processors to implement the embodiments described herein.
Bhanushali, 0018, emphasis added.
a communication situation monitoring process of detecting a communication situation in the reception of the image;
[0079] At step 708, the vehicle system determines, based on the quality of the connection to the remote computer system, whether selective compression of the visual representation generated in step 706 is required. The connection can be a wireless link as shown in FIG. 1, and can be the same connection through which the information was received in step 704, or a different connection (e.g., a wireless link dedicated to video streaming). The quality of the connection can be determined, for example, based on measurements of bandwidth, latency, and/or throughput. In some embodiments, the measurements are performed by the vehicle system 102. Alternatively, the measurements can be performed by the remote computer system, which may send the measurements or a request for selective compression to the vehicle system. The vehicle system or the remote computer system may determine that selective compression is required based on the quality falling below a certain threshold, for example, a certain amount of data per second. If selective compression is required, the method proceeds to step 710. Otherwise, the method proceeds to step 714.
Bhanushali, 0077-0079, emphasis added.
and a presented image control process of determining a central field-of- view region and a peripheral field-of-view region in a display region in which the image is displayed
[0077] At step 704, the vehicle system receives, from the remote computer system, information indicating a direction or area of focus of the remote operator. As explained earlier, the direction can be a direction in which the remote operator is gazing or facing with respect to an image output on a display device, where the direction is determined based on one or more sensors located at the remote computer system. An area of focus can correspond to, for example, a specific display device among a set of display devices, a display region extrapolated from the remote operator's gaze or face direction, or a specific position on a display screen (e.g., the coordinates of the cursor 448 in FIG. 4). Thus, the information received in step 704 can be formatted in various ways, including as display coordinates, an angular value, or using an identifier of a display device (e.g., “display number 1”).
Bhanushali, 0077-0079, emphasis added.
and determining, on the basis of the communication situation detected by the communication situation monitoring process and an operating situation of the remote machine,
[0063] FIG. 4 also includes a representation of a gaze or facing direction of a remote operator relative to the images being displayed on the display devices 410A-C. As shown in the figure, an eye 440 of the remote operator can move between different display devices. For example, the remote operator might be gazing in a direction 442 toward the display device 410A during a majority of the time that the vehicle is being remotely operated. Further, the remote operator may occasionally look away from the display device 410A, for example, in a direction 444 toward display 410B in preparation for executing a lane change maneuver across the lane 406.
[0064] Assuming that selective compression has been activated (e.g., based on a measured bandwidth falling below a certain threshold), the elements displayed by the display devices 410 can be compressed in different ways. First, the vehicle system (e.g., vehicle system 102 in FIG. 1) may define various regions of interest, including a first region 450 around the boundaries 402, 404 of the trajectory, a second region 452 around the lane 406, and a third region 454 around the vehicle 430. In some embodiments, portions corresponding to the vehicle being remotely operated (e.g., the front 412) are eliminated from consideration as regions of interest. As shown in FIG. 4, the regions of interest can be defined to encompass a certain amount of image area beyond the boundaries of the relevant feature. The vehicle system may determine the degree to which the regions of interest extend beyond the feature boundaries in order to meet a margin of error for safety. Alternatively, in some embodiments, the vehicle system may define the region of interest to approximate the boundaries of the feature as closely as possible.
Bhanushali, 0063-0067, emphasis added.
at least one image parameter related to a transmission rate of a central field-of-view image displayed in the central field-of-view region and a peripheral field-of-view image displayed in the peripheral field-of-view region,
[0079] At step 708, the vehicle system determines, based on the quality of the connection to the remote computer system, whether selective compression of the visual representation generated in step 706 is required. The connection can be a wireless link as shown in FIG. 1, and can be the same connection through which the information was received in step 704, or a different connection (e.g., a wireless link dedicated to video streaming). The quality of the connection can be determined, for example, based on measurements of bandwidth, latency, and/or throughput. In some embodiments, the measurements are performed by the vehicle system 102. Alternatively, the measurements can be performed by the remote computer system, which may send the measurements or a request for selective compression to the vehicle system. The vehicle system or the remote computer system may determine that selective compression is required based on the quality falling below a certain threshold, for example, a certain amount of data per second. If selective compression is required, the method proceeds to step 710. Otherwise, the method proceeds to step 714.
Bhanushali, 0079-0081, emphasis added.
however, Bhanushali fails to explicitly teach, but Kim teaches: wherein the at least one image parameter includes a plurality of image parameters, and the presented image control process changes the plurality of image parameters based on a predetermined priority.
[0082] According to an embodiment, the electronic device 101 comprises a camera 180, a display 160, a memory 130, a communication module 190, and a processor 120 configured to identify a plurality of parameter sets related to image capturing from an external device using the communication module 190, provide, in a first preview, at least part of one or more images using the display 160 during at least part of obtaining the one or more images using the camera 180, generate one or more first corrected images to which a first parameter set among the plurality of parameter sets is applied, using the one or more images, generate one or more second corrected images to which a second parameter set among the plurality of parameter sets is applied, using the one or more images, identify priority for the plurality of parameter sets, and provide, in a second preview, one or more among the one or more first corrected images and the one or more second corrected images according to the priority during at least part of providing the first preview.
Kim, 0082, 0128, 0154
Accordingly, it would have been obvious to one ordinary skill in the art before the effective filing date to combine the teaching of Kim with the system of Bhanushanli in order wherein the at least one image parameter includes a plurality of image parameters, and the presented image control process changes the plurality of image parameters based on a predetermined priority, as such, the method of providing quality-customized images based on learned user preference..--0002.
Note: The motivation that was applied to claim 1 above, applies equally as well to claims 1-3, 7-8, 17-18, 20-21 and 22 as presented blow.
In regarding to claim 2 Bhanushali and Kim teaches:
2. (Currently Amended) The remote machine manipulation system according to claim 1, furthermore, Bhanushali teaches: comprising wherein the program when executed by the processor performs: a gaze point setting process of setting a gaze point of the operator, wherein the presented image control process determines the central field-of-view region on the basis of the gaze point set by the gaze point setting process.
Bhanushali, 0063-0067
In regarding to claim 3 Bhanushali and Kim teaches:
3. (Currently Amended) The remote machine manipulation system according to claim 2, furthermore, Bhanushali teaches: wherein the gaze point setting process device is a line-of-sight measurement process device that measures a line of sight of the operator.
Bhanushali, 0063-0067
In regarding to claim 7 Bhanushali and Kim teaches:
7. (Currently Amended) The remote machine manipulation system according to claim 1, furthermore, Bhanushali teaches: wherein the image parameter includes at least one of a resolution of the peripheral field-of-view image, a frame rate of the peripheral field-of-view image, a color of the peripheral field-of-view region, or a size of the central field-of-view region.
Bhanushali, 0027, 0066, 0072
In regarding to claim 8 Bhanushali and Kim teaches:
8. (Currently Amended) The remote machine manipulation system according to claim 1, furthermore, Bhanushali teaches: wherein the program when executed by the processor performs an automatic tracking process of controlling a shooting direction of the camera such that a specified target is included in the central field-of-view image.
Bhanushali, 0028, 0044, 0058
Claims 17-18 list all similar elements of claim 1, but in apparatus and method form rather than system form. Therefore, the supporting rationale of the rejection to claim 1 applies equally as well to claims 17-18. Furthermore, Bhanushali teaches a system, apparatus and method processing the claimed invention.
Bhanushali, Fig. 1 vehicle system 102 and remote computer system 106.
In regarding to claim 20 Bhanushali and Kim teaches:
20. (New) The remote machine manipulation system according to claim 1, furthermore, Kim teaches: wherein the presented image control process changes the plurality of image parameters in a descending order of the predetermined priority until an image transmission rate corresponding to the communication situation is achieved.
Kim, 0071, 0082, 0128, 0154
Claims 21-22 list all similar elements of claim 20, but in apparatus and method form rather than system form. Therefore, the supporting rationale of the rejection to claim 20 applies equally as well to claims 21-22.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 9, 11-15 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Bhanushali et al. US 2020/0348665 and Kim et al. US 2019/0379837 as applied to claims 1-3 above, and further in view of Takhirov et al. US 2020/0172014.
In regarding to claim 9 Bhanushali teaches:
9. (Currently Amended) The remote machine manipulation system according to claim 1, however, Bhanushali and Kim fails to explicitly teach, but Takhirov teaches: wherein the program when executed by the processor performs a zoom control process of controlling zoom-in and zoom-out of the image shot by the camera.
[0047] In step S504, controller 150 determines the focused distance of each camera 130. Parameters and settings of cameras 130 may be pre-stored in controller 150 or provided by cameras 130 along with the image data. Camera parameters and settings may include, among other things, local length, angle of view, aperture, shutter speed, white balance, metering, and filters, etc. The focal length is usually determined by the type of lens used (normal, long focus, wide angle, telephoto, macro, fisheye, or zoom). Camera parameters may also include, e.g., a distance v between the camera's lens plane (e.g., 410 in FIG. 4) and the image plane (e.g., 430 in FIG. 4). Controller 150 determines the focused distance u for each camera based on its focal length f and the distance v, e.g., according to Equation (2).
Takhirov, 0019, 0047, emphasis added
Accordingly, it would have been obvious to one ordinary skill in the art before the effective filing date to combine the teaching of Takhirov with the system of Bhanushanli and Kim in order wherein the program when executed by the processor performs a zoom control process of controlling zoom-in and zoom-out of the image shot by the camera, as such, the target event can be clearly identified.
Note: The motivation that was applied to claim 9 above, applies equally as well to claims 11-15 and 16 as presented blow.
In regarding to claim 11 Bhanushali, Kim and Takhirov teaches:
11. (Currently Amended) The remote machine manipulation system according to claim 9, furthermore, Takhirov teaches: wherein the program when executed by the processor performs a gain adjustment process of adjusting a gain of manipulation for an action of the remote machine in accordance with a magnification set by the zoom control process.
Takhirov, 0019, 0047
In regarding to claim 12 Bhanushali and Kim teaches:
12. (Currently Amended) The remote machine manipulation system according to claim 1, furthermore, Takhirov teaches: wherein the camera is a compound-eye camera, and the image parameter includes information indicating whether or not each of the central field-of-view image and the peripheral field-of-view image is a stereoscopic image.
Takhirov, 0038, 0046, 0049
In regarding to claim 13 Bhanushali, Kim and Takhirov teaches:
13. (Currently Amended) The remote machine manipulation system according to claim 9, furthermore, Takhirov teaches: wherein the camera is a compound-eye camera, and the image parameter includes information indicating whether or not each of the central field-of-view image and the peripheral field-of-view image is a stereoscopic image.
Takhirov, 0038, 0046, 0049
In regarding to claim 14 Bhanushali, Kim and Takhirov teaches:
14. (Currently Amended) The remote machine manipulation system according to claim 13, furthermore, Takhirov teaches: wherein the program when executed by the processor performs a camera drive process of controlling least one of a direction of shooting by the compound-eye camera, a distance between cameras of the compound-eye camera, or an angle of convergence of the compound-eye camera.
Takhirov, 0019, 0038, 0047-0048
In regarding to claim 15 Bhanushali, Kim and Takhirov teaches:
15. (Currently Amended) The remote machine manipulation system according to claim 13, furthermore, Takhirov teaches: wherein the zoom control process controls the camera in conjunction with zoom-in and zoom-out of the image.
Takhirov, 0019, 0047
In regarding to claim 16 Bhanushali, Kim and Takhirov teaches:
16. (Currently Amended) The remote machine manipulation system according to claim 13, furthermore, Takhirov teaches: wherein the program when executed by the processor performs an apparatus peripheral situation monitoring process of detecting whether or not an object exists in a field of view of the operator corresponding to the central field-of-view image, wherein in a case where the central field-of-view image is a stereoscopic image,
Takhirov, 0038, 0049
Furthermore, Bhanushali teaches: the presented image control process reduces a resolution of the peripheral field-of-view image when the presence of the object is detected by the apparatus peripheral situation monitoring process.
Bhanushali, 0002, 0006, 0031
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL T TEKLE whose telephone number is (571)270-1117. The examiner can normally be reached Monday-Friday 8:00-4:30 ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, William Vaughn can be reached at 571-272-3922. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL T TEKLE/Primary Examiner, Art Unit 2481
Prosecution Insights