Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hama (US 20230177732 A1), and further in view of Adsumilli (US 11611699 B2).
Regarding claim 1, Hama teaches an image generation device configured to generate an image for assisting teleoperation of a work machine equipped with a work attachment (par. 0045: “In the remote operation device 20, in the case where the environment image data is received through the remote wireless communication device 224 (FIG. 4/C21), the environment image according to the environment image data is outputted to the remote image output device 221 by the remote controller 200 (FIG. 4/STEP214).”), the image generation device comprising:
a memory (par. 0034: “Each of components of the actual machine controller 400 and the image processing device 30 is configured by an arithmetic processing unit (a single-core processor or a multicore processor or a processor core configuring it), reads required data and software from a storage device such as a memory, and executes arithmetic processing according to the software for the data.”; par. 0044: “In the case where the image processing device 30 is configured by the remote operation support server 10, the environment image data may be generated by the image processing of the captured image data being performed by the image processing device 30.”); and
a processor coupled to the memory and configured to generate a composite image (par. 0034, as above) that includes
a main image included in an image captured at a work site where the work machine works by an imaging device (par. 0043: “In the working machine 40, in the case where the environment confirmation request is received through the actual machine wireless communication device 422 (FIG. 4/C40), the actual machine controller 400 acquires a captured image through the actual machine image capturing device 412 (FIG. 4/STEP410).”).
Hama fails to teach a complementary image that complements an outside of an imaging range of the main image, the complementary image not being included in the image captured at the work site.
Adsumilli teaches a complementary image that complements an outside of an imaging range of the main image, the complementary image not being included in the image captured at the work site (col. 3, lines 48-56: “A portion of the field of view 108A of the left camera 105A and a portion of the field of view 108B of the right camera 105B represent a common field of view, as illustrated by the shaded portion of the shared view 115. Within the common field of view are image features 116A and 118A. The images 120A and 122A can be stitched together using an overlap region 124A common to both images, forming stitched image 126A representative of the entire field of view 115.”).
It would have been obvious to one familiar in the art prior to the effective filing date of the claimed invention to utilize the image stitching of Adsumilli in the remote operation device of Hama, as both are in the same field of endeavor of image processing. Doing so would allow for a greater field of view while operating a remote device, which is clearly beneficial to Hama’s invention.
Regarding claim 2, Hama and Adsumilli teach the image generation device according to claim 1. Hama further teaches wherein
the complementary image is an image that complements areas that are leftward and rightward of the imaging range of the main image (par. 0031: “The image output device 221 is configured by a center image output device 2210, a left side image output device 2211 and a right side image output device 2212 each provided with a roughly rectangular screen and arranged respectively at the front, oblique left front and oblique right front of the seat St, as illustrated in FIG. 2 for example.”).
Regarding claim 3, Hama and Adsumilli teach the image generation device according to claim 1. Adsumilli further teaches wherein
the image generation device includes a trained model trained with images as training data (col. 7, lines 24-33: “The image sensor 210 is a hardware component is configured to capture image data based on light incident upon the image sensor at the time of capture. The captured image data can be stored in the memory 220 without further processing (as “raw” image data), or can undergo one or more image processing operations by the image processor 215. The image processor 215 is a hardware chip configured to perform image processing operations on captured image data and store the processed image data in the memory 220.”), the images each including
an image of an outside of the imaging range of the main image included in the image captured at the work site by the imaging device (col. 4, lines 53-58: “A first example of a image stitching algorithm can identity portions of each of two or more images representative of an overlap region between the two or more images, can align the identified portions of the images, and can average or feather the image data (such as the pixel color data) of the identified portions of the images to produce a stitched image.”), and
the main image (col. 4, lines 53-58, as above), and
the processor is further configured to generate the composite image in response to an input of by inputting the main image to the trained model (col. 9, lines 23-27: “In other embodiments, the feature detection module 250 only detects features in images to be stitched together, for instance in response to a request to stitch images together from the stitch engine 245 or from a user of the image server 205.”).
Regarding claim 4, Hama and Adsumilli teach the image generation device according to claim 1. Hama further teaches wherein the processor is further configured to generate the complementary image that includes an image of an object based on information regarding the object and position and posture information of the work machine that works at the work site, the object being recognized in the image captured at the work site by the imaging device (par. 0072: “According to the image processing device of the configuration, the continuity of the image outputted to the image output device of the remote operation device appropriately from a viewpoint of smoothness of the operation of the operator is improved, in consideration of a high probability of concentration of operator's attention on the image area at least partially demarcated by the window frame demarcating the crew cabin, in the captured image outputted to the image output device configuring the remote operation device.”).
Regarding claim 5, Hama and Adsumilli teach the image generation device according to claim 1. Hama further teaches wherein the processor is further configured to generate the complementary image that includes an image of the work machine is generated based on a shape, dimensions, and a posture of the work machine that works at the work site and a positional relationship between the work machine and the imaging device (par. 0068: “According to the image processing device of the configuration, by considering the first delay time until the remote operation mode of the working machine in the remote operation device is reflected in the captured image indicating the situation of the working machine and the environment and/or the second delay time until the moving mode of the working machine is reflected in the captured image indicating the situation of the working machine and the environment thereof, whether or not the data amount when the difference between the input image and the prediction image is encoded is to exceed the threshold is predicted.”).
Regarding claim 6, Hama and Adsumilli teach the image generation device according to claim 1. Hama further teaches wherein a difference between a feature of the main image and a feature of the complementary image is equal to or less than a predetermined threshold (par. 0052: “For the captured image of one frame as an input image, the captured image of another frame after second delay time τ2 as a prediction image is predicted based on the moving mode of the working machine 40, and whether or not the data amount when a difference between the input image and the prediction image is encoded is to exceed the threshold is determined.”).
Regarding claim 7, Hama teaches a teleoperation assistance system for a work machine, the teleoperation assistance system comprising:
a work machine equipped with a work attachment (par. 0022: “The remote operation support server 10, the remote operation device 20 and the working machine 40 are configured such that network communication with each other is possible.”); and
a teleoperation assistance device configured to assist a remote operation of the work machine (par. 0022, as above), wherein
the teleoperation assistance device includes
a display device configured to display the composite image generated at a remote location away from the work site by the image generation device (par. 0027: “The remote output interface 220 comprises a remote image output device 221 and a remote wireless communication device 224.”); and
a teleoperation device configured to operate, from the remote location, the work machine that works at the work site (par. 0027: “The remote input interface 210 comprises a remote operation mechanism 211.”).
Hama fails to teach wherein the teleoperation assistance device includes the image generation device of claim 1.
Hama and Adsumilli teach wherein the teleoperation assistance device includes the image generation device of claim 1 (as above in claim 1 rejection).
Regarding claim 8, Hama and Adsumilli teach the teleoperation assistance system for the work machine according to claim 7. Hama further teaches wherein
the work machine includes a transmitter configured to transmit the main image from the work site by wireless communication (par. 0043: “By the actual machine controller 400, captured image data to which the image processing is performed is transmitted to the remote operation support server 10 through the actual machine wireless communication device 422 (FIG. 4/STEP412).”), and
the teleoperation assistance device further includes a receiver configured to
receive at the remote location the main image transmitted from the transmitter device (par. 0045: “In the remote operation device 20, in the case where the environment image data is received through the remote wireless communication device 224 (FIG. 4/C21), the environment image according to the environment image data is outputted to the remote image output device 221 by the remote controller 200 (FIG. 4/STEP214).”), and
output the main image to the image generation device arranged at the remote location (par. 0045, as above).
Regarding claim 9, Hama and Adsumilli teach the teleoperation assistance system for the work machine according to claim 8. Hama further teaches wherein the transmitter is further configured to transmit only the main image (par. 0043: “By the actual machine controller 400, captured image data to which the image processing is performed is transmitted to the remote operation support server 10 through the actual machine wireless communication device 422 (FIG. 4/STEP412).”).
Regarding claim 10, Hama and Adsumilli teach the teleoperation assistance system for the work machine according to claim 8. Hama further teaches wherein the work machine further includes a processor configured to extract the main image from the captured image (par. 0064: “As illustrated in FIG. 11, in the case where a specified target object obj2 such as a truck or a worker is extracted on the left side of the left window frame Q3 or the like in the captured image, the image area on the inner side of a boundary line C2 surrounding the specified target object obj2 may be specified as a first specified image area A11.”), and the transmitter is further configured to transmit only the extracted main image (par. 0043, as above in claim 9 rejection).
Regarding claim 11, Hama and Adsumilli teach the image generation device according to claim 1. Hama further teaches wherein the complementary image is an extrapolation image generated by the processor (par. 0044: “The environment image data is image data indicating a simulated environment image generated based on the captured image other than the captured image data itself. In the case where the image processing device 30 is configured by the remote operation support server 10, the environment image data may be generated by the image processing of the captured image data being performed by the image processing device 30.”).
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-8 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RYAN A BARHAM whose telephone number is (571)272-4338. The examiner can normally be reached Mon-Fri, 8:30am-5pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu, can be reached at (571) 272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RYAN ALLEN BARHAM/Examiner, Art Unit 2613
/XIAO M WU/Supervisory Patent Examiner, Art Unit 2613
Prosecution Insights