DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This is a Final Office Action of the merits. Claims 1-8 are currently pending and are addressed below.
Response to Amendment/Arguments
The amendment filed 06/08/2026 has been entered.
Applicant' s arguments with respect to the 35 USC 103 rejection of the claims, see pages 10-14 of Remarks, have been considered but are moot in view of the new grounds of rejection provided below, in light of newly found prior art, which was necessitated based on Applicant's amendments which changed the scope of the claims.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 2, 3, 6, and 8 are rejected under 35 U.S.C. 103 as being unpatentable over Kurosawa (US 11926994 B2), and further in view of Kumakura et al. (US 20220075584 A1, hereinafter “Kumakura”).
Regarding claim 1, Kurosawa discloses a robot system (Kurosawa, see at least Fig. 14, col. 50, lines 47-56, an excavator management system SYS), comprising:
a self-propelled robot including a robotic arm having one or more joints (Kurosawa, see at least Fig. 1A, see at least Figs. 1A-B, cols. 2-3, col. 25, lines 43-56, col. 8, lines 18-30, col. 43, lines 50-61, an autonomous excavator 100 including an attachment AT having one or more joints);
a manipulating part that accepts operation by an operator to allow the operator to manipulate the self-propelled robot (Kurosawa, see at least Fig. 14, see at least cols. 51-52, “The terminal apparatus 200 may also be configured to remotely operate the excavator 100... In this case, the user may use an operation input means (e.g., a touch panel, a touch pad, a joystick, etc.) mounted on the terminal apparatus 200 or communicatively connected to the terminal apparatus 200”);
a display visible by the operator (Kurosawa, see at least Fig. 14, see at least cols. 51-52, “Specifically, the terminal apparatus 200 displays the image information captured by the imaging device 80 distributed from the management apparatus 300 or the excavator 100 on the display device, and the user may perform remote operation of the excavator 100 while viewing the image information”);
a plurality of circumference cameras that are mounted around the self-propelled robot and image a situation around the self-propelled robot (Kurosawa, see at least Figs. 1B, 5, col. 45, lines 53-67, col. 46, lines 1-19, the plurality of circumference cameras 70F/70B/70L/70R/80F/80B/80L/80R that are mounted around the excavator 100 to capture images around the excavator 100); and
processing circuitry (Kurosawa, see at least Fig. 4, col. 13, lines 20-44, display control unit D1a/controller 30), the processing circuitry being adapted to:
generate a self-propelled robot simulated image that imitates (Kurosawa, see at least Figs. 7, 8A, 8B, col. 21, lines 50-67, col. 22, lines 1-17, the excavator image 821/871 is a computer graphic simulating the excavator 100 viewed from the perspective virtual viewpoint/top viewpoint/bird’s eye view as illustrated in Figs. 8A-B, 11, 12); and
generate a synthesized image displayed on the display (Kurosawa, see at least Figs. 8A, 8B, synthesized image 820/870 displayed on the display device D1), the synthesized image including a circumference situation image captured by the plurality of circumference cameras in combination with the generated self-propelled robot simulated image (Kurosawa, see at least Figs. 6, 7, 8A, 8B, col. 16, lines 1-25, col. 21, lines 50-67, col. 22, lines 1-17, a synthesized image 820/870 is generated to display on the display D1, the synthesized image 820/870 including three-dimensional image representing the work area around the excavator 100 in combination with the excavator image 821/871 and the surrounding image 500/800/850 as illustrated in Figs. 5, 8A-B; col. 22, lines 27-37, “The three-dimensional image representing the work area around the excavator 100, including the road cone image 822, the utility pole image 823, and the fence image 824, etc., may be generated as a viewpoint conversion image, for example, by performing a known viewpoint conversion process based on the image captured by the imaging device 80”; col. 14, lines 12-28, “The surrounding image may be, for example, at least one output image (captured image) of the front camera 80F, the back camera 80, the left camera 80L, and the right camera 80R. Further, the surrounding image may be a viewpoint conversion image generated based on an output image of at least one of the front camera 80F, the back camera 80B, the left camera 80L, and the right camera 80R. The viewpoint conversion image may be, for example, a combination of a top view image viewing a relatively close area around the excavator 100 from directly above, and a horizontal image viewing a relatively far area around the excavator 100 from a horizontal direction with respect to the excavator 100”),
wherein the synthesized image displayed on the display is further converted into images of at least three kinds of viewpoints, including a bird's eye image (Kurosawa, see at least Fig. 11, col. 45, lines 65-67, col. 46, lines 1-19, bird's-eye image 1100; Fig. 8A, col. 21, 22, image 820 from a virtual viewpoint on the oblique upper side of the excavator 100), an upper viewpoint image (Kurosawa, see at least Figs. 8B, col. 22-23, the top viewpoint image of the surrounding area in combination with the top viewpoint 871 of the excavator 100 as illustrated in Fig. 8B), and a first person viewpoint image (Kurosawa, see at least Figs. 5, 8A, 8B, a first person viewpoint image 500/800/850), and
wherein each of the images of the at least three kinds of viewpoints include a separate circumference situation image in combination with a separately generated self-propelled robot simulated image (Kurosawa, see at least Figs. 8A-B, col. 22, lines 7-17, col. 23, lines 3-27, the surrounding area images including the road cone image 822, the utility pole image 823, and the fence image 824, etc., in combination with a computer graphic simulating the excavator 100 viewed from the perspective viewpoint of the excavator 100 and the first person viewpoint image 800 as illustrated in Fig. 8A, and a top view image of the surrounding area in combination with top viewpoint of the excavator 100 and the first person viewpoint image 800 as illustrated in Fig. 8B; Fig. 11, col. 45, lines 53-67, col. 46, lines 1-19, the simulated image of the excavator 100 is disposed at the center of the bird’s eye viewpoint that is generated by combining the images captured by the front camera 80F, the back camera 80B, the left camera 80L, and the right camera 80R after performing a known viewpoint conversion process), the separately generated self-propelled robot simulated images being different from each other such that the separately generated self-propelled robot simulated image in at least the bird's eye image includes an entirety of the self-propelled robot (Kurosawa, see at least Fig. 8A, col. 21, 22, image 820 includes an excavator image 821) and the separately generated self-propelled robot simulated image in the first person viewpoint image includes (Kurosawa, see at least Fig. 8B, col. 24, lines 47-62, “The setting confirmation image 880 may be displayed as a two-dimensional (planar) image, including the excavator 100 … In this case, the setting confirmation image 880 may display an image of an excavator representing the excavator 100”).
Kurosawa fails to explicitly teach a self-propelled robot simulated image that imitates every moment of a posture of the self-propelled robot; the separately generated self-propelled robot simulated image in the first person viewpoint image includes only a part of the self-propelled robot.
Kumakura teaches a self-propelled robot simulated image that imitates every moment of a posture of the self-propelled robot (Kumakura, see at least Figs. 1, 3, 5, par. [0065-0071], “The working machine model images G10 and G20 include a combination of element images reproducing positions and postures of the boom BM, the arm AR, and the bucket BK of the working machine 1 … Here, the CPU 20 of the control device 2 calculates the operating states of the moving parts of the working machine 1, that is, an angle of the boom BM with respect to the upper swing body 12, an angle of the arm AR with respect to the boom BM, and an angle of the bucket BK with respect to the arm AR, using sensors attached to the cylinders SL1, SL2, and SL3 or an angle sensor such as an IMU. The position designation receiving unit 201 applies the results of calculation of the operating states of the moving parts to the element images of the working machine model images G10, G20, and G30. Accordingly, the shapes of the working machine model images G10, G20, and G30 change in conjunction with actual operations of the moving parts of the working machine 1”); the separately generated self-propelled robot simulated image in the first person viewpoint image includes only a part of the self-propelled robot (Kumakura, see at least Fig. 5, par. [0069], the bucket front view image G3).
It would have been obvious to one of ordinary skill in the art at the time of invention to modify the system of Kurosawa to include, generate a self-propelled robot simulated image that imitates every moment of a posture of the self-propelled robot including a posture of the robotic arm and the separately generated self-propelled robot simulated image in the first person viewpoint image includes only a part of the self-propelled robot, as taught by Kumakura. This modification allows an operator to remotely monitor the state of the working machine and visually recognize the positional relationship between the teeth of the bucket and the design surface.
Regarding claim 2, the combination of Kurosawa and Kumakura teaches all the limitations of claim 1. The combination of Kurosawa and Kumakura further teaches wherein the robotic arm includes one or more motors that drive the one or more joints, respectively (Kurosawa, see at least Fig. 2, col. 3, lines 60-67, col. 4, lines 1-5, Sakuta, see at least Fig. 1A, col. 11, lines 59-67, hydraulic actuators driving the boom, bucket, and arm), and one or more rotation angle detectors that detect rotation angle(s) of the one or more motors, respectively (Kurosawa, see at least Figs. 1A, 3A-B, col. 7, lines 57-64, col. 11, lines 11-26, “a boom angle sensor S1, an arm angle sensor S2, a bucket angle sensor S3, a machine tilt sensor S4, a turning state sensor S5”; Sakuta, see at least Fig. 1A, col. 2, lines 64-67, col. 3, lines 1-7, “A boom angle sensor S1 is attached to the boom 4, and an arm angle sensor S2 is attached to the arm 5. A bucket angle sensor S3 is attached to the bucket 6. The excavation attachment may have a bucket tilt mechanism. The boom angle sensor S1, the arm angle sensor S2, and the bucket angle sensor S3 may be referred to as “orientation sensors””), and
wherein the processing circuitry generates the self-propelled robot simulated image based on at least the rotation angle(s) detected by the one or more rotation angle detectors (Kumakura, see at least Fig. 1, par. [0070-0071], the controller 2 is configured to generate the working machine model images G10 and G20 include a combination of element images reproducing positions and postures of the boom BM, the arm AR, and the bucket BK of the working machine 1 based on applying the results of calculation of operating states of the moving parts to the element images of the working machine model images G10, G20, and G30. The controller 2 is configured to calculate the operating states of the moving parts of the working machine 1, that is, an angle of the boom BM with respect to the upper swing body 12, an angle of the arm AR with respect to the boom BM, and an angle of the bucket BK with respect to the arm AR, using sensors attached to the cylinders SL1, SL2, and SL3 or an angle sensor such as an IMU).
Regarding claim 3, the combination of Kurosawa and Kumakura teaches all the limitations of claim 1. The combination of Kurosawa and Kumakura further teaches wherein, when the processing circuitry generates the synthesized image of the first person viewpoint that is looked from the self-propelled robot (Kurosawa, see at least Figs. 1A, 5, col. 14, lines 29-39, the controller 30 is configured to generate the synthesized image 500 of a first person viewpoint that is looked from the excavator 100 based on the captured images from the image device 80),
the processing circuitry generates the self-propelled robot simulated image so that the part of the self-propelled robot is an arm imitation part that imitates at least a part of a portion of the robotic arm of the self-propelled robot, that is not displayed in the circumference situation image, is connected with a part of the robotic arm displayed in the circumference situation image (Kurosawa, see at least Figs. 1A, 5, col. 14, lines 29-67, col. 15, lines 1-36, the controller 30 is configured to generate the excavator 100 simulated image 510a representing the shape of the excavator 100 with the arm imitation part as described in Fig. 5), and
the processing circuitry generates the synthesized image of the first person viewpoint so that the arm imitation part in the generated self-propelled robot simulated image is connected with the part of the robotic arm displayed in the circumference situation image (Kurosawa, see at least Figs. 1A, 5, col. 14, lines 29-67, col. 15, lines 1-36, the controller 30 is configured to generate the excavator 100 simulated image 510a representing the shape of the excavator 100 with the arm imitation part as described in Fig. 5).
Regarding claim 6, the combination of Kurosawa and Kumakura teaches all the limitations of claim 1. The combination of Kurosawa and Kumakura further teaches wherein the processing circuitry determines whether the robotic arm interferes with an object around the self-propelled robot based on the circumference situation image captured by the plurality of circumference cameras, and the posture of the self-propelled robot (Kurosawa, see at least Figs. 7, 8A-B, col. 24, lines 23-62, the controller 30 configured to determine whether the excavator 100 is likely to contact the virtual wall VW based on the image data captured by the plurality of circumference camera 70F/70B/70L/70R/80F/80B/80L/80R and the posture of the excavator 100), and when the processing circuitry determines that the robotic arm interferes with the object, the processing circuitry outputs an interference warning signal (Kurosawa, see at least col. 24, lines 63-67, col. 24, lines 63-65, “The controller 30 may output an alarm when the excavator 100 is likely to contact the virtual wall VW so as not to contact the non-existent virtual wall VW”).
Regarding claim 8, the combination of Kurosawa and Kumakura teaches all the limitations of claim 1. The combination of Kurosawa and Kumakura further teaches further comprising an interference warning informer that is disposed separately from the display and informs an interference warning according to the outputted interference warning signal (Kurosawa, col. 25, lines 1-14, “For example, when the distance between the virtual wall VW and the excavator 100 (the lower traveling body 1, the upper turning body 3, the attachment AT, or the like) falls below a predetermined threshold value, the controller 30 may output a control signal to the voice sound output device D2 to output an alarm sound. At this time, the controller 30 can identify the position of the excavator 100 in the setting coordinate system based on the positioning result of a positioning device such as the GNSS device mounted on the upper turning body 3 and determine the positional relationship with the virtual wall VW. Further, the controller 30 may output different alarm sounds of plural levels as the distance between the virtual wall VW and the excavator 100 decreases. Further, the controller 30 may output an alarm based on an information image through the display device D1”).
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Kurosawa (US 11926994 B2), in view of Kumakura et al. (US 20220075584 A1, hereinafter “Kumakura”) as applied to claim 1 above, and further in view of Hoffman et al. (US 9283674 B2, hereinafter “Hoffman”).
Regarding claim 4, the combination of Kurosawa and Kumakura teaches all the limitations of claim 1. The combination of Kurosawa and Kumakura fails to explicitly teach wherein the processing circuitry generates the synthesized image in which a scheduled moving route of the self-propelled robot is superimposed on the circumference situation image.
Hoffman teaches the controller is configured to superimpose drive lanes 118 and/or turn lanes 119 on a first-person viewpoint image 120 to indicating where the robot 200 is heading (Hoffman, see at least Figs. 2A, 4K, 4L, col. 16, lines 22-29, col. 17, lines 53-67, col. 18, lines 1-4).
It would have been obvious to one of ordinary skill in the art at the time of invention to modify the combination of Kurosawa and Kumakura to include, wherein the processing circuitry generates the synthesized image in which a scheduled moving route of the self-propelled robot is superimposed on the circumference situation image, as taught by Hoffman. This modification would allow to provide the operator to remotely control the robot with a better view of the trajectory toward which the robot is driving.
Claims 5 and 7 are rejected under 35 U.S.C. 103 as being unpatentable over Kurosawa (US 11926994 B2), in view of Kumakura et al. (US 20220075584 A1, hereinafter “Kumakura”) as applied to claims 1 and 6 above, and further in view of Sakuta et al. (US 11946223 B2, hereinafter “Sakuta”).
Regarding claim 5, the combination of Kurosawa and Kumakura teaches all the limitations of claim 1. The combination of Kurosawa and Kumakura fails to explicitly teach wherein the processing circuitry generates the synthesized image in which an arm animation indicative of a change in the posture of the robotic arm of the self-propelled robot is displayed so as to be superimposed on the circumference situation image or the self-propelled robot simulated image.
Sakuta teaches the graphic shapes 1431 and 1432 in the synthesized image 1430 that including the arm/attachment of the excavator 100, wherein the graphic shapes 1431 and 1432 are animations that move in conjunction with the actual movement of the excavator 100 and the synthesized image 1430 is superimposed on the circumference situation image 1420 (Sakuta, see at least Fig. 15, col. 29, lines 39-51, col. 28, lines 32-52).
It would have been obvious to one of ordinary skill in the art at the time of invention to modify the combination of Kurosawa and Kumakura to include, wherein the processing circuitry generates the synthesized image in which an arm animation indicative of a change in the posture of the robotic arm of the self-propelled robot is displayed so as to be superimposed on the circumference situation image or the self-propelled robot simulated image, as taught by Sakuta. This modification would allow the operator to visually identify whether the attachment is approaching an obstacle located in an outer region of the attachment.
Regarding claim 7, the combination of Kurosawa and Kumakura teaches all the limitations of claims 1 and 6. The combination of Kurosawa and Kumakura fails to explicitly teach wherein the display displays an image indicative of an interference warning according to the outputted interference warning signal.
Sakuta teaches a graphic shape 1436/1437/1438 indicates the position of a part of the excavator 100 having a possibility of contacting an object (Sakuta, see at least Fig. 16, col. 30, lines 15-56).
It would have been obvious to one of ordinary skill in the art at the time of invention to modify the combination of Kurosawa and Kumakura to include, wherein the display displays an image indicative of an interference warning according to the outputted interference warning signal, as taught by Sakuta. This modification would allow the operator to visually identify whether the attachment is approaching an obstacle located in an outer region of the attachment.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Seifert et al. (US 20210041878 A1) teaches a method and system for remotely controlling a robot using a graphical user interface.
Watanabe et al. (US 20210154826 A1) teaches a method and system for operating the virtual robot displayed on the display device based on instruction information input from the operating device, and thereafter operate the real robot in a state that the virtual robot is displayed on the display device when operation execution information to execute an operation of the real robot is input from the operating device.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TRANG DANG whose telephone number is (703)756-1049. The examiner can normally be reached Monday-Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Khoi Tran can be reached at (571)272-6919. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TRANG DANG/Examiner, Art Unit 3656 /KHOI H TRAN/Supervisory Patent Examiner, Art Unit 3656
Prosecution Insights