DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The disclosure is objected to because of the following informalities:
A “Cross Reference to Related Applications” section is missing in the specification of this application while a Foreign Priority is filed.
Appropriate correction is required.
Claim Objections
Claims 1, 7-9, and 16 objected to because of the following informalities:
In claim 1, “the captured image” should read “a captured image”
In claim 7, line 5, “the region” should read “a region”.
In claim 8, line 7, “the region” should read “a region”.
In claim 8, line 3, “;” should read “,”.
In claim 9, line 9, “the region” should read “a region”.
In claim 16, line 1, “The operation support system” should read “An operation support system”.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 6, and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Takahama et al (US 11280062 B2, hereinafter, “Takahama”) in view of Koga (JP 2019163670 A, hereinafter “Koga”). A machine-translated English version for Koga is attached.
Regarding claim 1, Takahama discloses An image generation device configured to acquire captured images from an imaging device installed on a work machine and generate images used to assist an operator in remotely controlling the work machine comprising: (col. 1 / line 60 – col. 2 / line 4, “a construction machine image generation unit which generates a construction machine image indicating a virtual viewpoint image of the construction machine viewed from the virtual viewpoint on the basis of detection data of a posture detection device which is mounted in the construction machine and detects a posture of the construction machine; a synthesis overlapping the object image and the construction machine image; and a display control unit which causes a real image captured by an image capture device mounted in the construction machine unit”; col. 2, lines 11-12, “provide a technology that an operator can comfortably operate a construction machine at a remote distance”).
a machine information acquisition section that acquires machine information representing the state of the work machine from a vehicle sensor installed on the work machine; (col. 2, lines 46-50, “The posture detection device 40 includes a swing body posture sensor 41 which detects the posture of the swing body 3 and working equipment posture sensor 42 which detects the posture of the working equipment 4”; FIG. 1: sensors “41” and “42” are installed on the work machine “1”). Note that: the posture detection device can be regarded as a machine information acquisition section for the posture information as machine information of the working equipment.
an environment information acquisition section that acquires environment information representing the state of the surroundings of the work machine from a 3D measurement device and an imaging device installed on the work machine and (col. 4, lines 3-13, “The distance detection device 20 is mounted in the excavator 1. The distance detection device 20 detects a distance Ld to an object WA in the surrounding area of the excavator 1 (see FIG. 6) Examples of the object WA include a construction target to be constructed in the work place, the ground, at least a part of the working equipment 4 located in the front direction from the swing body 3, a structure, a construction machine different from the excavator 1, and at least a part of a transport vehicle which carries a load stacked by the excavator 1”; col. 8, lines 25-32, “The distance detection device 20 detects the distance Ld to each of a plurality of portions of the surface of the object WA to detect the position Pd of each of the plurality of portions, so that a three-dimensional shape of the object WA in a distance detection coordinate system can be detected. The three-dimensional shape of the object WA is point group
data of a plurality of positions Pd of which the three-dimensional coordinates (Xd, Yd, Zd) are detected”; col. 4, lines 57-60, “The image capture device 50 is mounted in the excavator 1. The image capture device 50 captures the object WA in the surrounding area of the excavator 1, and acquires the image”). Note that: (1) the distance detection device 20 as a 3D measurement device detects the shape of the environment objects; (2) The image capture device 50 installed on the work machine is an imaging device capturing the images of the object WA; (3) the 3D shapes and images of the objects in the surroundings of the work machine can be regarded as environment information acquired by devices 20 and 50 as a machine information acquisition section.
However, Takahama fails to disclose, but in the same art of computer graphics, Koga discloses
an AR image generation section that generates an AR image indicating at least one of the machine information and the environment information by superimposing the AR image on the captured image based on the machine information and the environment information, wherein (Koga, page 13, lines 29-34, “FIG. 11 shows a scene viewed by an operator on the left side of the excavator 100. Specifically, the abnormal content notification image G8 is displayed on a part of the lens of the smart glass. By looking at the abnormality content notification image G8 as shown in FIG. 11, the operator can know that “engine water temperature / high temperature abnormality” has been reported”; page 14 / line 28 – page 15 / line 6, “the work support apparatus 200 repeats the determination in step ST12 until it is determined that the check target image is recognized … FIG. 12 shows a scene viewed by an operator on the right side of the excavator 100. Specifically, a state where the guidance image G9, the arrow image G10, and the check target image G11 are displayed on a part of the lens of the smart glass is shown. The guidance image G9 represents information related to the next check target. The arrow image G10 represents the direction in which the operator should move. The check target image G11 represents a rough position of the next check target. By looking at the guidance image G9, the operator knows that the next check target is a “dust-proof net”. Further, by looking at the arrow image G10 and the check target image G11, it is understood that the operator needs to open the left side door 3d that covers the dust-proof net and knows the direction to proceed”; FIG. 12: “G8” related to the machine information and “G9 / G10 / G11” related to the environment information (the next check object, a dust-proof net in the environment of the machine) are superimposed on the acquired right side image of the machine). Note that: (1) the abnormal content notification image G8 (water temperature – high water temperature alert notification) is an AR image indicating the machine information about high water temperature while G8 is superimposed on the acquired left side image of the machine in FIG. 11; (2) the G9 / G10 / G11 images are the AR image indicating the environment information for the operator to move to the next check target (a dust-proof net) in the environment of the machine while the G9 / G10 / G11 images are superimposed on the acquired right side image of the machine in FIG. 12; (3) both the G8 image and the G9 / G10 / G11 images are shown on the acquired right side image of the machine in FIG. 12; and (4) the work support apparatus 200 can be regarded as an AR image generation section.
the AR image generation section generates a machine information AR image, which is the AR image representing the machine information to be superimposed on the position corresponding to the part of the work machine indicated by the machine information in the captured image, or generates an environment information AR image, which is the AR image representing the environment information to be superimposed on the position corresponding to the surroundings of the work machine indicated by the environment information in the captured image. Note that: (1) the G8 AR image (high water temperature alert notification) is superimposed on the corresponding part of the machine above in FIGs. 11-12; and (2) the G9 / G10 / G11 images are the AR image indicating the environment information for the operator to move to the next check target (a dust-proof net) in the environment of the machine while the G9 / G10 / G11 images are superimposed on the acquired right side image of the machine at the positions pointing or close to the next check target (the dust-proof net) in the surroundings of the work machine in FIG.
Takahama and Koga are in the same field of endeavor, namely computer graphics. Before the effective filing date of the claimed invention, it would have been obvious to apply generating AR images corresponding to machine information and environment information and superimposing the AR images onto to the acquired images, as taught by Koga into Takahama. The motivation would have been “a work support apparatus for a construction machine that supports work related to the construction machine.” (Koga, page 1, lines 25-26). The suggestion for doing so would allow to use a process generating AR images corresponding to machine information and environment information and superimposing the AR images onto to the acquired images to support the operation of a construction machine. Therefore, it would have been obvious to combine Takahama and Koga.
Regarding claim 2, Takahama in view of Koga discloses The image generation device according to claim 1 further includes a free viewpoint image generation section that generates a free viewpoint image of the work machine from the captured image when viewed from any viewpoint; (Takahama, col. 10, lines 17-22, “The control device 90 generates an object image AG indicating a virtual viewpoint image of the object WA viewed from a virtual viewpoint of the outside of the excavator 1 on the basis of the detection data of the distance detection device 20”). Note that: (1) a virtual viewpoint image of the object WA and all objects of the work machine (the excavator 1) can be generated by the control device 90 as a free viewpoint image generation section; and (2) since the virtual view point outside of the excavator 1 is not limited in terms of location, it can be any viewpoint.
the AR image generation section generates the machine information AR image or the environment information AR image corresponding to the free viewpoint image. (Koga, page 9, lines 27-30, “The work support apparatus 200 recognizes that the area is a work target area by recognizing information 28 (such as a cover shape) related to a device around the place where the predetermined 29 object is installed without recognizing the predetermined object itself”; (Koga, page 13, lines 29-34, “FIG. 11 shows a scene viewed by an operator on the left side of the excavator 100. Specifically, the abnormal content notification image G8 is displayed on a part of the lens of the smart glass. By looking at the abnormality content notification image G8 as shown in FIG. 11, the operator can know that “engine water temperature / high temperature abnormality” has been reported”; page 14 / line 28 – page 15 / line 6, “the work support apparatus 200 repeats the determination in step ST12 until it is determined that the check target image is recognized … FIG. 12 shows a scene viewed by an operator on the right side of the excavator 100. Specifically, a state where the guidance image G9, the arrow image G10, and the check target image G11 are displayed on a part of the lens of the smart glass is shown. The guidance image G9 represents information related to the next check target. The arrow image G10 represents the direction in which the operator should move. The check target image G11 represents a rough position of the next check target. By looking at the guidance image G9, the operator knows that the next check target is a “dust-proof net”. Further, by looking at the arrow image G10 and the check target image G11, it is understood that the operator needs to open the left side door 3d that covers the dust-proof net and knows the direction to proceed”; FIG. 12: “G8” related to the machine information and “G9 / G10 / G11” related to the environment information (the next check object, a dust-proof net in the environment of the machine) are superimposed on the acquired right side image of the machine). Note that: (1) the abnormal content notification image G8 (water temperature – high water temperature alert notification) is an AR image indicating the machine information about high water temperature and generated by the AR image generation section while G8 is superimposed on the acquired left side image of the machine in FIG. 11; (2) the left side image of the machine in FIG. 11 can be regarded as the free viewpoint image while the viewpoint is on the left and outside of the machine; (3) The work support apparatus 200 can recognize the objects in the generated left side image; and (4) the generated AR image (G8 image) corresponding to the generated left side image can be shown on the generated left side image of the machine in FIG. 11.
The motivation to combine Takahama and Koga given in claim 1 is incorporated here.
Regarding claim 3, Takahama in view of Koga discloses The image generation device according to claim 2 includes a coordinate system processing section that converts coordinate systems; (Takahama, FIG. 7: “COORDINATE CONVERSION DATA MEMORY UNIT 94”).
the coordinate system processing section converts the coordinate system of the 3D measurement data measured by the 3D measurement device provided in the work machine to the coordinate system of the imaging device provided in the work machine; (Takahama, col. 8, lines 45-54, “ the point group data at the plurality of positions Pd (Xd, Yd, Zd) defined in the coordinate system of the distance detection device is convertible to the point group data at the plurality of positions Pm (Xm, Ym, Zm) defined in the coordinate system of the vehicle, convertible to the point group data at the plurality of positions Pg (Xg, Y g, Zg) defined in the global coordinate system, and convertible to the point group data at the plurality of positions Pc (Xc, Yc, Zc) defined in the coordinate system of the image capture device”). Note that: the measurement data Pd (Xd, Yd, Zd) measured by the 3D measurement device can be converted to positions Pc (Xc, Yc, Zc) defined in the coordinate system of the image capture device.
the coordinate system processing section further converts the 3D measurement data converted to the coordinate system of the imaging device to the coordinate system of the free viewpoint image generated by the free viewpoint image generation section; (Takhama, col. 1, “lines 60-66, “a construction machine image generation unit which generates a construction machine image indicating a virtual viewpoint image of the construction machine viewed from the virtual viewpoint on the basis of detection data of a posture detection device which is mounted in the construction machine and detects a posture of the construction machine”; col. 10, lines 17-22, “The control device 90 generates an object image AG indicating a virtual viewpoint image of the object WA viewed from a virtual viewpoint of the outside of the excavator 1 on the basis of the detection data of the distance detection device 20”; col. 10, lines 46-49, “The virtual viewpoint image is a virtual image which is overlooked from the virtual viewpoint with a position and a direction set in the virtual space”). Note that: (1) the free viewpoint image (a virtual viewpoint image) is generated by overlooking a scene with 3D objects from the virtual viewpoint with a position and a direction set in the space; and (2) it is obvious to one having ordinary skill in the art that: the overlooking involves converting 3D measurement data to the coordinate system of the imaging device (a pinhole camera) to the coordinate system of the image plane in 2D for the free viewpoint image as a 3D-to-3D-to-2D coordinate system conversion.
the free viewpoint image generation section generates the free viewpoint image by projecting the 3D measurement data converted to the coordinate system of the free viewpoint image onto the free viewpoint image. (Takahama, col. 10, lines 46-49, “The virtual viewpoint image is a virtual image which is overlooked from the virtual viewpoint with a position and a direction set in the virtual space”) Note that: overlooking a scene with 3D objects from the virtual viewpoint with a position and a direction set in the space to generate a free viewpoint image is equivalent to projecting the 3D measurement data converted to the coordinate system of the free viewpoint image onto the free viewpoint image.
Regarding claim 4, Takahama in view of Koga discloses In the image generation device according to claim 1:
the AR image generation section is configured to switch between the machine information, the environment information, or both the machine information and the environment information based on the machine information and the environment information, or according to the selection of the operator operating the work machine. (Koga, page 13, lines 29-34, “FIG. 11 shows a scene viewed by an operator on the left side of the excavator 100. Specifically, the abnormal content notification image G8 is displayed on a part of the lens of the smart glass. By looking at the abnormality content notification image G8 as shown in FIG. 11, the operator can know that “engine water temperature / high temperature abnormality” has been reported”; page 14 / line 28 – page 15 / line 6, “the work support apparatus 200 repeats the determination in step ST12 until it is determined that the check target image is recognized … FIG. 12 shows a scene viewed by an operator on the right side of the excavator 100. Specifically, a state where the guidance image G9, the arrow image G10, and the check target image G11 are displayed on a part of the lens of the smart glass is shown. The guidance image G9 represents information related to the next check target. The arrow image G10 represents the direction in which the operator should move. The check target image G11 represents a rough position of the next check target. By looking at the guidance image G9, the operator knows that the next check target is a “dust-proof net”. Further, by looking at the arrow image G10 and the check target image G11, it is understood that the operator needs to open the left side door 3d that covers the dust-proof net and knows the direction to proceed”; FIG. 12: “G8” related to the machine information and “G9 / G10 / G11” related to the environment information (the next check object, a dust-proof net in the environment of the machine) are superimposed on the acquired right side image of the machine). Note that: (1) the abnormal content notification image G8 (water temperature – high water temperature alert notification) is an AR image indicating the machine information about high water temperature while G8 is superimposed on the acquired left side image of the machine in FIG. 11; (2) the G9 / G10 / G11 images are the AR image indicating the environment information for the operator to move to the next check target (a dust-proof net) in the environment of the machine while the G9 / G10 / G11 images are superimposed on the acquired right side image of the machine in FIG. 12; (3) both the G8 image and the G9 / G10 / G11 images are shown on the acquired right side image of the machine in FIG. 12; and (4) the work support apparatus 200 as an AR image generation section can switch to have G8 indicating the machine information in FIG. 11, to have G9/G10/G11 indicating the environment information in FIG.12, and to have G8 and G9/G10/G11 indicating the environment information in FIG.12.
The motivation to combine Takahama and Koga given in claim 1 is incorporated here.
Regarding claim 6, Takahama in view of Koga discloses In the image generation device according to claim 2:
… or according to the selection of the operator operating the work machine. (Takahama, FIG. 1: a remote operator operating the work machine can a remote operation device “70” through a control device “90” to select the free viewpoint).
the free viewpoint image generation section is configured to change the viewpoint in the free viewpoint image based on the machine information and the environment information, (Koga, page 13, lines 29-34, “FIG. 11 shows a scene viewed by an operator on the left side of the excavator 100. Specifically, the abnormal content notification image G8 is displayed on a part of the lens of the smart glass. By looking at the abnormality content notification image G8 as shown in FIG. 11, the operator can know that “engine water temperature / high temperature abnormality” has been reported”; page 14 / line 28 – page 15 / line 6, “the work support apparatus 200 repeats the determination in step ST12 until it is determined that the check target image is recognized … FIG. 12 shows a scene viewed by an operator on the right side of the excavator 100. Specifically, a state where the guidance image G9, the arrow image G10, and the check target image G11 are displayed on a part of the lens of the smart glass is shown. The guidance image G9 represents information related to the next check target. The arrow image G10 represents the direction in which the operator should move. The check target image G11 represents a rough position of the next check target. By looking at the guidance image G9, the operator knows that the next check target is a “dust-proof net”. Further, by looking at the arrow image G10 and the check target image G11, it is understood that the operator needs to open the left side door 3d that covers the dust-proof net and knows the direction to proceed”; FIG. 12: “G8” related to the machine information and “G9 / G10 / G11” related to the environment information (the next check object, a dust-proof net in the environment of the machine) are superimposed on the acquired right side image of the machine). Note that: (1) the abnormal content notification image G8 (water temperature – high water temperature alert notification) is an AR image indicating the machine information about high water temperature, while the G9 / G10 / G11 images are the AR image indicating the environment information for the operator to move to the next check target (a dust-proof net) in the environment of the machine; and (3) the free viewpoint image generation section can change or switch the viewpoint from left side to the right side of the machine to generate FIG 11 with G8 superimposed and FIG 12 with G8/G9/G10/G11 superimposed.
The motivation to combine Takahama and Koga given in claim 1 is incorporated here.
Claim 16 reciting “The operation support system for assisting the operation of a work machine comprising: ” is corresponding to the image generation device according to claim 1 in claim 1. Therefore, claim 16 is rejected for the same rationale for claim 1.
In addition, Takahama in view of Koga discloses The operation support system for assisting the operation of a work machine comprising: (Koga, page 1, lines 12-13, “To provide a work assist apparatus for construction machine that can assist work regarding 12 a construction machine more efficiently”).
a display device for displaying the image generated by the image generation device. (Koga, page 1, lines 16-19, “a display device 26. The control device 20 is 16 configured to recognize an image of a predetermined object in an image obtained by the 17 imaging device 21 and display, in a manner of augmented reality, information on the 18 predetermined object so as to correspond to the predetermined object”).
The motivation to combine Takahama and Koga given in claim 1 is incorporated here.
Claims 5, 7-10, and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Takahama in view of Koga and Tsuji et al. (US 20220042283 A1, hereinafter, “Tsuji”).
Regarding claim 5, Takahama in view of Koga discloses In the image generation device according to claim 1:
However, Takahama in view of Koga fails to disclose, but in the same art of computer graphics, Tsuji discloses
the AR image generation section generates the AR image in which the drawing attributes of a part of the region are changed so that the part of the region is displayed more prominently than other regions based on the machine information and the environment information. (Tsuji, para. [0142], “through the processing part 301, the safety function control part 305 may highlight the detected monitoring target appearing in the monitoring image displayed on the display device 50, or may highlight the position of the detected monitoring target in the monitoring image, which corresponds to the position of the monitoring image viewed from the shovel 100. … may superimpose and display a frame that surrounds the monitoring target appearing in the monitoring image, or may superimpose and display a marker at a position in the monitoring image, which corresponds to the actual position of the detected monitoring target (see FIG. 10, FIG. 21, and FIG. 22, for example)”; para. [0148], “The safety function control part 305 may change the color, the shape, the size, the presence or absence of blinking, the blinking cycle, and the like of an image displayed on the display device 50, between the attention level alarm and the caution level alarm.”; para. [0121], “as illustrated in FIG. 7, the monitoring image MP2 including a shovel image CG and a viewpoint converted image EP, which is placed along the periphery of the shovel image CG, is displayed on the display device 50”). Note that: (1) the processing part 301can be regarded as the AR image generation section to highlight or make the part of region (the detected monitoring target) more prominent; (2) the position of the detected monitor target can be regarded as the environment information while a shovel image CG indicates the machine information for the AR image generation; and (3) the drawing attributes (color, shape, the size, the presence or absence of blinking, the blinking cycle, and the like of an image can be changed to highlight the region in the AR image displayed.
Takahama in view of Koga, and Tsuji, are in the same field of endeavor, namely computer graphics. Before the effective filing date of the claimed invention, it would have been obvious to apply highlighting a part of region in an AR image by changing the drawing attributes based on machine information and environment information, as taught by Tsuji into Takahama in view of Koga. The motivation would have been “through the processing part 301, the safety function control part 305 may highlight the detected monitoring target appearing in the monitoring image displayed on the display device 50, or may highlight the position of the detected monitoring target in the monitoring image,” (Tsuji, para. [0142]). The suggestion for doing so would allow to generate an AR image in which drawing attributes of a part of the region are changed to be more prominently based on the machine information and the environment information. Therefore, it would have been obvious to combine Takahama, Koga, and Tsuji.
Regarding claim 7, the combination of Takahama, Koga, and Tsuji discloses In the image generation device according to claim 1:
the AR image generation section acquires information on whether the work machine is performing a predetermined operation from the vehicle sensor, the 3D measurement device, or the imaging device installed on the work machine; (Takahama, col. 4, lines 43-51, “The posture of the swing body 3 includes an inclination angle of the swing body 3. The posture detection device 40 includes a swing body posture sensor 41 which detects the posture of the swing body 3 and working equipment posture sensor 42 which detects the posture of the working equipment 4. Detection data of the posture detection device 40 is output to the control device 60”; col. 9, lines 44-48, “The detection data indicating the distance Ld to the object WA detected by the distance detection device 20 is output to the control device 60 in a predetermined cycle. The input/output interface 60C of the control device 60 receives the detection data of the distance detection device 20.”). Note that: the swing posture sensor detects the posture of the swing body 3 including an inclination angle as information to determine whether the swing operation has reached the pre-determined angle;
the AR image generation section generates the AR image in which the region corresponding to the predetermined operation is emphasized. (Tsuji, para. [0142], “through the processing part 301, the safety function control part 305 may highlight the detected monitoring target appearing in the monitoring image displayed on the display device 50, or may highlight the position of the detected monitoring target in the monitoring image, which corresponds to the position of the monitoring image viewed from the shovel 100. … may superimpose and display a frame that surrounds the monitoring target appearing in the monitoring image, or may superimpose and display a marker at a position in the monitoring image, which corresponds to the actual position of the detected monitoring target (see FIG. 10, FIG. 21, and FIG. 22, for example)”). Note that: the region corresponding to the predetermined operation (e.g., predetermined swing operation reaching the declination angle can be highlighted or emphasized.
The motivation to combine Takahama, Koga, and Tsuji given in claim 5 is incorporated here.
Regarding claim 8, the combination of Takahama, Koga, and Tsuji discloses In the image generation device according to claim 2:
the AR image generation section and the free viewpoint image generation section, according to the operation performed by the work machine;
switch between the machine information AR image, the environment information AR image, and both the machine information AR image and the environment information AR image to be superimposed on the captured image, (Koga, page 13, lines 29-34, “FIG. 11 shows a scene viewed by an operator on the left side of the excavator 100. Specifically, the abnormal content notification image G8 is displayed on a part of the lens of the smart glass. By looking at the abnormality content notification image G8 as shown in FIG. 11, the operator can know that “engine water temperature / high temperature abnormality” has been reported”; page 14 / line 28 – page 15 / line 6, “the work support apparatus 200 repeats the determination in step ST12 until it is determined that the check target image is recognized … FIG. 12 shows a scene viewed by an operator on the right side of the excavator 100. Specifically, a state where the guidance image G9, the arrow image G10, and the check target image G11 are displayed on a part of the lens of the smart glass is shown. The guidance image G9 represents information related to the next check target. The arrow image G10 represents the direction in which the operator should move. The check target image G11 represents a rough position of the next check target. By looking at the guidance image G9, the operator knows that the next check target is a “dust-proof net”. Further, by looking at the arrow image G10 and the check target image G11, it is understood that the operator needs to open the left side door 3d that covers the dust-proof net and knows the direction to proceed”; FIG. 12: “G8” related to the machine information and “G9 / G10 / G11” related to the environment information (the next check object, a dust-proof net in the environment of the machine) are superimposed on the acquired right side image of the machine). Note that: (1) the abnormal content notification image G8 (water temperature – high water temperature alert notification) is an AR image indicating the machine information about high water temperature while G8 is superimposed on the acquired left side image of the machine in FIG. 11; (2) the G9 / G10 / G11 images are the AR image indicating the environment information for the operator to move to the next check target (a dust-proof net) in the environment of the machine while the G9 / G10 / G11 images are superimposed on the acquired right side image of the machine in FIG. 12; (3) both the G8 image and the G9 / G10 / G11 images are shown on the acquired right side image of the machine in FIG. 12; and (4) the work support apparatus 200 as an AR image generation section can switch to have G8 indicating the machine information in FIG. 11, to have G9/G10/G11 indicating the environment information in FIG.12, and to have G8 and G9/G10/G11 indicating the environment information in FIG.12. or change the drawing attributes of a part of the region in the AR image to generate the AR image in which the part of the region is emphasized more than other regions, (Tsuji, para. [0142], “through the processing part 301, the safety function control part 305 may highlight the detected monitoring target appearing in the monitoring image displayed on the display device 50, or may highlight the position of the detected monitoring target in the monitoring image, which corresponds to the position of the monitoring image viewed from the shovel 100. … may superimpose and display a frame that surrounds the monitoring target appearing in the monitoring image, or may superimpose and display a marker at a position in the monitoring image, which corresponds to the actual position of the detected monitoring target (see FIG. 10, FIG. 21, and FIG. 22, for example)”; para. [0148], “The safety function control part 305 may change the color, the shape, the size, the presence or absence of blinking, the blinking cycle, and the like of an image displayed on the display device 50, between the attention level alarm and the caution level alarm.”; para. [0121], “as illustrated in FIG. 7, the monitoring image MP2 including a shovel image CG and a viewpoint converted image EP, which is placed along the periphery of the shovel image CG, is displayed on the display device 50”). Note that: (1) the processing part 301can be regarded as the AR image generation section to highlight or make the part of region (the detected monitoring target) more prominent; (2) the position of the detected monitor target can be regarded as the environment information while a shovel image CG indicates the machine information for the AR image generation; and (3) the drawing attributes (color, shape, the size, the presence or absence of blinking, the blinking cycle, and the like of an image can be changed to highlight the region in the AR image displayed. or change the viewpoint by the free viewpoint image. Note that: it is obvious to one having ordinary skill in the art that: according to the free viewpoint image, a remote operator can use the input control device to change the viewpoint to examine machine operations.
The motivation to combine Takahama, Koga, and Tsuji given in claim 5 is incorporated here.
Regarding claim 9, the combination of Takahama, Koga, and Tsuji discloses In the image generation device according to claim 2:
the AR image generation section and the free viewpoint image generation section acquire information on a predetermined threshold in the machine information or the environment information; (Koga, page 13, lines 29-34, “FIG. 11 shows a scene viewed by an operator on the left side of the excavator 100. Specifically, the abnormal content notification image G8 is displayed on a part of the lens of the smart glass. By looking at the abnormality content notification image G8 as shown in FIG. 11, the operator can know that “engine water temperature / high temperature abnormality” has been reported”; page 14 / line 28 – page 15 / line 6, “the work support apparatus 200 repeats the determination in step ST12 until it is determined that the check target image is recognized … FIG. 12 shows a scene viewed by an operator on the right side of the excavator 100. Specifically, a state where the guidance image G9, the arrow image G10, and the check target image G11 are displayed on a part of the lens of the smart glass is shown. The guidance image G9 represents information related to the next check target. The arrow image G10 represents the direction in which the operator should move. The check target image G11 represents a rough position of the next check target. By looking at the guidance image G9, the operator knows that the next check target is a “dust-proof net”. Further, by looking at the arrow image G10 and the check target image G11, it is understood that the operator needs to open the left side door 3d that covers the dust-proof net and knows the direction to proceed”; FIG. 12: “G8” related to the machine information and “G9 / G10 / G11” related to the environment information (the next check object, a dust-proof net in the environment of the machine) are superimposed on the acquired right side image of the machine). Note that: (1) the abnormal content notification image G8 (water temperature – high water temperature alert notification) is an AR image indicating the machine information about high water temperature while G8 is superimposed on the acquired left side image of the machine in FIG. 11; (2) the G9 / G10 / G11 images are the AR image indicating the environment information for the operator to move to the next check target (a dust-proof net) in the environment of the machine while the G9 / G10 / G11 images are superimposed on the acquired right side image of the machine in FIG. 12; (3) since the “engine water temperature / high temperature abnormality” is reported and corresponding notification is generated as an AR image based on the machine information acquired, the machine has the water temperature sensor for the AR image generation section and the free viewpoint image generation section to acquire; and (4) it is obvious to one having skill in the art that: a) the notification is triggered by the water temperature information; and b) as G8 displayed in FIG. 11 for high temperature abnormality, there should be an predetermined or preset upper temperature threshold for comparison to generate the notification.
the AR image generation section and the free viewpoint image generation section, based on the threshold, switch between the machine information AR image, the environment information AR image, and both the machine information AR image and the environment information AR image to be superimposed on the captured image, (Koga, page 13, lines 29-34, “FIG. 11 shows a scene viewed by an operator on the left side of the excavator 100. Specifically, the abnormal content notification image G8 is displayed on a part of the lens of the smart glass. By looking at the abnormality content notification image G8 as shown in FIG. 11, the operator can know that “engine water temperature / high temperature abnormality” has been reported”; page 14 / line 28 – page 15 / line 6, “the work support apparatus 200 repeats the determination in step ST12 until it is determined that the check target image is recognized … FIG. 12 shows a scene viewed by an operator on the right side of the excavator 100. Specifically, a state where the guidance image G9, the arrow image G10, and the check target image G11 are displayed on a part of the lens of the smart glass is shown. The guidance image G9 represents information related to the next check target. The arrow image G10 represents the direction in which the operator should move. The check target image G11 represents a rough position of the next check target. By looking at the guidance image G9, the operator knows that the next check target is a “dust-proof net”. Further, by looking at the arrow image G10 and the check target image G11, it is understood that the operator needs to open the left side door 3d that covers the dust-proof net and knows the direction to proceed”; FIG. 12: “G8” related to the machine information and “G9 / G10 / G11” related to the environment information (the next check object, a dust-proof net in the environment of the machine) are superimposed on the acquired right side image of the machine). Note that: (1) the abnormal content notification image G8 (water temperature – high water temperature alert notification) is an AR image indicating the machine information about high water temperature while G8 is superimposed on the acquired left side image of the machine in FIG. 11; (2) the G9 / G10 / G11 images are the AR image indicating the environment information for the operator to move to the next check target (a dust-proof net) in the environment of the machine while the G9 / G10 / G11 images are superimposed on the acquired right side image of the machine in FIG. 12; (3) both the G8 image and the G9 / G10 / G11 images are shown on the acquired right side image of the machine in FIG. 12; (4) the work support apparatus 200 as an AR image generation section can switch to have G8 indicating the machine information in FIG. 11, to have G9/G10/G11 indicating the environment information in FIG.12, and to have G8 and G9/G10/G11 indicating the environment information in FIG.12.; and (5) the presentation of G8 is based on the predetermined temperature threshold. or change the drawing attributes of a part of the region in the AR image to generate the AR image in which the part of the region is emphasized more than other regions, (Tsuji, para. [0142], “through the processing part 301, the safety function control part 305 may highlight the detected monitoring target appearing in the monitoring image displayed on the display device 50, or may highlight the position of the detected monitoring target in the monitoring image, which corresponds to the position of the monitoring image viewed from the shovel 100. … may superimpose and display a frame that surrounds the monitoring target appearing in the monitoring image, or may superimpose and display a marker at a position in the monitoring image, which corresponds to the actual position of the detected monitoring target (see FIG. 10, FIG. 21, and FIG. 22, for example)”; para. [0148], “The safety function control part 305 may change the color, the shape, the size, the presence or absence of blinking, the blinking cycle, and the like of an image displayed on the display device 50, between the attention level alarm and the caution level alarm.”; para. [0121], “as illustrated in FIG. 7, the monitoring image MP2 including a shovel image CG and a viewpoint converted image EP, which is placed along the periphery of the shovel image CG, is displayed on the display device 50”). Note that: (1) the processing part 301can be regarded as the AR image generation section to highlight or make the part of region (the detected monitoring target) more prominent; (2) the position of the detected monitor target can be regarded as the environment information while a shovel image CG indicates the machine information for the AR image generation; and (3) the drawing attributes (color, shape, the size, the presence or absence of blinking, the blinking cycle, and the like of an image can be changed to highlight the region in the AR image displayed. or change the viewpoint by the free viewpoint image. Note that: it is obvious to one having ordinary skill in the art that: according to the free viewpoint image, a remote operator can use the input control device to change the viewpoint to examine machine operations.
The motivation to combine Takahama, Koga, and Tsuji given in claim 5 is incorporated here.
Regarding claim 10, the combination of Takahama, Koga, and Tsuji discloses In the image generation device according to claim 9:
the AR image generation section and the free viewpoint image generation section acquire information on (Koga, page 13, lines 29-34, “FIG. 11 shows a scene viewed by an operator on the left side of the excavator 100. Specifically, the abnormal content notification image G8 is displayed on a part of the lens of the smart glass. By looking at the abnormality content notification image G8 as shown in FIG. 11, the operator can know that “engine water temperature / high temperature abnormality” has been reported”; page 14 / line 28 – page 15 / line 6, “the work support apparatus 200 repeats the determination in step ST12 until it is determined that the check target image is recognized … FIG. 12 shows a scene viewed by an operator on the right side of the excavator 100. Specifically, a state where the guidance image G9, the arrow image G10, and the check target image G11 are displayed on a part of the lens of the smart glass is shown. The guidance image G9 represents information related to the next check target. The arrow image G10 represents the direction in which the operator should move. The check target image G11 represents a rough position of the next check target. By looking at the guidance image G9, the operator knows that the next check target is a “dust-proof net”. Further, by looking at the arrow image G10 and the check target image G11, it is understood that the operator needs to open the left side door 3d that covers the dust-proof net and knows the direction to proceed”; FIG. 12: “G8” related to the machine information and “G9 / G10 / G11” related to the environment information (the next check object, a dust-proof net in the environment of the machine) are superimposed on the acquired right side image of the machine). Note that: (1) the abnormal content notification image G8 (water temperature – high water temperature alert notification) is an AR image indicating the machine information about high water temperature while G8 is superimposed on the acquired left side image of the machine in FIG. 11; (2) the G9 / G10 / G11 images are the AR image indicating the environment information for the operator to move to the next check target (a dust-proof net) in the environment of the machine while the G9 / G10 / G11 images are superimposed on the acquired right side image of the machine in FIG. 12; (3) both the G8 image and the G9 / G10 / G11 images are shown on the acquired right side image of the machine in FIG. 12; (4) the work support apparatus 200 as an AR image generation section can switch to have G8 indicating the machine information in FIG. 11, to have G9/G10/G11 indicating the environment information in FIG.12, and to have G8 and G9/G10/G11 indicating the environment information in FIG.1; and (5) the presentation of G8 is based on the predetermined temperature threshold. The similar AR image as a notification for high pressure abnormality can be generated to substitute G8 in FIGs. 11-12.
the free viewpoint image generation section switches the viewpoint of the free viewpoint image (Koga, page 13, lines 29-34, “FIG. 11 shows a scene viewed by an operator on the left side of the excavator 100. Specifically, the abnormal content notification image G8 is displayed on a part of the lens of the smart glass. By looking at the abnormality content notification image G8 as shown in FIG. 11, the operator can know that “engine water temperature / high temperature abnormality” has been reported”; page 14 / line 28 – page 15 / line 6, “the work support apparatus 200 repeats the determination in step ST12 until it is determined that the check target image is recognized … FIG. 12 shows a scene viewed by an operator on the right side of the excavator 100. Specifically, a state where the guidance image G9, the arrow image G10, and the check target image G11 are displayed on a part of the lens of the smart glass is shown. The guidance image G9 represents information related to the next check target. The arrow image G10 represents the direction in which the operator should move. The check target image G11 represents a rough position of the next check target. By looking at the guidance image G9, the operator knows that the next check target is a “dust-proof net”. Further, by looking at the arrow image G10 and the check target image G11, it is understood that the operator needs to open the left side door 3d that covers the dust-proof net and knows the direction to proceed”; FIG. 12: “G8” related to the machine information and “G9 / G10 / G11” related to the environment information (the next check object, a dust-proof net in the environment of the machine) are superimposed on the acquired right side image of the machine). Note that: (1) the abnormal content notification image G8 (water temperature – high water temperature alert notification) is an AR image indicating the machine information about high water temperature while G8 is superimposed on the acquired left side image of the machine in FIG. 11; (2) the G9 / G10 / G11 images are the AR image indicating the environment information for the operator to move to the next check target (a dust-proof net) in the environment of the machine while the G9 / G10 / G11 images are superimposed on the acquired right side image of the machine in FIG. 12; (3) both the G8 image and the G9 / G10 / G11 images are shown on the acquired right side image of the machine in FIG. 12; and (4) the work support apparatus 200 as an AR image generation section can switch from the left side to the right side of the machine by changing the free viewpoint to have G8 indicating the machine information in FIG. 11, to have G9/G10/G11 indicating the environment information in FIG.12, and to have G8 and G9/G10/G11 indicating the environment information in FIG.12.
the pressure of the cylinder (Tsuji, para. [0050], “The control valve 17 is of, for example, a hydraulically driven type (a hydraulic pilot type), and pilot pressures, commensurate with the operation details of the hydraulic actuators and commensurate with an operation command corresponding to the automatic operation function, are input into the control valve 17”; para. [0094], “The operation information obtaining sensor is, for example, a pressure sensor that senses a pilot pressure on the secondary side of the operating device 26 of a hydraulic pilot type”). Note that: the hydraulic pressure sensor for cylinders for hydraulic actuators can substitute the water temperature sensors for the engine as the machine information.
… to include at least the cylinder that exceeds the threshold when the pressure exceeds the threshold; (Tsuji, para. [0048], “The hydraulic actuator includes a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9”). Note that: (1) the machine has a few hydraulic cylinders; and (2) similar to the predetermined temperature threshold for water, one can have the predetermined threshold for cylinder pressure to monitor the health status of cylinders with the pressure sensors, especially determining whether the pressure exceeds the threshold.
along with this, the AR image generation section generates the AR image in which the drawing attributes of the region of the cylinder where the pressure exceeds the threshold are changed to be displayed more prominently than other regions. (Tsuji, para. [0142], “through the processing part 301, the safety function control part 305 may highlight the detected monitoring target appearing in the monitoring image displayed on the display device 50, or may highlight the position of the detected monitoring target in the monitoring image, which corresponds to the position of the monitoring image viewed from the shovel 100. … may superimpose and display a frame that surrounds the monitoring target appearing in the monitoring image, or may superimpose and display a marker at a position in the monitoring image, which corresponds to the actual position of the detected monitoring target (see FIG. 10, FIG. 21, and FIG. 22, for example)”; para. [0148], “The safety function control part 305 may change the color, the shape, the size, the presence or absence of blinking, the blinking cycle, and the like of an image displayed on the display device 50, between the attention level alarm and the caution level alarm.”; para. [0121], “as illustrated in FIG. 7, the monitoring image MP2 including a shovel image CG and a viewpoint converted image EP, which is placed along the periphery of the shovel image CG, is displayed on the display device 50”). Note that: (1) the processing part 301can be regarded as the AR image generation section to highlight or make the part of region (the detected monitoring target) more prominent, where the region can be ; (2) the position of the detected monitor target can be regarded as the environment information while a shovel image CG indicates the machine information for the AR image generation; and (3) the drawing attributes (color, shape, the size, the presence or absence of blinking, the blinking cycle, and the like of an image can be changed to highlight the region in the AR image displayed. Here the highlighted region is the region of the cylinder where the pressure exceeds the threshold.
The motivation to combine Takahama, Koga, and Tsuji given in claim 5 is incorporated here.
Regarding claim 14, the combination of Takahama, Koga, and Tsuji discloses In the image generation device according to claim 9:
the AR image generation section and the free viewpoint image generation section acquire information on the movement range when the work machine turns from the vehicle sensor as the machine information; (Takahama, col. 6, lines 28-31, “The swing body posture sensor 41 detects the inclination angle of the swing body 3 with respect to an Xg-Yg plane (horizontal plane) defined in the global coordinate system”; col. 11, lines 55-67, “the construction machine image generation unit 92 generates the construction machine image BG defined in the coordinate system of the vehicle in a predetermined cycle. In addition, for example, when the swing body 3 is swung and the yaw angle 83 is changed, the construction machine image generation unit 92 generates the construction machine image BG in a predetermined cycle in conjunction with the swing of the swing body 3. Similarly, when at least one of the roll angle 81 and the pitch angle 82 is changed, the construction machine image generation unit 92 generates the construction machine image BG in a 65 predetermined cycle in conjunction with the change of at least one of the roll angle 81 and the pitch angle 82”). Note that: (1) the AR image generation section and the free viewpoint image generation section acquire information using the posture sensors; and (2) the tuning angles as the machine information with the swing formulate the machine movement range.
the AR image generation section and the free viewpoint image generation section acquire information on the position of objects existing around the work machine from the 3D measurement device or the imaging device as the environment information; (Takahama, col. 8, lines 25-32, “The distance detection device 20 detects the distance Ld to each of a plurality of portions of the surface of the object WA to detect the position Pd of each of the plurality of portions, so that a three-dimensional shape of the object WA in a distance detection coordinate system can be detected. The three-dimensional shape of the object WA is point group data of a plurality of positions Pd of which the three-dimensional coordinates (Xd, Yd, Zd) are detected”; col. 4, lines 57-60, “The image capture device 50 is mounted in the excavator 1. The image capture device 50 captures the object WA in the surrounding area of the excavator 1, and acquires the image”).
the AR image generation section or the free viewpoint image generation section calculates the risk of collision between the work machine and the object when the work machine turns based on the movement range and the position of the object; (Tsuji, para. [0314], “the safety function control part 305 may determine whether to provide a notification to the surroundings of the shovel 100 in accordance with the degree of risk when the predetermined object is detected by the detecting part 304. The degree of risk is determined based on the positional relationship between the shovel 100 and the predetermined object”). Note that: (1) the safety function control part 305 can be a part of the AR image generation section or the free viewpoint image generation section; (2) the risk can be calculated or determined based on the positional relationship (if there is a collision) between the machine and the predetermined object; and (3) it is well known that the machine has a turning angle range and radius range so that the collision between the machine and the predetermined object can be checked.
the free viewpoint image generation section switches the viewpoint of the free viewpoint image to include at least the movement range when the risk exceeds the threshold Note that: (1) similar to the predetermined temperature threshold for water, one can have the predetermined threshold for the calculated risk to monitor the risk for collision between the machine and the objects, especially when the risk exceeds the threshold; and (2) the free viewpoint can be changed switched to a free viewpoint to cover the whole movement range.
along with this, the AR image generation section generates the AR image in which the drawing attributes of the region of the part where the work machine and the object may collide in the movement range are changed to be displayed more prominently than other parts. (Tsuji, para. [0142], “through the processing part 301, the safety function control part 305 may highlight the detected monitoring target appearing in the monitoring image displayed on the display device 50, or may highlight the position of the detected monitoring target in the monitoring image, which corresponds to the position of the monitoring image viewed from the shovel 100. … may superimpose and display a frame that surrounds the monitoring target appearing in the monitoring image, or may superimpose and display a marker at a position in the monitoring image, which corresponds to the actual position of the detected monitoring target (see FIG. 10, FIG. 21, and FIG. 22, for example)”; para. [0148], “The safety function control part 305 may change the color, the shape, the size, the presence or absence of blinking, the blinking cycle, and the like of an image displayed on the display device 50, between the attention level alarm and the caution level alarm.”; para. [0121], “as illustrated in FIG. 7, the monitoring image MP2 including a shovel image CG and a viewpoint converted image EP, which is placed along the periphery of the shovel image CG, is displayed on the display device 50”). Note that: (1) the processing part 301can be regarded as the AR image generation section to highlight or make the part of region (the detected monitoring target) more prominent, where the region can be ; (2) the position of the detected monitor target can be regarded as the environment information while a shovel image CG indicates the machine information for the AR image generation; and (3) the drawing attributes (color, shape, the size, the presence or absence of blinking, the blinking cycle, and the like of an image can be changed to highlight the region in the AR image displayed. Here the highlighted region is region of the part where the work machine and the object may collide in the movement range.
The motivation to combine Takahama, Koga, and Tsuji given in claim 5 is incorporated here.
Allowable Subject Matter
Claims 11-13 and 15 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding dependent claim 11, in the context of claim as a whole, the prior art either alone or in combination does not teach or suggest the additional elements of: “the AR image generation section and the free viewpoint image generation section acquire information on the degree of unevenness of the arrival location where the loading vehicle that loads the earth and sand excavated by the work machine is scheduled to arrive from the 3D measurement device or the imaging device as the environment information;”.
Regarding dependent claim 12, in the context of claim as a whole, the prior art either alone or in combination does not teach or suggest the additional elements of: “the AR image generation section and the free viewpoint image generation section acquire information on the expected weight of the earth and sand on the loading vehicle when the earth and sand excavated by the work machine is transferred to the loading vehicle from the 3D measurement device or the imaging device as the environment information;”.
Regarding dependent claim 13, in the context of claim as a whole, the prior art either alone or in combination does not teach or suggest the additional elements of: “the AR image generation section and the free viewpoint image generation section acquire information on the distance between the foremost part in the traveling direction of the work machine and the slope existing in the traveling direction of the work machine from the 3D measurement device or the imaging device as the environment information; the AR image generation section or the free viewpoint image generation section calculates the risk of the work machine slipping off the slope based on the distance;”.
Regarding dependent claim 15, in the context of claim as a whole, the prior art either alone or in combination does not teach or suggest the additional elements of: “the AR image generation section and the free viewpoint image generation section acquire information on the distance between the front of the work machine and the descending surface existing in front of the work machine from the 3D measurement device or the imaging device as the environment information;”.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Ding et al. (WO 2022070720 A1) teaches “In the present invention, a captured image acquisition unit acquires, from a camera provided to a work machine, a captured image in which the work machine is shown”. A machine-translated English version for Ding is attached.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BIAO CHEN whose telephone number is (703)756-1199. The examiner can normally be reached M-F 8am-5pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee M Tung can be reached at (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Biao Chen/
Patent Examiner, Art Unit 2611
/KEE M TUNG/Supervisory Patent Examiner, Art Unit 2611
Prosecution Insights