DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of papers submitted under 35 U.S.C. 119(a)-(d), which papers have been placed of record in the file.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 03/18/2026 is in compliance with the provisions on 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Amendments
Acknowledgment of receiving amendments to the claims, which were received by the Office on 07/09/2026.
Response to Arguments
Applicant’s arguments with respect to claims 1-8 and 11-14 have been considered but are moot because the arguments do not apply to the same combination of references being used in the current rejection. Applicant’s arguments are directed solely to the claimed invention as amended 07/09/2026, which has been rejected under new ground of rejection necessitated by amendment. See rejection below for full detail.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-5, 14 and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kennedy et al. (US 2020/0071912 A1) in view of Fritz et al. (US 2016/0060825 A1) in view of Kirk (US 2014/0218291 A1).
Regarding claim 1, Kennedy et al. (hereafter referred as Kennedy) teaches a display device for a work machine (Kennedy, Fig. 3A), comprising:
a position senor configured to acquire position information of the work machine (Kennedy, Fig. 3A, position sensor 146, proximity sensor 148 and Location logic 156, Paragraph 0045-0047 and 0052); and
a processor (Kennedy, Fig. 3A, Machine control system 112, Paragraph 0050) configured to:
acquire peripheral information indicating an existing structure, terrain, or a boundary of a zone (Kennedy, Fig. 1, Paragraphs 0022-0025) around a position indicated by the position information of the work machine (Kennedy, Fig. 3A, Communication logic 172 and datastore 168, Fig. 4, Steps 430-435, Paragraph 0057 and 0090-0091), the peripheral information being provided based on the position information and including a coordinate value defining the existing structure, the terrain, or the boundary of the zone (Kennedy, Paragraphs 0076, 0080 and 0082, GPS coordinates or local coordinates of the objects/terrain in the worksite are peripheral information including coordinate values.);
generate an augmented image in which the existing structure, the terrain, or the boundary of the zone is viewed from a predetermined viewpoint of the work machine based on the coordinate value included in the peripheral information (Kennedy, Fig. 3A, visualization alert system 164, Fig. 4, Blocks 450-460, Paragraph 0095-0098);
display the augmented image of the existing structure, the terrain, or the boundary of the zone while superimposing the augmented image on a field of view from the predetermined viewpoint of the work machine or on an image of the field of view (Kennedy, Fig. 3A, interface generator logic 182, Fig. 4, Blocks 470-480, Paragraphs 0061 and 0099-0101).
However, Kennedy does not teach the peripheral information including a plurality of coordinate values defining an outer periphery of the existing structure, the terrain, or the boundary of the zone; generate the augmented image based on the plurality of coordinate values included in the peripheral information; generate, from augmented image data of the generated augmented image, a contour shape pattern of the augmented image data, the augmented image being generated based on the plurality of coordinate values included in the peripheral information; search real image data of the field of view for a pattern that is the same as or approximate to the contour shape pattern; calculate a distance difference between the pattern detected in the real image data by the search and the contour shape pattern, determine whether the distance difference exceeds a threshold set in advance; and in response to determining that the distance difference exceeds the threshold, correct a position of the entire contour shape pattern to a position of the pattern detected in the real image data by the search.
In reference to Fritz et al. (hereafter referred as Fritz), Fritz teaches acquire peripheral information (Fritz, Paragraph 0053, Project data is peripheral information.) indicating an existing structure, terrain, or a boundary of a zone around a position indicated by position information of the work machine (Fritz, Paragraph 0060), the peripheral information being provided based on the position information including a plurality of coordinate values defining an outer periphery of the existing structure, the terrain, or the boundary of the zone (Fritz, Fig. 3, Paragraphs 0053 and 0061, reference points P11-P12, P21-P23 and P31-P34 define outer peripheries of structures O.);
generate an augmented image in which the existing structure, the terrain, or the boundary of the zone is viewed from a predetermined viewpoint of the work machine based on the plurality of coordinate values included in the peripheral information (Fritz, Figs. 3-4, Paragraphs 0061-0063);
generate, from augmented image data of the generated augmented image (Fritz, Paragraphs 0053, Project data may be considered to be augmented image data.), a contour shape pattern of the augmented image data (Fritz, Figs. 3-4, contours 16, 17 and 18), the augmented image being generated based on the plurality of coordinate values included in the peripheral information (Fritz, Figs. 3-4, Paragraphs 0053 and 0061-0063);
search real image data of the field of view for a pattern that is the same as or approximate to the contour shape pattern (Fritz, Paragraph 0063);
compare the pattern detected in the real image data by the search and the contour shape pattern (Fritz, Paragraph 0064).
These arts are analogous since they are both related to augmented reality display. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to modify the invention of Kennedy with the peripheral information including a plurality of coordinate values as seen in Fritz to define the outer periphery of objects in the environment.to avoid collisions between objects.
However, the combination of Kennedy and Fritz does not teach calculate a distance difference between the pattern detected in the real image data by the search and the contour shape pattern, determine whether the distance difference exceeds a threshold set in advance; and in response to determining that the distance difference exceeds the threshold, correct a position of the entire contour shape pattern to a position of the pattern detected in the real image data by the search.
In reference to Kirk, Kirk teaches teach calculate a distance difference between a pattern detected in the real image data by the search and a contour shape pattern (Kirk, Paragraphs 0026-0027),
determine whether the distance difference exceeds a threshold set in advance; and
in response to determining that the distance difference exceeds the threshold, correct a position of the entire contour shape pattern to a position of the pattern detected in the real image data by the search (Kirk, Fig. 4, Steps 412-420, Fig. 5, Steps, 508-518, Paragraphs 0026-0030).
These arts are analogous since they are both related to augmented reality display. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to modify the combination of Kennedy and Fritz with the correction unit as seen in Kirk to perform alignment of virtual objects and real world images to ensure the augmented reality objects are in the correct placement.
Regarding claim 2, the combination of Kennedy, Fritz and Kirk teaches the display device for a work machine according to claim 1 (see claim 1 analysis), wherein the processor is configured to display a name of the existing structure (Kennedy, Fig. 5, text 506, Paragraph 0103 and 0109), the terrain (Kennedy, Fig. 5, Paragraphs 0106-0108), or the boundary of the zone in addition to the augmented image (Kennedy, Fig. 5, worksite boundary 505, Paragraph 0105).
Regarding claim 3, the combination of Kennedy, Fritz and Kirk teaches the display device for a work machine according to claim 1 (see claim 1 analysis), further comprising:
a projection device configured to perform image display on a transparent panel of a window of a driver's seat of the work machine (Kennedy, Paragraph 0103, “View 500 can be augmented either by…a device mounted on work machine 102 (e.g., projected onto the glass of the operator cab 101).”),
wherein the processor is configured to display the augmented image on the transparent panel of the window of the driver's seat of the work machine through the projection device (Kennedy, Paragraph 0061, 0099-0100).
Regarding claim 4, the combination of Kennedy, Fritz and Kirk teaches the display device for a work machine according to claim 1 (see claim 1 analysis), further comprising:
an image display device configured to display the image of the field of view from the predetermined viewpoint of the work machine (Kennedy, Paragraphs 0099-0100),
wherein the processor is configured to display the augmented image on the image display device while superimposing the augmented image on the image of the field of view (Kennedy, Paragraph 0099, “For example, an opaque display device can include an LCD or LED screen. In this latter example, the augmented display device components are superimposed over a live video feed from an image sensor mounted on work machine 102.”, The live video feed from the image sensor mounted on the work machine is a predetermined viewpoint of the work machine., Fig. 7, Paragraph 0117, The predetermined viewpoint may alternatively be the bird’s eye view.).
Regarding claim 5, the combination of Kennedy, Fritz and Kirk teaches the display device for a work machine according to claim 1 (see claim 1 analysis), wherein the image of the field of view is an overhead image of the work machine (Kennedy, Fig. 7, Paragraph 0117, The bird’s eye view is an overhead.).
Regarding claim 14, the combination of Kennedy, Fritz and Kirk teaches a work machine comprising: the display device for a work machine according to claim 1 (see claim 1 analysis).
Regarding claim 16, the combination of Kennedy, Fritz and Kirk teaches the display device for a work machine according to claim 1 (see claim 1 analysis), wherein the processor is configured to acquire the peripheral information from an external device through a predetermined network (Kennedy, Fig. 3B, Worksite server 300, Paragraph 0091).
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kennedy et al. (US 2020/0071912 A1) in view of Fritz et al. (US 2016/0060825 A1) in view of Kirk (US 2014/0218291 A1) in view of Skillsäter et al. (US 2022/0228341 A1).
Regarding claim 6, the combination of Kennedy, Fritz and Kirk teaches the display device for a work machine according to claim 4 (see claim 4 analysis). However, the combination of Kennedy, Fritz and Kirk does not teach wherein the image display device is an image display device for remote control of the work machine.
In reference to Skillsäter et al. (hereafter referred to as Skills) Skills teaches wherein an image display device is an image display device for remote control of the work machine (Skills, Fig. 1, display 112, Paragraphs 0027-0031).
These arts are analogous since they are both related to controls of construction machines. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to modify the combination of Kennedy, Fritz and Kirk with the teaching of a remote system as seen in Skills to allow an operator to control the machine from a remote location.
Claim(s) 7-8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kennedy et al. (US 2020/0071912 A1) in view of Fritz et al. (US 2016/0060825 A1) in view of Kirk (US 2014/0218291 A1) in view of Watanabe (US 2021/0107356 A1).
Regarding claim 7, the combination of Kennedy, Fritz and Kirk teaches the display device for a work machine according to claim 1 (see claim 1 analysis), wherein the predetermined viewpoint is a position of the eyes of an operator sitting in a cab seat of the work machine (Kennedy, Paragraph 0049).
However, the combination of Kennedy, Fritz and Kirk does not explicitly state wherein the predetermined viewpoint is an intermediate position between right and left eyes of the operator.
In reference to Watanabe, Watanabe teaches wherein a predetermined viewpoint for an augmented reality display is an intermediate position between right and left eyes of the operator (Watanabe, Paragraph 0077 and 0236).
These arts are analogous since they are both related to augmented reality display in vehicles. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to modify the combination of Kennedy, Fritz and Kirk with the teaching of placing the viewpoint at an intermediate position between right and left eyes as seen in Watanabe since it is a known location as a viewpoint for an operator of a vehicle and would provide similar and expected results for proper display perspective of the augmented reality projections.
Regarding claim 8, the combination of Kennedy, Fritz, Kirk and Watanabe teaches the display device for a work machine according to claim 7 (see claim 7 analysis), wherein the predetermined viewpoint is determined by detecting positions of the right and left eyes of the operator by imaging with a camera (Kennedy, Fig. 25, driver-image capturing unit 107, cameras 3, Paragraph 0236).
Claim(s) 11-13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kennedy et al. (US 2020/0071912 A1) in view of Fritz et al. (US 2016/0060825 A1) in view of Kirk (US 2014/0218291 A1) in view of Matsuzaki (US 2018/0170258 A1).
Regarding claim 11, the combination of Kennedy, Fritz and Kirk teaches the display device for a work machine according to claim 1 (see claim 1 analysis),
wherein the processor is configured to perform processing of displaying an image of a periphery of the work machine and the augmented image of the existing structure converted into an overhead image by the processor on an image display device while superimposing the augmented image on the image (Kennedy, Fig. 7, Paragraph 0117-0120).
However, the combination of Kennedy, Fritz and Kirk does not teach wherein the processor is configured to perform processing of displaying a real image of a periphery of the work machine converted into an overhead image.
In reference to Matsuki, Matsuki teaches wherein performing processing of displaying a real image of a periphery of a work machine converted into an overhead image (Matsuki, Figs. 2-3, overhead image generating unit 60, Paragraph 0024-0027).
These arts are analogous since they are both related to work vehicles. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to modify the combination of Kennedy, Fritz and Kirk with teaching of using real images as seen in Matsuki to allow the overhead view to show real time images around the work vehicle.
Regarding claim 12, the combination of Kennedy, Fritz, Kirk and Matsuki teaches the display device for a work machine according to claim 11 (see claim 11 analysis),
wherein the processor is configured to, after converting captured image data captured by a front camera, a rear camera, a right camera, and a left camera (Matsuki, Fig. 3, First to fourth cameras 4, Paragraph 0015) of an imaging device into captured image data viewed from a field of view from a point at a predetermined height vertically above, synthesize the captured image data to integrate the captured image data, and generate real image data of the overhead image (Matsuki, Figs. 2-3, overhead image generating unit 60, Paragraph 0024-0027).
Regarding claim 13, the combination of Kennedy, Fritz, Kirk and Matsuki teaches the display device for a work machine according to claim 11 (see claim 11 analysis),
wherein the processor is configured to fit and synthesize an overhead image of the work machine prepared in advance or an overhead viewing icon to generate the overhead image (Matsuki, Fig. 3, Paragraph 0027).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WESLEY JASON CHIU whose telephone number is (571)270-1312. The examiner can normally be reached Mon-Fri: 8am-4pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Twyler Haskins can be reached at (571) 272-7406. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WESLEY J CHIU/ Examiner, Art Unit 2639
/TWYLER L HASKINS/ Supervisory Patent Examiner, Art Unit 2639
Prosecution Insights