DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendment filed 2 January, 2026 has been entered. Claims 1-12 remain pending in the application.
Response to Arguments
Applicant's arguments, see Applicant’s Remarks, filed 2 January, 2026, with respect to the rejection of claims 1-5 and 11 under 35 USC 102 and the rejection of claims 6-10 and 12 under 35 USC 103 have been fully considered but they are not persuasive. Specifically, Applicant argues:
In the embodied implementations, the processor stores an MR view image that is captured while the vehicle is traveling in a memory, and then, when a specific condition is satisfied, outputs the stored MR view image as a replay image corresponding to what was played during the vehicle's prior travel. In other words, the replay image is the image stored/recorded and generated in real time as a traveling history, rather than preexisting content prepared in advance.
In contrast, Wang discloses that "augmentation can also be performed on the previously compiled content," such as sponsored content (e.g., ads) augmented onto previously recorded images or video. Wang therefore does not disclose or suggest a route guidance device that includes a processor configured to store MR view image captured while the vehicle is traveling in a memory and to output the stored MR view image when a specific condition is satisfied.
Thus, Wang's content is external data (Ads or the like) previously compiled, whereas this application provides replay of the traveling image which is the history generated/displayed when the vehicle traveled the corresponding point in the past, which makes this application clearly different from Wang.
The Examiner has carefully considered the argument, however it is not persuasive. Wang does teach capturing MR view images while the vehicle is traveling (See Wang Paragraph 0027 When requesting augmented virtual reality content, the request may further includes an image or images captured by one or more cameras of autonomous vehicle 101 in real-time. The image or images may represent surroundings of autonomous vehicle 101.) and storing the MR view images (See Wang Paragraph 0022 Data store 125 may be maintained in a persistent storage device (e.g., a non-volatile storage device such as a hard disk) storing a variety of data including, but is not limited to, a content database (e.g., multimedia content), a user profile, trip related information ( e.g., location and route information, POIs information). Data stored in data store 125 may be obtained from a variety of data sources, such as, server 102, traffic information server 104, map and POI server 105, and location server 106. For example, content data and user profiles may be provided by and cached from content database 131 and user profiles 132 of data store 130 of server 102. Data processing system 110 further includes other components, such as one or more processors (e.g., central processing units or CPUs), a system memory, or a communication interface (e.g., wireless communication interface), etc. [Wang teaches implementing the invention on a computing device, and therefore the MR view images are inherently stored in memory.]). As was set forth in the Non-Final Rejection filed 2 October, 2025, Wang further teaches outputting the MR view image when a specific condition is satisfied (See Wang Paragraph 0034 For example, a camera may capture an object on the side of the road, where the object may be a building of a retail shop. The image is then analyzed and recognized in view of location and route information, MPOI information, and/or real-time traffic data of the autonomous vehicle.).
Applicant further argues:
In addition to the deficiency noted above with respect to claim 1, the cited references, including Wang and Cho, do not disclose the features of claim 6, which recites inter alia, "the processor is configured to set the one point based on a vehicle object, the vehicle object corresponding to the vehicle, and to control the MR view image to be enlarged into a state where the POI object is located at a central area, by adjusting the predetermined viewing angle to be directed to the POI object, when the vehicle object is moving toward the POI object." (emphasis added). The Office Action alleges that Cho discloses this feature because it enlarges the size of an AR object with respect to a changing depth of the point where the AR object is located. However, this characterization is incorrect because the claims require enlarging the past image captured by the device based on a specific POI, not an AR object.
Thus, as Cho teaches on page 2 line 39: "processor increases the size of the AR object... when the depth of the point where the AR object is located increases", Cho is merely about improving visibility of the AR object according to a change in distance (adjust the size), which has nothing to do with the technical concept of enlarging the view and replaying the past image based on a specific point (POI) as defined in claim 6.
The Examiner has carefully considered the argument, however it is not persuasive. The broadest reasonable interpretation of the POI object recited in the claim, in light of the specification, includes the AR object taught by Cho. Therefore, Cho teaches controlling the MR view image to be enlarged into a state where a POI object is located at a central area, by adjusting the predetermined viewing angle to be directed to the POI object, when the vehicle object is moving toward the POI object. While Applicant argues that Cho does not teach the concept of enlarging the view and replaying the past image, claim 6 does not recite limitations regarding replaying a past image.
Finally, Applicant argues:
None of the cited references, including Stroila, teach or suggest these features related to the real time capture of a thumbnail image of the MR view image and generating a replay image to be overlaid on an area of the MR view image, which can then be used in various useful embodiments as described in the specification in paragraphs [0838]-[0844].
Thus, Stroila teaches at paragraph [0057]: "FIG. 4 shows an example graphic overlay 52. The point of interest within the view of the scene is identified as a men's restroom" so that Stroila relates to a general AR technology that shows POI information through an icon or overlay. Therefore, Stroila differs from claim 7 defining "capture a thumbnail image of the MR view image when a distance between a vehicle object and a Point of Interest (POI) object displayed on the MR view image is within a preset distance". In this regard, Stroila does not have the "capture a current view" function.
The Examiner has carefully considered the argument, however it is not persuasive. Stroila does teach capturing an MR view image when a distance between a vehicle object and a Point of Interest (POI) object displayed on the MR view image is within a preset distance (See Stroila Paragraph 0048 The selection of augmented reality information is based on the location. Points of interest adjacent to or within view of the location of the mobile device are selected. Augmented reality information for points of interest within the region around the location and/or for the part of the region in the direction of viewing is selected. The location indicates a region around the mobile device. Points of interest within a distance of the location may be selected.) and that the thumbnail image displayed may be based on the captured image (See Stroila Paragraph 0059 Alternatively, the displayed scene is a graphic derived from the image, such as a graphic showing boundaries or scene outlines.).
Claim Rejections - 35 USC § 102
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claim(s) 1-5 and 11 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Wang (U.S. Patent Application Publication 2017/0330034).
Regarding claim 1, Wang teaches a route guiding device comprising: a communication circuit that communicates with a cloud server (Paragraph 0041 At block 502, processing logic transmits the images to a cloud server over a network, optionally including one or more user IDs of one or more users riding the autonomous vehicle.); an interface circuit that receives, from at least one sensor disposed in a vehicle, a camera image including an image of a road on which the vehicle is traveling, and sensing information obtained by sensing a traveling state of the vehicle (Paragraph 0041 Referring to FIG. 5, at block 501, processing logic receives a stream of images representing one or more objects (e.g., buildings, signs) external to an autonomous vehicle using one or more cameras of the autonomous vehicle.); a memory; and a processor (Paragraph 0022 Data store 125 may be maintained in a persistent storage device (e.g., a non-volatile storage device such as a hard disk) storing a variety of data including, but is not limited to, a content database (e.g., multimedia content), a user profile, trip related information ( e.g., location and route information, POIs information). Data stored in data store 125 may be obtained from a variety of data sources, such as, server 102, traffic information server 104, map and POI server 105, and location server 106. For example, content data and user profiles may be provided by and cached from content database 131 and user profiles 132 of data store 130 of server 102. Data processing system 110 further includes other components, such as one or more processors (e.g., central processing units or CPUs), a system memory, or a communication interface (e.g., wireless communication interface), etc.) configured to execute a mixed reality (MR) module that renders MR information including at least one virtual object, based on the camera image, the sensing information, and map information received from the cloud server (Paragraph 0042 At block 503, processing logic receives a stream of augmented images from the cloud server. The augmented images are processed, augmented, and rendered by the cloud server based on the images sent to the cloud server. Paragraph 0022 Data stored in data store 125 may be obtained from a variety of data sources, such as, server 102, traffic information server 104, map and POI server 105, and location server 106.), control the interface circuit to display an MR view image including the MR information is displayed on a display of the vehicle (Paragraph 0042 At block 504, the augmented images are displayed on one or more display devices of the autonomous vehicle in a virtual reality manner. Some of the augmented images may be displayed on a display device that is common to all users, while other augmented images may be streamed to a private display device (e.g., mobile phones, tablets) of individual users.), store the MR view image, which is captured while the vehicle is traveling, in the memory (See Wang Paragraph 0022 Data store 125 may be maintained in a persistent storage device (e.g., a non-volatile storage device such as a hard disk) storing a variety of data including, but is not limited to, a content database (e.g., multimedia content), a user profile, trip related information ( e.g., location and route information, POIs information). Data stored in data store 125 may be obtained from a variety of data sources, such as, server 102, traffic information server 104, map and POI server 105, and location server 106. For example, content data and user profiles may be provided by and cached from content database 131 and user profiles 132 of data store 130 of server 102. Data processing system 110 further includes other components, such as one or more processors (e.g., central processing units or CPUs), a system memory, or a communication interface (e.g., wireless communication interface), etc. [Wang teaches implementing the invention on a computing device, and therefore the MR view images are inherently stored in memory.]), and output the stored MR view image (Paragraph 0042 At block 504, the augmented images are displayed on one or more display devices of the autonomous vehicle in a virtual reality manner. Some of the augmented images may be displayed on a display device that is common to all users, while other augmented images may be streamed to a private display device (e.g., mobile phones, tablets) of individual users.), which was played when the vehicle was traveling in the past, as a replay image when a specific condition is satisfied (Paragraph 0026 The augmentation can also be performed on the previously compiled content. For example, sponsored content (e.g., Ads) may be augmented onto an existing image or video that has been previously collected.).
Regarding claim 2, Wang teaches the system of claim 1 as set forth above. Wang further teaches wherein a Point of Interest (POI) object is output on the MR view image, and the specific condition includes a case where a vehicle object corresponding to the vehicle passes the POI object (Paragraph 0034 For example, a camera may capture an object on the side of the road, where the object may be a building of a retail shop. The image is then analyzed and recognized in view of location and route information, MPOI information, and/or real-time traffic data of the autonomous vehicle.).
Regarding claim 3, Wang teaches the system of claim 1 as set forth above. Wang further teaches wherein a replay card is overlaid on the MR view image when the specific condition is satisfied, and the specific condition includes a case where the replay card is selected by a user (Paragraph 0024 The user can specifically select the real-time actual content captured in real-time or previously rendered content to be displayed on the display device(s), for example, retrieved from data store 125.).
Regarding claim 4, Wang teaches the system of claim 1 as set forth above. Wang further teaches wherein the specific condition includes a case where a Point of Interest (POI) object displayed on the MR view image is selected (Paragraph 0024 The user can specifically select the real-time actual content captured in real-time or previously rendered content to be displayed on the display device(s), for example, retrieved from data store 125.). The Examiner notes that limitations in this claim are presented in the alternative and as such, prior art is not presented for all limitations.
Regarding claim 5, Wang teaches the system of claim 1 as set forth above. Wang further teaches wherein the MR view image is a scene looking at a digital-twin map, to which a current situation is reflected in real time, at one point at a predetermined viewing angle (Paragraph 0044 The techniques described throughout this application can be used to provide certain sponsored content, such as advertisements or Ads, to a user riding in an autonomous vehicle, based on the traveling environment at the point in time along a route of a trip (e.g., location and route information, real-time traffic condition, map and point of interests).).
Regarding claim 11, Wang teaches the system of claim 1 as set forth above. Wang further teaches wherein the replay image is played and overlaid on one area of the MR view image or is output in a form of a pop-up window (Paragraph 0018 The first stream media content is presented on a first display device, where the first display device is configured and mounted at a first position resembling a first window of the autonomous vehicle.).
Claim Rejections - 35 USC § 103
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Cho (WIPO Publication WO2021133053).
Regarding claim 6, Wang teaches the system of claim 5 as set forth above. However, Wang does not teach wherein the processor is configured to set the one point based on a vehicle object, the vehicle object corresponding to the vehicle, and control the MR view image to be enlarged into a state where a Point of Interest (POI) object is located at a central area, by adjusting the predetermined viewing angle to be directed to the POI object, when the vehicle object is moving toward the POI object.
Cho, in the same field of endeavor, teaches a system for displaying augmented reality images. The system controls the display of AR objects to adjust the size of the AR object as the user becomes closer or further from the object (Page 2 line 39 Here, the processor increases the size of the AR object to maintain the size at which the AR object is displayed on the display when the depth of the point where the AR object is located increases while the mode of the electronic device is the arrangement mode, and the AR object is positioned When the depth of the point of interest decreases, the size of the AR object may be reduced to maintain the size at which the AR object is displayed on the display.)
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to have modified Wang with the teachings of Cho which teaches controlling the display of AR objects to adjust the size of the AR object as the user becomes closer or further from the object in order to maintain the AR object at a size that is more convenient to interact with regardless of the user’s distance from the object (See Cho Page 10 line 42 Accordingly, the visibility of information may be improved in that the user can view an AR object to which an attribute having a fixed size, such as text-type information, is assigned regardless of a depth (distance) in an appropriate size. In addition, user convenience maybe improved in that the user does not need to separately adjust the size of the AR object.).
Claim(s) 7-10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Stroila (U.S. Patent Application Publication 2013/0147837).
Regarding claim 7, Wang teaches the system of claim 1 as set forth above. However, Wang does not teach wherein the processor is configured to capture a thumbnail image of the MR view image when a distance between a vehicle object and a Point of Interest (POI) object displayed on the MR view image is within a preset distance, the vehicle object corresponding to the vehicle, and generate the thumbnail image as the replay image overlaid on one area of the MR view image.
Stroila, in the same field of endeavor, teaches a system for displaying augmented reality information to a user. The system generates a graphic overlay for a POI (Paragraph 0057 FIG. 4 shows an example graphic overlay 52. The point of interest within the view of the scene is identified as a men's restroom. Other information may be provided, such as hours of operation. More than one graphic for the point of interest may be generated, and/or graphic overlays for more than one point of interest in the view may be generated.) to be displayed for POIs which are within a certain distance from the user (Paragraph 0048 The selection of augmented reality information is based on the location. Points of interest adjacent to or within view of the location of the mobile device are selected. Augmented reality information for points of interest within the region around the location and/or for the part of the region in the direction of viewing is selected. The location indicates a region around the mobile device. Points of interest within a distance of the location may be selected.).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to have modified Wang with the teachings of Stroila which teaches generating a graphic overlay for a POI to be displayed for POIs which are within a certain distance from the user in order to display only the most relevant information to the user (See Stroila Paragraph 0011 Personalization allows delivery of augmented reality content specific to the user, such as specific geographical points of interest and/or advertising relevant to the particular user. The personalization may increase and enhance the use of navigation data in augmented reality applications.).
Regarding claim 8, Wang in view of Stroila teaches the system of claim 7 as set forth above. Wang further teaches wherein the replay image is implemented as a replay card including a name corresponding to the POI object (Paragraph 0034 Other information related to the recognized retail shop, such as a user ratings of the recognized retail shop, other shops nearby, including the name or types of the shops, their user ratings, and/or the distance from the retail shop, may be obtained. The information may be annotated on the augmented image, which is then streamed back to the autonomous vehicle.). The Examiner notes that limitations in this claim are presented in the alternative and as such, prior art is not presented for all limitations.
Regarding claim 9, Wang in view of Stroila teaches the system of claim 8 as set forth above. Wang further teaches wherein the processor is configured to sequentially generate replay cards whenever the vehicle object sequentially passes different POI objects and output the generated replay cards to the MR view image (Paragraph 0034 Other information related to the recognized retail shop, such as a user ratings of the recognized retail shop, other shops nearby, including the name or types of the shops, their user ratings, and/or the distance from the retail shop, may be obtained. The information may be annotated on the augmented image, which is then streamed back to the autonomous vehicle.).
Regarding claim 10, Wang in view of Stroila teaches the system of claim 7 as set forth above. Wang further teaches wherein the processor is configured to play, as the replay image, the MR view image including that the vehicle has traveled for a predetermined time while looking at the POI object corresponding to the replay image, when the replay image is selected (Paragraph 0026 The augmentation can also be performed on the previously compiled content. For example, sponsored content (e.g., Ads) may be augmented onto an existing image or video that has been previously collected.).
Claim(s) 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Herz (U.S. Patent Application Publication 2018/0079427).
Regarding claim 12, Wang teaches the system of claim 1 as set forth above. However, Wang does not teach wherein the replay image includes at least one of service information available at a Point of Interest (POI) linked to the replay image or a button for setting a travel route to a place corresponding to the POI.
Herz, in the same field of endeavor, teaches a system for navigating a vehicle using augmented reality. The system displays augmented reality images of POIs nearby a vehicle and allows an occupant of the vehicle to select a POI for the vehicle to navigate to (Paragraph 0052 Respective icons 656, 657 and 658 may be generated to represent the destinations, while the lengths and directions of the arrows in display 620 may indicate the (approximate, not-necessarily-to-scale) relative distances and directions of the destinations. Since the restaurant is further away than the department store, a longer arrow may be used for the restaurant option than is used for the department store option, for example, and so on. In response to the selection of a particular one of the options displayed (e.g., using some of the kinds of gestures and/or other signals discussed above), the vehicle 610 may be caused to move to, and park near, the building or location corresponding to the selected option.).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to have modified Wang with the teachings of Herz which teaches displaying augmented reality images of POIs nearby a vehicle and allows an occupant of the vehicle to easily select a POI for the vehicle to navigate to in order to allow the occupant to select relevant locations as destinations for navigation See Herz Paragraph 0049 As shown in table 622, respective relevance scores or interest scores may be estimated for at least some of the destinations, indicating the likelihoods of each of the destinations being selected by the occupant(s) of vehicle 610.).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PATRICK D MOHL whose telephone number is (571)272-8987. The examiner can normally be reached M-Th 6:00AM-4:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Anne Antonucci can be reached at (313) 446-6519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PATRICK DANIEL MOHL/Examiner, Art Unit 3666
/ANNE MARIE ANTONUCCI/Supervisory Patent Examiner, Art Unit 3666
Prosecution Insights