DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This Office Action is in response to Applicant’s amendment/response filed on 05/11/2026, which has been entered and made of record.
Claim Objections
Claims 1, 21, and 27-28 are objected to because of the following informalities:
In claim 1, line 15, “the augmented video” should read “augmented video” due to the first appearance.
In claim 1, line 25, “the annotated image and augmented video” should read “annotated image and augmented video” due to the first appearance.
In claim 21, line 12, “the augmented video” should read “augmented video” due to the first appearance.
In claim 21, line 22, “the annotated image and augmented video” should read “annotated image and augmented video” due to the first appearance.
In claim 27, line 15, “the augmented video” should read “augmented video” due to the first appearance.
In claim 27, line 25, “the annotated image and augmented video” should read “annotated image and augmented video” due to the first appearance.
In claim 28, line 16, “the augmented video” should read “augmented video” due to the first appearance.
In claim 28, line 26, “the annotated image and augmented video” should read “annotated image and augmented video” due to the first appearance.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 21-23, and 26-28 are rejected under 35 U.S.C. 103 as being unpatentable over Nussbaum et al. (US 10832476 B1, hereinafter “Nussbaum”) in view of Wright et al. (US 11417091 B2, hereinafter “Wright”) and Siddell (What is Object Tracking and ReIdentification, archive.org, https://web.archive.org/web/20221121173124/https://alwaysai.co/blog/object-tracking-guide, hereinafter “Siddell”).
Regarding claim 1, Nussbaum teaches A method of using augmented reality to assess damage to an object comprising: (Abstract, “This application discloses methods, systems, and computer implemented virtualization software applications and computer- implemented graphical user interface tools for remote virtual visualization of structures”; col. 23, lines 55-58, “the preceding discussion primarily discusses damage assessment using Virtual Reality, Augmented Reality (AR), and/or or mixed reality by generating models representing sites, areas, structures, or portions thereof”).
… viewing damage to the object in the viewer from a first view; (FIGURE 12: a roof damage on structure 1224; col. 17, lines 58-62, “FIG. 12 illustrates presentation of an exemplary virtual 3D digital model of the structure 1110, by displaying a representation of the virtual 3D digital model 1224 of the structure 1110 within a virtual 3D digital environment 1228 on a user interface unit 1218”). Note that: (1) the damage to the object (roof) is displayed or viewed in the viewer of a user interface unit 1218; and (2) since the presentation in FIGURE 12 is rendered based on the virtual 3D digital model of the object (structure) within a virtual 3D digital environment, a perspective indicating a viewpoint (a first view) is taken as the viewpoint for the image view.
… damages ... (col. 23, lines 60-63, “The methods and systems described above apply equally to other uses with appropriate modifications, primarily modifications to the types of virtual objects and data sources used to estimate costs associated with the damage”).
creating a three-dimensional model of the object; (FIGURE 10: step 1008, “GENERATE ONE OR MORE VIRTUAL 3D MODELS INCLUDING VIRTUAL REPRESENTATION OF ONE OR MORE STRUCTURES”). Note that: there are no other specific limitations besides “creating 3-d model of the object” in this claim.
estimating loss based on the augmented video; (lines 60-63, “The methods and systems described above apply equally to other uses with appropriate modifications, primarily modifications to the types of virtual objects and data sources used to estimate costs associated with the damage”; col. 22, lines 28-36, “As the AR session may be recorded, the expert's annotations may 30 be provided for use by various users, and the "sticking" annotations can then be used to overlay or annotate or overlay over different video captured by the devices of those various users. Further, this may allow the various users to be at any perspective or position in their respective environments without losing the effect or message of the expert's annotations”). Note that: (1) the augmented video images including all images (scene images, the rendered images from 3D object model, video, annotations-overlaid images, etc.) can be recorded and stored in the corresponding memory, database, or storage as cited for claim 1; and (2) the damage loss or cost can be assessed or estimated based on the augmented video images that include virtual objects and data sources.
resizing and adjusting the object data to fit the object (col. 17, lines 58-62, “FIG. 12 illustrates presentation of an exemplary virtual 3D digital model of the structure 1110, by displaying a representation of the virtual 3D digital model 1224 of the structure 1110 within a virtual 3D digital environment 1228 on a user interface unit 1218”; col. 24, lines 7-17, “the virtual objects may correspond to components of a vehicle, such as a windshield, a bumper, a hood, a door, a side view mirror, a wheel, a light housing, a trunk, a roof. The user may select, position, and resize the virtual objects to match damaged components of the vehicle … the virtual objects may be generic virtual objects representing general components found in many vehicles. In other embodiments, the virtual objects may be selected for a specific make and model of vehicle, in order to better fit the design of the damaged
vehicle.”). Note that: (1) The damaged components can be regarded as the object; (2) the virtual objects can be regarded as object data (e.g., CAD files as disclosed later) corresponding to the real components of a vehicle; (3) the user can resize and position (adjust) the virtual to match or fit damaged components; and (4) the virtual objects are viewed on a user interface unit and can be viewed by the image capture device as disclosed below.
However, Nussbaum fails to disclose, but in the same art of computer graphics, Wright discloses
connecting to an augmented reality system comprising a first user and a second user; (Wright, FIG. 1: an augmented reality system consisting of “CAMERA”, “AR DISPLAY”, “VIRTUAL ANNOTATIONS, “VISUAL TRACKING”, “VIEW SYNTHESIS”, “VR DISPLAY”, etc. and “LOCAL USER” and REMOTE USER”; col. 3, lines 25-36, “FIG. 1 illustrates an example system for a particular example application that enables a remote user to explore a physical environment via live imagery from a camera that the local user holds or wears, such as a camera of a mobile device or a wearable computing device, which may be or include a networked, wearable camera. The remote user is able to interact with a model fused from images captured from the surroundings of the local user and create and add virtual annotations in it or transfer live imagery (e.g., of gestures) back”). Note that: (1) a local user is a first user while a remote user is a second user; and (2) the two users connect to an augmented reality system in FIG. 1 of Wright.
viewing the object through an image capture device; … as viewed by the image capture device; … images captured by the image capture device … (Wright, FIG.3: “LOCAL USER A” can view a car engine through a camera; col. 3, lines 28-33, “that enables a remote user to explore a physical environment via live imagery from a camera that the local user holds or wears, such as a camera of a mobile device or a wearable computing device, which may be or include a networked, wearable camera”). Note that: (1) a camera of a mobile device or a wearable computing device is an image capture device; and (2) a car engine is an object for viewing.
accepting comments from the second user to the first user; (Wright, col. 22, lines 22-26, “Accordingly, the expert's annotations may be augmented or added to the environment as viewed by the user while allowing the user to freely choose his/her viewpoint to look at the user's environment as well as the annotations”; col. 28, lines 11-15, “The first part allows the user …to receive the expert's instructions and comments and the second part allows the user to replay the expert's instructions and comments to guide him/her through the issue”). Note that: (1) the expert is the second user while the user or local user is the first user; (2) the expert’s comments, instructions, or annotations can be accepted or received for the local user to view; and (3) There are no other specific limitations on how the users communicate to each other in this claim.
determining, based on user input and contextual metadata, if an annotation is desired; (Wright, col. 3, lines 13-21, “At least one of the local user and the remote user may be able to add annotations (e.g., markings, notes, drawings, etc.) to certain objects within the environment captured within the images and/or video. These annotations and the shared images and/or video may improve the communication between the local user and the remote user by, for example, allowing the local user and the remote user to visually identify specific objects in the local user's environment.”; col. 4, lines 21-23, “Annotations may include point-based markers, more complex three-dimensional annotations, drawings, or live imagery, such as hand gestures”). Note that: (1) the hand gesture or its indication description is a contextual metadata that describes the environment data of the annotations; and (2) the local users and remote users can generate annotations based on user adding or inputting and the contextual metadata (hand gesture); and (3) since the annotations may improve the communication between the local user and the remote user, the users can determine they are as useful or desired for applications, resulting in adding them.
in response to determining an annotation is desired:
enabling a collaborative annotation interface that allows simultaneous input from both users; (Wright, col. 3, lines 13-21, “At least one of the local user and the remote user may be able to add annotations (e.g., markings, notes, drawings, etc.) to certain objects within the environment captured within the images and/or video ... may improve the communication between the local user and the remote user by, for example, allowing the local user and the remote user to visually identify specific objects in the local user's environment.”; col. 3/ line 63- col. 4 / line 9 , “the local user may hold or wear a device that integrates a camera and a display system (e.g., hand-held tablet, mobile device, digital eyewear, or other hardware with a camera), which is used to both sense the environment and display visual/spatial feedback from the remote user correctly registered to the real world. In the case of a hand-held device, the handheld device acts as sort of a "magic lens" (i.e., showing the live camera feed and virtual annotations, when the embodiment includes AR). Since a collaboration system typically aids the user an actual task being performed rather than distracts from it, an interface which is simple and easy to comprehend is typically provided such as to facilitate an active user who may be looking at and working in multiple areas”). Note that: (1) an annotation is determined as useful or desired so that the condition is triggered; (2) local user and the remote user work together with a provided interface which is simple and easy to comprehend, and enables the collaboration of the local and remote users to add annotations (e.g., markings, notes, drawings, etc.); and (3) the live collaboration of the local and remote users indicate the simultaneous input from them (adding annotations) are allowed and employed.
storing the annotated image and augmented video in a format configured for asynchronous review by third-party viewers with access controls (Wright, col. 22, lines 28-36, “As the AR session may be recorded, the expert's annotations may be provided for use by various users, and the "sticking" annotations can then be used to overlay or annotate or overlay over different video captured by the devices of those various users. Further, this may allow the various users to be at any perspective or position in their respective environments without losing the effect or message of the expert's annotations”; col. 3, lines 13-21, “At least one of the local user and the remote user may be able to add annotations (e.g., markings, notes, drawings, etc.) to certain objects within the environment captured within the images and/or video ... may improve the communication between the local user and the remote user by, for example, allowing the local user and the remote user to visually identify specific objects in the local user's environment”; col. 21, lines 26-34, “the AR or VR system can be programmed to communicate concurrently while also creating a recording for later review. Accordingly, in some embodiments, an AR or VR session may be generated based on a "live" or current issue being faced by the user and recorded for later review by the user or other users or recorded for later review by the users and other users”; FIG. 1: “OBJECT DATABASE”; FIG. 2: “NO-VOLATILE” memory 208 and “NON-REMOVABLE STORAGE 214”). Note that: (1) the expert’s annotations and annotation overlaid video (augmented video) can be stored when the AR session is recorded; (2) when the AR session including the annotations, captured videos, and overlays is recorded, the recorded contents of the AR session can be stored the corresponding database, memory, or/and storage; (3) it is obvious to one having ordinary skills in the art that the data are stored in a certain designated format (e.g., JPEG, MPEG, stream media formats, multi-media compression, transfer, storage formats) or formation that accommodates the stored AR data, the annotated image and augmented video data for concurrent user review or later user review in asynchronous review mode; (4) other users or additional parties can be allowed at any perspective or position to view the annotation along with augmented or overlaid video. It is obvious to one having ordinary skills in the art that the users have their corresponding access credentials or access control for data security and management; (5) other users or additional parties can be third party viewers reviewing the stored the annotated image and augmented video data; and (6) Examiner notices that there is no definition of “third party viewers” in the specification.
highlighting damages to the object in the three-dimensional model; (Wright, col. 8, lines 44-48. “the object may be highlighted on a display associated with the electronic device and the user may be able to select the object to bring up a menu of options to interaction with the object within an AR environment”; col. 7, lines 5-6, “this may include highlighting or otherwise visually identifying the recognized object”). Note that: (1) the damage to the object in the three-dimensional model 3D is included in the object and can be highlighted; and (2) the combination of Nussbaum and Wright here highlights the damages to the object.
identifying the object as a specific object type;
retrieving object data for the specific object type comprising a three-dimensional wireframe and component labels;
overlaying the object data on the image of the object; (Wright, col. 31, lines 11-27, “the object identification or recognition algorithm described above and implemented by the processing unit may be configured to identify the object, and the processing unit may further identify details regarding the identified object in a database … after identifying an object as being a sink faucet, may access a CAD file of the sink faucet and use the CAD file to identify hidden components ( e.g., components that are not readily visible) of the sink faucet. In some embodiments, this may be shown as a semi-transparent or transparent overlay view or layer. This overlay view may show all or some selectable parts. In some embodiments, the overlay view or the CAD file may include a menu or list of parts, where selection of an item in the menu or list by the user will highlight the part in the overlay view. In some embodiments, there may not be an overlay view and the parts will be selectable or highlighted on the main view itself”). Note that: (1) a faucet is identified as a specific object; (2) after identifying the object, the CAD file as a type of wireframe of the object can be accessed and retrieved from database, and can be overlayed onto the object as a semi-transparent or transparent overlay view or layer. This overlay view may show all or some selectable parts with object or part identifiers (labels or a menu or list of parts); and (3) since the overlay view may show all or some selectable parts, the CAD is overlaid on the image of the parts to form the overlay view.
allowing the first view to be changed to a second view; and (Wright, col. 22, lines 22-24, “the expert's annotations may be augmented or added to the environment as viewed by the user while allowing the user to freely choose his/her viewpoint to look at the user's environment as well as the annotations”). Note that: (1) the expert is the second user while the user or local user is the first user; and (2) the user is allowed to freely change his/her viewpoint, which is equivalent to allowing the view to be changed to a second view from the first view.
Nussbaum and Wright are in the same field of endeavor, namely computer graphics. Before the effective filing date of the claimed invention, it would have been obvious to apply viewing an object, accepting comments, determining if an annotation is desired, allowing an image to be annotated, and storing the annotation along with augmented video, as taught by Wright into Nussbaum. The motivation would have been “These annotations and the shared images and/or video may improve the communication between the local user and the remote user by, for example, allowing the local user and the remote user to visually identify specific objects in the local user's environment” (Wright, col. 3, lines 17-21). The suggestion for doing so would allow to improve the communication between the local user, the remote user, and additional users. Therefore, it would have been obvious to combine Nussbaum with Wright.
However, Nussbaum in view of Wright fails to disclose, but in the same art of computer graphics, Siddell discloses
anchoring the object data to the object such that movement of images captured by the image capture device causes the object data to follow corresponding movement of the object; (Siddell, page 1, para. 1, “In object tracking, the algorithm follows the movement of an object and tries to estimate or predict its position in a video. Re-identification is crucial in tracking moving objects because it enables us to identify the same object throughout a video sequence. Re-identification makes it possible to find objects, even if they are missing in several consecutive frames”; page 2, para. 1, “Object tracking enables us to track a unique object across a series of frames”; page 7, para. 4, “Training and feature extraction are critical steps to resolve these issues. Anchor boxes can be used to mitigate such problems”; page1, “
PNG
media_image1.png
416
822
media_image1.png
Greyscale
”). Note that: (1) the persons in the figure can be regarded as the objects for object tracking in video frames as images; (2) the text strings in green as the ID information the of persons above each green bounding box can be regarded as the object data and the green boxes follow the objects in the frames; and (3) object tracking is for tracking objects moving across a series of frames as movement of images while the green bounding boxes anchor the object data to the objects (persons) during the movements of the objects.
Nussbaum in view of Wright, and Siddell, are in the same field of endeavor, namely computer graphics. Before the effective filing date of the claimed invention, it would have been obvious to apply object tracking and reidentification, as taught by Siddell into Nussbaum in view of Wright. The motivation would have been “In object tracking, the algorithm follows the movement of an object and tries to estimate or predict its position in a video. Re-identification is crucial in tracking moving objects because it enables us to identify the same object throughout a video sequence. Re-identification makes it possible to find objects, even if they are missing in several consecutive frames” (Siddell, page 1, para. 1). The suggestion for doing so would allow to anchor object data to objects such that movement of images causes the object data to follow corresponding movement of the object. Therefore, it would have been obvious to combine Nussbaum, Wright, and Siddell.
Claim 21 is corresponding to claim 1. Therefore, claim 21 is rejected for the same rationale for claim 1.
Regarding claim 22, the combination of Nussbaum, Wright, and Siddell discloses The method of claim 21, wherein the object is a vehicle. (Nussbaum, col. 24, lines 4-11, “the methods and systems described above may be applied to vehicles, such as cars, trucks, boats, motorcycles, airplanes, or trains. For example, the virtual objects may correspond to components of a vehicle, such as a windshield, a bumper, a hood, a door, a side view mirror, a wheel, a light housing, a trunk, a roof panel, or a side panel.”)
Regarding claim 23, the combination of Nussbaum, Wright, and Siddell discloses The method of claim 22, wherein (Nussbaum, col. 24, lines 4-11, “the methods and systems described above may be applied to vehicles, such as cars, trucks, boats, motorcycles, airplanes, or trains. For example, the virtual objects may correspond to components of a vehicle, such as a windshield, a bumper, a hood, a door, a side view mirror, a wheel, a light housing, a trunk, a roof panel, or a side panel … Pointer objects may also be added to indicate further conditions, such as broken axels or water damage. From such virtual objects or pointer objects indicated by the user, the mobile computing device may determine the extent of damage and may generate a report, as discussed above”). Note that: (1) the user using the mobile computing device can capture the images for an interior (a roof panel, or a side panel) or hood portion; (2) it is obvious to one having ordinary skill that the under-hood portion can be checked or can be substituted by the hood portion for the first user to perform additional check and damage information (water damage and the extent of damage).
the comments from the second user comprise a request for the first user (Wright, col. 22, lines 22-26, “Accordingly, the expert's annotations may be augmented or added to the environment as viewed by the user while allowing the user to freely choose his/her viewpoint to look at the user's environment as well as the annotations”; col. 28, lines 11-15, “The first part allows the user …to receive the expert's instructions and comments and the second part allows the user to replay the expert's instructions and comments to guide him/her through the issue”). Note that: the comments from the second user (expert) can require or direct the first user to perform some actions.
Regarding claim 26, the combination of Nussbaum, Wright, and Siddell discloses The method of claim 21, wherein the annotated image and augmented video are reviewed to determine at least one of a value of damages to the object, whether a person may have been injured by the damage, fault in an accident, or whether the object is a total loss. (Nussbaum, col. 3, lines 14-25, “The remote user may access the representations of the virtual 3D digital models of the structure and/or location and view the representations of the virtual 3D digital models from different angles and perspectives in order to assess the extent of the damage to property. The remote user may be an insurance agent, or a third party, and may view the same images remotely and communicate remotely with the server and/or the other users. Both a first user and a second user may compare representations of the virtual 3D digital models of the structure and/or location that were captured before and after the damage, and they may compare the before and after images with each other.”; col. 24, lines 22-25, “From such virtual objects or pointer objects indicated by the user, the mobile computing device may determine the extent of damage and may generate a report, as discussed above.”). Note that: (1) the annotated image and augmented video can be regarded as the representations of the virtual 3D digital models of the structure and/or location and view the representations of the virtual 3D digital models from different angles and perspectives in order to assess the extent of the damage to property; and (2) the extent of the damage to property can be regarded as an assessment value of damages to the object (the property).
Claim 27 reciting “A non-transitory computer readable medium comprising computer executable instructions that physically configure a processor, the computer executable instruction comprising instructions for using augmented reality to assess damage to an object comprising instruction for:”, is corresponding to the method of claim 1. Therefore, claim 27 is rejected for the same rationale for claim 1.
In addition, the combination of Nussbaum, Wright, and Siddell discloses A non-transitory computer readable medium comprising computer executable instructions that physically configure a processor, the computer executable instruction comprising instructions for using augmented reality to assess damage to an object comprising instruction for: (Nussbaum, col. 30, lines 24-28, “18. A tangible, non-transitory computer-readable medium storing executable instructions for remote three-dimensional(3D) visualization of a location that, when executed by at least one processor of a computer system, cause the computer system to:”).
Claim 28 reciting “A computer system comprising: a processor that is physically configured according to computer executable instructions, a memory in communication with the processor; and an input-output circuit in communication with the processor, the computer executable instruction comprising instructions for using augmented reality to assess damage to an object comprising:”, is corresponding to the method of claim 1. Therefore, claim 17 is rejected for the same rationale for claim 1.
In addition, Nussbaum in view of Wright discloses A computer system comprising: a processor that is physically configured according to computer executable instructions, a memory in communication with the processor; and an input-output circuit in communication with the processor, the computer executable instruction comprising instructions for using augmented reality to assess damage to an object comprising: (Nussbaum, col. 29, lines 15-25, “11. A computer system for remote three-dimensional (3D) visualization of a physical structure, comprising: one or more processors; a communication component connected to the one or more processors and configured to send and receive electronic communications via a communication network; and a non-transitory program memory communicatively coupled to the one or more processors and storing executable instructions that, when executed by the one or more processors, cause the computer system to:”; col. 40-42, “ the communication unit 766 may provide input signals to the controller 750 via the I/O circuit 758”; FIGURE 7: “I/O” circuit 758 is in communication with “MICRO-PROCESSOR (MP)” 754).
Claim 24 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of Nussbaum, Wright, Siddell, and Wiki_Google_Maps (Google Maps, archive.org, https://web.archive.org/web/20230316111826/https://en.wikipedia.org/wiki/Google_Maps, hereafter “Wiki_Google_Maps”).
Regarding claim 24, the combination of Nussbaum, Wright and Siddell discloses The method of claim 21, wherein
However, the combination of Nussbaum, Wright and Siddell fails to disclose, but in the same art of computer graphics, Wiki_Google_Maps disclose the object data is overlaid on the image as separate layers comprising a wireframe layer and a component-label layer, and wherein the layers may be reordered or hidden depending on a purpose of analysis. (Wiki_Google_Maps, page 1, para. 1, “It offers satellite imagery, aerial photography, street maps, 360° interactive panoramic views of streets (Street View), real-time traffic conditions, and route planning for traveling by foot, car, bike, air (in beta) and public transportation.”; para. 3, “Google Maps' satellite view is a "top-down" or bird's eye view; most of the high-resolution imagery of cities is aerial photography taken from aircraft flying at 800 to 1,500 feet (240 to 460 m), while most other imagery is from satellites”; page 2, middle-right, Figure: “Google Maps Beta in 2005” showing wireframe of routes and icons (label) for start point and ending point of a drive). Note that: (1) it is known that a Google map for a city (e.g., Philadelphia, PA) shows a satellite imagery or aerial photography based bird’s eye view image (the image) when one selects a button of “Satellite” in group “Layers”; (2) The image is overlaid with a wireframe of route systems (wireframe layer) and icons of restaurants or hotels as layers (the object data; a city) for “Restaurants” or “Hotels” (component-label layer) when “Restaurants” or “Hotels” button is enabled; and (2) depending on finding “Restaurants” or “Hotels” as a destination information analysis, the “Restaurants” or “Hotels” layer can be toggled to show “Restaurants” or “Hotels” or be toggled to hide them.
The combination of Nussbaum, Wright, and Siddell, and Wiki_Google_Maps, are in the same field of endeavor, namely computer graphics. Before the effective filing date of the claimed invention, it would have been obvious to apply Google Maps’ function to have an image overlaid by a wireframe layer and a component-label layer and manipulating the layers for presentation or hiddenness, as taught by Wiki_Google_Maps into the combination of Nussbaum, Wright, and Siddell. The motivation would have been “Street map overlays, in some areas, may not match up precisely with the corresponding satellite images.” (Wiki_Foogle_Maps, page 1, para. 5). The suggestion for doing so would allow to overlay the object data as the layers of wireframe and labels on an image and manipulate the layers. Therefore, it would have been obvious to combine Nussbaum, Wright, Siddell, and Wiki_Google_Maps.
Claim 25 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of Nussbaum, Wright, Siddell, and Buscher et al. (Generating and Using Gaze-Based Document Annotations, CHI EA '08: CHI '08 Extended Abstracts on Human Factors in Computing Systems Pages 3045-3050, hereafter “Buscher”).
Regarding claim 25, the combination of Nussbaum, Wright, and Siddell discloses The method of claim 21, wherein the annotation is created using (Wright, col. 3, lines 13-21, “At least one of the local user and the remote user may be able to add annotations (e.g., markings, notes, drawings, etc.) to certain objects within the environment captured within the images and/or video. These annotations and the shared images and/or video may improve the communication between the local user and the remote user by, for example, allowing the local user and the remote user to visually identify specific objects in the local user's environment.”; col. 4, lines 21-23, “Annotations may include point-based markers, more complex three-dimensional annotations, drawings, or live imagery, such as hand gestures”). Note that: the local users (the first users) and remote users (the second user) can generate annotations based on user adding or inputting and the contextual metadata (hand gesture). to select and highlight a damaged region of the object. (Wright, col. 8, lines 44-48. “the object may be highlighted on a display associated with the electronic device and the user may be able to select the object to bring up a menu of options to interaction with the object within an AR environment”; col. 7, lines 5-6, “this may include highlighting or otherwise visually identifying the recognized object”). Note that: the damage to the object in the three-dimensional model 3D is included in the object and can be highlighted.
However, the combination of Nussbaum, Wright, and Siddell fails to disclose, but in the same art of computer graphics, Buscher discloses the annotation is created using eye-movement data (Buscher, Abstract, “In this paper we describe a prototypical system that is able to generate document annotations based on eye movement data”). Note that: (1) document annotations can be generated using eye movement data; and (2) eye-movement data can also substitute hand gesture to create an annotation.
The combination of Nussbaum, Wright, and Siddell, and Buscher, are in the same field of endeavor, namely computer graphics. Before the effective filing date of the claimed invention, it would have been obvious to apply using eye movement data to create an annotation, as taught by Buscher into the combination of Nussbaum, Wright, and Siddell. The motivation would have been “In this paper we describe a prototypical system that is able to generate document annotations based on eye movement data”” (Buscher, Abstract). The suggestion for doing so would allow to use eye movement data to create an annotation. Therefore, it would have been obvious to combine Nussbaum, Wright, Siddell, and Buscher.
Response to Arguments
Applicant's arguments with respect to claim rejection 35 U.S.C. 103 have been fully considered but they are not persuasive.
Applicant alleges, “The distinction is material. The amended claims require a system that (i) recognizes what the object is (a specific object type), (ii) retrieves technical reference data for that type (a three-dimensional wireframe and component labels), (iii) fits that data to the actual object as it appears in the image capture device, and (iv) locks the data to the object so it tracks with the object. This is fundamentally different from Nussbaum's approach of generating models from captured data and from Wright's approach of accepting user-drawn annotations on a video feed. Combining Nussbaum and Wright, with or without Mentis, would not yield this object-anchoring workflow, and the Office Action has not pointed to any teaching in the references that would supply the missing limitations or any rationale that would motivate a person of ordinary skill to arrive at them.” (page 10 / line 26, page 11 / line 5).However, Examiner respectfully disagrees about the respective allegations as whole because: (1) reference Siddell discloses the anchoring the object data to the object such that movement of images captured by the image capture device causes the object data to follow corresponding movement of the object; and (2) the combination of Nussbaum, Wright, and Siddell discloses all limitations of claim 1. Please see the prior art’s citations and rationale for claim 1 above. The arguments are not persuasive.
Applicant alleges, “For at least the foregoing reasons, amended claim 1 and new independent claims 21, 27, and 28, together with their respective dependent claims 22-26, are patentable over the cited art. Applicant respectfully requests withdrawal of the§ 103 rejections and allowance of the pending claims.” (page 11, lines 8-11). However, Examiner respectfully disagrees about the respective allegations as whole because: (1) independent claims 21, 27, and 28 are corresponding to claim 1. Therefore, they are rejected for the same rationale for claim 1; and (2) claim 22-26 are rejected for the respective rationale above. Please see the detailed citations and explanations above. The arguments are not persuasive.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BIAO CHEN whose telephone number is (703)756-1199. The examiner can normally be reached M-F 8am-5pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee M Tung can be reached at (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Biao Chen/
Patent Examiner, Art Unit 2611
/KEE M TUNG/Supervisory Patent Examiner, Art Unit 2611
Prosecution Insights