DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of the Claims
Claims 1-20 are currently pending in the present application, with claim 1, 16, and 18 being independent.
Response to Amendments / Arguments
Applicant’s arguments, see Pg. 8, filed 01/26/2026, with respect to the objection to the abstract have been fully considered and are persuasive. The objection to the abstract has been withdrawn.
Applicant's arguments filed 01/26/2026 have been fully considered but they are not persuasive.
Applicant argues: Wright (US 2020/0394012) does not disclose “sending the image and display parameters to a computing device, the display parameters enabling the computing device to simulate a display of the XR wearable device”.
Examiner replies: that Wright expressly discloses transmitting captured video images from the wearable device to a remote computing device together with spatial information describing the camera position and reconstructed environment. Par. 0212-213 discloses “The helper helps a helpee who is on the other side of a digital line, and both are using augmented reality to communicate with each other. The whole system consists basically of a mobile phone or a mobile device on the helpee side so that the helpee is able to point his camera to the environment that he needs help with and share this video stream with the helper…We are sharing not only the video feed with the helper but we also sharing a 3D representation with the helper so that whenever the helper looks at a video frame he will also know where his video frame is relative to the environment as if you image a 3D kind of a 3D representation lying behind any frame that he's receiving”. The transmitted spatial representation includes the 3D position feature points and camera positions, determined using triangulation and SLAM techniques (Wright Par. 0212-0213).
This spatial information enables the remote computing system to understand the viewpoint of the wearable device within the environment and generate annotations relative to that viewpoint. Wright explains that annotations drawn by the remote helper are projected onto the 3D representation of the environment and returned with associated 3D positions so that they can be displayed correctly in the user’s view, with the addition that annotations can be complex point-based markers or three-dimensional annotations (Par. 0023). Thus, Wright teaches “sending the image and display parameters to a computing device, the display parameters enabling the computing device to simulate a display of the XR wearable device”.
Applicant argues: Wright (US 2020/0394012) does not disclose “3D world coordinates”.
Examiner replies: that Wright expressly discloses determining three-dimensional positions of environmental features and the camera relative to the environment using simultaneous localization and mapping (SLAM). For example, Wright Par. 0212-213 discloses “If there are more than four feature points—and usually there are more than four—then we use triangulation to compute the camera's position relative to the environment for each frame. At the same time, we are able to reconstruct these feature positions and the camera position in three-dimensional space. This technique is called simultaneous localization and mapping (SLAM)…This map consists of feature points that we basically find in the image and find in the next image so we can do a triangulation and with that it allows us to have a 3D positional map of the environment”. Additionally, Wright Par. 0220 discloses “3D feature point positions of the environment plus camera positions of images…positions in 3D space”.
These reconstructed feature positions and camera locations define spatial positions within the real-world environment, and such positions, under broadest reasonable interpretation, correspond to 3D coordinates in the real-world environment. Accordingly, Wright teaches “3D world coordinates”.
Applicant argues: Wright (US 2020/0394012) does not disclose “displaying, at a fixed 3D world coordinate location”,” a video stream from the computing device”.
Examiner replies: that Wright expressly discloses feature points detected in successive frames are triangulated to determine both the 3D positions of the environmental features and the camera position in three-dimensional space (Par. 0212-0213), and further explains that the system shares the video feed with this spatial representation so that the video frame is associated with a position within the reconstructed environment “We are sharing not only the video feed with the helper but we also sharing a 3D representation with the helper so that whenever the helper looks at a video frame he will also know where his video frame is relative to the environment” (Par. 0213).
Because the video frame is associated with a position within the reconstructed three-dimensional environment, the video stream is effectively tied to a spatial location defined by the camera’s position relative to the environment. Accordingly, the video stream corresponds to a view originating from a particular location in the three-dimensional environment map generated by the system, and Wright therefore teaches “displaying, at a fixed 3D world coordinate location”,” a video stream from the computing device”.
Applicant’s arguments with respect to claim(s) 5 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Regarding the remaining arguments: Applicant argues with respect to the amended claim language, which is fully addressed in the prior art rejections set forth below.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claim 1, 15-16, and 18 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim 14 and 19 of U.S. Patent No. 12,266,060 (18/046,367) in view of Wright et al. (US 20200394012), hereinafter referred to as “Wright”.
Current Application (18/648,801)
U.S. Patent No. 12,266,060 (18/046,367)
1. An extended realist (XR) wearable device comprising:
at least one processor; and
at least one memory storing instructions that, when executed by the at least one processor, configure the XR wearable device to perform operations comprising:
capturing, by an image capturing device of the XR wearable device, an image corresponding to a user view of a real-world scene;
sending the image and display parameters to a computing device, the display parameters enabling the computing device to simulate a display of the XR wearable device;
receiving, from the computing device, an indication to send a plurality of three-dimensional (3D) world coordinates corresponding to a plurality of positions within the image;
determining the plurality of 3D world coordinates corresponding to the plurality of positions within the image;
sending, to the computing device, the plurality of 3D world coordinates corresponding to the plurality of positions within the image; and
receiving, from the computing device, an indication of an augmentation and 3D world coordinates associated with the augmentation
Although, the claim 1 is an extended reality (XR) device and patent claim 14 is an augmented reality (AR) device, application claim 1 and patent claim 14 are obvious variants. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention that XR devices are inclusive of VR, MR and AR devices, since the plain ordinary meaning of the XR term refers to all technologies that alter reality.
14. An augmented reality (AR) wearable device comprising:
a processor; and
a memory storing instructions that, when executed by the processor, configure the AR wearable device to perform operations comprising:
capturing, by an image capturing device of the AR wearable device, an image corresponding to a first user view of a real-world scene, the image comprising an identification;
sending the image and the identification to a computing device;
receiving, from the computing device, the identification;
determining a plurality of 3-dimensional (3D) coordinates corresponding to a plurality of positions within the image;
sending, to the computing device, the plurality of 3D coordinates corresponding to the plurality of positions within the image; and
receiving, from the computing device, an indication of an augmentation and 3D coordinates associated with the augmentation.
16. A method performed by an extended reality (XR) wearable device, the method comprising:
capturing, by an image capturing device of the XR wearable device, an image corresponding to a user view of a real-world scene;
sending the image and display parameters to a computing device, the display parameters enabling the computing device to simulate a display of the XR wearable device;
receiving, from the computing device, an indication to send a plurality of 3-dimensional (3D) world coordinates corresponding to a plurality of positions within the image;
determining the plurality of 3D world coordinates corresponding to the plurality of positions within the image;
sending, to the computing device, the plurality of 3D world coordinates corresponding to the plurality of positions within the image; and
receiving, from the computing device, an indication of an augmentation and 3D world coordinates associated with the augmentation.
Although, the claim 16 is a method performed by an extended reality (XR)device and patent claim 14 is an augmented reality (AR) device, application claim 16 is method that can be performed by AR device of patent claim 14, since it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention because XR is a term used to refer to all technologies that alter reality.
14. An augmented reality (AR) wearable device comprising:
a processor; and
a memory storing instructions that, when executed by the processor, configure the AR wearable device to perform operations comprising:
capturing, by an image capturing device of the AR wearable device, an image corresponding to a first user view of a real-world scene, the image comprising an identification;
sending the image and the identification to a computing device;
receiving, from the computing device, the identification;
determining a plurality of 3-dimensional (3D) coordinates corresponding to a plurality of positions within the image;
sending, to the computing device, the plurality of 3D coordinates corresponding to the plurality of positions within the image; and
receiving, from the computing device, an indication of an augmentation and 3D coordinates associated with the augmentation.
18. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by an extended reality (XR) wearable device, cause the XR wearable device to perform operations comprising:
capturing, by an image capturing device of the XR wearable device, an image corresponding to a user view of a real-world scene;
sending the image and display parameters to a computing device, the display parameters enabling the computing device to simulate a display of the XR wearable device;
receiving, from the computing device, an indication to send a plurality of 3-dimensional (3D) world coordinates corresponding to a plurality of positions within the image;
determining the plurality of 3D world coordinates corresponding to the plurality of positions within the image;
sending, to the computing device, the plurality of 3D world coordinates corresponding to the plurality of positions within the image; and
receiving, from the computing device, an indication of an augmentation and 3D world coordinates associated with the augmentation.
Claim 18 is a non -transitory computer readable storage medium the store instructions that are executed by an extended reality (XR)device. Patent claim 14 is an augmented reality (AR) device, and fails to specifically disclose a non-transitory CRM. However, it would have been obvious to one of ordinary skill in the art at the before effective filing date of the claimed invention
since storing instructions for execution by XR device on a non-transitory CRM is common practice in the art. Additionally, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, since XR devices are inclusive of VR, MR and AR devices because the plain ordinary meaning of the XR term refers to all technologies that alter reality.
14. An augmented reality (AR) wearable device comprising:
a processor; and
a memory storing instructions that, when executed by the processor, configure the AR wearable device to perform operations comprising:
capturing, by an image capturing device of the AR wearable device, an image corresponding to a first user view of a real-world scene, the image comprising an identification;
sending the image and the identification to a computing device;
receiving, from the computing device, the identification;
determining a plurality of 3-dimensional (3D) coordinates corresponding to a plurality of positions within the image;
sending, to the computing device, the plurality of 3D coordinates corresponding to the plurality of positions within the image; and
receiving, from the computing device, an indication of an augmentation and 3D coordinates associated with the augmentation.
15. The XR wearable device of claim 1 wherein the operations further comprise:
sending the image to a host computing device with an instruction to determine the plurality of 3D world coordinates for the image; and
receiving the plurality of 3D world coordinates from the host computing device
19. The AR wearable device of claim 15 wherein the operations further comprise:
sending the image to a host computing device with an instruction to determine the plurality of 3D coordinates for the image; and receiving the indications of the plurality of 3D coordinates from the host computer.
Although the conflicting claims are not identical, they are not patentably distinct from each other because independent claims 1, 15, 16, and 18 in the current application (18/648,801) are an obvious variant of the claims 14 and 19 in US Patent 18/046,367, and claims 1, 15, 16 and 18 in the current application does not appear to change the scope of the claimed invention.
Claims 1, 16, and 18 in the current application differs from claim 14 in that claims 1, 16, and 18 further recites “the display parameters enabling the computing device to simulate a display of the XR wearable device”. Wright discloses the display parameters enabling the computing device to simulate a display of the XR wearable device (Par. 0212-0213; The helper helps a helpee who is on the other side of a digital line, and both are using augmented reality to communicate with each other. The whole system consists basically of a mobile phone or a mobile device on the helpee side so that the helpee is able to point his camera to the environment that he needs help with and share this video stream with the helper…We are sharing not only the video feed with the helper but we also sharing a 3D representation with the helper so that whenever the helper looks at a video frame he will also know where his video frame is relative to the environment as if you image a 3D kind of a 3D representation lying behind any frame that he's receiving).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the system of claim 14 of US Patent 18/046,367 to include the transmission of display parameters as taught by Wright. Doing so enables a remote computing device to reconstruct and simulate the user’s viewpoint to improve remote interaction and spatial understanding between devices/users, and ensure consistent rendering of the scene across all device. Such modifications are a predictable use of known techniques to improve XR/AR system functionality for stereoscopic display.
Additionally, the recitation of “3D world coordinates” in claims 1, 15, 16, and 18 does not render the claims patentably distinct from claims 14 and 19 of US Patent 18/046,367, which recites “3D coordinates”, because one of ordinary skill in the art would have understood that 3D coordinates in XR/AR systems are defined with respect to a coordinate system, such as a world coordinate system. Accordingly, specifying “world” coordinates merely reflects an obvious designation or interpretation of the coordinates already recited and does not render a patentable distinction.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-4, 6-12 and 14-20 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Wright et al. (US 20200394012), hereinafter referred to as “Wright”.
Regarding claim 1, Wright discloses an extended realist (XR) wearable device comprising (Fig. 1; Local User. Par. 0019 and Par. 0022; shared AR system platform 100…live imagery from a camera that the local user holds or wears, such as…a wearable computing device. Par. 0270; head mounted display (HMD)):
at least one processor (Fig. 2; processing unit 202. Par. 0270; a processor),
and at least one memory storing instructions that, when executed by the at least one processor, configure the XR wearable device to perform operations (Fig. 2; memory 204. Par. 0270; a memory in communication with the processor and having stored thereon computer-executable instruction to cause the processor to…) comprising:
capturing, by an image capturing device of the XR wearable device, an image corresponding to a user view of a real-world scene (Par. 0005; capturing an image with a camera of the device, detecting the objects within the captured image. Par. 0075; the camera of the HMD may be configured to capture images of the field of view of the user),
sending the image and display parameters (Par. 0212-0213; If there are more than four feature points—and usually there are more than four—then we use triangulation to compute the camera's position relative to the environment for each frame. At the same time, we are able to reconstruct these feature positions and the camera position in three-dimensional space. This technique is called simultaneous localization and mapping (SLAM)…This map consists of feature points that we basically find in the image and find in the next image so we can do a triangulation and with that it allows us to have a 3D positional map of the environment) to a computing device, (Fig. 1; Remote User. Par. 0019 and 0023; remote user is able to interact with a model fused from images captured from the surroundings of the local user…mobile device, such as a handheld computer, a tablet, a smartphone), the display parameters enabling the computing device to simulate a display of the XR wearable device (Par. 0212-0213; The helper helps a helpee who is on the other side of a digital line, and both are using augmented reality to communicate with each other. The whole system consists basically of a mobile phone or a mobile device on the helpee side so that the helpee is able to point his camera to the environment that he needs help with and share this video stream with the helper…We are sharing not only the video feed with the helper but we also sharing a 3D representation with the helper so that whenever the helper looks at a video frame he will also know where his video frame is relative to the environment as if you image a 3D kind of a 3D representation lying behind any frame that he's receiving)
receiving, from the computing device, an indication to send a plurality of three-dimensional (3D) world coordinates (Par. 0220; 3D feature point positions of the environment plus camera positions of images…positions in 3D space) corresponding to a plurality of positions within the image (Par. 0096-0099; enable the remote individual to provide annotations to the portions of the environment in which the user is not currently located and/or to guide the user to a desired location within the environment…making an annotation. Par. 0023; Annotations may include point-based markers, more complex three-dimensional annotations, drawings, or live imagery, such as hand gestures. Examiner's note: annotations (that include point-based markers or a desired position) indicates to the HMD to identify and send corresponding 3D coordinates),
determining the plurality of 3D world coordinates (Par. 0220; 3D feature point positions of the environment plus camera positions of images…positions in 3D space) corresponding to the plurality of positions within the image (Par. 0083; starting the camera on the HMD, starting an AR engine to track and reconstruct the environment based on image received from the camera. Par. 0117; each 3D reconstruction of the environment may include one or more significant feature points that allow the user viewing or replaying the help session to identify and/or recognize the environment and track the environment and any movements or annotations within the environment),
sending, to the computing device, the plurality of 3D world coordinates (Par. 0220; 3D feature point positions of the environment plus camera positions of images…positions in 3D space) corresponding to the plurality of positions within the image (Par. 0083; sharing the video and audio feed received by the HMD with the remote individual, and sharing a 3D reconstruction of the environment with the remote individual. Par. 0093-0094; virtual navigation and digital visualization of a remote individual…Par. 0101; SLAM),
and receiving, from the computing device, an indication of an augmentation and 3D world coordinates (Par. 0220; 3D feature point positions of the environment plus camera positions of images…positions in 3D space) associated with the augmentation (Par. 0083; The “AR experience” may further include receiving audio and annotations from the remote individual, and displaying the annotations and playing audio via the speakers of the HMD anchored to objects in the environment tracked by the HMD. Par. 0023; Annotations may include point-based markers, more complex three-dimensional annotations, drawings, or live imagery, such as hand gestures).
Examiner’s note: Additionally, as set forth above in examiner’s response to applicant’s arguments, Par. 0212-0213 of Wright expressly discloses transmitting captured video images from the wearable device to a remote computing device together with spatial information describing the camera position and reconstructed environment. Par. 0212-213 discloses “The helper helps a helpee who is on the other side of a digital line, and both are using augmented reality to communicate with each other. The whole system consists basically of a mobile phone or a mobile device on the helpee side so that the helpee is able to point his camera to the environment that he needs help with and share this video stream with the helper…We are sharing not only the video feed with the helper but we also sharing a 3D representation with the helper so that whenever the helper looks at a video frame he will also know where his video frame is relative to the environment as if you image a 3D kind of a 3D representation lying behind any frame that he's receiving”. The transmitted spatial representation includes the 3D position feature points and camera positions, determined using triangulation and SLAM techniques (Wright Par. 0212-0213). Thus, Wright discloses “sending the image and display parameters to a computing device, the display parameters enabling the computing device to simulate a display of the XR wearable device”. Additionally, Wright Par. 0212-213 and 0220 discloses “If there are more than four feature points—and usually there are more than four—then we use triangulation to compute the camera's position relative to the environment for each frame. At the same time, we are able to reconstruct these feature positions and the camera position in three-dimensional space. This technique is called simultaneous localization and mapping (SLAM)…This map consists of feature points that we basically find in the image and find in the next image so we can do a triangulation and with that it allows us to have a 3D positional map of the environment”, and “3D feature point positions of the environment plus camera positions of images…positions in 3D space”. Accordingly, Wright teaches “3D world coordinates”.
Regarding claim 2, Wright discloses wherein the operations further comprise: receiving an indication from the computing device to send next images (Par. 0049; The communication session may be any type of real-time communication between the user and the remote individual. For example, the user may initiate a shared AR session with the remote individual in which a live video stream or sequence of images captured by a camera of the user's electronic device is transmitted to a remote user's interface (e.g., a remote computer system or other device capable of displaying the video stream or sequence of images). As discussed in detail above, one or more of the local users and the remote user may communicate verbally and/or by annotating objects within the environment. See more details in Par. 0115. Examiner's note: annotations or instructions for updated viewpoints act as an operational indication for the wearable device to capture and transmit a new/updated image set in order to continue the help session).
Regarding claim 3, Wright discloses wherein the operations further comprise:
receiving a video stream from the computing device, the video stream comprising a plurality of images captured by an image capturing device of the computing device (Par. 0227; a 3D recording of the environment plus the annotations of actions that occur. Par. 0266; receiving second AR graphics from the remote party via the remote computer system), and
displaying, at a fixed 3D world coordinates location (Par. 0212-0213), the video stream on a display of the XR wearable device (Par. 0049; a remote user's interface (e.g., a remote computer system or other device capable of displaying the video stream or sequence of images. Par. 0266; displaying the second AR graphics overlaid on the live video stream captured by the camera of the device on a display of the device).
Examiner’s note: Additionally, as set forth above in examiner’s response to applicant’s arguments, Wright expressly discloses feature points detected in successive frames are triangulated to determine both the 3D positions of the environmental features and the camera position in three-dimensional space (Par. 0212-0213), and further explains that the system shares the video feed with this spatial representation so that the video frame is associated with a position within the reconstructed environment “We are sharing not only the video feed with the helper but we also sharing a 3D representation with the helper so that whenever the helper looks at a video frame he will also know where his video frame is relative to the environment” (Par. 0213). Because the video frame is associated with a position within the reconstructed three-dimensional environment, the video stream is effectively tied to a spatial location defined by the camera’s position relative to the environment. Accordingly, the video stream corresponds to a view originating from a particular location in the three-dimensional environment map generated by the system, and Wright therefore teaches “displaying, at a fixed 3D world coordinate location”,” a video stream from the computing device”.
Regarding claim 4, Wright discloses wherein the displaying further comprises:
displaying the video stream in a video insert window, the video insert window positioned near the 3D world coordinates (Par. 0212-0213) of the augmentation (Par. 0022; display visual/spatial feedback from the remote user correctly registered to the real world. Par. 0266-0268; displaying the second AR graphics overlaid on the live video stream captured by the camera of the device on a display of the device…the AR graphics are displayed on the first device in motion relative to the object in the field of view).
Regarding claim 6, Wright discloses wherein the operations further comprise:
in response to a determination that the user view of a user of the XR wearable device comprises the 3D world coordinates associated with the augmentation (Par. 0212-0213), displaying the augmentation on a display of the XR wearable device to appear at a location indicated by the 3D world coordinates (Par. 0117; a user viewing or replaying any help session may use their device to recognize the environment and track the 3D location and timing of any expert annotation that has been drawn along with the voices of both the expert user and any other participants, including the spatial location of any user or expert device relative to the environment (e.g., the spatial location of recording position of the user device and the spatial location of the viewing position of the expert device).
Regarding claim 7, Wright discloses wherein the augmentations is a first augmentation (Par. 0131; initiating a first augmented reality session), and wherein the operations further comprise:
receiving, from a user of the XR wearable device, an indication of a second augmentation (Par. 0131; initiating a second augmented reality session where a second camera captures video of a second object with the camera's field of view; and superimposing the saved augmented reality annotations over the video captured by the second camera during the second augmented reality session),
and determining 3D world coordinates (Par. 0212-0213) for the second augmentation (Par. 0131; … wherein each annotation is superimposed: in spatial relationship to the second object based on the saved spatial relationship, and in temporal relationship to a time within the second augmented reality session based on the saved temporal relationship).
Regarding claim 8, Wright discloses wherein the plurality of 3D world coordinates are 3D world coordinates (Par. 0117; each 3D reconstruction of the environment may include one or more significant feature points that allow the user viewing or replaying the help session to identify and/or recognize the environment and track the environment and any movements or annotations within the environment. Par. 0212-0213) in a frame of reference (Par. 0098 on key frames; the shared representation of the environment may include a 3D reconstruction of the environment as well as so-called key frames (also referred to as virtual navigation “anchors”) including the pose of the camera that has taken those key frames. In certain implementations, a key frame may be a screen shot taken from the images captured by the local user's camera while the local user is navigating the environment. A key frame may also include information relating to the pose, e.g., the relative position and orientation of the camera in the environment when the image of the key frame was captured) of the XR wearable device (local user).
Regarding claim 9, Wright discloses wherein the operations further comprise:
determining a 3D world coordinate of the 3D world coordinates for the second augmentation based on a pixel of the second augmentation being closest, within the image, to the 3D world coordinate of the plurality of 3D world coordinates (Par. 0212-0213 and Par. 0219; the environment map may comprise key frames that are or may be linked together. The key frames may be part of the environment map and may be used to store the map and later used to re-localize the device in the environment. The key frame may be the camera frame that keeps together with the 2D locations of the point plot at 3D depth and the camera frames have also positions. A key point is described by its surrounding pixels or its surrounding).
Regarding claim 10, Wright discloses wherein the plurality of 3D world coordinates is a point cloud of 3D world coordinates (Par. 0178; simultaneous localization and mapping (SLAM) may provide tracking of the user device relative to the environment. This localization and mapping may be used to generate a rough 3D reconstruction of the environment. Accordingly, a rough 3D model of the environment, including many visual features describing the environment uniquely, may be generated. Par. 0237; SLAM with a very sparse point cloud that is generated)
Regarding claim 11, Wright discloses wherein the operations further comprise:
receiving a selection of a user interface item (Par. 0107; The remote individual may then draw a sketch using either a mouse or a touch interface or any other 2D input device. In this way, the remote individual can specify a sequence of 2D locations (describing lines or points) on the screen), the user interface item indicating to move the second augmentation back from the user or towards the user (Par. 0107; Based on the 2D pose of the current input, the 3D pose of the key frame selected by the remote individual and the local user's pose in the environment, the pose (e.g., the position and orientation) of the remote individual's hand can be determined relative to the local user's viewing position. Examiner's note: changing the pose relative to the user necessarily enables forward and backward depth movement),
and determining, based on the 3D world coordinates and the indication to move the second augmentation, new 3D world coordinates for the second augmentation (Par. 0093-0094; virtual navigation and digital visualization of a remote individual…Par. 0101; SLAM. Par. 0212-0213).
Regarding claim 12, Wright discloses wherein the augmentation is a first augmentation, and wherein the operations further comprise:
capturing, by the image capturing device, images corresponding to user views of the real-world scene (Par. 0075; the camera of the HMD may be configured to capture images of the field of view of the user),
processing the images to identify a gesture, the gesture indicating a second augmentation (Par. 0079; the user may be able to navigate the “AR experience” via gesture input detected using the camera of the HMD or via motion detection (e.g., the user may be able to move his/her head as input to the HMD)),
and determining 3D world coordinate (Par. 0212-0213) for the second augmentation (Par. 0083; starting an AR engine to track and reconstruct the environment based on image received from the camera, sharing the video and audio feed received by the HMD with the remote individual, and sharing a 3D reconstruction of the environment with the remote individual).
Regarding claim 14, Wright discloses wherein the augmentation comprises a geometric shape or a line drawn by user input (Par. 0107; The remote individual may then draw a sketch using either a mouse or a touch interface or any other 2D input device…Par. 0159; simultaneous localization and mapping (SLAM) may be used to generate 3D representations of the objects and annotations and track the target object and annotations. In some embodiments, the user may draw abstract circles/outlines and provide verbal comments, such as “do something to this object” while identifying an approximate location or drawing a rough shape of the object).
Regarding claim 15, Wright discloses wherein the operations further comprise:
sending the image to a host computing device with an instruction to determine the plurality of 3D world coordinates for the image (Par. 0033; The user may be able to communicate with a third-party (e.g., a remote user and/or automated electronic system) to receive assistance with any issues the user may be experiencing with the object(s) to diagnose the source of the issues and/or provide instructions to the individual to aid in the interaction with the object. Par. 0129; the user may initiate the AR or VR session by capturing a video of the object in the user's environment using the user device (e.g., camera). The video may be shared with the expert. Par. 0212-0213),
and receiving the plurality of 3D world coordinates from the host computing device (Par. 0115; The processing unit 202 or other component of either device may further include a sparse environment reconstruction method that allows the devices to share a “rough” or “coarse” 3D representation of the user's environment with the other user. Par. 0212-0213).
Regarding claim 16, claim 16 is the method claim of system claim 1 and is accordingly rejected using substantially similar rationale as to that which is set for with respect to claim 1.
Regarding claim 17, claim 17 has similar limitations as of claim 2, except it is a method claim, therefore it is rejected under the same rationale as claim 2.
Regarding claim 18, claim 18 is the CRM claim (Par. 00226-0027; computer storage) of system claim 1 and is accordingly rejected using substantially similar rationale as to that which is set for with respect to claim 1.
Regarding claim 19, claim 19 has similar limitations as of claim 2, except it is a CRM claim (Par. 00226-0027; computer storage), therefore it is rejected under the same rationale as claim 2.
Regarding claim 20, claim 20 has similar limitations as of claim 3, except it is a CRM claim (Par. 00226-0027; computer storage), therefore it is rejected under the same rationale as claim 3.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wright et al. (US 20200394012), hereinafter referred to as “Wright”, in view of Schuck et al. (US 20220317459).
Regarding claim 5, Wright discloses wherein the displaying further comprises:
displaying the video stream in a video insert window, the video insert window positioned at a far field location or a near field location of a user of the wearable device (Par. 0101; system can provide a spatial frame of reference and understanding of the environment's spatial layout…corresponding perspective. Par. 0142-0153; When the annotation is communicated from the user to another user, the annotation may include information about the annotation (e.g., 3D object tracking information and the annotation identifier) and/or information about the object annotated…such features or parts may be displayed in a “window” or other frame of the object that indicates that they are internal to the object).
Wright does not disclose the far field location and the near field location being at a distance from a user of the XR wearable device where both eyes of the user can resolve a menu item into a single overlapping range.
In the same art of stereoscopic display, Schuck discloses disclose the far field location and the near field location being at a distance from a user of the XR wearable device where both eyes of the user can resolve a menu item into a single overlapping range (Fig. 8A-8F and Par. 0024; The overlap of two virtual images generated by a respective display as seen by the user’s two eyes…binocular overlap of the two virtual images depends on the location of the accommodation plane and the depth of the content formed…).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to render Wright’s augmented annotations using Schuck’s stereoscopic head-mounted display system, which positions virtual objects at a depth where images presented to each eye overlap to produce a single perceived view. Doing so would improve visual comfort and depth perception when displaying augmentations in an XR environment (Shuck Par. 0024; By designing the optical system to place both the virtual image focus and the location of 100% binocular overlap at a depth in space where the virtual content is most likely to occur, the user comfort is improved. Par. 0069; Diffractive structures which place the respective virtual image 715 at infinity can have benefits, such as simpler design and manufacturing, wider field of view (FOV), better pupil replication, higher efficiency and improved color uniformity).
Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wright et al. (US 20200394012), hereinafter referred to as “Wright”, in view of Berliner et al. (US 20230139626), hereinafter referred to as “Berliner”.
Regarding claim 13, Wright discloses the XR wearable device of claim 1, but does not disclose wherein the augmentation is displayed using a first brightness, and wherein the operations further comprise:
displaying, after a predetermined duration, the augmentation at a second brightness, wherein the second brightness is less than the first brightness
In the same art of augmented and mixed reality, Berliner discloses wherein the augmentation is displayed using a first brightness, and wherein the operations further comprise:
displaying, after a predetermined duration, the augmentation at a second brightness, wherein the second brightness is less than the first brightness (Par. 0150; during a first time duration, the at least one processor may perform a first PWM procedure to increase the duty cycle from 50% to 75% to increase the brightness of content displayed via a wearable extended reality appliance. During a second time duration, the at least one processor may perform a second PWM procedure to decrease the duty cycle from 75% to 50% to dim the display of content via the wearable extended reality appliance. As another example, the at least one processor controlling the display of content via a wearable extended reality appliance may perform a PWM procedure to display incoming messages of a messaging application according to a 75% duty cycle, e.g., to draw the attention of the user, and display a weather application according to a 50% duty cycle, e.g., as a background application)
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Wright’s augmented reality system to include temporally adjusting the brightness of the augmentation as taught by Berliner. The motivation lies in the advantage of minimizing visual distraction and improve user experience. Temporally modifying display brightness is a well-known technique for conserving battery life and reducing display clutter for an application that augments content such as Wright’s teachings, yielding predicting results to improve display-based content.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JENNY NGAN TRAN whose telephone number is (571)272-6888. The examiner can normally be reached Mon-Thurs 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alicia Harrington can be reached at (571) 272-2330. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JENNY N TRAN/Examiner, Art Unit 2615
/ALICIA M HARRINGTON/Supervisory Patent Examiner, Art Unit 2615