DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of the Claims
Claims 1-20 are currently pending in the present application, with claim 1, 16, and 18 being independent.
Response to Amendments / Arguments
Applicant's arguments, see Pg. 8, filed 04/27/2026, have been fully considered but they are not persuasive.
Applicant argues: Wright does not disclose “displaying, at a fixed 3D world coordinates location, the video stream on the display of the XR wearable device,” and further states that Wright does not disclose the opposite direction in which the XR wearable device receives a video stream from the remote computing device and displays that video stream at a fixed 3D world coordinate location on the XR wearable device’s display.
Examiner replies: Claim 3 broadly recites receiving a video stream from a computing device and displaying the video stream at a fixed three-dimensional (3D) world coordinate’s location, and the claim does not require any particular source of the video stream beyond the recited computing device, nor does it require any specific communication protocol or distinguish between locally captured and remotely generated imagery.
Wright expressly discloses reconstructing a three0dimensional representation of the environment using feature points and camera positions determined through SLAM, sharing the video feed together with the reconstructed three-dimensional representation, and presenting annotations anchored relative to that reconstructed environment. Wright discloses that “we are sharing not only the video feed with the helper but also sharing a 3D representation with the helper so that whenever the helper looks at a video frame he will also know where the video frame is relative to the environment” (Par. 0213). Wright further discloses determining camera positions and feature positions in three-dimensional space using triangulation and generating a three-dimensional positional map of the environment (Par. 0212), and communicating annotations associated with positions within that three-dimensional environment (Par. 0220, 0023).
Accordingly, Wright teaches associating displayed content with fixed positions in the reconstructed three-dimensional world coordinate system rather than merely displaying the content in screen coordinates. Under broadest reasonable interpretation, displaying the received video stream at a fixed 3D world coordinates location reads on displaying the received stream in association with the reconstructed world-space location maintained by Wright’s SLAM environment. The claim does not require any particular origin of the displayed stream beyond being received from the computing device in regards to applicant’s argument of the direction of the video stream. Therefore, Wright discloses “displaying, at a fixed 3D world coordinates location, the video stream on the display of the XR wearable device.”
Applicant’s arguments, see Pg. 8, filed 04/27/2026, with respect to the rejection(s) of claim(s) 5 under 35 U.S.C. § 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of newly found prior art.
Regarding the remaining arguments: Applicant argues with respect to the amended claim language, which is fully addressed in the prior art rejections set forth below.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claim 1, 15-16, and 18 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim 14 and 19 of U.S. Patent No. 12,266,060 (18/046,367) in view of Forutanpour et al. (US 20230144091), hereinafter referred to as “Forutanpour”.
Current Application (18/648,801)
U.S. Patent No. 12,266,060 (18/046,367)
1. An extended realist (XR) wearable device comprising:
at least one processor; and
at least one memory storing instructions that, when executed by the at least one processor, configure the XR wearable device to perform operations comprising:
capturing, by an image capturing device of the XR wearable device, an image corresponding to a user view of a real-world scene;
sending the image and display parameters to a computing device, comprising at least one of an aspect ratio or a screen resolution of a display of the XR wearable device, the display parameters enabling the computing device to simulate a display of the XR wearable device on the computing device;
receiving, from the computing device, an indication to send a plurality of three-dimensional (3D) world coordinates corresponding to a plurality of positions within the image;
determining the plurality of 3D world coordinates corresponding to the plurality of positions within the image;
sending, to the computing device, the plurality of 3D world coordinates corresponding to the plurality of positions within the image; and
receiving, from the computing device, an indication of an augmentation and 3D world coordinates associated with the augmentation
Although, the claim 1 is an extended reality (XR) device and patent claim 14 is an augmented reality (AR) device, application claim 1 and patent claim 14 are obvious variants. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention that XR devices are inclusive of VR, MR and AR devices, since the plain ordinary meaning of the XR term refers to all technologies that alter reality.
14. An augmented reality (AR) wearable device comprising:
a processor; and
a memory storing instructions that, when executed by the processor, configure the AR wearable device to perform operations comprising:
capturing, by an image capturing device of the AR wearable device, an image corresponding to a first user view of a real-world scene, the image comprising an identification;
sending the image and the identification to a computing device;
receiving, from the computing device, the identification;
determining a plurality of 3-dimensional (3D) coordinates corresponding to a plurality of positions within the image;
sending, to the computing device, the plurality of 3D coordinates corresponding to the plurality of positions within the image; and
receiving, from the computing device, an indication of an augmentation and 3D coordinates associated with the augmentation.
16. A method performed by an extended reality (XR) wearable device, the method comprising:
capturing, by an image capturing device of the XR wearable device, an image corresponding to a user view of a real-world scene;
sending the image and display parameters to a computing device, the display parameters comprising at least one of an aspect ratio or a screen resolution of a display of the XR wearable device, the display parameters enabling the computing device to simulate the display of the XR wearable device on the computing device;
receiving, from the computing device, an indication to send a plurality of 3-dimensional (3D) world coordinates corresponding to a plurality of positions within the image;
determining the plurality of 3D world coordinates corresponding to the plurality of positions within the image;
sending, to the computing device, the plurality of 3D world coordinates corresponding to the plurality of positions within the image; and
receiving, from the computing device, an indication of an augmentation and 3D world coordinates associated with the augmentation.
Although, the claim 16 is a method performed by an extended reality (XR)device and patent claim 14 is an augmented reality (AR) device, application claim 16 is method that can be performed by AR device of patent claim 14, since it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention because XR is a term used to refer to all technologies that alter reality.
14. An augmented reality (AR) wearable device comprising:
a processor; and
a memory storing instructions that, when executed by the processor, configure the AR wearable device to perform operations comprising:
capturing, by an image capturing device of the AR wearable device, an image corresponding to a first user view of a real-world scene, the image comprising an identification;
sending the image and the identification to a computing device;
receiving, from the computing device, the identification;
determining a plurality of 3-dimensional (3D) coordinates corresponding to a plurality of positions within the image;
sending, to the computing device, the plurality of 3D coordinates corresponding to the plurality of positions within the image; and
receiving, from the computing device, an indication of an augmentation and 3D coordinates associated with the augmentation.
18. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by an extended reality (XR) wearable device, cause the XR wearable device to perform operations comprising:
capturing, by an image capturing device of the XR wearable device, an image corresponding to a user view of a real-world scene;
sending the image and display parameters to a computing device, the display parameters comprising at least one of an aspect ratio or a screen resolution of a display of the XR wearable device, the display parameters enabling the computing device to simulate the display of the XR wearable device on the computing device;
receiving, from the computing device, an indication to send a plurality of 3-dimensional (3D) world coordinates corresponding to a plurality of positions within the image;
determining the plurality of 3D world coordinates corresponding to the plurality of positions within the image;
sending, to the computing device, the plurality of 3D world coordinates corresponding to the plurality of positions within the image; and
receiving, from the computing device, an indication of an augmentation and 3D world coordinates associated with the augmentation.
Claim 18 is a non -transitory computer readable storage medium the store instructions that are executed by an extended reality (XR)device. Patent claim 14 is an augmented reality (AR) device, and fails to specifically disclose a non-transitory CRM. However, it would have been obvious to one of ordinary skill in the art at the before effective filing date of the claimed invention
since storing instructions for execution by XR device on a non-transitory CRM is common practice in the art. Additionally, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, since XR devices are inclusive of VR, MR and AR devices because the plain ordinary meaning of the XR term refers to all technologies that alter reality.
14. An augmented reality (AR) wearable device comprising:
a processor; and
a memory storing instructions that, when executed by the processor, configure the AR wearable device to perform operations comprising:
capturing, by an image capturing device of the AR wearable device, an image corresponding to a first user view of a real-world scene, the image comprising an identification;
sending the image and the identification to a computing device;
receiving, from the computing device, the identification;
determining a plurality of 3-dimensional (3D) coordinates corresponding to a plurality of positions within the image;
sending, to the computing device, the plurality of 3D coordinates corresponding to the plurality of positions within the image; and
receiving, from the computing device, an indication of an augmentation and 3D coordinates associated with the augmentation.
15. The XR wearable device of claim 1 wherein the operations further comprise:
sending the image to a host computing device with an instruction to determine the plurality of 3D world coordinates for the image; and
receiving the plurality of 3D world coordinates from the host computing device
19. The AR wearable device of claim 15 wherein the operations further comprise:
sending the image to a host computing device with an instruction to determine the plurality of 3D coordinates for the image; and receiving the indications of the plurality of 3D coordinates from the host computer.
Although the conflicting claims are not identical, they are not patentably distinct from each other because independent claims 1, 15, 16, and 18 in the current application (18/648,801) are an obvious variant of the claims 14 and 19 in US Patent 18/046,367, and claims 1, 15, 16 and 18 in the current application does not appear to change the scope of the claimed invention.
Claims 1, 16, and 18 in the current application differs from claim 14 in that claims 1, 16, and 18 further recites “the display parameters comprising at least one of an aspect ratio or a screen resolution of a display of the XR wearable device, the display parameters enabling the computing device to simulate the display of the XR wearable device on the computing device”. Forutanpour discloses sending the image and display parameters to a computing device (Fig. 2, 8 and Par. 0005-0016; The imaging system can determine, based on the level of perception of the virtual content by the user and/or the level of comprehension of the virtual content by the user, a modification to the display settings corresponding to the virtual content. The modification to the display settings, can, for example, hide, remove, shrink, reduce prominence of, reduce priority of, or terminate display of the virtual content. The modification to the display settings, can, for example, emphasize, enlarge, move, reorient, increase prominence of, and/or increase priority of, the virtual content…causing virtual content to be displayed using a display according to display settings associated with the virtual content, wherein an environment is viewable using the display as the virtual content is displayed by the display; determining, based on one or more perception-related attributes of a user, a level of perception of the virtual content that is displayed using the display by the user; and determining, based on the level of perception of the virtual content by the user, a modification to the display settings corresponding to the virtual content…), comprising at least one of an aspect ratio or a screen resolution of a display of the XR wearable device (Par. 0143; the XR system can monitor other external factors, such as user input, display duration(s) of previously-presented virtual content, device status (e.g., power status, such as low power), AR device screen resolution, traveling speed of the XR system (and thus the user), and/or other external factors), the display parameters enabling the computing device to simulate the display of the XR wearable device (Par. 002; An extended reality (XR) device is a device that displays an environment to a user, for example through a head-mounted display (HMD) or other device) on the computing device (Par. 0049-0052).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the system of claim 14 of US Patent 18/046,367 to include the transmission of display parameters as taught by Forutanpour. Doing so allows the remote computing device to present the received image in a manner consistent with the display characteristics of the XR wearable device, predictably improving the accuracy and consistency of the remote user’s view of the shared XR environment in remote collaboration sessions. Such modifications are a predictable use of known techniques to improve XR/AR system functionality for stereoscopic display.
The recitation of “3D world coordinates” in claims 1, 15, 16, and 18 does not render the claims patentably distinct from claims 14 and 19 of US Patent 18/046,367, which recites “3D coordinates”, because one of ordinary skill in the art would have understood that 3D coordinates in XR/AR systems are defined with respect to a coordinate system, such as a world coordinate system. Accordingly, specifying “world” coordinates merely reflects an obvious designation or interpretation of the coordinates already recited and does not render a patentable distinction.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-4, 6-12 and 14-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wright et al. (US 20200394012), hereinafter referred to as “Wright”, in view of Forutanpour et al. (US 20230144091), hereinafter referred to as “Forutanpour”.
Regarding claim 1, Wright discloses an extended realist (XR) wearable device comprising (Fig. 1; Local User. Par. 0019 and Par. 0022; shared AR system platform 100…live imagery from a camera that the local user holds or wears, such as…a wearable computing device. Par. 0270; head mounted display (HMD)):
at least one processor (Fig. 2; processing unit 202. Par. 0270; a processor),
and at least one memory storing instructions that, when executed by the at least one processor, configure the XR wearable device to perform operations (Fig. 2; memory 204. Par. 0270; a memory in communication with the processor and having stored thereon computer-executable instruction to cause the processor to…) comprising:
capturing, by an image capturing device of the XR wearable device, an image corresponding to a user view of a real-world scene (Par. 0005; capturing an image with a camera of the device, detecting the objects within the captured image. Par. 0075; the camera of the HMD may be configured to capture images of the field of view of the user),
receiving, from the computing device, an indication to send a plurality of three-dimensional (3D) world coordinates (Par. 0220; 3D feature point positions of the environment plus camera positions of images…positions in 3D space) corresponding to a plurality of positions within the image (Par. 0096-0099; enable the remote individual to provide annotations to the portions of the environment in which the user is not currently located and/or to guide the user to a desired location within the environment…making an annotation. Par. 0023; Annotations may include point-based markers, more complex three-dimensional annotations, drawings, or live imagery, such as hand gestures. Examiner's note: annotations (that include point-based markers or a desired position) indicates to the HMD to identify and send corresponding 3D coordinates),
determining the plurality of 3D world coordinates (Par. 0220; 3D feature point positions of the environment plus camera positions of images…positions in 3D space) corresponding to the plurality of positions within the image (Par. 0083; starting the camera on the HMD, starting an AR engine to track and reconstruct the environment based on image received from the camera. Par. 0117; each 3D reconstruction of the environment may include one or more significant feature points that allow the user viewing or replaying the help session to identify and/or recognize the environment and track the environment and any movements or annotations within the environment),
sending, to the computing device, the plurality of 3D world coordinates (Par. 0220; 3D feature point positions of the environment plus camera positions of images…positions in 3D space) corresponding to the plurality of positions within the image (Par. 0083; sharing the video and audio feed received by the HMD with the remote individual, and sharing a 3D reconstruction of the environment with the remote individual. Par. 0093-0094; virtual navigation and digital visualization of a remote individual…Par. 0101; SLAM),
and receiving, from the computing device, an indication of an augmentation and 3D world coordinates (Par. 0220; 3D feature point positions of the environment plus camera positions of images…positions in 3D space) associated with the augmentation (Par. 0083; The “AR experience” may further include receiving audio and annotations from the remote individual, and displaying the annotations and playing audio via the speakers of the HMD anchored to objects in the environment tracked by the HMD. Par. 0023; Annotations may include point-based markers, more complex three-dimensional annotations, drawings, or live imagery, such as hand gestures).
Wright does not appear to explicitly disclose sending the image and display parameters to a computing device, comprising at least one of an aspect ratio or a screen resolution of a display of the XR wearable device, the display parameters enabling the computing device to simulate the display of the XR wearable device on the computing device.
In the same art of displaying virtual content on XR systems, Forutanpour discloses sending the image and display parameters to a computing device (Fig. 2, 8 and Par. 0005-0016; The imaging system can determine, based on the level of perception of the virtual content by the user and/or the level of comprehension of the virtual content by the user, a modification to the display settings corresponding to the virtual content. The modification to the display settings, can, for example, hide, remove, shrink, reduce prominence of, reduce priority of, or terminate display of the virtual content. The modification to the display settings, can, for example, emphasize, enlarge, move, reorient, increase prominence of, and/or increase priority of, the virtual content…causing virtual content to be displayed using a display according to display settings associated with the virtual content, wherein an environment is viewable using the display as the virtual content is displayed by the display; determining, based on one or more perception-related attributes of a user, a level of perception of the virtual content that is displayed using the display by the user; and determining, based on the level of perception of the virtual content by the user, a modification to the display settings corresponding to the virtual content…), comprising at least one of an aspect ratio or a screen resolution of a display of the XR wearable device (Par. 0143; the XR system can monitor other external factors, such as user input, display duration(s) of previously-presented virtual content, device status (e.g., power status, such as low power), AR device screen resolution, traveling speed of the XR system (and thus the user), and/or other external factors), the display parameters enabling the computing device to simulate the display of the XR wearable device (Par. 002; An extended reality (XR) device is a device that displays an environment to a user, for example through a head-mounted display (HMD) or other device) on the computing device (Par. 0049-0052).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Wright’s XR collaboration system to additionally transmit display parameters, such as display settings, as taught by Forutanpour. Doing so allows the remote computing device to present the received image in a manner consistent with the display characteristics of the XR wearable device, predictably improving the accuracy and consistency of the remote user’s view of the shared XR environment in remote collaboration sessions.
Regarding claim 2, Wright in view of Forutanpour discloses the XR wearable device of claim 1, and Wright further discloses wherein the operations further comprise: receiving an indication from the computing device to send next images (Par. 0049; The communication session may be any type of real-time communication between the user and the remote individual. For example, the user may initiate a shared AR session with the remote individual in which a live video stream or sequence of images captured by a camera of the user's electronic device is transmitted to a remote user's interface (e.g., a remote computer system or other device capable of displaying the video stream or sequence of images). As discussed in detail above, one or more of the local users and the remote user may communicate verbally and/or by annotating objects within the environment. See more details in Par. 0115. Examiner's note: annotations or instructions for updated viewpoints act as an operational indication for the wearable device to capture and transmit a new/updated image set in order to continue the help session).
Wright and Forutanpour are combined for the reasons set forth above with respect to claim 1.
Regarding claim 3, Wright in view of Forutanpour discloses the XR wearable device of claim 1, and Wright further discloses wherein the operations further comprise:
receiving a video stream from the computing device, the video stream comprising a plurality of images captured by an image capturing device of the computing device (Par. 0227; a 3D recording of the environment plus the annotations of actions that occur. Par. 0266; receiving second AR graphics from the remote party via the remote computer system), and
displaying, at a fixed 3D world coordinates location (Par. 0212-0213), the video stream on a display of the XR wearable device (Par. 0049; a remote user's interface (e.g., a remote computer system or other device capable of displaying the video stream or sequence of images. Par. 0266; displaying the second AR graphics overlaid on the live video stream captured by the camera of the device on a display of the device).
Wright and Forutanpour are combined for the reasons set forth above with respect to claim 1.
Examiner’s note: As mentioned above in examiner’s response to arguments, claim 3 broadly recites receiving a video stream from a computing device and displaying the video stream at a fixed three-dimensional (3D) world coordinate’s location, and the claim does not require any particular source of the video stream beyond the recited computing device, nor does it require any specific communication protocol or distinguish between locally captured and remotely generated imagery.
Wright teaches associating displayed content with fixed positions in the reconstructed three-dimensional world coordinate system rather than merely displaying the content in screen coordinates (Par. 0212-213, 0220, and 0023). Under broadest reasonable interpretation, displaying the received video stream at a fixed 3D world coordinates location reads on displaying the received stream in association with the reconstructed world-space location maintained by Wright’s SLAM environment. The claim does not require any particular origin of the displayed stream beyond being received from the computing device in regards to applicant’s argument of the direction of the video stream. Therefore, Wright discloses “displaying, at a fixed 3D world coordinates location, the video stream on the display of the XR wearable device.”
Regarding claim 4, Wright in view of Forutanpour discloses the XR wearable device of claim 3, and Wright further discloses wherein the displaying further comprises:
displaying the video stream in a video insert window, the video insert window positioned near the 3D world coordinates (Par. 0212-0213) of the augmentation (Par. 0022; display visual/spatial feedback from the remote user correctly registered to the real world. Par. 0266-0268; displaying the second AR graphics overlaid on the live video stream captured by the camera of the device on a display of the device…the AR graphics are displayed on the first device in motion relative to the object in the field of view).
Wright and Forutanpour are combined for the reasons set forth above with respect to claim 1.
Regarding claim 6, Wright in view of Forutanpour discloses the XR wearable device of claim 1, and Wright further discloses wherein the operations further comprise:
in response to a determination that the user view of a user of the XR wearable device comprises the 3D world coordinates associated with the augmentation (Par. 0212-0213), displaying the augmentation on a display of the XR wearable device to appear at a location indicated by the 3D world coordinates (Par. 0117; a user viewing or replaying any help session may use their device to recognize the environment and track the 3D location and timing of any expert annotation that has been drawn along with the voices of both the expert user and any other participants, including the spatial location of any user or expert device relative to the environment (e.g., the spatial location of recording position of the user device and the spatial location of the viewing position of the expert device).
Wright and Forutanpour are combined for the reasons set forth above with respect to claim 1.
Regarding claim 7, Wright in view of Forutanpour discloses the XR wearable device of claim 1, and Wright further discloses wherein the augmentations is a first augmentation (Par. 0131; initiating a first augmented reality session), and wherein the operations further comprise:
receiving, from a user of the XR wearable device, an indication of a second augmentation (Par. 0131; initiating a second augmented reality session where a second camera captures video of a second object with the camera's field of view; and superimposing the saved augmented reality annotations over the video captured by the second camera during the second augmented reality session),
and determining 3D world coordinates (Par. 0212-0213) for the second augmentation (Par. 0131; … wherein each annotation is superimposed: in spatial relationship to the second object based on the saved spatial relationship, and in temporal relationship to a time within the second augmented reality session based on the saved temporal relationship).
Wright and Forutanpour are combined for the reasons set forth above with respect to claim 1.
Regarding claim 8, Wright in view of Forutanpour discloses the XR wearable device of claim 7, and Wright further discloses wherein the plurality of 3D world coordinates are coordinates (Par. 0117; each 3D reconstruction of the environment may include one or more significant feature points that allow the user viewing or replaying the help session to identify and/or recognize the environment and track the environment and any movements or annotations within the environment. Par. 0212-0213) in a frame of reference (Par. 0098 on key frames; the shared representation of the environment may include a 3D reconstruction of the environment as well as so-called key frames (also referred to as virtual navigation “anchors”) including the pose of the camera that has taken those key frames. In certain implementations, a key frame may be a screen shot taken from the images captured by the local user's camera while the local user is navigating the environment. A key frame may also include information relating to the pose, e.g., the relative position and orientation of the camera in the environment when the image of the key frame was captured) of the XR wearable device (local user).
Wright and Forutanpour are combined for the reasons set forth above with respect to claim 1.
Regarding claim 9, Wright in view of Forutanpour discloses the XR wearable device of claim 8, and Wright further discloses wherein the operations further comprise:
determining a 3D world coordinate of the 3D world coordinates for the second augmentation based on a pixel of the second augmentation being closest, within the image, to the 3D world coordinate of the plurality of 3D world coordinates (Par. 0212-0213 and Par. 0219; the environment map may comprise key frames that are or may be linked together. The key frames may be part of the environment map and may be used to store the map and later used to re-localize the device in the environment. The key frame may be the camera frame that keeps together with the 2D locations of the point plot at 3D depth and the camera frames have also positions. A key point is described by its surrounding pixels or its surrounding).
Wright and Forutanpour are combined for the reasons set forth above with respect to claim 1.
Regarding claim 10, Wright in view of Forutanpour discloses the XR wearable device of claim 9, and Wright further discloses wherein the plurality of 3D world coordinates is a point cloud of 3D world coordinates (Par. 0178; simultaneous localization and mapping (SLAM) may provide tracking of the user device relative to the environment. This localization and mapping may be used to generate a rough 3D reconstruction of the environment. Accordingly, a rough 3D model of the environment, including many visual features describing the environment uniquely, may be generated. Par. 0237; SLAM with a very sparse point cloud that is generated)
Wright and Forutanpour are combined for the reasons set forth above with respect to claim 1.
Regarding claim 11, Wright in view of Forutanpour discloses the XR wearable device of claim 7, and Wright further discloses wherein the operations further comprise:
receiving a selection of a user interface item (Par. 0107; The remote individual may then draw a sketch using either a mouse or a touch interface or any other 2D input device. In this way, the remote individual can specify a sequence of 2D locations (describing lines or points) on the screen), the user interface item indicating to move the second augmentation back from the user or towards the user (Par. 0107; Based on the 2D pose of the current input, the 3D pose of the key frame selected by the remote individual and the local user's pose in the environment, the pose (e.g., the position and orientation) of the remote individual's hand can be determined relative to the local user's viewing position. Examiner's note: changing the pose relative to the user necessarily enables forward and backward depth movement),
and determining, based on the 3D world coordinates and the indication to move the second augmentation, new 3D world coordinates for the second augmentation (Par. 0093-0094; virtual navigation and digital visualization of a remote individual…Par. 0101; SLAM. Par. 0212-0213).
Wright and Forutanpour are combined for the reasons set forth above with respect to claim 1.
Regarding claim 12, Wright in view of Forutanpour discloses the XR wearable device of claim 1, and Wright further discloses wherein the augmentation is a first augmentation, and wherein the operations further comprise:
capturing, by the image capturing device, images corresponding to user views of the real-world scene (Par. 0075; the camera of the HMD may be configured to capture images of the field of view of the user),
processing the images to identify a gesture, the gesture indicating a second augmentation (Par. 0079; the user may be able to navigate the “AR experience” via gesture input detected using the camera of the HMD or via motion detection (e.g., the user may be able to move his/her head as input to the HMD)),
and determining 3D world coordinate (Par. 0212-0213) for the second augmentation (Par. 0083; starting an AR engine to track and reconstruct the environment based on image received from the camera, sharing the video and audio feed received by the HMD with the remote individual, and sharing a 3D reconstruction of the environment with the remote individual).
Wright and Forutanpour are combined for the reasons set forth above with respect to claim 1.
Regarding claim 14, Wright in view of Forutanpour discloses the XR wearable device of claim 1, and Wright further discloses wherein the augmentation comprises a geometric shape or a line drawn by user input (Par. 0107; The remote individual may then draw a sketch using either a mouse or a touch interface or any other 2D input device…Par. 0159; simultaneous localization and mapping (SLAM) may be used to generate 3D representations of the objects and annotations and track the target object and annotations. In some embodiments, the user may draw abstract circles/outlines and provide verbal comments, such as “do something to this object” while identifying an approximate location or drawing a rough shape of the object).
Wright and Forutanpour are combined for the reasons set forth above with respect to claim 1.
Regarding claim 15, Wright in view of Forutanpour discloses the XR wearable device of claim 1, and Wright further discloses wherein the operations further comprise:
sending the image to a host computing device with an instruction to determine the plurality of 3D world coordinates for the image (Par. 0033; The user may be able to communicate with a third-party (e.g., a remote user and/or automated electronic system) to receive assistance with any issues the user may be experiencing with the object(s) to diagnose the source of the issues and/or provide instructions to the individual to aid in the interaction with the object. Par. 0129; the user may initiate the AR or VR session by capturing a video of the object in the user's environment using the user device (e.g., camera). The video may be shared with the expert. Par. 0212-0213),
and receiving the plurality of 3D world coordinates from the host computing device (Par. 0115; The processing unit 202 or other component of either device may further include a sparse environment reconstruction method that allows the devices to share a “rough” or “coarse” 3D representation of the user's environment with the other user. Par. 0212-0213).
Wright and Forutanpour are combined for the reasons set forth above with respect to claim 1.
Regarding claim 16, claim 16 is the method claim of system claim 1 and is accordingly rejected using substantially similar rationale as to that which is set for with respect to claim 1.
Regarding claim 17, claim 17 has similar limitations as of claim 2, except it is a method claim, therefore it is rejected under the same rationale as claim 2.
Regarding claim 18, claim 18 is the CRM claim (Par. 00226-0027; computer storage) of system claim 1 and is accordingly rejected using substantially similar rationale as to that which is set for with respect to claim 1.
Regarding claim 19, claim 19 has similar limitations as of claim 2, except it is a CRM claim (Par. 00226-0027; computer storage), therefore it is rejected under the same rationale as claim 2.
Regarding claim 20, claim 20 has similar limitations as of claim 3, except it is a CRM claim (Par. 00226-0027; computer storage), therefore it is rejected under the same rationale as claim 3.
Claim(s) 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wright et al. (US 20200394012), hereinafter referred to as “Wright”, in view of Forutanpour et al. (US 20230144091), hereinafter referred to as “Forutanpour”, and in further view of Blaha et al. (US 9706910 B1).
Regarding claim 5, Wright in view of Forutanpour discloses the XR wearable device of claim 3, and Wright further discloses wherein the displaying further comprises:
displaying the video stream in a video insert window, the video insert window positioned at a far field location or a near field location of a user of the wearable device (Par. 0101; system can provide a spatial frame of reference and understanding of the environment's spatial layout…corresponding perspective. Par. 0142-0153; When the annotation is communicated from the user to another user, the annotation may include information about the annotation (e.g., 3D object tracking information and the annotation identifier) and/or information about the object annotated…such features or parts may be displayed in a “window” or other frame of the object that indicates that they are internal to the object).
Wright in view of Forutanpour does not disclose the far field location and the near field location being at a distance from a user of the XR wearable device where both eyes of the user can resolve a menu item into a single overlapping range.
In the same art of head-mounted VR display/devices, Blaha discloses disclose the far field location and the near field location being at a distance from a user of the XR wearable device where both eyes of the user can resolve a menu item into a single overlapping range (Fig. 13A-13B and Col. 25-27; The activity can be repeated until the acquired user input indicates that the user seeing a double of the displayed object, which is the outer limit of the user's fusional range. The disparity can then be reduced until the user indicates that the target object is blurry, then until the user indicates that the target object is perceived as a single object again…measuring fusional ranges…nearsighted acuity…farsighted acuity)
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to position a video insert window in the combined system of Wright and Forutanpour at a near-field or far-field depth within a user’s fusional range, as taught by Blaha. Doing so allows the corresponding left-eye and right-eye representations of the window and its menu items to be perceived as a single image, thereby improving readability and reducing visual discomfort.
Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wright et al. (US 20200394012), hereinafter referred to as “Wright”, in view of Forutanpour et al. (US 20230144091), hereinafter referred to as “Forutanpour”, and in further view of Berliner et al. (US 20230139626), hereinafter referred to as “Berliner”.
Regarding claim 13, Wright in view of Forutanpour discloses the XR wearable device of claim 1, but does not disclose wherein the augmentation is displayed using a first brightness, and wherein the operations further comprise:
displaying, after a predetermined duration, the augmentation at a second brightness, wherein the second brightness is less than the first brightness
In the same art of augmented and mixed reality, Berliner discloses wherein the augmentation is displayed using a first brightness, and wherein the operations further comprise:
displaying, after a predetermined duration, the augmentation at a second brightness, wherein the second brightness is less than the first brightness (Par. 0150; during a first time duration, the at least one processor may perform a first PWM procedure to increase the duty cycle from 50% to 75% to increase the brightness of content displayed via a wearable extended reality appliance. During a second time duration, the at least one processor may perform a second PWM procedure to decrease the duty cycle from 75% to 50% to dim the display of content via the wearable extended reality appliance. As another example, the at least one processor controlling the display of content via a wearable extended reality appliance may perform a PWM procedure to display incoming messages of a messaging application according to a 75% duty cycle, e.g., to draw the attention of the user, and display a weather application according to a 50% duty cycle, e.g., as a background application)
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Wright and Forutanpour’s combined augmented reality system to include temporally adjusting the brightness of the augmentation as taught by Berliner. The motivation lies in the advantage of minimizing visual distraction and improve user experience. Temporally modifying display brightness is a well-known technique for conserving battery life and reducing display clutter for an application that augments content such as Wright’s teachings, yielding predicting results to improve display-based content.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JENNY NGAN TRAN whose telephone number is (571)272-6888. The examiner can normally be reached Mon-Thurs 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alicia Harrington can be reached at (571) 272-2330. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JENNY N TRAN/Examiner, Art Unit 2615
/ALICIA M HARRINGTON/Supervisory Patent Examiner, Art Unit 2615
Prosecution Insights