DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 5, 6, 10, 11, 14, 15, 19, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Bina et al. (US Pub. 2021/0136342), hereinafter Bina, in view of Kimber et al. (US Patent 10,250,813), hereinafter Kimber.
Regarding claim 1, Bina discloses a non-transitory computer-readable storage medium, comprising instructions that, when executed by one or more processors (Paragraph [0034]: location 24 includes a video processing server 56 which can be any of a wide variety of computing devices, including an appropriately programmed general purpose computing device, a purpose built computing device or a general purpose computing device equipped with special purpose hardware, such as a server equipped with a set of GPU (graphic processor units) coprocessors or FPGA coprocessors to provide additional computational resources, if necessary, to perform the computing operations described herein below. Video processing server 56 can also be a combination of two or more computing devices, each of which processes a subset of the data or performs a subset of the functions herein attributed to video processing server), cause the processors to: obtain local data about a local physical area in which a user is wearing an artificial-reality (AR) headset (Fig. 1; Paragraph [0093]: a scene is rendered based on video captured at the display device 48. Such capture may be performed by a point-of-view or “look-through” camera of a headset, a camera of a smartphone or tablet computer, or similar. As such, augmented reality may be provided. In various embodiments, an optical see-through AR headset is used, in which case the synthetic video is displayed on a transparent display allowing the user to see the actual environment as background through the display; Paragraph [0116]: these local video signals are obtained by cameras mounted on headset 48a and are chromakey processed, by telepresence server 52a or by an FPGA or computing device included in VR headset 48a, to obtain processed local videos which are then composited with the synthetic videos constructed by telepresence server 52a of user 40b and the rendered scene and the resultant composited videos are displayed to user 40a); identify, via the local data, a local orientation of the user relative to the local physical area (Fig. 1; Paragraph [0041]: telepresence server 52 may create synthetic videos of subject 36 for the left and right eyes of user 40 which correspond closely to what the eyes of user 40 would see if user 40 was physically present at the source location position at source location 24, with their head orientated as determined from headset 48 and/or an external head location and orientation tracking system; Paragraph [0059]: the viewpoint determined by telepresence server 52 can be forwarded to video processing server 56 which can then select a subset of cameras 32, which most closely correspond to the received determined viewpoint, and only process and transmit the selected subset of cameras 32 to telepresence server 52, to reduce the required amount of bandwidth required to transmit the videos and to reduce computational needs at telepresence server 52); obtain remote data indicating a remote orientation of a remote user relative to a remote physical landmark within a remote physical area different than the local physical area (Fig. 1; Paragraph [0032]: Telepresence server 52 is operable, amongst other functions, to receive data representing the head location and orientation of user 40 from headset 48 and/or an external head position and orientation tracking system to determine the user's head position and orientation and, as described further below, provides the video signals to VR headset 48 of user 40. Further, telepresence server 52 maintains a source location position for the head of user 40. Specifically, when user 40 first enters system 20, telepresence server 52 determines the head position of user 40 in remote location 44 and sets a source location position, within source location 24, to correspond to the determined head position location of user 40 (e.g.—three feet in front of subject 36) and telepresence server 52 uses this source location position in constructing the video displayed to user 40 in VR headset 48, as described below);
Bina does not explicitly disclose based on the local data, the local orientation of the user, and the remote data: identify a local physical landmark that satisfies similarity criteria with respect to the remote physical landmark, and present, at the AR headset, a visual representation of the remote user at a co-present position within the local physical area with a representative orientation relative to (i) the local physical landmark and (ii) the local orientation of the user.
However, Kimber teaches a shared view visual representation (Abstract), further comprising based on the local data, the local orientation of the user, and the remote data: identify a local physical landmark that satisfies similarity criteria with respect to the remote physical landmark, and present, at the AR headset, a visual representation of the remote user at a co-present position within the local physical area with a representative orientation relative to (i) the local physical landmark and (ii) the local orientation of the user (Fig. 9B; Column 11, lines 24-36: In the visualization 600, a marker 602 indicates the relative position of the camera device 108, and a marker 604 indicates the relative position of the local user 102 (or its proxy the sensory device 106). In some embodiments, the relative positions of the camera device 108 and the local user 102/sensory device 106 are known because the camera device 108 is placed or mounted at a fixed distance relative to the local user 102/sensory device 106 (e.g., mounted on an arm extending from a harness worn by the local user 102). The fixed distance may be provided as an initial input and/or a calibration input to the server system 120, the remote device 114, the camera device 108, and/or the sensory device 106; Column 12, lines 7-22: the local user 102 and/or the camera device 108 moves, the markers 602 and 604 remain stationary in the visualization if the camera device 108 has a fixed distance with respect to the local user 102 (e.g., local user 102 wears a harness with camera device 108 mounted on an arm extending from the harness). Either marker 602 or 604 moves if the camera device 108 does not have a fixed distance with respect to the local user 102 (e.g., camera device 108 is on a table and the local user 102 is mobile). This visualization is very helpful in communicating to both a remote user how a local user is oriented with respect to the local environment and therefore provides a frame of reference by which the local user can communicate about a point of interest to the remote user, and by which the remote user can direct the local user's attention to a particular object that is represented on the display 116 at the remote device 114; Column 14, line 55-Column 15, line 12: FIG. 9B illustrates view information 910 displayed at a remote device 114. The view information 910 includes video 912 from the camera device 108. The video 912 shows objects 914 in the local environment 101. The view information 910 also includes a first set of markers 916-A and 916-B that indicates a direction of the field of view 104 of the local user 102. Markers 916-A and 916-B that are aligned (as in FIG. 9B) indicate that the field of view 110 of the camera device 108 is in the same direction as the field of view 104 of the local user 102; the camera device 108 and the local user 102 are looking at the same objects. When the markers 916-A and 916-B are not aligned, the user would need to reorient the camera device 108 to be oriented toward the same objects as the local user 102. A second set of markers 918-A and 918-B indicate where to look to see the local user 102. Markers 918-A and 918-B that are not aligned (as in FIG. 9B) indicate that the user would need to reorient the camera device 108 to be oriented toward the local user 102. Thus, the view information 910 provides the remote user 112 information on how much to orient the camera device 108 to align with the field of view 104 of the local user 102 or to see the local user 102, as well as a general direction to orient the camera device 108 towards to align with the field of view 104 of the local user 102 or to see the local user 102). Kimber teaches that this visualization is very helpful in communicating to both a remote user how a local user is oriented with respect to the local environment and therefore provides a frame of reference by which the local user can communicate about a point of interest to the remote user (Column 12). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Bina with the features of above as taught by Kimber so as to allow for better communication between users as presented by Kimber.
Regarding claim 2, Bina, in view of Kimber teaches the non-transitory computer-readable storage medium of claim 1, Kimber discloses wherein: the local physical landmark is identified based on determining that the local physical landmark satisfies one or more similarity criteria, the one or more similarity criteria based on one or more of: comparing semantic properties of the local physical landmark to other semantic properties of the remote physical landmark; a physical relationship between the local physical landmark and one or more local real-world objects in the local physical area; and/or a location of the remote user or the user in relation to a real-world boundary within the local physical area or the remote physical area (Fig. 6; Column 11, lines 3-23: FIG. 6 illustrates an example visualization of views in accordance with some embodiments. FIG. 6 illustrates a visualization 600 that shows a bird's eye perspective of the relative positions of the camera device 108 and local user 102 (with respect to each other), and their respective fields of view 110 and 104. In some embodiments, the sensory device 106, and its relative position and field of view, serves as a proxy for the local user 102, and thus what is shown in the visualization 600 are the relative positions of the camera device 108 and the sensory device 106, and their respective fields of view. For convenience of description, unless described otherwise, the field of view 104 of the local user is also the field of view of the sensory device 106. In some embodiments, the visualization 600 is displayed on the display 116 of the remote device 114. For example, at the remote device 114, the visualization 600 is displayed along with video from the camera device 108. In some embodiments, the visualization 600 (or a similar visualization) is displayed on the display 414 of the sensory device 106, as well as at the remote device).
Regarding claim 5, Bina, in view of Kimber teaches the non-transitory computer-readable storage medium of claim 1, Bina discloses further comprising instructions for causing the one or more processors to: based on at least the local data and the remote data obtained from the local physical area and the remote physical area: generate a virtual scene, distinct from the local physical area and the remote physical area, wherein: the virtual scene includes respective visual representations of the user and the remote user and at least one other visual representation corresponding to the local physical landmark and/or the remote physical landmark, and the co-present position of the visual representation of the remote user is determined, at least in part, based on a virtual co-present position of the remote user within the virtual scene (Paragraph [0112]: a further enhancement of the virtual environment shared by users 40a and 40b, props, such as a table (not shown), can be provided at each of locations 204a and 204b and that prop can be colored with the preselected “key” color and the prop is positioned to correspond to the location of a corresponding prop (table, bench, etc.) 224 in virtual environment 220, then users 40a and 40b can “sit” at prop 224 and actually feel it with their hands, further improving their perception of actually being present in virtual environment 220; Paragraph [0120]: present invention provides a telepresence system and method that enhances a user's feeling of actual presence in a telepresence created remote location. The system and method provide for one-way (single subject and at least one remote user); two-way (first user and second user, each being a subject to the other); multi-way (a plurality of remote users, each being a subject to all other users); and one-to-many (one subject, many viewers) implementations. The system and method provide a remote user with synthetic monoscopic or stereoscopic views of the subject(s) which are formed from one or more views captured by one or more cameras at the location of the subject. In some embodiments, chromakey (or depth-based) techniques are used to remove unwanted backgrounds. In such cases, a rendered virtual reality scene can be used to replace the unwanted backgrounds. Backgrounds may be omitted or partially provided in augmented reality. Also, each remote user can capture local video, which is processed, via chromakey or depth based techniques, to remove undesired backgrounds and combined with the synthetic videos and, if present, a rendered scene to provide the remote user with a view of the subject including the remaining local video features).
Regarding claim 6, Bina, in view of Kimber teaches the non-transitory computer-readable storage medium of claim 5, Bina discloses further comprising instructions for causing the one or more processors to: responsive to an input by the user to modify a presentation mode of the AR headset, such that the AR headset is caused to present a virtual-reality scene that encompasses substantially all of a particular field of view of the user: present, at the AR headset, the virtual scene that includes the respective visual representations of the user and the remote user (Paragraph [0022]: a location may contain other users. In augmented reality, remote users can be rendered in a local user's view, so that the local user may view the remote users in the local user's environment. Further, virtual reality systems tend to use stereoscopic displays, whereas augmented reality systems may use stereoscopic displays, monoscopic displays, such as one a screen of a phone or tablet computer. It should be understood that the techniques discussed herein, when described in terms of virtual reality, may be used in augmented reality systems with the appropriate modification without departing from the scope of this disclosure. Likewise, techniques described in terms of augmented reality may be used in virtual reality systems with the appropriate modification without departing from the scope of this disclosure; Paragraph [0033]: As user 40 moves within remote location 44, the source location position is updated to track any changes of the head position of user 40, as determined by VR headset 48 and/or any external head position and orientation tracking system. Thus, in the example mentioned above, if user 40 is initially determined to be three feet in front of subject 36 and then moves their head back one foot to a distance of four foot from the front of subject 36, telepresence server 52 updates the source location position, and the corresponding rendered videos displayed in VR headset 48, to a value corresponding to four feet in front of subject 36).
Regarding claim 10, Regarding claim 10, the limitations of this claim substantially correspond to the limitations of claim 1; thus they are rejected on similar grounds.
Regarding claim 11, the limitations of this claim substantially correspond to the limitations of claim 2; thus they are rejected on similar grounds.
Regarding claim 14, the limitations of this claim substantially correspond to the limitations of claim 5; thus they are rejected on similar grounds.
Regarding claim 15, the limitations of this claim substantially correspond to the limitations of claim 6; thus they are rejected on similar grounds.
Regarding claim 19, the limitations of this claim substantially correspond to the limitations of claim 1; thus they are rejected on similar grounds.
Regarding claim 20, the limitations of this claim substantially correspond to the limitations of claim 2; thus they are rejected on similar grounds.
Allowable Subject Matter
Claims 3, 4, 7-9, 12, 13, and 16-18 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is an examiner’s statement of reasons for allowability:
Claims 3 and 12 would be allowable over the prior art of record since the cited references taken individually or in combination fails to particularly disclose or suggest a medium or system, comprising instructions for causing the one or more processors to: responsive to a different remote user joining a shared AR experience being performed by the user and the remote user: obtain other remote data identifying another remote position of the different remote user relative to another remote physical landmark, positioned within another remote physical area in which the remote user is located; and present within the local physical area, based on the local data and the other remote data, another visual representation of the different remote user, as presented in the environment of the remaining limitations of claim 3 (and substantially similar limitations in claim 12). It is noted that the closest prior art, Bina, shows the non-transitory computer-readable storage medium of claim 1. However, Bina fails to disclose or suggest instructions for causing the one or more processors to: responsive to a different remote user joining a shared AR experience being performed by the user and the remote user: obtain other remote data identifying another remote position of the different remote user relative to another remote physical landmark, positioned within another remote physical area in which the remote user is located; and present within the local physical area, based on the local data and the other remote data, another visual representation of the different remote user.
Claims 7 and 16 would be allowable over the prior art of record since the cited references taken individually or in combination fails to particularly disclose or suggest a medium or system, comprising wherein: the representative orientation of the visual representation of the remote user is based on a respective remote object, different than the remote physical landmark, that the remote user is interacting with in the remote physical area; and the visual representation of the remote user is oriented to represent a remote-object orientation of the remote user with respect to the remote object, as presented in the environment of the remaining limitations of claim 7 (and substantially similar limitations in claim 16). It is noted that the closest prior art, Bina, shows the non-transitory computer-readable storage medium of claim 1. However, Bina fails to disclose or suggest wherein: the representative orientation of the visual representation of the remote user is based on a respective remote object, different than the remote physical landmark, that the remote user is interacting with in the remote physical area; and the visual representation of the remote user is oriented to represent a remote-object orientation of the remote user with respect to the remote object.
The remaining claims depend from one of the above dependent claims, either directly or indirectly, and would be accordingly allowable.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW D SALVUCCI whose telephone number is (571)270-5748. The examiner can normally be reached M-F: 7:30-4:00PT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, XIAO WU can be reached at (571) 272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW SALVUCCI/Primary Examiner, Art Unit 2613
Prosecution Insights