Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1, 10, and 18 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Stahl (US 20200066046 A1).
Regarding claim 1, Stahl teaches a method comprising, by one or more computing systems:
accessing, from a client system associated with a first user (par. 0091: “In particular embodiments, the computing system may access a data store that includes AR content. As an example and not by way of limitation, the data store may be a local data store of a client computing device (e.g., the first computing device)”; as the following citation will demonstrate, this client system is associated with a first user.), a plurality of augmented reality surfaces in an artificial reality scene, wherein the plurality of augmented reality surfaces are projected for display to a viewpoint of the first user based on an eye pose of the first user (par. 0090: “In particular embodiments, a computing system may determine one or more suggested AR content items for the first user. In particular embodiments, the computing system may be a client computing device (e.g., the first computing device) or a remote server. The suggested AR content item may be an AR content item from a subset of AR content items that are accessible to the first user (e.g., content with privacy settings that allow the first user to access the content).” These AR content items are considered analogous to the mentioned “augmented reality surfaces.”);
capturing a frame of the real-world environment while displaying the artificial reality scene to the first user, wherein the captured frame of the real-world environment is associated with a viewpoint of a camera pose (par. 0029: “In particular embodiments, an AR application may be configured to obtain images (e.g., video frames) of a real-world environment captured by a camera communicatively coupled to the device on which the AR application is running.”);
reprojecting the plurality of augmented reality surfaces for display to the viewpoint of the camera pose (par. 0030: “In particular embodiments, the AR application may support one or more types of tracking algorithms, which may be used to create a map of a real-world environment to create an AR environment (e.g., an AR representation of the real-world environment). Users may view, interact, or otherwise engage with the AR environment on any AR compatible computing device.”);
generating an aligned artificial reality scene for display to the viewpoint of the camera pose based on the reprojected plurality of augmented reality surfaces and the captured frame of the real-world environment (par. 0057: “Conceptually, given a video frame, SLAM may estimate the relative position and orientation of the camera and features of interest in the scene (e.g., often edges, corners, etc.) and iteratively update the estimates based on motion and the resulting feature observations. Based on positional deviations of those features due to movement, SLAM may use triangulation techniques to generate a 3D model of the recognizable objects in the captured scene. For example, when the camera moves, a landmark point associated with a feature of interest may move.”); and
sending the aligned artificial reality scene to a second user for display from the viewpoint of the camera pose (par. 0047: “In this example, a user may view an AR environment with the first computing device and interact with the AR environment (e.g., by placing an AR content item in the AR environment) with the second computing device. In particular embodiments, the first computing device may be operated by a first user and the second computing device may be operated by a second user.”).
Regarding claim 10, Stahl teaches one or more computer-readable non-transitory storage media embodying software that is operable when executed to execute the method of claim 1 (par. 0158: “Herein, a computer-readable non-transitory storage medium or media may include one or more semiconductor-based or other integrated circuits (ICs) (such, as for example, field-programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard drives (HHDs), optical discs, optical disc drives (ODDs), magneto-optical discs, magneto-optical drives, floppy diskettes, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM-drives, SECURE DIGITAL cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these, where appropriate.”).
Regarding claim 18, Stahl teaches a system comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to execute the method of claim 1 (In particular embodiments, memory 1804 includes main memory for storing instructions for processor 1802 to execute or data for processor 1802 to operate on.).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 2-3 and 11-12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Stahl (US 20200066046 A1) as applied to claims 1 and 10 above, and further in view of Malia (US 11227446 B2).
Regarding claim 2, Stahl teaches a plurality of surfaces in an artificial reality scene, but fails to teach timestamps.
Malia teaches wherein the plurality of augmented reality surfaces are associated with a first timestamp, wherein the captured frame of the real-world environment is associated with a second timestamp that is different from the first timestamp (col. 117, lines 16-23: “In some embodiments, the proximity criteria include a capture time proximity requirement, requiring that the first image and the second image were captured within a predefined threshold amount of time from each other (e.g., a timestamp corresponding to a time of capture of the first image is within the predefined threshold amount of time from a timestamp corresponding to a time of capture of the second image).”).
It would have been obvious to one familiar in the art prior to the effective filing date of the claimed invention to include Malia’s timestamps in Stahl’s content-sharing method, as this would allow a predictive algorithm to prepare future AR image renders based on information gathered at previous timestamps, as well as recording the positioning data of a plurality of surfaces in order to share the recording at a higher resolution later, as recognized by Malia. Stahl teaches predictive algorithms, so there is precedent to suggest that timestamps would be desirable in some iterations of Stahl’s invention.
Regarding claim 3, Stahl teaches the method of claim 1, but fails to teach a second frame rate higher than a first frame rate.
Malia teaches varying levels of graphical fidelity (col. 51, lines 36-45, “The method includes, concurrently displaying: graphical representations of the plurality of primary features that are generated with a first level of fidelity to the corresponding plurality of primary features of the physical environment (808); and one or more graphical representations of secondary features that are generated with a second level of fidelity to the corresponding one or more secondary features of the physical environment, wherein the second level of fidelity is lower than the first level of fidelity in the user interface”).
Graphical fidelity is an inversely proportional quality to frame rate; a lower level of graphical fidelity would necessarily lead to a higher frame rate, as anyone familiar in the art would be aware. Therefore, Malia teaches wherein the plurality of surfaces are previously rendered at a first frame rate, wherein the reprojected plurality of surfaces are generated at a second frame rate that is higher than the first frame rate.
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to incorporate Malia’s lower-fidelity graphical representation into Stahl’s invention so as to improve frame rate and thus increase interactive responsiveness in the embodiment.
Claim 11 is functionally identical to claim 2, save that it depends on claim 10 rather than claim 1. Therefore, it is rejected on the same basis as claim 2.
Claim 12 is functionally identical to claim 3, save that it depends on claim 10 rather than claim 1. Therefore, it is rejected on the same basis as claim 3.
Claim(s) 4 and 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Stahl (US 20200066046 A1) as applied to claims 1 and 10 above, and further in view of Li (US 9588593 B2).
Regarding claim 4, Stahl teaches the method of Claim 1, wherein reprojecting the plurality of augmented reality surfaces for display to the viewpoint of the camera pose further comprising:
reprojecting the plurality of surfaces into a two-dimensional rectilinear frame buffer based on the camera pose (Paragraph 0089, “In particular embodiments, the objects may be identified using any suitable methods such as the tracking algorithms disclosed herein (e.g., SLAM, region tracking, face tracking)”). It fails to teach disabling one or more display corrections when reprojecting the plurality of surfaces based on the camera pose.
Li teaches disabling one or more display corrections when reprojecting the plurality of augmented reality surfaces for display to the viewpoint of the camera pose, wherein the one or more display corrections are used for projecting the plurality of surfaces for display to the first user based on the eye pose of the first user (col. 10, lines 18-26: “The function of the image warper is to reduce this latency by performing a rapid “rerendering approximation” that provides a relatively quick and efficient update to the images 110 and 111 based on changes to the pose. This rerendering approximation is not a complete rendering as would be performed by the scene renderer 142; instead it uses approximations to reduce the calculations and communications required to update the display, thereby reducing latency.”).
By rendering an approximation of a virtual object, Li demonstrates a method by which a virtual object can be observed without the need for constant display corrections despite a constantly-shifting pose. Li uses display corrections elsewhere for larger pose adjustments, but these display corrections are not in use for smaller changes in pose (col. 3, lines 19-23, “For large changes in pose, the approximations used by the image warper may become inadequate, and it may be preferable to perform a full 3D rendering despite the high latency. For small changes in pose, the rerendering approximations may be sufficiently realistic.”).
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to have modified Stahl with the features of reduced graphical corrective features as taught by Li, as this would allow for the playback to run more smoothly and reduce latency for the user.
Claim 13 is functionally identical to claim 4, save that it depends on claim 10 rather than claim 1. Therefore, it is rejected on the same basis as claim 4.
Claim(s) 5-9, 14-17 and 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Stahl (US 20200066046 A1) as applied to claims 1, 10, and 17 above, and further in view of Groten (US 9761052 B2) and Malia (US 11227446 B2).
Regarding claim 5, Stahl teaches the method of claim 1, but fails to teach positioning metadata or frame buffers.
Groten teaches recording positioning metadata associated with each of the plurality of surfaces (col. 17, lines 43-45: “Advantageously, sizing and positioning metadata enables the graphics generation function to generate augmentations for the object.”); and storing the plurality of surfaces to a single frame buffer (col. 6, lines 31-35, “The image sensor may generate one or more image frame(s) capturing the real world scenery, which may be stored in an image frame buffer in memory 124 (that is accessible by the AR application)”); it fails to teach three-dimensional playback.
Malia teaches repositioning the stored plurality of surfaces based on the positioning metadata for three-dimensional playback (col. 71, lines 55-64, “the annotation is not displayed until playback of the video reaches an initial frame (e.g., the second representation) of the second media that corresponds to at least the first portion of the physical environment, corresponds to at least the first portion of the physical environment, and the annotation is displayed on the portion of the second representation of the second media in response to receiving the input corresponding to selection of the second media, in combination with displaying the second media”). Note that in order to correspond to any portion of a physical environment as explained in the citation, Malia would necessarily require this playback be three-dimensional and rely on three-dimensional positioning data (col. 35, lines 22-25, “These cameras (optionally in combination with a time-of-flight sensor such as time-of-flight sensor 220, FIG. 2B) acquire depth data of the room, that is used for creating a three dimensional representation of the scanned room.”).
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to have modified Stahl with the features of positioning metadata as taught by Groton and Malia to record positioning metadata if one desired to share a three-dimensional virtual environment that accurately recorded the location of virtual objects, as positioning metadata would create a convenient reference point to draw from when rendering this environment.
Regarding Claim 6, Stahl teaches encoding a single frame buffer (Paragraph 0041: “As an example and not by way of limitation, in the case where the sensor is a smartphone camera, the first information may encode an image in a raw image format (e.g., an unprocessed pixel buffer directly sourced from the camera) or any other suitable image file format (e.g., JPEG, BMP, GIF, PNG).”), but fails to teach sending it and the encoded positioning metadata to a second user for three-dimensional playback.
Groten teaches encoding the positioning metadata and sending the encoded single frame buffer and the encoded positioning metadata to a second user (col. 18, Lines 8-11, “In certain embodiments, the metadata for the relative positioning of original reference image (or object) and/or slices are described in three dimensions and/or in another reference system.”). It fails to teach three-dimensional playback.
Malia teaches three-dimensional playback (as above, in claim 5 rejection).
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to record positioning metadata, as explained above in the Claim 5 rejection. This same metadata would prove obviously useful in transferring data to a second user in an easily-packaged way for convenient playback.
Regarding Claim 7, Stahl, Groten, and Malia teach the method of Claim 5. Groten further teaches wherein the positioning metadata comprises a surface three-dimensional pose (col. 25, line 63 – col. 26, line 2: “In some embodiments, the processor includes a graphics processing unit specialized for rendering and generating computer-generated graphics, e.g., to generate augmentations based on the pose information and/or metadata associated with the reference features set from which the pose information is determined. Preferably, processor is configured to communicate, via a communication bus with other components of device.”) in the single frame buffer (col. 6, lines 31-35: “The image sensor may generate one or more image frame(s) capturing the real world scenery, which may be stored in an image frame buffer in memory 124 (that is accessible by the AR application).”), a surface offset in the single frame buffer (col. 16, lines 52-60: “Exemplary metadata is reproduced below: …Offset: (0,0)”), a surface size in the single frame buffer (col. 16, lines 52-58: “Exemplary metadata is reproduced below: …Size: (240,320)”), a surface scaling factor in the single frame buffer (col. 18, lines 17-21: “The transformation may be expressed by a 3×3 homogeneous transformation matrix, where the upper left 2×2 submatrix may describe the rotation and scaling, and the upper right 2×1 column vector may describe the translation (e.g., the last row of the matrix is (0,0,1)).”), and a camera pose (col. 1, lines 42-45: “Furthermore, the AR application may use a pose estimation process to determine position and/or orientation (pose information) of the object based on information in the image frame and sensor and/or camera parameters.”).
Stahl further teaches tracking the head pose and eye pose of the first user (par. 0100: “In particular embodiments, the focal region of the display may be center of the display, a region around a reticle displayed on the display, or a predicted focus of the first user's gaze (e.g., as determined by a camera or other sensor that may track the first user's head orientation or pupils).”).
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to have modified Stahl and Malia with the features of the above elements in any positioning metadata being recorded as taught by Groten because all of these elements are useful in determining an accurate position for every three-dimensional surface, as demonstrated by Groten.
Regarding claim 8, Stahl, Groten, and Malia teach the method of claim 5.
Stahl further teaches a plurality of surfaces stored in a texture format (par. 0039: “In particular embodiments, the region-tracking algorithm in particular embodiments mainly process 2D pixel or texture information (aside from possibly generating a plane to model the location of an object in a video frame and using gyroscope data to determine the plane's orientation in 3D space).”).
Regarding Claim 9, Stahl, Groten, and Malia teach the method of claim 5.
Stahl further teaches interactable surfaces during a three-dimensional playback (par. 0007: “In particular embodiments, the first user may be able to view or interact with the AR content item in intuitive ways (e.g., picking them up, moving them, modifying them, etc.).”).
Claim 14 is functionally identical to claim 5, save that it depends on claim 10 rather than claim 1. Therefore, it is rejected on the same basis as claim 5.
Claim 15 is functionally identical to claim 6, save that it depends on claim 14 rather than claim 5. Therefore, it is rejected on the same basis as claim 6.
Claim 16 is functionally identical to claim 7, save that it depends on claim 14 rather than claim 5. Therefore, it is rejected on the same basis as claim 7.
Claim 17 is functionally identical to claim 8, save that it depends on claim 14 rather than claim 5. Therefore, it is rejected on the same basis as claim 8.
Claim 19 is functionally identical to claim 5, save that it depends on claim 18 rather than claim 1. Therefore, it is rejected on the same basis as claim 5.
Claim 20 is functionally identical to claim 7, save that it depends on claim 18 rather than claim 5. Therefore, it is rejected on the same basis as claim 7.
Response to Amendment
The amendment filed 29 September, 2025 has been entered. Claims 1-20 remain pending in the application. Applicant’s amendments to the Specification, Drawings, and Claims have overcome each and every objection and 112(b) rejection previously set forth in the Non-Final Office Action mailed 28 April, 2025.
Response to Arguments
Applicant's arguments filed 29 September, 2025 have been fully considered but they are not persuasive.
With respect to Stahl (see applicant’s response, pages 8-10, “REJECTION OF CLAIMS 1, 10, AND 18 UNDER 35 U.S.C. 102(A)(1) AS BEING ANTICIPATED BY STAHL”), applicant argues that the examiner has failed to establish a prima facie case of anticipation for claims 1, 10, and 18. The requirements for a proper response to a rejection may be found in 37 CFR 1.111(b) and MPEP §714.02; see also MPEP §707.07(a).
Regarding the substance of the examiner’s anticipation rejection over Stahl as argued on page 8-10 of the remarks, the requirements for anticipation are discussed in MPEP § 2131. MPEP § 2131 notes that “To reject a claim as anticipated by a reference, the disclosure must teach every element required by the claim under its broadest reasonable interpretation.” In the most recent office action, the examiner provided multiple rejections under 35 U.S.C. 102 with specific mappings addressing every element of the claimed invention for the pertinent claims. Applicant submits that the cited references are not understood to disclose or suggest each and every feature of independent claims 1, 10, and 18, particularly with respect to at least the features common between all three claims of “accessing, from a client system associated with a first user, a plurality of augmented reality surfaces in an artificial reality scene, wherein the plurality of augmented reality surfaces are projected for display to a viewpoint of the first user based on an eye pose of the first user; capturing a frame of a real-world environment while displaying the artificial reality scene to the first user, wherein the captured frame of the real-world environment is associated with a viewpoint of a camera pose; reprojecting the plurality of augmented reality surfaces for display to the viewpoint of the camera pose; generating an aligned artificial reality scene for display to the viewpoint of the camera pose based on the reprojected plurality of augmented reality surfaces and the captured frame of the real-world environment; and sending the aligned artificial reality scene to a second user for display from the viewpoint of the camera pose.”
Examiner has reformatted the independent claim rejections for claims 1, 10, and 18 in order to provide a more legible claim mapping, and has referenced new sources from Stahl in the hopes that these better illustrate the claimed features of independent claim 1 as they appear in Stahl.
With respect to Stahl, Malia, Li, and Groten (see applicant’s response, p. 10-11, “REJECTION OF CLAIMS 2-3 AND 11-12 UNDER 35 U.S.C. 103 AS BEING UNPATENTABLE OVER STAHL IN VIEW OF MALIA, REJECTION OF CLAIMS 4 AND 13 UNDER 35 U.S.C. 103 AS BEING UNPATENTABLE OVER STAHL IN VIEW OF LI, AND REJECTION OF CLAIMS 5-9, 14-17 AND 19-20 UNDER 35 U.S.C. 103 AS BEING UNPATENTABLE OVER STAHL IN VIEW OF GROTEN AND FURTHER IN VIEW OF MALIA”), applicant asserts that claims 2-9, 11-17, and 19-20, depending from claim 1, are believed to be allowable because each dependent claim is deemed to define an additional aspect of the invention. Since the rejections of claims 1, 10, and 18 are maintained, so too is the rejection of all dependent claims.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RYAN A BARHAM whose telephone number is (571)272-4338. The examiner can normally be reached Mon-Fri, 8:30am-5pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu, can be reached at (571) 272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RYAN ALLEN BARHAM/Examiner, Art Unit 2613
/XIAO M WU/Supervisory Patent Examiner, Art Unit 2613