DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
2. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
3. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
4. Claims 1, 21, 56-57 are rejected under 35 U.S.C. 103 as being unpatentable over Meng (Meng, Jiayi, Sibendu Paul, and Y. Charlie Hu. "Coterie: Exploiting frame similarity to enable high-quality multiplayer vr on commodity mobile devices." Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems. 2020.), hereinafter Meng.
Regarding claim 1, Meng teaches an apparatus comprising: at least one processor; and at least one memory storing instructions (Fig. 9 section 6, wherein the invention is implemented as software on a VR device, section 7.3, wherein the software is executed on GPU/CPU, which is interpreted as an apparatus containing at least on processor executing memory storing instructions) that, when executed by the at least one processor, cause the apparatus at least to: determine a radius of a sphere centered at a virtual camera that has a correspondence to the apparatus (Fig. 4, section 4.3, wherein determining a cutoff radius from a viewing point is interpreted as determining a radius centered at a virtual camera, wherein determining a radius suggests determining a radius of a sphere); wherein preferential rendering is used for objects within the sphere defined with the radius (Fig. 4, Fig. 9, Section 4.3 paragraph 2-6, wherein near background environment objects within the cutoff radius are rendered locally on the device, which is interpreted as using preferential rendering for objects within the sphere defined with the radius); wherein remote rendering is used for objects outside the sphere defined with the radius (Fig. 4, Fig. 9, Section 4.3 paragraph 2-6, wherein far background environment objects outside the cutoff radius are rendered on an external server, which is interpreted as using remote rendering for objects outside the sphere defined with the radius); and draw on a display the objects within the sphere and objects outside the sphere, based on the objects being used for preferential rendering and remote rendering.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have used the teachings of Meng to disclose this apparatus for preferential and remote rendering. Meng discusses split rendering approach for 360-degree VR systems, by first determining a radius from a virtual camera, and using that radius to determine which background objects are considered “near”, and which are considered” far”. It then discusses rendering the near objects locally, and the far objects on a cloud-based server. The teachings described in Meng pertain to maintaining a high-quality user experience in VR systems, using their proposed split rendering system to split workloads accordingly in order to maintain a low input-to-display latency, while providing high quality visuals and frame rates. As the split rendering teachings of Meng describe improving a user’s VR experience by reducing latency and maintaining video quality, it would have been obvious to have used its teachings to disclose the claimed invention.
Regarding claim 21, Meng discloses the apparatus of claim 1. Additionally, Meng teaches the apparatus of claim 1, wherein preferential rendering comprises at least one or more of: rendering the objects within the sphere defined with the radius locally at the apparatus, and the objects outside the sphere defined with the radius being rendered at a split rendering server (Fig. 4, Fig. 9, Section 4.3 paragraph 2-6, wherein near background environment objects within a cutoff radius are rendered locally on the mobile device, and far background environment objects outside the cutoff radius are rendered on an external server, which is interpreted as rendering objects within the radius of a sphere locally, and rendering objects outside the radius of a sphere on a split rendering server); or the objects within the sphere defined with the radius being rendered at a split rendering server as a composition layer with higher warp sensitivity; or the objects within the sphere defined with the radius being rendered at a quality higher than the objects outside the sphere defined with the radius, and higher than any other objects that are not within the sphere defined with the radius.
Regarding claim 56, Meng discloses the apparatus of claim 1. Additionally, Meng teaches the apparatus of claim 1, wherein: objects with a high-level of interaction are rendered at the apparatus (Fig. 9, Section 5, wherein foreground interactions (FI) are rendered locally on the mobile device; Section 2.2, wherein foreground interactions are VR content operated by a user, which is interpreted as objects with a high-level of interaction being rendered at the apparatus), and a level of interaction of an object is determined based on actions comprising one or more of eye gaze, or controller input, or a hand gesture (Section 2.2, wherein foreground interaction objects are determined as objects triggered by players operating a controller).
Regarding claim 57, Meng discloses the apparatus of claim 1. Additionally, Meng teaches the apparatus of claim 1, wherein objects that have predictable motion or slow response to actions are rendered at a server (Section 2.2, wherein background environment (BE) VR content are predictable and can be prerendered and prefetched on a remote rendering engine, which is interpreted as objects with predictable motion are rendered at a server).
5. Claims 2-3 are rejected under 35 U.S.C. 103 as being unpatentable over Meng in view of Phillips et al. (US 12558618 B2), hereinafter Phillips.
Regarding claim 2, Meng discloses the apparatus of claim 1. Additionally, Phillips teaches the apparatus of claim 1, wherein the apparatus is further caused to: determine a restricted field of view relative to the radius, wherein preferential rendering is used for objects within the restricted field of view (Fig. 2, Col. 10 lines 31-52, wherein a rendering engine can generate model view projection data based on where a viewport is pointing within a 360 space, which is interpreted as determining a restricted field of view relative to a 360 space, and suggests being based on the 360 space's radius, and wherein performing the model view projection locally to render the visual information based on where the viewport is pointing is interpreted as performing preferential rendering for objects within the restricted field of view).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Meng with the teachings of Phillips for this apparatus for preferential rendering. Both Meng and Phillips discuss rendering systems for VR systems involving splitting rendering between local rendering, and remote server rendering for 360-degree spaces. Meng discusses this split rendering technique for the purposes of reducing latency and maintaining high quality video and frame rates to provide a better VR user experience. Similarly, Phillips discusses splitting rendering between local and remote rendering and dynamically adjusting what gets rendered remotely based on shifts in latency or bandwidth, in order to maintain consistent video quality. As both references discuss splitting rendering between local and remote devices to improve VR video quality and latency, and Phillips discusses dynamic improvements to split rendering that Meng would appreciate, it would be obvious to combine these references.
Regarding claim 3, Meng in view of Phillips discloses the apparatus of claim 2. Additionally, Phillips teaches the apparatus of claim 2, wherein the restricted field of view is based on one or more of: a gaze of a user of the apparatus (Fig. 2, Col. 10 lines 31-52, wherein a rendering engine can generate model view projection data based on where a viewport is pointing within a 360 space, which is interpreted as determining a restricted field of view relative to a 360 space; Col. 8 lines 25-29, wherein the orientation of a headset is used as data as the viewport, which is interpreted as the restricted field of view being based on a gaze of a user wearing a headset), or an application state, or an action of another user in a mixed reality experience shared with the apparatus, or a pose of another user in a mixed reality experience shared with the apparatus, or an environmental condition, or a position of at least one light source, or a movement of at least one object.
The motivation to combine would be the same as that set forth for claim 1.
6. Claims 17-19, 22-23, 34-36, 46 are rejected under 35 U.S.C. 103 as being unpatentable over Meng in view of Harviainen et al. (US 12231483 B1), hereinafter Harviainen.
Regarding claim 17, Meng discloses the apparatus of claim 1. Additionally, Harviainen teaches the apparatus of claim 1, wherein the apparatus is further caused to: determine a seamless adaptation parameter that indicates that an object is rendered at a split rendering client when at least one requirement for preferential rendering is met (Fig. 2, Col. 11 lines 20-44, wherein additional parameters such as power consumption, battery, or network bandwidth are provided to determine whether to offload rendering capabilities from the viewing client to an edge node, wherein an edge node is defined as an external processor for graphics rendering managed by the content server 215; Col. 3 line 59 – Col. 4 line 8, wherein handing over processing step from one processing entity to another processing entity is described as seamless, which is interpreted as the apparatus determining a seamless adaptation parameter to determine whether objects are rendered on the client device, which is interpreted as preferential rendering, or offloaded to a split rendering node.), and that the object is rendered at a split rendering server when the at least one requirement for preferential rendering is not met, without use of additional signaling between the split rendering client and the split rendering server (Fig. 2, Col. 11 lines 45-62, wherein edge nodes analyze and process what assets are to be rendered locally in the viewing client instead of being offloaded to be rendered by an edge node, which is interpreted as determining requirements for whether objects should be rendered at a split rendering client or rendered in a split rendering server, and suggests that since assets are only sent to the client device when the requirements for preferential client rendering is met, that no additional signaling is used when the requirements is not met).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Meng with the teachings of Harviainen. Both Meng and Harviainen discuss split rendering techniques for different extended reality content. Meng discusses this split rendering technique for the purpose of reducing latency and maintaining high quality video and frame rates to provide a better user experience for VR headsets. Similarly, Harviainen discusses dynamically responding to operating conditions while performing split rendering, adapting to changes in conditions to better transfer data between servers and a user’s device more seamlessly, without abruptly changing or interrupting the XR experience. Meng focuses on using split rendering to reduce latency and maintain video quality, while Harviainen focuses on adapting to environmental and operational conditions to more seamlessly transfer data without disrupting user experience. As both references discuss analogous art in split rendering for XR and VR content for the purposes of improving user experience, it would be obvious to combine these two references.
Regarding claim 18, Meng discloses the apparatus of claim 1. Additionally, Harviainen teaches the apparatus of claim 1, wherein the apparatus is further caused to: determine a type of preferential rendering to be used, wherein the type of preferential rendering comprises one of: local rendering where objects are rendered on a client (Fig. 2, Col. 11 lines 20-44, wherein determining whether to offload rendering capabilities from the viewing client to an edge node, wherein an edge node is defined as an external processor for graphics rendering managed by the content server 215, is interpreted as determining a preferential rendering consisting of local rendering where objects are rendered on a client), or level of detail rendering where objects with a higher level of detail are rendered, or composition layer rendering where objects are rendered as a separate composition layer, or remote rendering when level of detail rendering is used.
The motivation to combine would be the same as that set forth for claim 17.
Regarding claim 19, Meng discloses the apparatus of claim 1. Additionally, Harviainen teaches the apparatus of claim 1, wherein the apparatus is further caused to: transmit a split rendering configuration message with at least one offered value for split rendering (Fig. 2, Col. 11 lines 20-44, wherein parameters of a viewing client determining the viewing client capabilities are sent to edge nodes managed by the content server in order to determine whether graphics rendering should be offloaded to the edge nodes, which is interpreted as transmitting a split rendering configuration message with at least one offered value for split rendering); receive a rendering description message comprising at least one value for the split rendering; and render objects based on the at least one value for the split rendering (Fig. 2, Fig. 2, Col. 11 lines 45-62, wherein determining whether to render assets on the viewing client based on the viewing client capabilities and parameters, is interpreted as receiving a rendering description message comprising at least one value for split rendering, and rendering objects based on the values for the split rendering).
The motivation to combine would be the same as that set forth for claim 17.
Regarding claim 22, Meng discloses the apparatus of claim 1. Additionally, Harviainen teaches the apparatus of claim 1, wherein the apparatus comprises an extended reality device, or the extended reality device comprises the apparatus (Fig. 1, Col. 3 line 59 – Col. 4 line 8, wherein the apparatus comprises an XR viewing device).
The motivation to combine would be the same as that set forth for claim 17.
Regarding claim 23, Meng teaches an apparatus comprising: at least one processor; and at least one memory storing instructions (Fig. 9 section 6, wherein the invention is implemented as software on a VR device, section 7.3, wherein the software is executed on GPU/CPU, which is interpreted as an apparatus containing at least on processor executing memory storing instructions) that, when executed by the at least one processor, cause the apparatus at least to: determine the at least one value for the split rendering comprising at least one radius of a sphere centered at a virtual camera that has a correspondence to a client device (Fig. 4, section 4.3, wherein determining a cutoff radius from a viewing point on a local mobile device is interpreted as determining a radius centered at a virtual camera corresponding to a client device, wherein determining a radius suggests determining a radius of a sphere); wherein preferential rendering is used for objects within the sphere defined with the at least one radius (Fig. 4, Fig. 9, Section 4.3 paragraph 2-6, wherein near background environment objects within the cutoff radius are rendered locally on the device, which is interpreted as using preferential rendering for objects within the sphere defined with the radius); wherein remote rendering is used for objects outside the sphere defined with the at least one radius (Fig. 4, Fig. 9, Section 4.3 paragraph 2-6, wherein far background environment objects outside the cutoff radius are rendered on an external server, which is interpreted as using remote rendering for objects outside the sphere defined with the radius).
Meng does not teach receiving a split rendering configuration message with at least one offered value for split rendering; and determining at least one value for the split rendering; and transmitting a rendering description message comprising the at least one value for the split rendering.
Harviainen teaches receiving a split rendering configuration message with at least one offered value for split rendering (Fig. 2, Fig. 2, Col. 11 lines 45-62, wherein determining whether to render assets on the viewing client based on the viewing client capabilities and parameters, is interpreted as receiving a rendering description message comprising at least one value for split rendering, and rendering objects based on the values for the split rendering); and determining at least one value for the split rendering; and transmitting a rendering description message comprising the at least one value for the split rendering (Fig. 2, Col. 11 lines 20-44, wherein parameters of a viewing client determining the viewing client capabilities are sent to edge nodes managed by the content server in order to determine whether graphics rendering should be offloaded to the edge nodes, which is interpreted as determining at least one value for split rendering, and transmitting a split rendering configuration message comprising at least one value for split rendering).
The motivation to combine would be the same as that set forth for claim 17.
Regarding claim 34, Meng in view of Harviainen discloses the apparatus of claim 23. Additionally, Meng teaches the apparatus of claim 23, wherein preferential rendering comprises at least one or more of: rendering the objects within the sphere defined with the radius locally at the apparatus, and the objects outside the sphere defined with the radius being rendered at a split rendering server (Fig. 4, Fig. 9, Section 4.3 paragraph 2-6, wherein near background environment objects within a cutoff radius are rendered locally on the mobile device, and far background environment objects outside the cutoff radius are rendered on an external server, which is interpreted as rendering objects within the radius of a sphere locally, and rendering objects outside the radius of a sphere on a split rendering server); or the objects within the sphere defined with the radius being rendered at a split rendering server as a composition layer with higher warp sensitivity; or the objects within the sphere defined with the radius being rendered at a quality higher than the objects outside the sphere defined with the radius, and higher than any other objects that are not within the sphere defined with the radius.
Regarding claim 35, Meng in view of Harviainen discloses the apparatus of claim 23. Additionally,, Harviainen teaches the apparatus of claim 23, wherein the apparatus comprises an extended reality device, or the extended reality device comprises the apparatus (Fig. 1, Col. 3 line 59 – Col. 4 line 8, wherein the apparatus comprises an XR viewing device).
Regarding claim 36, Meng teaches an apparatus comprising: at least one processor; and at least one memory storing instructions (Fig. 9 section 6, wherein the invention is implemented as software on a VR device, section 7.3, wherein the software is executed on GPU/CPU, which is interpreted as an apparatus containing at least on processor executing memory storing instructions) that, when executed by the at least one processor, cause the apparatus at least to: receive the at least one value for the split rendering comprising at least one radius of a sphere centered at a virtual camera that has a correspondence to a client device (Fig. 4, section 4.3, wherein determining a cutoff radius from a viewing point on a local mobile device is interpreted as receiving a radius centered at a virtual camera corresponding to a client device, wherein determining a radius suggests determining a radius of a sphere); wherein preferential rendering is used for objects within the sphere defined with the at least one radius (Fig. 4, Fig. 9, Section 4.3 paragraph 2-6, wherein near background environment objects within the cutoff radius are rendered locally on the device, which is interpreted as using preferential rendering for objects within the sphere defined with the radius); wherein remote rendering is used for objects outside the sphere defined with the at least one radius (Fig. 4, Fig. 9, Section 4.3 paragraph 2-6, wherein far background environment objects outside the cutoff radius are rendered on an external server, which is interpreted as using remote rendering for objects outside the sphere defined with the radius).
Meng does not teach receiving, from a client, a split rendering configuration message with at least one offered value for split rendering; forwarding, to a server, the split rendering configuration message with the at least one offered value for split rendering; receive, from the server, a rendering description message comprising at least one value for the split rendering; and forward, to the client, the rendering description message comprising the at least one value for the split rendering.
Harviainen teaches receiving, from a client, a split rendering configuration message with at least one offered value for split rendering (Fig. 2, Fig. 2, Col. 11 lines 45-62, wherein determining whether to render assets on the viewing client based on the viewing client capabilities and parameters, is interpreted as receiving a split rendering configuration message from a client comprising at least one offered value for split rendering, and rendering objects based on the values for the split rendering); forwarding, to a server, the split rendering configuration message with the at least one offered value for split rendering (Fig. 2, Col. 11 lines 20-44, wherein parameters of a viewing client determining the viewing client capabilities are sent to edge nodes managed by the content server in order to determine whether graphics rendering should be offloaded to the edge nodes, which is interpreted as forwarding, to a server, the split rendering configuration message with at least one offered value for split rendering); receive, from the server, a rendering description message comprising at least one value for the split rendering (Fig. 2, Fig. 2, Col. 11 lines 45-62, wherein the edge nodes managed by the content server determining whether to render assets and which assets to render on the viewing client based on the viewing client capabilities and parameters is interpreted as receiving, from the server, a rendering description message comprising at least one value for split rendering, and rendering objects based on the values for the split rendering); and forward, to the client, the rendering description message comprising the at least one value for the split rendering (Fig. 2, Fig. 2, Col. 11 lines 45-62, wherein the edge nodes managed by the content server selecting the assets to be rendered and updating asset offloading settings for the viewing client is interpreted as forwarding, to the client, the rendering description message comprising the at least one value for split rendering).
The motivation to combine would be the same as that set forth for claim 17.
Regarding claim 46, Meng in view of Harviainen discloses the apparatus of claim 23. Additionally, Harviainen teaches the apparatus of claim 23, wherein the apparatus comprises a simple web real time communication application protocol server, or the simple web real time communication application protocol comprises the apparatus (Col. 22 lines 35-50, wherein the remote server may comprise an Internet based web browser, which is interpreted as the apparatus comprising a web real time communication application protocol server).
The motivation to combine would be the same as that set forth for claim 17.
7. Claims 24, 37 are rejected under 35 U.S.C. 103 as being unpatentable over Meng in view of Harviainen as applied to claims 1, 23 above, and further in view of Phillips.
Regarding claim 24, Meng in view of Harviainen discloses the apparatus of claim 23. Additionally, Phillips teaches the apparatus of claim 23, wherein the apparatus is further caused to: determine a restricted field of view relative to the radius, wherein preferential rendering is used for objects within the restricted field of view (Fig. 2, Col. 10 lines 31-52, wherein a rendering engine can generate model view projection data based on where a viewport is pointing within a 360 space, which is interpreted as determining a restricted field of view relative to a 360 space, and suggests being based on the 360 space's radius, and wherein performing the model view projection locally to render the visual information based on where the viewport is pointing is interpreted as performing preferential rendering for objects within the restricted field of view).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Meng in view of Harviainen with the teachings of Phillips for this apparatus for preferential rendering. Meng, Harviainen, and Phillips each discuss rendering systems for VR systems involving splitting rendering between local rendering, and remote server rendering. Meng discusses this split rendering technique for the purposes of reducing latency and maintaining high quality video and frame rates to provide a better VR user experience. Similarly, Harviainen discusses dynamically responding to operating conditions while performing split rendering, adapting to changes in conditions to better transfer data between servers and a user’s device more seamlessly, without abruptly changing or interrupting the XR experience. Additionally, Phillips discusses splitting rendering between local and remote rendering and dynamically adjusting what gets rendered remotely based on shifts in latency or bandwidth, in order to maintain consistent video quality. Both Harviainen and Phillips discuss dynamically adjusting split rendering systems based on system conditions. As all three references discuss analogous art on split rendering for VR devices, and Harviainen and Phillips discuss novel improvements that can further enhance user experience, it would be obvious to combine these references.
Regarding claim 37, Meng in view of Harviainen discloses the apparatus of claim 36. Additionally, Phillips teaches the apparatus of claim 36, wherein the apparatus is further caused to: determine a restricted field of view relative to the radius, wherein preferential rendering is used for objects within the restricted field of view (Fig. 2, Col. 10 lines 31-52, wherein a rendering engine can generate model view projection data based on where a viewport is pointing within a 360 space, which is interpreted as determining a restricted field of view relative to a 360 space, and suggests being based on the 360 space's radius, and wherein performing the model view projection locally to render the visual information based on where the viewport is pointing is interpreted as performing preferential rendering for objects within the restricted field of view).
The motivation to combine would be the same as that set forth for claim 24.
Allowable Subject Matter
8. Claims 20, 33, are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
9. Any inquiry concerning this communication or earlier communications from the examiner should be directed to JORDAN W YICK whose telephone number is (571)272-4063. The examiner can normally be reached M-F 8-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Said Broome can be reached at (571) 272-2931. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JORDAN WAN YICK/Examiner, Art Unit 2612
/Said Broome/Supervisory Patent Examiner, Art Unit 2612
Prosecution Insights