DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Allowable Subject Matter
Claims 2-4, 10-12,18 objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Information Disclosure Statement
The information disclosure statement (IDS) submitted is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1,7,9,15,17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang(US-20220012842-A1), Nguyen(US-20200126291-A1) and Hladky(US-20200327713-A1).
As per claim 1, “A method for rendering a display element performed by a computer device,” Zhang teaches at (para. [0093]) “A graphics rendering method in the embodiments of this application may be executed by an electronic device. The electronic device may be a mobile terminal (for example, a smartphone), a computer, a personal digital assistant, a wearable device, a vehicle-mounted device, an Internet of things device, or another device that can perform image rendering processing.” This confirms that the claimed method is performed by a computer device.); “determining world vertex coordinates of the display element in a world space,” Zhang teaches transforming local vertex coordinates into world-space vertex coordinates using a model matrix (Zhang teaches at (para. [0116]) “After being transformed, the local coordinates are sequentially changed into world coordinates (World Coordinate), view coordinates (View Coordinate), and clip coordinates (Clip Coordinate), and a coordinate transformation process finally ends in a form of screen coordinates (Screen Coordinate).” This teaches the ordered coordinate-space transformation of graphical vertex data. Zhang further teaches at (para. [0123]) “coordinates in the world space are coordinates of a vertex relative to the (game) world. If the objects need to be scattered in the world (especially in a very real form), the world space is a space to which the objects need to be transformed. Coordinates of the objects are to be transformed from the local to the world space. The transformation is implemented by using a model matrix (Model Matrix).” This expressly teaches determining world vertex coordinates in world space. Zhang further teaches at (para. [0188-0189]) “301: Transform the to-be-processed vertex data from the local coordinate system to the world coordinate system to obtain the vertex data in the world coordinate system. In step 301, the vertex data in the local coordinate system may be multiplied by the model matrix to obtain the vertex data in the world coordinate system. ” These disclosures reinforce determining world vertex coordinates from source vertex data.); “performing coordinate space transformation based on the world vertex coordinates and the world position offset to obtain screen vertex coordinates of the display element in a screen space; and ,” Zhang teaches transforming world-space vertex coordinates through view, clip, and screen coordinate spaces (Zhang teaches at (para. [0126]) “The view space is a result generated by converting world space coordinates into coordinates in front of a field of view of a user. Therefore, the view space is a space viewed from a field of view of a camera. In this case, a panning/rotation scenario is usually completed by using a series of displacement and rotation combinations, to enable a specific object to be transformed to the front of the camera. These combined transformations are usually stored in a view matrix (View Matrix) used to transform the world coordinates to the view space.” This teaches transforming the world vertex coordinates according to a camera view transformation. Zhang further teaches at (para. [0183]) “the auxiliary data may include a model matrix, a view matrix (or may also be referred to as a viewer matrix), and a projection matrix. These matrices are matrices that match the vertex data obtained by the CPU, and the vertex data that is in the local coordinate system and that is obtained by the CPU can be transformed to the clip coordinate system by using these matrices.” This teaches using model, view, and projection transformations to obtain clip-space vertex data. Zhang further teaches at (para. [0117]) “Finally, the clip coordinates are transformed into the screen coordinates. Next, a process referred to as viewport transform (Viewport Transform) is used. The viewport transform transforms the coordinates within the range from −1.0 to 1.0 to a coordinate range defined by a glViewport function.” This expressly teaches obtaining screen vertex coordinates in screen space.); “and rendering the display element based on the screen vertex coordinates,” Zhang teaches rendering graphical vertex data after transformation into screen coordinates (Zhang teaches at (para. [0117]) “The finally transformed coordinates are sent to a rasterizer to convert the coordinates into segments (after the coordinates are converted into the segments, a video image can be displayed based on the segments).” This expressly teaches rendering the display element based on the transformed screen coordinates. Zhang further teaches at (para. [0149]) “After the vertex data clipping is completed, clipped vertex data may be updated to vertex data. Specifically, after the clipping is completed, corresponding vertex (vertex) data and index (indices) data may be updated, and the data is sent to a rendering pipeline (pipeline) of a GPU as data input of a draw call instruction.” This teaches providing the transformed vertex data to the GPU rendering pipeline to render the resulting display element.).
However, Zhang does not expressly disclose “the display element being an element displayed at an unchanged position on a screen,” “determining a world position offset of the display element based on an offset mode of the world vertex coordinates in a view transformation process,” or “performing coordinate space transformation based on the world vertex coordinates and the world position offset to obtain screen vertex coordinates of the display element in a screen space.” Nguyen supplies “the display element being an element displayed at an unchanged position on a screen” by teaching a view-fixed display element maintained at a fixed position in the user’s field of view as the viewing orientation changes (Nguyen teaches at (para. [0051]) “View-fixed, as used herein, refers to a VR element being attached to a user’s viewing direction. As a user changes the orientation of her view, a view-fixed element remains in a fixed position in the user’s field of view and appears to move relative to the background VR environment.” This expressly teaches a display element whose screen-relative position remains unchanged during a change in the view. Nguyen further teaches at (para. [0078]) “In some embodiments, the instructor view overlay widget is view-fixed to provide a convenient, fixed point of reference representing a VR tutorial from the instructor’s point of view.” This reinforces maintaining a rendered display element at a fixed position relative to the user’s field of view. Nguyen further teaches at (para. [0088]) “While the video player overlay widget may be world-fixed, the awareness overlay widget may be rendered view-fixed.” This teaches selectively rendering a display element in a view-fixed mode so that the element remains fixed relative to the screen rather than moving with the virtual environment.). However, Nguyen still does not disclose the remaining claim elements.
Hladky supplies “determining a world position offset of the display element based on an offset mode of the world vertex coordinates in a view transformation process” by teaching calculation of camera offsets associated with camera movement, alteration of the view matrix, and changes in the projected location of vertices (Hladky teaches at (para. [0012]) “the invention introduces a camera offset space, which for a given point or sample location on the image plane provides information under which camera offset the point is covered by a given polygon.” This teaches determining offset information associated with a camera and relating that offset information to a graphical point or polygon projected at a location on the image plane. Hladky further teaches at (para. [0033]) “Considering a potential camera movement C′=[Cx′ Cy′ Cz′], the camera (view) matrix is altered.” This expressly teaches that camera movement causes an offset in the view-transformation process and that the offset is reflected in an altered view matrix. Hladky further teaches at (para. [0043]) “one may also consider points in COS, i.e., computing under which offsets a vertex’ projection exactly coincides with the sample location.” This teaches calculating an offset based on the relationship between a vertex projection and a selected screen location, thereby supplying the claimed determination of an offset associated with vertex displacement during view transformation.). Hladky further supplies or reinforces “performing coordinate space transformation based on the world vertex coordinates and the world position offset to obtain screen vertex coordinates of the display element in a screen space” by teaching coordinate transformation of object-space vertex coordinates through model, view, and projection matrices while accounting for camera offset, followed by conversion of the resulting clip-space coordinates into normalized device coordinates corresponding to screen locations (Hladky teaches at (para. [0034-35]) “A point in object space p0 is transferred to its clip space location p using the model (M), view (V) and projection matrix (P): p=P·V·M·p0.” This teaches coordinate-space transformation of vertex coordinates through the model, view, and projection matrices. Hladky further teaches at (para. [0043]) “To compute the normalized device coordinate (NDC) representation of a point in clip space, one simply needs to perform the perspective division.” This teaches obtaining screen-related vertex coordinates from the transformed clip-space coordinates. Hladky further teaches at (para. [0046]) “the camera offset space provides information about under which offsets a fragment is covered by a triangle or exactly meets the projection of a vertex.” This reinforces determining the resulting projected screen location of a vertex based on the applied camera offset.).
It would have been obvious to one of ordinary skill in the art before the effective filing data of the claimed invention to combine Zhang’s world-to-view-to-clip-to-screen vertex-rendering pipeline with Nguyen’s view-fixed display element and Hladky’s camera-offset-space calculations because all three references address rendering graphical elements relative to a changing camera or user viewpoint, and Nguyen expressly identifies the desirability of maintaining selected graphical elements as fixed points of reference in the user’s field of view. Implementing Nguyen’s view-fixed display element using Zhang’s world-space rendering pipeline would have been motivated to use Hladky’s teaching that camera movement alters the view matrix and that offsets may be calculated based on when a vertex projection coincides with a selected screen location. Applying a world position offset corresponding to the camera-induced displacement before completing Zhang’s view, projection, viewport, and rasterization operations would have been a predictable use of known coordinate-transformation techniques to maintain the desired screen position. The predictable technical benefit would be a computer-rendered display element that remains at an unchanged and readily visible position on the screen during camera translation or rotation while still being processed through the ordinary world-to-view-to-clip-to-screen graphics-rendering pipeline.
As per claim 7, the combination of Zhang, Nguyen and Hladky discloses all the elements of claim 1 as discussed above. Nguyen also discloses “disabling a depth test of the display element,” Nguyen provides the closest support by teaching that a VR-embedded widget is rendered as an overlay on top of the VR environment rather than as ordinary scene geometry subject to scene occlusion (Nguyen teaches at (para. [0003]) “one or more VR-embedded widgets can be rendered on top of a VR environment for an active VR host application.” This teaches rendering the display element above the scene content. Nguyen further teaches at (para. [0064]) “For example, after VR host application 220 has rendered its VR environment, but before outputting to VR display 210, injection component 240 can intercept the rendered environment, render one or more VR-embedded widgets on top, and provide this composite rendering for output on VR display 210.” This teaches compositing the display element after the underlying scene has been rendered, which is functionally consistent with preventing the display element from being rejected or hidden by the scene depth buffer.); “and after the depth test is disabled, rendering the display element to an upper layer of a scene picture,” Nguyen expressly teaches rendering the widget on top of the completed scene picture (Nguyen teaches at (para. [0037]) “VR-embedded widgets can inject a VR application’s rendering and event system so that the widgets can be rendered on top of the VR application.” This teaches placing the display element in an upper visual layer relative to the scene picture.). It teaches the claimed result—rendering the display element on top of the scene—through post-scene compositing or overlay rendering. It would have been obvious to one of ordinary skill in the art before the effective filing data of the claimed invention to implement Nguyen’s overlay rendering by disabling depth testing for the display element because disabling the depth test was a conventional way to ensure that overlay or UI geometry is not occluded by previously rendered scene geometry. This would be allow for reliably presenting the display element in an upper layer of the scene picture regardless of the depth values of objects in the underlying three-dimensional scene.
Claims 9 and 17 similar in scope to claim 1 thus rejected under the same rationale.
Regarding claim 9, claim 9 recites similar claim limitations to claim 1. However, claim 9 recites “A computer device, comprising a processor and a memory, the memory having at least one computer instruction stored therein, the at least one computer instruction being loaded and executed by the processor to implement a method for rendering a display element including.” Zhang also discloses this at para.[0064] “ According to a third aspect, a graphics rendering apparatus is provided. The apparatus includes a processor, and the processor is configured to invoke program code stored in a memory, to execute some or all operations in any one of the first aspect or the manners of the first aspect.” The rest of claim 9 is rejected under the same rationale as claim 1.
Regarding claim 17, claim 17 is similar in scope to claim 1. However, claim 17 recites “A non-transitory computer-readable storage medium, having at least one computer instruction stored therein, the at least one computer instruction being loaded and executed by a processor of a computer device to implement a method for rendering a display element rendering including.” Zhang also discloses this at para.[0071] “According to a fifth aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores program code, and the program code includes instructions used to execute some or all operations of the method described in the first aspect.” The rest of claim 17 is rejected under the same rationale as claim 1.
Claim 15 is similar in scope to claim 7, thus rejected under the same rationale.
Claim(s) 5,13, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang as modified by Nguyen and Hladky as applied to claim 1 above, and further in view of Canada(US-20190251753-A1).
As per claim 5, the combination of Zhang, Nguyen and Hladky disclose all the elements of claim 1 as discussed above. However, the combination does not disclose creating a UI camera at an origin of coordinates in the world space in response to a camera creation instruction, the UI camera being configured to photograph the display element;
presenting, when the UI camera photographs the display element within a scene, an element display position of the display element on the screen in a photographing picture of the UI camera; and
adjusting the element display position of the display element in the photographing picture when the display element is moved.
Canada does disclose creating a UI camera at an origin of coordinates in the world space in response to a camera creation instruction, the UI camera being configured to photograph the display element(Canada teaches at (para. [0047]) “In this example, the process can involve activating 604 a camera-inclusive device in response to the request. This can include, for example, activating or contacting control software for an overhead camera, or other such device capable of capturing data for the space to be mapped .” This teaches activating a camera in response to an instruction or request. Canada further teaches that a user-selectable element may activate an augmented-reality camera view. Although Canada does not expressly disclose creating a separate virtual UI camera, locating that camera at the origin of coordinates in world space, or configuring the camera specifically to photograph the display element because Canada teaches activating a camera in response to a request for previewing a rendered object, while Zhang, Nguyen, and Hladky provide the world-space, view-fixed, and camera-transformation framework in which that preview camera would operate. Positioning the UI camera at the world-space origin would have been a predictable implementation choice that provides a fixed reference frame, simplifies the camera transformation by avoiding an additional translation, and enables consistent preview and adjustment of the display element. );
presenting, when the UI camera photographs the display element within a scene, an element display position of the display element on the screen in a photographing picture of the UI camera(Canada teaches at (para. [0058]) “a camera view of the physical space may be displayed that includes a rendering of the physical space including the planar surface as captured by an image capture device (e.g., a live video stream), and a digital representation of an object displayed as if the object were sitting on the planar surface.”This expressly teaches presenting a camera-generated picture of a scene on a display and showing the display position of a rendered object within that camera picture. The digital representation corresponds to the claimed display element, while the displayed live camera view corresponds to the claimed photographing picture. ); and
adjusting the element display position of the display element in the photographing picture when the display element is moved(ef_4 teaches at (para. [0029]) “A user can interact with the sticker. For example, the user can rotate, enlarge, reduce in size, change the location of the sticker, etc.” This expressly teaches moving a rendered display element and changing its displayed position within the camera view. Canada further teaches at paragraph [0032] that virtual sticker renderings are placed within a live camera view and that a user may “change the location of the virtual sticker,” directly mapping to adjustment of the element display position in the photographing picture when the element is moved. ).
It would have been obvious to one of ordinary skill in the art before the effective filing data of the claimed invention to incorporate Canada’s camera-activated object-preview functionality into Zhang’s world-space rendering pipeline, Nguyen’s view-fixed UI arrangement, and Hladky’s camera/view transformation system because Canada provides a known mechanism for presenting and repositioning a rendered display element within a camera-generated scene view. The predictable technical benefit would be a stable and user-adjustable UI preview in which the display element is shown from a known camera reference and can be repositioned within the displayed camera picture while remaining integrated with the existing 3D rendering pipeline.
Claims 13 and 19 are similar in scope to claim 5, thus rejected under the same rationale.
Claim(s) 6 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang as modified by Nguyen, Hladky and Canada as applied to claim 5 above, and further in view of Sheblak (US-20110306417-A1)
As per claim 6, “setting, when an adopted particle model is a sprite particle, facing of the sprite particle to a fixed facing, the fixed facing being a direction towards the UI camera,” Sheblak teaches using a 2D sprite or imposter model in three-dimensional space and orienting that planar sprite toward a virtual camera (Sheblak teaches at (para. [0011]) “the 2D model can be oriented so its plane is perpendicular to the direction of the virtual camera.” This teaches setting the planar sprite to face the camera. Sheblak further teaches at (para. [0068]) “Draw camera facing quad (i.e., the 2D surface can be defined with different orientation depending on virtual camera location) with sprite sheet texture cell.” This expressly teaches a camera-facing quad carrying a sprite-sheet texture and adjusting the quad orientation based on the virtual camera location.). Sheblak also teaches that its 2D images may be “derived from or used to create respective sprite sheets” and that a sequence of sprite-sheet images may be texture-mapped onto a planar surface for animation, thereby supporting that the adopted model is a sprite-based graphical model (Sheblak teaches at (para. [0047]) “FIG. 7A shows respective 2D images for two respective characters 50, 52. These respective 2D images or “cels” can be derived from or used to create respective sprite sheets, i.e., collections or sequences of 2D images of each character in different positions that can be used to supply animation. ” This teaches the claimed sprite-based model.).
Accordingly, Sheblak substantially discloses claim 6 by teaching a sprite-sheet-based planar model whose facing is set toward a virtual camera. Sheblak does not expressly call the virtual camera a UI camera, but in the context of claim 5’s separately created UI camera, directing Sheblak’s known camera-facing sprite quad toward that UI camera would have been an obvious application of the same camera-facing orientation technique, with the predictable benefit of maintaining the sprite in a stable, front-facing orientation in the UI camera view.
Claim 14 is similar in scope to claim 6, thus rejected under the same rationale.
Claim(s) 8,16 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang as modified by Nguyen, and Hladky as applied to claim 1 above, and further in view of Moreton (US-20170323475-A1)
As per claim 8, the combination of Zhang, Nguyen and Hladky disclose all the elements of claim 1 as discussed above. However, the combination does not fully disclose “performing coordinate offset processing on the world vertex coordinates based on the world position offset to obtain updated world vertex coordinates; and performing observation space transformation, clip space transformation, and screen space transformation on the updated world vertex coordinates to obtain the screen vertex coordinates,”
Moreton provides the closest support by teaching performing coordinate offset processing on the world vertex coordinates based on the world position offset to obtain updated world vertex coordinates; (Moreton teaches at (para. [0032]) “an arithmetic operation is performed on the vertex coordinates to produce modified vertex coordinates in homogeneous coordinate space.” This teaches applying a coordinate-modification operation to original vertex coordinates to obtain updated vertex coordinates. Moreton further teaches at (para. [0074]) “The projective geometric distortion unit 715 transforms the vertex coordinates using the projective distortion factors to compute view-specific modified vertex coordinates.” This reinforces obtaining updated vertex coordinates through an offset-like, view-specific transformation.); “and performing observation space transformation, clip space transformation, and screen space transformation on the updated world vertex coordinates to obtain the screen vertex coordinates,” Zhang teaches the ordered transformation of vertex coordinates from world space through observation/view space and clip space into screen space (Zhang teaches at (para. [0116]) “After being transformed, the local coordinates are sequentially changed into world coordinates (World Coordinate), view coordinates (View Coordinate), and clip coordinates (Clip Coordinate), and a coordinate transformation process finally ends in a form of screen coordinates (Screen Coordinate).” This expressly teaches the claimed sequence of observation-space, clip-space, and screen-space transformations. Zhang further teaches at (para. [0116]) “Finally, the clip coordinates are transformed into the screen coordinates.” This confirms obtaining screen vertex coordinates after the clip-space transformation.). Zhang also teaches transforming world-coordinate vertex data into viewer-coordinate data and then into clip-coordinate data (Zhang teaches at (paras. [0188]-[0193]) transforming vertex data from the local coordinate system into the world coordinate system, then “from the world coordinate system to the viewer coordinate system,” and then “from the viewer coordinate system to the clip coordinate system.” This directly maps to the claimed world-to-observation-to-clip transformation sequence.)
It would have been obvious to one of ordinary skill in the art before the effective filing data of the claimed invention to apply Moreton’s known vertex-coordinate modification before Zhang’s conventional world-to-view-to-clip-to-screen transformation pipeline because vertex-coordinate adjustments are ordinarily performed upstream of projection and rasterization so that the modified geometry is reflected in the final screen coordinates. The predictable technical benefit would be updated screen vertex coordinates that incorporate the desired world-space positional adjustment while retaining the conventional rendering pipeline.
Claims 16 and 20, are similar in scope to claim 8, thus rejected under the same rationale.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHRIS ALEJANDRO PUNTIER whose telephone number is (703)756-1893. The examiner can normally be reached M-F 7:30-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Hajnik can be reached at 571-272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHRIS ALEJANDRO PUNTIER/ Examiner, Art Unit 2616
/DANIEL F HAJNIK/ Supervisory Patent Examiner, Art Unit 2616
Prosecution Insights