DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 6-7, 8-10, 13-14, 15-17, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nishimoto et al. (US 20220365660 A1), and in view of Kellogg et al. (US 20190026922 A1).
Regarding Claim 8. Nishimoto discloses An electronic device for automatically converting a two-dimensional (2D) user interface (UI) feature to a three-dimensional (3D) UI feature suitable for an extended reality (XR) environment (ABST reciting “Systems and methods for generating an extended reality (XR) user interface are disclosed. A two-dimensional data set is imported. The two-dimensional data set defines a two-dimensional user interface design layout. The two-dimensional data set includes a transition data set corresponding to a user interface element included in the design layout. The two-dimensional data set is converted into a three-dimensional data set. ”), the electronic device comprising:
at least one processing device configured to: (¶39 reciting “shown in FIG. 2, is a flowchart of a method 200 for designing an XR user interface. The method 200 may be used in conjunction with the XR user interface design system 100 as described with respect to FIG. 1A and FIG. 1B.”)
obtain a design for the 2D UI feature; (¶41 reciting “ At operation 202, a two-dimensional user interface design layout, such as a wireframe layout, is received”)
analyze elements of the design for the 2D UI feature;
determine spatial positions in a 3D XR space corresponding to each element of the design for the 2D UI feature;
(¶44 reciting “When images and data are loaded, they may often have Anchors and positioning associated with them. . . Various calculations and processes may be used to solve this problem. For one, if the repository is shared over a network, usage statistics of how designers manipulate the image and objects may be stored and useful data extracted from the usage on how the image should render. For example, say a file of an image is rendered, but always rendered upside down and too far left of a user's workspace in the UI. If, when accessing the file, users usually rotate the image and move it to the middle of a workspace, the system 100 may learn (e.g., through generation and application of a machine-learned model) to begin rendering the object the “proper” way in accordance with the typical behavior of the users of the system.”)
and render elements for the 3D UI feature at the determined spatial positions to form the 3D UI feature. (¶42 reciting “At operations 204, the two-dimensional user interface design is converted into a three-dimensional design (e.g. for each user interface element in the design).”)
However, Nishimoto does not explicitly disclose to determine spatial coordinates in a 3D XR space.
The feature is obvious to a person skilled in the art. In addition, Kellogg teaches “AR systems and methods that enable efficient tracking of feature points among images (e.g., consecutive frames within a video) of natural and/or never-before-seen surroundings, 3D model matching based on the tracked feature points, and AR content rendering using positional information of the matched 3D model. ” (¶12). Further, ¶35 recites “ the AR system 100 generates 3D point cloud corresponding to the target frame based at least in on the 2D point cloud. As discussed above, the AR system 100 can convert the 2D point cloud into a 3D point cloud in a 3D reference system (e.g., 3D coordinate system) associated with the real-world environment or associated with the AR device 102 or its camera. ”
It would have been obvious to one with ordinary skill, before the effective filing date of the claimed invention, to modify the device (taught by Nishimoto) to render elements for the 3D UI feature at the determined spatial coordinates to form the 3D UI feature (taught by Kellogg). The suggestions/motivations would have been “In order to accurately and promptly superimpose AR content in various contexts ” (¶11), and to apply a known technique to a known device (method, or product) ready for improvement to yield predictable results.
Regarding Claim 9. Nishimoto in view of Kellogg discloses The electronic device of claim 8, wherein the at least one processing device is further configured to:
determine interactions for the elements of the 3D UI feature in the 3D XR space; and
translate the interactions into commands based on corresponding commands for the elements of the design for the 2D UI feature, wherein the translation is based at least in part on one or more user considerations.
(Nishimoto, ¶48 reciting “ the converted three-dimensional version of the user interface layout is presented within an XR environment for viewing and editing using one or more XR devices. In example embodiments, a user may use a writing implement or controller to select and modify an AR object. The controller can further be used to select UI features that a user may further use to change AR or VR objects. For example, the user can make selections such as color, texture, size, actions, or other features associated with the AR or VR object by selecting them from a UI generated in the AR/VR environment. In other instances, certain gestures may be preprogrammed to be associated with the selection of certain features.”)
Regarding Claim 10. Nishimoto in view of Kellogg discloses The electronic device of claim 8, wherein, to analyze the elements of the design for the 2D UI feature, the at least one processing device is configured to identify, using an artificial intelligence (AI) or machine learning (ML) model, at least one of: buttons, text, graphics, backing, statistics, time/date, chunking, decorative elements, or media display. (Nishimoto, Claim 6 reciting “ wherein the converting of the two-dimensional data set into a three-dimensional data set includes using statistics of previous content manipulation to determine the three-dimensional user interface design layout.” Further, Claim 7 reciting “ The system of claim 6, wherein the converting of the two-dimensional data set includes applying a machine learning model to the statistics to determine the three-dimensional user interface design layout.”)
Regarding Claim 13. Nishimoto in view of Kellogg discloses The electronic device of claim 8, wherein the at least one processing device is further configured to adjust at least one of a size or an orientation of the elements for the 3D UI feature in the 3D XR space based on a user experience during interactions with the 3D UI feature. (Nishimoto, ¶44 reciting “For example, say a file of an image is rendered, but always rendered upside down and too far left of a user's workspace in the UI. If, when accessing the file, users usually rotate the image and move it to the middle of a workspace, the system 100 may learn (e.g., through generation and application of a machine-learned model) to begin rendering the object the “proper” way in accordance with the typical behavior of the users of the system.”)
Regarding Claim 14. Nishimoto in view of Kellogg discloses The electronic device of claim 8, wherein: the at least one processing device is further configured to add one or more dynamic effects to one or more of the elements of the 3D UI feature; and
the one or more dynamic effects include at least one of: animation, shadowing, reflection, or ambient interaction based on user presence.
(Nishimoto, ¶56 reciting “They are also able to replace the graphics with 3D extrusions, or objects from a library, and apply behaviors and animations to each individual graphic, or to groups of graphics.”)
Claim 1, has similar limitations as of Claim(s) 8, therefore it is rejected under the same rationale as Claim(s) 8.
Claim 2, has similar limitations as of Claim(s) 9, therefore it is rejected under the same rationale as Claim(s) 9.
Claim 3, has similar limitations as of Claim(s) 10, therefore it is rejected under the same rationale as Claim(s) 10.
Claim 6, has similar limitations as of Claim(s) 13, therefore it is rejected under the same rationale as Claim(s) 13.
Claim 7, has similar limitations as of Claim(s) 14, therefore it is rejected under the same rationale as Claim(s) 14.
Claim 15, has similar limitations as of Claim(s) 8, therefore it is rejected under the same rationale as Claim(s) 8. (Nishimoto, ¶22 reciting “ The present invention includes apparatuses which perform one or more operations or one or more combinations of operations described herein, including data processing systems which perform these methods and computer readable media which when executed on data processing systems cause the systems to perform these methods, the operations or combinations of operations including non-routine and unconventional operations.”)
Claim 16, has similar limitations as of Claim(s) 9, therefore it is rejected under the same rationale as Claim(s) 9.
Claim 17, has similar limitations as of Claim(s) 10, therefore it is rejected under the same rationale as Claim(s) 10.
Claim 20, has similar limitations as of Claim(s) 13, therefore it is rejected under the same rationale as Claim(s) 13.
Claim(s) 4, 11, and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nishimoto et al. (US 20220365660 A1), in view of Kellogg et al. (US 20190026922 A1), and further in view of Chan (US 20230144893 A1).
Regarding Claim 11. Nishimoto in view of Kellogg discloses The electronic device of claim 8.
However, Nishimoto in view of Kellogg does not explicitly disclose wherein, to determine the spatial coordinates in the 3D XR space, the at least one processing device is configured to identify, using an artificial intelligence (AI) or machine learning (ML) model, depth values for the elements for the 3D UI feature based on one or more depth profiles.
Chan teaches “Methods and systems described herein are directed to creating an artificial reality environment having elements automatically created from source images.” (ABST). ¶20 recites “creating an artificial reality environment by analyzing one or more source images and automatically creating corresponding elements in the artificial reality environment.” More specifically, Chan recites “To generate the 3D model, the creation system can implement a second machine learning model trained to predict depth for real objects to be represented by the created second virtual objects. In some cases, depth data can be included in the image, in which case no depth estimation may be needed or may only be needed for portions of the object occluded in the image(s). Here, the second machine learning model can be trained using training data where the depth profiles for real objects can be determined from a multitude of prior training images. ” (¶26).
It would have been obvious to one with ordinary skill, before the effective filing date of the claimed invention, to modify the device (taught by Nishimoto in view of Kellogg) to predict depth value of a 3D object based on depth profiles (taught by Chan). The suggestions/motivations would have been to apply a known technique to a known device (method, or product) ready for improvement to yield predictable results.
Claim 4, has similar limitations as of Claim(s) 11, therefore it is rejected under the same rationale as Claim(s) 11.
Claim 18, has similar limitations as of Claim(s) 11, therefore it is rejected under the same rationale as Claim(s) 11.
Claim(s) 5, 12, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nishimoto et al. (US 20220365660 A1), in view of Kellogg et al. (US 20190026922 A1), and further in view of Chan (US 20230144893 A1), and further in view of Taylor et al. (US 20220244903 A1).
Regarding Claim 12. Nishimoto in view of Kellogg and Chan discloses The electronic device of claim 11.
However, Nishimoto in view of Kellogg and Chan doesn’t explicitly disclose wherein the one or more depth profiles are based on a hierarchy of the elements for the 3D UI feature.
The features is obvious to a person of skilled in the art. In addition, Taylor teaches “For example, a layer tree may include and/or may be associated with a hierarchy of layers that describes each layer of the UI for display. For example, the UI may include content in conjunction with a backdrop, which may utilize one or more blur layers and/or other filter layers. Thus, the tree may include a node and/or subtree that contains one or more attributes describing the blur layer, such as depth, size, placement, and the like.” (¶47).
It would have been obvious to one with ordinary skill, before the effective filing date of the claimed invention, to modify the device (taught by Nishimoto in view of Kellogg and Chan) to include a depth value in nodes of a UI hierarchy tree (taught by Taylor). The suggestions/motivations would have been to apply a known technique to a known device (method, or product) ready for improvement to yield predictable results.
Claim 5, has similar limitations as of Claim(s) 12, therefore it is rejected under the same rationale as Claim(s) 12.
Claim 19, has similar limitations as of Claim(s) 12, therefore it is rejected under the same rationale as Claim(s) 12.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YI WANG whose telephone number is (571)272-6022. The examiner can normally be reached 9am - 5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jason Chan can be reached at (571)272-3022. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YI WANG/Primary Examiner, Art Unit 2619
Prosecution Insights