DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-8, 10-15 and 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over Vorhees et al. (US PGPUB 2017/0109933; hereinafter “Vorhees”) in view of Stump et al. (US PGPUB 2021/0096553; hereinafter “Stump”), Bond et al. (US Patent 11,449,189; hereinafter “Bond”) and Lonial et al. (US PGPUB 2020/0319857; hereinafter “Lonial”).
Claim 1: (Currently Amended)
Vorhees teaches a computer-implemented method for collaborative software development in mixed reality, the method comprising:
rendering a three-dimensional integrated development environment (3D IDE) within a mixed-reality environment ([0027] “virtual representation of code is provided in virtual reality or in augmented reality environment that a user accesses using a display goggle device. Using one or more controllers, a user is able to navigate (e.g., expand and collapse nodes) and explore the virtual representation of code to gain an understanding of the code. The user is able to also execute the code and visualize and animate the execution in the virtual reality or augmented reality environment,” wherein “augmented reality” is a type of “mixed-reality”.),
wherein code comprising the 3D IDE is rendered as a plurality of virtual objects within the mixed-reality environment ([0050] “At 204, the processed code is utilized to generate a hierarchical graphical representation data of the code… and maps one or more elements of the code as nodes (e.g., vertices) and relationships between the objects are represented as edges (e.g., connections)… elements that the nodes represent include file directories, files, packages, classes, methods, functions, expressions, statements, data structures, fields, variables, etc. The edges/links between the nodes may represent a hierarchical relationship. For example, edges may represent a hierarchical file directory structure, a categorization structure, a programming language level abstraction, a layer structure, and any other hierarchical level relationship.” [0136] “FIG. 7E is an illustration of an embodiment of a rendered animation visualization of an execution of computer code.” [0137] “The example animation sequence concludes with fourth image (735) wherein the point of execution escapes the entire module as shown by the highlighted edge (736).”);
monitoring interactions between the plurality of developers and the 3D IDE within the mixed-reality environment ([0027] “This visual environment may also be shared with other users/programmers to facilitate a virtual collaboration between the users. For example, each user is placed in the same virtual environment and interaction by one user with the virtual representation is reflected in the virtual representation provided to all users.” [0039] “two or more users utilize the same client (e.g., using separate interface displays (110) connected to the same client (106)) to share the same and collaborate using a shared graphical representation of the code.”); and
wherein the interactions include one or more of the developers assisting another of the developers in creating a program by placing corresponding virtual objects into the mixed-reality environment ([0077] “When a program object is created during execution, a rendered node corresponding to the program object is created in visual animation. In some embodiments, even in object oriented rendering mode, starting nodes corresponding to package(s), director(ies), class(es), and/or static method(s) are visually rendered to serve as an anchoring basis for created program object nodes,” wherein the “code representations” which correspond to the “virtual objects” are taught by Lonial, as further discussed below.),
or by manipulating one or more of the placed virtual objects or one or more connectors ([0085] “User input from the user (106) from the virtual graphical environment (316), for example from a wand (112), is used to edit/analyze program to edit and/or analyze code (318). In the event that code is to be changed, code (306) is modified… revise the tree (314) and subsequent graphical representation in the virtual graphical environment (316).” [0037] “When a user modifies a state of the graphical representation (e.g., user interacts with a node/object of the graphical representation), this modification may be indicated to server (102) by a client of the user that modified the state and an updated graphical representation is provided to all of the clients to allow other users to visualize the state change,” wherein the “user”, i.e. “developer”, is able to assist another one of the “users” by modifying/interacting with a “node” representing code, i.e. “manipulating one or more placed virtual objects”.); and
responsive to the interactions, updating and displaying changes to the 3D IDE within the mixed-reality environment ([0085] “In the event that code is to be changed, code (306) is modified, triggering a re-input to the DLL (312) to revise the tree (314) and subsequent graphical representation in the virtual graphical environment (316).”).
With further regard to Claim 1, Vorhees does not teach the following, however, Stump teaches:
responsive to confirming credentials of a plurality of developers, admitting the plurality of developers to the mixed-reality environment ([0111] “a wearable AR/VR appliance 1510 can interface with industrial IDE system 202 via user interface component 204… user interface component 204 may be configured to verify an authorization of the wearable appliance 1510 to access the IDE system 202 prior to allowing VR presentations to be delivered to the wearable appliance 1510. User interface component 204 may authenticate the wearable appliance 1510 or its owner using password verification, biometric identification (e.g., retinal scan information collected from the user by the wearable appliance 1510 and submitted to the user interface component 204), cross-referencing an identifier of the wearable appliance 1510 with a set of known authorized devices, or other such verification techniques.”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method as disclosed by Vorhees with the developer credentials as taught by Stump so that “The IDE system can implement appropriate security features to ensure that users at a first enterprise do not share sensitive proprietary system information with users at other enterprises, while still allowing a managed degree of information sharing to facilitate crowdsourced development assistance” (Stump [0125]).
With further regard to Claim 1, Vorhees in view of Stump does not teach the following, however, Bond teaches:
wherein the plurality of developers are represented as a plurality of avatars within the mixed-reality environment; and monitoring interactions between the plurality of developers and the 3D IDE through the plurality of avatars (Col. 6 Ln. 8: “augmented reality content development system 22 may render developer 8 and/or a hypothetical end-user of the augmented reality system in the simulated augmented reality location under development as an avatar.” Col. 8 Ln. 44: “In the multi-developer environment represented by VR system 20, augmented reality IDEs executing on console 16 and/or HMDs 12 present respective VR content (optionally, with augmented reality content overlays) to each of developers 8A-8C (collectively, ‘developers 8’) based on a current viewing perspective of a corresponding frame of reference for the respective developer 8 and based on the parameters and present development state of the augmented reality tool with respect to a target physical location and, optionally, target future augmented reality viewing devices, such as augmented reality glasses 2. That is, in the example of FIG. 1B, each of developers 8 may contribute to the development of the augmented reality tool, by providing input via respective HMDs 12 and/or combinations of respective HMDs 12 with respective peripheral devices communicatively coupled to the respective HMDs 12.”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method as disclosed by Vorhees in view of Stump with the developer avatars as taught by Bond in order to “virtual reality-based development tools that aid in the development of augmented reality systems” (Bond Col. 1 Ln. 7).
With further regard to Claim 1, Vorhees in view of Stump and Bond does not teach the following, however, Lonial teaches:
creating a program by selecting one or more code representations from a menu and placing corresponding virtual objects ([0073] “the code notebook 320 includes a code editor 355.” [0074] “code editor 355 enables the user to enter computer code text and save it for interpretation and execution through the graphical user interface 105.” [0125] “GUI 900 includes several selectable menus, such as back-end data objects menu 905, data analytics functions menu 910, and visualization & reporting menu 915. GUI 900 includes a code notebook 920 for drafting code 925 and displaying data visualizations 930 or other results.” [0126] “Objects listed in the menus or the code notebook can be selected for placement into the application window 935 (for example by drag and drop) to assemble applications. For example, object icons 940 and 945 were selected from back-end data objects menu 905 and placed into application window 935,” wherein the “Objects listed in the menus or the code notebook” are the “one or more code representations,” and further wherein the “object icons”, as shown in Lonial Fig. 9, are the “corresponding virtual objects,” i.e. similar to the virtual object nodes as taught above in Vorhees.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method as disclosed by Vorhees in view of Stump and Bond with the selecting of code representations from a menu as taught by Lonial in order to “enable data scientists to employ a broad range of analyses, including the latest artificial intelligence advancements, without requiring specialized programming skills” (Lonial [0018]) and further since this “makes it easy to build intelligent, cloud-based applications” (Lonial [0017]).
Claim 3:
Vorhees in view of Stump, Bond and Lonial teaches the method of claim 1, and Vorhees further teaches:
wherein the 3D IDE supports one or more visual programming languages ([0147] “In one embodiment, the graphical programming software and the computer hardware are the only components needed to begin graphical debugging.” [0148] “One or more users could visualize code either written with the graphical programming software or written in another language and then imported into the graphical programming software.”).
Claim 4:
Vorhees in view of Stump, Bond and Lonial teaches the method of claim 1, and Vorhees further teaches:
wherein the interactions comprise gestures performed by the plurality of developers ([0140] “A radial pointer arm (744) is a pointer that can be controlled by a user to point to a location on timeline arc (741) by providing an input (e.g., circular gesture) via a controller (e.g., controller (112) of FIG. 1A).”).
Claim 5:
Vorhees in view of Stump, Bond and Lonial teaches the method of claim 1, and Vorhees further teaches:
wherein the 3D IDE comprises one or more visualizations illustrating dependencies, tasks, and/or components comprising a plurality of software under development within the 3D IDE ([0065] “the virtual machine has been configured to output information that can be used to visually indicate current execution state (e.g., stack trace, dependency information, memory information, field/variable values, current method/function being executed, machine state, etc.).” [0098] “The graphical representation program (360) includes the virtual graphical environment (316) configured to render static code, show dependency trees using the FDGs, and animate the code as it is running on an external VM.” See also Vorhees Figs. 1B and 7A-7F.).
Claim 6:
Vorhees in view of Stump, Bond and Lonial teaches the method of claim 1, and Vorhees further teaches:
wherein a plurality of the developers are enabled to simultaneously modify a plurality of code comprising a module of the 3D IDE ([0027] “This visual environment may also be shared with other users/programmers to facilitate a virtual collaboration between the users. For example, each user is placed in the same virtual environment and interaction by one user with the virtual representation is reflected in the virtual representation provided to all users.” [0039] “the server (102) may be also configured to enable collaborative interaction between a first user (106a) and a second user (106b). For example, within the graphical representation, each user may be assigned a color in terms of a cursor, breakpoints, watchpoints, etc. In some embodiments, two or more users utilize the same client (e.g., using separate interface displays (110) connected to the same client (106)) to share the same and collaborate using a shared graphical representation of the code.” [0149] “Multiple users may view the same code in the graphical programming environment and either with traditional displays or head mounted displays, and hand/physical trackers could explore and modify the code collaboratively.”).
Claim 7:
Vorhees in view of Stump, Bond and Lonial teaches the method of claim 1, and Vorhees further teaches:
wherein the changes to the 3D IDE are updated and displayed in real time ([0027] “This visual environment may also be shared with other users/programmers to facilitate a virtual collaboration between the users. For example, each user is placed in the same virtual environment and interaction by one user with the virtual representation is reflected in the virtual representation provided to all users.” [0037] “When a user modifies a state of the graphical representation (e.g., user interacts with a node/object of the graphical representation), this modification may be indicated to server (102) by a client of the user that modified the state and an updated graphical representation is provided to all of the clients to allow other users to visualize the state change.”).
Claims 8 and 10-14: (Currently Amended)
With regard to Claims 8 and 10-14, these claims are equivalent in scope to Claims 1 and 3-7 rejected above, merely having a different independent claim type, and as such Claims 8 and 10-14 are rejected under the same grounds and for the same reasons as discussed above with regard to Claims 1 and 3-7.
With further regard to Claim 8, the claim recites additional elements not specifically addressed in the rejection of Claim 1. The Vorhees reference also anticipates these additional elements of Claim 8, for example, wherein the computer system comprises:
one or more processors, one or more computer readable memories, one or more mixed-reality devices, one or more computer readable tangible storage medium, and program instructions stored on at least one of the one or more tangible storage medium for execution by at least one of the one or more processors via at least one of the one or more memories, wherein the computer system is capable of performing a method ([0023] “The invention can be implemented in numerous ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer readable storage medium; and/or a processor, such as a processor configured to execute instructions stored on and/or provided by a memory coupled to the processor,” see also Claim 20 of Vorhees which recites, “A computer program product, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for…” [0030] “Each client (106) is connected to a user interface display (110) for a user to view and interact with the graphical representation of code.” [0031] “Examples of user interface display (110) include:” [0033] “2. an augmented reality display (110b) that allows the viewer to view images that are positionally-tracked to user motion and permits a user to view virtual images atop of a physical view, for example, an augmented reality headset such as the Microsoft HoloLens, Meta headset, castAR Headset, Epson Moverio, etc.”).
Claims 15 and 17-20: (Currently Amended)
With regard to Claims 15 and 17-20, these claims are equivalent in scope to Claims 1 and 3-6 rejected above, merely having a different independent claim type, and as such Claims 15 and 17-20 are rejected under the same grounds and for the same reasons as discussed above with regard to Claims 1 and 3-6.
With further regard to Claim 15, the claim recites additional elements not specifically addressed in the rejection of Claim 1. The Vorhees reference also anticipates these additional elements of Claim 15, for example, wherein the computer program product comprises:
one or more computer readable tangible storage medium and program instructions stored on at least one of the one or more tangible storage medium, the program instructions executable by a processor to cause the processor to perform a method ([0023] “The invention can be implemented in numerous ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer readable storage medium; and/or a processor, such as a processor configured to execute instructions stored on and/or provided by a memory coupled to the processor,” see also Claim 20 of Vorhees which recites, “A computer program product, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for…”).
Response to Arguments
Applicant's arguments, see Pages 8-9 of the Remarks filed January 20, 2026, with respect to the rejections under 35 U.S.C. 103 of Claims 1, 3-8, 10-15 and 17-20 have been fully considered but they are not persuasive.
With respect to the Applicant’s argument that the newly amended language of Claims 1, 8 and 15 which recites, “creating a program by selecting one or more code representations from a menu and placing corresponding virtual objects” is not taught by the previously cited prior art, this argument has been fully considered but is moot in view of the newly cited Lonial et al. (US PGPUB 2020/0319857) reference as discussed above in the respective rejections.
With respect to the newly amended language of Claims 1, 8 and 15 which goes on to recite, “placing corresponding virtual objects into the mixed-reality environment,” the Office contends that the previously cited Vorhees et al. (US PGPUB 2017/0109933) reference does teach this portion of the newly amended language. For example, Vorhees recites the following:
[0077] “When a program object is created during execution, a rendered node corresponding to the program object is created in visual animation. In some embodiments, even in object oriented rendering mode, starting nodes corresponding to package(s), director(ies), class(es), and/or static method(s) are visually rendered to serve as an anchoring basis for created program object nodes,” wherein the “code representations” which correspond to the “virtual objects” are taught by Lonial, as discussed above.
As such, in view of at least the above citations, the Office has shown that Vorhees in view of Lonial does teach and make obvious the newly amended limitation of Claims 1, 8 and 15 which recites, “wherein the interactions include one or more of the developers assisting another of the developers in creating a program by selecting one or more code representations from a menu and placing corresponding virtual objects into the mixed-reality environment”. Therefore, the Office now contends that Vorhees in view of Stump, Bond and Lonial teaches and makes obvious the limitations in newly amended Claims 1, 8 and 15.
With respect to the Applicant’s further argument, Page 9 of the Remarks, that the features of the dependent claims are not taught by the cited prior art, the Office respectfully disagrees. These arguments rely upon the arguments as presented in relation to the independent claims discussed above, and as such the Office directs the Applicant to the responses above regarding these arguments.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure is as follows:
Kerber, III et al. (US PGPUB 2021/0042023) discloses systems and methods for projecting a mixed reality environment having a plurality of objects to a headset, and displaying in the mixed reality view an action menu having a plurality of selectable objects associated with various coding functions.
Dominic et al. (“Remote Pair Programming in Virtual Reality,” 2020) discusses an exploration of remote pair programming and code comprehension in virtual reality, including discussion regarding an experiment wherein half of the participants used VR as a tool and the other half used a screen sharing system.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Joanne G. Macasiano whose telephone number is (571)270-7749. The examiner can normally be reached Monday to Thursday, 10:30 AM to 6:00 PM Eastern Standard Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Bradley Teets can be reached at (571) 272-3338. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOANNE G MACASIANO/ Examiner, Art Unit 2197
Prosecution Insights