DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-6, 9-10 and 14-15 and 17-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kokkevis et al1 (“Kokkevis”) in view of McKegney et al2 (“McKegney”).
Regarding claim 1, Kokkevis teaches a computer implemented method of simulating a three-dimensional (3D) object for display in a browser on a client’s device (note that method steps below address the simulating of the 3D object for display as well as a browser and client’s device as functioning below and thus the limitations of the preamble are addressed in the body of the claim below), the method comprising the initial steps of:
generating a 3D object and storing the 3D object in memory (note that the claim does not define that the memory is necessarily of the client’s device or any other specific device and thus any storing of such data in any memory which allows the loading recited below is such storing in memory – furthermore note that this memory is not considered to necessarily be the same “memory” as the “physics libraries in memory” as it is not recited as “a memory” and the following “memory” is also not ever referred to more specifically as “the memory” such that recitation of such “memory” is a broad limitation that storage in memory must occur in some manner at least; see Kokkevis, paragraphs 0052-0056 and figure 3 teaching “3D application 302 may provide an input file that describes a graphics model to graphics plugin 306. Alternatively, 3D application 302 may make a set of method calls that describe the graphics model to graphics plugin 306. The graphics model may then be loaded into the internal memory of graphics plugin 306” such that here the “graphics model” is a 3D object which has been generated at this time or previously in order for it to be “loaded” from somewhere and is stored in the “internal memory” after being loaded from elsewhere meaning that it was unloaded from some other memory as well);
storing a plurality of physics libraries in memory (note that a physics library in the context of the claims is a library in the sense that it is a collection of something to be referenced and in which that thing to be refenced deals with the subject of physic in some way, and in the context of the claims this may take a more narrow form such as for example any collection of code, scripting or the like which can be used for physics calculations; see Kokkevis, paragraphs 0052-0058 teaching “3D application 302 coordinates the joint execution of physics engine 310 and rendering engine 312” and “may make method calls to both plugin 304 and graphics plugin 306 for loading a graphics model into rendering engine 312 and a corresponding physics model into physics engine 310” such that here a “physics” model that corresponds to the 3D graphic model is stored through such loading into the internal memory as well and thus all such collections of physics parameters and related data correspond to a plurality of physics libraries in memory as for example “To animate the graphics model, physics engine 310 may read from IMC buffers 314-316 to create a physics model corresponding to the graphics model in graphics plugin 306. Additional information related to the physics model, such as parameters, may be obtained from 3D application 302 by plugin 304. Next, physics engine 310 may perform a series of physics simulation calculations that update the physics model. For example, physics engine 310 may calculate vertex positions and velocities based on a set of forces acting on objects in the physics model. Plugin 304 may then update IMC buffers 314-316 with new vertex positions, velocities, and/or other data” such that here for any given objects or its interaction of the multiple objects and interactions disclosed, physics libraries are stored in memory which allow the proper calculations by the physics engine);
storing an in-browser simulation application programming interface (API) linking an in-browser physics engine and an in-browser rendering engine, and a simulation computer program code, in memory (here a simulation API is a set of rules or code or even an arrangement of elements as an interworking architecture that allows different components to communicate through facilitating communication in some manner; see Kokkevis, paragraphs 0052-0058 teaching such simulation application programming interface as the “3D Application 302” as configured to interact and communication with the physics and rendering engines as in figure 3 for example as “3D application 302 may correspond to a web application that executes in a web browser 300” and “3D application 302 may provide 3D graphics rendering and animation capabilities to a user of 3D application” where this may further comprise “CAD system, and/or a scientific modeling and/or simulation application” according to the particular application 302 where such arrangement as described with respect to figure 3 is a stored simulation API linking a physics engine “physics engine 310” and a rendering engine “rendering engine 312” and for example this is stored at the user device upon downloading from the internet where “physics engine 310 corresponds to a native code module that is executed within a secure runtime environment provided by plugin 304. Physics engine 310 may be provided by 3D application 302 (e.g., downloaded over the Internet) and validated prior to execution in plugin 304” such that here the API and linked engines and code are stored in the memory of the user device as well as from the internet where they were stored in some functional memory in order to be downloaded ); and
on a web page load request issued through the browser on the client’s device (see Kokkevis, paragraphs 0059-0062 teaching “Initially, a web application is loaded into a web browser (operation 402). The web application may be obtained from a server by the web browser. Furthermore, the web application may be used to provide computationally intensive features, such as financial modeling, computational math or science, and/or AI, to a user. To implement such features in a practical manner, a native code module associated with the web application may be obtained (operation 404). For example, the native code module may be downloaded from a source specified by the web application” such that here a web page load request issued through a browser on the client’s device is considered to have taken place which is a request which lead to loading of the API as explained above; see also paragraphs 0052-0056 teaching a web page load request issued through the browser on the client’s device where the “web-based 3D application 302” is so loaded to execute on the client’s device on the browser):
loading the an in-browser simulation application programming interface (API) and the simulation computer program code onto the client’s device (see Kokkevis, paragraphs 0052-0058 teaching such loading in connection with the storing explained above teaching such simulation application programming interface as the “3D Application 302” as configured to interact and communication with the physics and rendering engines as in figure 3 for example as “3D application 302 may correspond to a web application that executes in a web browser 300” and “3D application 302 may provide 3D graphics rendering and animation capabilities to a user of 3D application” where this may further comprise “CAD system, and/or a scientific modeling and/or simulation application” according to the particular application 302 where such arrangement as described with respect to figure 3 is a stored simulation API linking a physics engine “physics engine 310” and a rendering engine “rendering engine 312” and for example this is stored at the user device upon loading from the internet where “physics engine 310 corresponds to a native code module that is executed within a secure runtime environment provided by plugin 304. Physics engine 310 may be provided by 3D application 302 (e.g., downloaded over the Internet) and validated prior to execution in plugin 304” such that here the API and application 302 which may be for “scientific modeling and/or simulation” is loaded to the client’s device in order that they might run the simulation through the browser of the client device);
importing the 3D object and at least one of the physics libraries to the client’s device (see Kokkevis, paragraphs 0052-0058 where the 3D object model is imported to the client’s device as explained above teaching “3D application 302 may provide an input file that describes a graphics model to graphics plugin 306. Alternatively, 3D application 302 may make a set of method calls that describe the graphics model to graphics plugin 306. The graphics model may then be loaded into the internal memory of graphics plugin 306” such that here the “graphics model” is a 3D object imported to the device where it will then be operated on in connection with the imported physics library such as the “physics model” where “3D application 302 coordinates the joint execution of physics engine 310 and rendering engine 312” and “may make method calls to both plugin 304 and graphics plugin 306 for loading a graphics model into rendering engine 312 and a corresponding physics model into physics engine 310” such that here a “corresponding physics model” is a physics library particular to the simulation of the model being performed and is thus imported to the client’s device and for example such libraries are also imported to the client’s device as well where “Graphics plugin 306 may then load data relevant to physics simulation into IMC buffers 314-316. For example, graphics plugin 306 may copy vertex positions, normals, triangle indices, and/or transformation matrices into IMC buffers 314-316” and “physics engine 310 may read from IMC buffers 314-316 to create a physics model corresponding to the graphics model in graphics plugin 306. Additional information related to the physics model, such as parameters, may be obtained from 3D application 302 by plugin 304. Next, physics engine 310 may perform a series of physics simulation calculations that update the physics model. For example, physics engine 310 may calculate vertex positions and velocities based on a set of forces acting on objects in the physics model”);
simulating the 3D object wholly in-browser using the in-browser simulation API, the in-browser rendering engine, and the in-browser physics engine having access to the imported physics library (see Kokkevis, paragraphs 0052-0058 as explained above teaching “3D application 302 may provide an input file that describes a graphics model to graphics plugin 306. Alternatively, 3D application 302 may make a set of method calls that describe the graphics model to graphics plugin 306. The graphics model may then be loaded into the internal memory of graphics plugin 306” such that here the “graphics model” is a 3D object imported to the device where it will then be operated on in connection with the imported physics library such as the “physics model” where “3D application 302 coordinates the joint execution of physics engine 310 and rendering engine 312” and “may make method calls to both plugin 304 and graphics plugin 306 for loading a graphics model into rendering engine 312 and a corresponding physics model into physics engine 310” and “physics engine 310 may read from IMC buffers 314-316 to create a physics model corresponding to the graphics model in graphics plugin 306. Additional information related to the physics model, such as parameters, may be obtained from 3D application 302 by plugin 304. Next, physics engine 310 may perform a series of physics simulation calculations that update the physics model. For example, physics engine 310 may calculate vertex positions and velocities based on a set of forces acting on objects in the physics model” such that here simulating the 3D object is done using the simulation API, and both engines with the physics engine accessing the physics library imported so that “physics engine 310 may perform a series of physics simulation calculations that update the physics model. For example, physics engine 310 may calculate vertex positions and velocities based on a set of forces acting on objects in the physics model. Plugin 304 may then update IMC buffers 314-316 with new vertex positions, velocities, and/or other data”);
rendering the 3D object wholly in-browser in the client’s browser (see Kokkevis, paragraphs 0052-0058 where of course the purpose of such simulation is to view a rendering of the simulated object in the client’s browser as where “Rendering engine 312 may then pass the updated graphics model to GPU 320 for rendering” where this is all occurring in the client’s browser which is executing the application) , and, for each subsequent frame (see Kokkevis, paragraphs 0052-0058 as explained above where this process updates continuously and for each subsequent frame as “Graphics rendering and animation may continue to be provided by rendering engine 312 and physics engine 310 during execution of 3D application 302. For example, physics engine 310 may continue to update the graphics model as long as forces are felt by objects in the graphics model. Additional objects and/or forces may also be introduced into the graphics model and/or physics model by 3D application 302. Similarly, rendering engine 312 may render the graphics model at a frame rate specified by 3D application 302 and/or supported by GPU 320. As a result, physics engine 310 and rendering engine 312 may run at different frequencies. For example, physics engine 310 may run four times faster than rendering engine 312. As a result, the graphics model may be rendered once by rendering engine 312 for every four updates to the graphics model made by physics engine 310”):
updating the in-browser rendered 3D object by using the physics engine in-browser to generate position and rotation data for the 3D object and sending that position and rotation data to the in-browser rendering engine (see Kokkevis, paragraphs 0052-0058 teaching “Graphics rendering and animation may continue to be provided by rendering engine 312 and physics engine 310 during execution of 3D application 302. For example, physics engine 310 may continue to update the graphics model as long as forces are felt by objects in the graphics model. Additional objects and/or forces may also be introduced into the graphics model and/or physics model by 3D application 302” and such updating of the rendered objects to feel “forces” is in line with the physics model and simulation provided by the engine where “To animate the graphics model, physics engine 310 may read from IMC buffers 314-316 to create a physics model corresponding to the graphics model in graphics plugin 306. Additional information related to the physics model, such as parameters, may be obtained from 3D application 302 by plugin 304. Next, physics engine 310 may perform a series of physics simulation calculations that update the physics model. For example, physics engine 310 may calculate vertex positions and velocities based on a set of forces acting on objects in the physics model. Plugin 304 may then update IMC buffers 314-316 with new vertex positions, velocities, and/or other data. Finally, the new data is read from IMC buffers 314-316 by graphics plugin 306 and used to update the graphics model” such that this also shows sending such position and rotation data describing the position and rotation of an object in the simulation to the rendering engine in order to properly update the rendered object);
the in-browser rendering engine updating wholly in-browser the rendered 3D object based on the position and rotation data received from the physics engine and displaying the updated rendered 3D object (see Kokkevis, paragraphs 0052-0058 as explained above teaching “Graphics rendering and animation may continue to be provided by rendering engine 312 and physics engine 310 during execution of 3D application 302. For example, physics engine 310 may continue to update the graphics model as long as forces are felt by objects in the graphics model. Additional objects and/or forces may also be introduced into the graphics model and/or physics model by 3D application 302” and such updating of the rendered objects to feel “forces” is in line with the physics model and simulation provided by the engine where “To animate the graphics model, physics engine 310 may read from IMC buffers 314-316 to create a physics model corresponding to the graphics model in graphics plugin 306. Additional information related to the physics model, such as parameters, may be obtained from 3D application 302 by plugin 304. Next, physics engine 310 may perform a series of physics simulation calculations that update the physics model. For example, physics engine 310 may calculate vertex positions and velocities based on a set of forces acting on objects in the physics model. Plugin 304 may then update IMC buffers 314-316 with new vertex positions, velocities, and/or other data. Finally, the new data is read from IMC buffers 314-316 by graphics plugin 306 and used to update the graphics model” such that this passing of the information to the rendering engine allows the rendering engine to update the model and for example “Rendering engine 312 may then pass the updated graphics model to GPU 320 for rendering” such that this is an updated and rendered 3D object which is rendered for display to the browser which comprises a display of course in order to view the content which is being browsed and for example note paragraph 0026 also makes clear that rendered objects are rendered for the purpose of display as the “web application may be loaded in a web browser and executed on a computing system such as a personal computer (PC), a mobile phone, a personal digital assistant (PDA), a graphing calculator, a portable media player, a global positioning system (GPS) receiver, and/or another electronic computing device” such that here of course a web browser for such use includes a display for display on such devices as computing system 102 of paragraphs 0028-0032 teaching that “web application 116 may function as an email client, document editor, media player, computer-aided design (CAD) system, and/or computer game. Web application 116 may also include dynamic user interface elements such as menus, buttons, windows, sub-windows, icons, animations, and/or other graphical objects that emulate analogous user interface elements in native applications. In other words, web application 116 may correspond to a rich Internet application (RIA)” where all of such applications thus involve use of display of such objects being generated/simulated/rendered).
Kokkevis teaches all that is required as explained above, but fails to specifically teach that the operations above are specifically performed “in-browser” as rather while Kokkevis clearly teaches the technique as browser based and locally executed such that a user would experience the operation and control the method wholly in-browser, the performance of calculations could be seen as offloaded to a plugin to the browser such that the operations are not performed “in-browser” and “wholly in-browser”. Thus Kokkevis stands as a base device upon which the claimed invention can be seen as an improvement through providing complex calculations to a user through engines operating wholly in-browser allowing for greater ease of use for a user that does not have to install a plugin while still allowing for local wholly in-browser execution, which additionally would give the benefits of in-browser or native execution such as avoiding compatibility and support issues of plugging in to different browser types as well as enhanced security given that plug-ins can represent a security risk to users.
In the same field of endeavor relating to techniques for performing complex calculations/simulations for rendering of 3D objects simulated using a browser, McKegney teaches techniques for storing an in-browser simulation application programming interface (API) linking an in-browser physics engine and an in-browser rendering engine, and a simulation computer program code, in memory (see McKegney, paragraphs 0035-0037 teaching “3D scene browser interaction editing technique 100 and the herein disclosed techniques can provide a “Web-based 3D Framework” layer to the web designer 105 1, as shown, to bridge the gap between the web design environment and the 3D capabilities of the browser and device graphics hardware, without demanding that the web designer 105 1 learn and/or write low order 3D drawing and/or rendering code” such that this framework constitutes an API that links in-browser 3D engines and a simulation computer program code for simulating how some 3D asset appears based on the interactive scene definitions and inputs where “the 3D scene browser interaction editing technique 100 can be implemented in a web application (e.g., visual editor 106 1) operating in a browser 104 1 or operating in a web agent that is accessible to the web designer 105 1 (e.g., from a user device)” and as in paragraphs 0039-0043 the “the 3D runtime engine 210 1 can be an instance of certain 3D runtime engines 210 (e.g., various versions of the engine) stored in an external storage 246” and such engines are in-browser engines as the “visual editor” is running in the “browser 104” as seen in figures 1-2B and the 3D runtime engines are operating wholly in-browser as “rendering of, and interaction with, the embedded 3D project 268 in the web page 206 is enabled, in part, by an instance of the 3D runtime engines 210 (e.g., 3D runtime engine 210 2) running in the browser or web agent 204 1” and for example “the output of the 3D asset processor 242 can be composed to enable loading by the 3D runtime engine 210 1 for rendering in the browser 104 2” such that again it is clear that operations are occurring wholly in browser); and on a web page load request issued through the browser on the client's device (see McKegney, paragraphs 0044-0051 teaching “the 3D runtime engine 210 1 comprising the web-based 3D framework 230 that operates on certain backbone entities (e.g., 3D web project data 144) loaded at initialization of the 3D runtime engine 210 1 (e.g., at page load)”): loading the in-browser simulation application programming interface (API) and the simulation computer program code onto the client's device and importing the 3D object and at least one of the physics libraries to the client's device (see McKegney, paragraphs 0044-0051 teaching “3D runtime engine 210 1 handles the loading, rendering, and execution of web-based interactive 3D scenes” and “the 3D runtime engine 210 1 can include an asset load manager 232. The asset load manager 232 can help simplify loading of 3D assets for web designers. For example, many 3D assets comprise large volumes of data (e.g., for textures, models, animations, etc.)” and “When the 3D runtime engine 210 1 is initialized with a reference to a 3D project, the 3D runtime engine 210 1 can load the 3D project data (e.g., entity data, component data, etc.) associated with the 3D project into an asset registry 238” such that here this loads the simulation API and simulation code onto the clients device and imports the 3D objects being worked on); simulating the 3D object wholly in-browser using the in-browser simulation API, the in-browser rendering engine, and the in-browser physics engine having access to the imported physics library (see McKegney, paragraph 0043 teaching wholly in-browser execution of the relevant operations on the 3D object where the engines have access to the appropriate libraries to perform their functions as where “The rendering of, and interaction with, the embedded 3D project 268 in the web page 206 is enabled, in part, by an instance of the 3D runtime engines 210 (e.g., 3D runtime engine 210 2) running in the browser” and paragraph 0042 teaching “entity data 262 represents the building blocks of a 3D project and instances correspond to objects such as rendering library objects, textures, meshes, scenes, shots, cameras, events, etc. For example, in the “Three.js” rendering library, entity objects might include THREE.Texture, THREE.Object3D, etc. Entities can contains certain persisted settings, a loading state, and a reference to an underlying rendering library object.”); rendering the 3D object wholly in-browser in the client's browser (see McKegney, paragraph 0043 again teaching wholly in-browser execution of the rendering where “The rendering of, and interaction with, the embedded 3D project 268 in the web page 206 is enabled, in part, by an instance of the 3D runtime engines 210 (e.g., 3D runtime engine 210 2) running in the browser” ), and, for each subsequent frame: updating the in-browser rendered 3D object by using the physics engine in-browser to generate position and rotation data for the 3D object and sending that position and rotation data to the in-browser rendering engine (see McKegney, paragraph 0045 teaching “3D runtime engine 210 1 operates the main update and render loops of the interactive 3D scene and processes web page events (e.g., button clicks, object picks, etc.) associated with the 3D scene. For example, the 3D runtime engine 210 1 can process events that trigger object resize, focus, blur, etc”); the in-browser rendering engine updating wholly in-browser the rendered 3D object based on the position and rotation data received from the in-browser physics engine and displaying the updated rendered 3D object (see McKegney, paragraph 0045 teaching “3D runtime engine 210 1 operates the main update and render loops of the interactive 3D scene and processes web page events (e.g., button clicks, object picks, etc.) associated with the 3D scene. For example, the 3D runtime engine 210 1 can process events that trigger object resize, focus, blur, etc” and for example note paragraph 0048 teaching “Each component in the component library can listen to predefined events (e.g., specified by the web designer 105 1 in the visual editor 106 1), such as frame updates, window resizing, viewer input, etc. The component library 236 can further include built in components (e.g., a rotate component, an explode component, a translate/rotate/scale component, an animate component, etc.) to provide common functionality without demanding that the web designer write any code” such that again here the updates take place within the browser to render an updated rendered 3D object). Thus McKegney teaches the above applicable techniques showing that performing complex operations on 3D objects may be done wholly in-browser with in-browser rendering engines which is applicable to the Kokkevis system which also seeks to render and perform complex operations on a local computer under the control of a browser.
Therefore it would have been obvious for one of ordinary skill in the art before the effective filing date of the invention to modify Kokkevis with the techniques of McKegney such that instead of performing the complex physics and rendering calculations utilizing a plugin, the operations of such engines can be done wholly in-browser as in McKegney where an in-browser 3D engine is utilized. Such a modification would be no more than applying a known technique to a known device ready for improvement to yield predictable results. Here one of ordinary skill in the art would have recognized that applying the known technique would yield the predictable result that instead of accessing a physics plugin that runs in a secure runtime environment locally as in Kokkevis the relevant libraries and rendering and physics engines would be obtained to execute at the behest of a 3D engine executing these in-browser components wholly in-browser. This would result in an improved system as it would eliminate installation needs and issues associated with plugin, eliminate a need to maintain compatibility with multiple browser plugin installation requirements and increase security by removing the known risks associated with the download and installation of plugins. Additionally, as suggested by McKegney, such web-based operation allows for “enabling a web designer to specify camera shots and interaction events of a web-based 3D scene without demanding 3D graphics programming” (see paragraph 0006) such that one of ordinary skill in the art would be motivated to modify Kokkevis to enable the same specification of physics, rendering or other simulation without demanding 3D graphics programming, instead taking advantage of such engines and API to perform such calculations.
With regard to the remaining limitations, McKegney as modified as explained above further teaches and in which the method comprises the steps of:
simulating wholly in-browser the 3D object using a multi- threading technique in the in-browser physics engine (note that multi-threading is interpreted as any technique in which multiple threads or sets of instructions or units of processing work are sharing the same processing resource; see Kokkevis, paragraphs 0052-0056 teaching such multi-threading being described where “physics engine 310 may include native code that executes directly on CPU 318 within the constraints set by the secure runtime environment. The execution of physics engine 310 on CPU 318 may thus provide 3D application 302 with real-time animation of objects within 3D application 302” and “3D application 302 coordinates the joint execution of physics engine 310 and rendering engine 312” and “specifically, 3D application 302 may make method calls to both plugin 304 and graphics plugin 306 for loading a graphics model into rendering engine 312 and a corresponding physics model into physics engine 310. 3D application 302 may also facilitate the passing of data between plugin 304 and graphics plugin 306 using shared memory 308, as described below. On the other hand, plugin 304 and graphics plugin 306 may interface directly with one another to synchronize the animation and rendering of the graphics model” where “Graphics rendering and animation may continue to be provided by rendering engine 312 and physics engine 310 during execution of 3D application 302” and “3D application 302 may request the allocation of one or more IMC buffers 314-316 in a region of shared memory 308 by an IMC runtime provided by plugin 304 and/or graphics plugin 306. Graphics plugin 306 may then load data relevant to physics simulation into IMC buffers 314-316. For example, graphics plugin 306 may copy vertex positions, normals, triangle indices, and/or transformation matrices into IMC buffers 314-316” such that here threads for the physics simulation are run concurrently with threads of the rendering engine with both sharing memory and CPU resources while the physics engine also runs at the same time as rendering engine through the coordination of the 3D Application 302 facilitating this multi-threading of the work threads of the physics engine and rendering engine; additionally as modified by McKegney to utilize the wholly in-browser technique above McKegney’s in-browser operation of engines such as the physics engine uses a multi-threading technique in which input calls to the engine cause different threads of execution to be run such that invoking some call to change rendering or physics simulation triggers such multiple threads to be run concurrently as taught in paragraphs 0070-0078 further detailing the in-browser execution taught above where “one example of web page code 512 that can be included in a web page comprising an interactive 3D scene composed according to the herein disclosed techniques. Specifically, the web page code 512 can comprise certain loader scripts 272 2 (e.g., 3D.loader.js) that can be loaded from a server extStorage.com at page load. A project load call 274 2 (e.g., 3D.load) can further load project.json to the project canvas 276 2 (e.g., 3Dcanvas) in the body element of the page. The 3D runtime engine 210 3 (e.g., 3D-runtime-0.7.6.min.js) can also be included in the web page code 512 to render and enable interaction with the 3D scene. Specifically, the 3D runtime engine 210 3 can use the built-in selector code 534 and the custom button code 532 associated with the built-in shot selectors 434 2 and the custom button 432 2, respectively, to enable web page interaction with the 3D scene 506. As an example, when the “Reset” button (e.g., custom button 432 2) is clicked, the API call 3DAPI.events.trigger (‘resetScene’) is sent to the 3D runtime engine 210 3 to invoke a certain response corresponding to the event handler component having a resetScene listener. In some cases, the web page may want to listen to an event from the 3D runtime engine 210 3 using an API command such as 3DAPI.globalEvents.on(‘MyCustomEvent’,function( ){/*action*/})” such that any calls would be made in the same way including those as in paragraphs 0064-0065 teaching “the visual editor 106 3 can comprise a 3D scene setup window 402, a 3D component specification window 404, a 3D object selection window 406, and a scene shot window 408 that can receive interactions from the web designer 105 3 to define the shots and events. More specifically, the user interface view 4A00 exemplifies the web designer 105 3 specifying a first camera shot for a 3D scene by dragging and dropping a 3D object 410 from the 3D object selection window 406 to the 3D scene setup window 402. The web designer 105 3 can then select a camera (e.g., default camera “camera1”) for the first shot from certain cameras provided by the visual editor 106 3. Certain shot and/or camera attributes can then be specified. For example, the web designer 105 3 can use a mouse to adjust the position and/or orientation of the shot (see position “1”), and/or use the sliders in the 3D component specification window 404 to adjust other camera attributes (e.g., FOV, near plane, far plane, etc.) and set parameters. The web designer 105 3 might add further shots. Parameters are mapped into a shot template 405, as shown” such that each respect function corresponds to a thread carried out by the appropriate engine) and of compiling a version for each of a plurality of thread configurations, and on start of simulation, determining the version of thread configuration that is most suited to the client computer and running that version on the client computer (note that “compiling” as broadly recited here is interpreted to refer to any process whereby some source code is translated to another programming language; thus see Kokkevis, paragraphs 0052-0056 teaching “Physics engine 310 may be provided by 3D application 302 (e.g., downloaded over the Internet) and validated prior to execution in plugin 304” and “may include native code that executes directly on CPU 318 within the constraints set by the secure runtime environment” and “plugin 306 may then load data relevant to physics simulation into IMC buffers 314-316. For example, graphics plugin 306 may copy vertex positions, normals, triangle indices, and/or transformation matrices into IMC buffers 314-316” and “To animate the graphics model, physics engine 310 may read from IMC buffers 314-316 to create a physics model corresponding to the graphics model in graphics plugin 306” and “Plugin 304 may then update IMC buffers 314-316 with new vertex positions, velocities, and/or other data. Finally, the new data is read from IMC buffers 314-316 by graphics plugin 306 and used to update the graphics model. Rendering engine 312 may then pass the updated graphics model to GPU 320 for rendering” and “rendering engine 312 may render the graphics model at a frame rate specified by 3D application 302 and/or supported by GPU 320. As a result, physics engine 310 and rendering engine 312 may run at different frequencies. For example, physics engine 310 may run four times faster than rendering engine 312. As a result, the graphics model may be rendered once by rendering engine 312 for every four updates to the graphics model made by physics engine 310” such that here compiling for each of a plurality of thread configurations occurs when the 3D application 302 has commands translated to the native code module by the respective plugin to cause the plug to for example “load data relevant to physics simulation into IMC buffers 314-316” such that this is a plurality of thread configurations being compiled as the original command from the 3D application 302 is performed by the plugin instead and the thread version that is most suited to the client computer is chosen and run for that computer such as for example shown where “rendering engine 312 may render the graphics model at a frame rate specified by 3D application 302 and/or supported by GPU 320. As a result, physics engine 310 and rendering engine 312 may run at different frequencies. For example, physics engine 310 may run four times faster than rendering engine 312. As a result, the graphics model may be rendered once by rendering engine 312 for every four updates to the graphics model made by physics engine 310” such that here the system requirements are used to determine the most suitable arrangement of threads such as running certain threads on the GPU and certain threads on the CPU for rendering and physics for example; see also paragraph 0048 teaching “plugin 108 and/or native code module 118 may also include mechanisms for executing on a variety of instruction set architectures, including the use of “fat binaries” and binary translators” such that as explained above, the plugin is able to translate a variety of instructions from the 3D application 302; furthermore note that again as already modified Kokkevis in view of McKegney already teaches such a limitation in the combination above as in paragraphs 0045-0047 which specifically teach such a concept where “3D runtime engine 210 1 also contains a reference to the current renderer (e.g., browser) and device (e.g., desktop computer, tablet, smart phone, etc.) capabilities” and “when the 3D runtime engine 210 1 invokes the loading of the 3D project data for a given embedded 3D project, the hardware detector 234 can detect the capabilities of the current platform and load the appropriate version of the data. For example, certain texture sizes, compression formats, and/or shader features might be selected based on the current platform detected”).
Regarding claim 2, Kokkevis as modified teaches all that is required as applied to claim 1 above and further teaches in which the step of importing the physics library to the client’s device comprises importing a subset of the available physics libraries to the client’s device (see Kokkevis as modified, with Kokkevis already at paragraphs 0052-0058 teaching “3D application 302 may make method calls to both plugin 304 and graphics plugin 306 for loading a graphics model into rendering engine 312 and a corresponding physics model into physics engine 310” and “Graphics plugin 306 may then load data relevant to physics simulation into IMC buffers 314-316” such that this “data relevant to physics simulation” is a subset of relevant (for example as opposed to the irrelevant unloaded models) physics libraries which is imported to the IMC buffer 204 as they are fetched in accordance with their use with an object and furthermore for example “To animate the graphics model, physics engine 310 may read from IMC buffers 314-316 to create a physics model corresponding to the graphics model in graphics plugin 306. Additional information related to the physics model, such as parameters, may be obtained from 3D application 302 by plugin 304. Next, physics engine 310 may perform a series of physics simulation calculations that update the physics model. For example, physics engine 310 may calculate vertex positions and velocities based on a set of forces acting on objects in the physics model. Plugin 304 may then update IMC buffers 314-316 with new vertex positions, velocities, and/or other data. Finally, the new data is read from IMC buffers 314-316 by graphics plugin 306 and used to update the graphics model” such that here again the libraries imported are a subset of the physics simulation information as only the relevant simulation information is updated and where McKegney has already been combined and shown to teach such importing of the relevant libraries when running the 3D engine wholly in-browser).
Regarding claim 3, Kokkevis as modified teaches all that is required as applied to claim 2 above and further teaches in which the method comprises the step of determining which of the physics libraries is required for simulation of the 3D object and importing only those physics libraries required for the simulation of that 3D object (see Kokkevis, paragraphs 0052-0058 where as explained above a subset of the libraries is determined corresponding to the data relevant to the physics simulation which is imported to the IMC buffers as the objects is updated in the simulation as “3D application 302 may make method calls to both plugin 304 and graphics plugin 306 for loading a graphics model into rendering engine 312 and a corresponding physics model into physics engine 310” and “Graphics plugin 306 may then load data relevant to physics simulation into IMC buffers 314-316” such that this “data relevant to physics simulation” is that required for the simulation and is the only collection of data imported and again Kokkevis has already been modified in respect to this limitation as McKegney also teaches to import the relevant libraries required for rendering and simulating the 3D object as in paragraph 0042 teaching “entity data 262 represents the building blocks of a 3D project and instances correspond to objects such as rendering library objects, textures, meshes, scenes, shots, cameras, events, etc. For example, in the “Three.js” rendering library, entity objects might include THREE.Texture, THREE.Object3D, etc. Entities can contains certain persisted settings, a loading state, and a reference to an underlying rendering library object”).
Regarding claim 4, Kokkevis as modified teaches all that is required as applied to claim 1 above and further teaches storing the 3D object and the physics libraries on a content provider’s web server (see Kokkevis, paragraph 0026 teaching “the web application may be downloaded over the Internet from a website” and paragraphs 0029-0031 teaching “computing system 102 may obtain web application 116 from one or more servers (e.g., server 1 104, server x 106) using a network connection with the server(s) and load web application 116 in web browser 110. For example, web application 116 may be downloaded from an application server over the Internet by web browser 110” and “To provide computationally intensive features to the user, a native code module 118 associated with web application 116 may be used to execute computationally intensive code on behalf of web application 116. Like web application 116, native code module 118 may be obtained from one or more servers (e.g., server 1 104, server x 106) by web browser 110. For example, web application 116 may provide a hyperlink to native code module 118 on the Internet. Web browser 110 may then download native code module 118 from the Uniform Resource Locator (URL) specified in the hyperlink” such that here the content which includes the web application 3D modeling program and associated files and the physics libraries is from the “servers” which are web servers providing content ).
Regarding claim 5, Kokkevis as modified teaches all that is required as applied to claim 1 above and further teaches the method comprises the step of storing the in-browser simulation application programming interface (API) and the simulation computer program code in a content delivery network (CDN) (see Kokkevis, paragraph 0026 teaching “the web application may be downloaded over the Internet from a website” and paragraphs 0029-0031 teaching “computing system 102 may obtain web application 116 from one or more servers (e.g., server 1 104, server x 106) using a network connection with the server(s) and load web application 116 in web browser 110. For example, web application 116 may be downloaded from an application server over the Internet by web browser 110” and “To provide computationally intensive features to the user, a native code module 118 associated with web application 116 may be used to execute computationally intensive code on behalf of web application 116. Like web application 116, native code module 118 may be obtained from one or more servers (e.g., server 1 104, server x 106) by web browser 110. For example, web application 116 may provide a hyperlink to native code module 118 on the Internet. Web browser 110 may then download native code module 118 from the Uniform Resource Locator (URL) specified in the hyperlink” and as in paragraph 0052 and figure 3 the “3D application 302 may correspond to a web application that executes in a web browser 300. In addition, 3D application 302 may provide 3D graphics rendering and animation capabilities to a user of 3D application. For example, 3D application 302 may be a 3D computer game, CAD system, and/or a scientific modeling and/or simulation application” such that again the simulation API and the simulation computer program code come from the server network providing content delivery services such that these function as a content delivery network and note that Kokkevis as modified by McKegney is compatible with such storing and also teaches such storing of the API and simulation code in a CDN as in paragraphs 0045-0049 teaching “the 3D runtime engine 210 1 comprising the web-based 3D framework 230 that operates on certain backbone entities (e.g., 3D web project data 144) loaded at initialization of the 3D runtime engine 210 1 (e.g., at page load). The 3D runtime engine 210 1 handles the loading, rendering, and execution of web-based interactive 3D scenes” and “the 3D runtime engine 210 1 can include an asset load manager 232. The asset load manager 232 can help simplify loading of 3D assets for web designers. For example, many 3D assets comprise large volumes of data (e.g., for textures, models, animations, etc.)”).
Regarding claim 6, Kokkevis teaches all that is required as applied to claim 1 above and further teaches the step of running the in-browser physics engine in a web worker (note that a web worker can be recognized as a term of art that corresponds to some javascript element that runs in the background independently of other scripts in a manner that does not affect the page viewing performance for example and essentially may be interpreted as a worker for the web site; thus see Kokkevis, paragraphs 0052-0058 teaching “3D application 302 may be written in a web-based scripting language such as Javascript” and “Physics engine 310 may be provided by 3D application 302 (e.g., downloaded over the Internet) and validated prior to execution in plugin 304. Moreover, physics engine 310 may include native code that executes directly on CPU 318 within the constraints set by the secure runtime environment. The execution of physics engine 310 on CPU 318 may thus provide 3D application 302 with real-time animation of objects within 3D application 302” and “3D application 302 coordinates the joint execution of physics engine 310 and rendering engine 312” and “specifically, 3D application 302 may make method calls to both plugin 304 and graphics plugin 306 for loading a graphics model into rendering engine 312 and a corresponding physics model into physics engine 310. 3D application 302 may also facilitate the passing of data between plugin 304 and graphics plugin 306 using shared memory 308, as described below. On the other hand, plugin 304 and graphics plugin 306 may interface directly with one another to synchronize the animation and rendering of the graphics model” where “Graphics rendering and animation may continue to be provided by rendering engine 312 and physics engine 310 during execution of 3D application 302” such that as already combined with McKegney above the physics engine of Kokkevis is operating in-browser as in McKegney and as such is running as a web worker as the in-browser 3D operations operate as web workers as paragraphs 0057-0070 teaching “the web pages associated with the web application have been built (see step 318) and the 3D embed code included, certain 3D interface controls in one or more web pages can be connected to the embedded 3D project (see step 320). Such connections can be enabled by web page calls to an API associated with the 3D runtime engine. For example, a button in the 2D portion of the web page might trigger an action in the 3D scene when clicked (e.g., onclick=“3DAPI.events.trigger (‘action’)”). The resulting web application and 3D web pages can be tested and/or previewed by the web designer 105 3 in the web design environment 302” and “as examples, input events can include input events originating from the 3D scene as well as input events originating from the web application or from outside the web application and passed into the web application. Strictly as examples, the embed code can listen to user events (e.g., scroll events, orientation changes, screen touches, etc.) that are signaled through the web application. Such events can originate from user interface devices that are present in a desktop setting, a laptop setting, a mobile setting, etc” and as in paragraphs 0070-0078 the functions of the engine operate as a web worker with asynchronous bi-directional calls as described where “the web page code 512 can comprise certain loader scripts 272 2 (e.g., 3D.loader.js) that can be loaded from a server extStorage.com at page load. A project load call 274 2 (e.g., 3D.load) can further load project.json to the project canvas 276 2 (e.g., 3Dcanvas) in the body element of the page. The 3D runtime engine 210 3 (e.g., 3D-runtime-0.7.6.min.js) can also be included in the web page code 512 to render and enable interaction with the 3D scene. Specifically, the 3D runtime engine 210 3 can use the built-in selector code 534 and the custom button code 532 associated with the built-in shot selectors 434 2 and the custom button 432 2, respectively, to enable web page interaction with the 3D scene 506. As an example, when the “Reset” button (e.g., custom button 432 2) is clicked, the API call 3DAPI.events.trigger (‘resetScene’) is sent to the 3D runtime engine 210 3 to invoke a certain response corresponding to the event handler component having a resetScene listener. In some cases, the web page may want to listen to an event from the 3D runtime engine 210 3 using an API command such as 3DAPI.globalEvents.on(‘MyCustomEvent’,function( ){/*action*/})”).
Regarding claim 9, Kokkevis as modified teaches all that is required as applied to claim 1 above and further teaches the step of exporting the simulation as a shareable web link (note that the claim does not specify or limit the form or format of such a shareable web link and for example if a simulation is output or exported and is shareable over the web such that the web links the content to the user such as through some server network or other online network or web link; thus see Kokkevis, paragraph 0043 teaching “Output data obtained from the processed input data may be provided to web application 116 for use by web application 116” and “output data may also be provided as input data to other components associated with web application 116, such as a native application, a trusted plugin, and/or one or more servers (e.g., server 1 104, server x 106)” such that here the simulation output is exported as a shareable web link as it is exported and thus shareable to the server which provides a link through the web network storing such data, and see also paragraph 0050 teaching “Output data 210 may then be obtained from IMC buffer 206 for use by web application 116 and/or a trusted plugin 202 associated with web application 116. In particular, web application 116 and/or trusted plugin 202 may use output data 210 to perform additional tasks for a user of web application 116 or an entity associated with web application 116. For example, output data 210 may be stored in a file that is provided to the user, plotted in a chart or graph, uploaded to a database for a distributed computing cluster, and/or used to modify the execution of other applications” such that the output is able to be exported to a variety of outside sources such that they are exported as shareable as the ability to access such output from files for example makes them shareable and for example at least particularly upload to a database also teaches export of the output simulation as a shareable web link as accessibility of such simulation through the web and server database linking the user to the output file; note that while not necessary to be relied upon, but could be in the future, Kokkevis clearly teaches exporting the outputs of the 3D engine as a shareable web link which allows collaboration with other users as in paragraph 0085 teaching “The creator collaborator 625 2 might employ a cloud-based shared content storage service that might use the collaboration server 652 to store the data related to the interactive 3D scene on certain storage devices 612 (e.g., external storage 246, 3D assets 142, 3D project resource data 244, and 3D web project data 144). The collaboration server 652 can further interface with the sync server 620 to assist in managing conflicts among various active versions of the interactive 3D scene. The creator collaborator 625 2 and the user collaborator 623 2 can interact with the cloud-based shared content storage service using instances of a visual editor (e.g., visual editor 106 2 and visual editor 106 3) operating on various instances of user devices 602 (e.g., user device 602 7 and user device 602 8) and can operate within respective editing sessions (e.g., editing session 673 1, and editing session 673 2). For example, a visual editor 106 1 and a visual editor 106 2 can be operated within respective editing sessions. Such visual editors can communicate with the collaboration server 652 to invoke and execute certain operations (e.g., content uploads, content downloads, content viewing, content tracking, etc.) provided by the cloud-based shared content storage service”).
Regarding claim 10, Kokkevis teaches an in-browser robotics 3D object simulator comprising (note that robotics is not mentioned any further in the body of the claim and neither is 3D object, although at least a 3D object may be considered to be related to a rendering engine below and thus the “simulator” must be for at least a “3D object” where “robotics” is only given patentable weight in so far as the 3D object must be related to “robotics” in any way such that for example even the simulation or animation of any automated object, virtual or real, may correspond to a robotics 3D object; thus see Kokkevis, paragraphs 0052-0056 teaching an in-browser 3D object simulator in which robotic actions or robotic objects may be simulated as a 3D objects simulated with respect to other objects and physics which control movement and interaction between objects such using the in-browser “3D application 302” where “3D application 302 may correspond to a web application that executes in a web browser 300” and such simulator simulates the properties of such robotics objects as well as simulates physics related to any object in a scene as “3D application 302 may be a 3D computer game, CAD system, and/or a scientific modeling and/or simulation application” and “3D application 302 may offload graphics rendering to a graphics plugin 306 and animation to a plugin 304” where “plugin 304 includes a physics engine 310 that executes on a CPU 318”):
an in-browser rendering engine configured for operation wholly in-browser (see Kokkevis, paragraphs 0052-0056 as explained above teaching “graphics plugin 304 includes a rendering engine 312 that communicates with a graphics-processing unit (GPU) 320” and “rendering engine 312 may provide graphics hardware acceleration by performing calculations related to graphics rendering using GPU 320” such that this graphics rendering engine is configured for operation in-browser as it is a plugin to the browser such that it operates “in browser” as for example in a secure runtime environment);
an in-browser physics engine configured for operation wholly in-browser (see Kokkevis, paragraphs 0052-0056 ase explained above teaching “3D application 302 may offload graphics rendering to a graphics plugin 306 and animation to a plugin 304” where “plugin 304 includes a physics engine 310 that executes on a CPU 318” such that again as executing as a plug-in of the browser it is operating in-browser); and
an in-browser application programming interface (API) mapping the functions of the an in-browser physics engine to the an in-browser rendering engine wholly in-browser (here an API is a set of rules or code or even an arrangement of elements as an interworking architecture that allows different components to communicate through facilitating communication in some manner; see Kokkevis, paragraphs 0052-0058 teaching such application programming interface as the “3D Application 302” as configured to interact and communication with the physics and rendering engines as in figure 3 for example as “3D application 302 may correspond to a web application that executes in a web browser 300” and “3D application 302 may provide 3D graphics rendering and animation capabilities to a user of 3D application” where this may further comprise “CAD system, and/or a scientific modeling and/or simulation application” according to the particular application 302 where such arrangement as described with respect to figure 3 is a stored simulation API linking a physics engine “physics engine 310” and a rendering engine “rendering engine 312” and for example this is stored at the user device upon downloading from the internet where “physics engine 310 corresponds to a native code module that is executed within a secure runtime environment provided by plugin 304. Physics engine 310 may be provided by 3D application 302 (e.g., downloaded over the Internet) and validated prior to execution in plugin 304”).
Kokkevis teaches all of the above, but fails to teach that the in-browser experience of the user utilizes specifically an “in-browser physics” and “in-browser rendering engine” which are for operation “wholly in-browser” as rather while Kokkevis clearly teaches the technique as browser based and locally executed such that a user would experience the operation and control the method wholly in-browser, the performance of calculations could be seen as offloaded to a plugin to the browser such that the operations are not performed “in-browser” and “wholly in-browser”. Thus Kokkevis stands as a base device upon which the claimed invention can be seen as an improvement through providing complex calculations to a user through engines operating wholly in-browser allowing for greater ease of use for a user that does not have to install a plugin while still allowing for local wholly in-browser execution, which additionally would give the benefits of in-browser or native execution such as avoiding compatibility and support issues of plugging in to different browser types as well as enhanced security given that plug-ins can represent a security risk to users.
In the same field of endeavor relating to techniques for performing complex calculations/simulations for rendering of 3D objects simulated using a browser, McKegney teaches techniques for storing an in-browser simulation application programming interface (API) linking an in-browser physics engine and an in-browser rendering engine, and a simulation computer program code, in memory (see McKegney, paragraphs 0035-0037 teaching “3D scene browser interaction editing technique 100 and the herein disclosed techniques can provide a “Web-based 3D Framework” layer to the web designer 105 1, as shown, to bridge the gap between the web design environment and the 3D capabilities of the browser and device graphics hardware, without demanding that the web designer 105 1 learn and/or write low order 3D drawing and/or rendering code” such that this framework constitutes an API that links in-browser 3D engines and a simulation computer program code for simulating how some 3D asset appears based on the interactive scene definitions and inputs where “the 3D scene browser interaction editing technique 100 can be implemented in a web application (e.g., visual editor 106 1) operating in a browser 104 1 or operating in a web agent that is accessible to the web designer 105 1 (e.g., from a user device)” and as in paragraphs 0039-0043 the “the 3D runtime engine 210 1 can be an instance of certain 3D runtime engines 210 (e.g., various versions of the engine) stored in an external storage 246” and such engines are in-browser engines as the “visual editor” is running in the “browser 104” as seen in figures 1-2B and the 3D runtime engines are operating wholly in-browser as “rendering of, and interaction with, the embedded 3D project 268 in the web page 206 is enabled, in part, by an instance of the 3D runtime engines 210 (e.g., 3D runtime engine 210 2) running in the browser or web agent 204 1” and for example “the output of the 3D asset processor 242 can be composed to enable loading by the 3D runtime engine 210 1 for rendering in the browser 104 2” such that again it is clear that operations are occurring wholly in browser); and on a web page load request issued through the browser on the client's device (see McKegney, paragraphs 0044-0051 teaching “the 3D runtime engine 210 1 comprising the web-based 3D framework 230 that operates on certain backbone entities (e.g., 3D web project data 144) loaded at initialization of the 3D runtime engine 210 1 (e.g., at page load)”): loading the in-browser simulation application programming interface (API) and the simulation computer program code onto the client's device and importing the 3D object and at least one of the physics libraries to the client's device (see McKegney, paragraphs 0044-0051 teaching “3D runtime engine 210 1 handles the loading, rendering, and execution of web-based interactive 3D scenes” and “the 3D runtime engine 210 1 can include an asset load manager 232. The asset load manager 232 can help simplify loading of 3D assets for web designers. For example, many 3D assets comprise large volumes of data (e.g., for textures, models, animations, etc.)” and “When the 3D runtime engine 210 1 is initialized with a reference to a 3D project, the 3D runtime engine 210 1 can load the 3D project data (e.g., entity data, component data, etc.) associated with the 3D project into an asset registry 238” such that here this loads the simulation API and simulation code onto the clients device and imports the 3D objects being worked on); simulating the 3D object wholly in-browser using the in-browser simulation API, the in-browser rendering engine, and the in-browser physics engine having access to the imported physics library (see McKegney, paragraph 0043 teaching wholly in-browser execution of the relevant operations on the 3D object where the engines have access to the appropriate libraries to perform their functions as where “The rendering of, and interaction with, the embedded 3D project 268 in the web page 206 is enabled, in part, by an instance of the 3D runtime engines 210 (e.g., 3D runtime engine 210 2) running in the browser” and paragraph 0042 teaching “entity data 262 represents the building blocks of a 3D project and instances correspond to objects such as rendering library objects, textures, meshes, scenes, shots, cameras, events, etc. For example, in the “Three.js” rendering library, entity objects might include THREE.Texture, THREE.Object3D, etc. Entities can contains certain persisted settings, a loading state, and a reference to an underlying rendering library object.”); rendering the 3D object wholly in-browser in the client's browser (see McKegney, paragraph 0043 again teaching wholly in-browser execution of the rendering where “The rendering of, and interaction with, the embedded 3D project 268 in the web page 206 is enabled, in part, by an instance of the 3D runtime engines 210 (e.g., 3D runtime engine 210 2) running in the browser” ), and, for each subsequent frame: updating the in-browser rendered 3D object by using the physics engine in-browser to generate position and rotation data for the 3D object and sending that position and rotation data to the in-browser rendering engine (see McKegney, paragraph 0045 teaching “3D runtime engine 210 1 operates the main update and render loops of the interactive 3D scene and processes web page events (e.g., button clicks, object picks, etc.) associated with the 3D scene. For example, the 3D runtime engine 210 1 can process events that trigger object resize, focus, blur, etc”); the in-browser rendering engine updating wholly in-browser the rendered 3D object based on the position and rotation data received from the in-browser physics engine and displaying the updated rendered 3D object (see McKegney, paragraph 0045 teaching “3D runtime engine 210 1 operates the main update and render loops of the interactive 3D scene and processes web page events (e.g., button clicks, object picks, etc.) associated with the 3D scene. For example, the 3D runtime engine 210 1 can process events that trigger object resize, focus, blur, etc” and for example note paragraph 0048 teaching “Each component in the component library can listen to predefined events (e.g., specified by the web designer 105 1 in the visual editor 106 1), such as frame updates, window resizing, viewer input, etc. The component library 236 can further include built in components (e.g., a rotate component, an explode component, a translate/rotate/scale component, an animate component, etc.) to provide common functionality without demanding that the web designer write any code” such that again here the updates take place within the browser to render an updated rendered 3D object). Thus McKegney teaches the above applicable techniques showing that performing complex operations on 3D objects may be done wholly in-browser with in-browser rendering engines which is applicable to the Kokkevis system which also seeks to render and perform complex operations on a local computer under the control of a browser.
Therefore it would have been obvious for one of ordinary skill in the art before the effective filing date of the invention to modify Kokkevis with the techniques of McKegney such that instead of performing the complex physics and rendering calculations utilizing a plugin, the operations of such engines can be done wholly in-browser as in McKegney where an in-browser 3D engine is utilized. Such a modification would be no more than applying a known technique to a known device ready for improvement to yield predictable results. Here one of ordinary skill in the art would have recognized that applying the known technique would yield the predictable result that instead of accessing a physics plugin that runs in a secure runtime environment locally as in Kokkevis the relevant libraries and rendering and physics engines would be obtained to execute at the behest of a 3D engine executing these in-browser components wholly in-browser. This would result in an improved system as it would eliminate installation needs and issues associated with plugin, eliminate a need to maintain compatibility with multiple browser plugin installation requirements and increase security by removing the known risks associated with the download and installation of plugins. Additionally, as suggested by McKegney, such web-based operation allows for “enabling a web designer to specify camera shots and interaction events of a web-based 3D scene without demanding 3D graphics programming” (see paragraph 0006) such that one of ordinary skill in the art would be motivated to modify Kokkevis to enable the same specification of physics, rendering or other simulation without demanding 3D graphics programming, instead taking advantage of such engines and API to perform such calculations.
With regard to the remaining limitations, McKegney as modified as explained above further teaches simulating wholly in-browser the 3D object using a multi- threading technique in the in-browser physics engine (note that multi-threading is interpreted as any technique in which multiple threads or sets of instructions or units of processing work are sharing the same processing resource; see Kokkevis, paragraphs 0052-0056 teaching such multi-threading being described where “physics engine 310 may include native code that executes directly on CPU 318 within the constraints set by the secure runtime environment. The execution of physics engine 310 on CPU 318 may thus provide 3D application 302 with real-time animation of objects within 3D application 302” and “3D application 302 coordinates the joint execution of physics engine 310 and rendering engine 312” and “specifically, 3D application 302 may make method calls to both plugin 304 and graphics plugin 306 for loading a graphics model into rendering engine 312 and a corresponding physics model into physics engine 310. 3D application 302 may also facilitate the passing of data between plugin 304 and graphics plugin 306 using shared memory 308, as described below. On the other hand, plugin 304 and graphics plugin 306 may interface directly with one another to synchronize the animation and rendering of the graphics model” where “Graphics rendering and animation may continue to be provided by rendering engine 312 and physics engine 310 during execution of 3D application 302” and “3D application 302 may request the allocation of one or more IMC buffers 314-316 in a region of shared memory 308 by an IMC runtime provided by plugin 304 and/or graphics plugin 306. Graphics plugin 306 may then load data relevant to physics simulation into IMC buffers 314-316. For example, graphics plugin 306 may copy vertex positions, normals, triangle indices, and/or transformation matrices into IMC buffers 314-316” such that here threads for the physics simulation are run concurrently with threads of the rendering engine with both sharing memory and CPU resources while the physics engine also runs at the same time as rendering engine through the coordination of the 3D Application 302 facilitating this multi-threading of the work threads of the physics engine and rendering engine; additionally as modified by McKegney to utilize the wholly in-browser technique above McKegney’s in-browser operation of engines such as the physics engine uses a multi-threading technique in which input calls to the engine cause different threads of execution to be run such that invoking some call to change rendering or physics simulation triggers such multiple threads to be run concurrently as taught in paragraphs 0070-0078 further detailing the in-browser execution taught above where “one example of web page code 512 that can be included in a web page comprising an interactive 3D scene composed according to the herein disclosed techniques. Specifically, the web page code 512 can comprise certain loader scripts 272 2 (e.g., 3D.loader.js) that can be loaded from a server extStorage.com at page load. A project load call 274 2 (e.g., 3D.load) can further load project.json to the project canvas 276 2 (e.g., 3Dcanvas) in the body element of the page. The 3D runtime engine 210 3 (e.g., 3D-runtime-0.7.6.min.js) can also be included in the web page code 512 to render and enable interaction with the 3D scene. Specifically, the 3D runtime engine 210 3 can use the built-in selector code 534 and the custom button code 532 associated with the built-in shot selectors 434 2 and the custom button 432 2, respectively, to enable web page interaction with the 3D scene 506. As an example, when the “Reset” button (e.g., custom button 432 2) is clicked, the API call 3DAPI.events.trigger (‘resetScene’) is sent to the 3D runtime engine 210 3 to invoke a certain response corresponding to the event handler component having a resetScene listener. In some cases, the web page may want to listen to an event from the 3D runtime engine 210 3 using an API command such as 3DAPI.globalEvents.on(‘MyCustomEvent’,function( ){/*action*/})” such that any calls would be made in the same way including those as in paragraphs 0064-0065 teaching “the visual editor 106 3 can comprise a 3D scene setup window 402, a 3D component specification window 404, a 3D object selection window 406, and a scene shot window 408 that can receive interactions from the web designer 105 3 to define the shots and events. More specifically, the user interface view 4A00 exemplifies the web designer 105 3 specifying a first camera shot for a 3D scene by dragging and dropping a 3D object 410 from the 3D object selection window 406 to the 3D scene setup window 402. The web designer 105 3 can then select a camera (e.g., default camera “camera1”) for the first shot from certain cameras provided by the visual editor 106 3. Certain shot and/or camera attributes can then be specified. For example, the web designer 105 3 can use a mouse to adjust the position and/or orientation of the shot (see position “1”), and/or use the sliders in the 3D component specification window 404 to adjust other camera attributes (e.g., FOV, near plane, far plane, etc.) and set parameters. The web designer 105 3 might add further shots. Parameters are mapped into a shot template 405, as shown” such that each respect function corresponds to a thread carried out by the appropriate engine) and of compiling a version for each of a plurality of thread configurations, and on start of simulation, determining the version of thread configuration that is most suited to the client computer and running that version on the client computer (note that “compiling” as broadly recited here is interpreted to refer to any process whereby some source code is translated to another programming language; thus see Kokkevis, paragraphs 0052-0056 teaching “Physics engine 310 may be provided by 3D application 302 (e.g., downloaded over the Internet) and validated prior to execution in plugin 304” and “may include native code that executes directly on CPU 318 within the constraints set by the secure runtime environment” and “plugin 306 may then load data relevant to physics simulation into IMC buffers 314-316. For example, graphics plugin 306 may copy vertex positions, normals, triangle indices, and/or transformation matrices into IMC buffers 314-316” and “To animate the graphics model, physics engine 310 may read from IMC buffers 314-316 to create a physics model corresponding to the graphics model in graphics plugin 306” and “Plugin 304 may then update IMC buffers 314-316 with new vertex positions, velocities, and/or other data. Finally, the new data is read from IMC buffers 314-316 by graphics plugin 306 and used to update the graphics model. Rendering engine 312 may then pass the updated graphics model to GPU 320 for rendering” and “rendering engine 312 may render the graphics model at a frame rate specified by 3D application 302 and/or supported by GPU 320. As a result, physics engine 310 and rendering engine 312 may run at different frequencies. For example, physics engine 310 may run four times faster than rendering engine 312. As a result, the graphics model may be rendered once by rendering engine 312 for every four updates to the graphics model made by physics engine 310” such that here compiling for each of a plurality of thread configurations occurs when the 3D application 302 has commands translated to the native code module by the respective plugin to cause the plug to for example “load data relevant to physics simulation into IMC buffers 314-316” such that this is a plurality of thread configurations being compiled as the original command from the 3D application 302 is performed by the plugin instead and the thread version that is most suited to the client computer is chosen and run for that computer such as for example shown where “rendering engine 312 may render the graphics model at a frame rate specified by 3D application 302 and/or supported by GPU 320. As a result, physics engine 310 and rendering engine 312 may run at different frequencies. For example, physics engine 310 may run four times faster than rendering engine 312. As a result, the graphics model may be rendered once by rendering engine 312 for every four updates to the graphics model made by physics engine 310” such that here the system requirements are used to determine the most suitable arrangement of threads such as running certain threads on the GPU and certain threads on the CPU for rendering and physics for example; see also paragraph 0048 teaching “plugin 108 and/or native code module 118 may also include mechanisms for executing on a variety of instruction set architectures, including the use of “fat binaries” and binary translators” such that as explained above, the plugin is able to translate a variety of instructions from the 3D application 302; furthermore note that again as already modified Kokkevis in view of McKegney already teaches such a limitation in the combination above as in paragraphs 0045-0047 which specifically teach such a concept where “3D runtime engine 210 1 also contains a reference to the current renderer (e.g., browser) and device (e.g., desktop computer, tablet, smart phone, etc.) capabilities” and “when the 3D runtime engine 210 1 invokes the loading of the 3D project data for a given embedded 3D project, the hardware detector 234 can detect the capabilities of the current platform and load the appropriate version of the data. For example, certain texture sizes, compression formats, and/or shader features might be selected based on the current platform detected”).
Regarding claim 14, Kokkevis as modified teaches all that is required as applied to claim 10 above and further teaches the in-browser rendering engine comprises a web graphics library (WebGL) (see Kokkevis as modified where Kokkevis teaches the rendering engine comprises a WebGL library as in paragraph 0050 teaching “a DefaultRenderer component can be included to handle the lifecycle of the default renderer of the web-based 3D framework 230 (e.g., THREE.WebGLRenderer)” and paragraph 0061 teaching “the 3D content developer 305 can build the 3D application using a low order drawing API such as WebGL” such that here it is clear WebGL is used as the rendering engine in an example which includes its libaries).
Regarding claim 15, Kokkevis as modified teaches all that is required as applied to claim 10 above and further teaches the in-browser rendering engine comprises Three.js (see Kokkevis as modified by McKegney with McKegney already teaching in the combination above the use of a browser rendering engine that comprises Three.js as in paragraph 0035 teaching “the “Rendering Library” layer (e.g., Three.js, Babylon.js, etc.)” and paragraph 0042 teaching “entity data 262 represents the building blocks of a 3D project and instances correspond to objects such as rendering library objects, textures, meshes, scenes, shots, cameras, events, etc. For example, in the “Three.js” rendering library, entity objects might include THREE.Texture, THREE.Object3D, etc”).
Regarding claim 17, Kokkevis teaches all that is required as applied to claim 10 and further teaches in which the in-browser physics engine is configured to operate wholly in-browser in a web worker (note that a web worker can be recognized as a term of art that corresponds to some javascript element that runs in the background independently of other scripts in a manner that does not affect the page viewing performance for example and essentially may be interpreted as a worker for the web site; thus see Kokkevis, paragraphs 0052-0058 teaching “3D application 302 may be written in a web-based scripting language such as Javascript” and “Physics engine 310 may be provided by 3D application 302 (e.g., downloaded over the Internet) and validated prior to execution in plugin 304. Moreover, physics engine 310 may include native code that executes directly on CPU 318 within the constraints set by the secure runtime environment. The execution of physics engine 310 on CPU 318 may thus provide 3D application 302 with real-time animation of objects within 3D application 302” and “3D application 302 coordinates the joint execution of physics engine 310 and rendering engine 312” and “specifically, 3D application 302 may make method calls to both plugin 304 and graphics plugin 306 for loading a graphics model into rendering engine 312 and a corresponding physics model into physics engine 310. 3D application 302 may also facilitate the passing of data between plugin 304 and graphics plugin 306 using shared memory 308, as described below. On the other hand, plugin 304 and graphics plugin 306 may interface directly with one another to synchronize the animation and rendering of the graphics model” where “Graphics rendering and animation may continue to be provided by rendering engine 312 and physics engine 310 during execution of 3D application 302” such that as already combined with McKegney above the physics engine of Kokkevis is operating in-browser as in McKegney and as such is running as a web worker as the in-browser 3D operations operate as web workers as paragraphs 0057-0070 teaching “the web pages associated with the web application have been built (see step 318) and the 3D embed code included, certain 3D interface controls in one or more web pages can be connected to the embedded 3D project (see step 320). Such connections can be enabled by web page calls to an API associated with the 3D runtime engine. For example, a button in the 2D portion of the web page might trigger an action in the 3D scene when clicked (e.g., onclick=“3DAPI.events.trigger (‘action’)”). The resulting web application and 3D web pages can be tested and/or previewed by the web designer 105 3 in the web design environment 302” and “as examples, input events can include input events originating from the 3D scene as well as input events originating from the web application or from outside the web application and passed into the web application. Strictly as examples, the embed code can listen to user events (e.g., scroll events, orientation changes, screen touches, etc.) that are signaled through the web application. Such events can originate from user interface devices that are present in a desktop setting, a laptop setting, a mobile setting, etc” and as in paragraphs 0070-0078 the functions of the engine operate as a web worker with asynchronous bi-directional calls as described where “the web page code 512 can comprise certain loader scripts 272 2 (e.g., 3D.loader.js) that can be loaded from a server extStorage.com at page load. A project load call 274 2 (e.g., 3D.load) can further load project.json to the project canvas 276 2 (e.g., 3Dcanvas) in the body element of the page. The 3D runtime engine 210 3 (e.g., 3D-runtime-0.7.6.min.js) can also be included in the web page code 512 to render and enable interaction with the 3D scene. Specifically, the 3D runtime engine 210 3 can use the built-in selector code 534 and the custom button code 532 associated with the built-in shot selectors 434 2 and the custom button 432 2, respectively, to enable web page interaction with the 3D scene 506. As an example, when the “Reset” button (e.g., custom button 432 2) is clicked, the API call 3DAPI.events.trigger (‘resetScene’) is sent to the 3D runtime engine 210 3 to invoke a certain response corresponding to the event handler component having a resetScene listener. In some cases, the web page may want to listen to an event from the 3D runtime engine 210 3 using an API command such as 3DAPI.globalEvents.on(‘MyCustomEvent’,function( ){/*action*/})”).
Regarding claim 18, Kokkevis as modified teaches all that is required as applied to claim 10 and further teaches in which there is provided an accessible memory and a plurality of 3D objects stored in the accessible memory (see Kokkevis, paragraphs 0049-0058 and figures 2 and 3 teaching multiple examples of an accessible memory and a plurality of 3D objects stored in the accessible memory as “FIG. 2 shows the flow of data through native code module 118. As shown in FIG. 2, input data 208 is obtained by native code module 118 from a first inter-module communication (IMC) buffer 204. As described above, input data 208 may be specified by web application 116, a user, and/or a native application. Input data 208 may also be obtained from a variety of sources for placement in IMC buffer 204, including a host on a network, a disk, an input device (e.g., camera, microphone, etc.), and/or a hardware device (e.g., sound card, video card, etc.)” such that here buffers are accessible memory which store a plurality of 3D objects in order to process them, and the data input to such buffers is the “input data 208” which may be from a variety of accessible memories such as “sources for placement in IMC buffer 204, including a host on a network, a disk” before being stored on the accessible local memory IMC buffer for example; further note “execution of physics engine 310 on CPU 318 may thus provide 3D application 302 with real-time animation of objects within 3D application 302” such that this shows that of course multiple objects are loaded in order to be animated and simulated such as those loaded when “3D application 302 may make method calls to both plugin 304 and graphics plugin 306 for loading a graphics model into rendering engine 312 and a corresponding physics model into physics engine 310” and as already taught by McKegney in combination with Kokkevis such an accessible memory and 3D objects stored therein are as in paragraph 0040 teaching “the 3D assets 142 can be processed by a 3D asset processor 242 to convert 3D assets (e.g., 3D models, textures, audio, video, etc.) to a web-friendly object serialization format (e.g., JSON, XML, etc.) for use by the visual editor 106 1. Specifically, the output of the 3D asset processor 242 can be composed to enable loading by the 3D runtime engine 210 1 for rendering in the browser 104 2. For example, the 3D asset processor 242 can receive a 3D model, image, audio, video or zip file as input, and generate an array of JSON entities (e.g., 3D asset JSON 252) and resource files (e.g., object mesh data 254, object texture data 256, animation data, audio, video, etc.) that can be stored as 3D project resource data 244. In some cases, the 3D asset processor 242 can be a “node.js” application that converts various formats of 3D assets (see Table 1) to a format that the 3D runtime engine 210 1 can interpret”).
Regarding claims 19-20, the instant claim is directed toward a computer program product including a non-transitory medium which performs functions which correspond to the functions as in claim 1 above (note that method of claim 1 is computer implemented and thus as recited includes a computer program product in the form of the computer so programmed in connection with the memory and modules so configured; see Kokkevis, paragraphs 0023-0024, teaching “data structures and code described in this detailed description are typically stored on a computer-readable storage medium, which may be any device or medium that can store code and/or data for use by a computer system. The computer-readable storage medium includes, but is not limited to, volatile memory, non-volatile memory, magnetic and optical storage devices such as disk drives, magnetic tape, CDs (compact discs), DVDs (digital versatile discs or digital video discs), or other media capable of storing computer-readable media now known or later developed”). Essentially claim 19 is a broader version of claim 1 such that the more specific claim 1 limitations then also anticipate the broader version of these limitations. Claim 20 is then coextensive with claim 1 such that it is also rejected on the same grounds as claim 1. Thus the limitations of claims 19-20 correspond to the limitations of claim 1; therefore they are rejected on the same grounds as claim 1 above using a combination of Kokkevis and McKegney as explained above.
Claim(s) 11-12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kokkevis as modified by McKegney as applied to claim 10 above, and further in view of Murphy-Chutorian et al3 (“Murphy”) .
Regarding claim 11, Kokkevis as modified teaches all that is required as applied to claim 10 but fails to specifically teach in which the physics engine is a port of a C++ physics engine (note that a “port” is interpreted as some software module or the like which has been specifically adapted to the purpose of the system it is being ported to and thus a special version of a physics engine designed for use with the system is a physics engine port which must be C++ based). Kokkevis does teach that the native application being run as a plugin for physics and rendering calculations are those “written in a combination of general purpose programming languages such as C or C++ and low-level languages such as assembly language” (see paragraph 0003) and “code execution in the applications may further be optimized by writing the applications in a combination of general-purpose programming languages (e.g., C, C++, etc.) and assembly language, as well as utilizing libraries that provide hardware acceleration (e.g., graphics hardware acceleration) to the applications”. Furthermore, while McKegney teaches to perform complex 3D rendering and transform operations wholly in-browser that would be done by a native program previously, it is not specified that such programs are necessarily ports of C++ physics programs such as the native programs utilized in Kokkevis. Thus Kokkevis as modified stands as a base device upon which the claimed invention can be seen as an improvement through not only providing a physics engine that operates wholly in browser as well as a 3D rendering engine that operates wholly in browser but also would offer the more complex operations enabled by a C++ based native application for physics where such an improvement would be the ability to not only utilize capabilities already designed into C++ applications but to also reduce development time from writing such programs from scratch instead of porting them.
In the same field of endeavor relating to performing complex 3D rendering of 3D objects in a web browser (see Murphy, paragraph 0018 teaching “providing augmented reality (AR) in a web browser. More specifically, a system utilizes various web technologies to achieve simultaneous localization and mapping (SLAM) and six degrees of freedom (6DoF) markerless tracking. The system uses these techniques to execute an AR web application in a web browser. Implementations achieve AR in the web browser without need to customize the browser, and without the need to install a native application. Implementations enable the AR web application to work with existing web standards to provide fast AR in current and future web browsers”), Murphy teaches that it is known to port native applications written in C++ to web-based ports of these native applications to take advantage of such pre-existing libraries (see Murphy, paragraphs 0060-0065 teaching “the system performs operations to provide AR in a browser based at least in part on a predetermined subset of JavaScript transpiled or compiled from source code of another programming language that has a similar or different level of abstraction” where for example “in some implementations, a source-to-source compiler runs as a back end to a low-level programming language (LLVM) compiler to output the predetermined set JavaScript. An example source-to-source transpiler or compiler or may include Emscripten or another suitable source-to-source compiler” and “A compiler or transpiler such as Emscripten, for example, may consume the output of a bitcode generator/compiler front end, and compile or transpile it to the predetermined subset of JavaScript for yet faster speed for the purpose of markerless 6DOF tracking” such that here it can be seen that the ported code is ported to the web-browser based Javascript to implement the complex processes). Thus Murphy provides a known technique applicable to the base system of Kokkevis as modified.
Therefore it would have been obvious for one of ordinary skill in the art before the effective filing date of the invention to modify Kokkevis as modified with the known teaches of Murphy as doing so would be no more than application of a known technique to a base system ready for improvement that yields predictable results and results in an improved system. The predictable result of the combination would be that the native C++ physics libraries/engines used in Kokkevis would be ported using Emscripten or the like to run native program code on a web browser in the manner of Murphy where these would be run wholly in-browser as in McKegney and Murphy and the in-browser engines would then be operating wholly in browser as in McKegney and Murphy using ported C++ physics code as in Kokkevis. This would result in an improved system as it would be ensured that the in-browser physics engine used in Kokkevis as modified has the capabilities of native C++ programs that already exist and can be ported and would allow the system to take advantage of programs that have already been written in C++ saving the time of writing such programs from scratch. Furthermore it should be noted that Murphy provides powerful evidence that the concept of in-browser performance of specialized previously natively executed applications is well-known as for example the fact that emscripten is already known to exist for exactly the purpose of porting C++ to in-browser code such as Javascript.
Regarding claim 12, Kokkevis as modified teaches all that is required as applied to claim 10 above but fails to teach the in-browser physics engine is an emscripten port of a C++ physics engine. Kokkevis does teach that the native application being run as a plugin for physics and rendering calculations are those “written in a combination of general purpose programming languages such as C or C++ and low-level languages such as assembly language” (see paragraph 0003) and “code execution in the applications may further be optimized by writing the applications in a combination of general-purpose programming languages (e.g., C, C++, etc.) and assembly language, as well as utilizing libraries that provide hardware acceleration (e.g., graphics hardware acceleration) to the applications”. Furthermore, while McKegney teaches to perform complex 3D rendering and transform operations wholly in-browser that would be done by a native program previously, it is not specified that such programs are necessarily ports of C++ physics programs such as the native programs utilized in Kokkevis. Thus Kokkevis as modified stands as a base device upon which the claimed invention can be seen as an improvement through not only providing a physics engine that operates wholly in browser as well as a 3D rendering engine that operates wholly in browser but also would offer the more complex operations enabled by a C++ based native application for physics where such an improvement would be the ability to not only utilize capabilities already designed into C++ applications but to also reduce development time from writing such programs from scratch instead of porting them.
In the same field of endeavor relating to performing complex 3D rendering of 3D objects in a web browser (see Murphy, paragraph 0018 teaching “providing augmented reality (AR) in a web browser. More specifically, a system utilizes various web technologies to achieve simultaneous localization and mapping (SLAM) and six degrees of freedom (6DoF) markerless tracking. The system uses these techniques to execute an AR web application in a web browser. Implementations achieve AR in the web browser without need to customize the browser, and without the need to install a native application. Implementations enable the AR web application to work with existing web standards to provide fast AR in current and future web browsers”), Murphy teaches that it is known to port native applications written in C++ to web-based ports of these native applications to take advantage of such pre-existing libraries (see Murphy, paragraphs 0060-0065 teaching “the system performs operations to provide AR in a browser based at least in part on a predetermined subset of JavaScript transpiled or compiled from source code of another programming language that has a similar or different level of abstraction” where for example “in some implementations, a source-to-source compiler runs as a back end to a low-level programming language (LLVM) compiler to output the predetermined set JavaScript. An example source-to-source transpiler or compiler or may include Emscripten or another suitable source-to-source compiler” and “A compiler or transpiler such as Emscripten, for example, may consume the output of a bitcode generator/compiler front end, and compile or transpile it to the predetermined subset of JavaScript for yet faster speed for the purpose of markerless 6DOF tracking” such that here it can be seen that the ported code is ported to the web-browser based Javascript to implement the complex processes). Thus Murphy provides a known technique applicable to the base system of Kokkevis as modified.
Therefore it would have been obvious for one of ordinary skill in the art before the effective filing date of the invention to modify Kokkevis as modified with the known teaches of Murphy as doing so would be no more than application of a known technique to a base system ready for improvement that yields predictable results and results in an improved system. The predictable result of the combination would be that the native C++ physics libraries/engines used in Kokkevis would be ported using Emscripten or the like to run native program code on a web browser in the manner of Murphy where these would be run wholly in-browser as in McKegney and Murphy and the in-browser engines would then be operating wholly in browser as in McKegney and Murphy using ported C++ physics code as in Kokkevis. This would result in an improved system as it would be ensured that the in-browser physics engine used in Kokkevis as modified has the capabilities of native C++ programs that already exist and can be ported and would allow the system to take advantage of programs that have already been written in C++ saving the time of writing such programs from scratch. Furthermore it should be noted that Murphy provides powerful evidence that the concept of in-browser performance of specialized previously natively executed applications is well-known as for example the fact that emscripten is already known to exist for exactly the purpose of porting C++ to in-browser code such as Javascript.
Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kokkevis as modified as applied to claim 11 above, and further in view of Shah et al4 (“Shah”).
Regarding claim 13, Kokkevis as modified teaches all that is required as applied to claim 11 but fails to teach in which the physics engine is PhysX. Rather, Kokkevis does not specify the physics engine, rather referring to it as “native code” such that the physics engine ported is not meant to be limited to any specific physics engine such as PhysX. Note that Physx is known to one of ordinary skill in the art as referring to a specific physics engine initially created by Nvidia but which may take many forms. Regardless, Kokkevis teaches the equivalent of the Physx engine as such “native code module may be used to process data for the web application to provide functionality associated with computationally intensive tasks such as simulation, signal processing, artificial intelligence, and/or modeling” and such “native code module” is to perform “Simulation: computational fluid dynamics (CFD), rigid body dynamics, collision detection, molecular dynamics, three-dimensional (3D) animation, etc” and “native code module 118 may provide one or more of the computationally intensive features listed above to a user of web application 116 by processing input data associated with web application 116” and “physics engine 310 corresponds to a native code module that is executed within a secure runtime environment provided by plugin 304. Physics engine 310 may be provided by 3D application 302 (e.g., downloaded over the Internet) and validated prior to execution in plugin 304. Moreover, physics engine 310 may include native code that executes directly on CPU 318.” (see paragraphs 0027, and 0030-0045 and 0053). Thus Kokkevis as modified provides a base device ready for improvement through utilization of any graphics engine as providing the native code where such an improvement may comprise at least use of a known and stable physics engine such as Physx instead of designing one or choosing another option which may be more unreliable or less supported. Furthermore as already combined with McKegney and Murphy, the system can utilize any native C++ code and port it to a wholly in-browser version such that the native physics application can run wholly in-browser as explained above.
In the same field of endeavor relating to simulation of 3D objects and rendering related graphics according to physics applied to such objects in the form of movement or animation or other interaction rules, Shah teaches providing 3D object simulation such as with regard to simulated robotic 3D objects and teaches that such simulation would involve the physics engine Physx (see Shah, paragraphs 0029-0032 teaching “environment engine 210 instantiates and generates a virtual environment in which robot structure and behavior can be simulated” and “virtual environment includes computer graphics representing objects and materials within the virtual environment, and includes a set of property and interaction rules that govern characteristics of the objects within the virtual environment and interactions between the objects” and “environment engine 210 can also include a physics engine configured to generate and implement a set of property and interaction rules within the virtual environment” where this includes the “PhysX” engine where “examples of physics engines include but are not limited to the Open Dynamics engine, Bullet, PhysX, the Havok engine, and the like”). Thus Physx is taught as utilized to be adapted in a simulation and rendering program and corresponds to native code such as in Kokkevis as modified above. Thus Shah teaches known techniques applicable to the base device of Kokkevis as modified explained above such as through physics simulation of 3D objects utilizing the PhysX engine.
Therefore it would have been obvious for one of ordinary skill in the art before the effective filing date of the invention to modify Kokkevis as modified such that the native code module is one such as the PhysX engine as such modification would yield predictable results and result in an improved model where this would be ported from the C++ based version of the code to be in-browser code such as in Murphy. The result of the modification would be predictable as the PhysX engine is simply an example of native code which may be C++ ported as already taught by Kokkevis as modified such that the PhysX engine would be the native code module utilized for the physics engine and ported using known porting techniques and solutions such as emscripten.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1, 19 and 20 are provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1 of copending Application No. 18245118 (reference application). Although the claims at issue are not identical, they are not patentably distinct from each other as explained below.
Note the following table showing the independent claims with similarities rendered in bold.
Conflicting Reference Application 18245118
Pending Application 17996981
Claim 1.
A computer implemented method of simulating a three-dimensional (3D) object for simultaneous display in a browser on each of a plurality of client devices, the method comprising the initial steps of:
generating a 3D object and storing the 3D object in memory;
storing a plurality of physics libraries in memory;
storing a simulation application programming interface (API) linking a physics engine and a rendering engine, and a simulation computer program code, in memory; and
on a web page load request issued through the browser on each of the client devices:
establishing a communication link between the client device and the other client devices;
loading the simulation application programming interface (API) and the simulation computer program code onto each of the client devices;
importing the 3D object and at least one of the physics libraries to each of the client devices;
simulating the 3D object using the simulation API, the rendering engine, and the physics engine having access to the imported physics library;
rendering the 3D object in each of the client device browsers, and, for each subsequent frame:
receiving 3D object manipulation instructions from one client device;
updating the rendered 3D object by using the physics engine to generate position and rotation data for the 3D object and sending that position and rotation data to the rendering engine on each of the client devices;
the rendering engine updating the rendered 3D object based on the position and rotation data received from the physics engine and displaying the updated rendered 3D object on each of the client devices.
Claim 1.
A computer implemented method of simulating a three-dimensional (3D) object for display in a browser on a client's device, the method comprising the initial steps of:
generating a 3D object and storing the 3D object in memory;
storing a plurality of physics libraries in memory;
storing an in-browser simulation application programming interface (API) linking an in-browser physics engine and an in-browser rendering engine, and a simulation computer program code, in memory; and
on a web page load request issued through the browser on the client's device:
loading the an in-browser simulation application programming interface (API) and the simulation computer program code onto the client's device;
importing the 3D object and at least one of the physics libraries to the client's device;
simulating the 3D object wholly in-browser using the in-browser simulation API, the in-browser rendering engine, and the in-browser physics engine having access to the imported physics library;
rendering the 3D object wholly in-browser in the client's browser, and, for each subsequent frame:
updating the in-browser rendered 3D object by using the physics engine in-browser to generate position and rotation data for the 3D object and sending that position and rotation data to the in-browser rendering engine;
the in-browser rendering engine updating wholly in-browser the rendered 3D object based on the position and rotation data received from the in-browser physics engine and displaying the updated rendered 3D object, and in which the method comprises the steps of:
simulating wholly in-browser the 3D object using a multi- threading technique in the in-browser physics engine, and
of compiling a version for each of a plurality of thread configurations, and on start of simulation, determining the version of thread configuration that is most suited to the client computer and running that version on the client computer.
Thus it can be seen that the instant claim 1 is a broader version of the conflicting reference claim and thus is effectively a genus to the species recited in claim 1 of the conflicting application as well as the other conflicting independent claims 19-20 reciting similar limitations as conflicting claim 1. The other difference is specific recitation of the engines as “in-browser” and “wholly in-browser” operation of engines and the multi-threading limitations. However, Kokkevis, McKegney and Murphy’s teachings as applied above render such limitations obvious for the same reasons they render such limitations obvious in view of Kokkevis as applied above. Thus the claims are an obvious variant in view of the prior art. Again it is important to note that claims 19 and 20 are rendered obvious for the same reasons.
This is a provisional nonstatutory double patenting rejection because the patentably indistinct claims have not in fact been patented.
Response to Arguments
Applicant's arguments filed 10/07/2025 have been fully considered but they are not persuasive.
Applicant begins by arguing on page 8 of “REMARKS” filed 10/7/2025 that in general one would not combine the teachings of Kokkevis and McKegney to arrive at the claimed invention because allegedly “Kokkevis teaches away from the use of web-based applications.” Applicant points to paragraph 0008 of Kokkevis as supposed evidence of this teaching away, however, all that Kokkevis discloses is that “web applications may lack the performance capabilities” and at that time were “currently unable to implement computationally intensive features that are available in native applications.” Thus not only is Kokkevis equivocal (“may lack the performance” and “may run one or two orders of magnitude slower” for example”), but the only teaching away that could be suggested from this disclosure would be that one would be discouraged from using those “web applications” that at that time were “currently unable to implement” the computational features. McKegney however, is not used to provide teachings of techniques of web applications that run orders of magnitude slower and its teachings are not those mentioned by Kokkevis as, “currently unable to implement computationally intensive features” as instead McKegney provides the 3D runtime engine utilizing WebGL, Three.js, Babylon.js, etc that of course one of ordinary skill in the art recognizes as giving the ability to implement computationally intensive features utilizing the 3D capabilities of the browser and the device graphics hardware (see McKegney, paragraph 0035). Applicant argues that Kokkevis’ “modus operandi” is “to process outside a browser, in a native environment, using a plug-in to address those slow processing times” and that apparently this means that one would not try to address slow processing times in browser-based methodologies. The two do not logically or necessarily follow each other at all. Simply because Kokkevis chose one solution to this problem does not mean that a later person having ordinary skill in the art would not choose another solution, when viewing the prior art at that time. Furthermore, with the benefit of teachings as in McKegney, the skilled artisan at the time of invention would be able to solve the process of slow browser processing times that could not be addressed with browser based technologies at Kokkevis’ time, and would not be forced to only rely on plug-in type solutions such as in Kokkevis. Thus all that is shown is that Kokkevis stands as a base device ready for improvement, with McKegney supplying the improvement that was currently unavailable at the time of Kokkevis’ invention. Here then, neither Kokkevis or McKegney can be seen in any way to criticize, discredit, or otherwise discourage the solution claimed, as all that is criticized, discredited or otherwise discouraged would be to utilize a web application, available at the time of Kokkevis, that lacks the performance capabilities for computationally intensive calculations and that runs orders of magnitude slower. McKegney’s web-based application and 3D runtime engine do not lack performance capabilities of web-based applications such as those available at the time of Kokkevis’ invention and are able to implement computational intensive features in a wholly browser-based environment, thus Kokkevis does not teach away in any manner from these techniques nor utilizing such techniques to improve other devices and techniques.
Applicant then argues that the cited references fail to disclose or suggest the respective claim limitations as previously applied to claims 7 and 8. Applicant then points to Applicant's Specification at various locations which has support for the limitations argued. However, instead of arguing the broadest reasonable interpretation of the claim language, Applicant repeatedly argues that various limitations which are not present in the claims show that the broadest reasonable interpretation of the claim language is not met by the prior art.
For instance, Applicant argues that "the physics engine simulation operations can be divided out into multiple threads and the multiple physics engine simulation operations are processed in parallel". In response to applicant's argument that the references fail to show certain features of the invention, it is noted that the features upon which applicant relies (i.e., “physics engine simulation operations can be divided out into multiple threads and the multiple physics engine simulation operations are processed in parallel”) are not recited in the rejected claim(s). Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993). As noted by the Examiner, the broadest reasonable interpretation is that a multi-threading technique is utilized in the in-browser physics engine, and this does not mean specifically that multiple physics simulation operations are processed in parallel or that the multi-threading must only be of multiple physics simulation operations. Rather, as explained by the Examiner, if multiple threads of work are processed and at least one of these multiple threads is processed by the in-browser physics engine then the processor has simulated wholly in-browser the 3D object using a multi-threading technique in the in-browser physics engine as the processing of the multiple threads, including by the in-browser physics engine is used in the simulation of the 3D object. Thus in Kokkevis as modified, the in-browser physics calculations comprise a thread of work for simulating the 3D object and the rendering calculations comprise another thread of work and these multiple threads are processed by the same processing unit simultaneously to produce the output such that the in-browser physics engine utilizes the ability of the processor to operate on multiple threads. There is no requirement to divide out multiple physics engine simulation operation into multiple threads where then multiple physics engine simulation operations can be divided out into multiple threads.
Applicant then argues with respect to the "compiling" and asserts the examiner has used "an overly broad interpretation and contends that it is not warranted in the circumstances." Applicant argues that Kokkevis as modified as applied is not "compiling a version for each of a plurality of thread configurations within the meaning of the application in suit." Applicant then cites portions of their Specification with far greater detail than is required by the claims and asserts that the claims require “providing a plurality of compiled versions.” Related to this, Applicant argues against the Application of Kokkevis to the compiling step, arguing, “Kokkevis, as far as it is understood, does not use data from one of the IMC buffers 314-316 and disregard the data from the other IMC buffers" and that apparently this means that the claims distinguish over the prior art. In response to applicant's argument that the references fail to show certain features of the invention, it is noted that the features upon which applicant relies (i.e., “providing a plurality of compiled versions” and “a version for each typical thread configuration (1,2,4,8) is compiled”) are not recited in the rejected claim(s). Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993). Rather than requiring necessarily a plurality of compiled versions of threads or a version for each typical thread configuration, the claim much more broadly requires to compile “a version for each of a plurality of thread configurations” such that all that is required is a single version so long as it is “for each of a plurality of thread configurations.” However the claims do not describe, define, nor limit what these “a plurality of thread configurations” necessarily refer to. Thus in line with the broadest reasonable interpretation of a thread as some specific work operation or processing job, a plurality of thread configurations may refer to any plurality of work operations or processing jobs that define any given simulation of the 3D object. When Kokkevis as modified loads a given simulation to the 3D runtime engine for execution, the simulation commands/threads and rendering commands/threads are compiled in order to be executed, such that on the start of the simulation, the version of thread configuration that has been created to execute the simulation is determined as the most suited to the client computer for executing that simulation and then that version is run on the client computer. Thus Applicant’s arguments are not persuasive.
Note that if such limitations as dividing out multiple physics engine calculations so that multi-threading is with respect to performing the multiple physics engine calculations and the multiple physics engines calculations are processed in parallel, then this would distinguish over the prior art of record and necessitate further search for more fine-grained programming techniques. Similarly, additionally defining that multiple thread versions of a plurality of thread configurations are compiled and there is selection from among the multiple thread configurations would define over the prior art or record and necessitate a search for these more specific techniques. Finally, it is possible that defining the manner in which the version of thread configuration is determined to be most suited to the client computer, such as based on a number of cores of the client computer such that different compiled configurations must be chosen from based on some specific aspect of the client computer making it the most suited.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SCOTT E SONNERS whose telephone number is (571)270-7504. The examiner can normally be reached Mon-Friday 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached at (571) 272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SCOTT E SONNERS/Examiner, Art Unit 2613
/XIAO M WU/Supervisory Patent Examiner, Art Unit 2613
1 US PGPUB NO. 20100017461
2 US PGPUB No. 20170024112
3 US PGPUB No. 20200082624
4 US PGPUB No. 20200276708
Prosecution Insights